Allocating Space slow under Linux: SOLVED
|
Yep, it's due to fsync. The installer probably does something stupid like synchronizing the disk after every write. Thanks to JukiJulma in the "What is Allocating Space at the install?" thread (https://www.pathofexile.com/forum/view-thread/36934/page/2) for pointing me in the right direction.
[edit] Tracked it down further. Apologies for blaming the devs, it's actually due to a Wine bug (FlushViewOfFile ends up synchronous instead of asynchronous as it is in Windows). Patch submitted. If you want a speedy install or patch, grab "libeatmydata" http://www.flamingspork.com/projects/libeatmydata/, or apt-get install eatmydata, or emerge libeatmydata, which is a library that hijacks fsync() calls and prevents them from doing anything, then run wine like so: " Distro-specific instructions Gentoo: emerge libeatmydata; LD_PRELOAD=/usr/lib/libeatmydata.so Debian, 64 bit: apt-get install eatmydata:i386; LD_PRELOAD=/usr/lib/libeatmydata/libeatmydata.so (thanks, Drakier!) If you have another distro, tell me and I'll add it to the list. Speedy as it should be. Happy installing/updating! Last edited by FeepingCreature on Mar 8, 2013, 12:22:42 PM
|
|
|
Another very effective solution for those with ~8GB of memory or more: set WINEPREFIX to some path under /dev/shm when running the installer and copy the results to wherever you want afterward.
In fact, copying Content.ggpk to /dev/shm and making it into a symlink while the game runs (which also works with the updater, though for some odd reason the updater will complain and you'll have to run the game again after a patch) is a very nice way to run the game, and cuts down loading times immensely. Good to know the real cause of this though! Perhaps this knowledge could be used to improve the performance of the updater on all platforms. Last edited by MesostelZe on Mar 7, 2013, 7:53:55 PM
|
|
|
This is AWESOME.
If you're using PlayOnLinux, you can just add export LD_PRELOAD=/your/path/to/libeatmydata.so to the commands to exec before running the program in the Miscellaneous tab in the Configuration screen. I had to apt-get install eatmydata:i386 because my system is a 64-bit distro, but it worked great. I even moved my old Content.ggpk just to see how much faster it was, and the Allocating Space only took a couple of minutes. Very Awesome my libeatmydata.so was not in the same place as the OP, but it was easy enough to locate and adjust the settings. | |
|
Where was it for you, and what distro are you on? I'll add it to the first post.
|
|
" I'm on 64-bit debian sid, so I had to apt-get install eatmydata:i386 an it stuck it in /usr/lib/libeatmydata/libeatmydata.so | |
|
Thank you, added. (Sorry to bump but this is important)
|
|
|
Does this only help on ext3 or ext4 filesystems? I don't understand why there's a library to queue up ignore fsync() calls when it should just be a FS setting.
Also, I've heard of people using Windows 7 stuck at "Allocating Space" with empty mechanical hard drives. Is this related to them? Last edited by ionface on Mar 8, 2013, 8:24:06 AM
| |
|
You generally don't want to disable fsync globally - most apps that call it (databases, partition editors…), call it for a very good reason.
I don't know if this is a problem under Windows. It seems likely, depending on how they implement it, that it could be causing some slowdown. Probably Windows implements it differently for performance reasons? I don't actually know which Windows function Wine translates into an fsync call, but I'll try and find out. Okay, I checked and apparently it's FlushFileBuffers(), which Wine forwards to fsync(). Some enterprising spirit may want to try and replace the call to NtFlushBuffersFile in FlushFileBuffers in dlls/kernel32/file.c:789 or thereabouts in their wine tree with a return TRUE; to see if that has the same effect as libeatmydata. Or just run wineserver in gdb and break on fsync. :) [EDIT didn't work. I'll try the FlushFileBuffers thing.] Last edited by FeepingCreature on Mar 8, 2013, 8:53:55 AM
|
|
|
Okay, it's not FlushFileBuffers. libeatmydata does some stuff involving opening files in O_SYNC mode too, I'll check if wine uses that anywhere.
[edit] Apparently Path of Exile uses a read buffer size of 64 during the "Checking resources" phase (according to strace log)? Devs, maybe try increasing that by a few orders of magnitude? [edit] No use of anything SYNC/fsync related in the strace log. I wonder what else libeatmydata does. [edit] I built a logging version of libeatmydata and the function it prevents that causes the slowdowns is msync(), not fsync(). Progress! I'll try to figure out what of Wine calls that. [edit] NtFlushVirtualMemory. Trying to trace further. [edit] I don't think Wine calls this, I think PoE calls it. Devs, could you check that please? [edit] OH. FlushViewOfFile. Trying to confirm .. [edit] Confirmed! With FlushViewOfFile disabled, Allocating Space on a fresh install completes in two minutes. Last edited by FeepingCreature on Mar 8, 2013, 9:51:37 AM
|
|
|
Devs, can you please not call FlushViewOfFile when you detect you're running under Wine (as per http://www.winehq.org/pipermail/wine-devel/2008-September/069387.html?) It would make the Linux user experience a lot better.
[edit] About msync(): http://lists.freebsd.org/pipermail/freebsd-stable/2008-June/043392.html [edit] I'm trying to see if I can fix it in wine by switching msync to ASYNC mode. [edit] ASYNC mode confirmed working. Devs: nevermind, I'll try to get it fixed in Wine instead. [edit] I submitted a patch to wine-patches. Hopefully we'll see some improvement the next version. Last edited by FeepingCreature on Mar 8, 2013, 11:06:26 AM
|
|
