Residue wrote on Jan 22
nd, 2014 at 8:40pm:
Maj; mystyfi almost has it. But it's not a fragmentation problem.
This is what's going down:
The .dat files are filled with thousands and thousands of small files, and are growing with each update. The size of the .dat files is not the problem. Nor is fragmentation a problem, or at least not what it used to be.
The problem is where the small files are physically located within the .dat structure. When you load up the game, a whole shit load of these files need to be accessed, and the hard disks need to seek through the large .dat files to get at the smaller ones.
Here's an example. You login, pick a character, and zone into the harbor. Client needs to load graphics stuph. So texturefile1 is in the middle of the .dat file, so the hard disk has to physically seek there to get the file. texturefile2 is near the end of the .dat file, so the hard disk has to seek there. Texturefile3 is near the beginning of the .dat file, so the hard disk has to seek there.
I'm going to guess that this file access isn't even happening sequentially, but maybe parallel, so multiple texture files are being called for at the same time, and even sound files, which are in another .dat file entirely. Do you see where I am going with this? Now throw in memory management and paging to disk, and the result is a hard drive with read/write heads thrashing all across the platters (you'll see this as a near solid hdisk LED on your box) and the client takes forever to load into the zone, and eventually the server times out because the client is taking way too long.
Oh it's sequential enough.
go to sysinternals website, download process monitor, launch it, create a filter with just the Dndclient.exe ( you can eventually extend to awesomiumprocess.exe and turbinelauncher.exe ).
It will give you all the calls made to files, registry, DLLs and so on. The one to look at for the DAT files is FASTIO_READ.
Then you'll start crying.
Residue wrote on Jan 22
nd, 2014 at 8:40pm:
Modern hard drives with larger read ahead caches are probably not as much of a problem, but will still be slow. Even the fastest hard disks with high data transfer rates will perform poorly because of the physical seeking required to access these thousands of small files.
Running the game from an SSD drive, or even a slower lower transfer rate USB thumb drive will perform much better than a traditional hard disk, because the physical seeking of read/write heads across a platter is eliminated.
It's not just the seek time, it's the amount of data that is pre-loaded, and how they are organized in the file. ( thus requiring to go through the whole directory/sub directory structure every time you look for something and you don't already know at which offset in the file it's hiding. )
Residue wrote on Jan 22
nd, 2014 at 8:40pm:
Maj, look into seeing if the following is possible:
- Your client is just too fucking fat dude! Find out if there's a way to purge any old texture/audio/whatever the hell files aren't used anymore and get them out of there. The .dat files would somehow have to be defragmented (and shrunk) afterward.
The dat files needs to be defragmented anyway, or at least they need to be rehashed to cut down all the duplicate entries there's in them. I already gave some details that are deep enough in the files, I don't want to give more since all those that have done that have earned a permaban from the game, but with an Hex Editor it's relatively easy to look at them and get a fair idea of their structure. And there's a problem with duplicate in the file structure.
Removing these duplicate would reduce the file size, thus improving the performances. And that does not include removing the unused things ( there's probably some unused things in those files )
Residue wrote on Jan 22
nd, 2014 at 8:40pm:
- the texture data / audio / whatever files need to be consolidated together within the .dat files. I have no doubt that many files are shared among dungeons, but you have to consolidate them to match the main public areas that players would zone into, in order to resolve this specific problem.
The Texture and Audio files are quite easy to look at, since they are raw third party files... there's an unpacker tool that lets you get them all extracted. There's not that much need to consolidate there. Really the pigs are gamelogic, general and surface.
Residue wrote on Jan 22
nd, 2014 at 8:40pm:
- as a work around to my first point, perhaps the client can check where the player needs to zone into (it knows, it's displayed on the character selection screen) and pre-load / cache that zone data before logging into that zone.
use Process explorer, really you will be amazed by all the things done at DDO startup.
Residue wrote on Jan 22
nd, 2014 at 8:40pm:
This problem usually only occurs upon the first zone login. Subsequent logins are successful because the windows OS has cached the required data already after the first failed attempt, and so the login process is faster and there is no server time-out.
Because of the way the client is installed and updated/patched not everyone will have this problem. This is because no two client .dat files are the same. The .dat file size and physical locations of the many small files within will vary from client to client depending on when the client was first installed, and then when patched. And this is different for each client.
It's a bit more complex, but that's more or less what happens.
Residue wrote on Jan 22
nd, 2014 at 8:40pm:
I'll give a thumbs up to your engineering team for adding in some defragmentation. I think this happened somewhere around u14, and since then I have periodically analyzed internal .dat file fragmentation, and it's been looking much better. Any one running the game on an original install prior to u14 should re-install. But fragmentation really isn't the problem here, at least if your client install was post u14.
There's still fragmentation, even if it's not what is the usual acceptation of the term.
If you looked at the structure you know that there's duplicate directory entries as well as sub directories that are almost the same, and last duplicate file entries.
It's going to depends on the file, some are clean ( from what I've seen sounds seems to be one of the cleanest, with gamelogic being the most dirty )