majmalphunktion wrote on Jan 21^st, 2014 at 9:01pm:

What Maj said.  It's logins, meaning -account logins-.  When you first launch DDO and login with your username and password? That's 1 login.

Had the nasty crash bug that finally got fixed when entering Sands?  That's a reload, and a fresh login.  For a good 6-9 months the Oracle's "login" count was double to triple what it should have been, so compared to some people's ideas of 'players', the number is way high.

On the other hand, play 12 different characters on one account for the next 5 hours?  That's still -1- login, during the hour you initially logged in, and nothing more.  [You people I'm pointing at know who you are.]  Check your LFM's 4 times a day?  That's 4 logins.

So how does it all fit?  Match things to your in-game experience.  If you frequently play more than 1 hour, then you could consider yourself underrepresented, unless you take into account the guys that are crashing and logging in N times.  In practice, it appears to balance out pretty well.   The numbers are what they are, but if you're looking to log in in the busiest time of day it pays to pay attention to the graphs.

One of the things the Oracle finds interesting is the relatively small changes that occurred as these bugs were fixed.  I know firsthand issues were addressed, as it was pretty bad at one point, but it got better.  I also know better than most that the game traffic didn't change much afterwards.   That suggests that the people with problems were a small albeit vocal minority and that most people can just play the game.

The Oracle wrote on Jan 22^nd, 2014 at 1:40am:



just like in real life?

majmalphunktion wrote on Jan 21^st, 2014 at 9:01pm:


I wonder if this is the answer to the G-land mystery. 

The Oracle wrote on Jan 22^nd, 2014 at 1:40am:

Or that people who got the bug didn't bother to log back in the second time. In most of 2013, I was in that mode. Unless I'd set up a guild run and scheduled people to come on and play, if it died the first time, I just played something else.

NOTpopejubal wrote on Jan 22^nd, 2014 at 9:03am:

Or may be the G-Land mystery is tied to the failed first login bug? A malformed authentication token gets routed to Ghallanda, where it counts as a login but doesn't make it back to the client to let them into the world they tried to log into?

Flav wrote on Jan 22^nd, 2014 at 3:36am:


I figure that maj is the only sob at turbine that gives 2 shits about ddo. Some of the older dev's did, but they are long gone, replaced with useless fucks like ole "not good at his job, but hired for min wage so..."

If they want to ban my account now, so be it. I have already violated the eula a thousand times over, whats a few more times?

btw maj, you should nominate me for lamma/mournlands2. I would NOT agree to an NDA and would NOT agree to your Eula, but you would get more value from just having me instead of a hundred mouth breathers from the motherforums.

mystafyi wrote on Jan 21^st, 2014 at 7:31pm:


Maj;  mystyfi almost has it.  But it's not a fragmentation problem.

This is what's going down:

The .dat files are filled with thousands and thousands of small files, and are growing with each update.  The size of the .dat files is not the problem.  Nor is fragmentation a problem, or at least not what it used to be.

The problem is where the small files are physically located within the .dat structure.  When you load up the game, a whole shit load of these files need to be accessed, and the hard disks need to seek through the large .dat files to get at the smaller ones.

Here's an example.  You login, pick a character, and zone into the harbor.  Client needs to load graphics stuph.  So texturefile1 is in the middle of the .dat file, so the hard disk has to physically seek there to get the file.  texturefile2 is near the end of the .dat file, so the hard disk has to seek there.  Texturefile3 is near the beginning of the .dat file, so the hard disk has to seek there.

I'm going to guess that this file access isn't even happening sequentially, but maybe parallel, so multiple texture files are  being called for at the same time, and even sound files, which are in another .dat file entirely.  Do you see where I am going with this?  Now throw in memory management and paging to disk, and the result is a hard drive with read/write heads thrashing all across the platters (you'll see this as a near solid hdisk LED on your box) and the client takes forever to load into the zone, and eventually the server times out because the client is taking way too long.

Modern hard drives with larger read ahead caches are probably not as much of a problem, but will still be slow.  Even the fastest hard disks with high data transfer rates will perform poorly because of the physical seeking required to access these thousands of small files.

Running the game from an SSD drive, or even a slower lower transfer rate USB thumb drive will perform much better than a traditional hard disk, because the physical seeking of read/write heads across a platter is eliminated.

Maj, look into seeing if the following is possible:

- Your client is just too fucking fat dude!  Find out if there's a way to purge any old texture/audio/whatever the hell files aren't used anymore and get them out of there.  The .dat files would somehow have to be defragmented (and shrunk) afterward.

- the texture data / audio / whatever files need to be consolidated together within the .dat files.  I have no doubt that many files are shared among dungeons, but you have to consolidate them to match the main public areas that players would zone into, in order to resolve this specific problem.

- as a work around to my first point, perhaps the client can check where the player needs to zone into (it knows, it's displayed on the character selection screen) and pre-load / cache that zone data before logging into that zone.

This problem usually only occurs upon the first zone login.  Subsequent logins  are successful because the windows OS has cached the required data already after the first failed attempt, and so the login process is faster and there is no server time-out.

Because of the way the client is installed and updated/patched not everyone will have this problem.  This is because no two client .dat files are the same.  The .dat file size and physical locations of the many small files within will vary from client to client depending on when the client was first installed, and then when patched.  And this is different for each client.

I'll give a thumbs up to your engineering team for adding in some defragmentation.  I think this happened somewhere around u14, and since then I have periodically analyzed internal .dat file fragmentation, and it's been looking much better.  Any one running the game on an original install prior to u14 should re-install.  But fragmentation really isn't the problem here, at least if your client install was post u14.

There are workarounds to this problem, sure, like running from ssd/usb flash, or parking your toon in a small low populated zone (pick any bar) or ddopre-loader, but someone made a point with regards to players (new or otherwise) getting frustrated over this issue and not knowing the workarounds.

I hope this info helps.

Res

edit:  um, so, um, what were we talking about again?

Residue wrote on Jan 22^nd, 2014 at 8:40pm:


Oh it's sequential enough.
go to sysinternals website, download process monitor, launch it, create a filter with just the Dndclient.exe ( you can eventually extend to awesomiumprocess.exe and turbinelauncher.exe ).
It will give you all the calls made to files, registry, DLLs and so on. The one to look at for the DAT files is FASTIO_READ.
Then you'll start crying.

Residue wrote on Jan 22^nd, 2014 at 8:40pm:


It's not just the seek time, it's the amount of data that is pre-loaded, and how they are organized in the file. ( thus requiring to go through the whole directory/sub directory structure every time you look for something and you don't already know at which offset in the file it's hiding. )

Residue wrote on Jan 22^nd, 2014 at 8:40pm:


The dat files needs to be defragmented anyway, or at least they need to be rehashed to cut down all the duplicate entries there's in them. I already gave some details that are deep enough in the files, I don't want to give more since all those that have done that have earned a permaban from the game, but with an Hex Editor it's relatively easy to look at them and get a fair idea of their structure. And there's a problem with duplicate in the file structure.
Removing these duplicate would reduce the file size, thus improving the performances. And that does not include removing the unused things ( there's probably some unused things in those files )

Residue wrote on Jan 22^nd, 2014 at 8:40pm:


The Texture and Audio files are quite easy to look at, since they are raw third party files... there's an unpacker tool that lets you get them all extracted. There's not that much need to consolidate there. Really the pigs are gamelogic, general  and surface.

Residue wrote on Jan 22^nd, 2014 at 8:40pm:


use Process explorer, really you will be amazed by all the things done at DDO startup.

Residue wrote on Jan 22^nd, 2014 at 8:40pm:


It's a bit more complex, but that's more or less what happens.

Residue wrote on Jan 22^nd, 2014 at 8:40pm:


There's still fragmentation, even if it's not what is the usual acceptation of the term.
If you looked at the structure you know that there's duplicate directory entries as well as sub directories that are almost the same, and last duplicate file entries.
It's going to depends on the file, some are clean ( from what I've seen sounds seems to be one of the cleanest, with gamelogic being the most dirty )

Hint : It's basically the same structure as the Asheron's Call DAT files...  There's a few differences, but they are minor.

Edit : that's how I went to look at th'e files, I googled AC Dat File Format and then started to look at them with an Hex Editor. ( and downloaded the Datunpacker from the Lotrowhatever website )

I'm wondering what the advantage to using a monolithic dat file is versus just dropping all of the files into a normal directory structure.  Is it because they're afraid it would make hacking much easier?  If so, it's a failed method; GW2 uses the same dat file system and it's consistently botted by gold sellers.

usb flash drive
pylotro

Durk wrote on Jan 23^rd, 2014 at 11:22am:


It sort of looks that way, but it's not what I mean.  I was trying to keep the post as basic as possible, so bear with me.

A fragmented file means that the file itself is spread out throughout different areas of a disk, instead of being written sequentially and in a contiguous area of the disk.  So having a file with all it's parts grouped together will be transferred much faster because the hard disk doesn't have to seek to other parts of a disk to retrieve the file.  For us biologicals, seeking may seem quick, but in hard disk technology, it's a freaking eternity to get from one part of the disk to another.  And this is why ssd/flash technology excels, because there are no moving parts.

What I am implying is that even though the files within the .dat files are not fragmented (as flav pointed out, they are, but just not nearly what it used to be prior to u14) the files that the client needs within the .dat are not consolidated, or close together so to speak, but spread out across the .dat file.

There may not be fragmentation in file1, file2, and file3.  But if file1 is in the middle of the disk, file2 is at the end, and file3 is at the beginning, the symptom will still appear as though there is fragmentation because the hard disk has to seek to these file locations.

This is where consolidation comes in.  This means placing file1 somewhere on the disk, doesn't matter where, but file2 placed after it, and file3 placed after file2.  Or at the very least, have those files placed closely together on the disk to avoid the seeking.

Flav obviously has more insight into this problem as I, and has indicated there are other problems within the .dat file structure.  And certainly, there are many other things happening when you load into a zone.  I was just trying to keep my example simple, and to offer explanations as to why for DDO flash tech performs better, and why some see the problem and others don't.

Res

Yeah, Residue has a point here...

There's Fragmentation and Fragmentation.

There's the fragmentation you can do something about : Your file is spread over the hard disk in small chunk and when the game needs to read a bit of the file it needs to move it's little head all over the place... that takes time and is bad.

You can do something about it by defragmenting your disks and telling your defragmenting tool that the DAT files needs to be contiguous. ( I won't go into über performance seeking and geeking everybody talking about position of data segments on the physical disk to minimize the seek time, it would be pointless to 90% of the audience  Lets just say that there's a step beyond defragmenting a disk and that it involves lots of technical weirdness )

In computer timeline the seek time ( the time it takes for the little head to move at the location where it can read the data and for the disk to spin so that the tiny spot with said data is under the head ) is like waiting a decades for something to happen [seek time is in milliseconds ( ok, micro seconds for really good disks), modern computers works with nanoseconds reaction times ]

What I meant by the file is fragmented is that in a file like gamelogic.dat you will have at one index the scripts to fire when somebody jumps ( the jump action ), next to it you will have the script to fire for swimming actions, then you will have 10 instances of loot tables, then the script for the spell fireball, then 3 more loot table, then the script for the Druid wolf form, then the script for an enhancement SLA for the Monk, then the script for Blade barrier, then 20 more loot tables,
and so on...
There is no consistency, and I'm not sure there's a reason, since all the new stuff gets added at the first free spot offset if they fit in the hole at that offset or at the end of the file.
And if a given 'file' gets changed the offset pointing towards it is changed and the new version points towards the end of the file... the old version stay in at it's old location as long as nothing writes over it ( since the old location is marked a free spot. )... and if there was several call to the script in the index structure, then only the first occurrence of the offset gets updated, the other occurrences will still point towards the old location. ( even after said location has been overwritten with new stuff ) ... BTW MajMal, this might be the reason for the Doors Are Mushrooms problems you keep encountering and for all the bugs that were fixed before that reappears after a patch.
( as well as fun stuff like Epic GH Loot Table appearing in VON1 )