Page Index Toggle Pages: [1] 2  Send TopicPrint
Hot Topic (More than 35 Replies) Data Files Discussion (Read 19726 times)
Mr.Brown
Dragon Raider
***
Offline


I Love Drama!

Posts: 205
Joined: May 4th, 2013
Data Files Discussion
May 7th, 2013 at 6:49pm
Print Post  
Posting to discuss the data files.

Some clues are found here
https://www.asheronscall.com/en/forums/showthread.php?46283-Monthly-Portal-DAT-i...

Code
Select All
0x0000 Start File

[Super  Block]
0x0100 DWord : 0x4C5000 ( HexDump : 00 50 4C 00 )
0x0104 DWord : ???
0x0108 DWord : ???
0x010C DWord : ???
[...]
0x0130 DWord : ???
0x0134 DWord : ???
0x0138 DWord : ???
0x013C DWord : ???
0x0140 DWord : 0x5442 ( HexDump : 42 54 00 00 )
0x0144 DWord : Block_Size
0x0148 DWord : Size
0x014C DWord : Version
0x0150 DWord : Version-2
     !  0 : client_anim, client_gamelogic, client_general, client_highres, client_mesh, client_sound, client_surface
     !  1 : client_local_English, client_cell_1, client_map_1;
     !  2 : client_cell_2, client_map_2
0x0154 DWord : FirstFreeLoc  ! File offset of first free block !
0x0158 DWord : LastFreeLoc  ! File offset of last free block !
0x015C DWord : FreeBlkCount ! Free Block Count !
0x0160 DWord : DirOffset ! Offset to Directory !
 



Check if DWords between 0x104 and 0x13C checksums and CRCs?

Next group :

Directory :
[0x0000] DWord : 00 00 00 00
[0x0004] DWord : 9C 09 00 00
[0x0008] DWord : Directory Pointer 1
[0x000C] DWord : 9C 09 00 00
[0x0010] DWord : Directory Pointer 2
[0x0014] DWord : 9C 09 00 00

Repeat until a pointer to 0x00000000.

The hex addresses above are relative to the start of the directory.  If the DWord at address 0x0160 of the file contains 0x00D00, the 0x0000 address above is 0x00D00 and the first directory pointer is at address 0x00D08.

Mr.B
  
Back to top
 
IP Logged
 
wetspot
Korthos Resident
*
Offline


I Love Drama!

Posts: 4
Joined: Apr 13th, 2013
Re: Data Files Discussion
Reply #1 - May 7th, 2013 at 7:30pm
Print Post  
That's what SHE said
  
Back to top
 
IP Logged
 
kavinsky
Dragon Raider
***
Offline


I Love Drama!

Posts: 213
Joined: Jun 24th, 2012
Re: Data Files Discussion
Reply #2 - May 8th, 2013 at 6:43am
Print Post  
To avoid all the rubbishy posts from the tards, might want to move thsi to the other bit.

And I still want to get around to re-creating Rodrak's Program.
  
Back to top
 
IP Logged
 
Flav
Vault Frog
*
Offline


One Frog to Rule them
All!

Posts: 9969
Location: Land of the Frogs
Joined: Aug 29th, 2010
Gender: Male
Re: Data Files Discussion
Reply #3 - May 8th, 2013 at 7:17am
Print Post  
kavinsky wrote on May 8th, 2013 at 6:43am:
To avoid all the rubbishy posts from the tards, might want to move thsi to the other bit.

And I still want to get around to re-creating Rodrak's Program.


You're not alone...
At least it looks like, it's the same format as Asheron's Call, so maybe there's an AC modder around that want to look into things and come up with a good tool.
  

Yes my avatar is an Hermine eating a Greenland Lemming for brunch.
Back to top
 
IP Logged
 
kavinsky
Dragon Raider
***
Offline


I Love Drama!

Posts: 213
Joined: Jun 24th, 2012
Re: Data Files Discussion
Reply #4 - May 8th, 2013 at 7:45am
Print Post  
It is the same format - it's just figuring out the data dump from the .dat files that hte AC decrypter posts up. Something for the weekend perhaps.
  
Back to top
 
IP Logged
 
Inkblack
Puppy Farmer
Vault WDA
****
Offline


_,|,,

Posts: 1553
Joined: Sep 10th, 2009
Gender: Male
Re: Data Files Discussion
Reply #5 - May 8th, 2013 at 8:07am
Print Post  
Interesting. 

Any posters that help are likely to attract Turbine's attention.  But on the other hand, we don't want to lock this down so that it is so private it isn't useful.  I'll take suggestions via PM.
  
Back to top
 
IP Logged
 
Mr.Brown
Dragon Raider
***
Offline


I Love Drama!

Posts: 205
Joined: May 4th, 2013
Re: Data Files Discussion
Reply #6 - May 9th, 2013 at 12:43am
Print Post  
There are subdirectories for each pointer, with up to 62 file list pointers.

Code
Select All
Sub Directory :
[0x0000] DWord : 00 00 00 00
[0x0004] DWord : 9C 09 00 00
[0x0008] DWord : Pointer to File List 1
[0x000C] DWord : 9C 09 00 00
[0x0010] DWord : Pointer to File List 2
[0x0014] Dword : 9C 09 00 00
[0x0018] Dword : Pointer to File List 3
[0x001C] Dword : 9C 09 00 00
[...]
Repeat up to Pointer to file list 62
[0x0xxx] DWord : 32 00 00 00 ! End of Data (need to confirm 0x32 is the end marker)
 



Organization of file lists:
Code
Select All
File List :
[0x0000] DWord : 00 00 00 00
[..]
[0x01F0] DWord : 00 00 00 00
[0x01F4] DWord : 00 00 00 00
[0x01F8] DWord : 1E 00 00 00
[0x01FC] DWord : 00 00 06 00
[0x0200] DWord : <file name>
[0x0204] DWord : <file address>
[0x0208] DWord : 00 10 00 00
[0x020C] DWord : <modification time> (see calculation below)
[0x0210] DWord : 4C 02 00 00
[0x0214] DWord : 08 10 00 00
[0x0218] DWord : 00 00 00 00
 [code]

Need to figure out why DWords at 0x1FC, 0x208, 0x210 and 0x214 have specific values.

The Asheron’s Call people say time is in 0x20C. (see calculation example below)

Each file record in the file list is 8 DWord Long (16 words Long, or 32 Bytes long)
Example  (from client_local_English, a few updates ago)

 [code]
0x000005FC DWord : 00 00 06 00
0x00000600 Dword : 4F 3E 00 0A (0x0A003E4F) file 'name'
0x00000604 DWord : 9C 11 00 00 (0x0000119C) file address
0x00000608 Dword : 00 10 00 00
0x0000060C DWord : 61 8B 5E 45  Modification Time in Seconds since 1970 ????
0x00000610 Dword : 4C 02 00 00
0x00000614 DWord : 08 10 00 00
0x00000618 Dword : 00 00 00 00
 



Time calculation:
0x455E8B61
In Decimal  : 1163823969
1163823969/3600 = 323284
323284/24 = 13470
13470/365 = 36 (rounded)
1970+36 = 2006

Now we have the files.... they start like this:

Code
Select All
[0x0000] DWord : <file name>   
[0x0004] DWord :   
[0x0008] DWord :
 



Files seem to vary by type.  The LOTRO unpacker works ok, but not great on DDO files.

The tough part will be figuring out the file types and what they mean.
  
Back to top
 
IP Logged
 
karatekid
Self-Proclaimed Puppy Farmer
*****
Offline



Posts: 4830
Joined: Aug 13th, 2012
Re: Data Files Discussion
Reply #7 - May 11th, 2013 at 8:54am
Print Post  
Once extracted, do the file headers align with known file types in something like etc/magic?
  
Back to top
 
IP Logged
 
Flav
Vault Frog
*
Offline


One Frog to Rule them
All!

Posts: 9969
Location: Land of the Frogs
Joined: Aug 29th, 2010
Gender: Male
Re: Data Files Discussion
Reply #8 - May 11th, 2013 at 9:48am
Print Post  
well if the .dat file name is a hint on file type I'd say the following :

*look at the ddo directory content*

client_anim.dat : should contain the animation information.
client_cell_1 & 2.dat : no idea
client_gamelogic.dat : well game logic whatever that means.
client_general.dat : general game information, whatever that means too.
client_highres.dat : High Resolution Textures
client local_xx.dat : DE Kraut local stuff, FR Frog Local stuff.
client_local_English.dat : English local stuff. Local stuff meaning descriptions, fluff texts.
client_map_1 & 2.dat : maps ? maybe the world map... but that's a WAG.
client_mesh.dat : the mesh ( whatever that means I know it has some meaning in 3D rendering. )
client_sound.dat : Sound files, music, kobold still hates you, and so on.
client_surface : the surface, again whatever that means... I has some meaning when used in conjunction with the mesh in 3D Rendering

But that's just looking at the file names I didnd't check the real content.

Now another Wild Assed Guess is that what is called file in the posts from Mr Brown is not a real file, but it may contain, encapsulated in some DDO stuff real files... It's just because it's the way things are done nowadays, you encapsulate things you want in the format you want it and make software decapsulate it so that it can throw the result at another bit of software for use.

So I'm sure that if we look at the sound dat file we will see lots of vorbis files ( there's 2 Vorbis DLLS in the DDO directory, so it's not a WAG, it's almost a certainty ) stacked one after the other with some DDO relevant information between them so that DDO can retrieve the "Kobold Remember Waterworks" one line when needed.

Ok, I just spend my time encapsulating PPPoE ( or ATM)  into VLANs then into Pseudowires, then into MPLS and decapsulating it back at the other end... I even encapsulate X.25 into IP then into VLANS then... well get the drift, there's even weird stuff like X.25 encapsulated into IP encapsulated into X.25 encapsulated into ATM encpasulated into IP encapsulated into pseudowires and then into MPLS... and I've come to the conclusion that most of the modern computing is just about encapsulating old ( as in 10+ year old ) stuff into something more shiny to make it look new.
  

Yes my avatar is an Hermine eating a Greenland Lemming for brunch.
Back to top
 
IP Logged
 
Mr.Brown
Dragon Raider
***
Offline


I Love Drama!

Posts: 205
Joined: May 4th, 2013
Re: Data Files Discussion
Reply #9 - May 13th, 2013 at 11:44pm
Print Post  
Mr.Brown wrote on May 9th, 2013 at 12:43am:
The tough part will be figuring out the file types and what they mean.

I have ideas. Does this look correct? PM feedback.

Each file appears to have its own field separator which is defined at offset 0x01A4 in the file header. I have checked it in the files below.

client_local_English : DWord  0x9C090000
client_general : DWord 0x9C090000 (0x9C0D0000 also?)
client_sound : DWord 0x00440000
client_anim : DWord 0x00440000
client_cell_1 : DWord 0x00140000
client_cell_2 : DWord 0x9C090000
client_gamelogic : DWord 0x9C090000
client_highres : DWord  0x00100000
client_map_1 : DWord 0x9C090000
client_map_2 : DWord 0x9C090000
client_mesh : DWord 0x00440000
client_surface : DWord 0x00100000

It may be defined in another area of the header with the location at offset 0x1A4 marking the end of the header. It is the last data in the header.
  
Back to top
 
IP Logged
 
stainer
Arkat's Bitch
*
Offline


Gawna's big sexy crony!

Posts: 12345
Location: gawna's nuts
Joined: Jan 28th, 2010
Gender: Male
Re: Data Files Discussion
Reply #10 - May 14th, 2013 at 12:13am
Print Post  
No one plays this game anymore, or cares. Post this on reddit or something.
  

Nilazgrc: For all those familiar with loreseekers/sentinels.... Dont run with Stainer.... Guy is a tool. Black list his toons.
kmack can drive the tractor.
The Vault donates to charity.
JDollar wrote on Apr 25th, 2013 at 4:47pm:
Stainer likes tractors
Back to top
 
IP Logged
 
arjiwan
Titan Demolisher
****
Offline


I Love Comedy!

Posts: 320
Joined: Nov 17th, 2011
Re: Data Files Discussion
Reply #11 - May 14th, 2013 at 1:23am
Print Post  
stainer wrote on May 14th, 2013 at 12:13am:
No one plays this game anymore, or cares. Post this on reddit or something.


You make me sad.

also that new sub-forum for NWO, it makes me even sadder.
  
Back to top
 
IP Logged
 
Terebinthia
Horoluth Raider
****
Offline


All your cupcakes are
belong to me! :D

Posts: 2499
Joined: Jan 6th, 2011
Re: Data Files Discussion
Reply #12 - May 14th, 2013 at 3:01am
Print Post  
arjiwan wrote on May 14th, 2013 at 1:23am:
You make me sad.

also that new sub-forum for NWO, it makes me even sadder.


I miss the 5 players you never run with thread, but I think it illustrates what an empty place DDO is becoming. The forum FUBAR isn't exactly doing them any favours either. That plus the character copy issues in Lamma have disconnected me from the game still further. Both my running partners have left or are on hiatus.  It just feels that I'm running through a ghost town when I log in, although admittedly I don't play in primetime.

If they want to deliver a casual MMO there are games that do it very much better with fewer bugs. I for one am enjoying the hell out of Neverwinter, and think the Foundry has real potential to give it legs.
  
Back to top
 
IP Logged
 
rest
One Man Wolfpack
*
Offline



Posts: 7223
Joined: Aug 13th, 2010
Gender: Male
Re: Data Files Discussion
Reply #13 - May 14th, 2013 at 10:54am
Print Post  
stainer wrote on May 14th, 2013 at 12:13am:
No one plays this game anymore, or cares. Post this on reddit or something.


I do.
  
Back to top
 
IP Logged
 
karatekid
Self-Proclaimed Puppy Farmer
*****
Offline



Posts: 4830
Joined: Aug 13th, 2012
Re: Data Files Discussion
Reply #14 - May 15th, 2013 at 9:10am
Print Post  
One could post an absolutely undetectable plat dupe exploit here and I wouldn't log on to do it. 

THAT'S bad.
  
Back to top
 
IP Logged
 
whothefuckami
Stormreaver Piker
*
Offline


Oppan Gangnam Style

Posts: 599
Joined: Oct 6th, 2010
Re: Data Files Discussion
Reply #15 - May 15th, 2013 at 8:23pm
Print Post  
Inkblack wrote on May 8th, 2013 at 8:07am:
Interesting. 

Any posters that help are likely to attract Turbine's attention.  But on the other hand, we don't want to lock this down so that it is so private it isn't useful.  I'll take suggestions via PM.


Ever wondered why I use an anonymous name on the Vault?
  

Troll account, please ignore
Back to top
 
IP Logged
 
IMARANGER
Completionist (i.t.p.)
******
Offline


I Love Drama!

Posts: 5034
Joined: Jan 17th, 2013
Re: Data Files Discussion
Reply #16 - May 15th, 2013 at 10:36pm
Print Post  
whothefuckami wrote on May 15th, 2013 at 8:23pm:
Ever wondered why I use an anonymous name on the Vault?


Because you're not stupid?
  

OnePercenter wrote on Jan 19th, 2014 at 5:32pm:
I agree with 0bama...


OnePercenter wrote on Feb 9th, 2015 at 8:18am:
Not saying that others weren't better, just that quantity does not always indicate quality...  I mean, look at my post count...
Back to top
 
IP Logged
 
Munkenmo
Epic Poster
*****
Offline



Posts: 4339
Location: A land under down under
Joined: Nov 10th, 2010
Re: Data Files Discussion
Reply #17 - May 15th, 2013 at 11:56pm
Print Post  
whothefuckami wrote on May 15th, 2013 at 8:23pm:
Ever wondered why I use an anonymous name on the Vault?


Because you're a pussy who lacks the required intelligence to post here otherwise?
« Last Edit: May 15th, 2013 at 11:56pm by Munkenmo »  

So you want to know about an exploit?
PM Epoch For Details. Or, in case you don't already know, OnePercenter controls the Exploits Board. Lastly, if you're truly desperate, Vendui Tells Everyone
Back to top
 
IP Logged
 
Bill Z Bubb
Boobies Controller
*
Offline


I only know three chords!

Posts: 1238
Joined: Jul 2nd, 2012
Re: Data Files Discussion
Reply #18 - May 16th, 2013 at 11:07am
Print Post  
I'm no pussy, I got away with a lot of posts on the DDO forum by not naming my account here GrandpaBill.  I just figured that if they could definitely tie a post there to an account here they'd infract with extreme prejudice.
  

            
Back to top
 
IP Logged
 
stainer
Arkat's Bitch
*
Offline


Gawna's big sexy crony!

Posts: 12345
Location: gawna's nuts
Joined: Jan 28th, 2010
Gender: Male
Re: Data Files Discussion
Reply #19 - May 16th, 2013 at 11:08am
Print Post  
Bill Z Bubb wrote on May 16th, 2013 at 11:07am:
I'm no pussy, I got away with a lot of posts on the DDO forum by not naming my account here GrandpaBill.  I just figured that if they could definitely tie a post there to an account here they'd infract with extreme prejudice.


Yes.
  

Nilazgrc: For all those familiar with loreseekers/sentinels.... Dont run with Stainer.... Guy is a tool. Black list his toons.
kmack can drive the tractor.
The Vault donates to charity.
JDollar wrote on Apr 25th, 2013 at 4:47pm:
Stainer likes tractors
Back to top
 
IP Logged
 
stainer
Arkat's Bitch
*
Offline


Gawna's big sexy crony!

Posts: 12345
Location: gawna's nuts
Joined: Jan 28th, 2010
Gender: Male
Re: Data Files Discussion
Reply #20 - May 16th, 2013 at 11:19am
Print Post  
The bonus of getting banned is you no longer have to participate in that the thing they call a forum.
  

Nilazgrc: For all those familiar with loreseekers/sentinels.... Dont run with Stainer.... Guy is a tool. Black list his toons.
kmack can drive the tractor.
The Vault donates to charity.
JDollar wrote on Apr 25th, 2013 at 4:47pm:
Stainer likes tractors
Back to top
 
IP Logged
 
rest
One Man Wolfpack
*
Offline



Posts: 7223
Joined: Aug 13th, 2010
Gender: Male
Re: Data Files Discussion
Reply #21 - May 16th, 2013 at 12:08pm
Print Post  
I'm nicer on the Vault than I am on the regular forums, so I used the same name.
  
Back to top
 
IP Logged
 
Flav
Vault Frog
*
Offline


One Frog to Rule them
All!

Posts: 9969
Location: Land of the Frogs
Joined: Aug 29th, 2010
Gender: Male
Re: Data Files Discussion
Reply #22 - May 16th, 2013 at 1:30pm
Print Post  
Bill Z Bubb wrote on May 16th, 2013 at 11:07am:
I'm no pussy, I got away with a lot of posts on the DDO forum by not naming my account here GrandpaBill.  I just figured that if they could definitely tie a post there to an account here they'd infract with extreme prejudice.


Well, I haven't been really hiding who I am on the main forum, and any regular Vaultie should know it by now. So far I haven't been infracted with extreme prejudice.
Actually I have even been asked if I would join The Mournlands Puppet Show. ( I never answered that one... for obvious reasons )

Now I hope Mr Brown cannot be tied in anyway to an account, because what he has been giving us is pure gold.
I couldn't resist having a look at the file to check the information, so far it match, if only I was better at programming in visual <insert crapastic modern language here>...  Roll Eyes
Coding a tool in assembler is going to take me ages Cry
« Last Edit: May 16th, 2013 at 1:33pm by Flav »  

Yes my avatar is an Hermine eating a Greenland Lemming for brunch.
Back to top
 
IP Logged
 
mudfud
Shroud Slacker
***
Offline


I Love Drama!

Posts: 1123
Joined: Nov 17th, 2011
Re: Data Files Discussion
Reply #23 - May 17th, 2013 at 5:18pm
Print Post  
Bill Z Bubb wrote on May 16th, 2013 at 11:07am:
I'm no pussy, I got away with a lot of posts on the DDO forum by not naming my account here GrandpaBill.  I just figured that if they could definitely tie a post there to an account here they'd infract with extreme prejudice.


Do what I did. Find someone who posts regularly on your server forums, then see if they have a username here. If not great for you, you got a ddo vault username. Bad for them because if turbine does infract for things said here over there, that person will be wondering why the hell they getting in trouble.
  
Back to top
 
IP Logged
 
Bill Z Bubb
Boobies Controller
*
Offline


I only know three chords!

Posts: 1238
Joined: Jul 2nd, 2012
Re: Data Files Discussion
Reply #24 - May 17th, 2013 at 11:43pm
Print Post  
Now THAT sounds like the actions of a pussy.
  

            
Back to top
 
IP Logged
 
Page Index Toggle Pages: [1] 2 
Send TopicPrint