Solid hypothesis

That people seem to be paying about 1/50th the price for one of their big sellers and sources of revenue, leads me to believe they must be trying to do SOMETHING about it.

I would, if I were them.

If their intent was to go after duping in DDO, I find it breathtakingly funny that they've managed to completely shut down LOTRO until further notice while keeping DDO running like a drunk in a potato sack race.

noamineo wrote on Jul 22^nd, 2020 at 6:16pm:


Very nearly a coffee-sprayed-over-monitor moment.  I was picturing oompa-loompas singing this in the SSG stable.

Skoodge wrote on Jul 22^nd, 2020 at 4:41pm:


Interesting notion.

I ( we ) know that there's a frontend and a backend side for each server.  ( as noticed by Techno here : http://www.ddovault.com/cgi-bin/yabb2/YaBB.pl?num=1461363393 after one of my post. )

From the same post we can see that on the frontend side there's different IPs for different things ( even if physically they may all end up in the same virtual server ).
We only have hints of the backend side with the login queue and the status server IPs, but there's also probably a database server for each game world. ( along with an O&M server, a log server and a few other servers )

My understanding of the current dupe method is that it's using the fact that the database content is updated sequentially with a delay and if timed right some of those updates are discarded by the shutdown.
The best method to get rid of the duping is to dump everybody ( kick all the players out ) and force a database queue emptying but that can be annoying if it's not automated and/or if there's no tool to perform the database queues flush ( especially if there's a queue per character/user... which would make it a pita to do )... you need a guy to do it on all the servers. ( and that guy needs to know what he's doing ).

Now add to that the fact that apparently the backend notwork seems to be common for all the game servers, and you have the recipe for some nice fail.

one thing they could try to llimit the latency between the database and the game world is to separate the database transaction network from the main back end network and to upgrade it... and to separate it on a game world basis... since each game world has it's own database server and everything is supposed to be virtualized, they could host the game server and the database server on the same physical hardware using the internal virtualizer network switch for added bandwidth ( instead of going out to a physical network and another box ). But this requires some devious and nasty optimization on the virtual machine side.
If not done correctly all the data writes in the database files would impact every virtual machine hosted on the physical server ( concurrent writes on the physical HDD. [ because it's probably not SSD, but a RAID5 array or something sdimilar ] ), creating lag, long loading screens and more... ( which strangely is what we see )


TLDR : I think they tried something with the database servers in the backend, namely hosting them on the same physical host as the game world, to limit the
duping cases, and fubared it, because they didn't take into account the DB writes.



As for LoTRO... Remember it's exactly the same game as DDO when it comes to backend and server setup the only difference is in the universe. ( the DAT files used by the game client.  [ hell it's also exactly the same setup that was used for AC2 ]
The games were devlopped concurently and in the early days things added to DDO were added later to LoTRO ( we beta tested lots of things for them that way ).
Nowadays it's more LoTRO that beta test things later added to DDO ( because there's more money in LoTRO to develop things... and once developped, since the games are identical, adding it to DDO is cheap and easy )

noamineo wrote on Jul 22^nd, 2020 at 6:16pm:


Two words:  Because Turbine.

noamineo wrote on Jul 23^rd, 2020 at 1:53am:


They are on the same hardware... that's the most obvious theory.

Now seriously, I suspect that it's more complex. And one of the thing they might be doing is to upgrade their network infrastructure.... And they didn't take into account all the flows between each and every elements.. ( virtual or not )
If they want I can do some consluting for them on DC notwork... been there, done that, got enough complex project under the belt ( most of them being tied to $TELCO charging & Billing ). I'm cheap compared to an US network consultant.

Notanarc wrote on Jul 23^rd, 2020 at 5:43am:


That would be the smart, easy fix.

Sev doesn't do the smart, easy fixes.

Skoodge wrote on Jul 22^nd, 2020 at 5:15pm:


They might share the same hardware.  But they are on separate virtuals. That has been stated that they virtualized the servers.   Don't know what they are using (Hyper-V, VMWare, etc) but nothing they did in their applications would be able to impact performance on the other virtuals.  The more likely issue would be an issue at the hardware level or a bad patch/update to the virtualization software.   But nothing they do to "Fix duping" in DDO would affect LoTRO, the forums, etc.   

An interesting thing I notice, though, is that when I freeze, my hirelings, pets, the mobs, and all the info coming from the server is still moving without any impact or lag.   This is different than other freeze lag (like DoJ or Deathwyrm, etc).   It almost seems like my keyboard and mouse input just aren't being received.  It feels more communications related, but not just pure latency/packet loss, which affects all communication. 

Flav wrote on Jul 23^rd, 2020 at 5:23am:


That would be really funny. Especially if you went through the whole job and then told them right at the end "oh, hey, I'm Flav on the Vault". The loo on sev's face....

Flav wrote on Jul 23^rd, 2020 at 1:27am:

I disagree.

They use MS SQL server.   SQL does virtual writes that then get asynchronously written to disk in separate functions that are independent of the application interaction.  The game would never be waiting for a physical write to disk.   Given how SQL Server handles logical writes, the write-ahead transaction log, dirty pages, batched physical writes in lazy writes, eager writes, and checkpoints, physical IO would not be a factor.   And all that assumes they are running DB on the same hardware as the front end application.      

What you describe hasn't been an issue for 20 years.

Asheras wrote on Jul 23^rd, 2020 at 2:26pm:


Should I start to mention that I did Sybase DBA stuff for more than 10 years ? ( MS SQL Server is the bastard cousin of Sybase ASE )
Should I start to mention that the game is 14 years old ?
Should I start to mention that the infrastructure setup is as old as Asheron's Call 2 development ? ( since they basically resused the same type of infrastructure for DDO and LotrO )
Trust me... differences between Sybase and MS SQL Server are tiny... and most of them are underlying OS related. ( after all they started their life as siblings )

As for disk writes... if SQL Server manages it's own devices ( reported as raw devices in the hosting OS ) and said disks are on a raid unit somewhere in a SAN  you can have a whole list of wonky things happening... Not to mention forcing synchronous writes to limit the dupe options...

But overall it all boils down to Virtual machines optimizations and of the database server disks are dedicated ( and local/SAN with priority )
Physical IO will always be a factor when you work with databases, as you can have lockouts that basically forces you to wait for the dataset to have no more activity on it before you can do something on it. This is especially important in large databases, as it can block a large amount of data to be read as there's a write operation going on at the same time.
It's all a matter of DB Optimization and VM optimization and since the move to Virtual Server we all know they have no clue about optimization and manage it by the seat of the pant.