So apparently the game worlds stayed down extra long today because SSG literally forgot to turn them back on.

Then downtime for Sarlona was extended because, according to jerry:

Quote:


Now, we know DDO's servers were (badly)virtualized a few years ago. The theory back then was that the virtualization was meant to reduce the game's hardware footprint by replacing 8 independent machines with some smaller number of virtual hosts.

However, the fact that 1 individual server can stay down "due to hardware maintenance" would seem to punch a hole in that theory.

Here's my new belief: MMOs in particular are usually optimized at the hardware-level, E.G. the code is actually written around the server's specific architecture and memory topology. This is because in order to get the kind of lag-free gameplay we all so desperately want, making the server software work is a lot more complicated than just "put a faster CPU in there and give it more RAM".

This is why things like WoW getting a server upgrade is such a big deal: its not just a matter of installing the software on a new machine. It means rewriting the code for that hardware, a task requiring elite veteran programmers to carry out.

Enter DDO, which is apparently funded entirely by a blind orphan sitting on the sidewalk with a tin cup. Hardware ages, companies stop producing spares, and support contracts expire(and I feel bad for you if you've ever had to work in a data center trying to keep out of warranty hardware running - its not fun). A few years back, SSG was in a pickle: they could no longer feasibly keep the DDO server hardware running, but they also no longer had the technical expertise in-house to migrate to a new hardware platform.

The simple solution to this problem was virtualization. Build a virtual server that mimics the hardware DDO was written to run on. In practice, you would never do something like that in a production environment; but in practice no one would ever hire jerry, so it makes sense in context.

This would then mean each individual server still has it's own physical machine to run on, but that hardware is running a hypervisor which in turn runs the virtual server. This is far less effective than running the servers "on metal" - but it does work, AND it does get you into a place where some deficiencies can be made up simply by cramming in more CPU/memory/disk I/O resources.

Keyword there is "some" of course.

Remember that 4 minutes or so 12 years ago when DDO and WoW were competitors? Yeah, me neither.

noamineo wrote on Oct 28^th, 2020 at 9:40pm:

LOL, since decades no application server software has been optimized at the hardware-level.

does editing still not work or am i retarded

SweetKitten wrote on Oct 29^th, 2020 at 1:01pm:


i agree, a game would have to be from like 1999 and made in someones basement or something to even consider the thought LOL amirite

Quote:

Asheras wrote on Oct 29^th, 2020 at 11:29am:


Always a possibility but there's not a lot of ambiguity in jerry's statement. One server down due to hardware would tend to mean that hardware ran just that server, otherwise it'd be 2+ servers down.

Then again jerry's also not the sharpest tool in the shed, so who knows what the statement was actually based on?

noamineo wrote on Oct 28^th, 2020 at 9:40pm:


Oh, come on be serious a minute, you really think that a game server is actually one single server ? At a glance from the old olgs I keep and doing a few wild assed guesses, there's at least 3 servers for one game server : the login server, the game server itself and the backend database server that stores our characters. I'd add a fourth one the queing server but I've never been able to prove it was a separate function from the login server.
Once upon a time all those where physical boxes... with a front end network switch and a backend network switch... Now they are all virtualized on the same physical box with front end vlans and back end vlans in teh virtualizer switching function.

noamineo wrote on Oct 28^th, 2020 at 9:40pm:


They just did a RAM upgrade on all the servers and Sarlona hardware failed to come up with a checksum error on a stick... ( really wild guess ) So they are troubleshooting it.
Since what I called Sarlona abovev is a box that contains 3 or 4 ( maybe more ) VMs defining Sarlona it's not far from truth.

noamineo wrote on Oct 28^th, 2020 at 9:40pm:


Well compared to 2005 servers the servers we have right now are light years ahead in capacity to the point that more RAM and more Core can work...( and not Faster CPUs... Modern CPUS are generally slower than the 2006 ones ) You don't even need to upgrade the software to multicore... you just assign a core to a virtual server in the hypervizor.

noamineo wrote on Oct 28^th, 2020 at 9:40pm:


Server upgrade is always a big thing... To the point that I once spent a week end in 8 Hours shifts to perform a software upgrade ( yep no hardware involved, but critical telecom equipment... so critical that we twinned it before the operation so that the twin took over while we upgraded the system ). Getting the 99.9999% uptime in a yearout of any system ( aka Telecom Grade ) does not come cheap.

noamineo wrote on Oct 28^th, 2020 at 9:40pm:


we don't really know in details how a given DDO Server is built... but taking into account the fact that the bare metal nowadays is lightyears better than the 2006 one... Even without multicore/multithreading software you can assign a core to a specific function ( say login ) another one to another function ( say database ) and another one to a third function ( the game itself ) and still have power ( and RAM to spare )

SweetKitten wrote on Oct 29^th, 2020 at 1:01pm:


Definitely, the time when optimization at hardware level was something belongs to before Y2K. Since then it's the Time of Visual This or That that spew code generated by moving boxes around on a screen and linking them together. Said code being horribly unoptimized... even in the source code. ( so don't expect it to be better in the compiled code )
But who cares... we have computing power to waste, RAM to waste and Mass storage to waste... now try to make some graphical library for a VGA screen fit in less than 4K and we will talk about code optimization.

just want to thank you ppl for the tech talk. generally, not only this one. i understand less than nothing but it is interesting anyway.

Notanarc wrote on Oct 29^th, 2020 at 4:17pm:


I wouldn't discount it. Given how much SSG likes to troll its paying customers its easy to assume their office politics are even worse. After jerry had to requisition that special keyboard with the double extra-large keys for his fat fingers, things haven't been the same.

Flav wrote on Oct 30^th, 2020 at 12:26pm:


True. But then again you also log in before server-selection. Maybe its one login server to rule them all then independent queueing servers for each server?

Flav wrote on Oct 30^th, 2020 at 12:26pm:


A server swap is also doable. Pull drives out of hot-swap, pull server off rack, rack new server, put drives back in. But that's less "maintenance" and more "catastrophic failure". I wonder if SSG is working off of mirrored physical hosts or playing fast and loose? They don't seem to mind days of down-time.

Flav wrote on Oct 30^th, 2020 at 12:26pm:


Yeah I laughed pretty hard myself. After talking for him a bit more it turned out to be one of those "shoot for the moon"-type scenarios. Funny thing is our wireless APs could actually get there - we had a few units in my tenure that logged over 2 years of contiguous operation. of course, the rest of the wired LAN getting to that point is not very likely, and that's before you even get to telco. AND this was a venue that could handle a little down time without the world ending.

Flav wrote on Oct 30^th, 2020 at 12:26pm:


Tough to say, this is SSG we're talking about. Remember some of DDO's code is unaltered from the Asheron's Call beta.

Flav wrote on Oct 30^th, 2020 at 12:26pm:


That's why I theorize the project to virtualize the DDO servers went so poorly. If there wasn't *some* level of hardware optimization happening, simply moving the system to a virtual host and adding much more power should not have caused the level of fuck-up that we bore witness too.

Flav wrote on Oct 31^st, 2020 at 7:43am:


Well if its a 2U and you are replacing like for like you don't have to replace the rails, but thats neither here nore there. Man I want one of those SUN fire V880... 8 physical CPUs! I don't need one, but damn I miss those days...

Flav wrote on Oct 31^st, 2020 at 7:43am:

 
I believe the wayfinder had to do with systemic DB corruption and they had to roll back s far because that was the most recent backup that didn't have the corruption. Its also entirely possible jerry miscounted the doors and pissed in the server room again thinking it was the bathroom. Exactly how SSG's storage is configured we may never know, but they have to have one or more SANs, there's just no way of knowing what or how they are configured.


Flav wrote on Oct 31^st, 2020 at 7:43am:


True.

Flav wrote on Oct 31^st, 2020 at 7:43am:


Oh that is funny.

Flav wrote on Oct 31^st, 2020 at 7:43am:


There are a lot of ways and we can generally assume SSG hit... most of them.