Digimonk wrote on Mar 29^th, 2016 at 1:24pm:

You describe a failure in configuration management.  Your job might be at risk, but even more so should be the person who allowed you to make that change without proper supervision.

I've been cursed at many times by techs who refused to accept my mandate that they document the changes they were going to make.  In most cases they were the ones who left the organization, and I continued to improve it.  The cases where I was the one who was forced out are a constant source of humor at cocktail parties.

wogoflove wrote on Mar 29^th, 2016 at 4:20pm:

https://www.youtube.com/watch?v=BKorP55Aqvg

Not to be a turbine apologist, but it's clearly not a simple fix.  I'm guessing the devs are playing a game of CYOA and not admitting they don't have a handle of the situation, but suggesting that a fresh grad with an IT degree could just come in and fix it in like a day, is really naive. 

Quote:

Now that's definitely on your end.  DDO or LOTRO has never done anything remotely like that on any of the 5+ computers I've run. 

Digimonk wrote on Mar 29^th, 2016 at 1:24pm:


On dedicated hardware optimized for VMs, with people that have more than a clue. One Word : Turbine.

Digimonk wrote on Mar 29^th, 2016 at 1:24pm:


At a guess, the beancounters or the manglers lookad at the specs of the old swervers in the datacenter, looked at the specs of the new server and just made some Shade Maths so that several old servers could be put on a single one in a virtualized environment. ( because it's the hype of the moment )
Nobody asked the people that know better, what to do, how to do it, and how to plan the hardware to get it right.

Frank wrote on Mar 29^th, 2016 at 3:31pm:


Oh, they are not an all encompassing evil... They have their use in deeply redundant clusters of webservers or in distributed datacenter ( read : cloud ), and a few more applications... But in real time ( or at least in plesiochronous ) application with a non negligible ammount of data exchanged between a single server and a large amount of clients, it's not the best choice... unless you have one of the best VM expert team at hand.

All in all that VM thing boils down to : ( or TL;)DR )
- VMs are not always a bad thing but are mostly inadequate for the specific MMO use called DDO.
- Turbine... If we can Fail at it we will make sure we do.

P.S.  poorly done VM environments suck balls.  Well done VM environments can give amazing performance.  The same is true of hardware environments.  Do them poorly and they suck ass.  The only difference is that more people have experience with hardware solutions than proper VM designs, so the probability of a mis-configuration is higher.  But, the plus of the virtual environment is that reconfiguration is usually easier than with hardware solutions.

noamineo wrote on Mar 29^th, 2016 at 5:52pm:


You never know.  Morons here and on the MoBo constantly claim that people at Turbine must be uneducated and unskilled.  While I'm not a fan of their work at times, the few resumes I've ever seen linked all had respectable backgrounds and work experience.  None of them screamed "total hack". 

Flav wrote on Mar 29^th, 2016 at 5:36pm:


Well, I, for one, learned a new word today.

Asheras wrote on Mar 29^th, 2016 at 5:58pm:


The way they treat their customers and DDO screams "total hack".

It's too bad we can't play "DDO Developer Resume" instead.

OldCoaly wrote on Mar 29^th, 2016 at 7:49pm:

LMAO, awesome!

Flav wrote on Mar 29^th, 2016 at 5:36pm:

You realize you went to the trouble of quoting me when I had already added in the same summation as you just did after quoting me, right? 


*Hint*  It's where I said:
Quote:

So apparently they did different things to different servers and the reports are in, it didn't do shit. People are still lagging or even completely freezing. I wonder how would the player base feel that now they're being used like rats?

AtomicMew wrote on Mar 29^th, 2016 at 5:11pm:

In the form of a kitten.

Not to be a Turbine apologist, but I loled. 

So much awesome there.  I've been on conference calls where a tech had to (wanted to, more likely) explain that the reason that a particular site was below the average in performance was because that was that average meant.  You could have heard a pin drop, but I was on mute because I was afraid of chuckling loudly enough for the customer to hear.

Flav wrote on Mar 29^th, 2016 at 10:09am:


I think you're right. I actually have personal experience with this as we created VM's for our business software to create a SaaS solution for our customers. Originally we paid a company for those services but we transitioned to a solution in which we owned the servers and just paid for them to be housed with a hosting company.

When we transitioned we bought a physical server to run the VM's that we thought would be plenty. It had enough VPU's to create the same number of servers we had with the original company so we figured we'd be fine. So we slowly moved one VM at a time from the original company to our own servers. As everything went better and better though we got cocky and started moving 2 or 3 VM's a night.

You can guess what happened. While there were plenty of VPU's the VM's share the CPU resources. Just because there's 8 VPU's on each server doesn't mean that all 8 can be maxed out on every single VM at the same time and not cripple the system. We solved our problem very simply - we added more physical servers and migrated some VM's to those. What had started as a plan for one physical server to serve 11 VM's morphed into 4 physical servers to serve up to 13.

Turbine could do this. Provisioning extra physical servers, upgrading CPU's, etc. are all very easy to do. You have to acknowledge the problem first though and be willing to spend money to fix it. Our company prides itself on its customer service and therefore it was a no-brainer - we spent whatever it would take and put in an infrastructure that could support our customer's needs. Not so sure I see Turbine going that route or even figuring out that's what the problem is.

Darkrok wrote on Mar 30^th, 2016 at 9:52am:


I'll admit that I have never played with production grade full fledged VMs ( well beyond a custom tailored, heavy clusterized, telecom grade thing called TSP ...), I've installed lots of ESXI ( and the VMS that needs to be put on them, but it wasn't hard watching the script doing it ), and most of my playing around with Virtualization has been with Sun Solaris 10 Zones ( they are not realy VMs, they share the Kernel with the hosting Solaris 10  ).

But knowing how computers are made and how they work it's obvious it can happens once you add virtualization.
And it's probably one of the most common mistake people ( read beancounters ) do when they tell people to go Virtualized.
IOs ( specifically memory and disks ) have a limited bandwidth... that bandwidth is not and cannot be virtualized.

On Sparc Servers once they stopped frying processors trying to catch up with X86 frequencies ( which is pointless with a RISC system ), they went to heavy core/thread... So a single UltraSparc T1 processor carries 8 Cores each with 16 Threads... So for a Windows comparison : that's 256 Processors on one chip.
You can do a lot of things with that many processors ( especially when you have single processed, single threaded applications ), but the IO ( read : Disk and Memory ) is shared between all the (virtual) processors... and IO hasn't much evolved in throughput compared to the previous generation - in SPARC case : Ultra 320 15KRPM SCSI disks or SAS 15KRPM disks for mass storage, and DDR3 for Memory -

On our personnal computer we can have SSD drives, but most of the time they are just too expensive for server applications.
( and in most case use they are a waste of money )

Anyway, it's wine time for me ( if it wasn't obvious ) and I need to get all those free quivers done before they shutdown the anniversary.