okay, then if they just look at the bug what does that do?with no diagnostic tools it is impossible to fix.

currently, they look at reported bugs on a non-live build. When they determine that everything is WAI, since it isnt feather_of_sun'ed on their internal testing build that doesnt match lamma or live, they discard the reports and go on lunch. been this way forever with turbine.

deal with incompetent coders or find another game, it will only get worse till WB dissolves turbine into their gaming division. then it wont matter 

So you dismiss my point by comparing me to chai? Isn't that like godwin's law of ddo forums?

You mean cunt? You'll have to see gawna for the rights to that.

The way Eladrin explained it is awful.  The concept is sound.  You do not run diagnostics on production.  You do have a testing platform that is identical to live for the purpose of reproducing bugs and isolating.  Also used for stress testing. 

Anyone thinking that a dev should "pop onto live", fire up their diagnostics and logging and test out some bugs is asking for major lag, amongst other things, and probably doesn't have any idea what they are talking about.  I would fire a developer for running such on a production blade with paying clients.  Now, where I work, we have the ability to redirect customers to other blades and run tests on the environment that the error was reported on in addition to our testing blades/virtuals, but no well designed evironment includes "popping onto live" and firing off diagnostics.

All the above said, the idea would be stupid in my environment, but that may or may not be the case for Turbine.  We make a TON of assumptions as players about how they have things designed and architected and what their procedures are and it is very easy to pass judgement based on our assumptions.  But really, we are uninformed as to what their configuration is, testing options and procedures are, and whether they are well designed, implemented or utilized.

Of course, there is one thing we CAN evaluate.  The results, in terms of mean number of failures per release and mean time to repair, are shockingly poor and indicate either poor architecture or poor procedures, or both.  Probably combined with poor execution by the team for the triple play of software.    

In today's day and age, with the easy of use of VM's not having multiple test platforms at your finger tips is shoddy at best.  Anyone from the comfort of their desk who does sustaining engineering should be able to fire up a personal VM of live with all the tools in place to test something like this. 

I managed a lab before VM's where I kept images of dozens of production products on in dozens of conceivable configurations as ghost images so a tech could just reimage machine and start testing.

This sort of "Oh, it worked in my build."  sloppiness is why we see reoccurring bugs and ridiculous bugs that should have been caught.  Everyone needs to be working on the same code base.  If you have custom or side projects they should be branched and on separate VM's or even physical machines.  If everyone is building like this it's no wonder why QA can't adequately test shit.  It's sloppy as hell.  It so easy to accidentally port in changes that were never meant to go or overwrite changes you never merged in.

Fucking hardware cost...

I have what I think is an accurate idea of infrastructure involved in the Live server....

Lets just say that :
- 1 Oracle Cluster ( 2 powerful nodes at least )
- 1 EMC² Symetrix ( to hold the several terabytes of data )
- 1 SAN Infrastructure ( to connect the Symetrix to the Oracle Cluster and the Game Cluster )
- 1 Load Balancer for the Oracle Cluster
- 1 dedicated back end network infrastructure ( Gigagyte net, probably fiber)
- 1 Game Cluster ( around 100/200 blades, probably more )
- 1 Billing/accounting/authentication cluster ( several nodes .... 5/6 as it's common infrastructure )
- 1 Load Balancer infrastructure for the game server.
- 1 Front End Network ( common to all the servers )

Should be IMFO representative of what we call one single game server. ( say G-Land )

The number of blades in thee Game Cluster can be dynamically allocated to another server if needed... Eventually the Symetrix can be shared if they went for a big one, but it add complexity at the SAN and back end network.

not many companies can waste several hundred thousand dollars ( or euros ) for a test system.
And the few that can ( and do )  are usually way bigger than Turbine. ( lets say : in Telco : ATT, Verizon, BT, FT, DT, Telia, Vodafone, ... Ericsson, Alcatel-Lucent, Nokia-Siemens, Huawei, ZTE, ... HP, IBM, Oracle, SAP, ... )

A lot of that stuff can be mimicked virtually, but I am not sure if everything can be. Regardless, what they have been doing for the last 7 years seems to be working well for them.

This.

Hardware costs, yo. It is fun to watch the mouthbreathers apply their extensive knowledge of how the fries station works to all aspects of technology and science, though.

Turbine is owned by WB Interactive, a division of Time Warner.  142 on the Fortune 500.  Even if you ignore the size of the parent company, they were acquired for 160 million in 2010.  They had a staff of somewhere around 250-300 people, depending.  Those two facts place their gross revenue in a 30-40 million dollar range.  I'd guess their IT budget is somewhere in the 2+ million per year range.  More than enough to squeeze in a testing platform. 

Sorry.  I don't buy it.  I've worked at companies with revenues below 50 million a year in revenue and they had proper testing environments. 

This from the guy who nearly wet himself and cried like a girl because some guy on an internet forum told him he hacked his router.   Yes, bitch I want that supersized.