Page Index Toggle Pages: 1 2 3 [4]  Send TopicPrint
Very Hot Topic (More than 75 Replies) Player's Council Application Open (Read 7099 times)
Asheras
Completionist (i.t.p.)
******
Offline


This is why we can't have
nice things.

Posts: 10230
Location: Ohio
Joined: Jun 9th, 2010
Gender: Male
Re: Player's Council Application Open
Reply #75 - Feb 8th, 2021 at 7:28am
Print Post  
Flav wrote on Feb 7th, 2021 at 10:43am:
I know, but it doesn't means that they are not stringy on the hardware.
After all it costs some money tp recpnfigure the office space for LARP Pac-Man.

As for the Hosting, in Europe the servers were hosted in an AT&T datacenter in Amsterdam. ( both for DDO and LoTRO ), but that was long ago. Right now they are hosted in the Newark area. And the most obvious culprit there
is Equinix.

Since I work on and off with a MVNO that has it's network hosted by Equinix, I have a good idea of what they provide.

It can be barebone hosting ( you tell them how many rack cabinets you want and they set up a separate area for you with the racks and you fill them up with whatever you want, they only provide the environment and the external connectivity )
It can be full hosting ( you tell them what you need in term of RAM/CPU/Storage and they give you admmin access to something that meets your requirement, but you have no idea what the underlying hardware is )

or a few variations in between.
+

Summary:  You have no idea what kind of storage system they use nor if they have DBA(s) on staff.  Just making guesses.

Which is fine.  Theorizing to support confirmation bias on the internet is as old as the internet itself.

Just looking at it from a "What do we think, what do we know, what can we prove" aspect, we have their revenue as a fact.  So theorizing that they don't have the revenue to support a decent storage array or a DBA doesn't track.  That's all I'm saying.

If you keep your theorizing in line with the few facts that exist, people will buy it better.   Like you can say Jewish space lasers and sex trafficking pizzerias are a thing, but when you call school shootings and 9-11 pentagon plane attacks a hoax, you lose people. 




  
Back to top
 
IP Logged
 
Dragonborn
Shroud Slacker
***
Offline


Savaging the masses...
one dumbfuck at a time

Posts: 1243
Joined: Jul 25th, 2020
Re: Player's Council Application Open
Reply #76 - Feb 8th, 2021 at 7:46am
Print Post  
Oh that's gonna leave a mark, roflmfao!
  

noamineo wrote on Dec 24th, 2021 at 12:04pm:
Sadly only scammers are "selling" boxes now. But if you do want to get scammed, I know a guy.


OSeanessy wrote on Dec 26th, 2021 at 6:03am:
Just for the record, I bought ten stones from Dragonborn a few days back. Fair and civil trade which went smoothly
Back to top
 
IP Logged
 
noamineo
Completionist (i.t.p.)
******
Offline


All men fear time, but
time fears the pyramids

Posts: 9036
Location: Titan
Joined: Jul 2nd, 2014
Re: Player's Council Application Open
Reply #77 - Feb 8th, 2021 at 12:54pm
Print Post  
Flav wrote on Feb 6th, 2021 at 9:16am:
Do you really think they can afford that kind of disk array... They are probably still on physically rotating disks with SAS interfaces ( ok, I'll even give them the 15K version, and they might have, for some specific purposes, a few SSD ( SAS ) )

No way they are on an NVMe or a M2 infrastructure. It's just too expensive for a game company as small as SSG.


Oh no definitely not. I was saying "this technology exists" not "I think this is what SSG is using" sorry for being unclear.

As to what they are using all I can tell you is I didn't sell it them. The rest is just educated guesswork but the database definitely lives on a SAN. They might be using a single SAN for all the databases or one for each server. The database definitely doesn't live on the same VM host, to get the kind of I/O needed, you need a SAN.

Flav wrote on Feb 7th, 2021 at 10:43am:
Right now they are hosted in the Newark area.


Based on some of the dev comments it made it sound like the datacenter was actually at SSG headquarters.
  

I'll never understand the propensity of people to brag about being good at a video game. Its a toy you play with for fun. The only person who should be proud of you is your mother. If you're 3.
Back to top
 
IP Logged
 
Flav
Vault Frog
*
Offline


One Frog to Rule them
All!

Posts: 9986
Location: Land of the Frogs
Joined: Aug 29th, 2010
Gender: Male
Re: Player's Council Application Open
Reply #78 - Feb 8th, 2021 at 1:54pm
Print Post  
noamineo wrote on Feb 8th, 2021 at 12:54pm:
Oh no definitely not. I was saying "this technology exists" not "I think this is what SSG is using" sorry for being unclear.

As to what they are using all I can tell you is I didn't sell it them. The rest is just educated guesswork but the database definitely lives on a SAN. They might be using a single SAN for all the databases or one for each server. The database definitely doesn't live on the same VM host, to get the kind of I/O needed, you need a SAN.

Once upon a time in the logs you could find things like :
Code
Select All
World status servers:                                                                                23/08/2014 19:56:36
    Cannith @ http://gls.ddo.com/GLS.DataCenterServer/StatusServer.aspx?s=10.67.112.59               23/08/2014 19:56:36
    Wayfinder @ http://gls.ddo.com/GLS.DataCenterServer/StatusServer.aspx?s=10.67.112.53             23/08/2014 19:56:36
    Argonnessen @ http://gls.ddo.com/GLS.DataCenterServer/StatusServer.aspx?s=10.67.112.56           23/08/2014 19:56:36
    Thelanis @ http://gls.ddo.com/GLS.DataCenterServer/StatusServer.aspx?s=10.67.112.21              23/08/2014 19:56:36
    Sarlona @ http://gls.ddo.com/GLS.DataCenterServer/StatusServer.aspx?s=10.67.112.22               23/08/2014 19:56:36
    Khyber @ http://gls.ddo.com/GLS.DataCenterServer/StatusServer.aspx?s=10.67.112.23                23/08/2014 19:56:36
    Ghallanda @ http://gls.ddo.com/GLS.DataCenterServer/StatusServer.aspx?s=10.67.112.64             23/08/2014 19:56:36
    Orien @ http://gls.ddo.com/GLS.DataCenterServer/StatusServer.aspx?s=10.67.112.60                 23/08/2014 19:56:36
 



So it's obvious there's a backend side. And the database server(s) is living on that backend side.
From the above IPs it was probably a shared backend at that time... nowadays we can guess that
the backend network is virtualized and that the backend server resides on the same hardware as the frontend ones.
( with maybe one physical box per game server... which could explain some shennanigans like the 15 days rollback )


[quote author=7776787470777C76190 link=1611958065/77#77 date=1612806865]
Based on some of the dev comments it made it sound like the datacenter was actually at SSG headquarters.


well my traceroute says the contrary. It ends up with a Level 3 connexion somewhere in the Newark area.
Code
Select All
 2    15 ms    15 ms    14 ms  lac-net1.net1.bsmso651.rbci.orange.net [80.10.123.200]
 3    15 ms    14 ms    14 ms  10.123.109.10
 4    14 ms    14 ms    14 ms  ae41-0.niidf201.rbci.orange.net [193.252.98.161]
 5    15 ms    14 ms    15 ms  81.253.184.182
 6    15 ms    16 ms    15 ms  hundredgige0-1-0-11.partr2.saint-denis.opentransit.net [193.251.131.164]
 7    15 ms    16 ms    15 ms  ae-19.edge7.Paris1.Level3.net [4.68.73.61]
 8    92 ms    92 ms    93 ms  ae-6-3506.ear2.Newark1.Level3.net [4.69.214.37]
 9    95 ms    94 ms    94 ms  4.31.43.210
10     *        *        *     Délai d'attente de la demande dépassé.
11     *        *        *     Délai d'attente de la demande dépassé.
 



They used to have them hosted in a Datacenter in the Boston area, but they moved them a bit before Turbine spawned SSG and sold it to Daybreak.
  

Yes my avatar is an Hermine eating a Greenland Lemming for brunch.
Back to top
 
IP Logged
 
noamineo
Completionist (i.t.p.)
******
Offline


All men fear time, but
time fears the pyramids

Posts: 9036
Location: Titan
Joined: Jul 2nd, 2014
Re: Player's Council Application Open
Reply #79 - Feb 8th, 2021 at 2:32pm
Print Post  
Flav wrote on Feb 8th, 2021 at 1:54pm:
From the above IPs it was probably a shared backend at that time... nowadays we can guess that
the backend network is virtualized and that the backend server resides on the same hardware as the frontend ones.


An ordinary server won't have the non-sequential read/write speeds or the needed I/O to host a game with that many users. A SAN isn't just about lots and lots of space - by striping the data across many drives it makes it possible to read and write much faster than a single server even with a RAID6 array.


Flav wrote on Feb 8th, 2021 at 1:54pm:
well my traceroute says the contrary. It ends up with a Level 3 connexion somewhere in the Newark area.


Ah ok. Well that sounds on-brand. I bet they chose the colocation based entirely on the price. I can tell you some fun horror stories...
  

I'll never understand the propensity of people to brag about being good at a video game. Its a toy you play with for fun. The only person who should be proud of you is your mother. If you're 3.
Back to top
 
IP Logged
 
Asheras
Completionist (i.t.p.)
******
Offline


This is why we can't have
nice things.

Posts: 10230
Location: Ohio
Joined: Jun 9th, 2010
Gender: Male
Re: Player's Council Application Open
Reply #80 - Feb 8th, 2021 at 4:05pm
Print Post  
My feeling is that if they could fix the lag/performance issues by throwing money at hardware/infrastructure upgrades, that would have happened by now.  Again, looking at their revenues, cost doesn't need to be a concern.

I doubt the problem is as simple as "they are unsophisticated enough to run the app and db server on the same hardware" or "they don't have a DBA on staff who knows how to manage indexes".   Both of those are 90's and early/mid 2000's, pre-ITIL best practices and outdated storage  and N-Tier architecture tech issues. 

The one thing I look at from a facts standpoint is that they have been hiring DevOps people for a few years.   If you are building DevOps practices in house then that says to me that you are a bit more sophisticated than that.   

As for their Data Center, it may end in Newark or the trace may drop there.   That's not definitive.  It seems to end with Level 3.  They have facilities in Newark.   Level 3/Lumen is the largest datacenter provider in the US (not including clouded, like AWS and Azure, etc). 

  
Back to top
 
IP Logged
 
noamineo
Completionist (i.t.p.)
******
Offline


All men fear time, but
time fears the pyramids

Posts: 9036
Location: Titan
Joined: Jul 2nd, 2014
Re: Player's Council Application Open
Reply #81 - Feb 8th, 2021 at 5:44pm
Print Post  
Asheras wrote on Feb 8th, 2021 at 4:05pm:
My feeling is that if they could fix the lag/performance issues by throwing money at hardware/infrastructure upgrades, that would have happened by now. 


You dramatically under-estimate how expensive that equipment is. I don't recall if that posting a while back was gross or net, but DDO brings in something like $9,7 million annually, the upgrades would cost about $1 million, easy. And that's assuming SSG doesn't SSG it up badly.

There is a whole lot more to systems architecture than just "which number is higher". We are making fun because SSG's servers are probably still running on spinning disks but the reality is those spinning disks couldbe plenty fast enough, the bottleneck might be elsewhere.


Asheras wrote on Feb 8th, 2021 at 4:05pm:
Again, looking at their revenues, cost doesn't need to be a concern.


Thats where I want to know if that report was gross(all the money DDO is bringing in) or net(all the money - expenses, eg profit).


Asheras wrote on Feb 8th, 2021 at 4:05pm:
I doubt the problem is as simple as "they are unsophisticated enough to run the app and db server on the same hardware" or "they don't have a DBA on staff who knows how to manage indexes". 


They definitely have someone who knows their way around the database on staff. All the talk of DBAs was suggesting that whoever it is has more important things to do with their time than scrub player inventories. Simple as that.

Asheras wrote on Feb 8th, 2021 at 4:05pm:
 Both of those are 90's and early/mid 2000's, pre-ITIL best practices and outdated storage  and N-Tier architecture tech issues. 


You forget: DDO is based on Asheron's Call which was coded in.... you guessed it, the mid 90s. Just as your modern intel Core i9 CPU still has DNA left over from the venerable 8086, so too does DDO carry the seed of poor tech decisions.


Asheras wrote on Feb 8th, 2021 at 4:05pm:
The one thing I look at from a facts standpoint


Hey that's no fair. You're looking from a "facts" stantpoint while the rest of us are just making wild speculations. "Ooooh, look at me, I'm Asheras with my fancy facts! Ooh la la!" busy lording it over us.
  

I'll never understand the propensity of people to brag about being good at a video game. Its a toy you play with for fun. The only person who should be proud of you is your mother. If you're 3.
Back to top
 
IP Logged
 
Asheras
Completionist (i.t.p.)
******
Offline


This is why we can't have
nice things.

Posts: 10230
Location: Ohio
Joined: Jun 9th, 2010
Gender: Male
Re: Player's Council Application Open
Reply #82 - Feb 8th, 2021 at 7:38pm
Print Post  
noamineo wrote on Feb 8th, 2021 at 5:44pm:
You dramatically under-estimate how expensive that equipment is. I don't recall if that posting a while back was gross or net, but DDO brings in something like $9,7 million annually, the upgrades would cost about $1 million, easy. And that's assuming SSG doesn't SSG it up badly.


Hey that's no fair. You're looking from a "facts" stantpoint while the rest of us are just making wild speculations. "Ooooh, look at me, I'm Asheras with my fancy facts! Ooh la la!" busy lording it over us.


It’s a cap ex.   They don’t pay for it all in one year.  Over 10 years it’s 1% of gross.  Meh.  Tech companies only have two major expenditures .  Salaries and infrastructure.  If you don’t know how finance works, that’s not my fault.  (Btw, that’s a better example of being a duck/ lording it over someone.   My boobies was mostly just busting flavs balls.  Which anyone who has been around her long enough knows he and I do)

On the other:

A). The fair is something that comes to town once a year.   You sound like a child.
B).  I never said I was the only one using facts.  Try fixing your reading comprehension fail, if you can.    I said the only fact I was using for my comments.  Which include some speculation the same as everyone else.  I’m just not making speculation that they don’t have the money to do the simple stuff.   Which means either the simple stuff is happening or, if it isn’t, it’s not happening for a reason that isn’t “too poor”.    Try to keep up. 



  
Back to top
 
IP Logged
 
Asheras
Completionist (i.t.p.)
******
Offline


This is why we can't have
nice things.

Posts: 10230
Location: Ohio
Joined: Jun 9th, 2010
Gender: Male
Re: Player's Council Application Open
Reply #83 - Feb 8th, 2021 at 7:39pm
Print Post  
Lol.  Ducking word sub still getting me after 11 years.  With no edit ability.   Angry Grin Cheesy
  
Back to top
 
IP Logged
 
Shunned
Stormreaver Piker
*
Offline


Hi! You have multiple
personality disorder.

Posts: 528
Location: Behind the edit button
Joined: Nov 21st, 2020
Re: Player's Council Application Open
Reply #84 - Feb 8th, 2021 at 8:05pm
Print Post  
Asheras wrote on Feb 8th, 2021 at 7:38pm:
My boobies was mostly just busting flavs balls.  Which anyone who has been around her long enough knows he and I do)


I am so gender confused.  Tits, balls, her, he, wtf?
  
Back to top
 
IP Logged
 
Charon
Ex Member


Re: Player's Council Application Open
Reply #85 - Feb 8th, 2021 at 8:57pm
Print Post  
Shunned wrote on Feb 8th, 2021 at 8:05pm:
I am so gender confused.  Tits, balls, her, he, wtf?

Quiet you'll set off a vault orgy.
  
Back to top
 
IP Logged
 
noamineo
Completionist (i.t.p.)
******
Offline


All men fear time, but
time fears the pyramids

Posts: 9036
Location: Titan
Joined: Jul 2nd, 2014
Re: Player's Council Application Open
Reply #86 - Feb 8th, 2021 at 9:58pm
Print Post  
Asheras wrote on Feb 8th, 2021 at 7:38pm:
It’s a cap ex.   They don’t pay for it all in one year.  Over 10 years it’s 1% of gross. 


It is not over ten years, man. A storage system is 3 years. 5 at the absolute max. Outside of MAYBE switches or high level telco equipment, nothing in the enterprise world stays in support for 10 years. You buy a server today you better already be shopping the the replacement. (or know how to support it yourself)


Asheras wrote on Feb 8th, 2021 at 7:38pm:
On the other:

A). The fair is something that comes to town once a year.   You sound like a child.
B).  I never said I was the only one using facts.  Try fixing your reading comprehension fail, if you can.    I said the only fact I was using for my comments.  Which include some speculation the same as everyone else.  I’m just not making speculation that they don’t have the money to do the simple stuff.   Which means either the simple stuff is happening or, if it isn’t, it’s not happening for a reason that isn’t “too poor”.    Try to keep up. 


Yeah I deserved that. It was a pathetic attempt at humor. I concede that it was dumb.
  

I'll never understand the propensity of people to brag about being good at a video game. Its a toy you play with for fun. The only person who should be proud of you is your mother. If you're 3.
Back to top
 
IP Logged
 
Flav
Vault Frog
*
Offline


One Frog to Rule them
All!

Posts: 9986
Location: Land of the Frogs
Joined: Aug 29th, 2010
Gender: Male
Re: Player's Council Application Open
Reply #87 - Feb 9th, 2021 at 2:45pm
Print Post  
Asheras wrote on Feb 8th, 2021 at 7:38pm:
My boobies was mostly just busting flavs balls.


OOOOOH !!! I'd loooooove to get my balls busted by your boobies !!!! Grin Grin Grin Grin Grin
  

Yes my avatar is an Hermine eating a Greenland Lemming for brunch.
Back to top
 
IP Logged
 
Flav
Vault Frog
*
Offline


One Frog to Rule them
All!

Posts: 9986
Location: Land of the Frogs
Joined: Aug 29th, 2010
Gender: Male
Re: Player's Council Application Open
Reply #88 - Feb 9th, 2021 at 3:01pm
Print Post  
noamineo wrote on Feb 8th, 2021 at 2:32pm:
A SAN isn't just about lots and lots of space - by striping the data across many drives it makes it possible to read and write much faster than a single server even with a RAID6 array.


I agree, and the SAN can be attached to the server through the backend... or through a deidcated interface.
here wwe can only speculate. but my gut guess is that the DB SAN array is directly connected to the DB servers through a dedicated SAN.


noamineo wrote on Feb 8th, 2021 at 9:58pm:
high level telco equipment, nothing in the enterprise world stays in support for 10 years.


hmm, you have no idea what telcos are able to pull off... I know one French/internationnal opeator that has 20+ years old HP9000 D Series HP-PA Servers for the national PSTN supervision and O&M. they have been out of support ( the solution in itself not the hardware ) for more than 15 years now. ( the hardware has been OOS for longer ).
But two years ago they came to us ( well it ended on my desk as the last still there that knew the system ) as they wanted to move the servers from a Datacenter in Lyon to one in Val de Reuil... and 20+ years old hardware might not take well a power off, a 600km plus trip and a power on. They wanted to know what we could give them as alternative in case of failure [ we didn't have anything decent to offer, except helping them getting the thing virtualized ]

Lifetime of Circuit switching telco equipment is in the order of 30+ years
Lifetime of packet switching equipments ( that are not just there to transport stuff ) is 10 to 30ish years. ( the time it takes to get the two next Mobile G up. With 5G we will be ditching the last remants of 2G )
lifetime in the ISP/ Internet access goes from less than 5 years ( triple play boxes ) to 10 ish years [ P Routers, RR and so on. ]

But for all the annexes ( Charging, Billing, Network Supervision, and so on ) we have times of life similar to the It world.

  

Yes my avatar is an Hermine eating a Greenland Lemming for brunch.
Back to top
 
IP Logged
 
noamineo
Completionist (i.t.p.)
******
Offline


All men fear time, but
time fears the pyramids

Posts: 9036
Location: Titan
Joined: Jul 2nd, 2014
Re: Player's Council Application Open
Reply #89 - Feb 9th, 2021 at 10:15pm
Print Post  
Flav wrote on Feb 9th, 2021 at 3:01pm:
I agree, and the SAN can be attached to the server through the backend... or through a deidcated interface.
here wwe can only speculate. but my gut guess is that the DB SAN array is directly connected to the DB servers through a dedicated SAN.


Depending on exactly how much processing power a DDO server requires, its entirely possible the virtual servers are in fact running directly on the SANs themselves. I highly doubt this, but its feasible. More likely all the VMs associated with a DDO server are running on a redundant blade system with dual multicore processors and a ton of RAM on the same rack as the SAN holding the database. Wither its going through an IB switch or direct, the interface is probably fiberoptic.


Flav wrote on Feb 9th, 2021 at 3:01pm:
hmm, you have no idea what telcos are able to pull off... I know one French/internationnal opeator that has 20+ years old HP9000 D Series HP-PA Servers for the national PSTN supervision and O&M. they have been out of support ( the solution in itself not the hardware ) for more than 15 years now. ( the hardware has been OOS for longer ).
But two years ago they came to us ( well it ended on my desk as the last still there that knew the system ) as they wanted to move the servers from a Datacenter in Lyon to one in Val de Reuil... and 20+ years old hardware might not take well a power off, a 600km plus trip and a power on. They wanted to know what we could give them as alternative in case of failure [ we didn't have anything decent to offer, except helping them getting the thing virtualized ]

Lifetime of Circuit switching telco equipment is in the order of 30+ years
Lifetime of packet switching equipments ( that are not just there to transport stuff ) is 10 to 30ish years. ( the time it takes to get the two next Mobile G up. With 5G we will be ditching the last remants of 2G )
lifetime in the ISP/ Internet access goes from less than 5 years ( triple play boxes ) to 10 ish years [ P Routers, RR and so on. ]

But for all the annexes ( Charging, Billing, Network Supervision, and so on ) we have times of life similar to the It world.


noamineo wrote on Feb 8th, 2021 at 9:58pm:
Outside of MAYBE switches or high level telco equipment, nothing in the enterprise world stays in support for 10 years.


See I think you may have missed the part where I directly said "high level telco equipment can stay in service for a longass time". Its ok, I'm aware I'm not great with words. Love the stories though, have any more? For some reason I'm a big fan of weird tech tales.

But yeah, game servers actually stay in service longer than they should. Typical lifespan for a normal database/app server is about 3 years. Thats why virtualization was so attractive: you can more easily transition the VMs than software running on metal. The real problem is support contracts. I've been in enterprise support for like 6 years now and believe me - sysadmins are LOST without guys like me.

Don't let the fancy job description fool you - if you can dial a telephone you're qualified to run a datacenter.
  

I'll never understand the propensity of people to brag about being good at a video game. Its a toy you play with for fun. The only person who should be proud of you is your mother. If you're 3.
Back to top
 
IP Logged
 
Page Index Toggle Pages: 1 2 3 [4] 
Send TopicPrint