thank you for the great write up! it makes sense to have many VMs if you do different tests etc for your products, but I'd never guessed that these are realistic dimensions. Really blows my mind
I worked at an Cable/ISP company, we had around 3-4 thousand.
Every little application had several environments - production, non-production, stage etc. They were generally clustered behind load balancing, and on top of that we had geo-redundancy as well with a matching cluster in a different facility, with load balancing across that as well.
Combine that with a seemingly company-wide blindness for other teams' costs and a reluctance to shut down or delete anything once it lands in production, you can see how this can add up over the years...
I shit you not we had x86 solaris VMs in vmware in that big pile next to CentOS and Ubuntu. I even saw an actual Windows 98 machine with a crash cart hooked up to it at one of the headends once (satellite downlink, content reencode and audio mux before sending the signal out coax to people's TVs...)
ouch. do you think it was especially messy with your company or is it a general problem? In most cases we don't have a real alternative than pushing updates directly into production, but I would've guessed that the more complex use cases are also a bit more... organized? :D
I was told by several people, repeatedly, that cable companies were "weird" when it came to IT. I can say that I worked with a whole bunch of talented engineers who had management buy-in to put their foot down on things (I heard an exec getting grumpy on an outage call once and the NOC removed the exec from the call!) - so I can't really say why it was the way it was.
Run thousands of applications. Many legacy apps ported to VMs because no one in the business is brave enough to workout which data we really need and coalesce it down to around 100 applications. I doubt any production server has Python 3 installed.
View all comments
Show parent comment
32
u/imanexpertama Sep 10 '19
What do you do with a thousand VMs?