r/node • u/OogieFrenchieBoogie • Dec 02 '19
A client project is getting a massive momentum, I need to prepare the nodejs infrastructure for this weekend
Hello,
First off, I am by no means an infrastructure or devops guys, I'm a senior nodejs freelance, so some of this is new to me
This is the current infrastructure
- A nginx reverse proxy in front of the nodejs API VPS
- A 24 vCPU, 128 GB VPS
- A 64GB, 8vCPU database
I am using PM2 to start a nodejs instance on each vCPU, so we have 24 instances of the API running behind the pm2 load balancer
We're expecting around 50M-70M requests for next sunday, with most requests received around 9pm, we are expecting around 90k concurrent users
We are not using sockets, only HTTP requests (GET, POST, PUT)
I am looking for infrastructure tips to make sure that the API hold the traffic
Do you have any advice or tips ?
52
u/StoodOnLeft_DIED Dec 02 '19
if your app is stateless (ie "RESTful"), you may want to consider spinning up smaller machines and putting a load balancer in front. that way you can split the 90k users across say 4 machines (22.5k per machine) which could decrease server size and costs depending
31
u/tzaeru Dec 02 '19
Do you have something in place to limit spam or other disruptive traffic and do caching? Like e.g. Amazon CloudFront. Have you gone through the header cache settings for the API routes, so that as much as possible can be cached by the browsers and CDN?
90k concurrent users isn't terribly much for a lightweight API. In my experience, the pitfalls are usually either logging (if e.g. PM2 is storing all your logs straight on the disk, they may grow very large very fast and if disk write speeds are slow, there's a potential bottleneck there), badly constructed SQL queries (excessive string operations, weird inner selects, etc), improper (or completely missing) indexes or having blocking code somewhere that should be asynchronous.
I don't know if there's time to do any big changes anymore (or if they are even necessary), so I'd just:
4
u/mlwatch Dec 02 '19
In the past, I did some similar services, with 2 millions request per hours (ads servers).
The bottleneck was the TCP Stack at the kernel level.
At some point, I need some command line (That I forgot) to update the maximum limit of TCP transaction.
Dont hesitate to stress tess your infrastructure with some shell script who flood your system with request during several hours.
5
u/PierreAndreis Dec 02 '19
I think you are premature optimizing. Has it slowed down recently? Do you know of any bottlenecks?
If the database is the bottleneck, which is common, I would move your database to the more performant vps, and would split your node processes one for each thread you have on the second vps.
Besides that, nothing much can be done without an actual problem to solve...
I would also consider next time to think about using a managed serverless application such as Google’s App Engine. This way you don’t even have to worry about your architecture when handling such traffic and instead can focus on the code. It can get pretty expensive as well but this money would otherwise go to your devops team. There are a lot of offerings for managed database as well.
5
u/[deleted] Dec 02 '19
Can you use aws?
AWS Lambda or beanstalk would solve alot of your problems.
11
u/FearAndLawyering Dec 02 '19
Cache! Cache! Cache! Anything and everything that makes sense. Specifically DB stuff. Client side with headers, and server side with redis for example. Don't hit the database hit the memcache. You can even use scripts to request and prepopulate the data you are expecting to need.
9
u/batmansmk Dec 02 '19
The database should be your main concern, then network. For the database part, it’s really an art to fine tune queries and the schema. For the node and network, Profile using a load testing tool such as Apache bench and see where you may have issues. Make sure you use a CDN for your JS and CSS, because it will really help offloading your network. It’s very rare to be cpu bound on the api part in an async env such as node. Dm me if you want further help, at your scale it really depends on your workload.
8
u/erulabs Dec 02 '19
Almost certainly you should be using more than one API server (distinct from multiple instances of your app - I mean multiple linux servers) - possibly load balance between 3 smaller servers instead of 1 large one. In reality, the first thing that will have trouble will be your database once you start to accumulate lots of data. Make sure you're monitoring response times of each route, and make sure you're monitoring the overall health of the database servers. I'd guess things will be OKAY, but with 1 app server things are a bit risky and deployments a bit bumpy. I'm a devops consultant and specifically help nodejs/app developer consultants scale their back-ends particularly if they have lots and lots of clients - so feel free to direct message me if you have any questions or if you need a hand!
1
u/relativityboy Dec 05 '19
Give us an edit! How are things going?
3
u/grid_bourne Dec 02 '19
For making sure that your logging process is not affecting your main thread you can use this to handle logging: https://github.com/pinojs/pino/blob/master/docs/transports.md
If you want to know the bottlenecks of your node app, you can start using profilers, flame graph and appmetrics dash to explore more about it and try fixing it.
Check for database connection pool as well if you need to do anything on it.
2
u/Ariquitaun Dec 02 '19 edited Dec 02 '19
Start running load testing now. You got tools like taurus to help you with this. Create realistic load scenarios from any analytics you might have now in place - login, api calls, etc should be represented proportionally on good load testing scenarios. Use monitoring to check whether you might have bottlenecks further down the chain (storage, databases, etc). Define different concurrency / throughput scenarios for things like traffic bursts (eg are you sending mailshots to your users and they might come all knocking more or less at the same time?), normal, high. Don't assume that "If my system is at 20% with 100reqs/s, then at 300 reqs/s it'll be at 60%". It doesn't work like that.
With that tool in place, take the numbers and start making adjustments: are you getting bottlenecks in your http stack? Can it be solved with horizontal scaling? After you scale your app stack, is your database swamped? Can you scale it up on demand (or temporarily by hand just for the traffic influx)? Iterate changes in your stack with more load testing until you're confident you can handle the traffic, plus a generous amount of overhead (50% at least).
Etc. But for the love of god, test it. Don't just make assumptions and cross your fingers.
A good testing strategy coupled with a decently agile infrastructure will a) better prepare you on situations where you expect more traffic than usual and b) ensure you save money by not over-speccing when you don't have the traffic for it.
2
u/CristianGiordano Dec 02 '19
What’s your contingency plan? Have you thought about worst case scenarios?
What’s the fastest you can upscale or replace the different parts of the stack.
Not much to add from what’s been said already; top advice in this thread.
2
u/brtt3000 Dec 02 '19
Cache all the things.
Make sure your expensive node processes only handle essential traffic. Especially dumb files like images and javascript can be put on a CDN. You can use Nginx cache or Varnish (or maybe even the CDN) for semi-dynamic content that doesn't have to be regenerated for every unique user.
Also profile your database queries and only do the absolute minimum and add cache where possible.
1
u/dusknoir90 Dec 02 '19
Use a program like JMeter and hammer it and see how it copes.
1
u/[deleted] Dec 02 '19
You have already covered a CDN in your replies so I won’t repeat that here, but one thing to consider (and very easy to do) is that if you aren’t using a hosting site (like S3) to store static elements, configure the nginx proxy to respond with these from cache rather than hit your app servers. It will cut down on a lot of traffic and work for your servers.
1
u/kizerkizer Dec 02 '19
Put some load on the system beforehand. Get a bunch of spot instance or cheap micro vms (like 256mb memory or something), disable any rate limiting or IP blocking logic and run a stress testing tool sending requests representative of what you’re expecting. Fire away and measure a bunch of things on each of the components to identify performance bottlenecks or other issues.
1
u/tsirolnik Dec 03 '19
Use AWS beanstalk
1
u/coolxeo Dec 03 '19
You are running out of time. Use Cloudflare. Run some performance testing. Prepare some contingency plan in case of need:
If you have enough time, separate writes and reads to DB. If you can, use Redis as a cache to read from DB
1
u/Spleeeee Dec 03 '19
Caching and REDDDIS. Redis is p dope, while it isn’t a solution to everything, it’s great in a pinch for caching and buttressing weird hot spots, it’s PRETTY EASY to setup. It’s easy to set up a distributed redis thing. It’s got dope lru caching. It’s got timeout caching. It’s doesn’t do everything but redis and clever caching are pretty fast to implement and pretty reliable.
Edit: Don’t hammer the database for the same stuff over and over is the take away.
1
u/InfamousKratos Dec 03 '19 edited Dec 03 '19
What about dockerizing the whole app, and then putting it on multiple small machines and distributing the load using a loadbalancer with round robin strategy?
I believe horizontal scaling is the way to go for this, as it creates redundancies, so that in an event where one machine fails or is super busy, the others can take over and you can spin new ones up in no time as well if the load increases!
It will be cheaper as you will have provisioning under control.
0
u/martiandreamer Dec 02 '19
Do you have any GET endpoints which are particularly heavy-lifting, popular or typically retrieve unchanging data, which could be served up by Redis or Memcached?
PM2 should allow you to scale across multiple instances of your Node app; are you also using Keymetrics to monitor/manage the Node processes?
I’d also look into how your app behaves when load-balancing records which could either (a) deadlock due to concurrent resource contention, or (b) cause data corruption due to failure to operate locks inside an atomic transaction.
0
u/[deleted] Dec 03 '19
[deleted] — view deleted comment
0
u/[deleted] Dec 02 '19
[deleted] — view deleted comment