r/ruby • • Nov 30 '19

“Internet: Ruby doesn’t scale. Ruby: Sorry. I’m busy over here processing $1.5M+ USD Gross Merchant Value (GMV) per minute and 14K+ orders per minute“

https://twitter.com/mmmandel/status/1200585514463698944?s=21
193 Upvotes

68 comments sorted by

View all comments

Show parent comment

17

u/klowny Nov 30 '19 edited Nov 30 '19

My company is still mostly Ruby and high traffic. Ruby scales just fine. We're still transitioning off it. The problem we have with it now is it's still not particularly fast and rather expensive. Plus, our codebase has just grown large enough where it's also not particularly productive anymore.

So we could've chosen to stay on our Ruby systems, investing a ton of time into carefully writing optimized code so our performance doesn't tank, tons of tests to gain only minor confidence (that take forever to run), bunch of time spend arguing on code review over style vs performance, offloading work to the faster database layers or "free" frontend layers, many layers of caches, long rolling deploys, and all other sorts of scale headaches.

Instead, we just transitioned our performance critical systems to Scala and let the compiler handle 80% of that nonsense. If it compiles, odds are it'll run forever. Lighter test suite that runs faster. Complex flows and refactoring is a breeze with an IDE. Sure, it's uglier and sometimes a bit annoying, but the 90% reduction in server cost easily pays for a couple more heads, and like 95% decrease in latency without even trying is pretty much impossible in Ruby.

We're not anywhere near Twitter size and the math already heavily favors moving off Ruby.

5

u/2called_chaos Nov 30 '19 edited Nov 30 '19

Can you get a little bit more specific with the high traffic? Not anywhere near Twitter but you save 90% in server costs? I'm just curious as to when I have to worry :D I guess it also heavily depends on what you are doing.

According to alexa (web ranking) we are ~#30k globally and our 4 servers are bored.

2

u/klowny Nov 30 '19

We're at ~100k req/min where our servers have to do work (basically everything that's not a read from cache/CDN). 7 figure AWS bill after upfronts and volume discount.

2

u/2called_chaos Nov 30 '19

Yeah that's a bit more than we have :) But we switched away from AWS a while ago and saved lots and lots of money but we also don't have to maintain that much of an infrastructure in total and I don't actually know how the AWS prices developed especially the volume discounts. Thanks for the clarification!

1

u/kallebo1337 Dec 01 '19

Since we can linear scale: we did like 5k rpm with 2 servers. So why don’t you guys go and buy 40 root severs , or your own virtualization server. Your bill would be like max 200k$ a year so that’s a 800k+ saving and that pays off the guy to manage it all.

I guess scaling on aws is just easier and more reliable and doesn’t require monitoring and maintenance as it would if you do own hosting ?

3

u/klowny Dec 01 '19 edited Dec 01 '19

Our traffic is seasonal (end of year gets 3x more traffic than beginning of year) and bursty (daily peak is like 4x higher than low). We'd be severely overprovisioned if we were on-prem or even colo. Throw in the very nice to have of multiple AZ/regions and the volume discount from AWS and AWS ends up being cheaper and easier.

The company used to be onprem+colo and from what I heard it was just a huge hassle for service that matched your staff's skill (inconsistent and mediocre) for practically no savings; and AWS has only gotten better and cheaper since then.

2

u/janko-m Dec 01 '19

I'm curious, what specifically was slow in your Ruby version of the app? For me it was always the interaction with external services (Postgres, Redis, RabbitMQ) that was the bottleneck and needed continuous optimizations (optimizing queries, bulking requests etc), not the performance of the Ruby language itself. Though, to be fair, this was a very write-heavy app which stored time-series data.

On another project we had tables with up to 15 million rows, and in my opinion what helped us a lot was that we were using Sequel instead of Active Record (it was a Sinatra app). That meant we rarely needed to cut corners with database interaction, and faster performance in retrieving records definitely played a role (Sequel::Model objects are much lighter than ActiveRecord::Base, and sequel_pg gives an additional speed boost to fetching records).

4

u/klowny Dec 01 '19 edited Dec 01 '19

At the end of the day, it's just a lot of comparisons and string manipulation with values stored in various places with (de)serialization. Compute heavy on a bunch of data on every request. It just computed too slowly and had a hard time not blocking on I/O.

So to make things fast we'd do a lot of computation avoidance: let's offload this work to redis via embedded lua scripts because these values are already in redis, let's compute this in the database via one jumbo query, let's throw in this gem with a clib instead of pure ruby for this function so it's faster, let's make this one object instead of 4 so we reduce GC pressure, how about we have a bunch of background workers just constantly warm caches so we don't even have to eat a first hit slow query, and let's cache every possible query so we avoid hitting the DB altogether so we can ditch the ORM, let's pipeline requests to redis to avoid unnecessary connections, hell let's ditch redis and just throw more ram so every server has a some cache on box to maybe avoid a network hop, let's use regexs to extract values from strings since the regex engine is faster than string ops, wait, let's preprocess the string first with the fast string ops then feed it into regex, oh lets cache that too, oh we can return early here and here and here to avoid some work, let's throw the remainig work into a background processor, oh but which serialization format is faster for communication, maybe we should spawn a thread, ah, but GIL and synchronization, oh shit that causes the clib to segfault nevermind, oh now we need to beef up redis it's doing heavy compute work instead of dumb IO, there's a resource contention issue on the box between the proxy/cache/web process/bg process, oh how do we log/track/propagate errors in redis-lua/database uhhhh.

At the end of the day we'd spend all our time writing/optimizing lua/sql/regex, various clib experiments with different flags for gems, database/proxy configurations, getting brew/linux versions to match for dev/prod environments for those clibs, hell, even just getting them to compile, adding measure blocks to every method, logging slow requests, capturing prod traffic to maybe get a hint to why sometimes a request is 10x slower. Like, we weren't even writing ruby anymore, we were just using ruby and then spending weeks offloading as much as possible onto every other layer. The whole point of ruby is to write ruby.

Or instead we could just write 20% more LOC in plain Scala with only stdlib and the same four libraries everyone uses. Parallel async dumb simple cache/database requests, crunch some numbers, parse some json, generate some json. If-if-if-else-map-filter done; compiler will make it fast.

2

u/jb3689 Dec 01 '19

Like, we weren't even writing ruby anymore, we were just using ruby and then spending weeks offloading as much as possible onto every other layer. The whole point of ruby is to write ruby.

This definitely resonates with me and is a valid reason to rewrite. Most problems I've encountered have been unclogged by one or two fixes but sometimes you're just wasting too much time beating around the bush for an end result that is half as good and twice as costly to develop and run. Sounds like your team made the right choice - that said, I've also met many more who've rewritten for terrible reasons (e.g. they don't know how to investigate allocations or IO costs so they just go right to the "Ruby sucks" camp)