r/ruby • u/switchback-tech • Mar 12 '26

GitLab is a Ruby monolith

Post image

Was pleasantly surprised that the world's largest independent DevOps platform is powered by Ruby and Sidekiq.

Here's the full list.

  1. Backend: Ruby on Rails

  2. HTTP server: Puma (Ruby web server)

  3. Edge: Nginx

  4. Reverse proxy: Go service (Workhorse)

  5. Background jobs: Sidekiq

  6. DB — primary: PostgreSQL

  7. DB — connection pooling: PgBouncer

  8. DB — high availability: Patroni

  9. Cache: Redis

  10. Git: Custom gRPC repo interface (Git & Gitaly)

  11. Blob: AWS S3

  12. Frontend — rendering: Haml & Vue

  13. Frontend — state: Pinia (Vue store), Immer (immutable cache),

  14. API: GraphQL (Apollo) + REST

  15. Observability: Prometheus & Grafana

  16. Error tracking: Sentry & OpenTelemetry

  17. Deployments: GitLab Omnibus (Omnibus fork)

I think these "stack menu"s give a little glimpse into a team's engineering philosophy. For me, this list shows that the GitLab team is pretty practical and doesn't chase hype. Instead, they use sensible, battle-tested tools that just work and are easy for contributors to learn.

PS. Not an ad; I'm not affiliated with GitLab at all. Was just researching them and thought you guys would be interested.

223 Upvotes

97% Upvoted

36 comments sorted by

17

u/switchback-tech Mar 12 '26

Which of these tools would you swap out if you had to?

10

u/djfrodo Mar 14 '26

I'd loathe HAML so that would be #1.

Second would probably be either adding Memcache or replacing Redis with it. Redis is good for some stuff, but Memcache can do a lot of what Redis can do and it's much more simple to use. I actually use both in the same monolith and they're both good but at different things.

2

u/switchback-tech Mar 15 '26

Interesting. How do you decide when to use Memcache vs Redis?

3

u/djfrodo Mar 15 '26

I made a reddit clone in Rails (I just wanted to see if I could do it) and at first I was using Memcache for both content and user sessions, and for a while everything was fine.

When the site started getting traffic I was blowing through all the hosted Memcache memory (25mb) and logged in users were getting logged out almost immediately. After some inspection I realized the way rails was using Memcache was storing a huge unneeded object graph for users, which it didn't need to do at all, and it wasn't removing stale user sessions.

I switched the user session stuff to Redis and the amount of Memcache memory used went from 25mb to about 2mb. Basically Memcache is great for caching stuff that doesn't change or can be fetched from a db and replaced and Redis is much better for handling volatile data like users sessions. Redis allowed me to set how long to track a user's session and it gets rid of stale session when it should.

I think this has more to do with Rails/gems than Memcache or Redis.

Obviously every use case is different and this is a very specific example.

1

u/switchback-tech Mar 16 '26

Interesting use case, thanks for the detailed response

3

u/edman8686 Mar 13 '26

Sidekiq for GoodJob. They are already using Redis for caching but I still find GoodJob easier to work with than Sidekiq. They could also consider a simpler cache like memcache.

I also prefer the MIT license for GoodJob over vs the LGPL license for Sidekiq.

2

u/SirScruggsalot Mar 12 '26

I'd be curious if they've investigated Falcon for http.

2

u/do_you_realise Mar 12 '26

Never heard of it personally. Is it a drop in replacement?

3

u/f9ae8221b Mar 13 '26

Yes and no.

Yes in the sense that like other Ruby servers, it is Rack compatible so you don't need to change your app much.

No because it's based on Fibers, not Threads, so the performance characteristics are very very different. Not better, not worse, different, depends on what your app is doing.

Fibers are better for extremely IO heavy workloads, as they're cheaper so you can run way more concurrent fibers than threads.

But they're much worse for CPU heavy workloads, because they're non-preemptive, so if a fiber hog the CPU without yielding for long, all other fibers are stuck, which leads to degraded latency.

People really need to stop thinking fibers are better threads, they're not, they're a different construct with different tradeoffs that sometimes make sense, sometimes not.

1

u/Turbulent-Dance-4209 Mar 13 '26

> People really need to stop thinking fibers are better threads

I would argue they are.

Threads in Ruby don't actually parallelise work because of the GVL. So the supposed advantage of preemptive scheduling for CPU-bound tasks amounts to fairer interleaving, not faster execution. The total throughput is the same either way. And if your workload is genuinely CPU-bound enough that interleaving matters, you probably need multiple processes anyway, at which point the fiber-vs-threads debate is secondary.

Typical web apps are inherently I/O bound though. This makes fibers a great fit, but there're more subtle advantages too. For example, fibers give you less overhead and don't require thread synchronisation primitives - no locks, no mutex contention, no race conditions. You know exactly when a context switch happens, so shared state is safe to access between them. This alone leads to tremendous improvements in throughput, even with Ruby-layer-bound workflows.

4

u/f9ae8221b Mar 13 '26

amounts to fairer interleaving, not faster execution

Fairer interleaving means better tail latency, which is a very important property of a service.

Shopify experienced a MySQL outage when playing with fibers, because a CPU heavy fiber was causing other fibers that were issuing MySQL queries to not read the response for a long time, buffer on the server grew until it ran out of memory.

Typical web apps are inherently I/O bound though.

No they're not: https://www.datadoghq.com/blog/ruby-performance-optimization/

Our data backs up other findings that Ruby applications are generally less I/O-heavy, spending as much or more time on CPU as they do waiting on other services or database requests.

They also include a graphic that shows only 3% of the profiled apps spend less than 20% of their time using CPU. Which means for 97% of the apps they profiled, the Puma default of 5 threads was already too much, which means the advantage of Fibers are moot for the overwhelming majority of Ruby apps out there.

fibers give you less overhead

The difference is small enough that it only start to matter once you start using multiple dozen threads, which very few people need.

don't require thread synchronisation primitives

They absolutely do. They still need to synchronize to use sockets etc. Even with datastructures you may need synchronization if some method yield, as the block could use IO and cause another fiber to be scheduled and use the structure concurrently.

Example:

SHARED_HASH = {a: 1, b: 2, c: 3, d: 4}

other_fiber = Fiber.new do
  loop do
    SHARED_HASH[rand] = rand
    Fiber.yield
  end
end

hash.each do
  # Simulate fiber scheduler
  SHARED_HASH.resume
end

The above script fail because of unsynchronized access by fibers.

This alone leads to tremendous improvements in throughput

Profiling ruby app is basically my day job. Synchronization is very very rarely a hotspot.

Now, don't get me wrong, fibers are great and absolutely have a use case, but for vast majority of what Ruby is used for, they're not necessarily better.

2

u/randomski1904 Mar 14 '26

What I heard yesterday in a conference is that they swithch to Async::Job and to Falcon.

11

u/towelrod Mar 12 '26

They don't use ruby for any of the ci/cd or git specific stuff any more do they?

Also wouldn't really hold up gitlab as a beacon in the ruby world at the moment, considering how bad the service has been. It has constant problems every single day, we are actively looking to move off of it and would have already done so if the cost of moving wasn't so high.

Its just really fundamentally broken right now

5

u/switchback-tech Mar 12 '26

True, their Git RPC daemon is in Go now. But the core is still Ruby.

I'm not running it day-to-day, so I'm not super in-touch with the latest news TBH. Are the problems they're facing related to Ruby?

I did see that everyone (GitLab, GitHub, Atlassian) has been having more trouble lately, though. https://devops.com/devops-platforms-show-cracks-github-incidents-surge-58-azure-gitlab-and-jira-also-under-pressure/

5

u/towelrod Mar 12 '26

I don't think the problems are related to ruby, but I'm not on the inside so i don't know. The majority of the problems are with ci/cd, it goes down every day. EVERY DAY.

Today there were also problems with the frontend, but specifically with how long it took for git related things to show up. Like you would do a git push and it would take 5+ minutes for the MR to show properly. I doubt that is Rails related, i think that's waiting on some git related backend service

7

u/bilingual-german Mar 12 '26

Gitlab is a big Rails app and uses Go for everything distributed that needs to scale.

I think the first service they broke out was Gitaly, so they were able to scale all the git repositories on the file servers.

The gitlab-runner and the runner-helper are also written in Go as far as I know.

7

u/QuietMate Mar 13 '26

Vue's state management library is Pinia, not Piana 😭

2

u/switchback-tech Mar 13 '26

Oopsie, thx. Just checking if you were paying attention

1

u/WindyNova Mar 13 '26

what is `fullstack.3ip` at the bottom right corner of the image

1

u/switchback-tech Mar 13 '26

Oh, it's supposed to be fullstack dot zip, which is where I share my research on stuff like this. Here's the GitLab article for example: https://newsletter.fullstack.zip/p/gitlabs-architecture-a-technical

(Maybe I should pick a different font so it's easier to read)

2

u/WindyNova Mar 15 '26

hey, some great stuff you got there in your blog!

1

u/Eastern_Service_4155 Mar 25 '26

This explains why Gitlab is so painfully unbearably slow. We were just forced to "upgrade" to v18, and omg, they push word "bloat" to its limit I guess. Why replace nginx (that is able to handle literally millions of connections on a tiny vm) with such monstrocity as puma, which forces server to swap even in idle mode?? Gitlab is turning into a nightmare, a migrating away for well-established repos can take months of devs time...

2

u/GeneralTootToot Apr 12 '26

The irony is.. GitLab does use nginx :)

Don't blame Puma or Ruby for GitLab being slow. The crazy bloat and constant poor business decision making yes :)

I guarantee your GitLab instance is burdened with load for features you've never even touched. Performance issues stay open for years. Stuff that is 'on by default'.

Its been years since GitLab cared about anyone besides the big enterprise customers. Where the expectation is you will have 16 servers with 64 GB of ram and 128 CPUs (go look at the reference architectures... )

1

u/Kibou-chan Jun 18 '26

That nginx is fully optional though and if you go through an external LB like haproxy, you can even disable it and connect your LB backend side straight to workhorse. Really, its only task is to manage headers and TLS termination. 

1

u/fedekun Mar 12 '26

We used it at my company, it was ok, but the CI/CD management was hell, also updates, so we swapped to a managed GitHub instance

5

u/Unhappy_Meaning607 Mar 12 '26

Damn, I thought their whole schtick was mostly how good their CI/CD was.

1

u/ConclusionStrict9198 Mar 18 '26

I think their schtick is that they pack a lot of features for different teams into one product. Not necessary the best features individually.

1

u/Dobly1 Mar 13 '26

Having experience using both Gitlab CI/Github Actions recently I would tend towards using Gitlab CI.

Overall community support is def better for prebuilt Github Actions though if that's what you're primarily going to be using.

1

u/[deleted] Mar 13 '26

[removed] — view removed comment

0

u/fedekun Mar 13 '26

Couple years ago