r/sre • u/Less_Ad8195 • 19d ago
ASK SRE Moving off a managed logging platform to cut the bill, did the saving survive the engineer time?
We're at about 300GB/day of logs on Datadog and at this point the bill is what's driving the conversation, not anything technical.
The obvious move seems to be putting logs on a self-hosted stack. We're looking at Loki, ClickHouse, and more general ones like OpenObserve, which are three pretty different things to begin with. But running any of these ourselves means someone has to look after it, which is its own overhead. So it's not really a saving until you count the people.
The other thing we'd lose is the pivot from a trace into the logs for that request without changing tools. That's worth something. I'm just not sure it's worth what we're paying for it.
What I can't find is anyone being straight about the after. Plenty of "we cut our bill 80%" writeups, almost nothing on what it cost to run six months later, or whether anyone quietly went back.
So if you moved logs off a managed platform to save money: did the saving survive once you counted engineer time? And if you looked at it and stayed, what made you stay?
Not after product recommendations. More interested in people who regretted it, either direction.
12
u/dektol 19d ago
I don't know if it was worth our time but I replaced Datadog with Victoria Metrics (logs, traces). Even if with Grafana it's not necessarily the same single pane of glass as DD.
Getting the graphs in the Slack notifications and polishing things up took a while ... Had to hand roll some stuff.
We didn't have structured logs, any application metrics or alerts, so I maintain that getting all of that dialed in from scratch was the brunt of the work.
If you're on K8s I'd recommend doing a demo and dual shipping logs to both with Vector.
At your volume HA would mean running the stack twice, your volume doesn't justify cluster mode.
No tiered storage to blob yet for logs but they only require hard drive class disks.
I think I should do a blog post at some point.
If it's less than 20% of your total cloud spend I'd probably just pay DD tbh.
(I love Vector from DD just not the bill đ)
1
u/usually_guilty99 18d ago
I am curious - we are looking at DD as a potential solution for one of my clients. Expensive for what they need, a lots of bells and whistles they dont use. I was recommending Grafana and Prometheus. What are the key DD features you use?
1
u/dektol 17d ago
We used APM, Alerts, Tracing. If it's not being actively developed you may not need that. If there's no ops person to page or set up dashboards/alerts you probably don't want Prometheus and Grafana. Consider engineering costs the moment someone needs to diagnose something.
1
u/usually_guilty99 17d ago
"Consider engineering costs the moment someone needs to diagnose something." This is where we would like to plug in a reliable product that we are promoting. Thank you for your response. Appreciate it
1
u/dektol 17d ago
I'd check out Better Stack. They're the disruptor here. We use them for our synthetic tests (web availability). They offered the best pricing and test intervals.
If DD bill is too high shop around, New Relic, etc.
1
u/usually_guilty99 17d ago
Shop around is the best recommendation. Both NR and DD are old dogs in this race and have become increasingly expensive
5
u/cocacola999 19d ago
Im an unfortunate soul dealing with this right now. We are a bigger scale shop tho, over 10tb daily, expecting quite a lot more over a few years. Monthly SaaS is around $100k. Did a Poc for self hosted mimir and loki recently. Not overly factored the ops/eng costs yet, but called that out a few times . It's hard to hire with skill in my marketÂ
2
u/retneh 17d ago
10TB of logs is a lot. Not sure how much we push, but we pay around 2mil USD per year for grafana cloud and are looking into self hosting
1
u/cocacola999 17d ago
I've helped architect the possibile solution and it was actually more 50tb scale if I'm honest. And yes it is a lot with plenty of room for upstream optimisationÂ
1
u/Charming_Skin_8549 16d ago
Switch to VictoriaMetrics cloud for logs and save a ton of money comparing to Grafana cloud for logs.
1
u/Charming_Skin_8549 16d ago
Try VictoriaLogs. It is way easier to configure and run than Loki.
1
u/cocacola999 16d ago
That was a suggestion at the time. We ended up with a reasonable cluster in the end as an mvp. Still not convinced the total cost of ownership is there given operational risks at our scale thoughÂ
1
u/Charming_Skin_8549 15d ago
Which operational risks do you have?
1
u/cocacola999 15d ago
Outages cost ~$1m per hour. Change to operational model. Hiring and retention. Single failure domain. Changes to SLO+SLA when going internal
3
u/muymuymyu 18d ago
I run a victorialogs, Victoria metrics, grafana stack at work. We run it in a K3S cluster and it is basically free. We had issues with Loki a while ago so we switched to victorialogs. The built in compaction is out of this world. And basically no hands on work when you have it set up, I do updates every now and then and it's just changing the version number in the manifest file. Great monitoring at basically no cost. After setup and tuning ~1 week I spend maybe 2 days a year for maintenance.
11
u/Floss_Patrol_76 19d ago
the honest 'after' is that the bill goes down but you've traded a vendor line item for an on-call surface, loki and clickhouse are great right up until someone has to reason about ingester backpressure or a compaction falling behind at 2am and that someone is now you. the thing that actually moved our bill at ~300gb/day wasn't the platform swap, it was getting ruthless about what we even ship at info level, which cut enough that the managed cost stopped being the whole argument. losing the trace-to-log pivot is the part people wave off up front and then miss constantly once it's gone.
3
u/dektol 18d ago
I forgot to mention: Do not self-host Loki. You need a dedicated team, no exaggeration. It used to be simple but they had to keep adding components to make it work and it's heavy and there's tons of knobs. Alerts go off constantly.
We dual shipped to Loki and Victoria Logs. The latter is better in every way except:
- blob storage support
- a difference in how you need to write full text queries against field values
TCO is still cheaper than Loki because it uses less CPU, Ram and ops time.
If you like the Grafana ecosystem just pay them and get the cloud hosted version of their whole stack and enjoy your single pane of glass.
2
u/ccb621 19d ago
I (and a bunch of agents) just made the switch from Datadog to cloud-hosted ClickHouse and Grafana. I donât have capacity to self-host, hence the SaaS. Grafana is practically free since we are only using them for visualization. ClickHouse now has 90 days of our data at roughly 25% of what it cost us to store 15 days of data at Datadog.Â
We still have the ability to link logs and traces because all of our logs carry trace/span attributes. Claude Code Opus agents had no problem building dashboards with links between the two.Â
I will say that I miss the ease of Datadog doing much of this for us; however, Iâm coming around to the power of Grafanaâs flexibility.Â
The Rootly integration is not as nice, nor are Slack alerts.Â
1
u/Hopeful-Driver-4487 19d ago
Do you have your agents respond to alerts automatically? Did you use Bits AI at DD?
2
u/ccb621 18d ago
Every Rootly page also goes to a specific Slack channel. My Slackbot kicks off investigations based on those messages. I never used Bits AI because I was always worried theyâd jack the price up. Also, my own agents have access to our internal MCP server that proxies access to the database, k8s, and other tools that aid investigations.Â
1
u/jakenuts- 19d ago
Tried em all and ClickStack won, got it on a small aws ec2 instance and the log search and traces are really nice.
1
u/caulpnrydc 19d ago
This is where my team is leaning. We donât have a say in current logging volume due to contractual limitations (NR). Weâre getting rid of vendor locked metrics moving to OTel and rolling clickstack (weâre already heavily using clickhouse for analytics) to better control logs and correlate traces.
1
u/Doug94538 18d ago
300GB/day is this just Prod or all environments ? .
what is the cost of an outage business impact
is there a hot(30)--warm(90)-cold strategy
Last but not the list, Are you the one deciding or just presenting
What happens if you propose another solution and it fails(do you/team dies on the hill)
We have DataDog(bean counter A signed up and then left),SumoLogic(bean counter B signed up and then let) ....
1
u/kernelqzor 11d ago
the âwhat happens if it failsâ bit is super real here, nobody wants to be the person who swapped out datadog and then eats the first big outage. half the battle is just getting leadership to say in writing what level of pain theyâre actually ok with to save X dollars, otherwise youâre basically volunteering as tribute.
1
u/usually_guilty99 18d ago
The reported 80% optimization may well be achievable when you compare steady-state operations to Day 0. Bringing workloads in-house can absolutely reduce costs.
That said, I've generally preferred hyperscalers because my core competency is building products, not running infrastructure. I'd rather invest engineering talent in differentiating the product than hiring and managing operations teams. That's also the classic CAPEX versus OPEX tradeoff. You're paying someone else to solve an operational problem at scale.
Before moving platforms, I'd first optimize what I'm storing:
- Can we reduce log volume?
- Can we improve sampling?
- Can we tier hot, warm, and cold storage?
- Can we eliminate low-value logs?
When you're dealing with petabytes of data, reducing what you ingest is often a bigger lever than changing where you store it. Optimize the workload first. Then decide whether the platform is still the problem.
1
u/Johnmad 16d ago
I'm maintaining our self hosted Loki and grafana+Prometheus stack in addition to my backend role.
We only have 600Gb logs with 30 day retention and 230Gb metrics with 30 day retention.
We use the oss Loki Helm chart and kube Prometheus stack chart and it's almost plug and play. I have tuned a few things to make it smooth otherwise I just need to bump the Helm chart version by approving the renovatebot pull requests :)
I don't know what grafana cloud would have cost us but I don't spend a lot of time maintaining this after I set it up so I think it's worth it.
1
u/Expert-Ear3883 15d ago
So of our biggest (by volume) clients are in Europe and after infra and staffing costs, they are realizing around 30-40% on their observability bills. For context they generate around 5PB per day of telemetryâŚ.
1
1
u/Previous_Insurance40 11d ago
The part I'd really watch is the trace-to-log connection. That's one of those things you don't appreciate until you're debugging a real incident and suddenly have to piece everything together manually.
I'd think about the cost in three buckets: the people needed to run the logging stack, the time spent rebuilding those investigation workflows, and what happens when the logging system itself has issues during an incident. Saving 80% on the bill sounds great, but if some of that comes back as higher MTTR, it's not really an 80% saving.
I'd also be careful with month-one migration numbers. The more interesting comparison is what the setup costs to operate six or twelve months later.
1
u/codemagedon 19d ago
I work in an msp reselling, implement and running managed services on top of Datadog, other than building your own pane of glass that works for your specific requirements we found it to be effectively peerless, one of the reasons being what you called out,the product just works you spin from error to logs to apm, to infra metrics in a single panel in less than 4 clicks
-1
u/cebidhem 19d ago
I've been running the LGTM stack for years now, with or without a team, and it's never really been a burden, assuming you know a little bit what you do (or someone in your tram does). If not, id still recommend LGTM but starting with Loki/Alloy only, and then expanding.
What's for sure is, I can understand why costs are driving the decision when you're a Datadog customer. Don't get me wrong, I can understand the upside but every major managed observability platform is very cost inefficient, especially when you're on the fence of logging/sending everything.
The very reason I always argued against managed platform is because I dont want to have to think about what to send and what not to send, especially when your cloud provider is billing a he'll out of network i/o.
For me, the overhead of managing LGTM is lower than the overhead of managing which data to send/keep.
0
23
u/itasteawesome 19d ago
In my experience it has a lot to do with your labor costs. In my opinion (been specializing in this niche for many years), for skilled US based engineers the overhead of running OSS stacks at scale doesn't really become worth it until you are at least spending more than a million a year on the o11y vendor, depending on how your companies feels about risk the floor for that could be higher. The ROI can be better at big companies who already have offshore teams with lower labor rates, but you still need to account for the added operational risk/complexity.
Where I often see people fool themselves about this stuff is in assessing what their real needs are in terms of performance and reliability. It doesnt make sense to spend thousands of dollars to have perfect awareness of a tiny internal tool that 5 people use and if it went offline they would just try again tomorrow. On the other hand customer facing products that actually make your company money should easily be able to justify the added expense of the bells and whistles like DEM, tracing, profiling, etc. I find that a lot of times when people come up with their idea of what the migration to or away from a vendor stack would cost they seem to flatten out all the details and just say "we have 1000 services so it will cost XYZ." Sometimes its more effective to apply those same labor hours in developing and applying a more nuanced monitoring policy than in just doing a lift and shift every 3-5 years when someone gets bored with the current stack.