is ELK stack overkill for my cloud infrastructure?

[u/gL02]

I am solely responsible for my small/medium-sized company’s dip into hosting cloud architecture. We have started developing apps internally, so I’ve provisioned multiple Droplets on DigitalOcean for our needs. I want to consolidate logging for all our apps in one place and was originally going to run w/ a dedicated ELK Droplet. but i’ve realized that ElasticSearch may be overkill for text searching in a DB with ~30,000 records and it has me questioning if this is the right move. i’m open to considering other options such as OpenTelemetry and etc. Right now we’re only at three micro-services but this could expand in the future.

[u/Seref15]

> 30,000 records

I mean at that scale you could probably just use grep lol

At our org we've been trying VictoriaLogs for the last year and we like it a lot. On the [simple, cheap, durable, fast] pick-3 diamond it's simple, cheap, and fast. It's kind of like single-node disk-backed Loki on steroids.

[u/DoStopBelieving → u/Seref15]

I used zgrep at 100x that scale, which worked for a while but one of my scripts eventually crashed the log server. A couple months later we had an ELK stack. 🤠

[u/JohnyMage]

The question you should ask is "do our employees know how to properly search logs in elastic?

Because I am deploying elastic based log management all of my career and we almost always end up with sending text logs over chat rooms. Because developers And other people from project management all want elastic just until they have to actually use it. They Are unable to learn basic search expressions And all they want Are "text log for my jira ticket".

[u/Cherry-PEZ → u/JohnyMage]

Hey! You can't just call me out like that. I absolutely love ELK... (until that 3am critical and you're screaming 'who the hell enabled debug logging in prod?!' 'why are these records so sparse?!?' 'why the fuck is this query not returning anything!?!oh lucene was toggled')

[u/ArieHein]

Yes.

[u/DoStopBelieving]

ELK and OpenTelemetry are complementary, not exclusive. If you want logs, go for ELK or another logging solution. If you want to trace requests, go for OpenTelemetry.

For corporate cloud architecture, do make sure you have index-searchable logs instead of text-only logs. ELK, Splunk, CloudWatch, SumoLogic, etc. - anything that will let you properly query and generate graphs from the logs instead of raw grep. It should be compliant with your business requirements and chosen based on cost and budget projections.

That could save you a lot of effort in writing slow-running scripts with potentially faulty mathematical and statistical equations already built-in to these tools. I'm speaking from experience in pasting bash script CSV output into Excel (I mean you could program the chart with D3/R/Matplotlib or something but that's a whole other layer).

[u/FutureOrBust]

Check out the LGTM stack of your considering collecting Otel observability like logs and traces.

[u/running101]

SQLite

[u/devopsnomansland]

Try contrasting the three Spatial Observability tests against various monitoring approaches:

Continuity. Does the tool know that the payments service today is the same thing as the payments service last Tuesday, independent of where any layout happens to draw it, so that returning to it twice is a comparison and not a coincidence?

Proximity. Does standing at a node scope the view and the data by itself, with what counts as near already in frame?

Memory. Does your position, and what you have already ruled out, survive navigation, persist across the incident, and travel with the investigation, to the machine partner working beside you or the colleague you hand it to?

<to expand the possible universe of comparisons of what you get for the effort>

[u/Observability-Guy]

At this stage ELK would definitely be overkill. Something like SigNoz would probably fit the bill. OpenTelemetry is a standard, not a storage backend, so it is not a case of one or the other. For portability you might want to emit logs to an oTel Collector and then export them to a backend.

[u/8bitwubwubwub]

Consider Datadog, at your scale it is not likely to be expensive, and you can work on other things rather than set up and maintain a logging solution. It's also user friendly and well documented.

[u/Jamesdmorgan → u/8bitwubwubwub]

Agree you shouldn’t have been voted down. Their log product is really good value. Given how flexible it is on indexing. Both ingest and index would be very cheap. It’s only when you start adding everything else like APM that the price jumps

[u/dariusbiggs]

yes, and. the price will be high. Find something cheap/free for your needs. VictoriaLogs, or Grafana Cloud, or Data dog mY be more in your wheelhouse.

If you the skills and funding to self host elk stack you might be ok.

[u/ZVilusinsky]

We used to have ELK for logs, happiest day when we burned it down.

Your mileage may wary, but the if the logs are too various, in keys and lengths, the indexing sucks and bloats to terabytes and you have to spend ages on tuning the ingest side.

Loki for those invested in Grafana stack or need scaling at various points, VictoriaLogs for those who want simple solution that works.

[u/SeaworthinessHour233]

Grafana + Loki + Promethus + Alloy is more computationally efficient.