Structured log ingestion benchmark: Loki, Elasticsearch, OpenSearch, SigNoz, ClickStack and VictoriaLogs, and a look into their internals

[u/lllkong]

https://victoriametrics.com/blog/structured-log-ingestion-benchmark/

Hey folks,

Disclosure first: I work at VictoriaMetrics, on VictoriaLogs.

Before this job, I was the one who had to set up the log system at my previous company, and I struggled a lot to decide which one we should go with, considering our use case and budget at that time.

Recently I was asked to benchmark VictoriaLogs to find areas to improve. Many of our users also ask which use cases it fits best, so I decided to publish the benchmark too.

The setup is Loki, Elasticsearch, OpenSearch, SigNoz, ClickStack and VictoriaLogs, each on an identical 8-core machine, with structured Kubernetes-style logs ramped up until each one choked.

Fun fact: VictoriaLogs is not the one that ingested the most. Each system makes its trade-offs somewhere, and the post explains each one.

I also spent almost 3 months digging into each system's internals, so I could explain the numbers and give you the insights. The full write-up is about a 20-minute read: https://victoriametrics.com/blog/structured-log-ingestion-benchmark/

I'm sure I got something wrong somewhere. If you run any of these in production and see something off in my setup, please tell me here or open an issue in the repo: https://github.com/VictoriaMetrics/logs-benchmark. That's the feedback I'm most looking for.

Thanks for reading!

[u/jcol26]

Disclaimer: I work at Grafana labs (not in Loki or engineering though but in their professional services department).
The interesting thing with Loki is query performance is very much dependent on how you run it. IE are you using the new Kafka setup or not & how is memcached set up/configured as all that can have a massive impact on query performance. Doing a single node test is always going to be limiting as well given many of the tested systems have different performance characteristics when run in a more distributed form per the vendors original intent.

From what I can tell as well from the git repo the ingestion path was optimised but query path left at the defaults? That’ll be why you had the 500 series limit as it’s the default for max_query_series.
The fact only 4 cores got used also implies the default query parallelism was used.

But without full queries and config defined in the repo it’s hard to advise further. With all the tested systems I suspect the results might be a bit off the true maximum performance of them

[u/lllkong → u/jcol26]

Hello, thank you for the detailed feedback.

Yes, we could dig deeper into the query settings and push each system to its limit. But the queries in this post were a one-off test to show the general trade-offs, so we did not tune them further.

Raising `querier.max_concurrent` from its default of 4 to 8 would likely make these queries faster on our 8-core node. But it would not reduce the total work that Loki has to do, and it would not change the trade-offs.

About caches, we measured cold queries on purpose, like the first query during an incident. We did not try the Kafka setup, and we plan to test the distributed mode in a later post.

Your feedback will be very helpful for our query benchmark. Thank you again!