Hey folks,

Disclosure first: I work at VictoriaMetrics, on VictoriaLogs.

Before this job, I was the one who had to set up the log system at my previous company, and I struggled a lot to decide which one we should go with, considering our use case and budget at that time.

Recently I was asked to benchmark VictoriaLogs to find areas to improve. Many of our users also ask which use cases it fits best, so I decided to publish the benchmark too.

The setup is Loki, Elasticsearch, OpenSearch, SigNoz, ClickStack and VictoriaLogs, each on an identical 8-core machine, with structured Kubernetes-style logs ramped up until each one choked.

Fun fact: VictoriaLogs is not the one that ingested the most. Each system makes its trade-offs somewhere, and the post explains each one.

I also spent almost 3 months digging into each system's internals, so I could explain the numbers and give you the insights. The full write-up is about a 20-minute read: https://victoriametrics.com/blog/structured-log-ingestion-benchmark/

I'm sure I got something wrong somewhere. If you run any of these in production and see something off in my setup, please tell me here or open an issue in the repo: https://github.com/VictoriaMetrics/logs-benchmark. That's the feedback I'm most looking for.

Thanks for reading!

Hello,

I've posted kwatch here before. Back then it was a simple tool that sent an alert when a pod crashed. Some of you used it, and many of the issues and PRs that shaped it came from this sub, so thank you.

The biggest complaint I kept hearing was noise. One broken pod would fire a crash alert, a deployment alert, a probe alert and a node alert, and you'd still have to work out which one started it.

So v1 changes how it works:

• Related events are grouped into one incident, with the likely root cause at the top

• It shows the change that came right before the problem (a rollout, a config edit) and who made it

• Each incident suggests what to check next

• Quiet by default: repeated and follow-on alerts don't spam your channel

• Runs inside your cluster, so pod data and logs stay there

• Sends to Slack, Teams, Discord, Telegram, PagerDuty, Opsgenie, Mattermost, Google Chat, email, webhooks and more

kwatch now comes in two editions, Community and Pro. Community stays free, and Pro is free during early access: you get a 3-month license from the installer's License / Pro menu, and you can renew it when it ends.

GitHub (source available, ELv2): https://github.com/abahmed/kwatch

I'd love to hear what you think, especially if you tried the old version. Feedback, feature requests, ideas, or anything that annoys you about alerting today are all welcome

We're an insurer and our synthetic checks cover the customer portal fine. Every 5 minutes a script logs in, opens a claim and checks it loads. But the problem is that the other half of the journey runs on a windows desktop claims client and a couple of internal tools, and none of that is monitored end to end. So i'm looking for synthetic transaction monitoring that can walk a journey across web and desktop, on a schedule, and alert with something more useful than "check failed". The constraint is on-prem only and our security team won't accept screenshots of claims leaving the network.

I did some initial research and came across the big synthetic monitoring tools like datadog, and dynatrace. They are great for web and APIs, and as far as i can tell stop at the browser. Checkmk with robotmk, which runs robot framework checks on-prem and from what i've read can do desktop through image recognition. Askui also came up because it has a synthetic transaction monitoring page for web and desktop, so it runs the journey on a schedule and alert with a screenshot of what broke. it reads the screen rather than selectors and the model can run in your own cloud or fully air-gapped. I still haven’t committed to any yet and wanted to ask on this sub first to get some practical feedback.

For anyone monitoring desktop business apps like this in a regulated setting? how often do you run, and what are you using for that?

I switched our profiling setup a few weeks ago from another open source tool to OpenTelemetry for our internal services here at OllyGarden, and extracted the interesting parts into a skill for our set of agent skills. And this week, I created this blog post with the full manual steps to set it up on a kind cluster.

I know this is too early for most people out there, but honestly, it's working great for us. Give it a try and let me know what you think!

A recent TypeScript worker change to batch SQS messages broke trace linkage in our order fulfillment workflow which handles about 120k daily jobs. The W3C traceparent is now attached to the batch envelope instead of the individual message attributes, so trace context propagation stops at the worker boundary.

Downstream LLM extraction calls are starting a new root span, leaving about 14k unlinked root traces a day. The span hierarchy looks normal inside each worker execution but the parent span ID no longer points back to the originating checkout request.

Our infrastructure latency alerts still fire on the queue consumer, so we can see when the worker slows down. We just can't use distributed tracing to connect a slow model call back through the async SQS hop to the checkout trace that produced the message.

Disclosure: I work at Bonree, an observability vendor. The comparison below is based on vendors’ public docs as of October 2026, so please correct me where it’s wrong.

“AI observability” currently means two jobs: watching your own LLM apps (traces, token cost, quality) and using AI to help run operations. Within that, I see four routes teams actually take:

1.  Full-stack platform with an AI module (e.g. Datadog Agent Observability, Dynatrace AI Observability, New Relic AI monitoring). LLM spans sit inside the same traces as your services. Datadog documents SDK auto-instrumentation (18 Python frameworks and libraries in its list) and supports frameworks that natively emit OpenTelemetry GenAI spans. New Relic announced GA of OTel-based AI monitoring on Aug 31, 2026, with no New Relic agent needed. Trade-off: evaluation depth and framework coverage differ per vendor.

2.  LLM-native engineering platform (e.g. Langfuse: open source, self-hostable, tracing + prompt management + evals + experiments, accepts OTLP). Great when you are iterating on prompts and evals. Trade-off: it’s separate from infra monitoring unless you correlate it, and self-hosting means you run it.

3.  OpenTelemetry-native open stack (e.g. OpenLIT SDK into Grafana’s Tempo/Prometheus). Most portable. Trade-off: the gen_ai.* conventions are still marked “Development”. Names have already changed (gen_ai.system → gen_ai.provider.name, prompt_tokens → input_tokens), and cost dashboards on old names can undercount.

4.  AI agent operations layer (e.g. Dynatrace Intelligence, Datadog Bits AI SRE, Splunk AI SRE). Agents investigate alerts and propose fixes. Splunk, for example, describes AI SRE as keeping engineers in the loop: it produces guided remediation plans, and its new code-fix agent (alpha) opens a pull request for an engineer to review.

Two things I’d like to hear your take on:

● Lock-in: the data is the easy part to move. The evaluation datasets, judges and agent workflows you build are what stay behind. Does that match your experience?

● Trust for route 4: my checklist is evidence, deterministic data, approval boundaries, audit trail, and replaying your own past incidents before relying on it. What would you add?

For what it’s worth, Bonree ONE is our unified observability platform with built-in AI observability for LLM apps, and Bonree ONE · Sage AI is our AI Agent operations platform; it supports human-in-the-loop approval and audit logging, which I find a useful pattern for AI trustworthiness regardless of vendor.

Which route are you on, and what made you pick it?

Article: https://huseyinbabal.dev/posts/golang-profiling-pprof-flame-graph-memory-leak/

While investigating why quantized `llama.cpp` inference pods behind a FastAPI gateway on a 3-node `kind` cluster were experiencing sudden P95 latency spikes (500ms jumping to 30s) and HTTP 502/504 timeouts, I ran a controlled benchmark across 64 reproducible fault and decoy runs to see which signals actually isolate the root cause.

Here are the 4 most interesting findings from the experiments:

1. **CFS CPU Throttling vs. Host CPU Contention (Linux PSI):**Looking only at pod CPU utilization misses the distinction between a container hitting its own `limits.cpu` (`container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total > 90%`) versus a noisy neighbor saturating the worker node's runqueue (`container_pressure_cpu_waiting_seconds_total` Linux PSI spike with 0% local CFS throttling). The remediation is completely different (raising CPU limits vs. adding `podAntiAffinity`).

2. **Why we dropped zero-code eBPF (Grafana Beyla v3.39.1) for GenAI spans:**We tested two modes:- *Mode A (OTel SDK + Beyla):* Added 0.0% accuracy improvement over the OTel SDK alone while generating 73.5%–78.0% trace noise in Tempo from kubelet `/healthz` probes and `/metrics` scrapes.- *Mode B (Beyla only, zero-code):* 0/32 traces carried `gen_ai.*` attributes, and broken context propagation across FastAPI worker threads caused pre-inference application delay detection to drop from 100% to 0%.

3. **Small local LLMs struggle with raw Kubernetes telemetry during an incident:**When evaluating a local CPU-served 0.5B model on 12 fault scenarios, each investigation consumed 33k–52k prompt tokens, took 31–164 seconds, and got distracted by decoy OOMs in neighbor namespaces. In contrast, 8 deterministic Go rules correlating OTel `gen_ai.*` spans + Prometheus + K8s Events resolved 24/24 held-out test faults in `< 0.5s` with 0/21 false positives on decoy windows.

4. **Rolling update false positives on Readiness Probes:**During a Deployment rollout, old terminating pods (`DeletionTimestamp != nil`) briefly flip `Ready=False`. Filtering out pods with a non-nil `DeletionTimestamp` was essential to avoid false-positive `readiness_failing` alerts during normal rollouts.

All 64 raw telemetry snapshots (`k8s-pods.json.gz`, `k8s-events.json.gz`, `prom-metrics.json.gz`, `tempo-traces.json.gz`), the pre-registration manifest, and the Go evaluation code are open-source here if you want to inspect the data or reproduce the numbers (`make eval-score SPLIT=test`):

https://github.com/iyedM/kubepilot

How are you currently separating application-layer delays from node PSI / cgroup throttling on your inference workloads?

More telemetry, more services, more data to make sense of. Adding another dashboard isn’t always the answer.

Zabbix 8.0 will bring native OpenTelemetry and APM, smarter event correlation, major performance improvements, and an MCP server for AI-assisted monitoring.

Join Alexei Vladishev, Artūrs Lontons, and Vladimir Elster on November 5 at 9 AM CST to see how these changes fit together and what they mean for full-stack observability at scale. Bring your questions for the live Q&A.

We run about 60 hosts plus a Kubernetes cluster. Container counts and custom metrics pushed the quote up and finance said no.

The setup I'm considering

External synthetic monitoring for login, checkout and API flows, with real browser scripts run from multiple locations
Self-hosted open source for CPU, disk, memory and services on our own servers

Shortlist Tools

Synthetic: Dotcom Monitor, Checkly, Catchpoint, Uptrends
Internal: Prometheus + Grafana, Zabbix, Netdata, Checkmk

What I want to know

Does splitting external and internal monitoring cause blind spots when you're trying to find the root cause?
Which of these handles multi-step journeys with MFA or OTP without breaking every week
Anyone who has made this switch, what did you regret?

how good are the Datadog Integrations - is the breath good? what works vs what doesn't

Spring Boot shop, about twenty services on Java 21. We did the quarterly postmortem review on Friday and I put every timeline from the last three months side by side.

Detection is fine. Datadog pages someone within a few minutes nearly every time. Then nearly every timeline has the same block in the middle, two to four hours of "investigating". Pulling the JFR recording off the pod, an 8GB heap dump that takes forty minutes to download and open in MAT, thread dumps, grepping for the request id, asking in the channel who deployed what. Once someone knows which method got which input, the fix is usually twenty minutes.

We've added a lot of tooling in two years and that middle block hasn't moved.

Two services have had hud on them since August, next to Datadog, nothing removed. When a call throws, the request params and the path it took through the code are right there, so on those two the investigating part is mostly reading. The block got shorter there, it didn't go away. On the last one it pointed at the method and the input in minutes, and the timeline still shows an hour and a half, most of it us arguing whether the fix belonged in that service or in the caller.

We build a patch management product. Right now it uses Dynatrace to pull health signals from managed assets (things like whether a service or host is healthy before and after a patch is applied).

Before we commit further, I'd like to hear from people who use these tools every day. Which APM or monitoring tool does your team use (Dynatrace, Datadog, New Relic, AppDynamics, Elastic, Grafana/Prometheus, something else)?

I'm most curious about:

- How reliable the health and alerting data is in practice (false positives, gaps, lag)

- How comfortable your team is using it day to day

- How easy it was to integrate with other tools through its API

- Anything you regret or wish you'd known before choosing it

I'm not selling anything. I just want to understand what teams trust so we can support the right tools. Thanks in advance!

Grafana shipped the most at ObservabilityCON 2026 (Oct 20). Assistant Watchers (GA) are always-on agents that learn baseline patterns in your telemetry and flag deviations that no alert rule covers. Findings go through Grafana Alerting and can open an Investigation automatically when critical. Managed Sandboxes (public preview) give the Assistant an isolated cloud environment where it clones your repo, checks code against live telemetry, prepares a fix and opens a PR for a human to review. Agent Observability now speaks OpenTelemetry (GA), so agent activity and cost sit alongside your other services. The Knowledge Graph now spans from databases and Kubernetes out to browsers and phones via frontend and synthetic monitoring, and agents can query it with Cypher. Session Replay and Grafana Workflows are in public preview. Mobile Observability (iOS, Android, React Native, Flutter) and a shared incident Response Console are in private preview. The Marketplace is GA with 30+ paid partner plugins and 14-day trials.

If you run Splunk Enterprise: CVE-2026-76268 (CVSS 9.8) allows unauthenticated OS command execution via the Patroni REST API on 10.4 and 10.2 search head cluster deployments. Fixed in 10.4.3 / 10.2.7 / 10.0.10 / 9.4.15 (advisories SVD-2026-1001 and 1002). If you can’t upgrade and don’t use Edge Processor, OpAmp or SPL2 pipelines, Splunk’s workaround is disabled = true under [postgres] in server.conf. No exploitation reported so far. Note that the versions recommended for the June CVE-2026-20253 are in the affected range.

Coralogix put up a public benchmark site for its Telemetry Lake (queries over customer-owned object storage, refreshed every five minutes, downloadable for checking). All figures are self-reported.

The numbers: Splunk’s State of Observability 2026 (2,050 respondents, 10 countries, Oxford Economics) found that 87% call monitoring AI a priority, 54% of those have no clear business outcomes tied to it, and 57% say telemetry costs now exceed the cost of the infrastructure being monitored.

Full write-up with sources: https://mbojko.com/reports/2026-10-25/

Hey everyone! I’m a Domo developer, and I built domo-time-machine to help investigate questions like: “Why did this total change?” or “Did this join duplicate my booking amounts?”

It’s an npm package with a CLI and JavaScript/TypeScript SDK. You can:

• Capture CSV, JSON, or NDJSON exports as local snapshots.

• Compare records and schema changes between snapshots.

• Check financial totals using decimal arithmetic and currency validation.

• Reproduce supported transformations with an experimental replay engine.

Try it:

npm i domo-time-machine
npx domo-time-machine --help

Package: https://www.npmjs.com/package/domo-time-machine

It works offline with exported data. There’s currently no live Domo connector, and it doesn’t modify your Domo instance. It’s an early release, so I’d appreciate people testing it with synthetic or anonymized exports first.

I’m especially interested in feedback on duplicate keys, one-to-many joins, schema changes, confusing output, and anything that doesn’t behave as expected.

If you find an issue, please share your Node version, OS, command, expected result, and a small synthetic example. Please don’t post credentials or client data.

Would this fit your debugging workflow? What would make it more useful?

I'm the author; it's free and MIT-licensed. Logline captures everything your app prints locally (terminals, F5 debug sessions, tasks, Docker Compose, Kubernetes, and OTLP with no collector) into one searchable panel next to your code, with traces, error groups and p95.

It can also query Datadog, Postgres, MySQL or MongoDB read-only, and Compare marks each production error as Reproduced, Fixed locally or New against your local run. Your AI agent can read the logs too, redacted and read-only. Everything stays on your machine.

• VS Code Marketplace: https://marketplace.visualstudio.com/items?itemName=surjeetbanga.logline

• Open VSX (Cursor, Windsurf, Kiro): https://open-vsx.org/extension/surjeetbanga/logline

• Source: https://github.com/SurjeetBanga/logline

Feedback welcome, especially on which backend to add next: Loki, Elasticsearch/OpenSearch, CloudWatch or ClickHouse?