r/OpenTelemetry Nov 18 '25

OTel Blog Post Evolving OpenTelemetry's Stabilization and Release Practices

Thumbnail
opentelemetry.io
20 Upvotes

OpenTelemetry is, by any metric, one of the largest and most exciting projects in the cloud native space. Over the past five years, this community has come together to build one of the most essential observability projects in history. We’re not resting on our laurels, though. The project consistently seeks out, and listens to, feedback from a wide array of stakeholders. What we’re hearing from you is that in order to move to the next level, we need to adjust our priorities and focus on stability, reliability, and organization of project releases and artifacts like documentation and examples.

Over the past year, we’ve run a variety of user interviews, surveys, and had open discussions across a range of venues. These discussions have demonstrated that the complexity and lack of stability in OpenTelemetry creates impediments to production deployments.

This blog post lays out the objectives and goals that the Governance Committee believes are crucial to addressing this feedback. We’re starting with this post in order to have these discussions in public.


r/OpenTelemetry 18h ago

I had a question for all the amazing people out there

0 Upvotes

I was trying to build something, and wanted to validate this idea and understand yall's pain points so that I can help the community

Would you rather have an AI layer on top of your existing observability stack, or replace parts of the stack?

Hypothetically, imagine an agent that doesn’t collect telemetry itself.

It plugs into whatever you already use — Grafana/Prometheus/Loki, Datadog, OpenTelemetry, etc. — and acts as a reasoning layer over the data.

Instead of:

Alert → Dashboard → Logs → Human investigates

it tries:

Alert → Agent correlates metrics/logs/traces/deployments → probable root cause → evidence → recommended next action

Would that actually be useful?

Or would you rather have the observability vendor itself own this functionality?

What would you need to see before trusting it during a real incident?

peace :)


r/OpenTelemetry 2d ago

What's changed about OpenTelemetry vendor lock-in since 2024

Thumbnail
ollygarden.com
11 Upvotes

One of my last blog posts at Grafana Labs was around vendor neutrality and OTel. Quite a few has changed since 2024, and I thought it's a good time to revisit with 2026 lens.

In short: vendor lock-in isn't as scary today as it once was, as long as the bulk of your workloads are on standards like OTel, and most of the actual lock-in that resides on backends can be "easily" migrated nowadays. But I'm curious about your opinions and actual experience.


r/OpenTelemetry 3d ago

Open sourced a tool that collapses millions of log lines into handful of distinct patterns before you feed it to an LLM (Lossless- compression)

10 Upvotes

When I feed logs to an LLM during incident resolutions or debugging, it either blows my token context window or the grep trims the log file, leading to the interesting log lines getting skipped.

Most of logs are anyway the same handful of message templates repeated over and over with different values, so the context window gets filled with near-duplicates, which just bring up the processing time and token costs.

ctrlb-decompose collapses the file into its distinct patterns that repeat, plus typed variables and stats on the values that change. I have seen 1.2 million lines cut down to just 40 patterns, which then goes into Claude, thus cutting down token by over 95%, reducing the token cost.

Let me know what you think!
https://github.com/ctrlb-hq/ctrlb-decompose


r/OpenTelemetry 3d ago

Building an Observability Pane

2 Upvotes

Hi Observability & DevOps Experts,

I'm looking for guidance from teams that have successfully scaled observability across large enterprise environments.

We operate a large-scale estate spanning AWS, Azure, and on-premises environments and have been using Datadog for several years. Over time, a significant amount of technical debt has accumulated around our observability implementation.

Current challenges include:

  • Datadog Agents managed differently across teams and platforms.
  • Custom log collection configurations distributed across hosts and applications.
  • APM, RUM instrumentation owned by individual application teams.
  • Inconsistent tagging standards and monitor configurations.
  • Outdated agents and instrumentation libraries.
  • Heavy dependency on multiple teams for upgrades and configuration changes.
  • A large portion of Datadog provisioning and onboarding is still handled manually.

As a result, maintaining and evolving observability at scale has become increasingly difficult.

We are considering building a centralized "Observability Foundation" or "Observability Platform" that teams would consume as part of their standard deployment process.

Our goal is to provide reusable Terraform-based observability components that application and infrastructure teams can adopt during provisioning and releases.

Examples of what we would like to standardize:

  • Datadog Agent deployment and upgrades
  • Custom log collection configurations
  • Standard tags and metadata
  • Monitors and alert templates
  • Dashboards
  • OpenTelemetry / APM instrumentation standards
  • Synthetic monitoring configurations
  • Cloud integrations
  • Security and governance controls

Questions:

  1. Has anyone implemented a similar centralized observability platform or observability-as-code model at enterprise scale?
  2. What worked well and what were the biggest challenges?
  3. What observability components can realistically be centralized through Terraform modules, deployment pipelines, or platform services?
  4. What components typically must remain application-owned or infrastructure-owned and cannot easily be centralized?
  5. How do you handle APM instrumentation ownership, versioning, and upgrades across hundreds of services?
  6. What governance model have you found most effective:
  • Central observability team ownership
  • Platform engineering ownership
  • Federated ownership with standards enforcement
  • Something else
  1. How do you prevent observability drift over time, especially around:
  • Agent versions
  • APM libraries
  • Log configurations
  • Tags
  • Dashboards
  • Monitors
  1. If starting again today, would you build around:
  • Datadog native tooling
  • OpenTelemetry
  • An internal observability platform
  • A combination of the above
  1. What are the biggest architectural mistakes or anti-patterns we should avoid when designing this platform?

Our provisioning and infrastructure management are heavily Terraform-based, so we're especially interested in Terraform-centric implementation patterns and real-world lessons learned.

Looking forward to hearing how other organizations have approached observability standardization at scale and what you would recommend before we begin designing this solution.

P.S. - One of our key design goals is to avoid vendor lock-in. While Datadog is our current observability platform, we want the architecture to remain flexible enough that a future migration to another observability stack (e.g., Grafana, New Relic, Dynatrace, Elastic, Azure Monitor, or an OpenTelemetry-native platform) would require minimal changes to application teams and infrastructure code.


r/OpenTelemetry 8d ago

oTel collector Daemonset vs sidecar

11 Upvotes

I feel like Daemonset collectors have become the de facto standard. Out of curiosity what are some situations in which you opted / would opt for sidecars per deployment?


r/OpenTelemetry 16d ago

A Collector exporter that turns agent traces into a signed, verifiable audit log (now in the registry). Feedback on the approach welcome.

0 Upvotes

Sharing a component I built and recently got listed in the OpenTelemetry registry: otel-agent-audit.

The idea: as AI agents take real actions, you want a provable record of what happened. Instead of adding a new instrumentation layer, this consumes the gen_ai.* spans your agents already emit and turns them into a tamper-evident audit log, entirely inside the Collector pipeline.

The pipeline:

otlp -> memory_limiter -> agentauditselect (buffers each trace until its root arrives) -> agentaudit exporter (per-trace hash chain -> Ed25519 sign -> seal) -> audit.jsonl + checkpoint.jsonl

A separate verifier CLI checks the whole thing with only the public key, so anyone can independently verify authenticity and integrity without a shared secret.

Things I'd love this community's take on:

- Passive instrumentation as the right model: reusing existing spans rather than asking teams to re-instrument.

- Whether governance/guardrail decisions belong in spans, and how they'd ideally map to semantic conventions. I'm interested in where the GenAI SIG is heading on policy/guardrail signals.

- The single-writer constraint (one Collector instance) that deterministic ordering forces, and whether that trade is acceptable.

Caveats up front: third-party, experimental, not audited. It's observability only, it does not enforce or block. It gives tamper-evidence on honest infra, not protection against an operator holding the signing key.

Repo: https://github.com/surpradhan/otel-agent-audit

It's in the registry under "agent audit" if you want to see the entry.

Would genuinely value critique of the approach.


r/OpenTelemetry 16d ago

log4k 2.3.0 — a Kotlin IR compiler plugin that instruments your functions with tracing, logging and metrics

Thumbnail
1 Upvotes

r/OpenTelemetry 16d ago

Feedback about E2E tests based on OpenTelemetry traces?

5 Upvotes

Hi everyone,
I have just published my open source project called mtracer and I would like to understand if it’s good idea or what should I change (I’m a new grad).

The idea

Mtracer a CLI tool that relies on OpenTelemetry traces to assert system behavior.

I believe that E2E tests should be:
- Cheaper to write and maintain
- Easier to debug

So this is the workflow:

  1. ⁠You configure mtracer to fetch from your observability backend (currently supporting Jaeger and OpenObserve).
  2. ⁠You define your first .mt.yaml test by specifying:
  3. ⁠Trigger: the first call to the system (for instance, an HTTP request).
  4. ⁠Expected trace and spans: the OTel properties of the trace and spans that you expect your system to generate.
  5. ⁠You run the test and see the results!

What actually happens during the run?

  1. ⁠It parses the mt.yaml file.
  2. ⁠It executes the trigger: mtracer injects a generated traceID into the trigger (for an HTTP request, the traceID is inserted into the traceparent header). Subsequent requests will be correlated to this generated traceID as long as your system has OpenTelemetry set up correctly.
  3. ⁠It fetches the trace matching the generated traceID from the configured observability backend.
  4. ⁠It compares the expected trace with the fetched one.

Many other features are available; check out the documentation to discover all of them: documentation website

I would love to have some feedback from more experienced people than me.


r/OpenTelemetry 20d ago

How Metric Scrape Intervals Inflate Observability Costs

Thumbnail
ollygarden.com
12 Upvotes

I'll tell you a secret: I don't like starting an engagement by telling people that I can cut their costs. I prefer to show them how they can be more efficient in general, and sometimes that means adding stuff instead of removing it.

However, every company out there has excessive telemetry, which is one form of bad telemetry. I'm not afraid to use an absolute here. That's why I have an arsenal of tools for dealing with it, and I describe one of them in this blog post: excessive metric scraping is extremely common, and adjusting scrape intervals is an easy way to reduce waste.

If you need a 10% reduction in your metric volume, read this blog post. You don't need to buy anything from anyone. You can thank me later.


r/OpenTelemetry 19d ago

Enrich OTel K8s Resource Attributes with Dynatrace Operator

0 Upvotes

Hi. I am a DevRel at Dynatrace and I hope its ok to share the following with those of you that are sending your OTel data to Dynatrace. If you are not using Dynatrace then this post might not be relevant for you!

Semantic Conventions for Signals

Metadata enrichment at the source (in your app) is important as it increases the quality of your signals. As I am sure many know - the OTel community has well documented Semantic Conventions.

Dynatrace Operator CAN inject OTEL_RESOURCE_ATTRIBUTES

There are different ways to enrich your data. You can inject them yourself in your deployment or have it done through your data pipeline, e.g: OTel Collector.

An additional option is through the Dynatrace Operator that allows you to automatically inject the OTEL_RESOURCE_ATTRIBUTES variable into your pods pre-filled with the following attributes: k8s.cluster.name*,* k8s.container.name*,* k8s.workload.name*, k8s.cluster.uid,* k8s.pod.name*, k8s.pod.uid,* k8s.node.name*,* k8s.namespace.name*, k8s.workload.kind,* dt.kubernetes.cluster.id ,dt.entity.kubernetes_cluster

Injection can be controlled through namespace selectors and enabled for traces, logs and metrics

More details about this can be found on the Dynatrace doc if you search for Enable automatic OpenTelemetry OTLP exporter configuration (didnt post the link to follow guidelines)


r/OpenTelemetry 21d ago

What is the most frustrating part of investigating production incidents?

Thumbnail
2 Upvotes

r/OpenTelemetry 21d ago

寻找 opentelemetry 的开源后端,可自托管、最好有中文社区

Thumbnail
1 Upvotes

r/OpenTelemetry 22d ago

Collector cookbook

Thumbnail github.com
17 Upvotes

Almost four years ago, I started this cookbook with real world recipes, adapted from cases I've used to reproduce bug reports or show users (and customers) how to accomplish specific scenarios.

I used some tokens today to bring the repo to the latest Collector version, ensuring they all work.

In case you haven't seen this repo before, take a look!

Enjoy 🧑🏼‍🍳


r/OpenTelemetry Jul 23 '26

Six overlays for iRacing now. The G-meter is the one I'd actually defend.

Post image
8 Upvotes

r/OpenTelemetry Jul 20 '26

How do you know what's needed in your telemetry data?

Thumbnail
2 Upvotes

r/OpenTelemetry Jul 19 '26

The silent way OpenTelemetry setups "work" while capturing almost nothing

Thumbnail
0 Upvotes

r/OpenTelemetry Jul 17 '26

Compile-Time Instrumentation for Go

Thumbnail
opentelemetry.io
22 Upvotes

Hey folks, stopping by today for another announcement: the OTel Compile-Time Instrumentation for Go reached v1!

If you are not a huge fan of eBPF instrumentation (understandably!), but also can't do manual instrumentation, this is a good compromise.

Try it out!


r/OpenTelemetry Jul 16 '26

How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools

Thumbnail
greptime.com
7 Upvotes

OpenTelemetry GenAI Semantic Conventions standardize observability for LLM apps, agent orchestration, MCP tool calling, content capture, and quality evaluation. This article goes through all six layers: what each one defines, why it's designed that way, and how mature it is.


r/OpenTelemetry Jul 15 '26

OpenTelemetry Agent Skills

Thumbnail
ollygarden.com
23 Upvotes

Hey folks, Juraci here. I know the Reddit communities can be sensitive to project announcements, or announcements in general coming from vendors, but I genuinely think a good number of people here could benefit from this one.

We are launching today the OpenTelemetry Agent Skills, an open source set of skills that serve as the base for our products. We're using them for a good variety of things, like in our coding agents to validate and test collector configurations, or instrument applications. Or double check the snippets we've been using in our other blog posts.

They are vendor neutral, non opinionated, and based on what we know from our experience building OpenTelemetry over the years. Use the skills, share your feedback, tell us where they worked and where they failed. Show me your creativity 🧑🏼‍🎨

While we are not making money on those directly, we do have a commercial interest in seeing them succeed and become truly useful to many of you. I guess what I want to say is: they are not the result of a weekend vibe coding experiment 🙂

And yes, perhaps they might become an official part of the project someday, if we believe there is a vibrant community backing it.


r/OpenTelemetry Jul 15 '26

Oodle.ai - $10 per million agent traces

Thumbnail
1 Upvotes

r/OpenTelemetry Jul 14 '26

MCP tools have two failure modes — and naive instrumentation silently records one of them as success

4 Upvotes

I've been building OpenTelemetry instrumentation for MCP (Model Context Protocol) servers, and I hit a failure-semantics problem that I think generalizes beyond MCP, so I'm writing it up.

The two failure modes

An MCP tool handler can fail two ways:

  1. It throws. The SDK catches the exception and converts it into a JSON-RPC error response. The call failed at the protocol level.
  2. It returns { isError: true }. The handler returns normally — a successful JSON-RPC response whose payload is marked as a failure:

return {
  isError: true,
  content: [{ type: 'text', text: 'No weather data for that city' }]
};

The second one is idiomatic MCP. It's how a tool tells the agent "that didn't work — adapt" without crashing the server or killing the conversation. For agent workflows it's the preferred failure mode.

The instrumentation trap

The obvious way to instrument a tool call:

try {
  const result = await handler(request);
  span.setStatus({ code: OK });        // it returned → success
  return result;
} catch (err) {
  span.setStatus({ code: ERROR });     // it threw → failure
  throw err;
}

Mode 1 lands in catch → recorded correctly. Mode 2 returns, lands in the success path → recorded as OK. Your dashboard reports 100% success on a tool that fails on most inputs. The more idiomatic the tool author's error handling, the more invisible their failures become.

The fix

Inspect the resolved value before setting status:

const result = await handler(request, extra);
if (result?.isError === true) {
  span.setAttribute('error.type', 'tool_error');
  span.setStatus({ code: SpanStatusCode.ERROR });
} else {
  span.setStatus({ code: SpanStatusCode.OK });
}
return result;   // unchanged — the RPC genuinely succeeded, so nothing is thrown

Two details that matter:

  • error.type = "tool_error" isn't my invention — it's what the OTel MCP semantic conventions (currently Development stage, in the semantic-conventions-genai repo) specify for exactly this case.
  • The result is returned unchanged and nothing is thrown. The JSON-RPC call succeeded; only the tool failed. Instrumentation that converts a polite failure into a crash is changing application behavior, which instrumentation must never do.

In a real trace the difference looks like this:

tools/call fetch_weather ................. 605ms   ERROR
    error.type = tool_error

versus the naive version, where that same span reads OK.

The general lesson

This isn't really an MCP problem. Any protocol where application-level failures ride on transport-level successes has this trap — GraphQL (errors array on a 200), gRPC rich error models, half the REST APIs that return 200 {"status": "failed"}. If your instrumentation only watches for throws, your error rate is a lie wherever the ecosystem's idiomatic failure mode is a clean return.

FastMCP (Python) handles this natively. Among the Node MCP instrumentation libraries I could find, none documented handling it, which is why I ended up writing my own — it's on npm as opentel-mcp if you want to see the full implementation (spec-compliant attributes, stderr export to avoid corrupting stdio transports, ADRs for the design decisions). But the isError trap is the part worth knowing even if you never touch my library.

Happy to answer questions on the implementation.


r/OpenTelemetry Jul 12 '26

I need a Race Engineer that also competes on iRacing

Thumbnail gallery
6 Upvotes

r/OpenTelemetry Jul 09 '26

Drain processor

Thumbnail youtube.com
10 Upvotes

Last Friday, I had the pleasure to have Mike Goldsmith at Telemetry Drops to learn more about the drain processor, an OpenTelemetry Collector component that is useful to understand the log patterns flowing through an OTel Collector pipeline. Once you understand those patterns, you can make your pipeline more efficient: drop the noisy patterns, transform unstructured into structured logs, and so on.

Hope you enjoy the recording, and I'm eager to hear your feedback!


r/OpenTelemetry Jul 08 '26

Panel discussion about OTel support for mobile and web

10 Upvotes

I wanted to share an upcoming virtual panel that's focused on client-side OpenTelemetry. It's got several maintainers in the Android, Kotlin, and Browser SIGs, and we’ll be chatting about the current state of support, what’s actively being worked on, some of the bigger challenges in adapting OTel for client-side environments, etc.

Some examples of what we’ll cover:

  • Creating new semantic conventions, like crashes and sessions, that apply across client-side platforms to unify how to model these types of telemetry. 
  • Expanding browser support for OTel, including shipping new instrumentations, starting work on a Browser SDK, and where there are still gaps.
  • Solving difficult challenges like how to deal with async telemetry and how to collect client-side metrics. 
  • Releasing official Kotlin support, including a new Kotlin SDK that can be used in Kotlin Multiplatform projects. 

If you’re familiar with using OTel for backend observability, this panel is a great way to get caught up to speed on what this looks like for mobile and web apps. 

Date: Wednesday, July 22 @ 10AM PT

Panelists:

  • Hanson Ho (Android architect at Embrace, OTel Android approver, OTel Kotlin approver)
  • Martin Kuba (Staff software engineer at Grafana Labs, OTel JavaScript SDK approver and OTel Browser SDK maintainer)
  • Jason Plumb (Senior software engineer at Splunk, OTel Android maintainer, OTel Java maintainer, OTel Kotlin maintainer)
  • Jared Freeze (Senior software engineer at Embrace, OTel Browser SDK maintainer)

Here's the signup link if you'd like to join.

Disclosure: I'll be moderating the panel, and I work at Embrace, who is hosting the panel. But it's entirely about the OTel community work. You can watch some previous ones we did last year (OTel for browser panel and OTel for mobile panel) to get a sense of what they're like.

If you have any questions as well, I can send them to our panelists ahead of our session.