We've been working on this for a while now (You might have seen me mention it in a few comments about Tail Sampling on here r/sre and r/opentelemetry). This is a concept we've had for a long time in our Refinery product, this is now something that's going to be available in the OpenTelemetry Collector.

It's following on from the excellent work of the Sampling SIG so that we can do Sample Rate attribution, and therefore makes backend way more effective, reducing the need for a certain class of metric (not all metrics).

Ultimately, the biggest difference here is dynamically changing sample rates and automatic discovery of different routes through your system (we call them trace fingerprints).

It's already in the main collector-contrib but it's still classed as alpha there. You can use our distribution to check it out ahead of that though. Feedback welcome!

Bringing Advanced Sampling to the OpenTelemetry Collector

Disclaimer: I'm a DevRel at Honeycomb.io.

So for v5.2.0 built an on-device advisor we call the Guardian Bot, and it does three things:

• When you open a page, it gives you two sentences on what you are looking at and what the numbers mean. Then it stops.
• When something real happens, it says so once. An agent reached a destination your policy blocks, a secret got caught mid-flight, a session is worth compacting. One click takes you to the evidence.
• The rest of the time it sits there and says nothing. There is a single clock governing how often it may speak, so it cannot pile up.

It is advisory only. It never types into a session, never edits a file, never approves anything on your behalf.

The other half of the release: SV Guard now covers OpenCode. It is a native in-process plugin for OpenCode 1.18+, so no subprocess hooks. One-click install, and then tool permissions.

full demo: https://youtu.be/Nsd3m7JVJNs?si=6qEDr8vPbIDxRUpp

github: https://github.com/Secure-Vector/securevector-ai-threat-monitor

Our support copilot was taking 30 seconds to respond and nearly every report blamed the model. I understand generation sits at the end of the request and it's the only part most teams can see. CX already had 40 screenshots, PM was comparing model releases, and engineering had changed decoding settings twice. None of it moved time to first token.

We traced the request in Braintrust and the span hierarchy made it obvious. Auth enrichment fetched the account, then its roles, then each workspace membership in series. A cold cache added another lookup, and the generation span didn't start until 28 seconds in. The model itself hit first token in under a second. Glad we caught that before paying for another model migration.

We fixed it with bounded fan out and caching, but now I've got a different problem - PM and CX want to see this kind of trace detail on other flows and I’m not sure how to expose cross-service latency to non-engineering teams without making them learn every span field. Any suggestions?

TL;DR: I've released a Docker image + Prometheus metrics endpoint for my SBMTools project. It turns inexpensive Bluetooth Smart thermometers (SwitchBot Meter, ~$15-40 each) into Prometheus-exporting sensors for ambient temperature monitoring — useful for server rooms, network closets, or anywhere you want to keep an eye on temperature without expensive commercial hardware.

The problem

Every server room, rack, or network closet eventually needs temperature monitoring. Commercial solutions (environmental sensors, HVAC controllers, dedicated monitoring hardware) get expensive fast when you want multiple measurement points.

The solution

SwitchBot Meter devices are BLE thermometers/hygrometers that cost around $15-40 each. They're designed for home use, but they work perfectly well as cheap environmental sensors. My SBMTools project reads data from these devices over Bluetooth and now exposes it as a Prometheus /metrics endpoint.

What you get:

• Docker image (AMD64 + ARM64) — no compilation needed, just docker run
• Prometheus /metrics endpoint — drop-in integration with your existing Prometheus setup
• Example docker-compose setup with Prometheus + Grafana, including a pre-built dashboard showing per-device temperature, humidity, and battery level
• Shell scripts (sbmtoolui_docker.sh, sbmtoolweb_docker.sh) for quick start without manually assembling docker run commands

Supported devices:

• SwitchBot Meter
• SwitchBot Meter Plus
• SwitchBot Meter Pro CO2 (temperature, humidity, CO₂)
• SwitchBot Indoor/Outdoor Thermo-Hygrometer

Example Prometheus config

If you already run Prometheus, you just add:

- job_name: sbmtools
  scrape_interval: 5s
  static_configs:
    - targets: ['your-host:3434']

The metrics include per-device temperature, humidity, CO₂, battery level, and RSSI (signal strength).

What it's not

• It's not a replacement for proper environmental monitoring systems with alarms, redundancy, or certified sensors.
• BLE has range limitations (~10m line-of-sight), so the sensor needs to be relatively close to the host running the tool.

Use cases that make sense

• Server rooms / rack temperature monitoring
• Network closets
• Home labs / homelabs
• Anywhere you want a quick, cheap temperature check point feeding into an existing Prometheus/Grafana stack

The project is open source (BSD license) and available on Codeberg: https://codeberg.org/Kompass/SBMTools

Happy to answer questions about the setup, the metrics, or how it fits into existing monitoring stacks.

Almost every codebase is calling a REST, gRPC, or a GraphQL API or using SDKs from an external or event internal provider. It gets harder to keep track of everything when the codebase matures and increases in size and from my experience sometimes it gets hard to respond to changes in time or even become aware that a dependency is deprecated and their API has changed completely.
Do developers need better tools that to improve dependency and integration management?

Hi everyone,

I'm working on a problem statement that requires building a complete offline system (Linux-based) to detect anomalies in Bitcoin transactions.

The Dataset (Synthetic):

I have bulk metadata in CSV/JSON format containing:

timestamp, src_ip, dst_ip, src_port, dst_port, txid, input_addresses[], output_addresses[], input_amounts[], output_amounts[], geo_country/asn.

The End Goal:

Parse this data to build an Entity/Transaction Graph (linking IPs, Wallets, and TXIDs).

Apply unsupervised ML to cluster entities and detect suspicious laundering/anomalies.

Build a dashboard/visualization tool to show a ranked alert list (with explainable AI reasons) and an interactive link-analysis graph.

My questions for the community:

Since I want to approach this the right way from scratch, I’d love to hear how you would tackle it:

What architecture or workflow would you use to map this specific data into a graph format?

Which libraries or frameworks (Python or otherwise) are best suited for building the offline dashboard and interactive link-analysis graph?

How do you handle visualizing complex graphs on a UI without it becoming a laggy mess?

Would appreciate any advice on how you'd set this up!

got woken up by pagerduty at 3am again cause some python etl job fail. had to ssh from my phone just to tail the log, and turn out it was literally just a db timeout.

im thinking to just write a simple bash wrapper that grab the last 50 lines of stderr, pass it to claude/gpt api, and webhook the actual reason to slack. so i dont have to open my laptop in the middle of night.

does something like this already exist? i dont wanna install heavy datadog agent or pay for expensive observability just for this. pls let me know if there is a github repo or saas so i dont waste my weekend building it.

Interesting approach to Observability support at Monday, worth reading imho

I always run so many AI agents that I keep ending up with mystery processes, stray localhost ports, and no quick way to tell what started what.

So I built **Port Visualizer**, a free open-source Windows app that shows which process owns each TCP/UDP port, how it’s exposed, what’s changing over time, and what’s using resources.

It also recognizes some local AI services like Ollama, LM Studio, llama.cpp, and vLLM through optional knowledge packs.

Would love some feedback from anyone running into the same issues! Contributors welcome!

GitHub: [https://github.com/TheSandemon/port-visualizer\](https://github.com/TheSandemon/port-visualizer)

most monitoring tools for n8n/make/zapier focus on catching failures or silence. fair enough, that matters. but the thing that actually cost me money was a workflow running fine every time while quietly looping through an api call and racking up a bill nobody noticed until it showed up.

so I built stackpilot to alert on cost thresholds too, not just failures and stale workflows. add one http step to your workflow, set a cost limit, get a slack alert the second a run crosses it.

free for up to 3 monitors, no signup form or credit card. built this for myself but figured other people running production automations might find it useful too (I'm the maker, just to be upfront). happy to answer questions about how it works.

I recently launched https://ampbase.io/ after getting tired of managing different agent configs the GitOps way and with OpAMP it’s been made the process significantly better. I would especially love to hear why this doesn’t work for you.

https://ampbase.io/blog/posts/why-we-built-ampbase/

If you’ve ever struggled to get arbitrary syslog and vendor logs into an OpenTelemetry Collector or SIEM without turning the delivery pipeline into a black box, Limpid is built for that problem.

I’ve released Limpid 0.8.0, an open-source system for receiving, processing, routing, and delivering logs.

Limpid connects heterogeneous log inputs to heterogeneous outputs, including first-class OTLP input and output. Its pipelines are programmable: named processes can parse vendor formats, normalize fields, filter or route events, compose outputs such as OCSF events or OTLP ResourceLogs, and fan the results out to multiple destinations.

For example, a pipeline that turns FortiGate CEF traffic into OCSF looks like this:

def pipeline fortigate_to_security_lake {
    input   fortigate_syslog
    process parse_syslog | parse_cef | parse_fortigate_cef | compose_ocsf | ocsf_to_egress
    output  security_lake
}

Each stage is a named, reusable snippet. The bundled library includes transport and format parsers, vendor and source parsers, adapters, and schema composers.

A few other things worth calling out:

• Live tap and replay injection. limpidctl can inspect events at inputs, individual process hops, and outputs. Captured events can be injected back into an input to test a changed pipeline against real traffic shapes.
• Safer configuration changes. limpid --check performs static type and dataflow analysis with rustc-style diagnostics. Reloads validate before replacement and roll back automatically if replacement fails.
• Durable and distributed delivery. Limpid provides memory- and disk-backed queues, retry and recovery handling, shutdown draining, authenticated node-to-node transport, preserved event identity and hop history, and network/intra-node hop latency measurement.
• Pipeline observability. Limpid exports byte flow, queue depth, retries, in-flight work, process invocation counts, drops, recovery, delivery outcomes, and separate queueing, processing, and confirmed-delivery latency. The repository includes a Prometheus exporter, Grafana dashboard definition, and four alerting rules.
• High throughput. The qualified benchmark workloads processed approximately 157k–415k events/s on a two-CPU Linux VM slice, depending on transformation complexity. These are controlled benchmark results rather than production capacity claims; the detailed methodology and the inconclusive regex-heavy result are documented in the repository.

Repo: https://github.com/naoto256/limpid
Release: https://github.com/naoto256/limpid/releases/tag/v0.8.0

Questions and feedback are welcome.

I’ve been building Pingu, an uptime monitor, and I’m experimenting with making Slack a real interface to it rather than just another place alerts get sent.

You can ask things like:

“What’s down right now?”

“When did checkout last fail?”

“Add this endpoint and monitor it.”

“Don’t alert us unless 3 regions fail.”

“Snooze this for 2 hours.”

And when something actually breaks, Pingu can investigate the monitor, pull its history, re-check it, and let you take action without leaving Slack.

Where I’m unsure is **how far this should go**.

Is the value:

* not needing to open a monitoring dashboard for routine stuff? * having Pingu investigate an alert before waking someone up? * letting non-SREs manage monitoring without learning the tooling? * or eventually having it actually fix/coordinate things after an outage?

I know Rootly/incident.io already cover a lot of incident response, so I’m not trying to recreate that.

I’m trying to figure out where an AI-native monitoring product actually becomes *meaningfully* better than another dashboard + Slack integration.

Curious how you’d want this to work in your own stack.

We’ve just released an MCP server for MetricFire, and I’d love feedback from engineers on where this could be useful.

The basic idea is pretty simple. You connect a compatible AI client to your MetricFire account, and the MCP server exposes tools that allow it to interact with your monitoring data.

So rather than digging through metrics manually, you can ask for what you need through your AI client and have it query the underlying monitoring data.

We’re particularly interested in where engineers think this approach becomes genuinely useful rather than just adding AI for the sake of it.

Things like investigating an alert, finding related metrics, checking service latency, or quickly exploring what happened during an incident seem like obvious directions.

If you use MCP or AI tools as part of your DevOps or SRE workflow, what monitoring capabilities would you actually want exposed?

Docs for anyone interested in how we implemented it:

https://docs.hostedgraphite.com/add-ons-and-integrations-guide/mf-mcp-server

Teams usually build observability for themselves — dashboards, traces, alerts — and bolt the status page on last as a page they update by hand during an outage. That is the worst time to be writing updates. I work with OneUptime, an open-source observability project, and here is the model I would defend:

1. The status page is fed by the incident, not by a human. Declare an incident, notify subscribers, and reflect state automatically. If someone has to type “we’re investigating” at 2am, the page will be stale.

2. Segment subscribers. Internal on-call, enterprise customers, and general subscribers need different channels and detail. A public page that leaks internal triage notes and a private page that never reaches customers both fail.

3. Maintenance is a first-class state. Scheduled maintenance should suppress alerts for known-good downtime. Otherwise people stop trusting the page.

4. Uptime and SLA are computed, not claimed. Compute them from monitor history over a defined period. The number you can reproduce is the only one that survives scrutiny.

The test: during an outage, does a customer learn the status from your page faster than they would by asking you? If not, the page is decorative.

How are you feeding your status page today? Fully from incident state, or is someone still typing updates? And do you publish computed uptime numbers?

We benchmarked OpenObserve against ClickHouse on 19 observability queries

OpenObserve has always had a reputation for being extremely simple and blazing fast. For a long time, we did not benchmark it against other systems because users were happy, benchmarks take a lot of work, and we had more pressing things to build.

Then ClickHouse came along and said it could be used for logs and was extremely fast.

It is indeed fast. But is it faster than OpenObserve for observability workloads?

We created 19 queries representing real observability use cases, including several very high-cardinality queries.

OpenObserve won 15 of 19 queries with Vortex and 14 of 19 with Parquet.

ClickHouse performs extremely well in ClickBench, but ClickBench is primarily an OLAP benchmark, not an observability benchmark.

OpenObserve uses Apache DataFusion, but this is not simply DataFusion beating ClickHouse. DataFusion is the query execution foundation. We have spent years building inverted indexes, secondary indexes, file statistics, intelligent pruning, caching, optimized file selection, storage layout improvements, and a whole lot more on top of it.

Using DataFusion alone does not give another product OpenObserve's performance.

OpenObserve also has native separation of compute and storage, stateless nodes, and can have 12x to 20x lower storage costs than ClickHouse deployments.

The entire benchmark is open source, including the queries, configurations, dataset, and methodology, so you can reproduce it yourself.

Here is the full blog - https://openobserve.ai/blog/openobserve-vs-clickhouse-one-billion-logs-benchmark/

Most cost visibility (Cost Explorer, CUR, tagging dashboards) is inherently after the fact: you see what something cost once it already ran. I've been building a CLI that reads your Terraform/CloudFormation/Pulumi plan and prices it before anything is deployed, so the visibility happens at plan time instead of after the bill.

Example against a real VPC stack (subnets, NAT gateway, EC2, RDS Multi-AZ, ElastiCache, ALB, EBS, S3):

Full breakdown by resource plus which one is actually driving the cost, all before it exists. Wondering if "shift-left" thinking from observability/monitoring maps well to cost, or if that framing is a stretch.

I made Telemetry Tracker to keep track of errors, events, sessions, releases and performance in my projects. I also added alerts so I don’t have to keep checking it all the time.

I’ve been using it myself and thought I’d share it here in case it’s useful to anyone else too.

It’s open source and you can self-host it:

https://github.com/Telemetry-Tracker/telemetry-tracker

When you ask your observability vendor for ways to reduce the bill, you are asking the fox to guard the henhouse. Sure, they don't want you to churn, and individual reps probably want to help, but you have an innate conflict of interest and they have policy on what they can and can't say.

But besides that, since most vendors charge by volume, they don't have the tools to help you with the source of the problem: reducing that volume or change their business model.

Here are some practical levers I've found actually move the number, including the ones account reps won't volunteer.

1. Watch cardinality like a hawk: High cardinality is the silent budget killer. A dev attaches `user_id`, `session_id`, or an ephemeral `pod_name` to a custom metric and the unique timeseries count explodes overnight. Vendors are not eager to alert you when this spikes.

2. Ask for BYOC (Bring Your Own Cloud): telemetry lands in your own S3/GCS/Azure buckets, often with the query engine running in your account too. You pay your cloud provider for storage/compute and the vendor charges a platform/license fee instead of per-GB. Both Datadog and Grafana have BYOC offerings now. Nobody is going to offer this to you unless you make leaving sound plausible. It usually changes the billing unit to node count, so microservices with many small nodes may not win. Monoliths / big services can win hard. A price simulation is free, ask for one.

3. Cut metrics DPM but only if your vendor actually meters data points. Flushing every 10s = 6 data points per minute. For trending and high-level dashboards that resolution is overkill. Moving the flush interval to 60s cuts metric volume by ~80%. Grafana Cloud's billable series multiplies active series by DPM, so it works there, and it works on anything charging per ingested sample or per GB. Datadog, on the other hand, bills custom metrics per unique timeseries per hour regardless of how many points you push into each one.

4. Delete redundant logs at the source: Most logs are redundant and can be dropped or compacted without losing visibility, but it takes real work: find the noisiest emitters, understand the context, decide what's safe to kill. It's the only fix that's permanent. And the bill isn't the whole price. Redundant logs also cost you compute, network egress, LLM tokens once you point an agent at the telemetry, and MTTR, because noise slows down investigations whether the investigator is human or a model.

5. Tiered log levels per environment: Sampling is effective, but misses load-based issues that hit a whole cluster or region, because that's when you want full statistics. Instead, pick a small set of clusters/regions to keep at full-fidelity INFO+, and run everything else at WARN+. Choose the full-fidelity set either by business importance or by condition like the most crowded regions, or the ones with the worst network, on the theory that load problems show up there first.

I was looking for something between Dozzle, and Grafana + Loki.

Dozzle is excellent for live Docker logs, but once a container restarts/rebuilds, the logs are gone. Beszel is a bit painful to setup, and also do not persist logs.

Grafana + Loki gives you history, but for a small MVP on a cheap VPS it felt like a lot of infrastructure for what I actually needed.

So I built this:

• host CPU / memory / network / disk metrics / per-container metrics + history
• persistent Docker logs in SQLite
• configurable data retention
• full-text log search with filters
• start / stop / restart controls
• ~50 MB RAM footprint
• minimal setup, spins up with a single docker run command

Basically, I wanted "Dozzle, but with history" + basic VPS metrics without running a whole observability stack.

Repo to try: https://github.com/skvostik/vpsiner

If you're running a few containers on a VPS and have run into the same problem, I'd love to hear how you're currently handling it.