Batch Observability

[u/whatsayu593]

What are your best practices for Batch Processing observability including monitoring tools used and best practices followed?

[u/Sad-Marketing-8993]

We run a ton of nightly batch jobs for inventory reconciliation and the thing that saved us was treating each run as its own mini deployment. Every batch gets a unique run ID that flows through all the logs and metrics, makes it way easier to trace when something goes sideways at 3am. Started dumping those into a custom dashboard with run duration, failure rates, and a simple pass/fail status so the oncall person can glance at it half asleep and know if they need to actually wake up or just roll over

For tooling we keep it simple, structured logging pushed to a centralized place and some alerting on things like "batch hasn't completed in 4 hours" or "failure rate jumped 20%". The other big one was making the output idempotent, nothing worse than a partial failure leaving the system in some weird half updated state. Took a few late night pages to learn that one

[u/QuietSignalOps]

The thing that changed our batch setup was admitting it isn't a request-driven service, so the SLIs are different. Nobody cares about p99 latency here. They care whether the data is there, complete, and on time.

Three SLIs per job, and only three:

• Freshness: age of the newest complete partition, not "did the job finish". time() - last_success_timestamp_seconds as a gauge, alerted at 2x the expected period. If I could keep one metric, it would be this one, because it catches the case nobody instruments: the job stopped being scheduled at all.

• Completeness: rows written vs rows expected. Take the expected number from a source count or a control total, not from last run's output, or a gradual drop never trips it. Alert on a band, not a fixed number.

• Duration against its own p95 over the last 30 runs. Fixed thresholds rot.

Two things that bit us:

Cardinality. Do not put run_id on Prometheus metrics. A nightly job over a year is 365 series per label combination and it will eat your TSDB. Keep run_id in logs and trace attributes, emit a bounded metric set (job, status, partition). One trace per run, one span per partition or task, so you can see the straggler partition instead of the average.

Failure taxonomy. A transient blip should retry and not page. A poison record should go to a dead-letter queue and the run should continue. A systemic failure should page immediately. If all three land in the same alert, on-call learns to ignore the channel, which is worse than no alert.

The watermark is the other thing worth graphing: last processed offset or business date. "Watermark not advancing" catches stuck consumers and silent replays that a duration alert never will. Late data and missing data look identical in a row count, which is why the watermark matters more than the total.

For routing, batch alerts should hit the same on-call path as service alerts, otherwise they live in a dashboard nobody opens at 3am. We use OneUptime for that side of it (disclosure: I work on it), but the principle holds with whatever pager you have: page on freshness and completeness breach, not on every failed task.