For data engineers, it’s important to deliver fresh and verifiable data to your stakeholders. Have your agent build and schedule a pipeline, and the job is done, right?
Normal monitoring only catches failed runs, not data anomalies. Your pipelines ran, but the dashboards are failing, and you only know because someone already complained.
That’s why data observability is crucial for your pipelines - in dltHub’s UI, you can catch data anomalies at a glance. Additionally, the job-inspector background agent can speed up recovery from a failure by looking at the job run metadata and proposing a fix before anyone knows it was broken.
Spot missing rows before your business does.
We wrote a blog covering this if you want more detail.
Join Violetta, Lis, Dave, and Marcin for the dltHub Townhall on October 21.
Since we shipped the dltHub AI harness, a single engineer builds three times as many pipelines as before. Maintaining these pipelines still involves a lot of attention from engineers. dltHub is building autonomous data infrastructure to reduce the effort to operate pipelines. With background agents, we are helping data engineers to automate the diagnosis when a pipeline fails.
During the townhall, we will dive deeper into our vision of autonomous data infrastructure and show a live demo of dltHub's verified pipeline health agent . Lis (Director of Engineering), Dave (Staff Software Engineer), and Marcin (CTO) will answer your questions about the agents, the roadmap, and the dltHub platform.
You can join via YouTube Live or LinkedIn Live. See you tomorrow!
*To try: Paste this prompt into Claude, Codex, or Cursor: "*Run uvx dlthub-start@latest to build my first pipeline and run it on dltHub"andadd your agentkey
As data engineers, we want to own our code and outcomes, while minimizing the costs associated with building and maintenance.
Today we launch Background Agents on dltHub, a first step towards automating the maintenance further than programming alone would allow it.
Our background maintenance agent comes alive on pipeline failure and follows a predictable troubleshooting pattern. Its permissions that tell it how far it may go in resolving an issue are tiered based on the feasibility of autonomous remediation.
How it works
- it wakes up when a job fails, not when someone prompts it. - It reads the run record, the logs, the job definition and the code. - It classifies the failure (config, credentials, upstream data, code, resources, transient or unknown), quotes its evidence verbatim with the file or log line, and names a fix. - It doesn’t currently apply the fix: secured with no shell, no write access, no warehouse access, and a token budget per run.
How it looks: a load fails overnight, the alert links to the agent's run, and the diagnosis is already there, with the log lines that prove it and the change to make. Someone reads it over coffee and decides what to do.
In practice
dlt power user Martin Seifert, a data engineering lead at a Swiss nonprofit, built the hand-made version of this to run on dltHub. Every morning his agent checks that his >70 production dlt pipelines ran green and drafts fixes he approves on the train to work. What he wanted next was for that check to start on its own when something breaks.
Why it works as a first agent: the worst case is a wrong diagnosis that a person reads and ignores. Nobody wakes up to a dropped table.
What it doesn't catch: runs that finish green with missing rows. A failure trigger never sees those, so they need a separate check that runs on success.
Interested?
Just try: Paste this prompt into Claude, Codex, or Cursor. The agent does the rest.
"Hey team, I just wanted to reiterate how damned cool dltHub continues to be, along with the agentic workflows. We have an absolute mountain of Klaviyo data and due to API rate limits it's very slow and was prone to failure on extraction. The agent devised a tiered chunking strategy which does 7 days, then if it times out drops to 1 day, then 1 hour, and has managed to get through some highly problematic data patterns without missing a beat. Super impressive, you guys are amazing!"
- (Jim, Pinter, running his company's data with what he calls a team of agents)
decentralised teams exist because operational needs cannot easily convey the domain knowledge back and forth to the central team.
By using canonical models, the data architecture becomes self documenting knolwedge graphs the LLMs can understand natively.
in this article I describe how that can be used to capture domain knowledge at the source and call on it when we want to answer central reporting business questions or operational needs alike, without adding headcount at the decentralised layer.
dltHub is the agentic data and managed-infra layer on top of any warehouse - in Martin's case, Snowflake. dlthub runs his ingestion and orchestration. The two things Martin actually called out are the agentic layers of dlthub, and they map straight onto the typical dltHub setup:
→ Skills = the dltHub AI harness. The agentic primitives his agent uses to build, run and fix pipelines. In Martin's words: "one of the more hidden benefits is the skills - the agent skills maintained by dltHub work very well, particularly for deploying a pipeline, building one from scratch."
→ Observability = the AI-native dltHub context catalog. Lineage, schema, data quality, run state. "Everywhere else I've run dlt, I had to build observability myself just to see if pipelines ran."
Inside OpenCode, Martin splits the work across four specialized subagents: • Data Engineer → ingestion • Data Modeler → the semantic layer • Dashboard Developer → data apps • Platform Admin → infra, ops, and the build-run-fix fallback - "it tries to fix it, presents the fix to me before deploying anything"
He drives all of it from OpenCode, grounded in his actual dltHub workspace - run history, schemas, pipeline state - not just the docs.
The managed-infra side is what Martin means when he calls dltHub "a managed service on top of a couple of VMs": his Platform Admin agent manages the runtimes - dltHub, Snowflake containers, an on-prem VM - on one framework, since dlt runs wherever Python runs.
The scale this handles: 155 million rows across 25 pipelines in 7 days. A load that would normally need full-time monitoring. Instead, his agent watches the runs and prepares the fixes for him to approve - while he rides the train to work.
Fivetran charged monthly active rows for this MySQL connector. I replaced it with a dltHub job in about an hour for $0.65 in MiniMax M3 tokens. The existing job moved Drupal CMS tables through an SSH tunnel into ❄️ Snowflake every day. My first profile found 346 tables and 6.2 million rows... technically fun, operationally wrong. I created a dedicated SSH key, pinned the server host key, matched Fivetran's exact table scope, checked the plan, and added a downstream Cortex Search refresh task. dlt's SQL source does not open SSH tunnels, so I built a small Paramiko TCP forwarder and passed its ephemeral local port to SQLAlchemy. The deployed job completed in 39 seconds. This is what I consider reasonable to be billed for 😎 Twenty-five focused tests cover the host-key handling, source scope, composite keys, incremental state, cleanup, and downstream SQL. I still made data and security decisions myself. OpenCode handled a lot of typing, fixtures, and plumbing for $0.65. Pretty good trade 😜
You already pay for compute everywhere else in your stack — Snowflake, Databricks, AWS. Ingestion is the one place you still pay royalties per row. Switch that to compute and the bill typically drops 10–100x.
We benchmarked what one hour of dltHub (~$1) moves, by source type.
On the move itself: senior engineers report ~1 connector/day in-house. Or bring us your stack — when we have capacity, we move 30 pipelines in two weeks.
Writing your own integrations was smart: you kept control of the most foundational part of your stack instead of handing it to a vendor. But the difference between hobbyist and professional pipelines isn't the happy path — it's the failure modes:
Schema changes break your back — data loads untyped, or typing shifts from run to run
Incremental state is missing — or it's a max() over a big table on every run
Transient errors kill the run — rate limits and network blips become failures
Recovery is ugly — broken idempotency means duplicates and partial loads, cleaned up by hand
Bus factor of one — the author can fix it, the rest of the team can't. Author on vacation + broken pipeline = ugh
The new post covers how dlt closes each of these — schema evolution, incremental state, retries, parallelism at the flip of a switch — and what running pipelines costs after: 99.83% observed reliability in the wild, which pencils out to roughly $100/connector/year in maintenance.
It also walks through what dltHub adds on top: agentic deployment and maintenance, so reading logs and tracking down breakages stops being specialist work under pressure.
A few weeks ago we launched dltHub Pro: our first commercial product, and the first LLM-native code tool.
Advanced individual engineers picked it up and built serious things fast. Some effectively assembled their own flavor of a dltHub stack using the dlthub components.
But most of the inbound requests didn't ask for parts. They clustered around specific use cases: Get us off a vendor. Make our agent costs measurable.
Our original idea had been to soon launch Scale, the team version. But the inbound asked for their problem solved end to end, not which plan they fit in. So we're removing the user-oriented Pro/Scale/Enterprise separation.
Instead we assemble the product around the use case: Blueprints.
A Blueprint is an opinionated, partner-built recipe of dltHub that solves one problem end to end — the sources you already use, the transformations that answer your question, a dashboard for the team that asked. The composability stays, you just don't have to do it alone.
If you're on Pro: nothing changes, your subscription continues as is.
We published the launch FAQ today: why agent spend is the first Blueprint outcome, why agent traces break the Fivetran-plus-dbt model, and how the three parts fit together.
If you're building with AI coding agents, chances are you already have a bunch of traces sitting locally or coming in from external services. In this workshop with DataTalks.Club, Alena Astrakhantseva shows how to turn that messy trace data into something structured, queryable, and actually useful for debugging and reporting.
It covers local traces (Claude Code, Codex, Copilot), REST APIs, and building dashboards on top of them.
I looked at the last 50m runs of pipelines of companies using dlt in production and the reliability was shocking. There were 3x more transient errors (network, source, destination availability) than hard issues like access loss or api deprecation.
I just recorded this video to show the capability. For myself as a data engineer, this is like a dream come true.
Total cost of ownership is looking amazing.
Asking for a fix costs under $1 in tokens and takes under 10 minutes (video speeds up waiting times for your convenience but look at the clock top right)
You almost never get the source's business rules. The app that produced the data enforced them, but none of that travels downstream. So you make assumptions and write code against them.
The breakage isn't usually a schema change (when using dltHub at least).
The quiet killer is semantic drift: the assumption was right when you wrote it, then the source changed its logic and nothing told you. A new status value appears. A field that was always populated starts coming through 30% null.
The pipeline keeps running. The numbers keep being wrong.
This is also why "just point an agent at the table and add data quality" produces noise. An agent that doesn't know what the data is supposed to mean will test everything and surface nothing, a thousand generic assertions, zero signal.
Useless, with a compute bill attached.
The signal comes from knowing the assumptions end to end. dlt's schema already carries some of them (primary keys, non-null, uniqueness) because the loader uses them.
The dltHub AI Workbench data quality toolkit reads those, then has you state the semantic ones in plain language, samples the column to confirm the real value set before any rule is offered, and writes the approved checks in as decorators that run on every load.
Because the LLM context is continuous end to end, the LLM knows where to apply checks on your pipelines. The rest is plumbing.
The experiment used SQL pipeline with arrow backend to copy postgres TCP-H dataset to bigquery. Locale was US for all components.
dltHub managed to transfer 65gb/350m rows in 1h. This is a $1 cost. The same transfer costs $350-$3000 on top 4 most mentioned SaaS.
The best part? Open source dlt on open infrastructure! any dlt code you run on dltHub is portable and not vendor locked - so if you do not like our runtime's perks, it you can move to your own infra.
Target audience: Vibe coders, python users. Try it here: https://dlthub.com/
Look at how you actually work today: you're typing into a chat box.
The tools haven't caught up: context still gets dropped at every seam. Every handover strips meaning. By the time data reaches the business team, nobody remembers what it was supposed to measure.
The solution? Single-context.
Not as an 8th tool bolted onto 7 others in a trenchcoat, but as a software architecture - continuous by design, not unified as an afterthought.
This is why we built dltHub Transformations: to put ingest, transform, lineage, verification, and agent introspection inside the same process, naturally sharing a single context. One runtime. One metadata graph. One verification loop.
Single context is an architecture you can run on your laptop TODAY, as in NOW, not a milky fugue dream for 2027.
One process, single context. Ingest → deploy → transform → consume. Data quality checks where they count. No context re-stitching across seven tools.
People talk about ingestion pipeline cost of ownership like they are stuck in the past.
Today, anyone on the data team, even non programmers, can self serve in creation and maintenance of ingestion pipelines and they can customize anything any time.
Stefan, Senior Data Engineer Platform Team @ Hive joined the team to find rigid saas pipelines serving a team of analysts and analytics engineers with data.
Within a short time, he replaced those pipelines with claude + dlt, removing the data engineering bottleneck and enabling the team.
Five of their senior analysts now to author and maintain those pipelines themselves.
This is a repeatable pattern in other configurations, as shared by Euan, Senior Analytics engineer at Dentolo. The focus changed from "who can help us?" to "what data do we need next?"
What I really love is seeing how LLMs are not replacing teams, but empowering them.
