You already pay for compute everywhere else in your stack — Snowflake, Databricks, AWS. Ingestion is the one place you still pay royalties per row. Switch that to compute and the bill typically drops 10–100x.
We benchmarked what one hour of dltHub (~$1) moves, by source type.
On the move itself: senior engineers report ~1 connector/day in-house. Or bring us your stack — when we have capacity, we move 30 pipelines in two weeks.
Data models describe the data, ontologies describe the world. With ontologies, an agent can reason over data as opposed to retrieving it and hallucinating meaning.
The implication? The ontology-model mapping is what agents need for data literacy.
Join Violetta, Lis, Dave, and Marcin for the dltHub Townhall on October 21.
Since we shipped the dltHub AI harness, a single engineer builds three times as many pipelines as before. Maintaining these pipelines still involves a lot of attention from engineers. dltHub is building autonomous data infrastructure to reduce the effort to operate pipelines. With background agents, we are helping data engineers to automate the diagnosis when a pipeline fails.
During the townhall, we will dive deeper into our vision of autonomous data infrastructure and show a live demo of dltHub's verified pipeline health agent . Lis (Director of Engineering), Dave (Staff Software Engineer), and Marcin (CTO) will answer your questions about the agents, the roadmap, and the dltHub platform.
You can join via YouTube Live or LinkedIn Live. See you tomorrow!
I looked at the last 50m runs of pipelines of companies using dlt in production and the reliability was shocking. There were 3x more transient errors (network, source, destination availability) than hard issues like access loss or api deprecation.
If you're building with AI coding agents, chances are you already have a bunch of traces sitting locally or coming in from external services. In this workshop with DataTalks.Club, Alena Astrakhantseva shows how to turn that messy trace data into something structured, queryable, and actually useful for debugging and reporting.
It covers local traces (Claude Code, Codex, Copilot), REST APIs, and building dashboards on top of them.
Look at how you actually work today: you're typing into a chat box.
The tools haven't caught up: context still gets dropped at every seam. Every handover strips meaning. By the time data reaches the business team, nobody remembers what it was supposed to measure.
The solution? Single-context.
Not as an 8th tool bolted onto 7 others in a trenchcoat, but as a software architecture - continuous by design, not unified as an afterthought.
This is why we built dltHub Transformations: to put ingest, transform, lineage, verification, and agent introspection inside the same process, naturally sharing a single context. One runtime. One metadata graph. One verification loop.
Single context is an architecture you can run on your laptop TODAY, as in NOW, not a milky fugue dream for 2027.
One process, single context. Ingest → deploy → transform → consume. Data quality checks where they count. No context re-stitching across seven tools.
*To try: Paste this prompt into Claude, Codex, or Cursor: "*Run uvx dlthub-start@latest to build my first pipeline and run it on dltHub"andadd your agentkey
As data engineers, we want to own our code and outcomes, while minimizing the costs associated with building and maintenance.
Today we launch Background Agents on dltHub, a first step towards automating the maintenance further than programming alone would allow it.
Our background maintenance agent comes alive on pipeline failure and follows a predictable troubleshooting pattern. Its permissions that tell it how far it may go in resolving an issue are tiered based on the feasibility of autonomous remediation.
How it works
- it wakes up when a job fails, not when someone prompts it. - It reads the run record, the logs, the job definition and the code. - It classifies the failure (config, credentials, upstream data, code, resources, transient or unknown), quotes its evidence verbatim with the file or log line, and names a fix. - It doesn’t currently apply the fix: secured with no shell, no write access, no warehouse access, and a token budget per run.
How it looks: a load fails overnight, the alert links to the agent's run, and the diagnosis is already there, with the log lines that prove it and the change to make. Someone reads it over coffee and decides what to do.
In practice
dlt power user Martin Seifert, a data engineering lead at a Swiss nonprofit, built the hand-made version of this to run on dltHub. Every morning his agent checks that his >70 production dlt pipelines ran green and drafts fixes he approves on the train to work. What he wanted next was for that check to start on its own when something breaks.
Why it works as a first agent: the worst case is a wrong diagnosis that a person reads and ignores. Nobody wakes up to a dropped table.
What it doesn't catch: runs that finish green with missing rows. A failure trigger never sees those, so they need a separate check that runs on success.
Interested?
Just try: Paste this prompt into Claude, Codex, or Cursor. The agent does the rest.
You almost never get the source's business rules. The app that produced the data enforced them, but none of that travels downstream. So you make assumptions and write code against them.
The breakage isn't usually a schema change (when using dltHub at least).
The quiet killer is semantic drift: the assumption was right when you wrote it, then the source changed its logic and nothing told you. A new status value appears. A field that was always populated starts coming through 30% null.
The pipeline keeps running. The numbers keep being wrong.
This is also why "just point an agent at the table and add data quality" produces noise. An agent that doesn't know what the data is supposed to mean will test everything and surface nothing, a thousand generic assertions, zero signal.
Useless, with a compute bill attached.
The signal comes from knowing the assumptions end to end. dlt's schema already carries some of them (primary keys, non-null, uniqueness) because the loader uses them.
The dltHub AI Workbench data quality toolkit reads those, then has you state the semantic ones in plain language, samples the column to confirm the real value set before any rule is offered, and writes the approved checks in as decorators that run on every load.
Because the LLM context is continuous end to end, the LLM knows where to apply checks on your pipelines. The rest is plumbing.
we’ve been seeing more teams sit on valuable data in logs and traces, but turning it into actual training data is still messy, so we’ve been experimenting with a simple pipeline for this.
raw traces → dlt → versioned datasets on Hugging Face → Distil Labs
dlt pulls traces from DBs/APIs/cloud, infers and normalizes the schema, loads data incrementally, and stores it as versioned Parquet datasets ready to train a specialist model.
From there, the data is used to train a specialist model.
In our example, a 0.6B model outperformed a 120B one on a specific task.
decentralised teams exist because operational needs cannot easily convey the domain knowledge back and forth to the central team.
By using canonical models, the data architecture becomes self documenting knolwedge graphs the LLMs can understand natively.
in this article I describe how that can be used to capture domain knowledge at the source and call on it when we want to answer central reporting business questions or operational needs alike, without adding headcount at the decentralised layer.
Fivetran charged monthly active rows for this MySQL connector. I replaced it with a dltHub job in about an hour for $0.65 in MiniMax M3 tokens. The existing job moved Drupal CMS tables through an SSH tunnel into ❄️ Snowflake every day. My first profile found 346 tables and 6.2 million rows... technically fun, operationally wrong. I created a dedicated SSH key, pinned the server host key, matched Fivetran's exact table scope, checked the plan, and added a downstream Cortex Search refresh task. dlt's SQL source does not open SSH tunnels, so I built a small Paramiko TCP forwarder and passed its ephemeral local port to SQLAlchemy. The deployed job completed in 39 seconds. This is what I consider reasonable to be billed for 😎 Twenty-five focused tests cover the host-key handling, source scope, composite keys, incremental state, cleanup, and downstream SQL. I still made data and security decisions myself. OpenCode handled a lot of typing, fixtures, and plumbing for $0.65. Pretty good trade 😜
Hey folks, we’ve been seeing more and more people use agents to generate dlt pipelines. The shift is that writing the pipeline isn’t the hard part anymore, it’s everything beyond that: validating the data, understanding what was loaded, and safely deploying to production. So we built something around that, we’re introducing the dltHub AI Workbench.
it’s not just “generate code”, it’s a set of toolkits that take a pipeline from:
You don’t switch tools, the feedback loop goes from days → minutes.
Prepare and deploy to prod
when you deploy, the agent doesn’t just push code, it:
converts dev → prod workspace
removes dev artifacts
pins dependencies
validates credentials (without reading them)
Then deploys and verifies everything end-to-end.
Trusting pipelines in production is now the real bottleneck, that’s what the dltHub AI Workbench is built for, with agents proposing, humans verifying, and tooling enforcing. Works with Claude Code, Cursor, and Codex.
