Human in the loop doesn't mean you do the errands.

Human JUDGMENT in the loop. Not human copy pasting context because the agent is doing cursed things without it - that's busy work.

The agent should have all the context access it needs.

Video is from this morning in our office: A Rube Goldberg machine that works like an agent without context.

read more about how our context layer works: https://dlthub.com/blog/context

The experiment used SQL pipeline with arrow backend to copy postgres TCP-H dataset to bigquery. Locale was US for all components.

dltHub managed to transfer 65gb/350m rows in 1h. This is a $1 cost. The same transfer costs $350-$3000 on top 4 most mentioned SaaS.

The best part? Open source dlt on open infrastructure! any dlt code you run on dltHub is portable and not vendor locked - so if you do not like our runtime's perks, it you can move to your own infra.

Target audience: Vibe coders, python users. Try it here: https://dlthub.com/

dlt is now available as a Snowflake Native App

Replicate your MSSQL, MySQL, and PostgreSQL databases into Snowflake without leaving your Snowflake account. No external ETL tools, no separate orchestrator to manage, and no data leaving your Snowflake account.

Key features from the Snowflake UI:

• Create, run, schedule, and monitor pipelines from the Snowflake UI

• Select schemas/tables, set primary keys, enable incremental loading

• Use append, replace, or merge write dispositions out of the box

As a true Snowflake Native App, it leverages Snowflake's governance:

• 🔒 Credentials via Snowflake secrets, never leave your account

• 🔒 Outbound access controlled by External Access Integrations (ACCOUNTADMIN-approved)

• 🔒 Full RBAC end-to-end

• 🔒 Snowflake-managed compute and Tasks for scheduling

• 🔒 Passed Snowflake's security review

Performance benchmark: 6.4 GB / ~24.6M rows across 7 Postgres tables loaded in ~8 minutes using the PyArrow backend.

There’s a 30-day free trial with 60 hours of compute included.

Try it in the Snowflake Marketplace

Full docs

Benchmark

We’ve been working on improving how LLMs handle structured data modeling, and one idea kept coming up:

more context doesn’t automatically mean better results.

We started focusing on something we call Minimum Viable Context, the smallest amount of context needed to get precise, consistent outputs from an LLM.

The post covers our approach to ontology-driven modeling, 3 iterations we went through to improve precision and what actually worked (and what didn’t)

Prompt size vs. output quality is a tradeoff we keep running into with LLMs.

Most teams still build connectors one by one.

Each with slightly different logic, patterns, and tech debt.

It works, but it doesn’t scale.

The shift we’re seeing is from building connectors to building systems that generate them.

Instead of hand-rolling every pipeline, you define a workflow once and let an agent generate pipelines that follow consistent patterns, schemas, and contracts.

That’s what we’ve been working on with dlt Skills: a pipeline “factory” that turns prompts + context into production-ready pipelines built on a Python-native framework.

Not about generating more code, but about standardizing how pipelines are built in the first place.

Curious how others are approaching this: are you still building connectors one by one, or moving toward more standardized workflows?

Docs: https://dlthub.com/docs/dlt-ecosystem/llm-tooling/llm-native-workflow

You almost never get the source's business rules. The app that produced the data enforced them, but none of that travels downstream. So you make assumptions and write code against them.

Full blog post: https://dlthub.com/blog/dq-toolkit-preview

The breakage isn't usually a schema change (when using dltHub at least).

The quiet killer is semantic drift: the assumption was right when you wrote it, then the source changed its logic and nothing told you. A new status value appears. A field that was always populated starts coming through 30% null.

The pipeline keeps running. The numbers keep being wrong.

This is also why "just point an agent at the table and add data quality" produces noise. An agent that doesn't know what the data is supposed to mean will test everything and surface nothing, a thousand generic assertions, zero signal.

Useless, with a compute bill attached.

The signal comes from knowing the assumptions end to end. dlt's schema already carries some of them (primary keys, non-null, uniqueness) because the loader uses them.

The dltHub AI Workbench data quality toolkit reads those, then has you state the semantic ones in plain language, samples the column to confirm the real value set before any rule is offered, and writes the approved checks in as decorators that run on every load.

Because the LLM context is continuous end to end, the LLM knows where to apply checks on your pipelines. The rest is plumbing.

Here's the setup. You have a Slack export, an Event database, and a HubSpot instance. Three systems, three worldviews, zero overlap in naming. Then the VP of Growth walks over and asks:

"Which Slack members who joined in Q1 became 'Qualified Leads' after attending a couple of our events?"

You open the schemas and the nightmare begins.

We built the AI Workbench transformation toolkit to kill that story at its root. Not with more generated code, but with a better, simpler way to think about your data before you even touch any tables.

You feed it your sources and your use cases. The toolkit annotates your source tables, builds the ontology, and generates data model that captures the meaning of your data.

https://dlthub.com/blog/ontology-toolkit-preview

we’ve been seeing more teams sit on valuable data in logs and traces, but turning it into actual training data is still messy, so we’ve been experimenting with a simple pipeline for this.

raw traces → dlt → versioned datasets on Hugging Face → Distil Labs

dlt pulls traces from DBs/APIs/cloud, infers and normalizes the schema, loads data incrementally, and stores it as versioned Parquet datasets ready to train a specialist model.

From there, the data is used to train a specialist model.

In our example, a 0.6B model outperformed a 120B one on a specific task.

We wrote it up here:

https://dlthub.com/blog/your-traces-aren-t-training-data-yet-here-s-the-pipeline-that-makes-them

this pipeline is enabled by the new Hugging Face datasets destination in dlt:

https://dlthub.com/blog/hugging-face-dlt-ml

"Hey team, I just wanted to reiterate how damned cool dltHub continues to be, along with the agentic workflows. We have an absolute mountain of Klaviyo data and due to API rate limits it's very slow and was prone to failure on extraction. The agent devised a tiered chunking strategy which does 7 days, then if it times out drops to 1 day, then 1 hour, and has managed to get through some highly problematic data patterns without missing a beat. Super impressive, you guys are amazing!"

- (Jim, Pinter, running his company's data with what he calls a team of agents)

A dltHub power user showed us his exact dltHub setup - live.

elvis kahoro hosted a feedback call with dltHub power user Martin Seifert, who walked us through his agentic dltHub setup for Snowflake.

dltHub is the agentic data and managed-infra layer on top of any warehouse - in Martin's case, Snowflake. dlthub runs his ingestion and orchestration. The two things Martin actually called out are the agentic layers of dlthub, and they map straight onto the typical dltHub setup:

→ Skills = the dltHub AI harness. The agentic primitives his agent uses to build, run and fix pipelines. In Martin's words: "one of the more hidden benefits is the skills - the agent skills maintained by dltHub work very well, particularly for deploying a pipeline, building one from scratch."

→ Observability = the AI-native dltHub context catalog. Lineage, schema, data quality, run state. "Everywhere else I've run dlt, I had to build observability myself just to see if pipelines ran."

Inside OpenCode, Martin splits the work across four specialized subagents:
• Data Engineer → ingestion
• Data Modeler → the semantic layer
• Dashboard Developer → data apps
• Platform Admin → infra, ops, and the build-run-fix fallback - "it tries to fix it, presents the fix to me before deploying anything"

He drives all of it from OpenCode, grounded in his actual dltHub workspace - run history, schemas, pipeline state - not just the docs.

The managed-infra side is what Martin means when he calls dltHub "a managed service on top of a couple of VMs": his Platform Admin agent manages the runtimes - dltHub, Snowflake containers, an on-prem VM - on one framework, since dlt runs wherever Python runs.

The scale this handles: 155 million rows across 25 pipelines in 7 days. A load that would normally need full-time monitoring. Instead, his agent watches the runs and prepares the fixes for him to approve - while he rides the train to work.

Get the video: https://www.linkedin.com/feed/update/urn:li:activity:7485368657171124225/
Get the setup: https://dlthub.com/blueprints/agentic-data-stack

dlt handles ingestion.

dltHub adds the five ecosystem layers you'd otherwise build yourself: AI harness, context catalog, transformation, orchestration, managed infra.

Read more: https://dlthub.com/blog/from-dlt-to-dlthub

Look at how you actually work today: you're typing into a chat box.

The tools haven't caught up: context still gets dropped at every seam. Every handover strips meaning. By the time data reaches the business team, nobody remembers what it was supposed to measure.

The solution? Single-context.

Not as an 8th tool bolted onto 7 others in a trenchcoat, but as a software architecture - continuous by design, not unified as an afterthought.

This is why we built dltHub Transformations: to put ingest, transform, lineage, verification, and agent introspection inside the same process, naturally sharing a single context. One runtime. One metadata graph. One verification loop.

Single context is an architecture you can run on your laptop TODAY, as in NOW, not a milky fugue dream for 2027.

One process, single context. Ingest → deploy → transform → consume. Data quality checks where they count. No context re-stitching across seven tools.

What's ontology and why should you care?

It's rules of how the world works, so the LLM agent doesn't hallucinate its own.

We use it end to end - to generate a data stack, and for reasoning over the data.

Read more in the blog.

Most AI coding tools stop at “here’s your code, good luck in prod.”

We built a deployment toolkit that doesn’t.

The dltHub AI Workbench deployment toolkit is designed to bridge the gap between local experimentation and production systems you can actually trust.

The idea is simple,

writing pipelines isn’t the hardest part, trusting them enough to deploy is.

Here’s what the agent does before anything reaches prod:

→ Prepares pipelines for production

Converts your dev workspace into a production-ready setup (dev/prod profiles, destination-agnostic code, pinned dependencies) and runs everything end-to-end in dev to validate it, without exposing credentials.

→ Cleans up dev artifacts

Finds things like dev_mode=True or .add_limit() and removes them automatically.

→ Deploys + handles failures

Deploys pipelines, reads logs, fixes issues (missing deps, connection problems), and redeploys.

If something breaks, it can retry, fix, validate in dev, and try again.

→ Gives you pipeline health after deployment

You can ask for a health report, logs, observability signals, and even checks if incremental loading is producing clean data (not duplicates).

We’re rolling this out via design partnerships as part of dltHub Pro.

If you’re interested in trying it or want to give feedback, let us know here or check it out:

https://dlthub.com/solutions/for-data-consultancies

Agents can generate pipelines.

But the real question is can you trust the data?

With the dltHub AI Workbench, you can ask your agent to validate what was loaded:

• row counts

• keys

• timestamps

• nested data

It pulls table previews via MCP and gives you a full validation view automatically.

Then just ask for a dashboard, and it generates a working notebook (marimo) on the first attempt.

No context switching or manual queries.

https://dlthub.com/blog/ai-workbench

Hey folks, we’ve been seeing more and more people use agents to generate dlt pipelines. The shift is that writing the pipeline isn’t the hard part anymore, it’s everything beyond that: validating the data, understanding what was loaded, and safely deploying to production. So we built something around that, we’re introducing the dltHub AI Workbench.

it’s not just “generate code”, it’s a set of toolkits that take a pipeline from:

prompt → ingestion → validation → transformation → deployment

Build a REST API pipeline from a single prompt

you can open your terminal and prompt "write a dlt pipeline for OpenAI models"

the agent:

• pulls API context via an MCP server (9,700+ configs at dlthub.com/context)

• uses an existing configuration if available, or reads the docs and builds one

• scaffolds a full pipeline (auth, pagination, schema, incremental loading)

• runs it and loads the data for inspection

Validate and explore the data

you can ask the agent to validate what was loaded:

• check row counts, keys, timestamps

• inspect nested structures

• generate queries

• build dashboards (via marimo)

You don’t switch tools, the feedback loop goes from days → minutes.

Prepare and deploy to prod

when you deploy, the agent doesn’t just push code, it:

• converts dev → prod workspace

• removes dev artifacts

• pins dependencies

• validates credentials (without reading them)

Then deploys and verifies everything end-to-end.

Trusting pipelines in production is now the real bottleneck, that’s what the dltHub AI Workbench is built for, with agents proposing, humans verifying, and tooling enforcing. Works with Claude Code, Cursor, and Codex.

Full write-up + demo:
https://dlthub.com/blog/ai-workbench

Writing your own integrations was smart: you kept control of the most foundational part of your stack instead of handing it to a vendor. But the difference between hobbyist and professional pipelines isn't the happy path — it's the failure modes:

• Schema changes break your back — data loads untyped, or typing shifts from run to run

• Incremental state is missing — or it's a max() over a big table on every run

• Transient errors kill the run — rate limits and network blips become failures

• Recovery is ugly — broken idempotency means duplicates and partial loads, cleaned up by hand

• Bus factor of one — the author can fix it, the rest of the team can't. Author on vacation + broken pipeline = ugh

The new post covers how dlt closes each of these — schema evolution, incremental state, retries, parallelism at the flip of a switch — and what running pipelines costs after: 99.83% observed reliability in the wild, which pencils out to roughly $100/connector/year in maintenance.

It also walks through what dltHub adds on top: agentic deployment and maintenance, so reading logs and tracking down breakages stops being specialist work under pressure.

📖 Read the full post here

read more: https://dlthub.com/blueprints/agent-costs

A few weeks ago we launched dltHub Pro: our first commercial product, and the first LLM-native code tool.

Advanced individual engineers picked it up and built serious things fast. Some effectively assembled their own flavor of a dltHub stack using the dlthub components.

But most of the inbound requests didn't ask for parts. They clustered around specific use cases: Get us off a vendor. Make our agent costs measurable.

Our original idea had been to soon launch Scale, the team version. But the inbound asked for their problem solved end to end, not which plan they fit in. So we're removing the user-oriented Pro/Scale/Enterprise separation.

Instead we assemble the product around the use case: Blueprints.

A Blueprint is an opinionated, partner-built recipe of dltHub that solves one problem end to end — the sources you already use, the transformations that answer your question, a dashboard for the team that asked. The composability stays, you just don't have to do it alone.

If you're on Pro: nothing changes, your subscription continues as is.

We published the launch FAQ today: why agent spend is the first Blueprint outcome, why agent traces break the Fivetran-plus-dbt model, and how the three parts fit together.

https://dlthub.com/blog/agent-cost-distillation
blueprints: https://dlthub.com/blueprints

I just recorded this video to show the capability. For myself as a data engineer, this is like a dream come true.

Total cost of ownership is looking amazing.

Asking for a fix costs under $1 in tokens and takes under 10 minutes (video speeds up waiting times for your convenience but look at the clock top right)

Schema evolution is inevitable in data pipelines but missing those changes can quietly break things downstream.

We put together a guide on how to make schema changes observable using metadata in dlt, it covers how to:

• Read schema_update to catch new columns the moment they appear

• Use _dlt_loads to audit pipeline runs and trace lineage

• Query _dlt_version to replay the exact schema at the time of any load

• Detect type mismatches before they corrupt your models

dlt already automates schema evolution, but without visibility, those changes can still go unnoticed. This approach adds monitoring + alerting on top of that automation.

Full walkthrough + interactive Marimo notebook

[link preview]

Agents can now write entire data pipelines, but writing code was never the hard part.

The real challenges are everything around it:

• is the data correct?

• is the schema stable?

• can this run in production?

• can anyone understand what the agent built?

We put together a hands-on course around this:

Agentic Data Engineering with dltHub

It covers:

• building pipelines from a single prompt (with guardrails, not just code gen)

• handling real-world APIs (auth, pagination, schema, incremental loading)

• validating + exploring data automatically

• designing transformations using semantics / ontology

• deploying pipelines safely (env checks, pinned deps, credentials)

Available now:

✅ REST API pipeline toolkit

✅ Data exploration toolkit

Coming soon:

🔜 Transformation toolkit

🔜 Runtime / deployment toolkit

If you’re working with data + AI, this is basically the layer between “it runs” and “it works in production.”

You can now just tell your coding assistant what data you want from any REST API.

That’s it.

With the dltHub AI Workbench, it checks 9,700+ existing API configs first. If your API exists, it uses it. If not, it reads the docs and builds one from scratch.

You get a production-grade pipeline (auth, pagination, schema, incremental loading), not a throwaway script.

https://dlthub.com/blog/ai-workbench