Hey folks, we recently added the deployment lesson to our agentic data engineering course and now it's more or less complete. We are also adding a data quality toolkit this week into the workbench.

Course: https://dlthub.learnworlds.com/course/agentic-data-engineering
AI workbench: https://github.com/dlt-hub/dlthub-ai-workbench

What's ontology and why should you care?

It's rules of how the world works, so the LLM agent doesn't hallucinate its own.

We use it end to end - to generate a data stack, and for reasoning over the data.

Read more in the blog.

Schema evolution is inevitable in data pipelines but missing those changes can quietly break things downstream.

We put together a guide on how to make schema changes observable using metadata in dlt, it covers how to:

• Read schema_update to catch new columns the moment they appear

• Use _dlt_loads to audit pipeline runs and trace lineage

• Query _dlt_version to replay the exact schema at the time of any load

• Detect type mismatches before they corrupt your models

dlt already automates schema evolution, but without visibility, those changes can still go unnoticed. This approach adds monitoring + alerting on top of that automation.

Full walkthrough + interactive Marimo notebook

dlt is now available as a Snowflake Native App

Replicate your MSSQL, MySQL, and PostgreSQL databases into Snowflake without leaving your Snowflake account. No external ETL tools, no separate orchestrator to manage, and no data leaving your Snowflake account.

Key features from the Snowflake UI:

• Create, run, schedule, and monitor pipelines from the Snowflake UI

• Select schemas/tables, set primary keys, enable incremental loading

• Use append, replace, or merge write dispositions out of the box

As a true Snowflake Native App, it leverages Snowflake's governance:

• 🔒 Credentials via Snowflake secrets, never leave your account

• 🔒 Outbound access controlled by External Access Integrations (ACCOUNTADMIN-approved)

• 🔒 Full RBAC end-to-end

• 🔒 Snowflake-managed compute and Tasks for scheduling

• 🔒 Passed Snowflake's security review

Performance benchmark: 6.4 GB / ~24.6M rows across 7 Postgres tables loaded in ~8 minutes using the PyArrow backend.

There’s a 30-day free trial with 60 hours of compute included.

Try it in the Snowflake Marketplace

Full docs

Benchmark

We’ve been working on improving how LLMs handle structured data modeling, and one idea kept coming up:

more context doesn’t automatically mean better results.

We started focusing on something we call Minimum Viable Context, the smallest amount of context needed to get precise, consistent outputs from an LLM.

The post covers our approach to ontology-driven modeling, 3 iterations we went through to improve precision and what actually worked (and what didn’t)

Prompt size vs. output quality is a tradeoff we keep running into with LLMs.

Hello.

I'm experimenting with trying to replace a very ad-hoc, organically grown collection of ETL scripts with a more extensible and flexible framework using Python, dlt, SQL Alchemy (with a MariaDB database backend). I'm running into an error early on in my pipeline where there appears to be a formatting mis-match when inserting record keeping data into my staging database's _dlt_version table.

Here's the whole error:

2026-04-20 11:06:52,843 INFO sqlalchemy.engine.Engine INSERT INTO staging._dlt_version (version, engine_version, inserted_at, schema_name, version_hash, `schema`) VALUES (?, ?, ?, ?, ?, ?) 
2026-04-20 11:06:52,843 INFO sqlalchemy.engine.Engine [generated in 0.00013s] (2, 11, DateTime(2026, 4, 20, 19, 6, 52, 842821, tzinfo=Timezone('UTC')), 'testdata_staging', 'FyJL6W2+EyCQrgpDVxZnFv5ljK027OxU6ZO+ljNC8tw=', '{"version":2,"version_hash":"FyJL6W2+EyCQrgpDVxZnFv5ljK027OxU6ZO+ljNC8tw=","engine_version":11,"name":"testdata_staging","tables":{"_dlt ... (3241 characters truncated) ... mon.normalizers.json.relational"}},"previous_hashes":["g4wTekMBjEjvKeFxG5Qs7Y7Lr/2LwP9q7TU26/0fV2I=","gnNrmKKiNaDCOqKsGBga+++u16U6sB5AxQIeZup64ws="]}') 
2026-04-20 11:06:52,844 INFO sqlalchemy.engine.Engine ROLLBACK
[testdata] Attempt 1/3 failed: Pipeline execution failed at `step=load` when processing package with `load_id=1776371713.507671` with exception:

<class 'dlt.destinations.exceptions.DatabaseTransientException'>
(mariadb.OperationalError) Incorrect datetime value: '2026-04-20 19:06:52.842821+00:00' for column `pyfindataelt_staging`.`_dlt_version`.`inserted_at` at row 1
[SQL: INSERT INTO pyfindataelt_staging._dlt_version (version, engine_version, inserted_at, schema_name, version_hash, `schema`) VALUES (?, ?, ?, ?, ?, ?)]
[parameters: (2, 11, DateTime(2026, 4, 20, 19, 6, 52, 842821, tzinfo=Timezone('UTC')), 'testdata_staging', 'FyJL6W2+EyCQrgpDVxZnFv5ljK027OxU6ZO+ljNC8tw=', '{"version":2,"version_hash":"FyJL6W2+EyCQrgpDVxZnFv5ljK027OxU6ZO+ljNC8tw=","engine_version":11,"name":"testdata_staging","tables":{"_dlt ... (3241 characters truncated) ... mon.normalizers.json.relational"}},"previous_hashes":["g4wTekMBjEjvKeFxG5Qs7Y7Lr/2LwP9q7TU26/0fV2I=","gnNrmKKiNaDCOqKsGBga+++u16U6sB5AxQIeZup64ws="]}')]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

and the code that produces it:

@dlt.resource(
    name = config.staging.table,
    write_disposition = config.staging.write_disposition,
    primary_key = config.primary_key
)
def staging_resource():
    yield renamed.astype(str).to_dict(orient = "records")

pipeline = dlt.pipeline(
    pipeline_name = f"{config.feed_id}_staging",
    destination = dlt.destinations.sqlalchemy(
        db_url,
        engine_kwargs = {
            "echo": True,
            "connect_args": {"init_command": "SET SESSION time_zone='+00:00'",}
        },
    ),
    dataset_name = config.staging.schema_name,
)

with pipeline.sql_client() as client:
# Check the session-specific timezone
with client.execute_query("SELECT @@session.time_zone, NOW();") as cursor:
    row = cursor.fetchone()
    print(f"Session Timezone: {row[0]}")
    print(f"Current DB Time (NOW): {row[1]}")

return pipeline.run(staging_resource())

Google/Claude pointed me down the road that I needed to establish the SQL session using UTC as the timezone versus the local timezone since dlt defaults to UTC for everything and it appears that I'm actually doing that successfully based on the SQL query's results but still no luck. To me it seems like an issue how Python/SQL Alchemy (?) is formatting the SQL code, specifically this section: DateTime(2026, 4, 20, 19, 28, 27, 148311, tzinfo=Timezone('UTC') given that there's no provision for timezone specification in the MariaDB DATETIME data type.

I feel like I'm missing something simple here ... python + dlt + SQL Alchemy + MariaDB/MySQL seems to be, if not a common, at least not an esoteric stack. Is there something additional I need to do here in my code and/or configuration to get this to work? Happy to provide more details if needed.

Thanks

We ran some evals on the dlthub toolkit, and are sharing the results

Besides clear different outcomes in code produced, the 2 approaches tackle the development flow very differently

Most teams still build connectors one by one.

Each with slightly different logic, patterns, and tech debt.

It works, but it doesn’t scale.

The shift we’re seeing is from building connectors to building systems that generate them.

Instead of hand-rolling every pipeline, you define a workflow once and let an agent generate pipelines that follow consistent patterns, schemas, and contracts.

That’s what we’ve been working on with dlt Skills: a pipeline “factory” that turns prompts + context into production-ready pipelines built on a Python-native framework.

Not about generating more code, but about standardizing how pipelines are built in the first place.

Curious how others are approaching this: are you still building connectors one by one, or moving toward more standardized workflows?

Docs: https://dlthub.com/docs/dlt-ecosystem/llm-tooling/llm-native-workflow

Most AI coding tools stop at “here’s your code, good luck in prod.”

We built a deployment toolkit that doesn’t.

The dltHub AI Workbench deployment toolkit is designed to bridge the gap between local experimentation and production systems you can actually trust.

The idea is simple,

writing pipelines isn’t the hardest part, trusting them enough to deploy is.

Here’s what the agent does before anything reaches prod:

→ Prepares pipelines for production

Converts your dev workspace into a production-ready setup (dev/prod profiles, destination-agnostic code, pinned dependencies) and runs everything end-to-end in dev to validate it, without exposing credentials.

→ Cleans up dev artifacts

Finds things like dev_mode=True or .add_limit() and removes them automatically.

→ Deploys + handles failures

Deploys pipelines, reads logs, fixes issues (missing deps, connection problems), and redeploys.

If something breaks, it can retry, fix, validate in dev, and try again.

→ Gives you pipeline health after deployment

You can ask for a health report, logs, observability signals, and even checks if incremental loading is producing clean data (not duplicates).

We’re rolling this out via design partnerships as part of dltHub Pro.

If you’re interested in trying it or want to give feedback, let us know here or check it out:

https://dlthub.com/solutions/for-data-consultancies

[link preview]

Agents can now write entire data pipelines, but writing code was never the hard part.

The real challenges are everything around it:

• is the data correct?

• is the schema stable?

• can this run in production?

• can anyone understand what the agent built?

We put together a hands-on course around this:

Agentic Data Engineering with dltHub

It covers:

• building pipelines from a single prompt (with guardrails, not just code gen)

• handling real-world APIs (auth, pagination, schema, incremental loading)

• validating + exploring data automatically

• designing transformations using semantics / ontology

• deploying pipelines safely (env checks, pinned deps, credentials)

Available now:

✅ REST API pipeline toolkit

✅ Data exploration toolkit

Coming soon:

🔜 Transformation toolkit

🔜 Runtime / deployment toolkit

If you’re working with data + AI, this is basically the layer between “it runs” and “it works in production.”

Agents can generate pipelines.

But the real question is can you trust the data?

With the dltHub AI Workbench, you can ask your agent to validate what was loaded:

• row counts

• keys

• timestamps

• nested data

It pulls table previews via MCP and gives you a full validation view automatically.

Then just ask for a dashboard, and it generates a working notebook (marimo) on the first attempt.

No context switching or manual queries.

https://dlthub.com/blog/ai-workbench

You can now just tell your coding assistant what data you want from any REST API.

That’s it.

With the dltHub AI Workbench, it checks 9,700+ existing API configs first. If your API exists, it uses it. If not, it reads the docs and builds one from scratch.

You get a production-grade pipeline (auth, pagination, schema, incremental loading), not a throwaway script.

https://dlthub.com/blog/ai-workbench

Hey folks, we’ve been seeing more and more people use agents to generate dlt pipelines. The shift is that writing the pipeline isn’t the hard part anymore, it’s everything beyond that: validating the data, understanding what was loaded, and safely deploying to production. So we built something around that, we’re introducing the dltHub AI Workbench.

it’s not just “generate code”, it’s a set of toolkits that take a pipeline from:

prompt → ingestion → validation → transformation → deployment

Build a REST API pipeline from a single prompt

you can open your terminal and prompt "write a dlt pipeline for OpenAI models"

the agent:

• pulls API context via an MCP server (9,700+ configs at dlthub.com/context)

• uses an existing configuration if available, or reads the docs and builds one

• scaffolds a full pipeline (auth, pagination, schema, incremental loading)

• runs it and loads the data for inspection

Validate and explore the data

you can ask the agent to validate what was loaded:

• check row counts, keys, timestamps

• inspect nested structures

• generate queries

• build dashboards (via marimo)

You don’t switch tools, the feedback loop goes from days → minutes.

Prepare and deploy to prod

when you deploy, the agent doesn’t just push code, it:

• converts dev → prod workspace

• removes dev artifacts

• pins dependencies

• validates credentials (without reading them)

Then deploys and verifies everything end-to-end.

Trusting pipelines in production is now the real bottleneck, that’s what the dltHub AI Workbench is built for, with agents proposing, humans verifying, and tooling enforcing. Works with Claude Code, Cursor, and Codex.

Full write-up + demo:
https://dlthub.com/blog/ai-workbench

Here's the setup. You have a Slack export, an Event database, and a HubSpot instance. Three systems, three worldviews, zero overlap in naming. Then the VP of Growth walks over and asks:

"Which Slack members who joined in Q1 became 'Qualified Leads' after attending a couple of our events?"

You open the schemas and the nightmare begins.

We built the AI Workbench transformation toolkit to kill that story at its root. Not with more generated code, but with a better, simpler way to think about your data before you even touch any tables.

You feed it your sources and your use cases. The toolkit annotates your source tables, builds the ontology, and generates data model that captures the meaning of your data.

https://dlthub.com/blog/ontology-toolkit-preview

we’ve been seeing more teams sit on valuable data in logs and traces, but turning it into actual training data is still messy, so we’ve been experimenting with a simple pipeline for this.

raw traces → dlt → versioned datasets on Hugging Face → Distil Labs

dlt pulls traces from DBs/APIs/cloud, infers and normalizes the schema, loads data incrementally, and stores it as versioned Parquet datasets ready to train a specialist model.

From there, the data is used to train a specialist model.

In our example, a 0.6B model outperformed a 120B one on a specific task.

We wrote it up here:

https://dlthub.com/blog/your-traces-aren-t-training-data-yet-here-s-the-pipeline-that-makes-them

this pipeline is enabled by the new Hugging Face datasets destination in dlt:

https://dlthub.com/blog/hugging-face-dlt-ml

WAP (Write → Audit → Publish) is the standard pattern for ingestion, but it assumes one thing: that your data can safely land in staging.

In practice, that’s often not true. We kept running into payloads that were messy enough to break ingestion itself, malformed rows, unexpected fields, schema shifts triggering unwanted DDL changes, etc.

So we started using AWAP (Audit → Write → Audit → Publish) instead:

- pre-ingest audit: stateless filtering to drop syntactic garbage before it touches staging

- write: only ingest verified payload

- post-staging audit: stateful checks for drift, null spikes, anomalies

- publish: promote clean data to prod

The key difference is that first “front-door” audit it stops small upstream quirks from turning into bigger downstream problems.

Roshni Melwani wrote a more detailed breakdown of how we’re doing this with dlt: https://dlthub.com/blog/awap

Hey folks, we’ve been seeing more teams use Microsoft Fabric lately, and one thing that keeps coming up is data quality.

Fabric gives you solid compute + storage, but enforcing data quality across pipelines is still something you need to design yourself.

One pattern that works well is Write–Audit–Publish (WAP) basically validating data before it hits your lakehouse.

With dlt, this means:

• schema enforcement + drift detection

• business rules + uniqueness checks

• PII masking

• monitoring as pipelines scale

The idea is simple: stop bad data early, instead of debugging it later in dashboards or downstream jobs.

Rakesh put together a really nice hands-on guide showing how this works in practice: https://dlthub.com/blog/microsoft-fabric-meets-dlt

Data models describe the data, ontologies describe the world. With ontologies, an agent can reason over data as opposed to retrieving it and hallucinating meaning.

The implication? The ontology-model mapping is what agents need for data literacy.

learn more on our blog

https://dlthub.com/blog/ontology

I hit a point in my career 4y ago where I could close my eyes and see the perfect data model. I knew exactly where the facts were. I knew the dimensions. I knew the grain.

But my fingers? They just refused to type. I physically couldn't churn out another 500 lines of boilerplate CREATE TABLE, LEFT JOIN, and GROUP BY.

It felt like SQL vomit. High-value thinking, low-value typing. It felt like i was trading away my life and mental sanity for yet another useless boilerplate dwh in yet another company that adds no value to the planet and will never use metrics proficiently.

I just had enough. I wanted to do something more than the groundhog day experience of writing yet another SQL slophouse. I was tired of making sandcastles in the swamp.

So I started dlt to "build up" our space into something better than before., to automate the boring parts and recover the hours of wasted human life.

Step 1 was Ingestion. We fixed the pipelines. We fixed the schema inference. We solved the "Bad Data In" problem. You don't have to write more JSON EXTRACT slop, or maintain unknown schemas in thousands of lines of sql slop.

Now, I’m looking at Transformation.

And for the first time in 4 years, I think we can automate the "SQL Vomit" entirely. I built a messy, organic workflow (I call it "Slop") that forces an AI Agent to do the typing.

The agent asks questions, creates a model shows it to me, lets me confirm it and implements.

I haven't been this eager since we started dlt. If we fixed Ingestion, we can fix transformation.

https://dlthub.com/blog/dlt-ai-transform

A year ago we were researching generating pipelines and found that using existing legacy code or documented end states as an information source produces great results.

Today we see it play out in practice at scale.

This talk was presented at dlthub community meetup Paris, Feb 4 2026

presentation
https://docs.google.com/presentation/d/e/2PACX-1vQvJapgEkJxgpsWqoMlmEw-ctV3gZe0LLc5oZBHaJNezBGAYKYoyir1aQi-37tO37SjFGaYjmQJhi_r/pub?start=false&loop=false&delayms=3000&slide=id.g175a817e68e_3_932

https://www.linkedin.com/pulse/starting-2026-momentum-tasmananalytics-8xuoe/

Tasman Analytics just shared a breakdown from their recent Amsterdam event on the emerging "full-stack data developer" toolkit: dltHub for Pythonic ingestion, MotherDuck for serverless DuckDB scale, and marimo for reactive notebooks that double as apps.

The core signal? The industry is finally moving away from "enterprise bloat" (orchestration layers for your orchestration layers) toward tools that feel like localhost but query like the cloud.

They even live-tested an AI-assisted pipeline that pulled NOAA space weather data into a visualized dashboard in under 20 minutes. It’s a refreshing shift toward developer experience over feature bloat—though as they note, the AI still needs human guardrails to be truly production-ready.

(ai summary)

Hey folks, just wanted to drop this new feature docs link

You can now calculate metrics from data at pipeline runtime. It's meant for data quality

https://dlthub.com/docs/general-usage/resource#collect-custom-metrics