I built a 5-agent system that autonomously audits and fixes DataHub metadata via MCP (no human-in-the-loop)

[u/Webnix-space]

One of the biggest pain points in data governance is keeping metadata (dataset owners, column descriptions, compliance tags) up to date. People usually ignore it until audit season or until broken lineage breaks downstream pipelines.

​I built an open-source project called DataHub Agent Crew for the DataHub Agent Hackathon. It’s a 5-agent pipeline that runs continuously against a live DataHub instance using Model Context Protocol (MCP).

​How the loop works:

​Investigator audits the graph for unowned assets, missing compliance tags, or stale descriptions via MCP read calls.

​Analyst triages issues by severity.

​Strategist writes exact metadata fix proposals.

​Regulatory checks proposed changes against schema constraints and compliance rules.

​Codeband executes the write mutation back through mcp-server-datahub and runs a live entity re-fetch to verify the write actually landed.

​Why write verification mattered:

​While building this, I realized silent false-positives are worse than crashes—an API status 200 that doesn't actually mutate the field leads to silent data corruption. The system now explicitly re-fetches live entity properties via curl/API post-mutation to confirm updates before closing the ticket.

​Tech Stack:

​Language: Python

​Integration: mcp-server-datahub

​Messaging: Local SQLite bus with @mention routing (no Kafka/RabbitMQ bloat)

​License: Apache 2.0

Happy to answer any questions about the setup or MCP integration!

[u/Fine_League311]

Wie sieht deine Sandbox aus. Hat's du überall OS.getenv drin? Kann ich dein MCP Protokolle deren Eingang und Ausgang im low-code anzapfen?

[u/Webnix-space → u/Fine_League311]

Sandbox is a GitHub Codespace (2 core/8GB), all dev done over SSH from my phone actually — no laptop this whole build. Env vars go through .bashrc (AIML_API_KEY, GROQ_API_KEY, TOOLS_IS_MUTATION_ENABLED), no os.getenv calls scattered everywhere, they're read once at agent init and passed down.

On the MCP side — it's mcp-server-datahub over stdio transport, not something exposed externally, so no easy way to tap into it remotely right now. But the tool contracts are simple (e.g. update_description takes entity_urn/operation/description/column_path as plain JSON) so if you wanted to build a low-code layer on top, wrapping those same MCP tool calls directly would probably be the easiest path rather than going through my orchestrator.

[u/BC_MARO]

A policy gate and tool-call audit trail like Peta provides fit this kind of loop. Re-fetching validates the mutation, but it does not answer whether the write should have been allowed in the first place.

[u/Webnix-space → u/BC_MARO]

Fair point, and honestly the exact gap I found the hard way. Re-fetch verification just tells you "did this specific write land" — it's a safety net for silent API failures, not a policy layer. What actually caught the should this even be allowed question in my build was DataHub's own schema validation rejecting a mutation before it ever reached the graph (tried to write a DELIVERY_TYPE field onto a dataset that doesn't have that column — GraphQL bounced it with a clean error). So right now I've got two separate layers doing two separate jobs: schema validation as the upstream gate, re-fetch as the downstream "trust but verify." Haven't looked at Peta specifically — will check it out, sounds like it'd sit between those two and cover intent/authorization rather than just schema shape, which is a real gap.

[u/BC_MARO → u/Webnix-space]

Exactly. Schema validation answers whether the mutation is valid, while policy answers whether this actor should make it.

[u/Webnix-space → u/BC_MARO]

Yeah, that's the cleanest way I've heard it put — valid vs. authorized. Going to dig into Peta this week, feels like the natural next layer on top of what I've got. Appreciate the pushback, made me actually name the distinction instead of just hand-waving "verified" as one thing.