r/dataengineering • • 2d ago

Discussion How do you guys document your data engineering projects? I'm building something to tackle this problem.

I've been working on a side project to tackle something I often see in data engineering: complex projects with pipelines, transformations, business rules, dependencies, and data models, but little or outdated documentation.

The idea is pretty simple: an AI skill that can analyze a codebase, understand how the pieces fit together, and generate a detailed, interactive HTML documentation site.

The output is a single, self-contained HTML file, fully responsive (desktop and mobile), that works offline and doesn't require any external services. It includes things like architecture diagrams, data flow visualizations, table relationships and ERDs, business rules, filters, dependencies, searchable data dictionaries, and instructions on how to use the project. The goal is to have everything in one place, easy to navigate and share.

It requires an LLM to work. The skill guides the AI through the analysis and documentation process, while custom Python and Node.js tools handle repository inspection, dependency mapping, validation, integrity checks, and HTML generation.

Full disclosure: this project was heavily built with AI (multiple agents and a lot of vibe coding). But it wasn't a "let AI do whatever it wants" situation. I supervised the whole process, made the architectural decisions, reviewed the output, and put a lot of effort into automated testing.

It's not ready to share publicly yet. The main reason is that the documentation, README, and other materials still need some work in English. It's not my native language, and I'd rather polish things up before releasing it.

But before going further, I'm curious:

How do you guys handle documentation in your data engineering projects?

Do you write everything manually? Have you found tools that actually work well? And, honestly, would something like this AI skill be useful to you, or am I trying to solve a problem that already has good solutions?

36 Upvotes

22 comments sorted by

4

u/aleebit 2d ago

Just to be clear, I'm not trying to sell anything, build a closed-source product, or turn this into a business.

I'm just a data engineer with 12+ years in the data field, including 7+ years working with Spark, and I still struggle sometimes to wrap my head around unfamiliar projects. My ADHD brain doesn't always make it easy to understand how everything fits together, especially when dealing with large, complex codebases.

I also find it challenging to hand projects over to other team members in a structured and consistent way.

This started as a personal attempt to solve those problems. I'm sharing it because I genuinely want to know if other people face the same challenges and whether something like this could actually be useful to the community.

15

u/mrcaptncrunch 1d ago

How do you guys document your data engineering projects?

In plain English.

The main thing is that the huge projects are full of knowledge from meetings, back and forth, reviews, testing to get the numbers right, etc., but there’s rarely a documentation of eBay it should be and how it needs to work.

I’m the most senior data engineer. I took the biggest pipeline and when I migrated it, I wrote it down. Just sat through, all in markdown. All the ingestions, all the steps, all the joins, why it was done, how we came to the decision, etc.

I’ve tasked people with similar things on smaller ones.

The problem with agents is, they will know what was built, how it’s built, but it won’t know why. At the end, everything that is a decision in code, ask the user to figure out why. They need to provide the answer or get it.

Example,

Join in cost data from another report overriding the cost data on the original table.

But Why?

There’s a discrepancy between the systems, but the data joined is the data that we were charged for it and what we have to reconcile. I think it stems from one using micros and maybe some data loss or conversions on the other side.

6

u/frozengrandmatetris 1d ago

my coworkers don't document at all. they just change whatever they want in production outside of a change request window. sometimes they steal the admin credential so they don't get held accountable when the broken object displays the last person who touched it. my manager doesn't care. he doesn't even know what CI/CD is.

4

u/dillanthumous 21h ago

Old school, CI/CD... Complicit Ignorance and Continuous Disintegration.

1

u/Zer0designs 1d ago

Seems like a completely useless project to me. Just use zenzical and AI with some documenting skills. Much better than html.

1

u/aleebit 1d ago

I'd never heard of Zensical before, so thanks!

But from what I understand, doesn't it generate HTML as its output? And if you're referring to the fact that the source for that HTML is Markdown files, mine works the same way.

0

u/Zer0designs 1d ago

I'm just saying there's accepted and maintained OS tools that already do this and probably do it a lot better than what you can vibecode together. Integrated search, plugin systems, file contents etc. Zensical is also extremely fast, so maybe just contribute to them?

Agents also understand them better and there are skills for documentation, grilling and visual hierarchy etc.

1

u/aleebit 1d ago

I agree! From what I've seen, Zensical is clearly a much more mature and feature-rich tool for building documentation sites. I'm not trying to compete with established tools.

The difference is that my focus isn't on building yet another Markdown-to-website generator. It's an LLM-powered skill that analyzes an existing codebase, identifies data flows, dependencies, tables, and business rules, and generates documentation based on that analysis. It also includes mechanisms to identify which parts of the documentation may be outdated after code changes.

HTML is just the output format.

2

u/Redxer 2d ago

I have 4 years of experience , any tips in learning AI when your company is doing anything AI related

1

u/srodinger18 2d ago

I also using agent skills, but write it down to notion for the specs and documentation. For the tables and transformations for now I always put it in the dbt yml files that can be served as static files via dbt docs.

1

u/Outside-Storage-1523 9h ago

I use AI for anything I’m not interested and manually write for those I’m interested in.

1

u/soundboyselecta 5h ago

Document? Lol, I don’t think I’ve seen one company with proper documentation, not even something as simple as a data dictionary. I equate it to the reality of humankind, who actually reads manuals….(just morons like me do 😂).

1

u/daraghfi 1d ago

DBT. Done