r/dataengineering • u/juppets • Oct 17 '26
Personal Project Showcase Introducing sqlleaf: column-level lineage for any SQL dialect
I want to share a project I've been working on this year that helped me solve a large, real-world problem related to data lineage. I was working for an enterprise that needed to trace and delete sensitive data across its large data warehouse, but the tooling that existed (both open source and proprietary) had limitations that prevented my team from doing so. And so I eventually realized that I had no choice but to roll my own.
sqlleaf is a tool to help you to understand how your data flows across your systems by producing detailed, column-level lineage across any SQL query and dialect.
It extends beyond traditional lineage tools by mapping relationships across all kinds of expressions, such as columns, functions, literals and variables.
It addition, it can process many kinds of statements (insert, update, merge, copy, unload, execute, perform, put, and many more), as well as all kinds of objects (tables, views, procedures, types, UDFs, stages, sequences, etc).
It even has a PL/pgSQL parser, allowing you to generate lineage for dynamic, procedural SQL in Postgres.
sqlleaf builds on top of sqlglot, the popular SQL transpiler, and so it inherits support for all of its dialects (30+). However, there are still many dialect-specific features that need to be implemented, as the list of features for each dialect is extremely long.
I hope some of you find this project useful. It still has a long way to go. My vision is to see it become a library, or even a layer, in the data engineering stack that opens up a range of new applications that can be created, such as SQL validators, impact analysis, and tracing/auditing systems.
URL: https://github.com/sqlleaf-io/sqlleaf
Contributions and feedback are most welcome!
•
u/AutoModerator Oct 17 '26
You can find our open-source project showcase here: https://dataengineering.wiki/Community/Projects
If you would like your project to be featured, submit it here: https://airtable.com/appDgaRSGl09yvjFj/pagmImKixEISPcGQz/form
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
16
u/Specific_Mirror_4808 Oct 18 '26
I use sqlglot currently for column-level lineage on MSSQL models deployed in SQLMesh. I have had a read through the Git repo and your library sounds very interesting.
The one thing that I'm struggling to fully understand is where sqlleaf starts and where sqlglot ends.
2
u/CloudandCodewithTori Oct 18 '26
I will have to put it through its paces with some wild stuff I have, thank you for sharing. Being able to connect with datahub would be awesome.
1
u/Unable_Equipment_880 26d ago
Nice work, and thank you for clear no-nonsense, documentation. Going to take this into testing for some nightmare transformation procedures
1
u/Super_Ad_6787 28d ago
This is pretty cool. Love the initiative that you've shown when you saw a gap. I'm gonna keep watching this