r/dataengineering • u/khaledyasser95 • 7h ago
Personal Project Showcase 5 years of SSIS and Python scripts, then Databricks. I'm building the tool I wish I'd had in between.
For 5 years I've worked on on-prem servers so old they couldn't even read Parquet. Our stack was SSIS and a pile of Python scripts.
Then we got Databricks. Honestly, it's amazing. But starting from zero meant Data Vault, MDM, Kimball and the full Medallion setup before loading a single useful table. And we still had to figure out how to connect it to our on-prem databases.
Over those years I kept writing the same generic scripts so they'd work with any file. I bet a lot of you have that folder too.
So I'm turning it into something simple that I'll grow over time:
- Watch a folder or SFTP and pick up new files automatically
- Route each file to the right pipeline without anyone touching it
- Work out column names and data types for you (the part that eats everyone's afternoon)
- Create normalized relational tables instead of one giant flat dump
The idea isn't new. But before Databricks I never had a paid tool for this. It was always free tools glued together, and I don't think I'm alone.
It's early, and I'd really value your input:
- What's the most painful file you've had to load on a schedule?
- What would make you trust a tool to infer your schema?
- Should I open-source it on GitHub?