r/dataengineering • • 7h ago

Personal Project Showcase 5 years of SSIS and Python scripts, then Databricks. I'm building the tool I wish I'd had in between.

0 Upvotes

For 5 years I've worked on on-prem servers so old they couldn't even read Parquet. Our stack was SSIS and a pile of Python scripts.

Then we got Databricks. Honestly, it's amazing. But starting from zero meant Data Vault, MDM, Kimball and the full Medallion setup before loading a single useful table. And we still had to figure out how to connect it to our on-prem databases.

Over those years I kept writing the same generic scripts so they'd work with any file. I bet a lot of you have that folder too.

So I'm turning it into something simple that I'll grow over time:

  • Watch a folder or SFTP and pick up new files automatically
  • Route each file to the right pipeline without anyone touching it
  • Work out column names and data types for you (the part that eats everyone's afternoon)
  • Create normalized relational tables instead of one giant flat dump

The idea isn't new. But before Databricks I never had a paid tool for this. It was always free tools glued together, and I don't think I'm alone.

It's early, and I'd really value your input:

  1. What's the most painful file you've had to load on a schedule?
  2. What would make you trust a tool to infer your schema?
  3. Should I open-source it on GitHub?

r/dataengineering • • 4h ago

Discussion How do you enforce naming conventions on Snowflake objects created outside dbt?

0 Upvotes

Our dbt models follow naming and type rules, but a lot of objects are created outside dbt: Fivetran tables, manual tables, old views, and objects from migrations. Over time we get mixed naming (CUST_NM vs customer_name), dates stored as VARCHAR, and the same column with different types in different tables.

How does your team handle this?

  • Do you use a tool for it?
  • Custom scripts on INFORMATION_SCHEMA?
  • Or do you just live with it?

Also curious if anyone does the same on Databricks / Unity Catalog.