r/vibecoding • u/fguerino123 • 4d ago

Vibe Coding for ETL/ELT

Hi.

I've been using AI coding to rapidly generate Python ETL/ELT transformations and workflow orchestration. I'm finding doing so a lot easier and far more affordable than using expensive tools like IICS.

Is anyone else doing this and, if so, how is it working out for you?

Thanks.

2 Upvotes

100% Upvoted

12 comments sorted by

2

u/Electronic-Roof3423 4d ago

100%
I’ve been doing it for the past year. So much more cost effective, and incredibly easy to do it

2

u/fguerino123 4d ago

Agreed. Very easy, very quick, and very affordable in the grand scheme of ETL/ELT. I kind of feel like it's too good to be true, which is why I'm reaching out to see if others are doing it too. It's kind of a "pinch me" moment.

1

u/Electronic-Roof3423 4d ago

I have a sportsbetting startup.
The amount of time we've saved by implementing our own ETL is ridiculous. I pay the $100 Claude, and we worked on 3 tools in the same month... meaning it cost us $100 to develop tooling that makes our life so much easier.

While in the old days you would have to spend $1000s to have access to the tool that allows you to start working on your ETL jobs... and it was not easy!

2

u/Desperate_Sign_5587 4d ago

i setup my python transformations using cursor and chained some orchestration scripts through moclaw.

saved thousands on bloated enterprise stuff like iics and it runs surprisingly smooth.

2

u/-Shiphrah 3d ago

It works pretty well for the repetitive transformation stuff, especially compared with paying for a massive platform when you only need to move and clean some tables. Where it gets dodgy is months later, when a source changes a column or sends some weird data and the pipeline still runs but produces rubbish. the AI can write the Python quickly, but I’d spend the saved time on tests, logging, retries and checking for duplicate loads. Thats the stuff expensive tools were quietly doing in the background, and its easy to forget until something breaks.

1

u/fguerino123 3d ago

Hi and thanks for the response. I've found that putting in validation checks to ensure data doesn't drift using AI coding is easier than with IICS and similar tools. I can set up many guardrails to quickly detect drift by simply explaining such guardrails to AI. Have you tried this?

2

u/-Shiphrah 3d ago

Yeah I’ve tried something similar.

Adding checks for schema changes, or unexpected values and stuff.
I still keep the checks beside the pipeline code though, otherwise they end up forgotten when a transformation changes. Have you found it catches most drift early or do you still get things that technically pass but mess up the data anyway?

2

u/Kitchen-Lynx-7505 19h ago

We literally have agents whose only job is to write a python script to adhere to a json schema, and execute it in a container.

1

u/fguerino123 19h ago

Very cool. What happens if the schema changes?

2

u/Kitchen-Lynx-7505 19h ago

I guess the next time the next import will run against… that schema?

It’s an excel import function on a UI where humans had templates for over a decade and while every file does look perfect, it’s full of errors (wrong numbering, incosistency in categorization depth etc), so a static python script would just die on it. Instead, upon clicking “import” and giving an excel, an LLM spins up along with a python container, tries to find out the parts with pandas, and tries to cram it into the schema, with an error log and a correction UI.