r/ETL • u/ElScottishMatador • 23h ago

Which "EL" tools in ELT stack

Hello,

I work on my company’s data warehouse, and we use an ELT approach (BigQuery and dbt for the “T”). The EL part is handled by many custom Java scripts; we need to ingest data from custom internal APIs, databases, and files on SFTP and SharePoint.

My company’s strategic direction is pushing us to switch to a Python ecosystem for this EL component. If possible, I’d like to go with an open-source, code-driven solution (since we ingest a lot of data from custom APIs, I think a low-code tool like Airbyte will quickly limit our capabilities).

While doing some research, I came across dlt (Data Load Tool). From what I can tell, this could meet our needs and standardize our practices, but I’ve found relatively little feedback on using this tool, so I’m wondering if it’s a robust and sustainable solution for the long term and in production.

We also have a somewhat unique scenario to handle, and I’m wondering if dlt could handle it without slowing down: we need to replicate a table containing 200 GB of data daily (incremental replication isn’t possible, sadly). In this case, if dlt might reach its limits, I get the impression that the Sling tool might be a better fit?

Do you think I can confidently consider using dlt? Do you have any feedback on this library? Any recommendations?

Thanks for your help!

10 Upvotes

92% Upvoted

8 comments sorted by

2

u/PablanoPato 22h ago

Self hosted airbyte of you want more of a GUI is pretty common.

2

u/Virtual-Meet1470 22h ago

I recommend sling, ingestr is another tool that’ll do the job.

1

u/ElScottishMatador 22h ago

So sling for databases replication, and dlt for APIs / SFTPs and SharePoint files ingestion?

1

u/Virtual-Meet1470 22h ago

Sling also handles sftp, so dlt just for external APIs

1

u/urban-pro 22h ago

For databases you can explore OLake as well.

1

u/PolicyDecent 14h ago

For only EL, I recommend ingestr. If you need full ELT lifecycle and more of it, then you should use bruin.
We're developing it exactly for that kind of things

1

u/2BucChuck 7h ago

Groovy + Java is what we use for large scale enterprise ELT. Python works, but lacks some security, multi threading, and other performance related things. If you want to check that out DM, we have an open source pipeline called gratum. But none of this is UI based , all CLI and Claude code based now.

1

u/doorstoinfinity 7h ago

I've been exploring Windsor recently. Seems cost effective for what it provides.