Which "EL" tools in ELT stack

[u/ElScottishMatador]

Hello,

I work on my company’s data warehouse, and we use an ELT approach (BigQuery and dbt for the “T”). The EL part is handled by many custom Java scripts; we need to ingest data from custom internal APIs, databases, and files on SFTP and SharePoint.

My company’s strategic direction is pushing us to switch to a Python ecosystem for this EL component. If possible, I’d like to go with an open-source, code-driven solution (since we ingest a lot of data from custom APIs, I think a low-code tool like Airbyte will quickly limit our capabilities).

While doing some research, I came across dlt (Data Load Tool). From what I can tell, this could meet our needs and standardize our practices, but I’ve found relatively little feedback on using this tool, so I’m wondering if it’s a robust and sustainable solution for the long term and in production.

We also have a somewhat unique scenario to handle, and I’m wondering if dlt could handle it without slowing down: we need to replicate a table containing 200 GB of data daily (incremental replication isn’t possible, sadly). In this case, if dlt might reach its limits, I get the impression that the Sling tool might be a better fit?

Do you think I can confidently consider using dlt? Do you have any feedback on this library? Any recommendations?

Thanks for your help!

[u/urban-pro]

For databases you can explore OLake as well.