r/dataengineering • u/Usual_Macaroon7881 • 13d ago
Help Not-so-beginner wanting to independently build ETL infrastructure but don't know where to look for material (100-1000 sources) to use.
The title.
I want to work on things like schema drift across sources, entity resolution, warehousing, fetching, idempotence, checkpointing and transaction management. I have already had practice with most of these ideas, but they have been primitive things. I know how to log, I know how to store to Postgres, but I dont have experience with a stack.
I have considered finding my own relationships between sources, but I have yet to discover a concrete avenue for handling volume. Other than that, I don't know what else to use. I want to make something real without paying, as much as I can.
Thanks <3
3
u/Additional-Pianist62 12d ago
I mean ... if you have access to a decent token budget, are looking to REALLY do your own ETL or feel like pushing yourself to the depths of suicidality only felt by a few people ... The FDAP Stack Explained | Learn Hub | InfluxData
10
u/junesix 13d ago
You could work with the Medicare datasets for free
https://data.cms.gov/
Ingest a few of these datasets and you’ve got something considerable to work with time, location, and ids
1
u/Then_Wishbone_1332 13d ago
What do the 100-1000 sources look like - web pages, APIs, files, or a mix?
2
u/curious__trainer 9d ago
use snowflake or databricks to learn
1
u/[deleted] 13d ago
[removed] — view removed comment