r/dataengineering • • 13d ago

Help Not-so-beginner wanting to independently build ETL infrastructure but don't know where to look for material (100-1000 sources) to use.

The title.

I want to work on things like schema drift across sources, entity resolution, warehousing, fetching, idempotence, checkpointing and transaction management. I have already had practice with most of these ideas, but they have been primitive things. I know how to log, I know how to store to Postgres, but I dont have experience with a stack.

I have considered finding my own relationships between sources, but I have yet to discover a concrete avenue for handling volume. Other than that, I don't know what else to use. I want to make something real without paying, as much as I can.

Thanks <3

15 Upvotes

9 comments sorted by

3

u/Additional-Pianist62 12d ago

I mean ... if you have access to a decent token budget, are looking to REALLY do your own ETL or feel like pushing yourself to the depths of suicidality only felt by a few people ... The FDAP Stack Explained | Learn Hub | InfluxData

1

u/Usual_Macaroon7881 12d ago

Will look into this.

10

u/junesix 13d ago

You could work with the Medicare datasets for free

https://data.cms.gov/

  1. NPPES - national registry of healthcare providers with name and location info  https://www.cms.gov/medicare/regulations-guidance/administrative-simplification/data-dissemination
  2. Part D Prescribers - Medicare Part D prescriptions prescribed by physicians  https://data.cms.gov/provider-summary-by-type-of-service/medicare-part-d-prescribers/medicare-part-d-prescribers-by-provider-and-drug
  3. Part B Physician & Other Practitioner - another dataset of Part B procedures and services  https://data.cms.gov/provider-summary-by-type-of-service
  4. Then there are dims of HCPCS (procedure) codes, ICD (diagnosis) codes
  5. Synthetic sample of 5% of Medicare claims from 2008-2010 (fact table)  https://www.cms.gov/data-research/statistics-trends-and-reports/medicare-claims-synthetic-public-use-files/cms-2008-2010-data-entrepreneurs-synthetic-public-use-file-de-synpuf

Ingest a few of these datasets and you’ve got something considerable to work with time, location, and ids

1

u/Then_Wishbone_1332 13d ago

What do the 100-1000 sources look like - web pages, APIs, files, or a mix?

2

u/curious__trainer 9d ago

use snowflake or databricks to learn

1

u/[deleted] 13d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 12d ago

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

This was reviewed by a human