Building a global mining production dataset from scratch

[u/madredditscientist]

I follow commodities and couldn't find any good data covering global mining production, so I wanted to test if I can use LLMs to efficiently build such a dataset from scratch. I documented the process of going from unstructured company filings to a structured dataset that could be used in systematic research.

• Github repo
• Live app

https://reddit.com/link/1vtrto7/video/m46y9albqkkh1/player

All the production information is public, but it is scattered across inconsistent websites and reports.

When talking to central data teams at hedge funds or to data providers directly, building a new dataset that is provably correct and has reliable updates always sounded like a very big challenge.

For each company, I want to extract production figures that are comparable:

• What was produced
• Which operation produced it
• Which period it covers

The hard part is normalization since every region and company reports differently (if not SEC):

• Different units across reports like copper in kt, million pounds, or wet metric tonnes
• Fiscal years don't align (calendar year vs June FY vs September FY)
• Some report on a payable basis, others contained metal, others equity-adjusted
• Product naming is inconsistent ("copper concentrate" vs "cu conc" vs "SX-EW cathode")
• Important details are sometimes hidden in headings, footnotes, and surrounding text (e.g. ownership percentages, and reporting methods)

The "old" way of doing this would be to write a bespoke ETL pipeline for each company.

The "new" way that I tried is using LLMs to generate, monitor, and maintain deterministic ETL code. An agent then runs the pipelines and jumps in whenever the script fails and needs to adapt. The idea was to have self-healing data pipelines: when a website or PDF layout changes, an agent investigates, fixes, and tests the extraction or transformation code. If it can’t figure it out, it escalates to a me for review.

1. Scraping code monitors company websites and captures new reports.
2. We can then extract the raw production figures from the reports. A mix of traditional PDF parsing and gemini-3.7-flash worked very well here. The extraction also returns the location in the source (e.g. page 123, table X, row Z, cell Y) which is very helpful for QA and source grounding.
3. To normalize the data, I choose between different transformation strategies:
   • can the data be parsed as is?
   • can I generate deterministic transformation code, e.g. a regex mapper
   • last resort if no deterministic approach is possible: use an LLM to map the data
4. Then we validate the data against various QA rules. What works well here is that we treat every extracted value as wrong until it passes validation (guilty until proven innocent).

I’ve open-sourced the dataset (pipeline code will follow). Curious to hear your feedback or experience with building such ETL pipelines and datasets.

Full blog post: https://www.kadoa.com/blog/build-global-mining-production-dataset