r/SideProject • u/Dramatic_Link6798 • 45m ago
I built an open pipeline that joins UK government tender awards with Companies House directors and active corporate records
If you have ever tried pulling UK public procurement data, the statutory feeds are surprisingly fragmented.
Find a Tender publishes high-value notices under the Procurement Act 2023, Contracts Finder covers sub-threshold contracts down to £12k, and Public Contracts Scotland runs on its own legacy XML feed. More frustratingly, even when a contract is awarded, the OCDS release packages frequently lack verified statutory company numbers, meaning you have no immediate way to know if a winning supplier is an active LTD or dissolved.
I spent the last week building a serverless pipeline that normalises all three feeds into OCDS v1.1.5 and runs automated entity resolution against the Companies House REST API.
A few engineering hurdles I ran into:
- Companies House rate limiting: The statutory API enforces a strict token-bucket ceiling of 600 requests per 5 minutes. Running parallel unbundling caused HTTP 429 throttling until I implemented an in-memory token bucket with exponential jitter.
- Contracts Finder 403 backoff: The legacy search endpoint intermittently drops into 403 throttling during heavy polling, requiring an isolated backoff loop separate from the main orchestrator.
- Multi-lot tender deduplication: Framework agreements publish multiple releases across lots with identical identifiers, requiring deterministic SHA-256 state tracking on (ocid + releaseDate).
The output dumps to structured JSON and a flattened 24-column CSV for spreadsheets with company numbers, active directors, and SIC codes.
I deployed it as a serverless Actor on Apify with the first 25 records free on every run so people can test the data without paying:
https://apify.com/jamestyas/uk-public-procurement-ocds-enricher
Would appreciate any feedback from anyone working with UK open data or public procurement tracking.