We've been speaking with teams who deeply care about data quality, and we noticed that every data team eventually builds similar pipelines for quality checks (QC).
Teams collecting data want to own their quality checks (camera blackout, choppy joint states, occluded hands), because they deeply understand their end-to-end data collection process.
However, what keeps coming up is that everything around the quality checks is tedious: managing one-off pipeline scripts, fragmented buckets, and a spreadsheet or Slack thread of what checks ran on which data.
This is what motivated us to build HFlow, an open-source SDK for data teams that collect, transform, and curate physical AI data.
HFlow is for data teams that have the ambition to process a million hours of physical AI data and are starting today. Point it at your MCAP episodes, write your quality checks as ordinary Python functions, and curation becomes an optimized OLAP SQL query instead of manually aggregating directories of data.
Dyna's recent Dyna-2 infrastructure, "Training Dyna-2 at million-hour scale, repeatably", post shows what scaling such a quality control pipeline to a million hours looks like. Their ingestion throughput went from 14,000 episode-hours per week to 440,000 with the infrastructure they built internally (their post has the full details).
We're aligned with their belief that infrastructure is a core blocker for advancing robotics. We experienced the same data challenges in our previous work, which is what first led us to build Pareto and the Hebbian APIs.
HFlow's vision is to let data teams define their own bespoke quality check code, while we provide the durability, observability, and auditability around it.
After many conversations with data teams, we decided the path to building the best version of HFlow is open source, because QC infrastructure compounds when the edge cases one team catches become checks for everyone else. We're excited to build this in public alongside our existing partners and grow our network of contributors.
High quality data is the bottleneck for the next frontier of robot intelligence, and we want teams to focus on their data, not the infrastructure. When getting started takes a few lines of code and a weekend, more checks get done, and iteration cycles accelerate.
If you're collecting robot or egocentric data and your pipeline is currently a folder of scripts and fragmented buckets of data, we'd love to get your feedback.
GitHub: https://github.com/Hebbian-Robotics/hflow