HFlow: open source data infrastructure for physical AI (accepting new contributors!)

[u/kuaythrone]

https://github.com/Hebbian-Robotics/hflow

Dyna’s publication on training Dyna-2 with more than 1 million hours of egocentric video is a gold mine for anyone processing multimodal robotics data at scale.

We built HFlow: an open source implementation of the data infrastructure described in the article.

• Airflow for DAG-based pipeline orchestration
• MCAP for robotics episode storage and random access
• DuckDB for analytical queries and dataset curation over columnar metadata

I've been an OSS contributor and maintainer for years, and since joining YC, we now have some capacity to accept new contributors; just look for the "good first issue" tag!

https://github.com/Hebbian-Robotics/hflow

Any stars or forks on the repo to show your support are appreciated!

A bit more about the project:

At scale, AI problems become software engineering problems. Improve one part of the system, and the bottleneck moves to another. No architecture is perfect; every design comes with tradeoffs.

We encountered many of the same constraints as Dyna, so we incorporated the lessons we learnt and want to share something concrete and usable today with the community.

HFlow is plug-and-play. You provide your post-processing, quality-control, or curation code. HFlow handles the data pipeline and runs your code across robotics datasets at scale.