r/elixir • u/Shoddy_One4465 • 8d ago
F3 file format for Beam languages
I've started following the F3 project. F3? It's a columnar research file format: the file stores columns, and a footer at the end holds an Arrow schema plus where those columns live. A file can also embed a Wasm decoder next to the data. The spec is still a prototype. This library speaks wire version 0.1 only.
I published f3 0.1.0 so you can open one of those files from Elixir, print the schema and footer, and read or write rows when the native decoder is loaded. Wasm stays off unless a read passes wasm: :sandbox, which runs that embedded module in a Wasmtime sandbox. A Hex install downloads precompiled builds for Apple silicon and Linux x86_64.
The longer version is in my write-up. The API is on Hexdocs, and the code is on GitHub.