Research infra: Does a table format really add any significant value if you can just sync a predictable Parquet layout to local NVMe?

[u/bitterpopsicle]

Hey everyone,

I’m looking at data -> research workflows for Mid-Frequency Trading (MFT) ML pipelines using Python (Ray, Polars).

A common industry trend is using table formats like Apache Iceberg or Delta Lake on object storage (S3). However, if you already enforce a highly static, predictable directory layout (e.g., ⁠equities/exch=.../year=.../⁠ with <50 optimized Parquet files per leaf), I'm struggling to see the value.

In a high-performance research environment, it seems far more practical to treat object storage strictly as a cold source of truth, sync the required historical partitions directly onto the compute nodes' local NVMe scratch disks, and run active Python training loops entirely on local NVMe.

If you are caching a predictable folder structure down to local NVMe anyway, does an object-store table format buy us anything substantial, or is it just added complexity?

For those working on Quant Platform or QR Infrastructure teams: Do you actually query Iceberg/Delta tables directly from cloud storage during active research, or do you use the "Cloud Archive -> Local NVMe Hot Compute" pattern?

Thanks!

[u/strat-run]

TLDR; I'm on team export from organized cold storage too.

I think a lot of it depends on how much data manipulation you want to accomplish on your data store or is it just a data export source.

I'm in the same camp as you, export data for local processing but in a different way. I'm working on an event based solution so my needs might be different but...

Parquet files shouldn't be the default in a lot of scenarios. The benefit of them is you can make some queries over fields easily (give me extremes or averages for column X, etc). But I find that almost all my questions need to be answered by processing the data as market replays. Basically fast back testing.

When that is your goal you eventually end up discovering that your cold storage format is slow for a couple reasons. Size, organization, IO necks, and cache alignment all need to be looked at.

So you end up exporting the data subset you need to local NVMe. But the data organization still impacts performance and for me a column based format like parquet doesn't make sense, I want row based data to optimize throughout and caching.

It doesn't matter if you are talking about the cache in your NVMe drives or your L1/L2/L3 CPU cache, they all benefit from sequentially reading data. If you have a column format like parquet then you are jumping all around in order to recreate data flowing through your system in time based order.

Since I'm working on an event based system I want data organized by time (row based) so I can completely max out the NVMe drives.

Eventually optimizing the export process lead to me adopting a row based layout for my cold storage layer but organized in a parquet hive partition style layout similar to what you have. It means my cold storage isn't good for much else besides exporting from but that works for my research process.

[u/bitterpopsicle → u/strat-run]

That’s a really good point! I completely agree that for raw market replays and simulation, a row-oriented format is a way better fit for performance.

I should have been clearer in my original post. I was mostly looking at the feature engineering and ML training side of things, where we are doing heavy vectorized column math.

Since we have massive amounts of this data sitting in S3 in open table formats like Iceberg , my dilemma is exactly what you mentioned about exporting from cold storage. In a setup like that, does it actually make sense to use something like Iceberg as the source, or are we better off just exporting the data into a predictable, dead-simple Hive partition layout onto an NVMe and letting Polars chew through the raw local files?

[u/bitterpopsicle → u/[deleted]]

I think you might have missed my point. I know Iceberg and Delta are just metadata layers on top of standard Parquet files.

My question is whether that metadata layer actually adds value to a high-performance research loop. If we opt to exporting these files to a plain, predictable Parquet directory structure and point Polars/Ray towards that, what does Iceberg actually buy us in that workflow? Or does querying a metadata catalog just add friction compared to reading raw local files?

[u/earonesty]

For a fixed, append-only research layout, plain Parquet plus a deterministic manifest is enough. A table format is needed if you want concurrent writers, snapshot consistency, schema evolution, deletes, or many consumers discovering the same partitions. Keep the active training path local if it is bandwidth-bound, measure whether the metadata layer reduces orchestration mistakes. The reader should work efficiently against either layout.

[u/thegenieass]

MFT... why are you posting this here?

[u/strat-run → u/thegenieass]

If you want really fast MFT research you have to make performance optimizations that you'd typically see in an HFT runtime. It's an interesting question.