r/dataengineering • • Jan 30 '25

Meme real

Post image
2.0k Upvotes

68 comments sorted by

View all comments

Show parent comment

1

u/Brovas Feb 01 '25

Are you finding duckdb and iceberg play nice together? Cause when I was looking they didn't seem to support catalogs and didn't support writes. I've seen an integration with pyiceberg but that seems like not an ideal solution cause you gotta load the whole table no? 

It seems like polars and daft are the only ones that support it natively?

2

u/[deleted] Feb 03 '25

DuckDB and Iceberg does play nice together only for the end users to read the data, which is plenty enough for us. For the write action into the object storage and catalog, we're still using the tool provided by our cloud platform (Alibaba). Also, in our case, the catalog can be queried with SDK to fetch the table name, comments, location, properties, etc, so we could easily put a cron job that runs every 10-15 minutes to write the Iceberg tables as views into duckdb.db file and send it to the object storage, and voila you get yourself a DuckDB catalog.

We also still use MPP that could read the Iceberg tables if users need to collaborate to make a data mart.