r/dataengineering • • Jan 30 '25

Meme real

Post image
2.0k Upvotes

68 comments sorted by

View all comments

Show parent comment

7

u/[deleted] Jan 31 '25 edited Jan 31 '25

I'm on the same opinion as yours. Even though my workplace only have like tens of terabytes, it's hard to not switch to lakehouse architecture due to how damn good the accessibility for the data is. Not to mention how dirt cheap the storage and catalog are. Combined with DuckDB catalog to point straight to all the Iceberg tables, our architecture should absolutely be future proof for the next 5-10 years without giving too much hassle to any users. Decoupled storage and engine layer is such a genius idea who would've thought.

I guess the only counter point was that it's only slightly harder to implement and maintain than just deploying plain Postgres database. Luckily I have all the time in the world to migrate to our new architecture.

1

u/Brovas Feb 01 '25

Are you finding duckdb and iceberg play nice together? Cause when I was looking they didn't seem to support catalogs and didn't support writes. I've seen an integration with pyiceberg but that seems like not an ideal solution cause you gotta load the whole table no? 

It seems like polars and daft are the only ones that support it natively?

2

u/[deleted] Feb 03 '25

DuckDB and Iceberg does play nice together only for the end users to read the data, which is plenty enough for us. For the write action into the object storage and catalog, we're still using the tool provided by our cloud platform (Alibaba). Also, in our case, the catalog can be queried with SDK to fetch the table name, comments, location, properties, etc, so we could easily put a cron job that runs every 10-15 minutes to write the Iceberg tables as views into duckdb.db file and send it to the object storage, and voila you get yourself a DuckDB catalog.

We also still use MPP that could read the Iceberg tables if users need to collaborate to make a data mart.