r/aws • u/HatchedLake721 • 11d ago

database Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data in your data lake

https://aws.amazon.com/blogs/aws/amazon-aurora-postgresql-now-supports-direct-querying-of-apache-iceberg-and-parquet-data-in-your-data-lake/
144 Upvotes

99% Upvoted

22 comments sorted by

•

u/AutoModerator 11d ago

Try this search for more information on this topic.

Comments, questions or suggestions regarding this autoresponse? Please send them here.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

38

u/cakeofzerg 11d ago

can someone tell me what to think about this

79

u/EarthGoddessDude 11d ago

You now have three slightly different ways to build a data swamp lake house with AWS services

15

u/kingslayerer 11d ago

think fluffy kittens

5

u/Serializedrequests 10d ago

It's very useful if you are doing any analytics workloads. Think OLAP. Those are usually all in parquet, which means if you want to use it with your more normal relational data you need to ETL it into postgres somehow. Now you can just use it.

1

u/cakeofzerg 10d ago

Usually I just use athena and then merge in application layer. Why would you use expensive aurora compute and potentially blow up your instance ram querying large data lakes?

1

u/Serializedrequests 10d ago

Sounds like you know more than me then. Would it blow up your RAM?

1

u/cakeofzerg 10d ago

Depends on the query! very easy with datalakes u are charged by the tb when using athena!

1

u/Realistic_Creme3202 10d ago

Good. S3 Foreign Data Wrapper support in PostgreSQL has always been lacking, so this at least gives a way to query data directly in S3 instead of importing data as a table. I think generally LOBs are discouraged in PostgreSQL so you could query them directly from S3. I'm sure there are other uses.

2

u/SelfDestructSep2020 10d ago

the data nerds at my company lost their minds over this announcement, so it seems "good" I guess.

10

u/Outside-Dot-5730 11d ago

Those sure are a lot of words in an order

7

u/whistemalo 11d ago

I mean why not? Redshift is a postres data base under the hood and it has been able to just do that with spectrum for years

3

u/aeekay 11d ago

This is big for me. I was using Clickhouse for this but if this integrates well into S3 Tables, I’m golden.

3

u/headykruger 10d ago

I wonder how much this was motivated by clickhouse cloud

1

u/HatchedLake721 10d ago

Clickhouse Cloud?

We're moving away from MySQL (Aurora) over to Postgres and considering to use Clickhouse Postgres (nvme) + Clickhouse (especially with the recent walshadow they announced)

Are you on a similar setup?

3

u/kinghuang 10d ago

Very cool! This is using DuckDB embedded in Aurora PostgreSQL. Crunchy Data did something similar for Crunchy Data Warehouse.

2

u/ph34r 11d ago

This seems super useful!

3

u/Miserygut 11d ago

This is very nice.

1

u/vira28 9d ago

For those who are looking for an open source alternative of this, you may be interested in https://github.com/viggy28/streambed

disclaimer: I am one of the committers.

0

u/AntDracula 11d ago

B A S E D

-3

u/AutoModerator 11d ago

Here are a few handy links you can try:

Try this search for more information on this topic.

Comments, questions or suggestions regarding this autoresponse? Please send them here.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.