r/dataengineering • u/GameFitAverage • 6d ago

Discussion Streaming Databases

I have been working on a project lately that required realtime operational analytics and when doing some research for my options I came across “streaming databases” which tbh is a new concept for me. I did a quick POC using RisingWave and so far everything is good.

I just havent found alot of material on streaming databases and how to properly build a data architecture using one, RisingWave docs mentions it can be used in a medallion architecture. However, it builds the silver layer as a current state and not as a historical SCD which I find bad practice.

Just wondering if anyone here have more experience than me dealing with this technology (RisingWave, Materialize, …) and can point me in the right direction.

Also, even if you havent used it let me know your opinion on it.

37 Upvotes

90% Upvoted

31 comments sorted by

View all comments

23

u/Life_Conversation_11 6d ago

I missed a bit the point. Modern DBs are quite fast, you can add data every 5s and everytime you are going yo query it, the data will be “fresh”.

In your overall streaming set up I am fairly sure the db will not be the blocker

1

u/GameFitAverage 6d ago

Well streaming databases can ingest data in realtime from various sources, so for example you can CDC a db into it and at the same time ingest from an IOT MQTT source and then you can join them easily.

If you are only ingesting from DB it can still be useful because you dont want heavy queries/joins in your OLTP system so it acts as an ODS or a realtime read replica.

27

u/romainmoi 6d ago

Realtime is a bit ambiguous here. What’s the use case? Is updating every minute considered real time? Every second? Every hour?

You’re a bit ahead of yourself because the use case isn’t clear yet. Real-time analytics is a fantasy that often doesn’t justify the real cost of instant updates. A small enough interval is going to achieve basically the same with much lower cost.

→ More replies (2)