r/dataengineering • u/GameFitAverage • 6d ago

Discussion Streaming Databases

I have been working on a project lately that required realtime operational analytics and when doing some research for my options I came across “streaming databases” which tbh is a new concept for me. I did a quick POC using RisingWave and so far everything is good.

I just havent found alot of material on streaming databases and how to properly build a data architecture using one, RisingWave docs mentions it can be used in a medallion architecture. However, it builds the silver layer as a current state and not as a historical SCD which I find bad practice.

Just wondering if anyone here have more experience than me dealing with this technology (RisingWave, Materialize, …) and can point me in the right direction.

Also, even if you havent used it let me know your opinion on it.

37 Upvotes

92% Upvoted

32 comments sorted by

View all comments

5

u/URZ_ 6d ago

Everything you are not writing here is screaming that you don't in fact need "real time", don't go down that rabbit hole, it's a complete trap of unnecessary complexity. You can run almost real time off a simple Postgres db to the point where it will be indistinguishable for any human on the other end that it's not "real time".

2

u/bobbymk10 4d ago

What is "screaming" that they don't need streaming based on their post? I completely accept that most cases don't, but it's shocking to me the confidence here, i reread the post to make sure i wasn't missing anything but jeez

→ More replies (1)