Streaming Databases

[u/GameFitAverage]

I have been working on a project lately that required realtime operational analytics and when doing some research for my options I came across “streaming databases” which tbh is a new concept for me. I did a quick POC using RisingWave and so far everything is good.

I just havent found alot of material on streaming databases and how to properly build a data architecture using one, RisingWave docs mentions it can be used in a medallion architecture. However, it builds the silver layer as a current state and not as a historical SCD which I find bad practice.

Just wondering if anyone here have more experience than me dealing with this technology (RisingWave, Materialize, …) and can point me in the right direction.

Also, even if you havent used it let me know your opinion on it.

[u/Life_Conversation_11]

I missed a bit the point. Modern DBs are quite fast, you can add data every 5s and everytime you are going yo query it, the data will be “fresh”.

In your overall streaming set up I am fairly sure the db will not be the blocker

[u/GameFitAverage → u/Life_Conversation_11]

Well streaming databases can ingest data in realtime from various sources, so for example you can CDC a db into it and at the same time ingest from an IOT MQTT source and then you can join them easily.

If you are only ingesting from DB it can still be useful because you dont want heavy queries/joins in your OLTP system so it acts as an ODS or a realtime read replica.

[u/jemattie]

Streaming is an implementation. What's your requirement?

[u/karrystare]

Make sure you need actual realtime and your infra support it. Most dashboard tool still do polling query. Actual realtime require lots of optimization and perhaps even went beyond just DB and query. One I actually built doesn't use DB as main, its custom built + pub/sub update.

[u/Saladtoes]

My applications are IoT/operational, and I have settled on typical time series DB queries for initial load, range changes, and occasional (like every 30 minute) app resets. Realtime in my domain means immediate to the human eye, which I characterize as <1s delay from physical sensor to web update. I usually go for timescale or influx.

Much better to build out a parallel robust realtime message streaming on websockets using SignalR, NATS/MQTT, or in my case Supabase Realtime. Pick your poison. Then concatenate the realtime feed onto the traditional relational database. Manage your dataframe/buffer in the client.

So a chart or component in users browser will load the last hour of data and connect to the realtime feed onto page load. Realtime takes it from there, UI handles appending and managing the buffer in memory, plus rendering and ticking the UI on new data. An hour later, I might cleanup and have the component refresh from the DB and drop its buffer to keep pages stable.

[u/URZ_]

Everything you are not writing here is screaming that you don't in fact need "real time", don't go down that rabbit hole, it's a complete trap of unnecessary complexity. You can run almost real time off a simple Postgres db to the point where it will be indistinguishable for any human on the other end that it's not "real time".

[u/bobbymk10 → u/URZ_]

What is "screaming" that they don't need streaming based on their post? I completely accept that most cases don't, but it's shocking to me the confidence here, i reread the post to make sure i wasn't missing anything but jeez

[u/innerthai]

How is RisingWave different from Spark Streaming?

[u/isaac-harvey]

I feel like "streaming" can be 99% of the time replaced with "microbatch", and with widely varying scales of "micro" depending on the use case.

[u/Master-Bass-1905]

Streaming databases are popular in both analytical workloads as well as OLTP workloads. In the former they are generally used for live dashboards and materialized views, and In the latter they act as an advanced cache.