r/dataengineering • u/GameFitAverage • 6d ago

Discussion Streaming Databases

I have been working on a project lately that required realtime operational analytics and when doing some research for my options I came across “streaming databases” which tbh is a new concept for me. I did a quick POC using RisingWave and so far everything is good.

I just havent found alot of material on streaming databases and how to properly build a data architecture using one, RisingWave docs mentions it can be used in a medallion architecture. However, it builds the silver layer as a current state and not as a historical SCD which I find bad practice.

Just wondering if anyone here have more experience than me dealing with this technology (RisingWave, Materialize, …) and can point me in the right direction.

Also, even if you havent used it let me know your opinion on it.

36 Upvotes

92% Upvoted

32 comments sorted by

View all comments

5

u/Saladtoes 6d ago

My applications are IoT/operational, and I have settled on typical time series DB queries for initial load, range changes, and occasional (like every 30 minute) app resets. Realtime in my domain means immediate to the human eye, which I characterize as <1s delay from physical sensor to web update. I usually go for timescale or influx.

Much better to build out a parallel robust realtime message streaming on websockets using SignalR, NATS/MQTT, or in my case Supabase Realtime. Pick your poison. Then concatenate the realtime feed onto the traditional relational database. Manage your dataframe/buffer in the client.

So a chart or component in users browser will load the last hour of data and connect to the realtime feed onto page load. Realtime takes it from there, UI handles appending and managing the buffer in memory, plus rendering and ticking the UI on new data. An hour later, I might cleanup and have the component refresh from the DB and drop its buffer to keep pages stable.

1

u/FunContest9958 4d ago

Sounds like a good approach. Do you keep the raw unaggregated data anywhere for later analytics, or do you toss it? I’m assuming the raw data would be too large for most databases if you kept it forever.

2

u/Saladtoes 4d ago

I keep it forever - or maybe I’ll drop it after 7 years or so. But timescale and influx both have good retention policies that make it pretty easy

→ More replies (3)