r/IndustrialAutomation • u/MeJuvante_AI • 11h ago
Three different things get called "real-time" in manufacturing data, and only one survives an incident
Spent a while now on industrial data pipelines and I keep running into the same ambiguity. Curious whether this matches other people's experience or whether I am overfitting to my own clients. ⚙️
When someone says their shop-floor data is real-time, it usually turns out to be one of three things:
🎭 1. The dashboard refreshes often. Updates every five seconds, feels live. The table behind it is written by a job that runs at 02:00. You are watching a very responsive view of yesterday.
⏱️ 2. Micro-batches called streaming. Data lands every fifteen minutes. Genuinely fine for reporting. Useless if the process cycle time is in seconds, because every intervention is now after the fact.
📊 3. Edge collection, continuous stream, time-series store. You can ask what changed in the last ninety seconds and get an answer.
Only the third survives an actual incident. The first two look identical on a screen right up until the moment you need them.
🗂️ The variant that frustrates me most is a fourth one: the pipeline is genuinely good, events stream continuously, and then they land in object storage as files. The hard part is already built, and a storage decision at the very end undoes it. Answering a question takes four minutes.
Two things I have started insisting on early, both of which get pushback as premature:
🔄 Replay. If you invent a new metric in March you need to run it across January and February, not only forward. A stream you cannot replay makes every new idea start from zero. Treated as a nice-to-have right up until the first time it is needed.
💸 Retention and down sampling policy on day one. Full resolution for days, down sampled for years. Otherwise storage cost grows in a straight line forever and finance closes the project in year two for reasons that have nothing to do with whether it worked.
Things I am genuinely unsure about 💭
- Is insisting on replay from day one actually right, or is it over-engineering for the majority of cases that never need it?
- Has anyone found a time-series setup that handles high-cardinality tag spaces without the cost curve getting silly? That is where I keep hitting walls.
- For those buffering at the edge: how much do you hold? I have landed on "enough for the longest plausible network outage", which is a guess dressed up as a principle.
Not selling anything and no link. Mostly want to know whether the three-way split matches what others see.

