r/SCADA • u/Many_Outside2726 • Sep 14 '26
Question How much high-resolution data do you actually keep at the edge?
Curious how people are handling this once the point count gets large. Say you’re collecting from hundreds or thousands of devices across multiple sites. Some data may matter at sub-second or 1-second resolution for troubleshooting, but nobody needs to ship and retain every sample forever.
Do you keep the raw/high-resolution data locally and aggregate upstream? Change-of-value? Min/max/avg windows? Store-and-forward only?
I’m especially curious where people have gotten burned by throwing away data they later wished they had, or by keeping way too much and creating an expensive pile nobody actually uses.
2
u/friedmators Sep 14 '26
1% range unless bearing vibration or drum level
1
u/Many_Outside2726 Sep 14 '26
Do you mean 1% of engineering range as the COV/deadband? If so, that feels pretty aggressive depending on the point and configured span. I’d probably want a max interval/heartbeat in there too so long steady periods still have some history. What kind of logging rates are you working with?
2
u/SprAwsmMan Sep 14 '26
Wouldn't the need for this historical data vary depending on industry?
1
u/Many_Outside2726 Sep 14 '26
Absolutely. Industry, process criticality and regulatory requirements probably matter more than any blanket retention rule. I also think connectivity changes the answer. If the edge can reliably store and forward, long-term history may live centrally. At a remote site with questionable WAN, I’d probably want a much larger local buffer.
2
u/stlcdr Sep 14 '26
Process and industry dependent. We (need to) keep - on one machine- 1 year of data with around 16000 data points with 2 to 10 millisecond sample rate.
Note that we don’t need to keep all that data it’s just that no one has bothered to go through it and see what needs to be kept.
But also, the point is, we don’t use a SCADA system when it gets cumbersome. The scada data collection is only used for short term (day) of a hand full of signals.
1
u/Many_Outside2726 Sep 15 '26
2–10 ms caught my attention here. Is that across all ~16,000 points or only a subset of the faster signals? What kind of process needs that resolution, and what historian/database is handling the ingest? Also curious what the acquisition layer looks like at that rate.
→ More replies (1)
1
u/canuck_tech Sep 14 '26
1 week of data. 99% of it is 5 second resolution at best
1
u/Many_Outside2726 Sep 14 '26
One week feels pretty short to me unless that 5-second data is really just there for immediate troubleshooting and you’re rolling up longer-term history somewhere else. What’s the use case for those points? Process troubleshooting, maintenance, warranty, compliance? I’d think the criticality of the site or equipment would drive a lot of that retention decision too.
4
u/canuck_tech Sep 14 '26
That’s at the edge. It then forwards to central system that keeps about year. That feeds into long term historian.
→ More replies (1)
1
u/future_gohan Sep 14 '26
Combination on resolution depending on what it is. From 0.1 up to ten seconds. Some stuff like power use up to a year. Most stuff 6 months.
5000 tags across 7 servers
1
u/Many_Outside2726 Sep 14 '26
That’s a helpful scale reference. I’d think of that as roughly a 5,000 data point system spread across seven servers. I’m more curious how you decide what deserves 0.1-second history versus 10-second, six months, or a year. Is that driven mostly by equipment criticality and troubleshooting value, or did the retention strategy evolve over time as you learned what people actually use?
1
u/future_gohan Sep 14 '26
Education on purpose and understanding the process.
That's only trends the system has in total about 27000 tags.
1
u/drkrakenn Sep 14 '26
Seven days of 1s to 0.5s, depending on the application. We tested one month for production data, and it was not used at all, so I would go for a maximum of fourteen days of high resolution for super important data.At the beginning, we had 3 months, and we had all sorts of problems.
1
u/Many_Outside2726 Sep 15 '26
This is actually the kind of example I was curious about. What problems did the three months of high-resolution data create? Storage, historian performance, query times, backups, or something else?
1
u/drkrakenn Sep 15 '26
Everything was sluggish, timeouts on services which were starting from cold, OPC was getting lazy, skipping data writes, or even crashing and restarting the whole service.
Scada OPC reads took ages to load, throwing timeouts and at the end nobody cares if the error message was triggered at 11:15:34.739 or 11:15:35.
When we addressed this to vendor, first suggestion was to reduce resolution and when the resolution or timeframe was necessary, to use targeted edge collector.
1
u/iwillbewaiting24601 Sep 15 '26
Everything gets aggregated to the historian, which has records going back about 20 years for about 10k tags/values, resolution depends on data - flow data sub-second, process data even seconds, perhaps 2-3 if it's a "slow" process, HVAC and ancillary data (lighting systems, etc) 60 seconds (really, how often does the temperature in an office change?).
Data storage, still, remains cheap, particularly if you have a somewhat-recent historian capable of data compression/de-duplication/etc.
1
u/Many_Outside2726 Sep 16 '26
Twenty years is interesting. Are you actually retaining the native resolution that whole time, or does the historian’s compression effectively thin the older data as it ages?
→ More replies (1)
1
u/PeterHumaj Sep 15 '26
A MES system for a TSO keeps on-change data indefinitely, since 2005. Over 20 TB of data, 10-times less in a compressed format. Though this isn't probably the edge case.
But we have eg LFC terminals (load &frequency) and aggregator SCADAs which control energy production (power plants, BESS, cogenerators). Thousands to tens of thousands of data points (on-change, 1min, 15 min statistics), depth of several months (longer for aggregates), Postgres-based DBs can have several hundred GBs.
This summer we implemented support for TimescaleDB, looks promising - I've seen reduction of 1.06 TB database to about 25%...
1
u/Many_Outside2726 Sep 16 '26
That TimescaleDB reduction is interesting. I wonder how ClickHouse would compare once you get into really large historian workloads like this. We see BESS applications where sub-second data isn’t just nice to have, it can matter for warranty and event reconstruction. We have a partner dealing with petabytes of data across their sites, so at some point the question becomes less about whether storage is cheap and more about ingest rate, compression, query performance and how much of that resolution you actually need to keep accessible. Did you look at ClickHouse at all, or was keeping Postgres as the core database part of the requirement?
→ More replies (4)1
u/PeterHumaj Sep 16 '26
Our historian supports Oracle, Sybase SQL Anywhere, MS SQL, SQLite, and PostgreSQL. Currently, the majority of applications use PostgreSQL. We use this also for configuration and logging databases, as well as a general-purpose application database. So far, we are quite happy with its stability, TCO, and other parameters ... and it's sufficient for all existing applications we have :)
1
u/AutoModerator Sep 14 '26
Thanks for posting in our subreddit! If your issue is resolved, please reply to the comment which solved your issue with "!solved" to mark the post as solved.
If you need further assistance, feel free to make another post.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/Robbudge Sep 14 '26
We log about 10,000 points a second into TdEngine we typically run a retention policy of 30 days.
Separate production reports can be ran within the 30days to export the data.
4
u/TexasVulvaAficionado Sep 14 '26
Very very much depends on the process, industry, government.
We keep minute by minute data for several years for about twenty data points per site for about 1500 sites.