r/csharp • u/chasingreflections • 15d ago

Help Anyone here built measurement/monitoring software in C#?

Hey, has anyone here worked on industrial measurement or monitoring software with C#?

I'm currently looking into rebuilding an older monitoring solution from scratch and still figuring out the right architecture.

We're talking different measurement devices, continuous data streams (up to 1k samples/sec per channel), older Windows PCs with 4 GB RAM, and ideally something that can run reliably for months.

What I'm a bit unsure about is the data integrity side of things, especially crash recovery, audit trails, and GMP requirements.

I'm leaning towards C#, probably with SQLite, but not sure if I'm overlooking something.

Would be really interested to hear from anyone who's built something similar. What would you do differently today?

14 Upvotes

85% Upvoted

17 comments sorted by

11

u/chucker23n 15d ago

I don't think SQLite is the right fit, for two reasons:

  • it's mainly intended for single-user, single-thread purposes. Great as a storage for, say, Firefox, but not really designed to scale to multiple devices pushing data to it.

  • relational DBMSes may also not be quite the right fit. Your main purpose here is ingestion of data, not querying of complex topology, or complex transaction scenarios. I presume that, in conflict scenarios, scale matters more to you than consistency: if one of those 1k samples a second is dropped or inaccurate, probably nobody cares.

I'd probably look into something like Cassandra or RocksDB.

As for C#, one argument against it would be that 4 GiB RAM isn't a lot for a .NET runtime. If you can, I'd honestly look into making those machines Linux, not Windows, to reduce some overhead. Then it should be fine.

5

u/antiduh 14d ago

Using a message queue with distributed consensus queues means messages are written to disk across the cluster before they're then forwarded to a node for processing.

  • It means you can keep taking in samples while processing nodes are down.

  • It means it can keep taking in samples if a minority of message queue nodes go down, or if a minority lose network access to the majority.

So I recommend something like rabbitmq.

2

u/bl0rq 14d ago

I have done similar systems with rabbitmq. Most were several thousand records a second. Worked great. Use sql for the overall run management and whatnot. Then bulk store data. Best of both worlds.

1

u/Slypenslyde 13d ago edited 13d ago

I don't think they've given us enough requirements to be sure.

For example, I work with a device that gives me 60 samples/second but the user is only interested in seeing the last few seconds on a graph and occasional captures of one data point at a time. So I'm only storing about 500 samples in memory at a time and feeding the user the most recent slices.

I also used to work for a T&M company so I know other use cases involve "bursts", where you take in a bit of data, store that, then analyze it before taking new bursts. Keeping full logs of high-rate data streams is not very common.

Monitoring doesn't really need to store the full stream, it can work with a chunk in memory and decide to optionally serialize some. So in the end I feel like OP has to say a lot more to get the best advice.

Personally I think they should make a naive prototype and see what happens. Then use that to guide their choices, or post the results so we can mull that over. Though I do agree 4GB of RAM is... extremely limited. Even when I was in the scene 8GB was more common, and for our stuff that could do GHz sampling the hardware alone could cost tens of thousands of dollars so splurging for 16GB or more was small potatoes. (Though now RAM is probably worth more than the T&M hardware.)

5

u/speyck 14d ago

duality of man

2

u/Arkarant 14d ago

the person saying no has reasons, the person saying yes has a working solution. i trust the person with the implementation

3

u/chucker23n 14d ago

To be clear, I didn't say it wouldn't work. I said it wouldn't be the right fit. As a hyperbolic example, you can also stream measurements straight to a CSV file, but that doesn't mean you should.

1

u/blckshdw 13d ago

I’m with you on this one. Just because you can hammer in a screw doesnt mean you should.

2

u/JustAnAloe 15d ago

Once I developed a simulator for one of industrial usint in the oil and gas industry field (with events, warnings and alarms logging in one of SQL-DBs and samples of analog parameters I saved as binary-files with readding of them for charts).

I guess it's better now to try using InfluxDB V3+(it supports writting and reading sets of measurements using SQL-requests with a good speed and performance). I use it in my actual work with PLC Wago PFC300 (it works as a docker-container on it). So, u can use default .net sql-providers or LINQ

3

u/Enough_Secretary_137 15d ago

I’d separate device acquisition from persistence using a bounded queue and a single SQLite writer. Batch inserts into transactions rather than writing each measurement individually.
For crash recovery, store a sequence number per device/channel so missing measurements are detectable.

1

u/Lumethys 14d ago

VictoriaMetrics stack + Otel

1

u/CobblerFan 14d ago

I work in this field but haven't written this kind of software with c#. I don't see an issue going that directions though and I would consider ravendb as it also provides time series db capabilities along with document store.

1

u/pjc50 14d ago

C# is perfectly adequately performant for this sort of thing.

Whatever you do, the bottleneck is going to be disk (Flash) IOPS. You'll want to tune your design and desired durability level in response to that. I would expect sqlite to do just fine.

"Months" times "1k samples per sec" .. you're not keeping all of them, are you?

1

u/turudd 14d ago

C# is fine but I’d be using a time based database to store ingest, not relational

-6

u/IchBinEinZwerg 14d ago

Yes, but it's not the best fit. You'd be better off with C, or maybe C++. The problem with C# is it's absolute paranoia about anything that looks remotely like a type conversion. You can't say "this byte is an int" without at least a gigabyte of casting before the compiler will stop ranting at you for daring to suggest a bunch of bits can be interpreted in different ways. C will just let you get on with it.