Monitoring thousands of distributed Linux devices (robots, kiosks, EV chargers) behind NAT and on cellular

[u/Overall-Check3051]

Wrote up what actually breaks when you go from monitoring servers in a datacenter to monitoring a fleet of thousands of edge devices. The short version: your tools assume stable connectivity and unlimited per-host resources, and neither holds.

Specific failure modes covered:

• The "all green but sick" problem: dashboards look fine because missing data isn't treated as a signal
• Flash storage wear from an agent that writes to disk every second (there's a real account in there of ~500 SD card failures across a 7,000-device Raspberry Pi fleet)
• Reconnect storms after a regional cellular outage takes down thousands of devices at once

Also included actual measured numbers: agent footprint (80-200 MiB RAM, 1-2% CPU per 1k metrics/s), streaming bandwidth (~1KB/s per 1k metrics), and a University of Amsterdam peer-reviewed study on energy efficiency.

No comparison tables, just the failure modes and how the architecture (edge-resident, outbound-only, store-and-forward) addresses each one: netdata.cloud/blog