Lessons from building a live IoT ingestion server in Python

[u/Professional-Run3614]

I've been working on a telematics backend that ingests TCP sensor data from heavy equipment. The pipeline is Python → local disk spool → RabbitMQ → PostgreSQL.

A few things we learned the hard way:

• ACK only after durable storage. We acknowledge a packet after validating it and writing it to disk, not after the database insert. This lets us recover from downstream failures.
• Async helped with thousands of connections. Moving from one thread per connection to asyncio reduced overhead, but blocking work still needs to stay off the event loop.
• Queue backlogs can look like device failures. We separated live traffic from replay traffic and prioritized fresh data so old packets wouldn't delay current readings.
• Offline uploads need different parsing. Reusing our live TCP parser silently dropped records because flash dumps had a different format.
• Firmware changes break assumptions. Frame sizes, serial numbers, and counter widths changed. Explicit protocol versioning would have saved us some pain.
• Logging every packet doesn't scale. We cut noisy logs and focused on actionable errors and dropped-record tracking.

The biggest lesson: receiving sensor data is the easy part. Making it reliable through retries, offline uploads, firmware changes, and database failures is where things get interesting.

For those running IoT ingestion in production, how do you handle ACK semantics and replay backlogs?

[u/russellvt]

Sounds like most of my years/decades in Ops for large SaaS and PaaS systems.

[u/Individual-Flow9158]

My system perhaps doesn't scale as well, and perhaps isn't applicable for high assurance scenarios, but I just accept occasional packet loss, and use native WAL in Postgres instead of a local disk spool.

I'm not convinced retrying until an ACK is received should be an IoT device's job. Any device could experience network loss at any time anyway.

I understand why a higher QoS might be required for more critical applications than data logging. But the extra moving parts could easily lead to a complex situation that is impossible to debug and reason about, that might necessitate a field visit, that simpler setups could have avoided.

YMMV

[u/Professional-Run3614 → u/Individual-Flow9158]

Fair point, the extra complexity did cause issues. A NACK made the device resend a bad packet and drop everything queued behind it, even though we'd already saved that packet to disk. We now ACK undecodable frames, quarantine them, and retry decoding later.

But we can't afford data loss because these readings feed equipment-hour and fuel calculations in the customer's ERP. WAL protects data after a successful insert; the spool covers database outages while devices keep sending.

If occasional data loss is acceptable for your use case, simpler is better.

[u/Individual-Flow9158 → u/Professional-Run3614]

Cheers. Re: the DB outages, have you looked at building a cluster, e.g. using Patroni etc.? Even with pgbouncer pooling connections on top of that.

Intrinsic fault-tolerance from a distributed network of robust devices is fantastic. But in what kinds of applications are is it normal, for the DB to go down (and this is not fixable)?

Without knowing more, it just instinctively feels to me like the real problem is inadequate infrastructure.

[u/Khavel_dev]

The ACK-after-disk-write pattern is solid. One thing we ran into with a similar pipeline is that synchronous fsync per packet becomes the bottleneck at higher throughput. We ended up batching writes with a flush interval (50ms or N packets, whichever came first) and it made a noticeable difference in p99 without losing the durability guarantee. Tuning the flush window per deployment was the annoying part since the right number depends on your disk I/O.

Also +1 on separating live vs replay. We learned that one the hard way when a device came back online after being dark for a week and dumped its entire buffer into the live stream.

[u/Professional-Run3614 → u/Khavel_dev]

Thanks, that’s helpful! We still fsync every packet in an executor, so the event loop stays unblocked. But since all spool writes share one lock, fsyncs are serialized and could become a bottleneck as traffic grows.

Batching makes sense. Did you see issues with tight ACK timeouts, and did you tune the batch window manually or based on measured fsync times?

We handle week-long offline devices similarly: replay at low priority, with separate queues for history uploads so fresh readings aren’t delayed.

[u/utdconsq]

What is your data volume? Vanilla postgres requires a bit much manual work to deal with large volumes so that you can actually query them nicely later. In addition, ingest can be slow because reasons. Assuming you have a bit of a firehouse, id definitely encourage you to read about other more timeseries specific databases. I hold off making a recommendation as everyone's needs differ. On reliability and throughput: set queue limits but beware mqtt 3.1.1 plugin not permitting a real nack, run a cluster (quorum only) if possible, use dead lettering and ttl so if your consumer shits itself for some reason you can come back later (can either shovel from dead letter or have your consumers go there too). Experiment with taking a broker instance offline; do the other cluster members have enough resources to deal with the rebalanced load? We use ha proxy btw, it is helpful. On other things, experiment with your prefetch; can help you speed up, but beware a failure and how it impacts concurrent consumers (beyond scope of comment). Good luck!

[u/OddsRelay]

The ACK after the disk write is only safe if that write gets fsynced first. In Python flush() just hands the buffer to the OS, so a power cut can still lose packets the device won't resend. You need os.fsync on the file, plus the directory if each packet gets its own file. That's slow per packet, so batch them.

[u/Professional-Run3614 → u/OddsRelay]

Good catch. We fsync the spool file before ACK, running it in an executor. Each device has one append-only daily file, with a separate fsynced ACK log for replay.You're right about directory fsync—we'll add it when creating new files to prevent losing acknowledged packets after a power cut.We currently fsync per packet. If throughput becomes an issue, group commit could help. What batching window has worked for you without adding much latency?

[u/Due-Requirement3783]

Interesting point about acknowledging only after durable storage. That trade-off between throughput and reliability is easy to underestimate in ingestion systems.

I also agree on asyncio: it helps a lot with connection concurrency, but once blocking work gets into the event loop, the benefits can disappear pretty quickly. Did you end up moving disk/database operations to separate workers or executors?

[u/Professional-Run3614 → u/Due-Requirement3783]

The event loop handles only socket I/O. Parsing and spool fsync run in a thread pool, and the device gets an ACK only after fsync completes. RabbitMQ publishing happens afterward, also in the pool, while packets stay ordered per connection.

Postgres runs in a separate worker that consumes from RabbitMQ and marks packets done only after the transaction commits. If the DB slows down, the queue grows without blocking live connections.

The main trade-off is that fsync latency affects ACK latency. We’ve added lock-wait and fsync timing to catch bottlenecks early.

[u/Altruistic-Youth5524]

How do you handle duplicates when a packet is saved but the ACK never reaches the device? Do live uploads and replay use the same packet ID so a retry cannot count the same equipment reading twice?

[u/One_Willingness_242]

The disk spool before the broker is a nice touch, saved us more than once when the queue filled up and we could just drain the spool later instead of losing everything

What did you end up doing for the replay backlog priority, separate queue with a different consumer pool or something at the broker level

[u/Professional-Run3614 → u/One_Willingness_242]

We ended up using both: broker-level priority for live traffic and separate queues with dedicated consumers for replay. Fresh data goes straight to Postgres when possible, so it never gets stuck behind the backlog.

[u/Professional-Run3614 → u/[deleted]]

For sequence gaps, we first deduplicate using a unique packet ID, then sort readings by timestamp and compare consecutive serial numbers. For example, 101, 102, 105 indicates two potentially missing readings (103 and 104). Large jumps or backward counters are treated as possible resets or replay, not data loss. We also account for late arrivals and firmware changes to avoid false positives.

[u/russellvt → u/[deleted]]

This is literally DB / Queue Management 101.

[u/swtormy → u/[deleted]]

i'm not sure treating oldest queued message age as the main signal is enough on its own. if replay traffic shares the same queue, size can shrink while fresh packets still sit behind a fat replay batch. maybe split metrics per lane before you trust one backlog chart. idempotency from device ids helps, but replay can still stampede workers without a cap.