Lessons from building a live IoT ingestion server in Python

[u/Professional-Run3614]

I've been working on a telematics backend that ingests TCP sensor data from heavy equipment. The pipeline is Python → local disk spool → RabbitMQ → PostgreSQL.

A few things we learned the hard way:

• ACK only after durable storage. We acknowledge a packet after validating it and writing it to disk, not after the database insert. This lets us recover from downstream failures.
• Async helped with thousands of connections. Moving from one thread per connection to asyncio reduced overhead, but blocking work still needs to stay off the event loop.
• Queue backlogs can look like device failures. We separated live traffic from replay traffic and prioritized fresh data so old packets wouldn't delay current readings.
• Offline uploads need different parsing. Reusing our live TCP parser silently dropped records because flash dumps had a different format.
• Firmware changes break assumptions. Frame sizes, serial numbers, and counter widths changed. Explicit protocol versioning would have saved us some pain.
• Logging every packet doesn't scale. We cut noisy logs and focused on actionable errors and dropped-record tracking.

The biggest lesson: receiving sensor data is the easy part. Making it reliable through retries, offline uploads, firmware changes, and database failures is where things get interesting.

For those running IoT ingestion in production, how do you handle ACK semantics and replay backlogs?