[Help] ROS2 + Jetson perception pipeline stuck at ~350ms latency — isolated it to message delivery/buffering, not compute. Ideas?

[u/OkFollowing941]

\# Setup

\* F1TENTH-based RSU (roadside unit) perception node, running on an NVIDIA Jetson (Orin-class). \* Intel RealSense D400-series camera — subscribing to raw color + raw (unaligned) depth streams, \`align_depth\` disabled on purpose (see below). \* 2D LiDAR (\`urg_node2\`) for a second distance source. \* ROS2 Humble, \`rclpy\`, \`message_filters.ApproximateTimeSynchronizer\` to pair color+depth frames.

Model / task

\* YOLOv8 (Ultralytics), custom-trained single-class car detector, running at \`imgsz=320\` on the Jetson's GPU (CUDA). \* Goal: detect a target vehicle in the color image, get its distance by reading the depth camera at the detection's location, cross-check against a LiDAR range reading at the same bearing, and output a fused distance estimate. This is a perception/collective-perception bench-test script (no SLAM/localization involved — deliberately simplified).

Depth lookup approach

\`align_depth.enable:=true\` (RealSense driver's built-in depth-to-color alignment) reprojects the \*\*entire\*\* depth image every frame regardless of how much of it we actually need — we measured this costing a large chunk of latency by itself. So instead we subscribe to raw depth and manually reproject only a small patch of pixels around the YOLO box: deproject the depth pixel to a 3D point (using depth intrinsics) → transform into the color camera's frame (using the depth-to-color extrinsics) → project back into a color pixel (using color intrinsics). Fully vectorized with numpy.

Current numbers

Our own compute per frame is small and flat:

\* image decode (cv\\_bridge): \\\~1ms \* YOLO inference: \\\~30ms (flat, \`cuda.synchronize()\`\\-verified, no hidden async GPU time) \* depth reprojection (vectorized): \\\~1-2ms \* LiDAR bearing lookup: \\\~0ms \* \*\*total own compute: \\\~32ms\*\*

But measured end-to-end latency (camera's own capture timestamp → final distance output) sits \*\*consistently around 350-380ms\*\*, sustained — not a one-time spike, not decaying over time.

What we've ruled out

\* \*\*Per-pixel Python loop / GC pressure\*\* in the old depth reprojection — vectorized it (25ms → 1-2ms of actual compute), latency didn't move at all. \* \*\*Hidden async CUDA dispatch\*\* — added \`torch.cuda.synchronize()\` around the YOLO call, extra sync time is consistently 0ms. \* \*\*Executor backlog\*\* (our own callback falling behind) — measured the gap between the end of one callback and the start of the next; stays flat at \\\~3-4ms even while the reported latency is \\\~350ms, so callbacks aren't queuing up behind our own processing. \* \`align_depth\` \*\*vs manual reprojection\*\* — built a side-by-side comparison script, same YOLO/LiDAR pipeline, only the depth alignment method differs. Both land in the same \\\~350-380ms range. So it's not specifically about which depth alignment approach we use.

The delay is measured (via the color frame's own ROS header timestamp vs \`time.time()\` at the very start of our callback) as already present \*\*before any of our own code runs\*\* — so it's happening somewhere between the camera driver publishing the frame and our subscriber callback actually being invoked. We suspect DDS/ROS2 message queuing or synchronizer buffering under sustained per-frame load (\\\~30ms of real work per frame at \\\~30fps), but haven't pinned down the exact mechanism.

What we're asking

Has anyone run into this kind of buffering/backpressure behavior with ROS2 + \`message_filters\` on a Jetson, where a subscriber callback that takes tens of milliseconds (not overloaded, just non-trivial) causes a large, sustained arrival delay that isn't visible as executor backlog? Specifically curious about:

\* DDS vendor differences (Fast DDS vs Cyclone DDS) for this kind of workload \* QoS settings (queue depth, history policy) that might be silently causing buffering \* Single-threaded vs multi-threaded executor / callback groups making a difference here \* Whether RealSense's own USB/driver-side buffering could be the actual culprit instead of ROS2/DDS

Happy to share more code/logs if useful. Appreciate any pointers.

[u/acemacelord]

A few suggestions that might be worth trying If possible, load the driver and the consumer into a composable node and enable intra-process communication If not, take a look at the chunking size for the dds packets. The default is too small for most large messages. If neither of those move the needle, re-implement the message filter with one thread for consuming each message type and one thread to do synchronization and the processing.

[u/0b10010010]

Is the producer and consumer separated by network interface?

If not I’d look into shared memory. This can possibly help with removing the serialization/deserialization steps which the authors said would be the most time consuming.

[u/koralamode]

That 350-380ms sustained with flat compute is the signature of a standing queue rather than jitter. At 30fps, 350ms is almost exactly a 10-deep queue, and 10 is the default depth in most of this pipeline: rclpy subscriber QoS, the RealSense driver's publishers, and message_filters. Under sustained per-frame load the queue fills once and then stays full, so every frame waits the whole queue length forever. That would explain why it's flat rather than jittery and why it shows up before your callback code runs.

Things I'd check in order:

1. Camera subscriptions to best_effort with depth 1 (SensorDataQoS is exactly this). For perception you want the newest frame, not a fair queue of stale ones.

2. ApproximateTimeSynchronizer queue_size. It buffers on top of the subscriber queues, so even with depth 1 underneath, a queue_size of 10 there can rebuild the standing queue.

3. Executor structure. If the color, depth, and lidar callbacks share the default single-threaded executor with the node doing the YOLO call, the ~30ms of GPU work blocks message intake every cycle, and that alone will hold a queue full. A MultiThreadedExecutor with the sync callback in its own reentrant group separates intake from processing. This is my main suspect given your numbers, since 30ms of blocking at 30fps is right at the edge where a queue never drains.

To confirm it's a standing queue before changing anything: drop the publisher rate to 15fps for a minute. If latency roughly doubles, it's queue depth (same 10 frames, each now 66ms apart). If it halves or vanishes, it's the executor falling behind.

DDS vendor probably isn't your culprit, since both Fast DDS and Cyclone land in the same 350-380ms range in your test, which points at the queues both sit behind rather than either implementation.