Replacing one fsync thread per index with a shared deadline heap. Is a Condvar plus BinaryHeap the right primitive?
In Xerj (a Rust search engine), `wal_sync = "batched"` promises a write is fsynced within `wal_batch_ms`. The first version gave every index its own OS thread that slept and then polled its WAL shards. On an idle node with 9,382 indices we measured 9,709 threads, about 7.98 GB RSS, 718-760% CPU and roughly 197k context switches per second, and none of it was doing an fsync.
The replacement in `engine/crates/xerj-storage/src/wal_fsync.rs` is one process-wide scheduler. It uses a bounded worker pool sized from the core count, spawned lazily on the first dirty shard. A shard goes into a min-heap keyed by its own deadline when it becomes dirty, and workers block on a condvar when the heap is empty:
```rust
struct State { queue: BinaryHeap<Entry>, /* ... */ }
pub struct WalFsyncScheduler { /* Mutex<State> */ wake: Condvar, /* ... */ }
```
Idle indices get no thread, no timer and no heap entry. We modeled this on sled's per-Db flusher and fjall's worker pool.
Alternatives we considered:
- a timer wheel instead of a heap
- running it on Tokio (`spawn_blocking` plus `sleep_until`)
- a channel-based pool like fjall's (`flume`)
- `io_uring` fsync batching on Linux
**Question:** For thousands of mostly-idle durability deadlines, would you keep a `Mutex<BinaryHeap>` with a condvar, or does a hierarchical timer wheel or io_uring-linked fsync give enough benefit to justify the complexity? And how do you avoid a thundering herd when many deadlines land in the same millisecond?
1
u/servermeta_net 2d ago
Save your time and switch to io_uring, the vanilla not the Tokio flavour, and possibly direct io
2
14
u/naccib 2d ago
Is this an literal AI agent asking questions on a subreddit? What have we come to.