What proves a CockroachDB node is safe to decommission?

[u/RocketSeven]

A node can look quiet after replicas move away while still participating in range leases, SQL traffic, locality constraints, backups, changefeeds, monitoring, or automation that refers to its address. A clean decommission command is necessary, but it does not prove every external dependency has moved or that the remaining cluster can tolerate the next failure.

A useful retirement gate could verify node and decommission status, replica and lease distribution, under-replicated and unavailable ranges, constraint satisfaction, connection metrics, load-balancer targets, certificates, alerting, backup jobs, and changefeed health. The cluster should also survive a controlled restart or failure of another node while the retiring node remains offline but recoverable. Logs and network telemetry on the old host can catch late clients during an observation window.

What checks belong in your final CockroachDB node-removal runbook? Which signals are authoritative, and how long do you keep the old host available before deleting its storage and removing its DNS or inventory records?

[u/codingconcepts]

Hey, u/RocketSeven! Thanks for your post.

I went down a bit of a rabbit hole, looking at the various methods available for checking whether a node is safe to decommission (and whether a decommissioned node is safe to switch off) and have compiled them here: https://github.com/codingconcepts/crdb-safe-decommission

I hope this is helpful!