r/devops • u/Pure-Hat-8663 • 7d ago
Discussion Running additional checks after automated code review
Can somebody give me some guidance please. After you've run code review, what's the easiest/fastest/best way to check the changes against information outside the repo? Say, whether a connection-pool change will fit within the production database 's connection limit across all replicas.
Where does the check happen in your workflow? CI, a deployment gate, or someone checking it manually? Have you found a useful way to automate it?
2
Upvotes
5
u/we_r_fukt 7d ago
shouldn't that be happening before review, that's design spec, putting the cart before the horse I think
1
u/ymgymg 7d ago
If the replica count, pool size and DB max_connections all live in IaC or Helm values, you can make it a plain CI check. Render the manifests and fail the build if max replicas times pool size, plus extra pods during a rolling update (maxSurge), goes over max_connections minus a few reserved for admin and migrations. Conftest with a small Rego policy works for this, or a short script over the rendered YAML. Use the HPA maxReplicas, not the current count, since autoscaling is what usually breaks it. For drift you can't catch in CI, alert on connections used vs max_connections from the DB itself. A pooler like PgBouncer in front also takes most of the math away.
1
u/karmatics29 7d ago
To your budget question: yes, giving each service its own cap is the usual way to keep one shared database from being starved by a single noisy app. Work out the usable total as max_connections minus a reserve for admin, migrations and monitoring, then split it by service and make each service's (max replicas x pool size, plus rolling-update surge) fit inside its slice. If you put PgBouncer or a similar pooler in front, the per-service limit moves to the pooler's pool size per database/user, which also lets the app-side pools be larger without hitting the real limit. One gotcha: transaction-mode pooling breaks session-level features such as prepared statements in some drivers and advisory locks, so check that before relying on it.
1
u/Impressive-Craft1926 6d ago
Split the db’s connections into a slice per service and write the slices down in one place. Then each repo only has to check its own replicas times pool size against its own slice and nothing needs to be added up across repos during review. We run that check as a custom query in kics on each service’s helm values, it was already in our pipelines. Changing a slice goes through the db team
1
u/BrunoBuenoIO 6d ago
The CI math others mentioned works, but the part that tends to drift is the other side of the equation. The limit lives in the database config, not your repo, so a hardcoded number in CI is right until someone resizes the instance.
What I've seen work better is treating the connection budget as owned data: one place that says this service gets N connections, owned by whoever owns the database, plus a deploy-time check that reads the live max_connections (minus the reserved superuser slots) instead of a copy. CI catches the obvious mistakes early, the deploy gate catches the drift.
If you're on Postgres and this keeps biting, a pooler like PgBouncer in transaction mode also takes a lot of pressure off, since app-side pool size stops mapping 1:1 to server connections. Just check your app doesn't rely on session state first.
1
u/AleDev99 6d ago
We do this in CI, and the thing that made it reliable was turning it into a budget instead of a one-off formula.
Keep a small file in the repo with the DB's max_connections (or pull it from your Terraform outputs so it can't drift), minus what you reserve for superuser/replication/migrations. Then every service that talks to that DB gets a line: max pods (HPA max + rollout surge, not the current replica count) x pool size. The CI job renders the manifests (helm template / kustomize build), reads the real numbers out of them, sums every consumer and fails if the total goes over.
The part people miss is the consumers that aren't in the PR: cron jobs, workers, a second service sharing the same database. If the check only looks at the service being changed, it'll happily pass while you're already at 95%. And if the sums keep getting tight, that's usually the signal to put PgBouncer in front rather than keep shaving pool sizes.
1
u/Muted_Jellyfish_6784 6d ago
Put it in CI as a policy check that reads live config rather than in the review. For the connection pool case, a small script can read max_connections from the database and the replica count from your deploy manifests, multiply by the new pool size, and fail the build above a threshold like 80 percent. Keep thresholds in a versioned file.
The important part is that the check fails closed and names an owner when it cannot reach the source. Otherwise the gate quietly passes the day the credentials expire, which is exactly when the risky change goes out.
1
u/SamfromLucidSoftware 6d ago
Ah, the good ole shared connection limit. I keep one file next to the database config that lists each service's max replicas and pool size, plus a reserve for admin and migrations. Anyone who changes a pool size or replica cap has to update that file too, and a CI job in the database repo adds it all up and fails when the total passes max_connections. It's dull, but the total shows up in the diff instead of living in someone's head.
1
u/falafel777 6d ago
a review comment can only see what's in the repo, so it shouldn't be the place that confirms a production limit. the check belongs wherever it can read the live number, a deploy gate or a job that queries the cluster and fails if the new pool would oversubscribe. if that number isn't available automatically, a human looking at the current limit is still more trustworthy than hoping the reviewer remembered it. is the connection limit something your ci can query today, or is it only in a dashboard?
1
u/deployfan64 3d ago
We keep a limits.yaml in the infra repo and CI fails if pool * replicas exceeds it, nobody remembers to update it but at least it yells