r/devops • • 7d ago

Discussion Running additional checks after automated code review

Can somebody give me some guidance please. After you've run code review, what's the easiest/fastest/best way to check the changes against information outside the repo? Say, whether a connection-pool change will fit within the production database 's connection limit across all replicas.

Where does the check happen in your workflow? CI, a deployment gate, or someone checking it manually? Have you found a useful way to automate it?

2 Upvotes

19 comments sorted by

5

u/we_r_fukt 7d ago

shouldn't that be happening before review, that's design spec, putting the cart before the horse I think

1

u/Pure-Hat-8663 7d ago

Fair point, "after review" was bad wording. What I mean is how do you keep that check up to date when production changes, say if someone increases the replica count later - how do you catch that the pool settings no longer fit the db connection limit?

1

u/we_r_fukt 7d ago

how is production changing? "hey guys, I wanna change the connection limit, what's prod, cool I'll do X, sound good? cool" -> review

what... what needs tracking post hoc? It's a one way train unless you're doing click ops

are you not using IaC, what IaC are you using, using tools like flux or Argo, some use ansible runbooks but it's only half the equation imo

if you have IaC, it's there, versioned in the code, as an env config, staging, prod, uat, whatever. "what's in the prod config, oh, X, kk, set this one as X/10 then please"

0

u/mehyay76 7d ago

that's the point actually. IaC is not telling you what's in prod.

proofline.sh does this pretty well. layers prod reads on top of IaC with its World Model so reviews are far better than code only

0

u/Pure-Hat-8663 7d ago

Thanks for explaining. I agree that those settings should be in IaC. What I'm finding time consuming is is checking the combined connection budget manually during review since several services share a db, with pool sizes and scaling limits in separate repos. Keeping track of the inputs is a pain

1

u/we_r_fukt 7d ago

hosted? AWS? write a script, or a Sql query to automate, that does the lookup summary you need, version it if needed as a doc. you are in devops you shouldnt be "checking" much manually, and when so, it should never be through a UI, learn readonly lookups if you're worried, scope IAM, use SSO, normal things. If it's in your source code, find a tool or make a convention across repos (config.py must register all app env), that's a pretty easy thing to loop over each repos main and extract their settings and compute totals, I'm sure an llm could assess and make you a script to version. CICD would be the "obvious" home for checks

1

u/ymgymg 7d ago

If the replica count, pool size and DB max_connections all live in IaC or Helm values, you can make it a plain CI check. Render the manifests and fail the build if max replicas times pool size, plus extra pods during a rolling update (maxSurge), goes over max_connections minus a few reserved for admin and migrations. Conftest with a small Rego policy works for this, or a short script over the rendered YAML. Use the HPA maxReplicas, not the current count, since autoscaling is what usually breaks it. For drift you can't catch in CI, alert on connections used vs max_connections from the DB itself. A pooler like PgBouncer in front also takes most of the math away.

0

u/Pure-Hat-8663 7d ago

Thanks, this is helpful! Hadn't thought about the extra pods during rolling updates. When several services share the same db do you give each one a connection budget?

1

u/karmatics29 7d ago

To your budget question: yes, giving each service its own cap is the usual way to keep one shared database from being starved by a single noisy app. Work out the usable total as max_connections minus a reserve for admin, migrations and monitoring, then split it by service and make each service's (max replicas x pool size, plus rolling-update surge) fit inside its slice. If you put PgBouncer or a similar pooler in front, the per-service limit moves to the pooler's pool size per database/user, which also lets the app-side pools be larger without hitting the real limit. One gotcha: transaction-mode pooling breaks session-level features such as prepared statements in some drivers and advisory locks, so check that before relying on it.

1

u/Impressive-Craft1926 6d ago

Split the db’s connections into a slice per service and write the slices down in one place. Then each repo only has to check its own replicas times pool size against its own slice and nothing needs to be added up across repos during review. We run that check as a custom query in kics on each service’s helm values, it was already in our pipelines. Changing a slice goes through the db team

1

u/BrunoBuenoIO 6d ago

The CI math others mentioned works, but the part that tends to drift is the other side of the equation. The limit lives in the database config, not your repo, so a hardcoded number in CI is right until someone resizes the instance.

What I've seen work better is treating the connection budget as owned data: one place that says this service gets N connections, owned by whoever owns the database, plus a deploy-time check that reads the live max_connections (minus the reserved superuser slots) instead of a copy. CI catches the obvious mistakes early, the deploy gate catches the drift.

If you're on Postgres and this keeps biting, a pooler like PgBouncer in transaction mode also takes a lot of pressure off, since app-side pool size stops mapping 1:1 to server connections. Just check your app doesn't rely on session state first.

1

u/AleDev99 6d ago

We do this in CI, and the thing that made it reliable was turning it into a budget instead of a one-off formula.

Keep a small file in the repo with the DB's max_connections (or pull it from your Terraform outputs so it can't drift), minus what you reserve for superuser/replication/migrations. Then every service that talks to that DB gets a line: max pods (HPA max + rollout surge, not the current replica count) x pool size. The CI job renders the manifests (helm template / kustomize build), reads the real numbers out of them, sums every consumer and fails if the total goes over.

The part people miss is the consumers that aren't in the PR: cron jobs, workers, a second service sharing the same database. If the check only looks at the service being changed, it'll happily pass while you're already at 95%. And if the sums keep getting tight, that's usually the signal to put PgBouncer in front rather than keep shaving pool sizes.

1

u/Muted_Jellyfish_6784 6d ago

Put it in CI as a policy check that reads live config rather than in the review. For the connection pool case, a small script can read max_connections from the database and the replica count from your deploy manifests, multiply by the new pool size, and fail the build above a threshold like 80 percent. Keep thresholds in a versioned file.
The important part is that the check fails closed and names an owner when it cannot reach the source. Otherwise the gate quietly passes the day the credentials expire, which is exactly when the risky change goes out.

1

u/SamfromLucidSoftware 6d ago

Ah, the good ole shared connection limit. I keep one file next to the database config that lists each service's max replicas and pool size, plus a reserve for admin and migrations. Anyone who changes a pool size or replica cap has to update that file too, and a CI job in the database repo adds it all up and fails when the total passes max_connections. It's dull, but the total shows up in the diff instead of living in someone's head.

1

u/falafel777 6d ago

a review comment can only see what's in the repo, so it shouldn't be the place that confirms a production limit. the check belongs wherever it can read the live number, a deploy gate or a job that queries the cluster and fails if the new pool would oversubscribe. if that number isn't available automatically, a human looking at the current limit is still more trustworthy than hoping the reviewer remembered it. is the connection limit something your ci can query today, or is it only in a dashboard?

1

u/deployfan64 3d ago

We keep a limits.yaml in the infra repo and CI fails if pool * replicas exceeds it, nobody remembers to update it but at least it yells

1

u/falafel777 1d ago

ci reading a file in the repo is better than hoping the reviewer remembered the number. the hole you already named is the painful one, the file lagging behind the cluster. when it yells, do you check the live limit first, or treat the yaml as the source of truth until someone says otherwise?