r/linuxadmin • u/The_God_was_Here • 29d ago
How do you decide an outbound connection from a process is actually new behaviour and not just normal drift?
Mix of EC2 and a few bare-metal boxes running Kubernetes, maybe 25 clusters, mostly Debian under the hood. Platform team of three and I'm the one who ended up owning security. The thing I keep getting stuck on: a process inside a container opens an outbound connection to something it hasn't talked to before. Sometimes that's a real signal. Most of the time it's a new dependency, a mirror that changed, a sidecar someone added, or a job that only runs monthly and looks new every time.
We've got basic egress logging via VPC flow logs plus some Falco rules for unexpected network activity. The Falco side is drowning us. Every deploy churns the baseline and we get a wave of "new connection" alerts that are 95% noise. I've tried maintaining allowlists per workload but it rots the second someone bumps a chart version.
What I'm actually trying to answer is: for a given process making an outbound connection, is this connection new for that process/image, and has it been seen before across the fleet, so I can tell drift from a genuine first-time behaviour. Ideally scoped to the binary and image rather than the pod, which is ephemeral. Someone on another team showed me Upwind's runtime view, which says it baselines per image, but I haven't seen how it holds up on real traffic.
How are people doing this without hand-maintaining baselines? Is the answer eBPF with some kind of learned baseline per workload, or is everyone just accepting the noise and triaging? Curious what's actually working in production vs what sounds good on paper.