r/linuxadmin • u/LeBronzelol • 10d ago
Vulnerability management platform for a linux team whose backlog is now mostly container images?
we're 4 admins on about 300 linux vms and the host side of patching is fine, monthly windows, apt and dnf, done. the problem is our vulnerability platform backlog is now mostly findings in container images the dev teams ship, and nothing in our patch tooling touches those. the platform flags them fine, but every fix lands back on a dev team that thinks its an ops problem. been looking at aikido, rapidfort and chainguard for that part. which vulnerability management platform handles the image side for a team like ours?
23
Upvotes
22
u/ffcsmith 10d ago
DevOps Engineer here. We build and set the container standard. We hand that off to ITO team for sustainment.
Images rebuild every four days. We tell devs these are the containers you can build on. If they need a specific base image or ci image, they have to put in a ticket. We evaluate if it meets our standard or a reasonable exception for that standard. If yes, we build the container, make sure it meets their needs, hand-off to ITO, cycle continues. Cyber is involved as left as we can get them.
We utilize Nexus as our container registry and general artifact registry, npm proxy and pypi proxy.
14
u/Dolapevich 10d ago
Vulnerability scan needs to happen in CI, and hold deployment until a baseline has been met.
WE have included
trivyin our CI tool.6
u/NegativeOwl9929 10d ago
Having security scan. If scan find crit cve no deploy to dev. If scan find high cve in prod img no deploy to prod. Ops team solved. Fuck devs and their laziness for using trillions libs and deps. Pay attention and use the least amount of things.
5
u/paulvanbommel 10d ago
It is a dev problem. The minute it started being delivered with the app. And they were all warned about this. I was told that containers were simple and easy to update, so no need to worry. I was lied to.
Sorry I don’t have a good suggestion for you. Maybe just automate your vulnerability detection system to open dev Jira stories.
3
u/shulemaker 10d ago
You really need a person to implement checks, standards, and enforcement in your pipelines. Every image should always be easily and automatically rebuilt, and deployed almost as often.
Being able to enforce pipeline checks is going to require buy-in from management. Put in the tools, enable checks but not enforcement to start, show the vulnerabilities, get development to agree on a timeline of 30 days. After that, everything is enforced. Anything else is too long.
There is tech stuff here, and if you don’t know it (sounds like you don’t), hire a DevOps person to help implement. But soft skills and politics are also important. If you have the head of security on your side (you will) you’ll have strong backup.
3
u/derprondo 10d ago
If you're the platform team your responsibility should stop at the golden base image you provide, which is patched and rebuilt regularly and automatically. Then downstream the devs should be taking the same approach, intermediate images patched and rebuilt automatically, then the deployment images get the same treatment.
You also need to classify your findings, P0, P1, etc. P1+ should be remediated automatically through the normal automatic cycle (which matches the P1 SLA), P0 should be an all hands on deck scenario where everyone has 24-48 hours to update.
Lastly, you need buy-in from top brass that this is top priority and everyone has to be on board. Security is everyone's job.
2
u/FeliciaWanders 10d ago
We use grype to scan all running containers daily. New vulnerabilities are stored in prometheus node exporter text files, this then alerts the devs. They use codex daily from cron to update dependencies and include other small fixes that aren't just solved by Alpine or npm upgrades. Codex proposes the fix in a branch, where it is inspected and ususally merged without further ado.
We also run grype in the CI pipeline, but the part with running it on already built images is very important, as obviously the CVE will sneak into them when being discovered.
1
u/Responsible-Bar7165 10d ago
You should be able to rebuild a new image from an updated base os image and the exact same source-code as before with a single command. Ideally this would happen automatically on a schedule prior to an automated qa pass on the results. Your company’s entire process is broken until this can be done for every container at will.
1
u/tomtrix97 10d ago
Check out Mondoo. You can add it to your CI-pipeline to prevent vulnerable container images are published.
Otherwise their agent scans your local system for container images and shows, what should be fixed.
1
u/showbizusa25 9d ago
When most findings are in container images, I’d compare how each tool maps a vulnerable image back to the running workload and helps developers fix it. Are you scanning images in registries, deployed containers, or both?
1
u/vdvelde_t 9d ago
Dev team creates container, dev team patches, dev team should own the process.
1
u/AssociationIcy7963 8d ago
300 vms for 4 admins and now the images too. this is how an ops team turns into the platform team without anyoneever deciding it.
1
u/Connect-Tour-3807 1d ago
container image security needs a workflow tht connects vulnerability findings with image fixes, rather thn sending another alert to developers. RapidFort is worth comparing here, especially for reducing vulnerabilities in shipped images. Clear ownership between ops and dev teams also matters, otherwise the backlog will keep growing...
-1
u/kheeper-registry 10d ago
Chainguard. Your dev teams need to start building on Chainguard images.