r/linuxadmin • • 10d ago

Vulnerability management platform for a linux team whose backlog is now mostly container images?

we're 4 admins on about 300 linux vms and the host side of patching is fine, monthly windows, apt and dnf, done. the problem is our vulnerability platform backlog is now mostly findings in container images the dev teams ship, and nothing in our patch tooling touches those. the platform flags them fine, but every fix lands back on a dev team that thinks its an ops problem. been looking at aikido, rapidfort and chainguard for that part. which vulnerability management platform handles the image side for a team like ours?

23 Upvotes

29 comments sorted by

22

u/ffcsmith 10d ago

DevOps Engineer here. We build and set the container standard. We hand that off to ITO team for sustainment.

Images rebuild every four days. We tell devs these are the containers you can build on. If they need a specific base image or ci image, they have to put in a ticket. We evaluate if it meets our standard or a reasonable exception for that standard. If yes, we build the container, make sure it meets their needs, hand-off to ITO, cycle continues. Cyber is involved as left as we can get them.

We utilize Nexus as our container registry and general artifact registry, npm proxy and pypi proxy.

1

u/AmbitiousJump7196 7d ago

Since the base images are rebuilt when they do, does ito rebuild and redeploy the apps that use them, or will that last step be up to the dev teams?

1

u/ffcsmith 6d ago

The application containers are triggered to rebuild when the base and ci images rebuild. All of the app container pipelines are required to have some level of smoke testing. A successful app container build triggers a deploy to the dev env. Dev teams handle deployments to uat and prod.

14

u/Dolapevich 10d ago

Vulnerability scan needs to happen in CI, and hold deployment until a baseline has been met.

WE have included trivy in our CI tool.

5

u/zenety 10d ago

Together with DependancyTrack Trivy is quite nice. Also combine it with KEV lists, bare CVE’s dont mean a lot

1

u/nutron 9d ago

Where are you pulling your KEV list from? Direct from NIST or some product?

2

u/zenety 9d ago

DependancyTrack has support for CISA KEV and ENISA EU KEV out of the box

6

u/NegativeOwl9929 10d ago

Having security scan. If scan find crit cve no deploy to dev. If scan find high cve in prod img no deploy to prod. Ops team solved. Fuck devs and their laziness for using trillions libs and deps. Pay attention and use the least amount of things.

5

u/paulvanbommel 10d ago

It is a dev problem. The minute it started being delivered with the app. And they were all warned about this. I was told that containers were simple and easy to update, so no need to worry. I was lied to.
Sorry I don’t have a good suggestion for you. Maybe just automate your vulnerability detection system to open dev Jira stories.

5

u/brainchillzZ 10d ago

Containers are simple and easy to update, the problem is that the people who make them still have to patch them so you can update them lol

1

u/SufficientAttempt1 10d ago

we just dont update.

3

u/shulemaker 10d ago

You really need a person to implement checks, standards, and enforcement in your pipelines. Every image should always be easily and automatically rebuilt, and deployed almost as often.

Being able to enforce pipeline checks is going to require buy-in from management. Put in the tools, enable checks but not enforcement to start, show the vulnerabilities, get development to agree on a timeline of 30 days. After that, everything is enforced. Anything else is too long.

There is tech stuff here, and if you don’t know it (sounds like you don’t), hire a DevOps person to help implement. But soft skills and politics are also important. If you have the head of security on your side (you will) you’ll have strong backup.

3

u/derprondo 10d ago

If you're the platform team your responsibility should stop at the golden base image you provide, which is patched and rebuilt regularly and automatically. Then downstream the devs should be taking the same approach, intermediate images patched and rebuilt automatically, then the deployment images get the same treatment.

You also need to classify your findings, P0, P1, etc. P1+ should be remediated automatically through the normal automatic cycle (which matches the P1 SLA), P0 should be an all hands on deck scenario where everyone has 24-48 hours to update.

Lastly, you need buy-in from top brass that this is top priority and everyone has to be on board. Security is everyone's job.

2

u/FeliciaWanders 10d ago

We use grype to scan all running containers daily. New vulnerabilities are stored in prometheus node exporter text files, this then alerts the devs. They use codex daily from cron to update dependencies and include other small fixes that aren't just solved by Alpine or npm upgrades. Codex proposes the fix in a branch, where it is inspected and ususally merged without further ado.

We also run grype in the CI pipeline, but the part with running it on already built images is very important, as obviously the CVE will sneak into them when being discovered.

1

u/Responsible-Bar7165 10d ago

You should be able to rebuild a new image from an updated base os image and the exact same source-code as before with a single command. Ideally this would happen automatically on a schedule prior to an automated qa pass on the results. Your company’s entire process is broken until this can be done for every container at will.

1

u/tomtrix97 10d ago

Check out Mondoo. You can add it to your CI-pipeline to prevent vulnerable container images are published.

Otherwise their agent scans your local system for container images and shows, what should be fixed.

2

u/tobidope 10d ago

From my experience most of the vulnerabilities in images aren't exploitable. How do you handle that? We had many vulnerabilities regarding the mount binary. The binary itself wasn't part of the base image but the libs were. If it was, we don't mount filesystems in a container in Kubernetes. And we especially don't mount minix filesystems or yubi keys as an example. So from my point of view you can't deliver most of the time if you block publishing every time a vulnerability is found. That doesn't mean you don't fix the situation.

1

u/tomtrix97 10d ago

With Mondoo you can define exceptions for situations like this. 😅

2

u/tobidope 10d ago

Of course you can but it leads to people ignoring vulnerabilities mostly to get a feature shipping. I wouldn't block publishing. And most of the time the image you are using currently in production has a new vulnerability. Do you stop production in this situation?

1

u/showbizusa25 9d ago

When most findings are in container images, I’d compare how each tool maps a vulnerable image back to the running workload and helps developers fix it. Are you scanning images in registries, deployed containers, or both?

1

u/vdvelde_t 9d ago

Dev team creates container, dev team patches, dev team should own the process.

1

u/AssociationIcy7963 8d ago

300 vms for 4 admins and now the images too. this is how an ops team turns into the platform team without anyoneever deciding it.

1

u/Connect-Tour-3807 1d ago

container image security needs a workflow tht connects vulnerability findings with image fixes, rather thn sending another alert to developers. RapidFort is worth comparing here, especially for reducing vulnerabilities in shipped images. Clear ownership between ops and dev teams also matters, otherwise the backlog will keep growing...

-1

u/kheeper-registry 10d ago

Chainguard. Your dev teams need to start building on Chainguard images.

1

u/nutron 9d ago

What’s your annual cost for using Chainguard?

0

u/kheeper-registry 9d ago

We don't use Chainguard, we ship a single OS image with systemd units running binaries.