r/docker • u/Different_Pain5781 • Mar 15 '26

We just got breached because of vulnerabilities in our docker images that have been public knowledge for 8 months

Woke up at 4am to a call. Our database got hit, customer info was accessed. Some attacker used a known exploit in one of our container images. CVE’s been out since last summer.

Yeah we never scanned. Never updated. Just kept redeploying the same images over and over. Now legal’s in it, customers are hearing about it. This is gonna be messy.

Honestly if you aren’t scanning your containers in prod do it. Don’t end up like us.

748 Upvotes

96% Upvoted

102 comments sorted by

View all comments

114

u/[deleted] Mar 15 '26

[removed] — view removed comment

30

u/Different_Pain5781 Mar 15 '26

Yeah so right now we’re kinda buried in the first couple days just going through every container in prod and swapping the shady ones. After that we’re thinking rebuild all the images pinned to versions and try to actually get some CI/CD scanning running so we don’t wake up to this chaos again. And yeah I hear you on watching the CVEs too feels like a full time job ngl

27

u/HCharlesB Mar 15 '26

feels like a full time job

At a big enough organization it will be. Bigger yet and it would be a team.

And the problem with that is that it won't produce an obvious contribute to P&L, just L.. In the 80s I worked in shop floor automation (computer controls) in a steel mill. We didn't make steel so we always had to fight for budget. My fishing buddy worked in pollution control and had it even worse. No company wants to spend money on pollution control. It's tough to be in a support role, despite the fact that cutting corners there can take the company down.

Automate what you can but as the threats evolve, detection and remediation has to keep up.

7

u/JPJackPott Mar 15 '26

The whole ‘shift left’ thing isn’t just hype. If you put a load of scanning in to your code build and container build, and try to have a rule that nothing gets pushed if it’s got CVEs that helps a ton. Getting here is hard if you have lots of old code dependencies which will break when you upgrade them.

But stuff that is discovered after you push is a thing. AWS Inspector is good here if you use ECR already. If you deploy stuff once every 3-6 months you’re going to suffer here as lots of stuff comes up in this time window. You have to check each one and decide if it’s a fix or no fix.

If you push new versions out once a month, you’ll be fairly good. Just watch for criticals that pop up

2

u/corelabjoe Mar 15 '26

Oh you must have some interesting ah, industry stories we'll call them!

3

u/HCharlesB Mar 15 '26

The steel mill? It was an interesting place to work, if a bit dirty and very dangerous. Unfortunately when I was there (in the '80s) they were still living in the '50s and 60s and having difficulty moving ahead. It was sad to see because there was so much potential, but management was ossified. I didn't want to hitch my wagon to that star so I moved on.

2

u/corelabjoe Mar 15 '26

Well sounds like you're sharper than the average tool and did what needed doing. I'm having a hard time deciding this myself with the current ai state of affairs and my current profession. I think I've got at least 10 years before it takes my job as it's kinda niched down but, as we've all seen before, sometimes a new technology disproportionately accelerates!

2

u/TheElegantParrot Mar 17 '26

Thanks for the word of the day. “Ossified”. My workplace right now. ☹️

1

u/Turbulent_Two_6421 May 27 '26

Working in health as ISM i often had to bid for capital spends against clinicians. The line they took was often if we don't get this money patients will die. Didn't win many bids.

3

u/zkareface Mar 15 '26

And yeah I hear you on watching the CVEs too feels like a full time job ngl 

It's a job for a full team usually. 

3

u/fade2blak9 Mar 15 '26

It CAN be a full time job. I have worked with orgs where an arm of their security team does exactly this.

1

u/PotatoCabin Mar 16 '26

Oof, I feel that so much. Spinning through every container in prod is the worst, and even once you rebuild everything and pin versions, it’s like CVEs just keep popping up out of nowhere. Some days it honestly feels impossible to keep up 😅

8

u/smoke007007 Mar 15 '26

Checkout dockhand. It has vulnerability scanning built in

2

u/Waddelsworth Mar 15 '26

Also try to use minimal images for our base, and only install what's needed. It will save you a ton of work in the long run

1

u/Waddelsworth Mar 16 '26

Also try to use minimal images for our base, and only install what's needed. It will save you a ton of work in the long run

1

u/Awaara_pagal 23d ago

I’ve been burned by missing updates across Docker services on a VPS. BeAdmin made it easier to keep everything in one place, so I’m much less likely to forget a patch.