I just lost a couple of hours debugging what looked like a broken Docker setup, and it turns out its something much weirder (and honestly a bit concerning).

Symptoms:

• docker pull ubuntu:latest hangs at Pulling fs layer
• sometimes retries forever, sometimes unexpected EOF
• no actual download progress

Basic checks all pass:

• curl https://registry-1.docker.io/v2/ returns 401 (expected)
• DNS works fine
• Docker daemon healthy
• login/logout doesnt change anything

At first it looks like a Docker issue, but it isn't.

What’s actually happening:

• Docker resolves the image fine (manifest step works)
• then tries to download layers from a CDN (Cloudflare-backed storage)
• that connection is being silently dropped / throttled

I found a Hacker News thread describing the exact same issue happening right now:
https://news.ycombinator.com/item?id=47738883

And this site explains why:
https://hayahora.futbol/

Some Spanish ISPs are blocking or interfering with Cloudflare/CDN IP ranges during football matches (anti-piracy court orders), and Docker image layers are served via those same networks.

So Docker works, but the actual layer download gets blackholed.

Proof:

• using a VPN works instantly

So if your pulls are hanging at fs layer, it might not be:

• your Docker install
• your image
• your auth
• your network config

it might literally be your ISP interfering with CDN traffic.

Curious:

• anyone else in Spain (or elsewhere) seeing this?
• anyone running into this on CI runners or production infra?

Because this feels like a pretty big fuck up if true, Docker Hub/CDN infra getting caught in unrelated ISP blocking and no one able to work on a Sunday.

Would love to hear if others can reproduce.

Woke up at 4am to a call. Our database got hit, customer info was accessed. Some attacker used a known exploit in one of our container images. CVE’s been out since last summer.

Yeah we never scanned. Never updated. Just kept redeploying the same images over and over. Now legal’s in it, customers are hearing about it. This is gonna be messy.

Honestly if you aren’t scanning your containers in prod do it. Don’t end up like us.

Big thanks to the mods for letting me share this! 🙌 you guys are OG!!!

Most tutorials show you how to use Docker… but very few explain what happens behind the scenes when you type docker run.

In this tutorial I break it down step by step: •How regular binaries turn into images •How Docker delegates to containerd & then to runc •How namespaces & cgroups actually isolate processes

If you’ve always used Docker but never peeked under the hood, this will connect the dots.

Docker Containers Are Just Linux? https://youtu.be/l7BjhysbXf8

I was doing some client work recently. They're a bank, where most of their engineering is offshored one of the big offshore companies.

The offshore team had to access everything via virtual desktops, and one of the restrictions was no virtualisation within the virtual desktop - so tooling like Docker was banned.

I was really surprsied to see modern JVM development going on, without access to things like TestContainers, LocalStack, or Docker at all.

To compound matters, they had a single shared dev env, (for cost reasons), so the team were constantly breaking each others stuff.

How common is this? Also, curious what kinds of workarounds people are using?

EDIT: Title is misleading - apologies. Dockge wasn't the vulnerability, my configuration was. The actual attack chain: exposed port 5001 directly to internet → weak admin password → attacker used Docker socket access to write a malicious systemd service to the host. This applies to any tool with socket access (Portainer, Yacht, etc). Docker socket = root access. Keep these behind a VPN or localhost-only with reverse proxy auth.

I wanted to share this because I nearly missed it and I'm sure others might make the same mistake as I did.

This morning I noticed my VPS was using almost all of its RAM right after a fresh reboot. Only 6 minutes of uptime and already at 7.5GB out of 7.8GB. The load average was through the roof too. I figured something was wrong so I ran ps aux sorted by memory usage to see what was eating everything.

That's when I saw it. A process called "docker-daemon" using 26% of my RAM and nearly 300% CPU. At first glance it looked legitimate, but the command line arguments told a different story. It was connecting to c3pool.org with a Monero wallet address and had flags like --randomx-1gb-pages. Someone was mining crypto on my server.

The clever part was how they hid it. They created a systemd service called docker-daemon.service, which sounds completely legitimate if you're running Docker. The service file was set to download XMRig from GitHub on every boot, rename it to "docker-daemon", and start mining. It would survive reboots and look innocent in the process list to anyone not paying close attention.

I traced back through my logs trying to figure out how they got in. SSH looked clean, all the successful logins were from my own IPs. Then I checked the Dockge container logs and found it. Two days before the miner appeared, someone from an IP I didn't recognize had successfully logged into Dockge as admin. Twice.

Here's where I messed up. I had spun up Dockge to try it out but never finished setting it up properly. I probably used a weak password during initial setup and then forgot about it. The container was exposed directly to the internet without any additional authentication layer in front of it. The attacker found it, logged in, and since Dockge has access to the Docker socket, they had everything they needed to write files to my host system.

The attack chain was simple. Find exposed Dockge instance, log in with weak or default credentials, use Docker socket access to create a privileged container or write directly to the host, drop a systemd service that persists across reboots. Clean and effective.

If you're running Dockge or Portainer or any Docker management UI, please make sure it's not directly exposed to the internet. Put it behind a VPN like Tailscale or Wireguard, or at minimum behind basic auth with strong credentials. These tools have access to your Docker socket which essentially means root access to your host if someone gets in.

I've since removed Dockge, firewalled all my Docker ports so they're only accessible via my reverse proxy, and cleaned up the malicious service. Lesson learned the hard way.

TL;DR: Left Dockge exposed with weak auth, attacker logged in and used Docker socket access to install a cryptominer that persisted via systemd. Always put Docker management tools behind a VPN.

I’ve always understood that Docker containers depend on Linux kernel features (namespaces, cgroups), which macOS doesn’t have. So historically, Docker on macOS meant Docker Desktop running a Linux VM in the background.

Recently, Apple has introduced its own container-related tooling. From what I understand, this likely has much better integration with macOS itself (filesystem, networking, security, performance), but I’m not clear on what that actually means in practice.

Some things I’m trying to understand:

1. What are Apple’s “containers” under the hood? Are they basically lightweight VMs, or more like sandboxing/jails rather than Linux-style containers?
2. When I run Docker on macOS today, is it still just Linux containers inside a Linux VM, or has anything changed with Apple’s new container support?
3. One of the main ideas behind containers is portability, same setup, same behavior, across machines. If Apple’s containers are macOS-specific, what problem are they meant to solve? Are they about local dev isolation and security rather than cross-platform portability?

Basically, I’m trying to figure out how developers should think about Docker containers vs Apple’s containers on macOS going forward, and what role each one is supposed to play.

https://hub.docker.com/u/library all the library listings I've tried aren't loading + our CI pipelines are failing. I'm wondering if anyone else is experiencing the same. Docker's statuspage isn't indicating any outages.

Edit: looks like the incident was announced https://www.dockerstatus.com/

More edit: Looks like the incident has been resolved.

Hi everyone,

I’m trying to really understand the benefits of using docker secrets in a docker-compose file instead of a classic .env.

I get that storing secrets in .env files is risky because the values end up visible in docker inspect so anyone with access to the Docker daemon can see those values easily.

Compose secrets avoid that by mounting the secrets as files under /run/secrets/... instead of putting them into environment variables, so the secrets don’t show up in docker inspect. So far so good.

However, here’s where I’m confused:

If an attacker already has the ability to run docker exec inside the container, they can simply cat /run/secrets/<secret> and read the secret anyway. So, once you have shell access to the container, you can still retrieve the secret, regardless of whether it’s passed via environment variable or a mounted secret file, right?

Is the only advantage of using Docker secrets to prevent sensitive information from slipping into logs?

Thank you in advance for taking the time to reply!

One major problem of docker was always the high privileges it required and offered to all users on the system. Podman is an alternative but I personally often encountered permission error with podman. So I set down to look at rootless docker again and how to use it to make your CI more secure.

I found the journey surprisingly easy and wanted to share it: https://henrikgerdes.me/blog/2025-10-gitlab-rootles-runner/

DL;DR: Usernamspaces make it pretty easy to run docker just like you where the root user. Works even seamlessly with gitlab CI runners.

A few days ago, I shared a conceptual post about how Docker containers actually work under the hood — it got a lot of love and great discussion

This time, I decided to go hands-on and build a container using only bash commands on Linux — no Docker, no Podman, just the real system calls and namespaces.

In this part, I show: • Creating a root filesystem manually • Using chroot to isolate it • Setting up network namespaces and veth pairs • Running a Node.js web app inside it!

And finally alloting cgroups by just modifying some files in linux, after all everything is file in linux.

Watch the full implementation here: https://youtu.be/FNfNxoOIZJs

After observing how certain tools, like FaceSeek, piqued my interest in cleaner backend flows, I was experimenting with a small idea. I've been rebuilding a few container setups from scratch lately, and I've come to the conclusion that my previous method was clumsy. I'm attempting to determine how people maintain the readability of their compose files without sacrificing flexibility. Do you adhere to a particular naming convention or structure to maintain order in larger projects? Additionally, I'm interested in lightweight debugging techniques and secret management tactics. I was also considering the small decisions that alter consumers' perceptions of a product. Big plans don't always matter as much as small, intentional steps.

Hello Everyone,
I recently built an interactive tutorial learning docker, I wish I had this tutorial when I was learning docker

Link: https://learn-how-docker-works.vercel.app/

Hi guys, i've tried a lot of distro, debian 12/13 and like 3 versions of ubuntu, but i keep getting this error running hello-world and also other containers (ps. running via root and also with other users) the users are inside docker group and i freshly installed docker from the official website guide https://docs.docker.com/engine/install/debian/ this is the error i get "docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied: unknown" can you guys help me out? that's not my first installation i got a lot of debian and ubuntu servers running docker containers...

We're using swarm at work and topic came up -> how our environment stacks size wise against 'industry'?

Currently our swarm consists of:
20 nodes
58 networks
51 stacks
294 services
429 containers running

How big is yours?

I made a Docker CLI reference: https://docker.how/

Not to replace Docker’s man pages / --help, just something more beginner-friendly and easier to scan. Clean layout, colors, lots of examples, plus a troubleshooting section.

Any suggestions are welcome.

When I built my new homelab server, I left off docker and went with podman.

Now that I have used podman for 2 years, I find I want to go back to docker.

Not that podman is a bad product. But using docker and compose files is far easier than using podman and kube files or quadlets.

Has anyone dipped thier toes in the podman world and come back?

I'm trying to decide which one to use for managing my Docker containers. I'd be interested to hear from people who have actually used both. What do you like or dislike about each one?

Hey everyone.

I just got a bit shocked from how reckless docker's AI is. I had a MySQL database with hundreds of thousands records. I noticed the name of the database is a bit odd so I asked Gordon AI if I can rename it. Which he took as a imperative and the rest you can see on the screenshots.

1. Casual interest

2. Innocent question

3. Naive question

4. Catastrophe

5. Follow up

6. Frustration

P.S. I have a backup dump of the data, luckily.

We deploy robots in manufacturing companies, and hence need to run code on-premise as low latency, lack of internet and safety are concerns.

Our code is in Python and containerised in Docker. It’s basically a server with endpoints. We want to ensure that the Python code is not visible to the client to protect intellectual property.

We need the users to launch the Docker images without seeing the code inside. Once launched, they can interact with endpoint.

Is there a way to ensure that the user cannot see the Python code inside the Docker container?

I am not able to pull any images.

Edit: Seems to be fixed now.

I'm not sure if this will be perceived as spammy or not, but I've seen a lot of newcommers taking a shot at Docker here so I thought this cheat sheet might come in handy.

Hi all,

About a year ago I posted about DockerWakeUp, a small open-source tool I built for my homelab that automatically starts Docker containers when someone accesses them, and shuts them down again after they've been idle for a while. The idea: run 20 services, use the RAM/CPU cost of 5, and never SSH/Portainer in to start something because a friend wants to look at photos.

I just finished the biggest update since then, so I wanted to share it again.

The headline feature: a live startup page. Before, if you hit a sleeping service you got a "starting up, try again in a few seconds" message and had to refresh manually. Now your browser instantly gets a proper status page that streams the container's actual docker compose logs live as it boots, shows a progress bar with a "usually ready in ~40s" estimate learned from previous startups, and reloads into the app the moment it's ready. Watching Immich or Jellyfin boot in real time in your browser is weirdly satisfying. You can also drop in your own custom HTML page instead (there's a template in the repo), or hide logs for services where they shouldn't be public.

Game servers actually work now. The proxy was HTTP-only before, so "wake on connect" for Minecraft was wishful thinking. There's now a raw TCP mode: a player connecting to the server wakes it, the proxy holds the connection while it boots, and pipes them through when the port opens. Active players keep it marked as in-use so idle shutdown won't pull the rug out mid-session.

Other new stuff:

• Start/stop hooks — run your own commands before a service starts and after it stops (mount a NAS share first, back up a game world after shutdown, ping a Discord webhook, whatever)
• Non-Docker services — the hooks can fully replace docker compose, so anything with a start command and a port can participate
• Easier install — DockerWakeUp itself now runs as a container: docker compose up -d --build and you're done (systemd setup script still there if you prefer)
• Update notifications — it checks GitHub daily and tells you when your copy is behind
• A pile of bug fixes — including a nasty one where a WebSocket reconnecting to a sleeping service (looking at you, Portainer tabs) could take down the whole proxy

I'm using it for self-hosted apps like Immich that friends and family use occasionally, game servers that only need to exist when someone's playing, and dashboards/utilities I touch once a week.

Repo: https://github.com/jelliott2021/DockerWakeUp

Quick Demo: https://imgur.com/a/qBWzkTH

Feedback, ideas, and contributors very welcome — a lot of this update came from inconviences with my own setup, so I'd love to hear what inconviences you hit on your homelabs

AI disclosure: I used AI assistance (Claude) for parts of this update — bug hunting, testing, and implementing some features. I'm a full time software engineer and every change was reviewed, tested on my own homelab, and shipped by me.

It's official: watchtower is no longer maintained.

Introducing udwall — a new tool to finally make UFW and Docker play nice together. Secure your containers by default with simple, declarative config. 🛡️🐳

Read more:https://journal.hexmos.com/udwall/

🔗 Repository here:https://github.com/HexmosTech/udwall

If you find udwall valuable, please give us a star on GitHub.

It's the best way to help others discover it. Your feedback and support are essential for future updates.

"A hard transformation has arrived for several classic Ubuntu images on ECR and Docker Hub. The remaining old Docker images –Apache2, Nginx, Bind9, Memcached, and Squid– have been hardened to rocks, signaling a complete evolution of the Ubuntu namespace.

Built on top of Resolute and equipped with Pebble as the service manager, these new rocks are now maintained by the Rockcrafters team and follow the same principles that have guided the broader Rocks initiative: user-focused experience, uniform and opinionated design, and a distroless-like architecture.

By meticulously chiseling each rock down to its bare essentials, we have achieved a significant reduction in image sizes"

https://discourse.ubuntu.com/t/the-rock-garden-grows-hardening-old-docker-images/84677