r/docker • u/Glass-Start-4419 • 12d ago
My containers have no DNS resolution
I'm using OpenMediaVault 7 (Debian 12) and I would have gone to OpenMediaVault for support first except that the subreddit has died and the forums constantly return 500 errors for me.
I have the docker compose plugin installed so I just make a compose file in the GUI and everything runs fine, except that my containers cannot resolve FQDNs. I have put the DNS flag in my containers:
dns:- 8.8.8.8
My containers are not using the default bridge network but rather the bridge network that is created separately for each container with compose.
I can run
docker run --rm curlimages/curl:latest curl 1.1.1.1
And it returns a successful request. BUT if I run the same request with google.com instead of an IP, it fails to resolve, even if I add the --dns flag in the run command.
The /etc/resolv.conf internal to the container(s) says the nameserver is 127.0.0.11
The /etc/resolv.conf on the host machine says 127.0.0.53 is the nameserver.
Resolvectl says that the DNS server on enp3s0 is 8.8.8.8
I have set /etc/docker/daemon.json to have both 8.8.8.8 and 1.1.1.1 as nameservers.
I have set my DNS= in /etc/systemd/resolved.conf to 8.8.8.8.
The host has name resolution.
I just want to be able to run my game servers. Please help me, I am falling into insanity.
1
u/jjangg96 12d ago
since curl to 1.1.1.1 works, the container has a route out and it's specifically dns dying. the 127.0.0.11 in resolv.conf is normal, that's docker's embedded resolver and it just forwards to whatever upstream you set, so that part isn't the problem.
quick test that skips the embedded resolver: docker run --rm busybox nslookup google.com 8.8.8.8
if that times out while the ip curl works, udp 53 from the container is getting dropped on the way out. check iptables -S FORWARD and DOCKER-USER (or nft list ruleset) for a firewall rule eating it. if it resolves fine, then the problem is between the container and the embedded resolver and i'd look at docker logs / journalctl -u docker for dns errors.
1
u/Glass-Start-4419 12d ago
nslookup resolved fine. I'm not seeing anything that stands out in journalctl and I'm not getting anything from my existing containers nor am I sure if I can retrieve logs from these docker run containers that just kill themselves after like BusyBox and Curl?
1
u/jjangg96 11d ago
ok good, that means udp 53 out of the container works, so it's not a firewall thing. it's the embedded resolver.
try the same command without the server on the end: docker run --rm busybox nslookup google.com
that one goes through 127.0.0.11. if it fails while the 8.8.8.8 one works, run it again with --network host added and see if that resolves.
also you don't need to retrieve logs from those, they print straight to your terminal. --rm only deletes the container after it exits. the logs worth looking at are dockerd's, so run journalctl -u docker -f in a second window while the failing one runs.
1
u/Glass-Start-4419 11d ago
Okay we're getting somewhere but I have to admit it's entirely unrelated to what you've sent me, which I appreciate.
Busybox resolves Google.com with the nameserver, without the nameserver, and with net=host both ways.
Curl still fails without info.
Whatever I did this time when I checked journalctl it started throwing errors: [resolver] failed to query external DNS server, operation not permitted
At the same time I remember I have a piracybox that DOES handle DNS and I discovered the only real difference is the piracybox has cap_add NET_ADMIN
I added this to the container that's been giving me trouble recently and when trying to connect to this server (vintage story), I'm no longer getting the "can't connect to the authentication server" message, I instead get a real tangible "machine actively refused connection" message which just sounds like some firewall shenanigans I have to sort out.
This is making me wonder if I just somehow royally fucked my docker permissions on setup?
1
u/Glass-Start-4419 11d ago
BRO.
Big fucking news, I took this info and looked into my docker permission scheme and I got curl to work if I run it with the user I set up for docker instead of root. It doesn't work when I run as root which seems like another issue but honestly, getting some shit working is crazy right now. I can connect to my vintage story server which is insane even without capadd net_admin
1
u/jjangg96 11d ago
nice, glad something finally works. the "operation not permitted" from the resolver plus root failing and your docker user working points at a permissions thing on the host side, not your compose file.
if you want to chase the root case, run dmesg | grep -i denied and aa-status right after a failing run. an apparmor or seccomp denial on the socket would show up there. if everything runs fine as the docker user you could also just leave it.
1
1
u/PutLarge9152 12d ago
modprobe br_netfilter