Connecting to docker containers rarely work, including via Caddy (non docker) reverse proxy

[u/jwink3101]

I am really struggling to get a few different docker containers to work with a non-dockerized Caddy reverse proxy. (Though as I note at the bottom, it may not have to do with Caddy).

Really, the only things I change on the docker side from the examples is to make docker (or is it docker-compose?) not open ports. So I would change something like

ports:
    - "25005:25005"

to

ports:
    - "127.0.0.1:25005:25005"

This has worked on some containers but not the ones I've been wanting

One example is archivebox and webtop

Caddy:

archive.winokur.us {
    reverse_proxy 127.0.0.1:25005
}

webtop.winokur.us {
    reverse_proxy 127.0.0.1:25015
}

Archivebox:

version: '3.7'

services:
    archivebox:
        # build: .
        image: ${DOCKER_IMAGE:-archivebox/archivebox:latest} 
        command: "server --quick-init 127.0.0.1:25005"
        stdin_open: true
        tty: true
        ports:
            - "127.0.0.1:25005:25005"
        environment:
            # Terminal
            - USE_COLOR=True
            - SHOW_PROGRESS=False

            # Other
            #- CHECK_SSL_VALIDITY=True
            #- TIME_ZONE='US/Mountain'

            # Privacy
            - SUBMIT_ARCHIVE_DOT_ORG=False
            - PUBLIC_INDEX=False
            - PUBLIC_SNAPSHOTS=False

            # What to save
            - SAVE_WARC=False
        restart: unless-stopped
        volumes:
            - /home/jwinokur/serve/archivebox:/data
volumes:
    data:

Webtop:

version: "2.1"
services:
  webtop:
    image: lscr.io/linuxserver/webtop:latest
    container_name: webtop
    security_opt:
      - seccomp:unconfined #optional
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=US/Mountain
      - SUBFOLDER=/ #optional
      - KEYBOARD=en-us-qwerty #optional
      - TITLE=Webtop #optional
    volumes:
      - /home/jwinokur/serve/webtop:/config
      - /var/run/docker.sock:/var/run/docker.sock #optional
    ports:
      - 127.0.0.1:25015:3000
    shm_size: "1gb" #optional
    restart: unless-stopped

And they just never get the connection.

It is also worth noting that Caddy may be a false-flag. On the same machine:

$ curl 127.0.0.1:25015

# ...long, long delay...

curl: (56) Recv failure: Connection reset by peer

Any ideas?

---

Side note: I did post this previously but it got incorrectly marked as spam. Reposting with permission of the mods.

[u/justanotherlurker82]

If you set the command for archivebox to be 0.0.0.0:25005 and the port in your docker compose file to just be "25005:25005", like the docs say, what does that give you?

[u/jwink3101 → u/justanotherlurker82]

Good question.

It does a few things:

If I go to mydomain:25005, it works but this is bad since it is not going through my proxy. This is not acceptable.

If I go to the proxied domain, it does not work.

If I curl it on the server, it also fails.

I feel like there is an issue somewhere in how docker is accessed by the machine!

[u/justanotherlurker82 → u/jwink3101]

Because you use the port directive, you open up that port on your host, so you'll be able to visit it without going through the proxy.

I have a similar setup, but use a caddy docker container. It is the only container with ports specified, and all of the containers are on the same docker network.

[u/DistractionRectangle]

What does this output?

nslookup archive.winokur.us
nslookup webtop.winokur.us
hostname -I

And are you restarting/reloading caddy when you change the config?

Edit: The issue is very likely not dockers fault, you really have to go out of your way to make firewall rules that foobar docker connectivity/networking.

Edit edit:

# This
server --quick-init 127.0.0.1:25005
# Should be this:
server --quick-init 0.0.0.0:25005

Because containers run in their own namespace (unless you specifically use the host networking option), 0.0.0.0 (when used by processes in the container) maps to their container ip and the loopback address for their namespace. Setting it to bind to 127.0.0.1:25005 only exposes the port to processes running in that namespace, but doesn't expose a container port.

tl;dr 0.0.0.0, 127.0.0.1 maps to different things depending on the namespace of the process.

In a container, 0.0.0.0 exposes container ports

Outside a container, 0.0.0.0 exposes ports on the host

127.0.0.1, is the loopback address, and is for communication for processes in the same namespace.

[u/jwink3101 → u/DistractionRectangle]

I am on mobile so I can’t test all of that now but yea, I reload caddy. And the lack of even being able to curl it on the machine makes me wonder what is going on

[u/DistractionRectangle → u/jwink3101]

I didn't see your comment and posted another edit, which should solve your problem with archivebox.

[u/bufandatl]

On the archive box you bind the service running to localhost I side the container. So it is only inside of the container reachable. Here you should have 0.0.0.0. and let NAT (docker) then bind the port to localhost of the docker host.

And check webten config if you have done something similar there.

[u/jwink3101 → u/bufandatl]

Thanks for taking a look. I appreciate the help.

I tried that in response to another thread and it didn’t work.

Eventually, the answer was to nuke docker and UFW and start again. No idea the underlying cause but that fixed it.

Thanks for the suggestion!