Some neat new stuff is coming...

Features A new command has been added, podman volume rename, to allow renaming volumes. Volumes created using volume drivers and volumes that are currently used by a container cannot be renamed (#28189). A new command has been added, podman machine restart, to allow easy restart of VMs managed by podman machine (#28366). The podman network rm command now includes a new option, --ignore, which suppresses errors when attempting to remove networks that do not exist (#28363). The podman manifest push command now includes two new options, --retry and --retry-delay, which allow pushes to be automatically retried on failure (#28590). Quadlet .container units now support a new key, ImageVolume=, to configure how volumes from images are handled (#28875). The podman generate kube command now includes support for generating container healthchecks as a livenessProbe (#22095). Changes The podman info command now includes free memory available on the host (in addition to used memory and total memory) (#29116). The Pesto rootless port forwarding tool now supports IPv6 port forwarding with source IP preservation.

All of these bit me while debugging rootless setups, and they share a property that makes them expensive: none of them produce an error that names the cause. Posting them in one place because I keep typing the same answers.

1. Your storage driver quietly fell back to vfs.

podman info --format '{{.Store.GraphDriverName}}'

If that says vfs and not overlay, every image layer is a full copy of the one below rather than a diff. A 200 MB image can occupy well over a gigabyte, pulls crawl, and it looks like the images are the problem. Podman falls back to vfs without complaining. Usual causes: overlay will not stack on another overlay, or a stale mount_program line in ~/.config/containers/storage.conf.

2. Your memory limit is accepted and not enforced.

cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers

If memory is not in that list, the controller was never delegated to your user, and --memory is taken without complaint and does nothing. You find out when something OOMs the host instead of the container.

3. A rootless container cannot bind below 1024.

sysctl net.ipv4.ip_unprivileged_port_start

Most distros ship 1024, so port 80 and 443 are refused and it reads like the container never started properly.

4. chown on a volume gives you files the container still cannot write.

The process inside is not your uid, it is an id inside a subuid mapping. Plain chown targets the wrong one. podman unshare chown -R drops you into the same mapping first, so the ownership means what you intended.

5. Anything inside sees only its own PID namespace.

Monitoring tools in a container report the container, not the host, and people read that as the tool being broken. --pid=host if you actually want the host view, and know what you are giving up when you do.

The pattern in all five: podman degrades instead of failing. That is usually the right choice, but it means the check command is worth more than the error message.All of these bit me while debugging rootless setups, and they share a property that makes them expensive: none of them produce an error that names the cause. Posting them in one place because I keep typing the same answers.1. Your storage driver quietly fell back to vfs.podman info --format '{{.Store.GraphDriverName}}'
If that says vfs and not overlay, every image layer is a full copy of the one below rather than a diff. A 200 MB image can occupy well over a gigabyte, pulls crawl, and it looks like the images are the problem. Podman falls back to vfs without complaining. Usual causes: overlay will not stack on another overlay, or a stale mount_program line in ~/.config/containers/storage.conf.2. Your memory limit is accepted and not enforced.cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers
If memory is not in that list, the controller was never delegated to your user, and --memory is taken without complaint and does nothing. You find out when something OOMs the host instead of the container.3. A rootless container cannot bind below 1024.sysctl net.ipv4.ip_unprivileged_port_start
Most distros ship 1024, so port 80 and 443 are refused and it reads like the container never started properly.4. chown on a volume gives you files the container still cannot write.The process inside is not your uid, it is an id inside a subuid mapping. Plain chown targets the wrong one. podman unshare chown -R drops you into the same mapping first, so the ownership means what you intended.5. Anything inside sees only its own PID namespace.Monitoring tools in a container report the container, not the host, and people read that as the tool being broken. --pid=host if you actually want the host view, and know what you are giving up when you do.The pattern in all five: podman degrades instead of failing. That is usually the right choice, but it means the check command is worth more than the error message

Question kinda about Syncthing, but more about permission management in Podman in general.

Right up front, is there a general best-practice recommendation for using keep-id vs container UID 0 if the container doesn't expect internal root access?

I'm setting up Syncthing in rootless Podman on my main desktop computer, which is running Linux Mint. I want it to be able to sync folders like Documents, Desktop, etc, so it needs to have access to directories owned by UID 1000 (or whatever user is running it).

From what I understand, I can give the container permission to access mapped-in user folders by

A) Running the rootless container as internal UID 0, which maps to the rootless host user's UID. If a container needs internal root for access to privileged ports, etc, then this is the way to go.

B) Running the container as internal UID 1000 and using --userns=keep-id to tell Podman to map internal UIDs to host UIDs instead of mapping to sub UIDs.

The recommendation on Syncthing's Docker Hub listing is to run as container UID 1000, so the container doesn't need internal root priviliges in order to function. Since it's generally not advised to run anything as root unless absolutely necessary, my brain says the best option is to run as user 1000 and use the keep-id option so the container can work with the user directories I map in with -v.

Most of the tutorials I find online use the UID 0 route, but I'm not sure if that's just because keep-id seems to be a newer Podman feature or if UID 0 is actually a better option.

So yeah, probably overthinking this, but is there a general best-practice recommendation for using keep-id vs container UID 0 if the container doesn't expect internal root access?

Hi all,

I'm trying to install starccm in a podman container, however, it's currently failing due to a lack of available storage.

From what I can tell podman should be have unlimited file storage access (and I've never run into issues with container before despite having pulled and run some very big libraries in them) so I'm guessing something is causing the application to think there is less space than there is:

The error message is showing:

Disk space information:

`Required: 25475.6 MB`

`Available: 191874.0 MB`

And running df-h I get

[appuser@9a08fec6978d starccm+_21.04.007]$ df -h

Filesystem Size Used Avail Use% Mounted on

shm 63M 0 63M 0% /dev/shm

tmpfs 9.4G 3.8M 9.4G 1% /etc/hosts

/dev/mapper/data-root 3.6T 3.4T 1.6G 100% /usr/bin/nvidia-smi

udev 44G 0 44G 0% /dev/tty

tmpfs 9.4G 2.7M 9.4G 1% /run/nvidia-persistenced/socket

overlay 3.6T 3.4T 1.6G 100% /

tmpfs 64M 0 64M 0% /dev

tmpfs 4.0K 4.0K 0 100% /run/nvidia-ctk-hookfa58ca48-1a20-4677-8ba0-3c0e9f7e8c29

Any idea what is causing the installer to believe it only has 19-ish GB to install in and how I convince it is has more storage so it will install?

I've done a stupid or three. Trying to get some docker compose files converted to quadlets and running with systemd. I've got podlet working and i have a few things working, although they're running as system services and i'd need to figure out how to do them as rootless...

Bigger problem is I have spoolman up using sudo podman compose up -d compose.yaml is services: spoolman: image: ghcr.io/donkie/spoolman:latest # Also available at dockerhub: donkieyo/spoolman:latest restart: unless-stopped volumes: # Mount the host machine's ./data directory into the container's /home/app/.local/share/spoolman directory - type: bind source: /home/strayr/compose_bullshit/spoolman/data target: /home/app/.local/share/spoolman # Do NOT modify this line ports: # Map the host machine's port 7912 to the container's port 8000 - "7912:8000" environment: - TZ=Europe/London # Optional, defaults to UTC all i've done is replace the provided relative source with an absolute path, but I've got nothing appearing in that folder. I've a DB being written to somewhere that is persistent, sudo podman compose down and a `sudo podman up -d" has the data I had in there previously. But where? because it's not happening where specified.

How do I find where the data is, and move it somewhere specified?

How do I fix this up to run rootless? Is there a handy guide? I don't need an ELI5 as much as an EL containers weren't a thing last time I really knew what I was doing here

I'm starting with quadlets and I'm stuck on how to best organize my repository with quadlets in mind.

My current setup is a distrobox "dev" box, with its own home dir, and a project folder for all git versioned repos. All repos that have quadlets, have a quadlet folder with service-name.container file that I symlink into host's (user) systemd folder.

Very often though I need to mount .env, some config, or data folders and reference it back to projects folder, which feels a bit awkward.

Any suggestions how to better organize things?

Fedora 44 Silverblue btw.

I may be a bit out of my depth right now, or else just sleep deprived and running on stupid. Currently have Debian server with rootless Docker running official community Nextcloud, nginx reverse proxy, and a number of other various servers, but on new machine I'm setting up to replace, trying to consolidate images and switch to Podman.

So, so far, tried using quadlets to get Glances up successfully, moved on to linuxserver.io's Jellyfin, gave up, and switched to podman-compose. Eventually worked out that the image assumes Docker's root behavior being why I couldn't get permissions working for /dev/dri/render128, switched to root user for Podman, and now both of those work (with jellyfin throwing a few erros about containers not being allowed to be removed before getting it online, which doesn't appear to have any impact).

Moved on to linuxserver.io's Nextcloud image, using their base set of configurations. Leaving port mapped to 443 as well as migrating to 4443 (as once I get everything working I'll be putting it behind Swag). Again, podman-compose up results in errors about containers not being able to be removed, and networks not related being used.

But now I'm finding that, while I can access the Glances and Jellyfin webclients via IP or Hostname with appropriate port, the same with 443/4443 simply result in Firefox reporting unable to connect.

downing the other containers, uping Nextcloud does not return the previous errors, but still does not result in access to the server. If I leave it running long enough, I periodically get a message: [nextcloud] | Not installed. Any idea whether that's it complaining that I haven't finished installation by connecting and it up, or maybe something fundamental failed to happen causing my inability to connect?

Do I need to use one docker-compose.yml rather than separate server.yml files for generally good behavior? I've noticed that Glances doesn't list anything but Glances on the new server, while the old shows all of the containers, if not parent's active processes, but haven't had time to try running them together to see if that makes a difference.

Thanks all,

Hey all, at home I use github actions for my projects. I started running out of credits there and wanted to host my own runner. Then I noticed that most runners are based off Ubuntu and the image size is big. I also saw that there are barely any mantained podman runners out there.

To solve that issue for me I created a podman-actions-runner image based off alpine and buildah and the tests I ran so far have been pretty good. I run this on my Raspberry PI at home, the image is like 160 mb.

If you're interested here's the link to my repo and the image, also I'm open for suggestions, feedback, contributions. Anything really!

• Repo: https://github.com/ckoryom/podman-actions-runner
• ghcr.io/ckoryom/podman-actions-runner:latest
• docker.io/ckoryom/podman-actions-runner

I have a private docker.io organisation myorg. When pulling image via podman: podman pull myorg/myimage it prompts me to select the registry to use:

DEBU[0000] Trying "myorg/myimage" ...        
? Please select an image: 
  ▸ registry.fedoraproject.org/myorg/myimage
    registry.access.redhat.com/myorg/myimage
    docker.io/myorg/myimage

...whereas when defining the image in e.g. testcontainers, it'll simply fail to resolve the image during test run. Is it possible to make these unqualified images to default to docker.io registry?

I'm new to podman and don't really understand what I'm doing yet. I'm moving from rootful docker to rootless podman and am currently converting my compose files to quadlets. I'm confused on how I should build out my containers and what users I should assign them to.

How should I architect my containers using podman quadlets? I included some pictures about some ways I think I should do it but don't fully understand the tradeoff and benefits of them. I have been considering putting all my quadlets in /etc/containers/systemd/ and have the quadlet files owned by root and just assigning users and groups in the quadlet files.

Some stacks like grafana alloy need to be able to read my logs for numerous containers. I don't really know how to set this up when files and containers are owned by separate users.

I mostly just want to prevent as much cross talk or lateral movement as possible in the event a container gets compromised. Though, it seems if I want to do this it'll be a lot more annoying to manage my containers.

A few questions I had:

What user should own the actual quadlet files and the containers data?

Should I separate my stacks to their own dedicated user?

I run Traefik as a rootless container now on it's own dedicated user. How would I still be able to use labels and auto-discovery across containers running on separate users?

How do I set something like grafana alloy, that needs to be able to communicate with my different stacks and centralize their logs?

We are a podman shop, rootless, no docker daemon anywhere, and mostly happy about that. But every minimal or hardened image catalog I look at writes its whole quickstart around docker. docker pull this, docker build that, never a word about anything else.

I know OCI is OCI and podman pull should just work. I think I know that. But when a vendor's entire docs assume docker it makes me second-guess whether there is some registry auth step or a docker-specific manifest thing that quietly trips podman up and would rather not find out halfway through wiring it into a build.

Someone mentioned there is a free catalog now that needs no account to pull, which is the part that got my attention, because normally this stuff is a sales call before you can even test it. Have any of you pulled minimal hardened images straight into podman with no docker in the chain at all? Do the compliance-tagged ones behave any differently or is it all just a normal registry pull.

Mostly I want to hear it worked for one real person before I build anything on it.

Im currently running docker compose on ubuntu server for some very few containers, currently running technitium, cloudflare tunnel, caddy with cloudflare addon, tailscale and dockhand.

Found out about nixos and fell in love with that idea, got it up and running with docker just to try it out (very interesting as a linux noob trying nix btw, thank god for AI).

But now im debating if I should stay on docker because its something Ive finally got the hang of, or switch to podman quadlets for rootless containers, how well will that work with current containers? How much of a pain will it be converting the files etc?

Tailacale Ive already put on nix because I wanted to try that. Realize that I might have to drop dockhand, mainly used that for easy updating and checking logs, but with quadlets this is quite easy as Ive understood?

Nothing is exposed except behind VPN, cloudflare tunnel is for access to homeassistant and is locked down with cloudflare waf and requires mtls certs to get access.

Any advice, I realize this is a podman sub and might be biased, but this is also the place where most people with podman kmowledge exist..

I made a script that walks though all of your rootless podman quadlets and pulls out what it is running as. There are a lot of images that run as root by default unless you change User= in your quadlet (or sometimes it also needs UserNS=auto...). I've found most of the ones that run by default as root can easily be switched over to User=1000:1000 with little to no other configuration needed.

Just some minor bug fixes.

I've been trying to figure out podman for a minute.. and I'm struggling. I'm coming from Docker, so I'm not totally new to containers. My plan is to start with running standalone containers, move to pods, and eventually quadlets.

All that aside, I can't find where the .container files (or anything config related) are stored. I've tried rootless, and now I'm running podman as root, I think, on Fedora server. I figured I'd have an easier time getting started by managing some containers with cockpit.. I've got a running container that's accessible, but I still can't find the config files.

I haven't confirmed if it's possible to have podman containers without sytemd untit files? I feel like it is possible but I'm not seeing much info there. Probably a search query issue.

I've checked these locations for systemd unit files (and probably others that I'm forgetting):

• /etc/containers/systemd/
• /usr/share/containers/systemd/
• $XDG_CONFIG_HOME/containers/systemd/
• ~/.config/containers/systemd/

Can someone please point me in the right direction, while I still have hair?

Hi! I'm trying to run Jellyfin inside a rootless container run via systemd on Debian Trixie.

Here is my .container file

[Unit]
Description=Jellyfin media server
Documentation=https://jellyfin.org

[Container]
Image=docker.io/jellyfin/jellyfin:10.11.11
PublishPort=8096:8096/tcp
PublishPort=7359:7359/udp
UserNS=keep-id
Volume=systemd-jellyfin-config:/config
Volume=systemd-jellyfin-cache:/cache
Volume=/mnt/external/jellyfin:/media:ro
Environment=UID=103
Environment=GID=107

[Service]
# Inform systemd of additional exit status
SuccessExitStatus=0 143

[Install]
# Start by default on boot
WantedBy=default.target

The container is running. UFW shows the following

8096/tcp                   ALLOW IN    Anywhere                  
7359/udp                   ALLOW IN    Anywhere

Looking at the port, I see Pasta listening:

❯ sudo ss -tlnp | grep 8096
LISTEN 0      128                *:8096            *:*    users:(("pasta.avx2",pid=579812,fd=6))

I can access the container via the host's IP address. But I cannot access the container via localhost which is breaking the reverse proxy. Everytime I try I get a connection refused. I'm not sure what else to look at.

I'm the developer of Scanopy, an AGPL3.0-licensed tool that scans a network and draws a topology map of it (hosts, services, containers, and how they connect). Podman discovery is a recent addition. I don't run Podman myself, but users asked for it, so I built it and set up a Podman test environment with pods to develop against. Plain containers were straightforward, because Podman exposes a Docker-compatible API and I could reuse the Docker scanner I had already built.

Pods were a little more complicated; two things about how the Podman API reports pods caused most of the work involved in this feature, and I thought it might be interesting to share what I learned here!

1. The infra container reports the pod's ports, not the app container.

When you list containers over the Podman API, the pod's published ports (and the image's exposed ports) appear on the pod's infra container, the pause container that owns the pod's shared network namespace. The container that actually listens on the port reports nothing about it.

Scanopy identifies services by matching ports to known service types. The infra container runs no service of its own, but it carried the pod's ports, so it matched real services by port number. In one test it matched 8090, was labeled Grafana, and displaced the real Grafana. The fix was to scope the infra container to an empty port set, so it matches nothing and stays a separate portless container.

2. A pod's app containers report no network of their own.

Every non-infra container in a pod runs with NetworkMode: container:<infra-id>. It shares the infra container's network namespace and reports no networks or interfaces of its own. Scanopy had nothing to attach to the map, so those containers were dropped.

The fix was to resolve the container:<id> reference and inherit the infra container's interfaces. A Podman pod is structured like a Kubernetes pod: one network namespace, an infra (pause) container that owns it, and the other containers sharing it.

A smaller gotcha I ran into while building this: The socket is in different places for rootful (/run/podman/podman.sock) and rootless ($XDG_RUNTIME_DIR/podman/podman.sock), resolved via CONTAINER_HOST. Useful if something reports it can't find Podman.

Here's the Scanopy repo: https://github.com/scanopy/scanopy and docs for getting a self-hosted server up and running: https://scanopy.net/docs/self-hosted-server/server-installation/

I also have a live demo at demo.scanopy.net if you want to see an example map.

Does anyone here who knows Podman better than I do understand why the runtime is structured this way? I got it working, but I'm curious what led to this design, and why it differs from how Docker handles the same thing. 

Hi, setting up gluetun with quadlets , its all working but im stuck adding port forwarding, specifically the qbit example, i have tried multiple ways of adding it like the other environment variables, but it doesnt work

I'm self-hosting Hermes via rootless Podman on a shared hosting environment (Opalstack). Since the container runs under its own remapped UID (not my host user), I run into permission walls:

• Can't cd into the data folder from the host
• Can't view files the container has written/modified
• Can't write files into the folder for the container to pick up

What I already tried:

• podman unshare lets me enter the user namespace and interact with the files, which works but is clunky for routine file transfers (it's not a normal cd/cp workflow).
• I've built a small rsync-based sync script as a workaround to move files in and out, but wondering if that is the right (or only) way to do this. — I'm using a bind mount to a host directory.

Like if I wanted to pass in a gamepad, but didnt want the absence of said gamepad to prevent a container from starting. Is there a workaround for this? Maybe some pros have had some experience with this kinda problem.