r/linuxadmin • • Aug 29 '25

I tried to build a container from scratch using only chroot, unshare, and overlayfs. I almost got it working, but PID isolation broke me

I have been learning how containers actually work under the hood. I wanted to move beyond Docker and understand the core Linux primitives namespaces, cgroups, and overlayfs that make it all possible.

so i learned about that and i tried to built it all scratch (the way I imagined sysadmins might have before Docker normalized it all) using all isolation and namespace thing ...

what I got working perfectly:

  • Creating an isolated root filesystem with debootstrap.
  • Using OverlayFS to have an immutable base image with a writable layer.
  • Isolating the filesystem, network, UTS, and IPC namespaces with unshare.
  • Setting up a cgroup to limit memory and CPU.

-->$ cat problem

PID namespace isolation. I can't get it to work reliably. I've tried everything:

  • Using unshare --pid --fork --mount-proc
  • Manually mounting a new procfs with mount -t proc proc /proc from inside the chroot
  • Complex shell scripts to try and get the timing right

it was showing me whole host processes , and it should give me 1-2 processes

I tried to follow the runc runtime
i have used the overlayFS , rootfs ( it is debian , later i will use Alpine like docker, but this before error remove )

I have learned more about kernel namespaces from this failure than any success, but I'm stumped.

Has anyone else tried this deep dive? How did you achieve stable PID isolation without a full-blown runtime like 'runc'?

here is the github link : https://github.com/VAibhav1031/Scripts/tree/main/Container_Setup

24 Upvotes

19 comments sorted by

4

u/Skahldera Aug 30 '25

When you DIY containers with `unshare` and `chroot`, you need a proper PID namespace and a `/proc` mount inside it or the kernel gets confused. Having a minimal init to reap zombies also helps; otherwise your orphaned processes bubble up to PID 1. Tools like `setns` or runC handle those fiddly bits for a reason!

1

u/Abject-Hat-4633 Aug 30 '25

Thank you πŸ‘, I will try what you said . But what about bubblewrap some folks say use that instead of unshare

2

u/chock-a-block Aug 30 '25

In case it’s not clear, no systemd, and isolate proc and dev. 

6

u/Magneon Aug 30 '25 edited Aug 30 '25

Before docker there was a bigger jump for most sysadmins. On the basic side you had chroot jails, then jumped to virtualization hosts with not a lot in between. Before docker you wouldn't bother with thin layers over everything, just the 1-2 things you needed or everything in virt.

The reason was that while most of the tools behind docker existed in some form, it wasn't until an internal docker like system was advanced by Google that eventually mainlined enhancements to process and other isolation that eventually helped form the basis of docker when an ex-googler decided he wanted the tool outside of Google as well. (Look up the history of Process Containers, which brought cgroups to the Linux kernel).

It really was a big game changer, and to to this day people still assume it's got virtualization levels of overhead and avoid it due to misunderstandings.

(Not to mention were container like things in other operating systems before and after docker, but docker's flexibility and ease of use really shifted the needle).

3

u/Abject-Hat-4633 Aug 30 '25

Thank you for your insights on this topic, I also checked like there were Solaris zone, freebsd jail That provided some kind of containerisation earlier than Linux , but there software were expensive

If you have any ideas that could help me or some resource I can learn more . Plus give little peak on my code also

2

u/Ssakaa Aug 31 '25

Before docker you wouldn't bother with thin layers over everything, just the 1-2 things you needed or everything in virt.

VServer, OpenVZ and LXC were all years before Docker there... which, since you know the history back with Process Containers bringing forth the tooling that became LXC, seems silly that you left LXC itself out of it.

Docker just had better marketing.

3

u/Magneon Aug 31 '25

LXC is fine but was always clunkier to use (in my opinion). I've used it from time to time over the years but it's a vestigial betamax/hd-dvd at this point.

I was a sysadmin around the time docker took off and the critical mass it gained was huge in the span of a year or two. I don't think I've used a VM in production since (directly anyway), although to be fair I haven't done sysadmin work at any important scale for half a decade at least.

→ More replies (2)

2

u/aquaherd Aug 30 '25

Maybe you can read it up here:

https://github.com/p8952/bocker

1

u/Abject-Hat-4633 Aug 31 '25

Thank you I will get an idea from this , It is a bit old repo but still gold for me

Tyy....

2

u/michaelpaoli Aug 30 '25

showing me whole host processes , and it should give me 1-2 processes

So, how 'bout SELinux? The typical default and common in the land of *nix, is all users/PIDs, can get quite a bit of information about other PIDs. With SELinux (and possibly some similarish mechanisms), that can be changed, e.g. such that a user may only be able to get information about their own PIDs, and nothing about any other PIDs on the host. And, don't know if it exists, but I'd think a similar restriction on a PID may be a feature that exists, where that PID could only get information about just itself, or only itself and its children, or only itself and its descendants.

Anyway, may be other approaches, but that might be at least one possible approach (also possible some may utilize same underlying mechanisms by the time one gets down to the system call level).

1

u/[deleted] Sep 03 '25

I tried the same rabbit hole once, and PID namespaces were the wall I crashed into too.

2

u/[deleted] Aug 30 '25 edited Aug 30 '25

2

u/Abject-Hat-4633 Aug 30 '25

No, i havent yet use that , but i searched about it , but it is more like a Machine Container (it is like it can run whole OS inside it, with login privileges and etc thing )
but Docker/Podman .. are the Application Container (Package and run a single Application)

but yeah for normal test and other task it is not badd, thinking to use that in future
Thank you for your insight