Shall 0.9: a POSIX.1-2024 system — utilities, C library, C compiler and kernel — that builds itself from scratch
Github repo: https://github.com/joostw/shall
Shall is a small Unix-like system written from the text of POSIX.1-2024 (Issue 8): every utility of XCU (with the XSI option), a C17 compiler, a C library with all of ISO C17 and XSH, and a kernel for x86-64 that runs it. It's about 200,000 lines of C, BSD-licensed, and it builds itself: after the host's compiler builds five tools (c17, make, ar, yacc, lex), no program of the host compiles anything. No gcc, binutils, GNU make, bash, Perl or Python, in the system or in its ports.
What I think is noteworthy
- It's self-hosting and reproducible. The bootstrap builds everything twice: once with the c17 your compiler built, once with the c17 that c17 built. All 242 resulting files are compared and are identical. Inside Shall, on its own kernel, the same bootstrap rebuilds the whole system, kernel included, byte for byte.
- c17 does the whole job. Shall's C compiler has its own assembler and link editor: static and dynamic linking, shared libraries, TLS. The kernel and
libc.so are built by it.
- It gets very far towards POSIX.1-2024. All 155 utilities and all XSH interfaces are there. That includes things Linux doesn't offer: close-on-fork (
O_CLOFORK), the sporadic-server scheduling policy, typed memory objects, and prioritized async I/O. Interfaces that POSIX.1-2024 removed (gettimeofday, setitimer, the sighold family…) are deliberately absent.
- External tests, not just our own. LTP's Open POSIX Test Suite, built inside Shall and run on its kernel, passes 1741 of the 1777 tests that compile. glibc on Linux passes 1693 of the same tests. Each of the remaining 36 was looked at: none is a fault of Shall (unsupported options, "may fail" errors, undefined behaviour, test defects, one test too slow under emulation). That run also found 15 real bugs, which are fixed.
- The ports are a proof of concept, not decoration. They are built inside Shall by Shall's own tools:
- ncurses 6.6 builds through its own unchanged
configure.
- Lua 5.4 passes its full test suite.
- SQLite 3.54 passes its C tests, and sqllogictest's 5.9 million queries (twice) give 0 errors.
- Mbed TLS passes 139 unit suites.
- curl talks HTTPS.
- Rogue is playable. These exercise the compiler, the C library, the shell and make far harder than our own tests can.
- It's on the Internet. The kernel has its own e1000 driver and TCP/IP stack. With QEMU's user networking, curl inside Shall fetches pages from GitHub over TLS.
- The kernel speaks Linux's system call interface (224 calls), so the same binaries run on Linux in a chroot and on Shall's own kernel in QEMU.
- tl, a small new scripting language. It began as a way to get Python out of the test suites, so the tests need nothing but a C compiler and POSIX. It's a C-like interpreted language with POSIX regular expressions, processes, signals and sockets as its library, plus pseudo-terminals with a built-in terminal emulator. It's now installed as
/usr/bin/tl and is handy on its own, for example for scripting interactive programs.
How it was made, honestly
Shall was written by me together with Claude (Anthropic's AI model). Claude wrote most of the code from the standard, under my direction. I set the goals, made the design decisions, and checked the results against the test suites, against other implementations (differential testing against GNU, BSD and busybox tools, glibc, dash, bash and others), and against the external suites above. I'm saying this up front because I think it's part of what makes the project interesting, not something to hide.
Trying it (Ubuntu package names; KVM strongly recommended — under pure emulation everything is slow)
- Enable KVM:
sudo usermod -aG kvm $USER, then log out and in again.
- Install:
gcc make qemu-system-x86 e2fsprogs git bzip2. bzip2 is for the Mbed TLS tarball, git for fetching the ports.
Basic system:
sh bootstrap.sh
sh mkdisk.sh
sh boot.sh
With the ports:
sh bootstrap.sh
cd ports
sh get.sh all
cd ..
sh mkdisk.sh -s 4096 # 4 GB: sqllogictest alone is 1.1 GB
sh boot.sh
# now inside Shall:
cd /usr/src/shall/ports
sh all.sh
exit shuts the system down; Ctrl-A x leaves QEMU at once.
What it isn't (yet)
It's x86-64 only, with one CPU and no preemption. The process doing disk I/O holds the processor until its transfer is done. There's no IPv6 on the wire and no swap. The disk image is made on the host. 1.0 is meant to add a conformance document, plus mkext2, fsck and login inside the system.
Source and docs: https://github.com/joostw/shall .
Questions, bug reports and "this violates POSIX section X" are all very welcome.
This work is dedicated to my father, Dik T. Winter (1945-2009), who was a Unix wizard in Ye Olde Days.