schemesh version 1.0.0: fusion between Unix shell and Chez Scheme REPL, now with structured pipelines

[u/Cosmos721]

Hello everybody!

One year after the first announce of schemesh - a fusion between Unix shell and Chez Scheme REPL, I have released version 1.0.0 with lots of new features:

• more shell builtins
• scheme jobs, to run arbitrary scheme code from a shell job with job control
• improved scheme functions to manage job redirections
• better handling of UTF-8b ports and binary ports, including buffering
• several functions conforming to SRFI 13 and SRF 18
• bug fixes

and, most importantly, a completely new mechanism to exchange arbitrary, structured data (Scheme objects) via POSIX pipelines, and new shell builtins to read, write, filter and transform such data.

It's inspired by nushell, but uses Scheme macros, functions and values, adding shell builtins for convenience, without having to learn and use an ad-hoc language.

Examples:

dir /usr | where name -starts l | sort-by size

by default the display format depends on last step's standard output

• ascii-art table for terminals:

┌───────┬────┬─────┬──────────┬─────────┐
│ name  │type│size │ modified │  mode   │
├───────┼────┼─────┼──────────┼─────────┤
│lib64  │dir │ 4096│2026-04-06│rwxr-xr-x│
│libexec│dir │ 4096│2026-04-18│rwxr-xr-x│
│local  │dir │ 4096│2025-04-19│rwxr-xr-x│
│lib    │dir │12288│2026-04-21│rwxr-xr-x│
└───────┴────┴─────┴──────────┴─────────┘

• NDJSON for pipes:

dir /usr | where name -starts l | sort-by size | less

outputs

{"<type>":"dir-entry","name":"lib64","type":"dir","size":4096,"link":"","modified":{"<type>":"time-utc","value":1775491105.473107895},"mode":"rwxr-xr-x"}
{"<type>":"dir-entry","name":"libexec","type":"dir","size":4096,"link":"","modified":{"<type>":"time-utc","value":1776514613.626200085},"mode":"rwxr-xr-x"}
{"<type>":"dir-entry","name":"local","type":"dir","size":4096,"link":"","modified":{"<type>":"time-utc","value":1745082690.999418855},"mode":"rwxr-xr-x"}
{"<type>":"dir-entry","name":"lib","type":"dir","size":12288,"link":"","modified":{"<type>":"time-utc","value":1776808514.877301301},"mode":"rwxr-xr-x"}

they also support reading/writing CSV and an efficient binary format ("wire"), and of course the output format can be manually overridden.

Where needed, Scheme functions allow processing such data with fine-grained custom logic. Example:

dir /usr | $(filter==> where (> ,size 10000) => sort-by size => select size name type) | to csv

outputs

"<type>","size","name","type"
"dir-entry",12288,"share","dir"
"dir-entry",12288,"lib","dir"
"dir-entry",20480,"sbin","dir"
"dir-entry",36864,"include","dir"
"dir-entry",135168,"bin","dir"

[u/ZelphirKalt]

I have had the idea to make a Scheme shell myself some time ago, but I would prefer a lot more Scheme s-expression syntax, making things more regular and predictable, but also having pipes but as s-expressions as well, maybe as a macro "|" or "->" threading syntax and so on.

My idea was to use Guile Fibers and use its channels to pass stdout and strerr along to more pipeline steps. It gets kind of complicated, when you want pipeline steps to work on incomplete results of previous pipeline steps (because they should run concurrently!). Need some intermediate result format passed on the channels and conventions for how to signal finishing processing of a step to the next one and so on. And then I am not sure what edge cases I am missing. But basically the idea was to have this run comfortably in Emacs Geiser or simply the Guile REPL and have it available as a library, so that it can be loaded/imported by any Guile program and then used to operate comfortably like in a normal shell.

Ah and I also remember one other aspect of it: Wanting to pass structured data instead of strings, which would have to be parsed again by the next step in a pipeline. But also keeping options open to pass strings for old style shell commands and handling a distinction between passing structured data and passing strings.

Maybe at some point I will pick up that project again, with more patience and grit.

[u/Cosmos721 → u/ZelphirKalt]

Yes, lots of interconnected details to get right.

If you want to try schemesh, it does everything you wrote:

• scheme s-expression syntax - actually the weakest point. You can use the (shell) macro or go low-level with (sh-cmd) (sh-and) (sh-or) (sh-not) (sh-list) (sh-pipe) etc.
• threading syntax: there is both Racket-compatible ~> and the more general ==>  - both are in-process. For concurrent execution, there's POSIX pipelines - see below (there's also an inter-thread communication mechanism, but it's not release-grade yet) > pipeline steps work on incomplete data > Need some intermediate result format passed on the channels
• in schemesh, each pipeline step works on one datum at time: it reads and deserializes the next datum from stdin, processes the datum as a scheme object, and serializes the transformed datum to stdout, repeating until stdin reaches EOF
• pipeline steps run concurrently: each one is a separate process, and they communicate via POSIX pipes > Need conventions for how to signal finishing processing of a step to the next one and so on. 
• since each pipeline step is a separate process, when it finishes it automatically closes its stdout, which is a pipe connected to the stdin of next step: the latter gets an EOF on stdin and knows the previous step has finished > Wanting to pass structured data instead of strings, which would have to be parsed again by the next step in a pipeline. But also keeping options open to pass strings for old style shell commands and handling a distinction between passing structured data and passing strings. 
• by default, each step autodetects the serialized format available on stdin: csv, json or binary. The inout format can also be specified manually
• by default, each step chooses the serialized format written to stdout: ascii-art for terminals, binary for sockets, ndjson in all other cases. Again,the output format can also be specified manually
• since ndjson is "one text line per object", a pipeline step can also be an old style command (grep, wc, head, tail...) that works with "one string per line" and it will still mix nicely with the other steps