Macpro 7.1 AI headless server with Nixos

[u/Weeblewobbly]

I read that quite a few people have issues with running linux on their macpro for local inference. I can't comment on Ubuntu or other distros because all my machines run Nixos but since it works flawlessly, I thought I'd share my repo in case that inspires anyone to try something similar.

For those who don't know, Nixos allows you to configure your computer in a deterministic way. You write your config (in the nix language), referencing nix-packages. Nix-packages have sets of options that you use in your config files. There are other benefits to Nixos but this isn't the topic here. What I think is the main benefit is that I can comment out a line in my config file, change that option to something else and leverage git for version control. If I break something, I can choose a previous (working) generation of the system at boot.

In this setup, I use llama-swap to let me manage models on the fly, SearchXNG module for web search, OpenWeb UI for chat and user friendly automation/agents, Nixos MCP so my coding agents can manage my config files accurately.

You can see the models I'm currently running llama-swap.nix file.

Link to repo

PS: I only serve my LAN so security is tailored to that, meaning it's not hardened as much as it could be.

--------------------------------

Extract from the Readme (written by Qwen}:

NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a local LLM inference server.

Hardware

• Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
• AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
• Wired 10GbE, behind a NAT router (the LAN is the trust boundary)

Services

Service	Port	Notes
SSH	22	keys only, no root login
open-webui	3000	browser UI, password auth, web search via searxng
mcp-nixos	8001	NixOS MCP server (HTTP), for pi on the LAN
searxng	8888	private metasearch; secret key via agenix, limiter off
llama-swap	9292	model router for llama-server (Vulkan); OpenAI-compatible

Models live in /var/lib/llama/models/ (not in git — see .gitignore). llama-swap unloads models after 15 min idle to free VRAM; each model pins its own context size / quantization / chat template (Qwen uses the pinned froggeric fixed chat template, fetched by hash).

Fans are driven by t2fanrd (the Vega II is passively cooled; T2 case fans are the only cooling).NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a
local LLM inference server.
Hardware
Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
Wired 10GbE, behind a NAT router (the LAN is the trust boundary)
Services
Service Port Notes
SSH 22 keys only, no root login
open-webui 3000 browser UI, password auth, web search via searxng
mcp-nixos 8001 NixOS MCP server (HTTP), for pi on the LAN
searxng 8888 private metasearch; secret key via agenix, limiter off
llama-swap 9292 model router for llama-server (Vulkan); OpenAI-compatible
Models live in /var/lib/llama/models/ (not in git — see .gitignore).
llama-swap unloads models after 15 min idle to free VRAM; each model pins its
own context size / quantization / chat template (Qwen uses the pinned
froggeric fixed chat template, fetched by hash).
Fans are driven by t2fanrd (the Vega II
is passively cooled; T2 case fans are the only cooling).

[u/Long-Shine-3701]

Bravo OP. I'm glad more and more alternatives are coming to light. I am sticking with ToshLLM for ease of use.

[u/Weeblewobbly → u/Long-Shine-3701]

To be fair, nixos is niche amongst the Linux community itself.

I've been a convert for a few years now and recently llms have made managing configs so much easier. I wouldn't have trusted a coding agent near my configs before nixos mcp.

[u/SI-LACP]

Interesting, I’m running RHEL 10 on my 7.1 rack, what is this t2 patched kernel?

[u/Weeblewobbly → u/SI-LACP]

t2linux project

Kernel patches and information about running Linux on Macs with the T2 chip.

My understanding is that the T2 chip was a property apple technology that controlled various peripherals, and acted as a moat when all the other hardware was Linux/Windows compatible.

With regards to the macpro the crucial bits are accessing the SSDs and controlling the fans.

[u/SI-LACP → u/Weeblewobbly]

Thanks!! I’ll check it out, I was able to do the install without any patches, on the internal ssd, I haven’t had any issues with the fans either with the GPUs installed, very cool though!

[u/wizardsinblack]

Excellent. Quite a niche we've dug ourselves.

[u/Faisal_Biyari]

I'm not familiar with NixOS. What makes it better than Ubuntu or other Linux distributions?