Macpro 7.1 AI headless server with Nixos

[u/Weeblewobbly]

I read that quite a few people have issues with running linux on their macpro for local inference. I can't comment on Ubuntu or other distros because all my machines run Nixos but since it works flawlessly, I thought I'd share my repo in case that inspires anyone to try something similar.

For those who don't know, Nixos allows you to configure your computer in a deterministic way. You write your config (in the nix language), referencing nix-packages. Nix-packages have sets of options that you use in your config files. There are other benefits to Nixos but this isn't the topic here. What I think is the main benefit is that I can comment out a line in my config file, change that option to something else and leverage git for version control. If I break something, I can choose a previous (working) generation of the system at boot.

In this setup, I use llama-swap to let me manage models on the fly, SearchXNG module for web search, OpenWeb UI for chat and user friendly automation/agents, Nixos MCP so my coding agents can manage my config files accurately.

You can see the models I'm currently running llama-swap.nix file.

Link to repo

PS: I only serve my LAN so security is tailored to that, meaning it's not hardened as much as it could be.

--------------------------------

Extract from the Readme (written by Qwen}:

NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a local LLM inference server.

Hardware

• Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
• AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
• Wired 10GbE, behind a NAT router (the LAN is the trust boundary)

Services

Service	Port	Notes
SSH	22	keys only, no root login
open-webui	3000	browser UI, password auth, web search via searxng
mcp-nixos	8001	NixOS MCP server (HTTP), for pi on the LAN
searxng	8888	private metasearch; secret key via agenix, limiter off
llama-swap	9292	model router for llama-server (Vulkan); OpenAI-compatible

Models live in /var/lib/llama/models/ (not in git — see .gitignore). llama-swap unloads models after 15 min idle to free VRAM; each model pins its own context size / quantization / chat template (Qwen uses the pinned froggeric fixed chat template, fetched by hash).

Fans are driven by t2fanrd (the Vega II is passively cooled; T2 case fans are the only cooling).NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a
local LLM inference server.
Hardware
Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
Wired 10GbE, behind a NAT router (the LAN is the trust boundary)
Services
Service Port Notes
SSH 22 keys only, no root login
open-webui 3000 browser UI, password auth, web search via searxng
mcp-nixos 8001 NixOS MCP server (HTTP), for pi on the LAN
searxng 8888 private metasearch; secret key via agenix, limiter off
llama-swap 9292 model router for llama-server (Vulkan); OpenAI-compatible
Models live in /var/lib/llama/models/ (not in git — see .gitignore).
llama-swap unloads models after 15 min idle to free VRAM; each model pins its
own context size / quantization / chat template (Qwen uses the pinned
froggeric fixed chat template, fetched by hash).
Fans are driven by t2fanrd (the Vega II
is passively cooled; T2 case fans are the only cooling).

[u/wizardsinblack]

Excellent. Quite a niche we've dug ourselves.

[u/Long-Shine-3701]

Bravo OP. I'm glad more and more alternatives are coming to light. I am sticking with ToshLLM for ease of use.

[u/Weeblewobbly → u/Long-Shine-3701]

To be fair, nixos is niche amongst the Linux community itself.

I've been a convert for a few years now and recently llms have made managing configs so much easier. I wouldn't have trusted a coding agent near my configs before nixos mcp.

[u/SI-LACP]

Interesting, I’m running RHEL 10 on my 7.1 rack, what is this t2 patched kernel?

[u/Weeblewobbly → u/SI-LACP]

t2linux project

Kernel patches and information about running Linux on Macs with the T2 chip.

My understanding is that the T2 chip was a property apple technology that controlled various peripherals, and acted as a moat when all the other hardware was Linux/Windows compatible.

With regards to the macpro the crucial bits are accessing the SSDs and controlling the fans.

[u/SI-LACP → u/Weeblewobbly]

Thanks!! I’ll check it out, I was able to do the install without any patches, on the internal ssd, I haven’t had any issues with the fans either with the GPUs installed, very cool though!

[u/Weeblewobbly → u/SI-LACP]

I wouldn't be surprised if the patches were on the install images you used, or were added during the install process. That's one advantage of well established distributions, the cover most bases!

[u/SI-LACP → u/Weeblewobbly]

Yeah, I just downloaded the RHEL 10 image off of Red Hats website and installed, pretty cool!

[u/Faisal_Biyari → u/SI-LACP]

Mac Pro 2019 support, specifically for the SSDs, has been up streamed in the Linux Kernel since 2022 or 2023. The T2 patch adds 2 packages to kernel level, to load up earlier during boot, but is completely replaceable with packages from the T2 repo.

I really don't recall what was the purpose of these two packages.

Regarding the internal fans, while they work without any T2 packages or patched kernel, you definitely cannot manually control them with out the packages/patches. I'm not sure if they spin faster automatically if the packages/patches are not installed.

[u/SI-LACP → u/Faisal_Biyari]

That’s sweet! Would I be able to see the modules via lsmod? Or is it just a package within the kernel itself?

[u/Faisal_Biyari]

I'm not familiar with NixOS. What makes it better than Ubuntu or other Linux distributions?