Architecture

llama.cpp built from current source with two backends — Vulkan (RADV) and HIP (ROCm gfx1151) — benchmarked head-to-head, winner per workload serves. Model weights live on the dedicated 2TB NVMe (/srv/llm/models). Multiple models serve side-by-side where memory allows (Qwen3.8-27B at 19 GB + DeepSeek-V4-Flash at 103 GB ≈ 121 GB co-resident; the 27B + a smaller endpoint otherwise), coordinated by llama-swap — the same tier/eviction design proven on our ashborn-prime node. One OpenAI-compatible endpoint (:8080) fronts everything; a small embeddings model gets a dedicated endpoint; the CPU reranker pattern carries over verbatim. Everything runs as user wg via systemd user units — the dan setup account was removed 2026-09-11.

• HIP/ROCm — full-speed prefill (prompt processing) and the path with the most Strix Halo community tuning (draft-MTP, the top-k radix fix for Qwen3.8 — both included in today's source build).
• Vulkan/RADV — 2026 consensus: faster decode on Strix Halo than HIP for MoE models (~40% gap in llama.cpp issue #24438), with MTP spec-decode support merged. Built from the same source tree.
• Benchmarks (see Benchmarks) decide which one each model runs on; both binaries stay installed.

Rejected with evidence (don't re-add): amd_iommu=off (+5–12% prefill but breaks the NPU), xnack=1, vm_update_mode, gartsize=131072, GGML_CUDA_ENABLE_UNIFIED_MEMORY (CUDA-only knob), TDP/TDC/EDC chasing (box already sustains 120 W; decode is bandwidth-bound). Contested (bench before believing): rocWMMA flash-attn flag, ttm.page_pool_size, vm.compaction_proactiveness=20. Full report: research/amd-ai-playbook-strix-halo.md.

/srv/llm/                     (owned by wg — survives dan-account removal)
├── src/
│   ├── llama.cpp/            production tree — build-vulkan/ + build-hip/ (serving lane)
│   ├── llama.cpp-fp4/        pinned-branch build (d3ca537 + qwen4exp patches) for Flash-Next FP4
│   └── llama.cpp-glm/        PR-branch build for GLM-5.3-Flash (agent experiment)
├── bench/                    benchmark scripts  ·  logs/  →  all run outputs
├── venv/                     Python 3.14 + huggingface_hub CLI (model pulls)
└── models/                   weights, one directory per model (2TB NVMe):
    Qwen3.8-27B (main+mmproj+MTP) · Flash-Next IQ1_S set · Flash-Next ROCmFP4 ·
    GLM-5.3-Flash IQ2_XS · Llama-3.3-70B · Qwen3-Embedding · bge-reranker (4 quants)

The OS keeps its own 1TB NVMe; wiping/replacing models never touches it.

• llama-swap tier + evict_cost + startup-preload design — with 128G unified memory the VRAM-eviction pressure that shaped the prime config mostly vanishes; co-residency becomes the default.
• DRY sampling block (agent-safe): --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2, repeat-penalty 1.0, no presence penalty, --jinja, -n 8192 cap.
• Expert offload + big-ubatch ingest: -ot ".ffn_...=CPU" pattern, -ncmoe, -ub 2048 — re-validated on gfx1151 during benchmarking (CPU here is 16C/32T, strong for expert compute).
• CPU rerank service: bge-reranker-v2-m3 Q8_0, -ngl 0 -t 4 --rerank --pooling rank — unit file copied pattern.
• MTP draft via embedded nextn head (≤2 draft caveat, ~1.7 GB draft context).

Agent integration (Hermes profile template, model ids, client examples): Hermes Integration →

OpenWebUI is NOT installed on this box by design — the owner sets it up on his own machine after handback (guide: OpenWebUI →, with <BOX-IP> placeholders).

• /srv/llm tree owned by wg (owner's account) — the temporary setup account dan was removed 2026-09-11; the box is single-user (wg, wheel).
• All services run as systemd user units under wg (lingering enabled), same pattern as ashborn-prime — no root-owned services.
• firewalld: tailnet CGNAT rules currently allow 8080/8081/18190; planned lockdown keeps only AI ports (8081 gateway, 8080 admin, 18190 rerank, 8188 ComfyUI, 9621 LightRAG, 6006 Phoenix) + SSH. LAN rules for the owner's static-IP network are part of handback (see OpenWebUI guide).
• Access model after handback: the owner reaches every AI endpoint over his LAN (static IP); the operator reaches it via Tailscale — which STAYS on the box. Endpoint of record for clients is the gateway :8081; raw :8080 is the admin/benchmark lane.