IMPROVEMENT ROADMAP
What this rig already has
Six LLMs (262K–1M context) behind one authenticated endpoint, tool calling, vision, embeddings, reranking, live model-swap serving, an agent-profile integration, and a comparison playground. Media-gen research done (installing separately). What follows is what's missing — ranked by capability-per-effort.
Ranked: what to add next
1. Native MCP + built-in tools on the endpoint hours · ~0 RAM
llama-server became a self-contained agent host in July 2026 (upstream PR #26062):
--mcp-servers-config attaches stdio MCP servers, --tools/--agent
exposes filesystem/shell tools directly from the endpoint. Biggest capability jump per
minute spent — every connected agent (Hermes included) inherits server-side tools without
any client changes.
Caveat: built-in exec_shell_command is unsandboxed and the rig is
tailnet-exposed. Pair with the API key (done) + a prompt-guard pass before widening access.
2. Qdrant — persistent vector store ~1 hr · 1–2 GB
The rig has embeddings and reranking but nothing persists vectors. Qdrant
(Apache-2.0, single binary/container) is the foundation both RAG and memory layers want.
Feeds from the existing embed endpoint; queried through the existing reranker.
3. mem0 — persistent memory layer ~2 hrs · <1 GB
The remaining fully-self-hostable memory layer (Zep killed its Community Edition in 2026). Defaults to Qdrant — reuses #2 directly. Gives agents durable memory across sessions against the OpenAI-compatible endpoint.
Caveat: mem0's benchmark numbers are vendor-reported (independent LongMemEval score ≈ 49 vs claimed 94). Evaluate on your own data before trusting it.
4. Document RAG engine 1–2 days · pick by corpus
Three shapes, all consume the existing endpoints as-is:
| Engine | License | Footprint | Best for |
|---|---|---|---|
| Khoj | AGPL | ~4 GB | personal notes / Obsidian vault |
| RAGFlow | Apache-2.0 | 16 GB+ multi-container | scanned PDFs, tables, heavy docs |
| LightRAG | MIT | light | graph-style multi-hop Q&A |
Decision hinges on what corpus goes on this box first.
5. Observability — Arize Phoenix ~1 hr · <1 GB
Zero tracing today. Phoenix (OTel-native) traces every call through llama-swap and ships free RAG-relevance / hallucination evals. Langfuse (MIT) only if scale demands its ~6 GB ClickHouse stack.
6. Bonus: LoRA fine-tuning on this box feasible now
Unsloth QLoRA runs on gfx1151 (verified by community May 2026 via TheRock ROCm builds) — up to ~70B-class models at 4-bit within ~80 GB peak. Train while LLMs are swapped out. The pip environment is fragile — containerize if pursued.
Evaluated and not recommended (with reasons)
- LiteLLM gateway — llama-swap already covers routing/swapping.
- Semantic caches — solve a cloud-cost problem; local compute is free.
- Guardrail stacks — premature until MCP/shell exposure widens; then a Prompt-Guard-2 pre-check step is the move.
- Letta (stateful agent runtime) — only if adopting a full agent runtime; mem0 covers the memory need alone.
Full research
Per-gap tables, licenses, memory budgets, sources, and UNVERIFIED flags:
research/capability-gaps.md in the build repo. Media-gen track:
Media Gen Roadmap.