IMPROVEMENT ROADMAP

Six LLMs (262K–1M context) behind one authenticated endpoint, tool calling, vision, embeddings, reranking, live model-swap serving, an agent-profile integration, and a comparison playground. Media-gen stack installed and tested (see Media Gen). What follows is what was missing — ranked by capability-per-effort; every gap below is now shipped and marked.

1. Native MCP + built-in tools on the endpoint INSTALLED + verified incl. second-model regression (read-only tools + MCP echo live; exec off; note: file tools have no path jail — key-holders read as wg)

llama-server became a self-contained agent host in July 2026 (upstream PR #26062): --mcp-servers-config attaches stdio MCP servers, --tools/--agent exposes filesystem/shell tools directly from the endpoint. Biggest capability jump per minute spent — every connected agent (Hermes included) inherits server-side tools without any client changes.

Caveat: built-in exec_shell_command is unsandboxed and the rig is tailnet-exposed. Pair with the API key (done) + a prompt-guard pass before widening access.

2. Qdrant — persistent vector store INSTALLED + verified (1.19.1, native binary, local-only)

The rig has embeddings and reranking but nothing persists vectors. Qdrant (Apache-2.0, single binary/container) is the foundation both RAG and memory layers want. Feeds from the existing embed endpoint; queried through the existing reranker.

3. mem0 — persistent memory layer INSTALLED + verified (2.0.20, round-trip PASS)

The remaining fully-self-hostable memory layer (Zep killed its Community Edition in 2026). Defaults to Qdrant — reuses #2 directly. Gives agents durable memory across sessions against the OpenAI-compatible endpoint.

Caveat: mem0's benchmark numbers are vendor-reported (independent LongMemEval score ≈ 49 vs claimed 94). Evaluate on your own data before trusting it.

4. Document RAG engine INSTALLED + verified (LightRAG 1.5.7 — ingest + multi-hop queries with citations, :9621)

Three shapes, all consume the existing endpoints as-is:

Decision hinges on what corpus goes on this box first. Khoj/RAGFlow remain installable alternates. Verified gotcha: thinking-mode models burn token budgets before answering — LightRAG needed enable_thinking:false (set via OPENAI_LLM_EXTRA_BODY); reasoning tokens dropped 25k → 0.

5. Observability — Arize Phoenix INSTALLED + verified (20.9.0, :6006, OTLP ingest tested)

Zero tracing today. Phoenix (OTel-native) traces every call through llama-swap and ships free RAG-relevance / hallucination evals. Langfuse (MIT) only if scale demands its ~6 GB ClickHouse stack.

6. Bonus: LoRA fine-tuning on this box feasible now

Unsloth QLoRA runs on gfx1151 (verified by community May 2026 via TheRock ROCm builds) — up to ~70B-class models at 4-bit within ~80 GB peak. Train while LLMs are swapped out. The pip environment is fragile — containerize if pursued.

Problem it solves: memory, grounding, screening, and tracing live server-side — but nothing in the OpenAI protocol carries them. Any client that skips them loses them. The gateway makes them unavoidable.

Shape (shipped): one FastAPI service at :8081 (/srv/llm/gateway/, systemd user unit gateway.service, 158 MB RSS), proxying to llama-swap on :8080. Clients point at the gateway instead — the URL change is live: use :8081; everything OpenAI-compatible passes through, plus:

Verified (2026-09-11): 401 without/wrong API key; models passthrough; health; plain chat; memory round-trip (stored fact → "Emerald" answer); streaming SSE byte-for-byte (53 chunks). Full log: research/gateway-build.md.

Latency (verified post-fix, 2026-09-11): measurement surfaced a llama-swap config bug — the groups' models: key is silently ignored by llama-swap v248 (it wants members:), so every model sat in one exclusive group and embed↔chat traffic kept evicting each other's models (2.4s + 6-10s reloads). The lead fixed it (members: + persistent: true on embed); after the fix the gateway's own overhead is ~21 ms (mem0 search) with direct vs gateway round-trips statistically indistinguishable (1408 ms vs 929 ms means, 5 pairs) — far under the 200 ms target. Raw :8080 keeps paying the eviction tax until clients move to :8081 — one more reason to use the gateway.

Rules: streaming passes through untouched (SSE, verified); API-key auth enforced at the gateway (verified); raw :8080 stays available for admin/benchmark use only.

OpenWebUI note: the OpenWebUI guide will point at the gateway URL — one URL change, nothing else moves.

• LiteLLM gateway — llama-swap already covers routing/swapping.
• Semantic caches — solve a cloud-cost problem; local compute is free.
• Guardrail stacks — premature until MCP/shell exposure widens; then a Prompt-Guard-2 pre-check step is the move.
• Letta (stateful agent runtime) — only if adopting a full agent runtime; mem0 covers the memory need alone.

Known issue — found during multi-agent verification (2026-09-10)

Vulkan GTT release lag between chat-model swaps: when llama-swap swaps one chat model for another, the outgoing model's buffers are not fully reclaimed before the incoming model allocates — two swap attempts OOM-killed the incoming loader while the outgoing model was still resident. Mitigation until fixed upstream (llama.cpp Vulkan allocator): use models sequentially, allow a pause between swaps, or stop-start the server around forced swaps. Affected: multi-model concurrent load patterns; single-model serving unaffected.

Per-gap tables, licenses, memory budgets, sources, and UNVERIFIED flags: research/capability-gaps.md in the build repo. Media-gen track: Media Gen Roadmap.