# Local AI Infrastructure Namespace Instructions This is a compiled namespace source under `pixi-vault/wikis/local-ai-infrastructure/`. ## Rules - Follow the root `Wiki Compiler Maps/Namespace Wiki Compiler Map.md`. - Treat `Knowledge/` and `Projects/` as canonical authoring sources. - Treat Daily Notes as scratch chronology, not direct compiled content. - Keep this namespace scoped to: Local-first AI setup, local LLMs, LoRA/fine-tuning infrastructure, RAG over compiled wikis, deterministic local workflow offload, and hardware/software constraints. - Do not widen scope silently; propose a namespace promotion/routing update first. - Update `wiki/index.md` and `wiki/log.md` whenever compiled pages are added. --- title: Local AI Infrastructure created: 2026-06-16 updated: 2026-07-17 type: namespace-overview status: active category: infrastructure namespace: local-ai-infrastructure confidence: high --- # Local AI Infrastructure > Active namespace for local-first models, retrieval, fine-tuning, deployment, and hardware/software constraints. ## Scope ### Covers Local-first AI setup, local LLMs, LoRA/fine-tuning infrastructure, RAG over compiled wikis, deterministic local workflow offload, and hardware/software constraints. ### Not Covered Cloud-only infrastructure unless it affects local-first migration; product UX unless tied to local model constraints. ### Current As 2026-07-17 — active. Includes local retrieval/RAG concepts, VPS app operation, and the LKY Brain consumer-GPU QLoRA case study with checkout-portable launchers, documented PEFT/vLLM paths, executed downstream llama.cpp Q4_K_M serving, tuned-voice co-placement, and the application-layer audited retrieval seam used by LKY Avatar. ## Canonical Source Roots - `Knowledge/concepts/local-retrieval-agent-infrastructure.md` - `Knowledge/concepts/rag-over-agent-wikis.md` - `Projects/LKY Archive/Index.md` - `Projects/LKY Avatar/Index.md` ## Crosslinks - [[../pixi-vault/README|pixi-vault]] - [[../curated-tuning-datasets/README|curated-tuning-datasets]] - [[../rl-sim-labs/README|rl-sim-labs]] ## Public Output Contract When published to `pixi-wiki`, this namespace should expose: ```text /raw/local-ai-infrastructure/README.md /raw/local-ai-infrastructure/wiki/index.md /wiki/local-ai-infrastructure/README.md /wiki/local-ai-infrastructure/wiki/index.md ``` ## Maintenance - Edit canonical source notes first. - Use `Wiki Compiler Maps/Namespace Wiki Compiler Map.md` for routing decisions. - Do not compile Daily Notes directly unless promoted or verified. --- source_url: https://youtu.be/K8ZTlMaDfmQ?si=IuNRaHVc-fFHZw41 ingested: 2026-06-19 sha256: 3718867c216538e104df7998eb758988e9fd4f264cc916a554fb6a36efd339fb type: transcript --- # Transcript — The Autonomous Agentic Workflow that Runs My Web App w/ Hermes + a $6 VPS Source: https://youtu.be/K8ZTlMaDfmQ?si=IuNRaHVc-fFHZw41 Creator/channel: Tonbi AI Transcript supplied by Jamie in chat on 2026-06-19. ## Video description When you start vibe-coding apps, the tools push you toward Vercel, Supabase, and Railway — but I run everything on one small VPS with a Hermes agent instead, and I'll show you exactly how. Using my live site Agent Wikis (agentwikis.com — free knowledge bases for agents) as the example, I break down why git is the only "database" a docs/wiki/blog-class site needs, how the agent lives on the box and serves the site through Caddy, and the autonomous Cron pipeline that keeps every wiki fresh behind a single Telegram approval. Then I demo the new "demand loop": agents search the wikis via an MCP server, repeated misses (privacy-preserving, no prompts or IPs stored) become an authoring backlog, and the agent researches and fills those gaps automatically — live on the site in minutes. This is Part 1 (the site that needs no database); Part 2 deploys a game app that does need one, on the same box. ## Chaptered transcript notes ### 1. Replacing Vercel/Supabase/Railway with a VPS + agent The setup replaces common beginner deployment platforms — Vercel, Supabase, Railway — with a small VPS and one Hermes agent that maintains web apps, content, and business workflows. The video uses Agent Wikis as the live example and promises a later database-backed game app example. ### 2. The usual stack and what it really gives you Vercel, Railway, and Supabase provide deployment, managed Postgres, backups, auth, realtime, row-level security, storage, and beginner convenience. The core claim is not that they are bad, but that their main value is de-risking and convenience. A VPS can replace the core functions for some project classes, but not all. ### 3. Agent Wikis: when git is your database Agent Wikis has no application database. It is a folder of Markdown files, one git clone per wiki, rendered from disk. The server renders Markdown on each request and keeps a search index in memory, rebuilding only when files change. For docs, blogs, wikis, marketing pages, and similar content sites, git can act as database, CMS, and deploy pipeline. A git push plus server pull makes content live without app restart or build. ### 4. The VPS setup The site runs as a tenant on a small VPS that already hosts the Hermes agent. Caddy terminates TLS and adds a virtual host that reverse-proxies the public domain to localhost; Caddy auto-provisions certificates. The operator can SSH into the VPS and run Hermes directly against files on the box. The agent does not receive unrestricted access; the security model should scope it to the site/workspace areas it needs. ### 5. The content pipeline A cron job fires scout/research agents to check sources such as releases, changelogs, major features, and community notes for each wiki. They deduplicate, research what is genuinely new, and create a plan. A Telegram human gate approves the plan. After approval, the system ingests sources, lints, commits to git, and the site becomes live within roughly 15 minutes. Weekly cadence controls token cost. ### 6. The demand loop Agent Wikis learns from what agents ask for without storing prompts or identities. Agents query through an MCP server. The server receives a distilled query, records only aggregate hit/miss information, and does not store prompts, IPs, or identity. Repeated misses become an authoring backlog. Research agents turn those misses into candidate updates, propose page-vs-extension decisions, and wait for a human approval gate. ### 7. Live demo: filling demand gaps autonomously The demo parses five live demand gaps, each asked at least twice in seven days. It creates item files and child cards, routes them through research and verification, and blocks at the human proposal gate. One example adds a webhook route rate limit answer to the messaging gateway page: each webhook route defaults to 30 requests per minute using a fixed window. Lint checks verify formatting and conflicts before commit. ### 8. Two human gates + merging live The workflow has two gates: proposal approval before writing, then final diff review and merge to main. The author may eventually automate the merge gate after confidence grows, but currently keeps human review before live merge. Once merged, the live site updates automatically. ### 9. When a single box is not enough The VPS pattern is not universal. Use managed services when the project needs real auth/password resets, cannot-lose or regulated data, managed Postgres, global/viral traffic, email deliverability, team preview deploys, or sensitive credential handling. The video frames the VPS pattern as appropriate for solo/operator-controlled apps and content/wiki-class sites first. --- title: Local Retrieval Agent Infrastructure created: 2026-06-18 updated: 2026-06-18 type: concept status: compiled namespace: local-ai-infrastructure tags: [agent-systems, ai, architecture, workflow, knowledge-management] sources: - /root/.hermes/knowledge/concepts/local-retrieval-agent-infrastructure.md - Knowledge/concepts/local-retrieval-agent-infrastructure.md confidence: medium --- # Local Retrieval Agent Infrastructure ## Definition **Local retrieval agent infrastructure** is the reference pattern for giving agents private, source-grounded retrieval over local knowledge without sending the private corpus to hosted systems. Reference pipeline: ```text ingest → clean → chunk → embed → Qdrant/vector store → retrieve → rerank → MCP/agent access ``` The pattern is useful as architecture vocabulary. It is not an implementation approval. ## Current synthesis The vault source describes a local RAG stack inspired by Krill-style infrastructure: 1. ingest documents from approved local sources; 2. clean and neutralize sensitive or irrelevant material; 3. chunk by stable sections with metadata; 4. embed chunks with a local embedding model; 5. store vectors in Qdrant or a similar vector database; 6. retrieve a broad candidate set; 7. rerank to a small evidence packet; 8. expose results to an agent through MCP or another controlled local interface. For Jamie’s current agent workflow, the governance lesson matters more than the stack: improve explicit routing first, then only add retrieval infrastructure if static wiki/index/llms routing fails an important question set. See [[agent-wikis]], [[rag-over-agent-wikis]], and [[runtime-memory-knowledge-routing]]. ## Application Use this page to reason about: - when a markdown/`llms.txt` knowledge pack is enough; - when a static retrieval eval should be written before infrastructure; - how private-corpus retrieval should preserve provenance and freshness; - why compiled/synthesized wiki pages are usually a better retrieval corpus than raw transcripts; - how hardware/local-model lessons differ from agent workflow governance. Hardware/local-model lessons answer “can this run locally?” Workflow-governance lessons answer “should this be routed, verified, and maintained this way?” Keep those questions separate. ## Boundaries This concept does **not** authorize: - installing Qdrant, Ollama, MCP servers, vector databases, rerankers, or embedding services; - changing providers, gateways, profile config, cron jobs, webhooks, secrets, or deployment routing; - building or deploying RAG infrastructure; - generating or publishing public wiki output; - creating automatic memory pruning, model fine-tuning, profile creation, or provider changes; - treating raw transcripts or scratch notes as equivalent to compiled wiki pages; - replacing explicit routing with embeddings. If retrieval is needed, start with a separate approved issue that defines eval questions, privacy boundaries, corpus scope, verification, and rollback. ## Related pages - [[agent-wikis]] - [[rag-over-agent-wikis]] - [[runtime-memory-knowledge-routing]] - [[self-improving-agent-systems]] - [[moc-knowledge-cortex]] - [[bounded-context-tree-pattern]] ## Sources - `/root/ObsidianVault/Knowledge/concepts/local-retrieval-agent-infrastructure.md` - `/root/ObsidianVault/Knowledge/concepts/agent-wikis.md` - `/root/ObsidianVault/Knowledge/concepts/rag-over-agent-wikis.md` --- title: RAG over Agent Wikis created: 2026-06-16 updated: 2026-06-16 type: concept status: compiled namespace: local-ai-infrastructure tags: [local-ai-infrastructure, rag, agent-wikis, retrieval, infrastructure] sources: - Knowledge/concepts/rag-over-agent-wikis.md - Knowledge/concepts/local-retrieval-agent-infrastructure.md confidence: medium --- # RAG over Agent Wikis **RAG over Agent Wikis** means using maintained compiled wiki pages as the retrieval corpus for agents, instead of retrieving directly from raw source dumps or asking the model to answer from memory. ## Infrastructure shape The practical loop is: ```text compiled wiki pages → chunks/sections → embeddings → retrieval/rerank → cited answer ``` For Jamie's stack, the important design choice is that the wiki compilation layer comes before the vector layer. Retrieval quality should be improved by curated pages, frontmatter, source metadata, freshness, and namespace boundaries before adding infrastructure complexity. ## Local-first implementation posture A future local prototype can use: - compiled `pixi-wiki` Markdown pages as the corpus; - local embeddings such as `nomic-embed-text` or a similar sentence model; - FAISS or Qdrant for local vector storage; - optional reranking for quality; - source metadata preserved for citations and freshness checks. ## Current Pixi Wiki use `pixi-wiki` is the current public corpus candidate for this pattern. Its generated pages include namespace scopes, page metadata, raw Markdown mirrors, full-corpus exports, and navigable HTML. That structure should be used as the first retrieval/eval surface before adding a vector database or runtime RAG service. ## Quality gates Do not ship retrieval infrastructure because it sounds useful. Use evals: - recall@k for known routing questions; - citation support for factual answers; - fallback behavior when a namespace does not cover the query; - change-aware questions that test whether updated wiki metadata helps. ## Boundary No MCP server, vector DB, crawler, hosted search, or runtime RAG is approved by this compiled page. This page is an infrastructure concept and future implementation home, not a build authorization. ## Source Compiled from `Knowledge/concepts/rag-over-agent-wikis.md` and `Knowledge/concepts/local-retrieval-agent-infrastructure.md`. --- title: VPS Agent Web App Pattern created: 2026-06-19 updated: 2026-06-19 type: concept status: compiled namespace: local-ai-infrastructure tags: [local-ai-infrastructure, vps, hermes-agent, agent-workflows, gitops, web-apps] sources: - raw/transcripts/tonbi-vps-agent-web-app-workflow-2026.md - https://youtu.be/K8ZTlMaDfmQ?si=IuNRaHVc-fFHZw41 confidence: medium --- # VPS Agent Web App Pattern The **VPS Agent Web App Pattern** uses a small always-on VPS plus a resident agent to replace parts of the beginner Vercel/Supabase/Railway stack for simple, operator-controlled web apps. It works best when the app is mostly files, Markdown, static-ish content, or a small service where the agent can safely own the deploy loop. This is not “VPS beats managed platforms.” The useful distinction is: managed platforms buy convenience and de-risking; a VPS buys control, low recurring cost, direct file access, and an agent that can operate the system where it runs. ## Core shape ```text git-backed content/app repo ↓ VPS workspace with one app clone + one clone per knowledge base ↓ local web server / app process ↓ Caddy vhost + automatic TLS ↓ resident Hermes agent reachable by SSH or gateway ↓ cron/scout/research/update loops with human gates ``` For docs, blogs, wikis, and marketing pages, git can be the database, CMS, and deploy pipeline. The server can render Markdown from disk, keep a search index in memory, and rebuild only when files change. In that class of system, a database would answer a question the app does not actually ask. ## What the agent replaces The resident agent can cover a subset of what Vercel/Supabase/Railway usually provide: | Managed stack job | VPS-agent equivalent | Boundary | |---|---|---| | Frontend deploy | git push/pull, Caddy vhost, local app process | Good for small sites; weak for preview deploy/team workflows. | | CMS updates | Markdown files in git | Good for docs/wiki/blog content; weak for multi-user editing. | | Background jobs | Hermes cron / shell cron / systemd timers | Needs explicit logging and recovery. | | Content freshness | scout agents + source checks + lint + commit | Needs human gates until trust is earned. | | Admin console | SSH + Hermes CLI/gateway | Powerful but must be scoped. | | Search/retrieval feedback | aggregate MCP hit/miss demand loop | Must avoid prompt/IP/user logging. | ## Agent Wikis example The transcript's example is Agent Wikis: a public wiki-of-wikis where each wiki is a folder of Markdown files. The site renders from disk, uses git as the deploy mechanism, and avoids an application database. That maps closely to Pixi Wiki's source/output model: - `pixi-vault` holds source truth and compiled namespace source. - `pixi-wiki` publishes raw Markdown, rendered HTML, `llms.txt`, `llms-full.txt`, `index.json`, and MCP-facing corpus files. - [[rag-over-agent-wikis|RAG over Agent Wikis]] uses the compiled wiki as retrieval corpus before adding heavier vector infrastructure. - [[../../../pixi-vault/wiki/syntheses/pixi-vault-to-pixi-wiki-publishing-model|Pixi Vault to Pixi Wiki Publishing Model]] defines the publish boundary. ## Content freshness loop The durable workflow pattern is: ```text cron fires → scout checks sources → deduplicate and score newness → research what matters → propose page changes → human gate approves plan → ingest/update Markdown → lint and verify → commit to git → merge/pull makes site live ``` The human gate is important. It keeps the agent from turning noisy sources into public content just because it can write. A second merge/diff gate is useful until the operator trusts the loop. ## Demand loop The demand loop is the stronger product idea: ```text agent asks wiki via MCP → server records aggregate hit/miss for distilled query → repeated misses become backlog candidates → scout/research agents propose additions → human approves → wiki fills real usage gaps ``` The privacy contract matters more than the automation: record aggregate misses, not prompts, IP addresses, or identities. The loop should learn what the knowledge base lacks without becoming surveillance infrastructure. ## When to use this pattern Use it when: - the app is a docs/wiki/blog/marketing/content site; - git can be the source of truth; - one operator can tolerate SSH/Git-based operations; - the resident agent can be scoped to the app workspace; - freshness comes from source monitoring and editorial review; - downtime or rollback risk is acceptable for a small VPS. Avoid it, or add managed services, when: - the app needs real auth, password resets, or user account recovery; - it stores cannot-lose or regulated data; - it needs managed Postgres, backups, row-level security, or object storage guarantees; - it may face global/viral traffic; - it needs high email deliverability; - a team needs preview deploys and parallel environments; - secrets or credentials require stronger isolation than a general-purpose agent workspace. ## Pixi Wiki routing Primary namespace: `local-ai-infrastructure`, because the reusable pattern is about local/VPS deployment topology, file-backed web apps, Caddy/git operations, and deciding when a single box can replace managed infrastructure. Crosslinks: - [[../../../agent-workflows/wiki/syntheses/pixoid-crew-operating-model|Pixoid Crew Operating Model]] — route governance, human gates, verification, and durable state. - [[../../../hermes-agent/wiki/concepts/source-priority|Hermes Source Priority]] — use official Hermes docs/local source for commands and runtime behavior; this transcript is a pattern source, not command truth. - [[../../../pixi-vault/wiki/entities/pixi-wiki|Pixi Wiki]] — the public wiki surface where this pattern can inform source→generated publishing. ## Open questions - Should Pixi Wiki eventually have its own demand-loop telemetry over MCP queries, and if so what aggregate-only privacy contract is acceptable? - Which parts of the publish workflow are safe to automate beyond the current regenerate/test/push/live-verify loop? - Should a future `agent-workflows` page split out the human-gated content freshness loop from this infrastructure page? ## Source Compiled from Tonbi AI's video transcript, “The Autonomous Agentic Workflow that Runs My Web App w/ Hermes + a $6 VPS,” supplied by Jamie on 2026-06-19. --- title: Local AI Infrastructure — Master Index created: 2026-06-16 updated: 2026-07-17 type: index status: active namespace: local-ai-infrastructure --- # Local AI Infrastructure — Master Index > Scaffold index for `local-ai-infrastructure`. Add compiled pages here as they are created. ## Concepts - [[concepts/local-retrieval-agent-infrastructure|Local Retrieval-Augmented Agent Infrastructure]] — Local-first model/retrieval stack pattern for private corpora and agent memory. - [[concepts/rag-over-agent-wikis|RAG over Agent Wikis]] — Retrieval architecture over compiled wiki pages rather than raw source dumps. - [[concepts/vps-agent-web-app-pattern|VPS Agent Web App Pattern]] — Small-VPS + resident-agent pattern for git-backed web apps, Caddy/TLS hosting, human-gated content freshness loops, and when not to replace managed platforms. ## Entities ## Summaries - [[summaries/lky-brain-consumer-gpu-qlora|LKY Brain Consumer-GPU QLoRA Case Study]] — Executed Qwen3-14B/Unsloth training plus downstream llama.cpp Q4_K_M serving beside the tuned voice on one 16GB GPU, now with an application-layer audited retrieval seam; preserves config drift, live-proof, and reproducibility boundaries. ## Syntheses - Cross-namespace summary: [[../../hermes-agent/wiki/summaries/external-hermes-wikis-import-review|External Hermes Wikis Import Review]] — Marks Local Stack / Airplane Mode as local-infrastructure primary, with deployment backend tradeoffs crosslinked from Hermes Agent. ## Source Roots - `Knowledge/concepts/local-retrieval-agent-infrastructure.md` - `Knowledge/concepts/rag-over-agent-wikis.md` - `Projects/LKY Archive/Index.md` - `Projects/LKY Avatar/Index.md` - `raw/transcripts/tonbi-vps-agent-web-app-workflow-2026.md` - `https://youtu.be/K8ZTlMaDfmQ?si=IuNRaHVc-fFHZw41` --- title: Local AI Infrastructure — Activity Log created: 2026-06-16 updated: 2026-07-17 type: log status: active namespace: local-ai-infrastructure --- # Local AI Infrastructure — Activity Log > Append-only namespace log. ## 2026-07-17 refresh | LKY Avatar application-layer fact retrieval - Updated the LKY Brain case study after `lky-avatar` issue #45 / PR #47 merged a small audited fact sheet, deterministic per-turn retrieval, uncertainty instructions, Singapore STT keyterms, and a 12-question factuality eval. - Preserved the serving boundary: the Q4_K_M llama.cpp brain is unchanged; grounding belongs to the voice-agent application layer. - Preserved the proof boundary: tests cover the seams, while real-microphone STT and grounding-on/off local-brain quality results remain pending. ## 2026-07-15 refresh | LKY Brain downstream avatar serving path - Refreshed the LKY Brain consumer-GPU summary from the separate `lky-avatar` integration evidence. - Added the executed merged-LoRA Q4_K_M llama.cpp route, 80.5 tok/s warm decode, tuned-voice co-placement, and ten-turn no-failure/VRAM evidence. - Added the later interactive chat REPL and its temperature 0.7 / repetition-penalty 1.1 correction while preserving the historical evaluation boundary. - Kept the boundary that this verifies one local product path, not vLLM, multi-user capacity, factual reliability, or a reproducible locked training environment. ## 2026-06-16 create | Namespace scaffold initialized - Created README, CLAUDE instructions, raw folder, index/log, and typed wiki folders. - Source routing comes from `Wiki Compiler Maps/Namespace Wiki Compiler Map.md`. - No Daily Notes were copied or compiled. ## 2026-06-16 update | Add first local-AI infrastructure compiled concepts - Added `wiki/concepts/local-retrieval-agent-infrastructure.md`. - Added `wiki/concepts/rag-over-agent-wikis.md`. - Updated `wiki/index.md` from scaffold to active content index. - Preserved the boundary that this is not approval to build MCP/vector/search runtime infrastructure. - No Daily Notes were copied or compiled. ## 2026-06-16 update | Link Pixi Wiki to RAG-over-wiki posture - Updated `wiki/concepts/rag-over-agent-wikis.md` to name `pixi-wiki` as the current public corpus candidate. - Preserved the eval-first boundary: use the compiled wiki as the first retrieval/eval surface before adding vector DB or runtime RAG infrastructure. - No Daily Notes were copied or compiled. ## 2026-06-18 update | Crosslink Hermes local-stack routing - Added cross-namespace pointer to the Hermes Agent external wiki import review for Local Stack / Airplane Mode and deployment-backend routing. - No Daily Notes were copied or compiled. ## 2026-06-18 update | Refresh local retrieval infrastructure boundary - Refreshed `wiki/concepts/local-retrieval-agent-infrastructure.md` from the verified local Hermes KB closure synthesis. - Preserved the explicit no-install/no-MCP/no-RAG-build/no-provider/no-deploy authorization boundary. ## 2026-06-19 ingest | Tonbi VPS agent web app workflow - Added `raw/transcripts/tonbi-vps-agent-web-app-workflow-2026.md` from Jamie-supplied transcript of Tonbi AI video. - Added `wiki/concepts/vps-agent-web-app-pattern.md` as a local-infrastructure concept for small-VPS + resident-agent web app operation. - Updated `wiki/index.md` with the new concept and source handles. - Routed workflow/human-gate material by crosslink to `agent-workflows`; kept primary namespace as `local-ai-infrastructure` because the reusable decision is VPS/local deployment topology. ## 2026-07-10 add | LKY Brain consumer-GPU QLoRA case study - Added `wiki/summaries/lky-brain-consumer-gpu-qlora.md` from the verified public repo and training report. - Distinguished the successful Unsloth script from the retained but unexecuted Axolotl portability config. - Captured WSL2 + Blackwell failure isolation, non-packed training, plain-Transformers inference, checkpoint preservation, and environment-specific boundaries. - Crosslinked source/provenance to `curated-tuning-datasets` and evidence quality to `eval-trace`. ## 2026-07-10 refresh | LKY Brain public-use and portability milestone - Refreshed the consumer-GPU QLoRA summary from repo commit `da490f6`. - Recorded checkout-relative launchers, the documented 4-bit PEFT quick-test path, and the vLLM OpenAI-compatible serving path. - Preserved the boundary between the executed training run and newly documented but not review-executed inference/serving examples. - Confirmed that dataset, training, and evaluation numbers did not change in this prose/portability refresh. --- title: LKY Brain Consumer-GPU QLoRA Case Study created: 2026-07-10 updated: 2026-07-17 type: summary status: compiled namespace: local-ai-infrastructure tags: [local-ai-infrastructure, qlora, consumer-gpu, wsl2, blackwell, reproducibility, llama-cpp, voice-ai] sources: - Projects/LKY Archive/Index.md - Projects/LKY Avatar/Index.md - https://github.com/pixiiidust/lky-brain confidence: high --- # LKY Brain Consumer-GPU QLoRA Case Study `lky-brain` demonstrates a complete Qwen3-14B QLoRA run on one RTX 5070 Ti 16GB GPU under WSL2 Ubuntu 24.04. The useful infrastructure lesson is not one magic config; it is how to isolate the actual failing layer and preserve a working evidence path. ## Successful run - base: Qwen3-14B, non-thinking mode; - quantization: 4-bit QLoRA; - adapter: rank 64 / alpha 128 across linear projections; - actual trainer: Unsloth; - maximum sequence length: 2,048; - assistant-turn-only loss; - micro-batch 1, gradient accumulation 8; - three epochs, final train loss 1.247; - generation: plain Transformers + PEFT; - judging: separate Claude API pass. ## Portability config vs executed config The retained Axolotl YAML is not the executed evidence on this machine. It specifies 4,096 sequence length, packing, dropout 0.05, and in-loop eval. The successful Unsloth path used 2,048, `packing=False`, dropout 0.0, and no in-loop eval. Agents should quote the executed script (`train/train_unsloth.py`) when describing the result and treat `train/lky-qlora.yml` as a portability candidate only. ## Failure isolation pattern 1. Probe raw CUDA allocation, matmul, dtype, and 4-bit load. 2. Reproduce a manual quantized LoRA train step. 3. Compare framework paths. 4. Disable the smallest failing optimization: varlen packing, pinned memory, or custom inference kernel. 5. Preserve diagnostic scripts and working launchers. 6. Decouple GPU generation from API judging. 7. Save intermediate checkpoints because post-hoc quality may peak before final train loss. ## Environment-specific findings On this WSL2 + Blackwell stack: - Axolotl crashed during model loading even when plain Transformers/bitsandbytes worked. - Unsloth import order mattered because it patches TRL/Transformers. - Triton required a C compiler. - packed/varlen attention crashed; dense non-packed training worked. - pinned-memory loading and Unsloth's Qwen3 inference kernel were avoided. These are verified for this run, not universal prescriptions for every GPU, WSL version, or future package release. ## Portable use and serving path The public-readability refresh at repo commit `da490f6` replaced machine-specific checkout paths in the setup, training, generation, and upload launchers with paths derived from each script's location. The repo now documents two downstream paths: - **Quick local inference:** load Qwen3-14B in 4-bit with Transformers/bitsandbytes, attach `sjsim/lky-qlora` through PEFT, and run with about 16GB VRAM. - **OpenAI-compatible serving:** run vLLM with LoRA enabled, register the adapter as `lky`, keep maximum LoRA rank at 64, and disable Qwen3 thinking in request metadata. The serving guide estimates about 28GB for the full-precision 14B base and points readers toward a 40GB card, or a 24GB card with quantization. These examples make the project easier to try and integrate, but they were not executed as part of this VPS review and should not be promoted to verified serving evidence yet. ## Interactive chat path and sampling correction Repo commit `3a9f6a5` added a streaming Transformers + PEFT chat REPL with conversation history, `/date` era switching, and `/reset`. The documented interactive defaults are temperature 0.7 with repetition penalty 1.1 after temperature 0.8 without a penalty produced degenerate loops in field use. This is an executed local interaction path, not evidence for the unexecuted vLLM example. It also does not rewrite the historical n=24 style-evaluation contract, which used its original stochastic generation settings. ## Executed downstream serving in LKY Avatar The separate `lky-avatar` product has now executed a different serving route on the same RTX 5070 Ti: - merge the published epoch-2 LoRA into Qwen3-14B; - quantize the merged model to Q4_K_M GGUF; - serve through llama.cpp's OpenAI-compatible `llama-server`; - use the same client seam as the slower 4-bit Transformers + PEFT fallback. Measured warm brain performance was 80.5 tok/s p50 decode and about 0.05 s time to first token. The brain plus the fine-tuned Chatterbox voice then completed a ten-turn same-GPU placement run with zero failures; TTS RTF mean/max was 0.369/0.397 and total-card VRAM peaked at 15,813 MiB of 16,303 MiB. This verifies one local product-serving path. It does not validate the README's vLLM example, establish multi-user serving capacity, or make the style adapter factually reliable. Live use exposed confident biography/date/place hallucinations. The application layer now adds a small audited fact sheet, deterministic section retrieval per turn, a source-over-memory and uncertainty block immediately before the latest question, and Singapore proper-noun input boosts without changing llama-server. A 12-question eval separates factual accuracy, persona quality, and fabrication and supports matched grounding-on/off runs. Repository verification passed 148 root tests and 199 voice-agent tests with 3 live-service skips. That proves the retrieval, prompt, config, STT-construction, and eval seams as code; it does not prove the model-quality lift. Real-microphone STT and a local-brain grounding-on/off run remain the acceptance evidence. ## Evidence boundary The repo has no CI or unit-test suite for the data/training pipeline, and the generated datasets/checkpoints are intentionally gitignored. The `uv.lock` covers the data-pipeline environment, but the version-sensitive Unsloth/TRL/Transformers/PEFT/Torch/bitsandbytes training stack is installed separately without exact pins. Reproducibility therefore depends on live sources, external model artifacts, package resolution, and operator-run hydration/training steps. The portable shell paths reduce checkout-specific friction but do not solve dependency or artifact reproducibility. Add training locks, manifests, hashes, schema versions, a small CPU-only pipeline fixture, and GPU smoke checks for the documented inference/serving paths before calling the template fully reproducible.