# Eval Trace Namespace Instructions This is a compiled namespace source under `pixi-vault/wikis/eval-trace/`. ## Rules - Follow the root `Wiki Compiler Maps/Namespace Wiki Compiler Map.md`. - Treat `Knowledge/` and `Projects/` as canonical authoring sources. - Treat Daily Notes as scratch chronology, not direct compiled content. - Keep this namespace scoped to workflow, agent, and model-behavior evaluation; traces; context-overfitting detection; reliability metrics; evidence gates; style-transfer checks; and quality read-outs. Route dataset provenance to `curated-tuning-datasets` and training runtime to `local-ai-infrastructure`. - Do not widen scope silently; propose a namespace promotion/routing update first. - Update `wiki/index.md` and `wiki/log.md` whenever compiled pages are added. --- title: Eval Trace created: 2026-06-16 updated: 2026-07-17 type: namespace-overview status: active category: evaluation namespace: eval-trace confidence: high --- # Eval Trace > Active namespace for evidence quality, behavior/workflow evaluation, reliability, and claim boundaries. ## Scope ### Covers Workflow, agent, and model-behavior evaluation; traces; context-overfitting detection; reliability metrics; evidence gates; style-transfer checks; and quality read-outs. ### Not Covered General observability unrelated to behavior/workflow quality; product analytics unless used as an evaluation surface; model-training infrastructure except where it changes the evaluation contract. ### Current As 2026-07-17 — active. Includes context-overfitting, the Eval Trace prototype, workflow-quality mapping, LLM style-transfer evaluation, the LKY Voice objective-plus-blind-listening gate, and the LKY Avatar factual-accuracy/persona-quality/fabrication separation. ## Canonical Source Roots - `Projects/Eval Trace/Index.md` - `Knowledge/concepts/context-overfitting.md` - `Knowledge/concepts/style-transfer-evaluation.md` - `Projects/LKY Avatar/Index.md` ## Crosslinks - [[../agent-workflows/README|agent-workflows]] - [[../pixi-vault/README|pixi-vault]] - [[../ai-native-product-surfaces/README|ai-native-product-surfaces]] - [[../local-ai-infrastructure/README|local-ai-infrastructure]] ## Public Output Contract When published to `pixi-wiki`, this namespace should expose: ```text /raw/eval-trace/README.md /raw/eval-trace/wiki/index.md /wiki/eval-trace/README.md /wiki/eval-trace/wiki/index.md ``` ## Maintenance - Edit canonical source notes first. - Use `Wiki Compiler Maps/Namespace Wiki Compiler Map.md` for routing decisions. - Do not compile Daily Notes directly unless promoted or verified. --- title: Context Overfitting created: 2026-06-18 updated: 2026-06-18 type: concept status: compiled namespace: eval-trace tags: [agent-systems, governance, workflow, ai] sources: - /root/.hermes/knowledge/concepts/context-overfitting.md - Knowledge/concepts/context-overfitting.md confidence: high --- # Context Overfitting ## Definition **Context overfitting** is when an agent over-weights written context — memory, skills, dossiers, project notes, cron prompts, prior suggestions, or old issue state — and treats it as a hard constraint even when current user intent, live evidence, or scope boundaries should override it. The written rule may still be useful. The failure is the weighting and scope. ## Current synthesis Use the compact test: ```text Context overfit? YES / NO / UNSURE Suspect source: Evidence: - <1-2 concrete observations> Recommended action: ``` Verdicts: | Verdict | Meaning | Action | |---|---|---| | YES | Written context or stale truth overrode current user intent, live evidence, or scope boundaries. | Demote the rule, scope it, patch the source, update the canonical truth surface, or remove the stale consumer. | | NO | Context was used as a weak prior and the agent stayed steerable. | Keep the rule; no durable change unless another defect appears. | | UNSURE | Evidence is insufficient, or Jamie must judge whether the rule should generalize. | Ask Jamie, inspect the suspect source, or gather better trace evidence before changing durable state. | ## Source-layer priority When context conflicts, prefer current and canonical truth in this order: 1. current user instruction and explicit scope; 2. safety, secrets, data-loss, and approval boundaries; 3. live filesystem/GitHub/runtime evidence; 4. current PRD, issue, route contract, or handoff; 5. project/vault source of truth; 6. local Hermes knowledge concepts; 7. skills and procedures; 8. memory, dossiers, cron prompts, and older session recalls as weak priors. This is a routing rule, not a license to ignore durable context. Written context is useful when it is current, scoped, and backed by live evidence. ## Application Run this check when: - an agent refuses a reasonable correction because an older rule says otherwise; - project-local advice leaks into another project; - stale issue or repo state drives a current recommendation; - a self-improvement loop tries to promote a one-off suggestion into global behavior; - memory or a dossier sounds authoritative but live evidence disagrees; - a route/profile rule is treated as automatic approval when it is only a recommendation. For self-improvement, pair this page with [[self-improving-agent-systems]]: promote only evidence-graded, reversible, layer-fit lessons. For memory and knowledge routing, pair it with [[runtime-memory-knowledge-routing]] and [[profile-memory-boundaries]]. ## Boundaries - Do not use this rubric to bypass safety, secrets, destructive-action, merge, deploy, or live-posting approvals. - Do not patch memory, cron, profiles, providers, gateways, MCP, RAG, or GitHub state merely because a context-overfit risk exists; verify and use the correct change process. - Do not store raw eval traces, issue progress, PR numbers, or command logs in concept pages. - Do not call every stale page context overfitting. Ordinary knowledge rot is stale content; context overfitting is stale or over-scoped content being weighted too strongly during action. ## Related pages - [[self-improving-agent-systems]] - [[runtime-memory-knowledge-routing]] - [[profile-memory-boundaries]] - [[agent-ops-harness-health]] - [[agent-capability-route-pattern]] - [[matt-pocock-sdlc-rhythm]] ## Sources - `/root/ObsidianVault/Knowledge/concepts/context-overfitting.md` --- title: Style-Transfer Evaluation created: 2026-07-10 updated: 2026-07-17 type: concept status: compiled namespace: eval-trace tags: [eval-trace, style-transfer, lora, llm-judge, checkpoint-selection, uncertainty, voice-ai] sources: - Knowledge/concepts/style-transfer-evaluation.md - Projects/LKY Archive/Index.md - Projects/LKY Avatar/Index.md - https://github.com/pixiiidust/lky-brain - https://pixiiidust.github.io/lora-LKY-report/ confidence: high --- # Style-Transfer Evaluation Style-transfer evaluation asks whether an adapter changes named observable behaviors on held-out prompts without confusing persona imitation, lower train loss, or one judge's preference with general reasoning improvement. ## Minimum contract 1. Split by whole source document before creating overlapping windows. 2. Compare base and candidate checkpoints on the same prompts and decoding settings. 3. Score explicit observable traits plus a separate overall voice/style measure. 4. Use deterministic generation or repeated fixed seeds. 5. Blind candidate identity and randomize answer order. 6. Run both reference-free and reference-anchored judge passes when a real answer exists. 7. Calibrate with human review. 8. Aggregate or bootstrap by held-out document, not only by prompt row. ## LKY Brain result On 24 rows from 10 held-out interviews, the epoch-2 QLoRA checkpoint moved judged behavior relative to base Qwen3-14B: - voice: 2.04 → 2.88; - directness: 46% → 88%; - bounded uncertainty: 8% → 33%; - concrete analogy: 0% → 25%; - reframing: 29% → 38%. Epoch 3 reached voice 2.96 and analogy 38%, but lower reframing and bounded uncertainty. Epoch 2 is a defensible provisional checkpoint preference because it retains the subtler traits while tying on broad voice within likely noise. ## Parallel speech-style evaluation The LKY Voice run applies the same evidence discipline to speech generation, where identity, intelligibility, deployment speed, and human recognition must remain separate: | gate | contract | winning Chatterbox LoRA e14 | |---|---|---:| | speaker similarity | meet or beat baseline 0.8693 | 0.8900 | | intelligibility | WER ≤ 0.05 | 0.0390 | | deployability | RTF ≤ 0.6 | 0.381 | | human blind listen | tuned preferred ≥ 70% | 18/20 | | integrated placement | no failures; realtime on the shared GPU | RTF mean/max 0.369/0.397; 0 failures | The rejected GPT-SoVITS arm is the useful counterexample: similarity improved to 0.9049, but WER degraded to 0.1274. The operator's blind listen can veto a numerically attractive model; no one metric decides shipment. Integration also preserved a stock-model rollback and PerTh watermark confidence 1.0000. These results support a tuned-voice preference and deployment decision, not authentic speech, semantic correctness, factual fidelity, or robust pronunciation. Singapore proper nouns and mixed acoustic eras remain separate residual risks. ## Parallel factuality evaluation The first integrated LKY Avatar session passed interaction and voice gates while inventing constituencies, dates, and historical events. The application therefore added an independent factuality lane: | signal | question | |---|---| | factual accuracy | Does the answer contain only correct dates, places, offices, and relationships? | | persona quality | Does grounding preserve concise, recognizable reasoning style? | | fabrication | Did the model invent a date, quote, meeting, office, or constituency? | The implementation uses a small audited fact sheet, deterministic per-turn section retrieval, a source-over-memory block inserted immediately before the latest question, an uncertainty guardrail, and Singapore proper-noun STT keyterms. A 12-question subset supports matched grounding-on/off runs. Tests establish that these seams work as code. They do not establish factual lift. The remaining proof is a real-microphone keyterm check plus a local-brain comparison of factual accuracy, persona quality, and fabrication with grounding enabled and disabled. ## Claim boundary The evidence supports a directional behavioral shift. It does not yet establish statistically stable per-trait lift, factual fidelity, general capability gain, or definitive overfitting because: - the 24 rows are the longest-reference subset of 66 windows rather than a representative random sample; - generation sampled at temperature 0.8 without reported fixed/repeated seeds; - those rows are clustered within 10 source documents; - the same held-out subset selects the checkpoint and reports the final result; - one Claude judge sees a real LKY reference answer; - no confidence intervals, judge-reliability study, or human agreement rate is reported. ## Next evidence gate For model style, evaluate all 66 rows with fixed or repeated seeds, separate checkpoint selection from final testing, publish document-level aggregates and confidence intervals, add a blind/no-reference judge pass, manually review a stratified sample, and check same-event/date leakage across the dialogue and speech streams. For application factuality, run the 12 questions with grounding on/off and report factual accuracy, persona quality, and fabrication separately. Phrase epoch-3 as a possible overfit signal and the retrieval layer as implemented-but-unproven until those checks agree. --- title: Eval Trace created: 2026-06-16 updated: 2026-06-16 type: entity status: compiled namespace: eval-trace tags: [eval-trace, agent-workflows, evaluation, traces] sources: - Projects/Eval Trace/Index.md confidence: medium --- # Eval Trace **Eval Trace** is Jamie's local prototype for evaluating agent-workflow quality, especially context overfitting. It builds trace packets, runs deterministic prechecks, and can hand ambiguous cases to an LLM judge. ## Current artifact ```text .hermes/context-overfit/ ``` Current status: local prototype. It is not yet a public GitHub repo. The repo decision is deferred until the harness proves review-load reduction on real workflow traces. ## Current capabilities - Builds segmented trace packets from Hermes session exports. - Runs deterministic prechecks before any LLM judge. - Emits compact `judge_packets.jsonl` for reviewable `UNSURE` cases. - Supports real LLM judge passes when Jamie asks Pixoid to judge. - Reads `.hermes/knowledge/registry/project-status.json` for done/deprecated/replaced project status. ## Next gate Before a standalone `pixiiidust/eval-trace` repo exists, prove value on real workflow traces: add sequence-level cron checks, run on recent crew outputs, and show a confirmed real `YES` or useful `UNSURE` clustering. ## Source Compiled from `Projects/Eval Trace/Index.md`. --- title: Eval Trace — Master Index created: 2026-06-16 updated: 2026-07-17 type: index status: compiled namespace: eval-trace --- # Eval Trace — Master Index > Compiled index for `eval-trace`. ## Concepts - [[concepts/context-overfitting|Context Overfitting]] — Agent workflow failure mode where stale or overly rigid context overwhelms current intent/live evidence. - [[concepts/style-transfer-evaluation|Style-Transfer Evaluation]] — LLM document holdouts and trait rubrics; tuned-voice similarity, intelligibility, realtime, blind-listening, placement, watermark, and rollback; plus separate factual accuracy, persona quality, and fabrication signals for grounded applications. ## Entities - [[entities/eval-trace|Eval Trace]] — Local prototype for context-overfit trace packets, deterministic prechecks, and LLM judge review. - Cross-namespace entity: [[../../agent-workflows/wiki/entities/hermes-mission-control|Hermes Mission Control]] — Agent Workflows primary page for route governance and crew operating surfaces. ## Summaries - Cross-namespace summary: [[../../hermes-agent/wiki/summaries/external-hermes-wikis-import-review|External Hermes Wikis Import Review]] — Routes auxiliary-model and evaluation/trace-adjacent Hermes content back to Hermes Agent with Eval Trace crosslinks. ## Syntheses - [[syntheses/workflow-quality-evaluation-map|Workflow Quality Evaluation Map]] — Evaluation surfaces, gates, and evidence rules for agent workflow quality. ## Source Roots - `Projects/Eval Trace/Index.md` - `Knowledge/concepts/context-overfitting.md` - `Knowledge/concepts/style-transfer-evaluation.md` - `Projects/LKY Avatar/Index.md` --- title: Eval Trace — Activity Log created: 2026-06-16 updated: 2026-07-17 type: log status: compiled namespace: eval-trace --- # Eval Trace — Activity Log > Append-only namespace log. ## 2026-07-17 update | Separate factual grounding from persona quality - Extended the style-transfer page with the LKY Avatar factuality lane after issue #45 / PR #47 merged. - Added the audited evidence, per-turn retrieval, uncertainty, proper-noun input, and 12-question matched-eval contract. - Kept tests as implementation evidence only; live microphone and local-brain grounding-on/off results remain the quality gate. ## 2026-07-15 update | Extend style-transfer evaluation to tuned voice - Added the LKY Voice parallel case to `wiki/concepts/style-transfer-evaluation.md`. - Preserved separate gates for identity similarity, WER intelligibility, realtime factor, blind human preference, same-GPU placement, watermarking, and rollback. - Recorded GPT-SoVITS as the counterexample where higher similarity did not compensate for failed intelligibility, and kept pronunciation/authenticity/factuality outside the claim. ## 2026-06-16 create | Namespace scaffold initialized - Created README, CLAUDE instructions, raw folder, index/log, and typed wiki folders. - Source routing comes from `Wiki Compiler Maps/Namespace Wiki Compiler Map.md`. - No Daily Notes were copied or compiled. ## 2026-06-16 update | Pilot compiled namespace page - Added pilot concept `wiki/concepts/context-overfitting.md` and crosslink to Agent Workflows Hermes Mission Control entity. - Source pages remained in `Knowledge/` and `Projects/`. - No Daily Notes were copied or compiled. ## 2026-06-16 update | Compile eval-trace content pack v1 - Expanded `wiki/concepts/context-overfitting.md`. - Added entity `wiki/entities/eval-trace.md`. - Added synthesis `wiki/syntheses/workflow-quality-evaluation-map.md`. - Updated namespace index. - Source pages remain in `Knowledge/` and `Projects/`. - No Daily Notes were copied or compiled. ## 2026-06-18 update | Crosslink Hermes eval-adjacent imports - Added cross-namespace pointer to the Hermes Agent external wiki import review for auxiliary-model and trace/eval-adjacent routing. - No Daily Notes were copied or compiled. ## 2026-06-18 update | Refresh context-overfitting from local Hermes KB closure - Refreshed `wiki/concepts/context-overfitting.md` with the verified YES/NO/UNSURE rubric and source-layer priority from the local Hermes KB tracker closure. - Preserved the boundary that context-overfit checks do not bypass safety, approval, profile, cron, or deployment gates. ## 2026-07-10 add | Style-transfer evaluation contract from LKY Brain - Added `wiki/concepts/style-transfer-evaluation.md` from the completed LKY Brain base/epoch-2/epoch-3 comparison. - Preserved the strong parts: whole-document holdout, named behavior rubric, matched candidates, checkpoint preservation, and answer-level inspection. - Tightened the claim boundary around n=24, 10 source documents, stochastic generation, one reference-anchored judge, no confidence intervals, and non-random longest-reference selection. - Recorded the next gate: all 66 rows, fixed/repeated seeds, document-level uncertainty, blind/no-reference judging, and human calibration. --- title: Workflow Quality Evaluation Map created: 2026-06-16 updated: 2026-06-16 type: synthesis status: compiled namespace: eval-trace tags: [eval-trace, workflow-quality, evidence-gates, agent-workflows] sources: - Projects/Eval Trace/Index.md - Knowledge/concepts/context-overfitting.md - wikis/agent-workflows/wiki/syntheses/pixoid-crew-operating-model.md confidence: medium --- # Workflow Quality Evaluation Map Workflow quality evaluation asks whether an agent's output followed the right source-of-truth hierarchy, adapted to current user intent, and produced verified artifacts instead of plausible process talk. ## Evaluation surfaces | Surface | Quality question | Evidence | |---|---|---| | Source routing | Did the agent use the correct truth layer? | GitHub issue/PR state, Obsidian project hubs, Knowledge pages, Wiki Compiler Maps | | Context adaptation | Did the agent update from latest user intent and live checks? | Current message, live git/GitHub/tool output | | Artifact progress | Did it produce a working artifact? | Committed files, tests, generated output, live URL checks | | Review discipline | Did Pixoid verify before closing? | Test logs, API checks, issue comments, clean status | | Safety boundary | Did it stop before risky actions? | Explicit approvals for deletes, deploys, secrets, broad rewrites | ## Evidence gates 1. **Pre-flight gate:** inspect live repo/tracker/source state before editing. 2. **Revision gate:** if checks fail, fix and rerun focused verification. 3. **Escalation gate:** ask Jamie before destructive or ambiguous ownership changes. 4. **Closure gate:** close issues only after pushed commits, tests, and live/remote verification. ## Context-overfit signal A strong context-overfit signal appears when the agent follows a written prior that should have been overridden by a newer user instruction, current project status, or live tool output. The evaluation target is not whether context was used; it is whether context was treated as stronger than the right source of truth. ## Cross-namespace links - [[../../../agent-workflows/wiki/syntheses/pixoid-crew-operating-model|Pixoid Crew Operating Model]] — operational source hierarchy. - [[concepts/context-overfitting|Context Overfitting]] — primary failure-mode concept.