Workflow Quality Evaluation Map¶
Workflow quality evaluation asks whether an agent's output followed the right source-of-truth hierarchy, adapted to current user intent, and produced verified artifacts instead of plausible process talk.
Evaluation surfaces¶
| Surface | Quality question | Evidence |
|---|---|---|
| Source routing | Did the agent use the correct truth layer? | GitHub issue/PR state, Obsidian project hubs, Knowledge pages, Wiki Compiler Maps |
| Context adaptation | Did the agent update from latest user intent and live checks? | Current message, live git/GitHub/tool output |
| Artifact progress | Did it produce a working artifact? | Committed files, tests, generated output, live URL checks |
| Review discipline | Did Pixoid verify before closing? | Test logs, API checks, issue comments, clean status |
| Safety boundary | Did it stop before risky actions? | Explicit approvals for deletes, deploys, secrets, broad rewrites |
Evidence gates¶
- Pre-flight gate: inspect live repo/tracker/source state before editing.
- Revision gate: if checks fail, fix and rerun focused verification.
- Escalation gate: ask Jamie before destructive or ambiguous ownership changes.
- Closure gate: close issues only after pushed commits, tests, and live/remote verification.
Context-overfit signal¶
A strong context-overfit signal appears when the agent follows a written prior that should have been overridden by a newer user instruction, current project status, or live tool output. The evaluation target is not whether context was used; it is whether context was treated as stronger than the right source of truth.
Cross-namespace links¶
- Pixoid Crew Operating Model — operational source hierarchy.
- Context Overfitting — primary failure-mode concept.