mirror of
https://github.com/rennf93/roboco.git
synced 2026-08-03 07:23:24 +02:00
* feat(lifecycle): revision findings ledger — structured QA/PR/PM/CEO failure feedback, persisted and delivered down the chain Every bounce used to survive only as flattened prose: rounds overwrote each other in notes_structured, request_changes persisted nothing, two raw dev_notes appends were silently destroyed by the next handoff note, and the dev prompt pointed at fields (qa_notes via evidence(), pm_notes) the API never delivered. Agents re-interpreted and re-discovered every failure before they could start fixing it. - task_review_findings (migration 071, append-only): file/line/severity/ criterion(AC-id-validated)/expected/actual/fix/evidence per finding, with origin (qa|pr_gate|pm|ceo), round, and an open->addressed->verified lifecycle (waived reserved); new tasks.pm_notes + PmReviewContent give request_changes a structured home - producers: fail_review/pr_fail/request_changes take findings=[...] (prose issues shimmed+merged for one release, deprecation-logged); ceo_reject validates its reason (no 500), lands an origin=ceo finding, and bumps round+audit on branchless coordination roots; guardrails at the verb chokepoint (nudge >5, hard reject >10, field caps, traversal-safe file); the dev_notes data-loss appends are removed; new task.request_changes + task.ceo_reject audit events close rework attribution - delivery: qa_notes/pr_reviewer_notes/pm_notes carry the deterministic [F-id8] rendering; claim briefings, evidence(), the REVISION_REQUIRED spawn prompt, PM triage bounced-blocks, and A2A bodies deliver open findings; round-N+1 QA and gate reviewers get the full prior ledger; panel Findings tab + bounced-xN chip; metrics pm_rejects/ceo_rejects + findings counts; vault task notes render a Findings section (fail-open) - resolution closes for every origin: i_am_done and submit_up/submit_root take resolved_findings gated by FINDINGS_ADDRESSED (owner-gated so a stale non-owner PM can never mutate the ledger); pass_review/pr_pass/ complete verify-stamp same-transaction; ceo_approve stamps best-effort - 24 real-DB integration tests drive the full loop through the real choreographer; full suite 12856 green * docs: revision findings ledger sweep — CLAUDE.md, map, RAG corpus - CLAUDE.md: new ledger section + corrected request_changes row - docs/map/review-findings.md (new subsystem map) + surgical updates to task-service/pr-gate-review/metrics-observability/vault/panel maps - docs/rag: producers' findings contract across qa/pr-reviewer/developer/ cell-pm/main-pm/ceo role docs (the PM docs were missing request_changes entirely), verb references, and a new architecture/review-findings.md disambiguating ledger findings from convention findings * test(e2e): resubmit resolves the pr_fail finding per the ledger contract The scripted pr_fail revision loop resubmitted submit_up without resolved_findings — correctly rejected now that FINDINGS_ADDRESSED gates the PM resubmit verbs (green locally, red only in CI since the e2e suite skips without ROBOCO_E2E_SMOKE=1). The scripted PM now reads the open ledger row pr_fail persisted (new open_finding_ids arc helper) and resolves it on resubmit, asserting the open set drains — exercising the coordinator half of the new contract end to end. --------- Co-authored-by: Renn F <rennf93@users.noreply.github.com>
132 lines
5.1 KiB
Markdown
132 lines
5.1 KiB
Markdown
# QA Review Workflow
|
|
|
|
## Preconditions
|
|
|
|
- Task is in `awaiting_qa` status
|
|
- The developer's PR is open (the choreographer opened it during their `open_pr(task_id)` call)
|
|
- You are not the original developer of the task (self-review guard)
|
|
|
|
## Steps
|
|
|
|
```python
|
|
# 1. Pick up an awaiting-QA task
|
|
give_me_work()
|
|
|
|
# 2. Claim it for review (auto-checks-out the dev's branch in your
|
|
# workspace; auto-records original_developer for the self-review
|
|
# guard at pass/fail time)
|
|
claim_review(task_id="<task>")
|
|
|
|
# 3. Inspect the diff (project_slug is optional — omit it and your
|
|
# own project is used)
|
|
roboco_git_diff()
|
|
roboco_git_log(branch="<dev's branch>")
|
|
|
|
# 4. Run the relevant suite
|
|
# Backend: uv run pytest && uv run ruff check . && uv run mypy roboco/
|
|
# Frontend: pnpm test && pnpm lint && pnpm typecheck
|
|
|
|
# 5. Capture evidence (survives compaction; PMs can audit later)
|
|
note(text="Verified AC #1 (429 on 101st req), #2 (TTL match), #3 "
|
|
"(boundary tests). pytest 1635 passed; ruff clean; mypy clean.",
|
|
scope="evidence",
|
|
task_id="<task>")
|
|
```
|
|
|
|
There is no `roboco_task_claim / _start / _qa_pass / _qa_fail` and no `roboco_git_checkout`. The verbs above (`claim_review`, `pass`, `fail`) are the actual surface; branch checkout is a side-effect of `claim_review`.
|
|
|
|
You always claim the review yourself — the dispatcher spawns you against an `awaiting_qa` task without pre-claiming it. `claim_review` records your claim but keeps the status at `awaiting_qa` (there is no `claimed` detour), so `pass`/`fail` find the status they demand.
|
|
|
|
## Review Checklist
|
|
|
|
Before deciding:
|
|
|
|
- [ ] Read the dev's notes and journal entries on the task
|
|
- [ ] Walk every acceptance criterion against the diff
|
|
- [ ] Tests pass on the dev's branch
|
|
- [ ] Lint / typecheck clean
|
|
- [ ] No layer-separation regressions (routes/ vs services/ etc.)
|
|
- [ ] No silenced rules (`# noqa`, `# type: ignore`, `# pragma: no cover`)
|
|
- [ ] Code matches project standards in CLAUDE.md
|
|
|
|
## Passing QA
|
|
|
|
```python
|
|
pass(
|
|
task_id="<task>",
|
|
notes=(
|
|
"All 3 acceptance criteria verified against the diff. "
|
|
"pytest 1635 passed; ruff and mypy clean. "
|
|
"PR #123."
|
|
),
|
|
)
|
|
```
|
|
|
|
Result:
|
|
|
|
- Task advances to `awaiting_documentation`
|
|
- Documenter and the original dev work in parallel from here
|
|
- The PR stays open; it will be merged later by the Cell PM via `complete(task_id, ...)`
|
|
|
|
## Failing QA
|
|
|
|
```python
|
|
fail(
|
|
task_id="<task>",
|
|
findings=[
|
|
{
|
|
"file": "roboco/api/routes/rate_limit.py",
|
|
"line": 88,
|
|
"severity": "blocker",
|
|
"expected": "429 on the 101st request in the window",
|
|
"actual": "the 100th request also returns 429 — boundary off-by-one",
|
|
"fix": "use > not >= when comparing against the window limit",
|
|
},
|
|
{
|
|
"severity": "major",
|
|
"criterion": "AC #3 — Redis-down failover path",
|
|
"expected": "a test covering the Redis-down failover path",
|
|
"actual": "no such test exists in this diff",
|
|
},
|
|
],
|
|
)
|
|
```
|
|
|
|
Each finding is validated, persisted onto the task's append-only revision-findings ledger (`origin=qa`, `round=revision_count+1`), and rendered into `qa_notes` as `[F-xxxxxxxx] file:line (severity) — expected → actual → fix`. The old `issues=[...]` (plain strings) form still works this release but is deprecated — each becomes a file-less `severity=major` finding. A soft nudge fires above 5 findings in one call, a hard reject above 10.
|
|
|
|
Result:
|
|
|
|
- Task returns to `needs_revision`
|
|
- Re-assigned to the original developer (recorded at submit-for-qa time)
|
|
- Developer receives a notification, and the open findings arrive inline via `evidence()`'s `revision_findings` and the respawn prompt — see `docs/rag/architecture/review-findings.md`
|
|
|
|
## Re-reviewing a bounced task (round ≥2)
|
|
|
|
If the task has failed before, `claim_review` returns `prior_findings` — the FULL ledger, every round, newest first — alongside the usual PR diff. Check each prior finding against the current diff one at a time before deciding: a finding still unaddressed is a fail, not a pass with a note. Passing (`pass`) bulk-verifies every `addressed` QA-origin finding in the same transaction — that verification IS the confirmation the fix landed.
|
|
|
|
## Reflect (recommended)
|
|
|
|
After pass or fail, journal the review for future QA agents to learn from:
|
|
|
|
```python
|
|
note(
|
|
text=(
|
|
"Reviewed task <id>. Pattern: rate-limiter boundary tests "
|
|
"should always assert the off-by-one — caught it in this "
|
|
"review and last week's. Worth a regression checklist item."
|
|
),
|
|
scope="reflect",
|
|
task_id="<task>",
|
|
)
|
|
```
|
|
|
|
## Self-Review Prevention
|
|
|
|
The system blocks QA from reviewing their own dev work. The original developer is recorded in `quick_context` at submit-for-qa time. If `qa_agent_id == original_developer_id`, **all** QA actions on the task return `not_authorized`:
|
|
|
|
- `claim_review` — FORBIDDEN
|
|
- `pass` — FORBIDDEN (defence-in-depth even if claim somehow succeeded)
|
|
- `fail` — FORBIDDEN (same)
|
|
|
|
Enforced at the gateway layer in `roboco/services/gateway/choreographer/_impl.py`.
|