feat: integrated new playwright mcp

This commit is contained in:
Murat Ozcan
2025-10-15 17:14:39 -05:00
parent 633abf0c3b
commit ec486af453
50 changed files with 15437 additions and 2401 deletions

View File

@@ -1,4 +1,4 @@
# Requirements Traceability Workflow
# Requirements Traceability & Quality Gate Workflow
**Workflow ID:** `testarch-trace`
**Agent:** Test Architect (TEA)
@@ -8,14 +8,22 @@
## Overview
The **trace** workflow generates a comprehensive requirements-to-tests traceability matrix that maps acceptance criteria to implemented tests, identifies coverage gaps, and provides actionable recommendations for improving test coverage.
The **trace** workflow operates in two sequential phases to validate test coverage and deployment readiness:
**PHASE 1 - REQUIREMENTS TRACEABILITY:** Generates comprehensive requirements-to-tests traceability matrix that maps acceptance criteria to implemented tests, identifies coverage gaps, and provides actionable recommendations.
**PHASE 2 - QUALITY GATE DECISION:** Makes deterministic release decisions (PASS/CONCERNS/FAIL/WAIVED) based on traceability results, test execution evidence, and non-functional requirements validation.
**Key Features:**
- Maps acceptance criteria to specific test cases across all levels (E2E, API, Component, Unit)
- Classifies coverage status (FULL, PARTIAL, NONE, UNIT-ONLY, INTEGRATION-ONLY)
- Prioritizes gaps by risk level (P0/P1/P2/P3)
- Generates CI/CD-ready YAML snippets for quality gates
- Applies deterministic decision rules for deployment readiness
- Generates gate decisions with evidence and rationale
- Supports waivers for business-approved exceptions
- Updates workflow status and notifies stakeholders
- Creates CI/CD-ready YAML snippets for quality gates
- Detects duplicate coverage across test levels
- Verifies test quality (assertions, structure, performance)
@@ -25,33 +33,49 @@ The **trace** workflow generates a comprehensive requirements-to-tests traceabil
Use `*trace` when you need to:
### Phase 1 - Traceability
- ✅ Validate that all acceptance criteria have test coverage
- ✅ Identify coverage gaps before release or PR merge
- ✅ Generate traceability documentation for compliance or audits
- ✅ Ensure critical paths (P0/P1) are fully tested
- ✅ Detect duplicate coverage across test levels
- ✅ Assess test quality across your suite
- ✅ Create gate-ready metrics for CI/CD pipelines
### Phase 2 - Gate Decision (Optional)
- ✅ Make final go/no-go deployment decision
- ✅ Validate test execution results against thresholds
- ✅ Evaluate non-functional requirements (security, performance)
- ✅ Generate audit trail for release approval
- ✅ Handle business waivers for critical deadlines
- ✅ Notify stakeholders of gate decision
**Typical Timing:**
- After tests are implemented (post-ATDD or post-development)
- Before merging a PR (validate P0/P1 coverage)
- Before release (validate full coverage)
- Before release (validate full coverage and make gate decision)
- During sprint retrospectives (assess test quality)
---
## Prerequisites
**Required:**
### Phase 1 - Traceability (Required)
- Acceptance criteria (from story file OR inline)
- Implemented test suite (or acknowledged gaps)
**Recommended:**
### Phase 2 - Gate Decision (Required if `enable_gate_decision: true`)
- Test execution results (CI/CD test reports, pass/fail rates)
- Test design with risk priorities (P0/P1/P2/P3)
### Recommended
- `test-design.md` - Risk assessment and test priorities
- `nfr-assessment.md` - Non-functional requirements validation (for release gates)
- `tech-spec.md` - Technical implementation details
- Test framework configuration (playwright.config.ts, jest.config.js)
@@ -59,12 +83,13 @@ Use `*trace` when you need to:
- Story lacks any tests AND gaps are not acknowledged → Run `*atdd` first
- Acceptance criteria are completely missing → Provide criteria or story file
- Phase 2 enabled but test execution results missing → Warn and skip gate decision
---
## Usage
### Basic Usage (BMad Mode)
### Basic Usage (Both Phases)
```bash
bmad tea *trace
@@ -72,16 +97,15 @@ bmad tea *trace
The workflow will:
1. Read story file from `bmad/output/story-X.X.md`
2. Extract acceptance criteria
3. Auto-discover tests for this story
4. Generate traceability matrix
5. Save to `bmad/output/traceability-matrix.md`
1. **Phase 1**: Read story file, extract acceptance criteria, auto-discover tests, generate traceability matrix
2. **Phase 2**: Load test execution results, apply decision rules, generate gate decision document
3. Save traceability matrix to `bmad/output/traceability-matrix.md`
4. Save gate decision to `bmad/output/gate-decision-story-X.X.md`
### Standalone Mode (No Story File)
### Phase 1 Only (Skip Gate Decision)
```bash
bmad tea *trace --acceptance-criteria "AC-1: User can login with email..."
bmad tea *trace --enable-gate-decision false
```
### Custom Configuration
@@ -89,15 +113,25 @@ bmad tea *trace --acceptance-criteria "AC-1: User can login with email..."
```bash
bmad tea *trace \
--story-file "bmad/output/story-1.3.md" \
--output-file "docs/qa/trace-1.3.md" \
--test-results "ci-artifacts/test-report.xml" \
--min-p0-coverage 100 \
--min-p1-coverage 90
--min-p1-coverage 90 \
--min-p0-pass-rate 100 \
--min-p1-pass-rate 95
```
### Standalone Mode (No Story File)
```bash
bmad tea *trace --acceptance-criteria "AC-1: User can login with email..."
```
---
## Workflow Steps
### PHASE 1: Requirements Traceability
1. **Load Context** - Read story, test design, tech spec, knowledge base
2. **Discover Tests** - Auto-find tests related to story (by ID, describe blocks, file paths)
3. **Map Criteria** - Link acceptance criteria to specific test cases
@@ -105,11 +139,18 @@ bmad tea *trace \
5. **Verify Quality** - Check test quality (assertions, structure, performance)
6. **Generate Deliverables** - Create traceability matrix, gate YAML, coverage badge
### PHASE 2: Quality Gate Decision (if `enable_gate_decision: true`)
7. **Gather Evidence** - Load traceability results, test execution reports, NFR assessments
8. **Apply Decision Rules** - Evaluate against thresholds (PASS/CONCERNS/FAIL/WAIVED)
9. **Document Decision** - Create gate decision document with evidence and rationale
10. **Update Status & Notify** - Append to bmm-workflow-status.md, notify stakeholders
---
## Outputs
### Traceability Matrix (`traceability-matrix.md`)
### Phase 1: Traceability Matrix (`traceability-matrix.md`)
Comprehensive markdown file with:
@@ -119,32 +160,136 @@ Comprehensive markdown file with:
- Quality assessment for each test
- Gate YAML snippet
### Gate YAML Snippet
**Example:**
```yaml
traceability:
story_id: '1.3'
coverage:
overall: 85%
p0: 100%
p1: 90%
gaps:
critical: 0
high: 1
status: 'PASS'
```markdown
# Traceability Matrix - Story 1.3
## Coverage Summary
| Priority | Total | FULL | Coverage % | Status |
| -------- | ----- | ---- | ---------- | ------- |
| P0 | 3 | 3 | 100% | ✅ PASS |
| P1 | 5 | 4 | 80% | ⚠️ WARN |
Gate Status: CONCERNS ⚠️ (P1 coverage below 90%)
```
### Updated Story File (Optional)
### Phase 2: Gate Decision Document (`gate-decision-{type}-{id}.md`)
Adds "Traceability" section to story markdown with:
**Decision Document** with:
- Link to traceability matrix
- Coverage summary
- Gate status
- **Decision**: PASS / CONCERNS / FAIL / WAIVED with clear rationale
- **Evidence Summary**: Test results, coverage, NFRs, quality validation
- **Decision Criteria Table**: Each criterion with threshold, actual, status
- **Rationale**: Explanation of decision based on evidence
- **Residual Risks**: Unresolved issues (for CONCERNS/WAIVED)
- **Waiver Details**: Approver, justification, remediation plan (for WAIVED)
- **Next Steps**: Action items for each decision type
**Example:**
```markdown
# Quality Gate Decision: Story 1.3 - User Login
**Decision**: ⚠️ CONCERNS
**Date**: 2025-10-15
## Decision Criteria
| Criterion | Threshold | Actual | Status |
| ------------ | --------- | ------ | ------- |
| P0 Coverage | ≥100% | 100% | ✅ PASS |
| P1 Coverage | ≥90% | 88% | ⚠️ FAIL |
| Overall Pass | ≥90% | 96% | ✅ PASS |
**Decision**: CONCERNS (P1 coverage 88% below 90% threshold)
## Next Steps
- Deploy with monitoring
- Create follow-up story for AC-5 test
```
### Secondary Outputs
- **Gate YAML**: Machine-readable snippet for CI/CD integration
- **Status Update**: Appends decision to `bmm-workflow-status.md` history
- **Stakeholder Notification**: Auto-generated summary message
- **Updated Story File**: Traceability section added (optional)
---
## Coverage Classifications
## Decision Logic (Phase 2)
### PASS Decision ✅
**All criteria met:**
- ✅ P0 coverage ≥ 100%
- ✅ P1 coverage ≥ 90%
- ✅ Overall coverage ≥ 80%
- ✅ P0 test pass rate = 100%
- ✅ P1 test pass rate ≥ 95%
- ✅ Overall test pass rate ≥ 90%
- ✅ Security issues = 0
- ✅ Critical NFR failures = 0
**Action:** Deploy to production with standard monitoring
---
### CONCERNS Decision ⚠️
**P0 criteria met, but P1 criteria degraded:**
- ✅ P0 coverage = 100%
- ⚠️ P1 coverage 80-89% (below 90% threshold)
- ⚠️ P1 test pass rate 90-94% (below 95% threshold)
- ✅ No security issues
- ✅ No critical NFR failures
**Residual Risks:** Minor P1 issues, edge cases, non-critical gaps
**Action:** Deploy with enhanced monitoring, create backlog stories for fixes
**Note:** CONCERNS does NOT block deployment but requires acknowledgment
---
### FAIL Decision ❌
**Any P0 criterion failed:**
- ❌ P0 coverage <100% (missing critical tests)
- OR ❌ P0 test pass rate <100% (failing critical tests)
- OR ❌ P1 coverage <80% (significant gap)
- OR ❌ Security issues >0
- OR ❌ Critical NFR failures >0
**Critical Blockers:** P0 test failures, security vulnerabilities, critical NFRs
**Action:** Block deployment, fix critical issues, re-run gate after fixes
---
### WAIVED Decision 🔓
**FAIL status + business-approved waiver:**
- ❌ Original decision: FAIL
- 🔓 Waiver approved by: {VP Engineering / CTO / Product Owner}
- 📋 Business justification: {regulatory deadline, contractual obligation}
- 📅 Waiver expiry: {date - does NOT apply to future releases}
- 🔧 Remediation plan: {fix in next release, due date}
**Action:** Deploy with business approval, aggressive monitoring, fix ASAP
**Important:** Waivers NEVER apply to P0 security issues or data corruption risks
---
## Coverage Classifications (Phase 1)
- **FULL** ✅ - All scenarios validated at appropriate level(s)
- **PARTIAL** ⚠️ - Some coverage but missing edge cases or levels
@@ -156,12 +301,12 @@ Adds "Traceability" section to story markdown with:
## Quality Gates
| Priority | Coverage Requirement | Severity | Action |
| -------- | -------------------- | -------- | ------------------ |
| P0 | 100% | BLOCKER | Do not release |
| P1 | 90% | HIGH | Block PR merge |
| P2 | 80% (recommended) | MEDIUM | Address in nightly |
| P3 | No requirement | LOW | Optional |
| Priority | Coverage Requirement | Pass Rate Requirement | Severity | Action |
| -------- | -------------------- | --------------------- | -------- | ------------------ |
| P0 | 100% | 100% | BLOCKER | Do not release |
| P1 | 90% | 95% | HIGH | Block PR merge |
| P2 | 80% (recommended) | 85% (recommended) | MEDIUM | Address in nightly |
| P3 | No requirement | No requirement | LOW | Optional |
---
@@ -196,10 +341,47 @@ variables:
generate_coverage_badge: true
update_story_file: true
# Quality gates
# Quality gates (Phase 1 recommendations)
min_p0_coverage: 100
min_p1_coverage: 90
min_overall_coverage: 80
# PHASE 2: Gate Decision Variables
enable_gate_decision: true # Run gate decision after traceability
# Gate target specification
gate_type: 'story' # story | epic | release | hotfix
# Gate decision configuration
decision_mode: 'deterministic' # deterministic | manual
allow_waivers: true
require_evidence: true
# Input sources for gate
nfr_file: '' # Path to nfr-assessment.md (optional)
test_results: '' # Path to test execution results (required for Phase 2)
# Decision criteria thresholds
min_p0_pass_rate: 100
min_p1_pass_rate: 95
min_overall_pass_rate: 90
max_critical_nfrs_fail: 0
max_security_issues: 0
# Risk tolerance
allow_p2_failures: true
allow_p3_failures: true
escalate_p1_failures: true
# Gate output configuration
gate_output_file: '{output_folder}/gate-decision-{gate_type}-{story_id}.md'
append_to_history: true
notify_stakeholders: true
# Advanced gate options
check_all_workflows_complete: true
validate_evidence_freshness: true
require_sign_off: false
```
---
@@ -208,24 +390,34 @@ variables:
This workflow automatically loads relevant knowledge fragments:
**Phase 1 (Traceability):**
- `traceability.md` - Requirements mapping patterns
- `test-priorities.md` - P0/P1/P2/P3 risk framework
- `risk-governance.md` - Risk-based testing approach
- `test-quality.md` - Definition of Done for tests
- `selective-testing.md` - Duplicate coverage patterns
**Phase 2 (Gate Decision):**
- `risk-governance.md` - Quality gate criteria and decision framework
- `probability-impact.md` - Risk scoring for residual risks
- `test-quality.md` - Quality standards validation
- `test-priorities.md` - Priority classification framework
---
## Examples
## Example Scenarios
### Example 1: Full Coverage Validation
### Example 1: Full Coverage with Gate PASS
```bash
# Validate P0/P1 coverage before PR merge
bmad tea *trace --story-file "bmad/output/story-1.3.md"
# Validate coverage and make gate decision
bmad tea *trace --story-file "bmad/output/story-1.3.md" \
--test-results "ci-artifacts/test-report.xml"
```
**Output:**
**Phase 1 Output:**
```markdown
# Traceability Matrix - Story 1.3
@@ -237,17 +429,42 @@ bmad tea *trace --story-file "bmad/output/story-1.3.md"
| P0 | 3 | 3 | 100% | ✅ PASS |
| P1 | 5 | 5 | 100% | ✅ PASS |
Gate Status: PASS
Gate Status: Ready for Phase 2
```
### Example 2: Gap Identification
**Phase 2 Output:**
```markdown
# Quality Gate Decision: Story 1.3
**Decision**: ✅ PASS
Evidence:
- P0 Coverage: 100% ✅
- P1 Coverage: 100% ✅
- P0 Pass Rate: 100% (12/12 tests) ✅
- P1 Pass Rate: 98% (45/46 tests) ✅
- Overall Pass Rate: 96% ✅
Next Steps:
1. Deploy to staging
2. Monitor for 24 hours
3. Deploy to production
```
---
### Example 2: Gap Identification with CONCERNS Decision
```bash
# Find coverage gaps for existing feature
bmad tea *trace --target-feature "user-authentication"
# Find gaps and evaluate readiness
bmad tea *trace --story-file "bmad/output/story-2.1.md" \
--test-results "ci-artifacts/test-report.xml"
```
**Output:**
**Phase 1 Output:**
```markdown
## Gap Analysis
@@ -263,70 +480,201 @@ bmad tea *trace --target-feature "user-authentication"
- Impact: Users may not recover accounts in error scenarios
```
### Example 3: Duplicate Coverage Detection
```bash
# Check for redundant tests
bmad tea *trace --check-duplicate-coverage true
```
**Output:**
**Phase 2 Output:**
```markdown
## Duplicate Coverage Detected
# Quality Gate Decision: Story 2.1
⚠️ AC-1 (login validation) is tested at multiple levels:
**Decision**: ⚠️ CONCERNS
- 1.3-E2E-001 (full user journey) ✅ Appropriate
- 1.3-UNIT-001 (business logic) ✅ Appropriate
- 1.3-COMPONENT-001 (form validation) ⚠️ Redundant with UNIT-001
Evidence:
Recommendation: Remove 1.3-COMPONENT-001 or consolidate with UNIT-001
- P0 Coverage: 100% ✅
- P1 Coverage: 88% ⚠️ (below 90%)
- Test Pass Rate: 96% ✅
Residual Risks:
- AC-3 missing E2E test for email error handling
Next Steps:
- Deploy with monitoring
- Create follow-up story for AC-3 test
- Monitor production for edge cases
```
---
### Example 3: Critical Blocker with FAIL Decision
```bash
# Critical issues detected
bmad tea *trace --story-file "bmad/output/story-3.2.md" \
--test-results "ci-artifacts/test-report.xml"
```
**Phase 1 Output:**
```markdown
## Gap Analysis
### Critical Gaps (BLOCKER)
1. **AC-2: Invalid login security validation**
- Priority: P0
- Status: NONE (no tests)
- Impact: Security vulnerability - users can bypass login
```
**Phase 2 Output:**
```markdown
# Quality Gate Decision: Story 3.2
**Decision**: ❌ FAIL
Critical Blockers:
- P0 Coverage: 80% ❌ (AC-2 missing)
- Security Risk: Login bypass vulnerability
Next Steps:
1. BLOCK DEPLOYMENT IMMEDIATELY
2. Add P0 test for AC-2: 1.3-E2E-004
3. Re-run full test suite
4. Re-run gate after fixes verified
```
---
### Example 4: Business Override with WAIVED Decision
```bash
# FAIL with business waiver
bmad tea *trace --story-file "bmad/output/release-2.4.0.md" \
--test-results "ci-artifacts/test-report.xml" \
--allow-waivers true
```
**Phase 2 Output:**
```markdown
# Quality Gate Decision: Release 2.4.0
**Original Decision**: ❌ FAIL
**Final Decision**: 🔓 WAIVED
Waiver Details:
- Approver: Jane Doe, VP Engineering
- Reason: GDPR compliance deadline (regulatory, Oct 15)
- Expiry: 2025-10-15 (does NOT apply to v2.5.0)
- Monitoring: Enhanced error tracking
- Remediation: Fix in v2.4.1 hotfix (due Oct 20)
Business Justification:
Release contains critical GDPR features required by law. Failed
test affects legacy feature used by <1% of users. Workaround available.
Next Steps:
1. Deploy v2.4.0 with waiver approval
2. Monitor error rates aggressively
3. Fix issue in v2.4.1 (Oct 20)
```
---
## Troubleshooting
### "No tests found for this story"
### Phase 1 Issues
#### "No tests found for this story"
- Run `*atdd` workflow first to generate failing acceptance tests
- Check test file naming conventions (may not match story ID pattern)
- Verify test directory path is correct (`test_dir` variable)
### "Cannot determine coverage status"
#### "Cannot determine coverage status"
- Tests may lack explicit mapping (no test IDs, unclear describe blocks)
- Add test IDs: `{STORY_ID}-{LEVEL}-{SEQ}` (e.g., `1.3-E2E-001`)
- Use Given-When-Then narrative in test descriptions
### "P0 coverage below 100%"
#### "P0 coverage below 100%"
- This is a **BLOCKER** - do not release
- Identify missing P0 tests in gap analysis
- Run `*atdd` workflow to generate missing tests
- Verify P0 classification is correct with stakeholders
### "Duplicate coverage detected"
#### "Duplicate coverage detected"
- Review `selective-testing.md` knowledge fragment
- Determine if overlap is acceptable (defense in depth) or wasteful
- Consolidate tests at appropriate level (logic → unit, journey → E2E)
### Phase 2 Issues
#### "Test execution results missing"
- Phase 2 gate decision requires `test_results` (CI/CD test reports)
- If missing, Phase 2 will be skipped with warning
- Provide JUnit XML, TAP, or JSON test report path via `test_results` variable
#### "Gate decision is FAIL but deployment needed urgently"
- Request business waiver (if `allow_waivers: true`)
- Document approver, justification, mitigation plan
- Create follow-up stories to address gaps
- Use WAIVED decision only for non-P0 gaps
- **Never waive**: Security issues, data corruption risks
#### "Assessments are stale (>7 days old)"
- Re-run `*test-design` workflow
- Re-run traceability (Phase 1)
- Re-run `*nfr-assess` workflow
- Update evidence files before gate decision
#### "Unclear decision (edge case)"
- Switch to manual mode: `decision_mode: manual`
- Document assumptions and rationale clearly
- Escalate to tech lead or architect for guidance
- Consider waiver if business-critical
---
## Integration with Other Workflows
- **testarch-test-design** → `*trace` - Define priorities, then trace coverage
- **testarch-atdd** → `*trace` - Generate tests, then validate coverage
- `*trace`**testarch-automate** - Identify gaps, then automate regression
- `*trace` **testarch-gate** - Generate metrics, then apply quality gates
- `*trace`**testarch-test-review** - Flag quality issues, then review tests
### Before Trace
1. **testarch-test-design** - Define test priorities (P0/P1/P2/P3)
2. **testarch-atdd** - Generate failing acceptance tests
3. **testarch-automate** - Expand regression suite
### After Trace (Phase 2 Decision)
- **PASS**: Proceed to deployment workflow
- **CONCERNS**: Deploy with monitoring, create remediation backlog stories
- **FAIL**: Block deployment, fix issues, re-run trace workflow
- **WAIVED**: Deploy with business approval, escalate monitoring
### Complements
- `*trace`**testarch-nfr-assess** - Use NFR validation in gate decision
- `*trace`**testarch-test-review** - Flag quality issues for review
- **CI/CD Pipeline** - Use gate YAML for automated quality gates
---
## Best Practices
### Phase 1 - Traceability
1. **Run Trace After Test Implementation**
- Don't run `*trace` before tests exist (run `*atdd` first)
- Trace is most valuable after initial test suite is written
@@ -346,26 +694,105 @@ Recommendation: Remove 1.3-COMPONENT-001 or consolidate with UNIT-001
- Unit tests for logic, E2E for journeys
- Only overlap for defense in depth on critical paths
5. **Generate Gate-Ready Artifacts**
### Phase 2 - Gate Decision
5. **Evidence is King**
- Never make gate decisions without fresh test results
- Validate evidence freshness (<7 days old)
- Link to all evidence sources (reports, logs, artifacts)
6. **P0 is Sacred**
- P0 failures ALWAYS result in FAIL (no exceptions except waivers)
- P0 = Critical user journeys, security, data integrity
- Waivers require VP/CTO approval + business justification
7. **Waivers are Temporary**
- Waiver applies ONLY to specific release
- Issue must be fixed in next release
- Never waive: security, data corruption, compliance violations
8. **CONCERNS is Not PASS**
- CONCERNS means "deploy with monitoring"
- Create follow-up stories for issues
- Do not ignore CONCERNS repeatedly
9. **Automate Gate Integration**
- Enable `generate_gate_yaml` for CI/CD integration
- Use YAML snippets in pipeline quality gates
- Export metrics for dashboard visualization
---
## Configuration Examples
### Strict Gate (Zero Tolerance)
```yaml
min_p0_coverage: 100
min_p1_coverage: 100
min_overall_coverage: 90
min_p0_pass_rate: 100
min_p1_pass_rate: 100
min_overall_pass_rate: 95
allow_waivers: false
max_security_issues: 0
max_critical_nfrs_fail: 0
```
Use for: Financial systems, healthcare, security-critical features
---
### Balanced Gate (Production Standard - Default)
```yaml
min_p0_coverage: 100
min_p1_coverage: 90
min_overall_coverage: 80
min_p0_pass_rate: 100
min_p1_pass_rate: 95
min_overall_pass_rate: 90
allow_waivers: true
max_security_issues: 0
max_critical_nfrs_fail: 0
```
Use for: Most production releases
---
### Relaxed Gate (Early Development)
```yaml
min_p0_coverage: 100
min_p1_coverage: 80
min_overall_coverage: 70
min_p0_pass_rate: 100
min_p1_pass_rate: 85
min_overall_pass_rate: 80
allow_waivers: true
allow_p2_failures: true
allow_p3_failures: true
```
Use for: Alpha/beta releases, internal tools, proof-of-concept
---
## Related Commands
- `bmad tea *test-design` - Define test priorities and risk assessment
- `bmad tea *atdd` - Generate failing acceptance tests for gaps
- `bmad tea *automate` - Expand regression suite based on gaps
- `bmad tea *gate` - Apply quality gates using traceability metrics
- `bmad tea *nfr-assess` - Validate non-functional requirements (for gate)
- `bmad tea *test-review` - Review test quality issues flagged by trace
- `bmad sm story-approved` - Mark story as complete (triggers gate)
---
## Resources
- [Instructions](./instructions.md) - Detailed workflow steps
- [Instructions](./instructions.md) - Detailed workflow steps (both phases)
- [Checklist](./checklist.md) - Validation checklist
- [Template](./trace-template.md) - Traceability matrix template
- [Knowledge Base](../../testarch/knowledge/) - Testing best practices

View File

@@ -1,10 +1,17 @@
# Requirements Traceability - Validation Checklist
# Requirements Traceability & Gate Decision - Validation Checklist
**Workflow:** `testarch-trace`
**Purpose:** Ensure complete and accurate traceability matrix with actionable gap analysis
**Purpose:** Ensure complete traceability matrix with actionable gap analysis AND make deployment readiness decision (PASS/CONCERNS/FAIL/WAIVED)
This checklist covers **two sequential phases**:
- **PHASE 1**: Requirements Traceability (always executed)
- **PHASE 2**: Quality Gate Decision (executed if `enable_gate_decision: true`)
---
# PHASE 1: REQUIREMENTS TRACEABILITY
## Prerequisites Validation
- [ ] Acceptance criteria are available (from story file OR inline)
@@ -114,15 +121,6 @@
---
## Quality Gate Validation
- [ ] P0 coverage >= 100% (required) ✅ or BLOCKER documented ❌
- [ ] P1 coverage >= 90% (recommended) ✅ or HIGH priority gap documented ⚠️
- [ ] Overall coverage >= 80% (recommended) ✅ or MEDIUM priority gap documented ⚠️
- [ ] Gate status determined: PASS / WARN / FAIL
---
## Test Quality Verification
For each mapped test, verify:
@@ -149,7 +147,7 @@ Knowledge fragments referenced:
---
## Deliverables Generated
## Phase 1 Deliverables Generated
### Traceability Matrix Markdown
@@ -161,15 +159,6 @@ Knowledge fragments referenced:
- [ ] Quality assessment section included
- [ ] Recommendations section included
### Gate YAML Snippet (if enabled)
- [ ] YAML snippet generated
- [ ] Story ID included
- [ ] Coverage metrics included (overall, p0, p1, p2)
- [ ] Gap counts included (critical, high, medium, low)
- [ ] Status included (PASS / WARN / FAIL)
- [ ] Recommendations included
### Coverage Badge/Metric (if enabled)
- [ ] Badge markdown generated
@@ -180,11 +169,10 @@ Knowledge fragments referenced:
- [ ] "Traceability" section added to story markdown
- [ ] Link to traceability matrix included
- [ ] Coverage summary included
- [ ] Gate status included
---
## Quality Assurance
## Phase 1 Quality Assurance
### Accuracy Checks
@@ -213,6 +201,370 @@ Knowledge fragments referenced:
---
## Phase 1 Documentation
- [ ] Traceability matrix is readable and well-formatted
- [ ] Tables render correctly in markdown
- [ ] Code blocks have proper syntax highlighting
- [ ] Links are valid and accessible
- [ ] Recommendations are clear and prioritized
---
# PHASE 2: QUALITY GATE DECISION
**Note**: Phase 2 executes only if `enable_gate_decision: true` in workflow.yaml
---
## Prerequisites
### Evidence Gathering
- [ ] Test execution results obtained (CI/CD pipeline, test framework reports)
- [ ] Story/epic/release file identified and read
- [ ] Test design document discovered or explicitly provided (if available)
- [ ] Traceability matrix discovered or explicitly provided (available from Phase 1)
- [ ] NFR assessment discovered or explicitly provided (if available)
- [ ] Code coverage report discovered or explicitly provided (if available)
- [ ] Burn-in results discovered or explicitly provided (if available)
### Evidence Validation
- [ ] Evidence freshness validated (warn if >7 days old, recommend re-running workflows)
- [ ] All required assessments available or user acknowledged gaps
- [ ] Test results are complete (not partial or interrupted runs)
- [ ] Test results match current codebase (not from outdated branch)
### Knowledge Base Loading
- [ ] `risk-governance.md` loaded successfully
- [ ] `probability-impact.md` loaded successfully
- [ ] `test-quality.md` loaded successfully
- [ ] `test-priorities.md` loaded successfully
- [ ] `ci-burn-in.md` loaded (if burn-in results available)
---
## Process Steps
### Step 1: Context Loading
- [ ] Gate type identified (story/epic/release/hotfix)
- [ ] Target ID extracted (story_id, epic_num, or release_version)
- [ ] Decision thresholds loaded from workflow variables
- [ ] Risk tolerance configuration loaded
- [ ] Waiver policy loaded
### Step 2: Evidence Parsing
**Test Results:**
- [ ] Total test count extracted
- [ ] Passed test count extracted
- [ ] Failed test count extracted
- [ ] Skipped test count extracted
- [ ] Test duration extracted
- [ ] P0 test pass rate calculated
- [ ] P1 test pass rate calculated
- [ ] Overall test pass rate calculated
**Quality Assessments:**
- [ ] P0/P1/P2/P3 scenarios extracted from test-design.md (if available)
- [ ] Risk scores extracted from test-design.md (if available)
- [ ] Coverage percentages extracted from traceability-matrix.md (available from Phase 1)
- [ ] Coverage gaps extracted from traceability-matrix.md (available from Phase 1)
- [ ] NFR status extracted from nfr-assessment.md (if available)
- [ ] Security issues count extracted from nfr-assessment.md (if available)
**Code Coverage:**
- [ ] Line coverage percentage extracted (if available)
- [ ] Branch coverage percentage extracted (if available)
- [ ] Function coverage percentage extracted (if available)
- [ ] Critical path coverage validated (if available)
**Burn-in Results:**
- [ ] Burn-in iterations count extracted (if available)
- [ ] Flaky tests count extracted (if available)
- [ ] Stability score calculated (if available)
### Step 3: Decision Rules Application
**P0 Criteria Evaluation:**
- [ ] P0 test pass rate evaluated (must be 100%)
- [ ] P0 acceptance criteria coverage evaluated (must be 100%)
- [ ] Security issues count evaluated (must be 0)
- [ ] Critical NFR failures evaluated (must be 0)
- [ ] Flaky tests evaluated (must be 0 if burn-in enabled)
- [ ] P0 decision recorded: PASS or FAIL
**P1 Criteria Evaluation:**
- [ ] P1 test pass rate evaluated (threshold: min_p1_pass_rate)
- [ ] P1 acceptance criteria coverage evaluated (threshold: 95%)
- [ ] Overall test pass rate evaluated (threshold: min_overall_pass_rate)
- [ ] Code coverage evaluated (threshold: min_coverage)
- [ ] P1 decision recorded: PASS or CONCERNS
**P2/P3 Criteria Evaluation:**
- [ ] P2 failures tracked (informational, don't block if allow_p2_failures: true)
- [ ] P3 failures tracked (informational, don't block if allow_p3_failures: true)
- [ ] Residual risks documented
**Final Decision:**
- [ ] Decision determined: PASS / CONCERNS / FAIL / WAIVED
- [ ] Decision rationale documented
- [ ] Decision is deterministic (follows rules, not arbitrary)
### Step 4: Documentation
**Gate Decision Document Created:**
- [ ] Story/epic/release info section complete (ID, title, description, links)
- [ ] Decision clearly stated (PASS / CONCERNS / FAIL / WAIVED)
- [ ] Decision date recorded
- [ ] Evaluator recorded (user or agent name)
**Evidence Summary Documented:**
- [ ] Test results summary complete (total, passed, failed, pass rates)
- [ ] Coverage summary complete (P0/P1 criteria, code coverage)
- [ ] NFR validation summary complete (security, performance, reliability, maintainability)
- [ ] Flakiness summary complete (burn-in iterations, flaky test count)
**Rationale Documented:**
- [ ] Decision rationale clearly explained
- [ ] Key evidence highlighted
- [ ] Assumptions and caveats noted (if any)
**Residual Risks Documented (if CONCERNS or WAIVED):**
- [ ] Unresolved P1/P2 issues listed
- [ ] Probability × impact estimated for each risk
- [ ] Mitigations or workarounds described
**Waivers Documented (if WAIVED):**
- [ ] Waiver reason documented (business justification)
- [ ] Waiver approver documented (name, role)
- [ ] Waiver expiry date documented
- [ ] Remediation plan documented (fix in next release, due date)
- [ ] Monitoring plan documented
**Critical Issues Documented (if FAIL or CONCERNS):**
- [ ] Top 5-10 critical issues listed
- [ ] Priority assigned to each issue (P0/P1/P2)
- [ ] Owner assigned to each issue
- [ ] Due date assigned to each issue
**Recommendations Documented:**
- [ ] Next steps clearly stated for decision type
- [ ] Deployment recommendation provided
- [ ] Monitoring recommendations provided (if applicable)
- [ ] Remediation recommendations provided (if applicable)
### Step 5: Status Updates and Notifications
**Status File Updated:**
- [ ] Gate decision appended to bmm-workflow-status.md (if append_to_history: true)
- [ ] Format correct: `[DATE] Gate Decision: DECISION - Target {ID} - {rationale}`
- [ ] Status file committed or staged for commit
**Gate YAML Created:**
- [ ] Gate YAML snippet generated with decision and criteria
- [ ] Evidence references included in YAML
- [ ] Next steps included in YAML
- [ ] YAML file saved to output folder
**Stakeholder Notification Generated:**
- [ ] Notification subject line created
- [ ] Notification body created with summary
- [ ] Recipients identified (PM, SM, DEV lead, stakeholders)
- [ ] Notification ready for delivery (if notify_stakeholders: true)
**Outputs Saved:**
- [ ] Gate decision document saved to `{output_file}`
- [ ] Gate YAML saved to `{output_folder}/gate-decision-{target}.yaml`
- [ ] All outputs are valid and readable
---
## Phase 2 Output Validation
### Gate Decision Document
**Completeness:**
- [ ] All required sections present (info, decision, evidence, rationale, next steps)
- [ ] No placeholder text or TODOs left in document
- [ ] All evidence references are accurate and complete
- [ ] All links to artifacts are valid
**Accuracy:**
- [ ] Decision matches applied criteria rules
- [ ] Test results match CI/CD pipeline output
- [ ] Coverage percentages match reports
- [ ] NFR status matches assessment document
- [ ] No contradictions or inconsistencies
**Clarity:**
- [ ] Decision rationale is clear and unambiguous
- [ ] Technical jargon is explained or avoided
- [ ] Stakeholders can understand next steps
- [ ] Recommendations are actionable
### Gate YAML
**Format:**
- [ ] YAML is valid (no syntax errors)
- [ ] All required fields present (target, decision, date, evaluator, criteria, evidence)
- [ ] Field values are correct data types (numbers, strings, dates)
**Content:**
- [ ] Criteria values match decision document
- [ ] Evidence references are accurate
- [ ] Next steps align with decision type
---
## Phase 2 Quality Checks
### Decision Integrity
- [ ] Decision is deterministic (follows rules, not arbitrary)
- [ ] P0 failures result in FAIL decision (unless waived)
- [ ] Security issues result in FAIL decision (unless waived - but should never be waived)
- [ ] Waivers have business justification and approver (if WAIVED)
- [ ] Residual risks are documented (if CONCERNS or WAIVED)
### Evidence-Based
- [ ] Decision is based on actual test results (not guesses)
- [ ] All claims are supported by evidence
- [ ] No assumptions without documentation
- [ ] Evidence sources are cited (CI run IDs, report URLs)
### Transparency
- [ ] Decision rationale is transparent and auditable
- [ ] Criteria evaluation is documented step-by-step
- [ ] Any deviations from standard process are explained
- [ ] Waiver justifications are clear (if applicable)
### Consistency
- [ ] Decision aligns with risk-governance knowledge fragment
- [ ] Priority framework (P0/P1/P2/P3) applied consistently
- [ ] Terminology consistent with test-quality knowledge fragment
- [ ] Decision matrix followed correctly
---
## Phase 2 Integration Points
### BMad Workflow Status
- [ ] Gate decision added to `bmm-workflow-status.md`
- [ ] Format matches existing gate history entries
- [ ] Timestamp is accurate
- [ ] Decision summary is concise (<80 chars)
### CI/CD Pipeline
- [ ] Gate YAML is CI/CD-compatible
- [ ] YAML can be parsed by pipeline automation
- [ ] Decision can be used to block/allow deployments
- [ ] Evidence references are accessible to pipeline
### Stakeholders
- [ ] Notification message is clear and actionable
- [ ] Decision is explained in non-technical terms
- [ ] Next steps are specific and time-bound
- [ ] Recipients are appropriate for decision type
---
## Phase 2 Compliance and Audit
### Audit Trail
- [ ] Decision date and time recorded
- [ ] Evaluator identified (user or agent)
- [ ] All evidence sources cited
- [ ] Decision criteria documented
- [ ] Rationale clearly explained
### Traceability
- [ ] Gate decision traceable to story/epic/release
- [ ] Evidence traceable to specific test runs
- [ ] Assessments traceable to workflows that created them
- [ ] Waiver traceable to approver (if applicable)
### Compliance
- [ ] Security requirements validated (no unresolved vulnerabilities)
- [ ] Quality standards met or waived with justification
- [ ] Regulatory requirements addressed (if applicable)
- [ ] Documentation sufficient for external audit
---
## Phase 2 Edge Cases and Exceptions
### Missing Evidence
- [ ] If test-design.md missing, decision still possible with test results + trace
- [ ] If traceability-matrix.md missing, decision still possible with test results (but Phase 1 should provide it)
- [ ] If nfr-assessment.md missing, NFR validation marked as NOT ASSESSED
- [ ] If code coverage missing, coverage criterion marked as NOT ASSESSED
- [ ] User acknowledged gaps in evidence or provided alternative proof
### Stale Evidence
- [ ] Evidence freshness checked (if validate_evidence_freshness: true)
- [ ] Warnings issued for assessments >7 days old
- [ ] User acknowledged stale evidence or re-ran workflows
- [ ] Decision document notes any stale evidence used
### Conflicting Evidence
- [ ] Conflicts between test results and assessments resolved
- [ ] Most recent/authoritative source identified
- [ ] Conflict resolution documented in decision rationale
- [ ] User consulted if conflict cannot be resolved
### Waiver Scenarios
- [ ] Waiver only used for FAIL decision (not PASS or CONCERNS)
- [ ] Waiver has business justification (not technical convenience)
- [ ] Waiver has named approver with authority (VP/CTO/PO)
- [ ] Waiver has expiry date (does NOT apply to future releases)
- [ ] Waiver has remediation plan with concrete due date
- [ ] Security vulnerabilities are NOT waived (enforced)
---
# FINAL VALIDATION (Both Phases)
## Non-Prescriptive Validation
- [ ] Traceability format adapted to team needs (not rigid template)
@@ -225,42 +577,77 @@ Knowledge fragments referenced:
## Documentation and Communication
- [ ] Traceability matrix is readable and well-formatted
- [ ] All documents are readable and well-formatted
- [ ] Tables render correctly in markdown
- [ ] Code blocks have proper syntax highlighting
- [ ] Links are valid and accessible
- [ ] Recommendations are clear and prioritized
- [ ] Gate status is prominent and unambiguous
- [ ] Gate decision is prominent and unambiguous (Phase 2)
---
## Final Validation
**Phase 1 (Traceability):**
- [ ] All prerequisites met
- [ ] All acceptance criteria mapped or gaps documented
- [ ] P0 coverage is 100% OR documented as BLOCKER
- [ ] Gap analysis is complete and prioritized
- [ ] Test quality issues identified and flagged
- [ ] Deliverables generated and saved
- [ ] Gate YAML ready for CI/CD integration (if enabled)
- [ ] Story file updated (if enabled)
- [ ] Workflow completed successfully
**Phase 2 (Gate Decision):**
- [ ] All quality evidence gathered
- [ ] Decision criteria applied correctly
- [ ] Decision rationale documented
- [ ] Gate YAML ready for CI/CD integration
- [ ] Status file updated (if enabled)
- [ ] Stakeholders notified (if enabled)
**Workflow Complete:**
- [ ] Phase 1 completed successfully
- [ ] Phase 2 completed successfully (if enabled)
- [ ] All outputs validated and saved
- [ ] Ready to proceed based on gate decision
---
## Sign-Off
**Traceability Status:**
**Phase 1 - Traceability Status:**
- [ ] ✅ PASS - All quality gates met, no critical gaps
- [ ] ⚠️ WARN - P1 gaps exist, address before PR merge
- [ ] ❌ FAIL - P0 gaps exist, BLOCKER for release
**Phase 2 - Gate Decision Status (if enabled):**
- [ ] ✅ PASS - Deploy to production
- [ ] ⚠️ CONCERNS - Deploy with monitoring
- [ ] ❌ FAIL - Block deployment, fix issues
- [ ] 🔓 WAIVED - Deploy with business approval and remediation plan
**Next Actions:**
- If PASS: Proceed to `*gate` workflow or PR merge
- If WARN: Address HIGH priority gaps, re-run `*trace`
- If FAIL: Run `*atdd` to generate missing P0 tests, re-run `*trace`
- If PASS (both phases): Proceed to deployment
- If WARN/CONCERNS: Address gaps/issues, proceed with monitoring
- If FAIL (either phase): Run `*atdd` for missing tests, fix issues, re-run `*trace`
- If WAIVED: Deploy with approved waiver, schedule remediation
---
## Notes
Record any issues, deviations, or important observations during workflow execution:
- **Phase 1 Issues**: [Note any traceability mapping challenges, missing tests, quality concerns]
- **Phase 2 Issues**: [Note any missing, stale, or conflicting evidence]
- **Decision Rationale**: [Document any nuanced reasoning or edge cases]
- **Waiver Details**: [Document waiver negotiations or approvals]
- **Follow-up Actions**: [List any actions required after gate decision]
---

View File

@@ -1,7 +1,7 @@
# Requirements Traceability - Instructions v4.0
# Test Architect Workflow: Requirements Traceability & Quality Gate Decision
**Workflow:** `testarch-trace`
**Purpose:** Generate requirements-to-tests traceability matrix with coverage analysis and gap identification
**Purpose:** Generate requirements-to-tests traceability matrix, analyze coverage gaps, and make quality gate decisions (PASS/CONCERNS/FAIL/WAIVED)
**Agent:** Test Architect (TEA)
**Format:** Pure Markdown v4.0 (no XML blocks)
@@ -9,29 +9,40 @@
## Overview
This workflow creates a comprehensive traceability matrix that maps acceptance criteria to implemented tests, identifies coverage gaps, and provides actionable recommendations for improving test coverage. It supports both BMad-integrated mode (with story files and test design) and standalone mode (with inline acceptance criteria).
This workflow operates in two sequential phases to validate test coverage and deployment readiness:
**PHASE 1 - REQUIREMENTS TRACEABILITY:** Create comprehensive traceability matrix mapping acceptance criteria to implemented tests, identify coverage gaps, and provide actionable recommendations.
**PHASE 2 - QUALITY GATE DECISION:** Use traceability results combined with test execution evidence to make gate decisions (PASS/CONCERNS/FAIL/WAIVED) that determine deployment readiness.
**Key Capabilities:**
- Map acceptance criteria to specific test cases across all levels (E2E, API, Component, Unit)
- Classify coverage status (FULL, PARTIAL, NONE, UNIT-ONLY, INTEGRATION-ONLY)
- Prioritize gaps by risk level (P0/P1/P2/P3) using test-priorities framework
- Generate gate-ready YAML snippets for CI/CD integration
- Detect duplicate coverage across test levels
- Verify explicit assertions in test cases
- Apply deterministic decision rules based on coverage and test execution results
- Generate gate decisions with evidence and rationale
- Support waivers for business-approved exceptions
- Update workflow status and notify stakeholders
---
## Prerequisites
**Required:**
**Required (Phase 1):**
- Acceptance criteria (from story file OR provided inline)
- Implemented test suite (or acknowledge gaps to be addressed)
**Required (Phase 2 - if `enable_gate_decision: true`):**
- Test execution results (CI/CD test reports, pass/fail rates)
- Test design with risk priorities (P0/P1/P2/P3)
**Recommended:**
- `test-design.md` (for risk assessment and priority context)
- `nfr-assessment.md` (for release-level gates)
- `tech-spec.md` (for technical implementation context)
- Test framework configuration (playwright.config.ts, jest.config.js, etc.)
@@ -39,21 +50,26 @@ This workflow creates a comprehensive traceability matrix that maps acceptance c
- If story lacks any implemented tests AND no gaps are acknowledged, recommend running `*atdd` workflow first
- If acceptance criteria are completely missing, halt and request them
- If Phase 2 enabled but test execution results missing, warn and skip gate decision
---
## Workflow Steps
## PHASE 1: REQUIREMENTS TRACEABILITY
This phase focuses on mapping requirements to tests, analyzing coverage, and identifying gaps.
---
### Step 1: Load Context and Knowledge Base
**Actions:**
1. Load relevant knowledge fragments from `{project-root}/bmad/bmm/testarch/tea-index.csv`:
- `traceability.md` - Requirements mapping patterns
- `test-priorities.md` - P0/P1/P2/P3 risk framework
- `risk-governance.md` - Risk-based testing approach
- `test-quality.md` - Definition of Done for tests
- `selective-testing.md` - Duplicate coverage patterns
- `test-priorities-matrix.md` - P0/P1/P2/P3 risk framework with automated priority calculation, risk-based mapping, tagging strategy (389 lines, 2 examples)
- `risk-governance.md` - Risk-based testing approach: 6 categories (TECH, SEC, PERF, DATA, BUS, OPS), automated scoring, gate decision engine, coverage traceability (625 lines, 4 examples)
- `probability-impact.md` - Risk scoring methodology: probability × impact matrix, automated classification, dynamic re-assessment, gate integration (604 lines, 4 examples)
- `test-quality.md` - Definition of Done for tests: deterministic, isolated with cleanup, explicit assertions, length/time limits (658 lines, 5 examples)
- `selective-testing.md` - Duplicate coverage patterns: tag-based, spec filters, diff-based selection, promotion rules (727 lines, 4 examples)
2. Read story file (if provided):
- Extract acceptance criteria
@@ -160,7 +176,7 @@ This workflow creates a comprehensive traceability matrix that maps acceptance c
- P1 coverage >= 90% (recommended)
- Overall coverage >= 80% (recommended)
**Output:** Prioritized gap analysis with actionable recommendations
**Output:** Prioritized gap analysis with actionable recommendations and coverage metrics
---
@@ -191,7 +207,7 @@ This workflow creates a comprehensive traceability matrix that maps acceptance c
---
### Step 6: Generate Deliverables
### Step 6: Generate Deliverables (Phase 1)
**Actions:**
@@ -231,13 +247,442 @@ This workflow creates a comprehensive traceability matrix that maps acceptance c
- Include coverage summary
- Add gate status
**Output:** Complete traceability documentation ready for review and CI/CD integration
**Output:** Complete Phase 1 traceability deliverables
**Next:** If `enable_gate_decision: true`, proceed to Phase 2. Otherwise, workflow complete.
---
## PHASE 2: QUALITY GATE DECISION
This phase uses traceability results to make a quality gate decision (PASS/CONCERNS/FAIL/WAIVED) based on evidence and decision rules.
**When Phase 2 Runs:** Automatically after Phase 1 if `enable_gate_decision: true` (default: true)
**Skip Conditions:** If test execution results (`test_results`) not provided, warn and skip Phase 2.
---
### Step 7: Gather Quality Evidence
**Actions:**
1. **Load Phase 1 traceability results** (inherited context):
- Coverage metrics (P0/P1/overall percentages)
- Gap analysis (missing/partial tests)
- Quality concerns (test quality flags)
- Traceability matrix
2. **Load test execution results** (if `test_results` provided):
- Read CI/CD test reports (JUnit XML, TAP, JSON)
- Extract pass/fail counts by priority
- Calculate pass rates:
- **P0 pass rate**: `(P0 passed / P0 total) * 100`
- **P1 pass rate**: `(P1 passed / P1 total) * 100`
- **Overall pass rate**: `(All passed / All total) * 100`
- Identify failing tests and map to criteria
3. **Load NFR assessment** (if `nfr_file` provided):
- Read `nfr-assessment.md` or similar
- Check critical NFR status (performance, security, scalability)
- Flag any critical NFR failures
4. **Load supporting artifacts**:
- `test-design.md` → Risk priorities, DoD checklist
- `story-*.md` or `Epics.md` → Requirements context
- `bmm-workflow-status.md` → Workflow completion status (if `check_all_workflows_complete: true`)
5. **Validate evidence freshness** (if `validate_evidence_freshness: true`):
- Check timestamps of test-design, traceability, NFR assessments
- Warn if artifacts are >7 days old
6. **Check prerequisite workflows** (if `check_all_workflows_complete: true`):
- Verify test-design workflow complete
- Verify trace workflow complete (Phase 1)
- Verify nfr-assess workflow complete (if release-level gate)
**Output:** Consolidated evidence bundle with all quality signals
---
### Step 8: Apply Decision Rules
**If `decision_mode: "deterministic"`** (rule-based - default):
**Decision rules** (based on `workflow.yaml` thresholds):
1. **PASS** if ALL of the following are true:
- P0 coverage ≥ `min_p0_coverage` (default: 100%)
- P1 coverage ≥ `min_p1_coverage` (default: 90%)
- Overall coverage ≥ `min_overall_coverage` (default: 80%)
- P0 test pass rate = `min_p0_pass_rate` (default: 100%)
- P1 test pass rate ≥ `min_p1_pass_rate` (default: 95%)
- Overall test pass rate ≥ `min_overall_pass_rate` (default: 90%)
- Critical NFRs passed (if `nfr_file` provided)
- No unresolved security issues ≤ `max_security_issues` (default: 0)
- No test quality red flags (hard waits, no assertions)
2. **CONCERNS** if ANY of the following are true:
- P1 coverage 80-89% (below threshold but not critical)
- P1 test pass rate 90-94% (below threshold but not critical)
- Overall pass rate 85-89%
- P2 coverage <50% (informational)
- Some non-critical NFRs failing
- Minor test quality concerns (large test files, inferred mappings)
- **Note**: CONCERNS does NOT block deployment but requires acknowledgment
3. **FAIL** if ANY of the following are true:
- P0 coverage <100% (missing critical tests)
- P0 test pass rate <100% (failing critical tests)
- P1 coverage <80% (significant gap)
- P1 test pass rate <90% (significant failures)
- Overall coverage <80%
- Overall pass rate <85%
- Critical NFRs failing (`max_critical_nfrs_fail` exceeded)
- Unresolved security issues (`max_security_issues` exceeded)
- Major test quality issues (tests with no assertions, pervasive hard waits)
4. **WAIVED** (only if `allow_waivers: true`):
- Decision would be FAIL based on rules above
- Business stakeholder has approved waiver
- Waiver documented with:
- Justification (time constraint, known limitation, acceptable risk)
- Approver name and date
- Mitigation plan (follow-up stories, manual testing)
- Waiver evidence linked (email, Slack thread, ticket)
**Risk tolerance adjustments:**
- If `allow_p2_failures: true` → P2 test failures do NOT affect gate decision
- If `allow_p3_failures: true` → P3 test failures do NOT affect gate decision
- If `escalate_p1_failures: true` → P1 failures require explicit manager/lead approval
**If `decision_mode: "manual"`:**
- Present evidence summary to team
- Recommend decision based on rules above
- Team makes final call in meeting/chat
- Document decision with approver names
**Output:** Gate decision (PASS/CONCERNS/FAIL/WAIVED) with rule-based rationale
---
### Step 9: Document Decision and Evidence
**Actions:**
1. **Create gate decision document**:
- Save to `gate_output_file` (default: `{output_folder}/gate-decision-{gate_type}-{story_id}.md`)
- Use structure below
2. **Document structure**:
```markdown
# Quality Gate Decision: {gate_type} {story_id/epic_num/release_version}
**Decision**: [PASS / CONCERNS / FAIL / WAIVED]
**Date**: {date}
**Decider**: {decision_mode} (deterministic | manual)
**Evidence Date**: {test_results_date}
---
## Summary
[1-2 sentence summary of decision and key factors]
---
## Decision Criteria
| Criterion | Threshold | Actual | Status |
| ----------------- | --------- | -------- | ------- |
| P0 Coverage | ≥100% | 100% | ✅ PASS |
| P1 Coverage | ≥90% | 88% | ⚠️ FAIL |
| Overall Coverage | ≥80% | 92% | ✅ PASS |
| P0 Pass Rate | 100% | 100% | ✅ PASS |
| P1 Pass Rate | ≥95% | 98% | ✅ PASS |
| Overall Pass Rate | ≥90% | 96% | ✅ PASS |
| Critical NFRs | All Pass | All Pass | ✅ PASS |
| Security Issues | 0 | 0 | ✅ PASS |
**Overall Status**: 7/8 criteria met → Decision: **CONCERNS**
---
## Evidence Summary
### Test Coverage (from Phase 1 Traceability)
- **P0 Coverage**: 100% (5/5 criteria fully covered)
- **P1 Coverage**: 88% (7/8 criteria fully covered)
- **Overall Coverage**: 92% (12/13 criteria covered)
- **Gap**: AC-5 (P1) missing E2E test
### Test Execution Results
- **P0 Pass Rate**: 100% (12/12 tests passed)
- **P1 Pass Rate**: 98% (45/46 tests passed)
- **Overall Pass Rate**: 96% (67/70 tests passed)
- **Failures**: 3 P2 tests (non-blocking)
### Non-Functional Requirements
- Performance: ✅ PASS (response time <500ms)
- Security: ✅ PASS (no vulnerabilities)
- Scalability: ✅ PASS (handles 10K users)
### Test Quality
- All tests have explicit assertions ✅
- No hard waits detected ✅
- Test files <300 lines ✅
- Test IDs follow convention ✅
---
## Decision Rationale
**Why CONCERNS (not PASS)**:
- P1 coverage at 88% is below 90% threshold
- AC-5 (P1 priority) missing E2E test for error handling scenario
- This is a known gap from test-design phase
**Why CONCERNS (not FAIL)**:
- P0 coverage is 100% (critical paths validated)
- Overall coverage is 92% (above 80% threshold)
- Test pass rate is excellent (96% overall)
- Gap is isolated to one P1 criterion (not systemic)
**Recommendation**:
- Acknowledge gap and proceed with deployment
- Add missing AC-5 E2E test in next sprint
- Create follow-up story: "Add E2E test for AC-5 error handling"
---
## Next Steps
- [ ] Create follow-up story for AC-5 E2E test
- [ ] Deploy to staging environment
- [ ] Monitor production for edge cases related to AC-5
- [ ] Update traceability matrix after follow-up test added
---
## References
- Traceability Matrix: `bmad/output/traceability-matrix.md`
- Test Design: `bmad/output/test-design-epic-2.md`
- Test Results: `ci-artifacts/test-report-2025-01-15.xml`
- NFR Assessment: `bmad/output/nfr-assessment-release-1.2.md`
```
3. **Include evidence links** (if `require_evidence: true`):
- Link to traceability matrix
- Link to test execution reports (CI artifacts)
- Link to NFR assessment
- Link to test-design document
- Link to relevant PRs, commits, deployments
4. **Waiver documentation** (if decision is WAIVED):
- Approver name and role (e.g., "Jane Doe, Engineering Manager")
- Approval date and method (e.g., "2025-01-15, Slack thread")
- Justification (e.g., "Time-boxed MVP, missing tests will be added in v1.1")
- Mitigation plan (e.g., "Manual testing by QA, follow-up stories created")
- Evidence link (e.g., "Slack: #engineering 2025-01-15 3:42pm")
**Output:** Complete gate decision document with evidence and rationale
---
### Step 10: Update Status Tracking and Notify
**Actions:**
1. **Update workflow status** (if `append_to_history: true`):
- Append gate decision to `bmm-workflow-status.md` under "Gate History" section
- Format:
```markdown
## Gate History
### Story 1.3 - User Login (2025-01-15)
- **Decision**: CONCERNS
- **Reason**: P1 coverage 88% (below 90%)
- **Document**: [gate-decision-story-1.3.md](bmad/output/gate-decision-story-1.3.md)
- **Action**: Deploy with follow-up story for AC-5
```
2. **Generate stakeholder notification** (if `notify_stakeholders: true`):
- Create concise summary message for team communication
- Include: Decision, key metrics, action items
- Format for Slack/email/chat:
```
🚦 Quality Gate Decision: Story 1.3 - User Login
Decision: ⚠️ CONCERNS
- P0 Coverage: ✅ 100%
- P1 Coverage: ⚠️ 88% (below 90%)
- Test Pass Rate: ✅ 96%
Action Required:
- Create follow-up story for AC-5 E2E test
- Deploy to staging for validation
Full Report: bmad/output/gate-decision-story-1.3.md
```
3. **Request sign-off** (if `require_sign_off: true`):
- Prompt for named approver (tech lead, QA lead, PM)
- Document approver name and timestamp in gate decision
- Block until sign-off received (interactive prompt)
**Output:** Status tracking updated, stakeholders notified, sign-off obtained (if required)
**Workflow Complete**: Both Phase 1 (traceability) and Phase 2 (gate decision) deliverables generated.
---
## Decision Matrix (Quick Reference)
| Scenario | P0 Cov | P1 Cov | Overall Cov | P0 Pass | P1 Pass | Overall Pass | NFRs | Decision |
| --------------- | ----------------- | ------ | ----------- | ------- | ------- | ------------ | ---- | ------------ |
| All green | 100% | ≥90% | ≥80% | 100% | ≥95% | ≥90% | Pass | **PASS** |
| Minor gap | 100% | 80-89% | ≥80% | 100% | 90-94% | 85-89% | Pass | **CONCERNS** |
| Missing P0 | <100% | - | - | - | - | - | - | **FAIL** |
| P0 test fail | 100% | - | - | <100% | - | - | - | **FAIL** |
| P1 gap | 100% | <80% | - | 100% | - | - | - | **FAIL** |
| NFR fail | 100% | ≥90% | ≥80% | 100% | ≥95% | ≥90% | Fail | **FAIL** |
| Security issue | - | - | - | - | - | - | Yes | **FAIL** |
| Business waiver | [FAIL conditions] | - | - | - | - | - | - | **WAIVED** |
---
## Waiver Management
**When to use waivers:**
- Time-boxed MVP releases (known gaps, follow-up planned)
- Low-risk P1 gaps with mitigation (manual testing, monitoring)
- Technical debt acknowledged by product/engineering leadership
- External dependencies blocking test automation
**Waiver approval process:**
1. Document gap and risk in gate decision
2. Propose mitigation plan (manual testing, follow-up stories, monitoring)
3. Request approval from stakeholder (EM, PM, QA lead)
4. Link approval evidence (email, chat thread, meeting notes)
5. Add waiver to gate decision document
6. Create follow-up stories to close gaps
**Waiver does NOT apply to:**
- P0 gaps (always blocking)
- Critical security issues (always blocking)
- Critical NFR failures (performance, data integrity)
---
## Example Gate Decisions
### Example 1: PASS (All Criteria Met)
```
Decision: ✅ PASS
Summary: All quality criteria met. Story 1.3 is ready for production deployment.
Evidence:
- P0 Coverage: 100% (5/5 criteria)
- P1 Coverage: 95% (19/20 criteria)
- Overall Coverage: 92% (24/26 criteria)
- P0 Pass Rate: 100% (12/12 tests)
- P1 Pass Rate: 98% (45/46 tests)
- Overall Pass Rate: 96% (67/70 tests)
- NFRs: All pass (performance, security, scalability)
Action: Deploy to production ✅
```
### Example 2: CONCERNS (Minor Gap, Non-Blocking)
```
Decision: ⚠️ CONCERNS
Summary: P1 coverage slightly below threshold (88% vs 90%). Recommend deploying with follow-up story.
Evidence:
- P0 Coverage: 100% ✅
- P1 Coverage: 88% ⚠️ (below 90%)
- Overall Coverage: 92% ✅
- Test Pass Rate: 96% ✅
- Gap: AC-5 (P1) missing E2E test
Action:
- Deploy to staging for validation
- Create follow-up story for AC-5 E2E test
- Monitor production for edge cases related to AC-5
```
### Example 3: FAIL (P0 Gap, Blocking)
```
Decision: ❌ FAIL
Summary: P0 coverage incomplete. Missing critical validation test. BLOCKING deployment.
Evidence:
- P0 Coverage: 80% ❌ (4/5 criteria, AC-2 missing)
- AC-2: "User cannot login with invalid credentials" (P0 priority)
- No tests validate login security for invalid credentials
- This is a critical security gap
Action:
- Add P0 test for AC-2: 1.3-E2E-004 (invalid credentials)
- Re-run traceability after test added
- Re-evaluate gate decision after P0 coverage = 100%
Deployment BLOCKED until P0 gap resolved ❌
```
### Example 4: WAIVED (Business Decision)
```
Decision: ⚠️ WAIVED
Summary: P1 coverage below threshold (75% vs 90%), but waived for MVP launch.
Evidence:
- P0 Coverage: 100% ✅
- P1 Coverage: 75% ❌ (below 90%)
- Gap: 5 P1 criteria missing E2E tests (error handling, edge cases)
Waiver:
- Approver: Jane Doe, Engineering Manager
- Date: 2025-01-15
- Justification: Time-boxed MVP for investor demo. Core functionality (P0) fully validated. P1 gaps are low-risk edge cases.
- Mitigation: Manual QA testing for P1 scenarios, follow-up stories created for automated tests in v1.1
- Evidence: Slack #engineering 2025-01-15 3:42pm
Action:
- Deploy to production with manual QA validation ✅
- Add 5 E2E tests for P1 gaps in v1.1 sprint
- Monitor production logs for edge case occurrences
```
---
## Non-Prescriptive Approach
**Minimal Examples:** This workflow provides principles and patterns, not rigid templates. Teams should adapt the traceability format to their needs.
**Minimal Examples:** This workflow provides principles and patterns, not rigid templates. Teams should adapt the traceability and gate decision formats to their needs.
**Key Patterns to Follow:**
@@ -245,7 +690,9 @@ This workflow creates a comprehensive traceability matrix that maps acceptance c
- Prioritize by risk (P0 gaps are critical, P3 gaps are acceptable)
- Check coverage at appropriate levels (E2E for journeys, Unit for logic)
- Verify test quality (explicit assertions, no flakiness)
- Generate gate-ready artifacts (YAML snippets for CI/CD)
- Apply deterministic gate rules for consistency
- Document gate decisions with clear evidence
- Use waivers judiciously (business approved, mitigation planned)
**Extend as Needed:**
@@ -253,6 +700,8 @@ This workflow creates a comprehensive traceability matrix that maps acceptance c
- Integrate with code coverage tools (Istanbul, NYC)
- Link to external traceability systems (JIRA, Azure DevOps)
- Add compliance or regulatory requirements
- Customize gate decision thresholds per project
- Add manual approval workflows for gate decisions
---
@@ -323,7 +772,7 @@ Use selective testing principles from `selective-testing.md`:
### With test-design.md
- Use risk assessment to prioritize gap remediation
- Reference test priorities (P0/P1/P2/P3) for severity classification
- Reference test priorities (P0/P1/P2/P3) for severity classification and gate decision
- Align traceability with originally planned test coverage
### With tech-spec.md
@@ -338,9 +787,15 @@ Use selective testing principles from `selective-testing.md`:
- Verify acceptance criteria align with product goals
- Check for unstated requirements that need coverage
### With nfr-assessment.md
- Load non-functional validation results for gate decision
- Check critical NFR status (performance, security, scalability)
- Include NFR pass/fail in gate decision criteria
---
## Quality Gates
## Quality Gates (Phase 1 Recommendations)
### P0 Coverage (Critical Paths)
@@ -496,6 +951,7 @@ traceability:
Before completing this workflow, verify:
**Phase 1 (Traceability):**
- ✅ All acceptance criteria are mapped to tests (or gaps are documented)
- ✅ Coverage status is classified (FULL, PARTIAL, NONE, UNIT-ONLY, INTEGRATION-ONLY)
- ✅ Gaps are prioritized by risk level (P0/P1/P2/P3)
@@ -503,19 +959,32 @@ Before completing this workflow, verify:
- ✅ Duplicate coverage is identified and flagged
- ✅ Test quality is assessed (assertions, structure, performance)
- ✅ Traceability matrix is generated and saved
- ✅ Gate YAML snippet is generated (if enabled)
- ✅ Story file is updated with traceability section (if enabled)
- ✅ Recommendations are actionable and specific
**Phase 2 (Gate Decision - if enabled):**
- ✅ Test execution results loaded and pass rates calculated
- ✅ NFR assessment results loaded (if applicable)
- ✅ Decision rules applied consistently (PASS/CONCERNS/FAIL/WAIVED)
- ✅ Gate decision document created with evidence
- ✅ Waiver documented if decision is WAIVED (approver, justification, mitigation)
- ✅ Workflow status updated (bmm-workflow-status.md)
- ✅ Stakeholders notified (if enabled)
---
## Notes
**Phase 1 (Traceability):**
- **Explicit Mapping:** Require tests to reference criteria explicitly (test IDs, describe blocks) for maintainability
- **Risk-Based Prioritization:** Use test-priorities framework (P0/P1/P2/P3) to determine gap severity
- **Quality Over Quantity:** Better to have fewer high-quality tests with FULL coverage than many low-quality tests with PARTIAL coverage
- **Selective Testing:** Avoid duplicate coverage - test each behavior at the appropriate level only
- **Gate Integration:** Generate YAML snippets that can be consumed by CI/CD pipelines for automated quality gates
**Phase 2 (Gate Decision):**
- **Deterministic Rules:** Use consistent thresholds (P0=100%, P1≥90%, overall≥80%) for objectivity
- **Evidence-Based:** Every decision must cite specific metrics (coverage %, pass rates, NFRs)
- **Waiver Discipline:** Waivers require approver name, justification, mitigation plan, and evidence link
- **Non-Blocking CONCERNS:** Use CONCERNS for minor gaps that don't justify blocking deployment (e.g., P1 at 88% vs 90%)
- **Automate in CI/CD:** Generate YAML snippets that can be consumed by CI/CD pipelines for automated quality gates
---
@@ -542,15 +1011,33 @@ Before completing this workflow, verify:
- Determine if overlap is acceptable (defense in depth) or wasteful (same validation at multiple levels)
- Consolidate tests at appropriate level (logic → unit, integration → API, journey → E2E)
### "Test execution results missing" (Phase 2)
- Phase 2 gate decision requires `test_results` (CI/CD test reports)
- If missing, Phase 2 will be skipped with warning
- Provide JUnit XML, TAP, or JSON test report path via `test_results` variable
### "Gate decision is FAIL but deployment needed urgently"
- Request business waiver (if `allow_waivers: true`)
- Document approver, justification, mitigation plan
- Create follow-up stories to address gaps
- Use WAIVED decision only for non-P0 gaps
---
## Related Workflows
- **testarch-test-design** - Define test priorities (P0/P1/P2/P3) before tracing
- **testarch-atdd** - Generate failing acceptance tests for gaps identified
- **testarch-automate** - Expand regression suite based on traceability findings
- **testarch-gate** - Use traceability matrix as input for quality gate decisions
- **testarch-test-review** - Review test quality issues flagged in traceability
**Prerequisites:**
- `testarch-test-design` - Define test priorities (P0/P1/P2/P3) before tracing (required for Phase 2)
- `testarch-atdd` or `testarch-automate` - Generate tests before tracing coverage
**Complements:**
- `testarch-nfr-assess` - Non-functional requirements validation (recommended for release gates)
- `testarch-test-review` - Review test quality issues flagged in traceability
**Next Steps:**
- If gate decision is PASS/CONCERNS → Deploy and monitor
- If gate decision is FAIL → Add missing tests, re-run trace workflow
- If gate decision is WAIVED → Deploy with mitigation, create follow-up stories
---

View File

@@ -1,12 +1,14 @@
# Traceability Matrix - Story {STORY_ID}
# Traceability Matrix & Gate Decision - Story {STORY_ID}
**Story:** {STORY_TITLE}
**Date:** {DATE}
**Status:** {OVERALL_COVERAGE}% Coverage ({GAP_COUNT} {GAP_SEVERITY} gap{s})
**Evaluator:** {user_name or TEA Agent}
---
## Coverage Summary
## PHASE 1: REQUIREMENTS TRACEABILITY
### Coverage Summary
| Priority | Total Criteria | FULL Coverage | Coverage % | Status |
| --------- | -------------- | ------------- | ---------- | ------------ |
@@ -24,9 +26,9 @@
---
## Detailed Mapping
### Detailed Mapping
### {CRITERION_ID}: {CRITERION_DESCRIPTION} ({PRIORITY})
#### {CRITERION_ID}: {CRITERION_DESCRIPTION} ({PRIORITY})
- **Coverage:** {COVERAGE_STATUS} {STATUS_ICON}
- **Tests:**
@@ -47,7 +49,7 @@
---
### Example: AC-1: User can login with email and password (P0)
#### Example: AC-1: User can login with email and password (P0)
- **Coverage:** FULL ✅
- **Tests:**
@@ -62,7 +64,7 @@
---
### Example: AC-3: User can reset password via email (P1)
#### Example: AC-3: User can reset password via email (P1)
- **Coverage:** PARTIAL ⚠️
- **Tests:**
@@ -81,26 +83,9 @@
---
### Example: AC-7: Session timeout handling (P2)
### Gap Analysis
- **Coverage:** UNIT-ONLY ⚠️
- **Tests:**
- `1.3-UNIT-006` - tests/unit/session-manager.spec.ts:42
- **Given:** Session has expired timestamp
- **When:** isSessionValid is called
- **Then:** Returns false
- **Gaps:**
- Missing: E2E validation of timeout behavior in UI
- Missing: API test for session refresh flow
- **Recommendation:** Add `1.3-E2E-005` to validate that user sees timeout message and is redirected to login. Add `1.3-API-002` to validate session refresh endpoint behavior.
---
## Gap Analysis
### Critical Gaps (BLOCKER) ❌
#### Critical Gaps (BLOCKER) ❌
{CRITICAL_GAP_COUNT} gaps found. **Do not release until resolved.**
@@ -112,7 +97,7 @@
---
### High Priority Gaps (PR BLOCKER) ⚠️
#### High Priority Gaps (PR BLOCKER) ⚠️
{HIGH_GAP_COUNT} gaps found. **Address before PR merge.**
@@ -124,7 +109,7 @@
---
### Medium Priority Gaps (Nightly) ⚠️
#### Medium Priority Gaps (Nightly) ⚠️
{MEDIUM_GAP_COUNT} gaps found. **Address in nightly test improvements.**
@@ -134,7 +119,7 @@
---
### Low Priority Gaps (Optional)
#### Low Priority Gaps (Optional)
{LOW_GAP_COUNT} gaps found. **Optional - add if time permits.**
@@ -143,9 +128,9 @@
---
## Quality Assessment
### Quality Assessment
### Tests with Issues
#### Tests with Issues
**BLOCKER Issues**
@@ -161,7 +146,7 @@
---
### Example Quality Issues
#### Example Quality Issues
**WARNING Issues** ⚠️
@@ -174,26 +159,26 @@
---
### Tests Passing Quality Gates
#### Tests Passing Quality Gates
**{PASSING_TEST_COUNT}/{TOTAL_TEST_COUNT} tests ({PASSING_PCT}%) meet all quality criteria** ✅
---
## Duplicate Coverage Analysis
### Duplicate Coverage Analysis
### Acceptable Overlap (Defense in Depth)
#### Acceptable Overlap (Defense in Depth)
- {CRITERION_ID}: Tested at unit (business logic) and E2E (user journey) ✅
### Unacceptable Duplication ⚠️
#### Unacceptable Duplication ⚠️
- {CRITERION_ID}: Same validation at E2E and Component level
- Recommendation: Remove {TEST_ID} or consolidate with {OTHER_TEST_ID}
---
## Coverage by Test Level
### Coverage by Test Level
| Test Level | Tests | Criteria Covered | Coverage % |
| ---------- | ----------------- | -------------------- | ---------------- |
@@ -205,85 +190,459 @@
---
## Gate YAML Snippet
### Traceability Recommendations
```yaml
traceability:
story_id: "{STORY_ID}"
date: "{DATE}"
coverage:
overall: {OVERALL_PCT}%
p0: {P0_PCT}%
p1: {P1_PCT}%
p2: {P2_PCT}%
p3: {P3_PCT}%
gaps:
critical: {CRITICAL_COUNT}
high: {HIGH_COUNT}
medium: {MEDIUM_COUNT}
low: {LOW_COUNT}
quality:
passing_tests: {PASSING_COUNT}
total_tests: {TOTAL_TESTS}
blocker_issues: {BLOCKER_COUNT}
warning_issues: {WARNING_COUNT}
status: "{STATUS}" # PASS / WARN / FAIL
recommendations:
- "{RECOMMENDATION_1}"
- "{RECOMMENDATION_2}"
- "{RECOMMENDATION_3}"
```
---
## Recommendations
### Immediate Actions (Before PR Merge)
#### Immediate Actions (Before PR Merge)
1. **{ACTION_1}** - {DESCRIPTION}
2. **{ACTION_2}** - {DESCRIPTION}
### Short-term Actions (This Sprint)
#### Short-term Actions (This Sprint)
1. **{ACTION_1}** - {DESCRIPTION}
2. **{ACTION_2}** - {DESCRIPTION}
### Long-term Actions (Backlog)
#### Long-term Actions (Backlog)
1. **{ACTION_1}** - {DESCRIPTION}
---
### Example Recommendations
#### Example Recommendations
### Immediate Actions (Before PR Merge)
**Immediate Actions (Before PR Merge)**
1. **Add P1 Password Reset Tests** - Implement `1.3-API-001` for email service integration and `1.3-E2E-004` for error path validation. P1 coverage currently at 80%, target is 90%.
2. **Optimize Slow E2E Test** - Refactor `1.3-E2E-001` to use faster fixture setup. Currently 145s, target is <90s.
### Short-term Actions (This Sprint)
**Short-term Actions (This Sprint)**
1. **Enhance P2 Coverage** - Add E2E validation for session timeout (`1.3-E2E-005`). Currently UNIT-ONLY coverage.
2. **Split Large Test File** - Break `1.3-UNIT-005` (320 lines) into multiple focused test files (<300 lines each).
### Long-term Actions (Backlog)
**Long-term Actions (Backlog)**
1. **Enrich P3 Coverage** - Add tests for edge cases in P3 criteria if time permits.
---
## PHASE 2: QUALITY GATE DECISION
**Gate Type:** {story | epic | release | hotfix}
**Decision Mode:** {deterministic | manual}
---
### Evidence Summary
#### Test Execution Results
- **Total Tests**: {total_count}
- **Passed**: {passed_count} ({pass_percentage}%)
- **Failed**: {failed_count} ({fail_percentage}%)
- **Skipped**: {skipped_count} ({skip_percentage}%)
- **Duration**: {total_duration}
**Priority Breakdown:**
- **P0 Tests**: {p0_passed}/{p0_total} passed ({p0_pass_rate}%) {✅ | ❌}
- **P1 Tests**: {p1_passed}/{p1_total} passed ({p1_pass_rate}%) {✅ | ⚠️ | ❌}
- **P2 Tests**: {p2_passed}/{p2_total} passed ({p2_pass_rate}%) {informational}
- **P3 Tests**: {p3_passed}/{p3_total} passed ({p3_pass_rate}%) {informational}
**Overall Pass Rate**: {overall_pass_rate}% {✅ | ⚠️ | ❌}
**Test Results Source**: {CI_run_id | test_report_url | local_run}
---
#### Coverage Summary (from Phase 1)
**Requirements Coverage:**
- **P0 Acceptance Criteria**: {p0_covered}/{p0_total} covered ({p0_coverage}%) {✅ | ❌}
- **P1 Acceptance Criteria**: {p1_covered}/{p1_total} covered ({p1_coverage}%) {✅ | ⚠️ | ❌}
- **P2 Acceptance Criteria**: {p2_covered}/{p2_total} covered ({p2_coverage}%) {informational}
- **Overall Coverage**: {overall_coverage}%
**Code Coverage** (if available):
- **Line Coverage**: {line_coverage}% {✅ | ⚠️ | ❌}
- **Branch Coverage**: {branch_coverage}% {✅ | ⚠️ | ❌}
- **Function Coverage**: {function_coverage}% {✅ | ⚠️ | ❌}
**Coverage Source**: {coverage_report_url | coverage_file_path}
---
#### Non-Functional Requirements (NFRs)
**Security**: {PASS | CONCERNS | FAIL | NOT_ASSESSED} {✅ | ⚠️ | ❌}
- Security Issues: {security_issue_count}
- {details_if_issues}
**Performance**: {PASS | CONCERNS | FAIL | NOT_ASSESSED} {✅ | ⚠️ | ❌}
- {performance_metrics_summary}
**Reliability**: {PASS | CONCERNS | FAIL | NOT_ASSESSED} {✅ | ⚠️ | ❌}
- {reliability_metrics_summary}
**Maintainability**: {PASS | CONCERNS | FAIL | NOT_ASSESSED} {✅ | ⚠️ | ❌}
- {maintainability_metrics_summary}
**NFR Source**: {nfr_assessment_file_path | not_assessed}
---
#### Flakiness Validation
**Burn-in Results** (if available):
- **Burn-in Iterations**: {iteration_count} (e.g., 10)
- **Flaky Tests Detected**: {flaky_test_count} {✅ if 0 | ❌ if >0}
- **Stability Score**: {stability_percentage}%
**Flaky Tests List** (if any):
- {flaky_test_1_name} - {failure_rate}
- {flaky_test_2_name} - {failure_rate}
**Burn-in Source**: {CI_burn_in_run_id | not_available}
---
### Decision Criteria Evaluation
#### P0 Criteria (Must ALL Pass)
| Criterion | Threshold | Actual | Status |
| --------------------- | --------- | ------------------------- | -------- | -------- |
| P0 Coverage | 100% | {p0_coverage}% | {✅ PASS | ❌ FAIL} |
| P0 Test Pass Rate | 100% | {p0_pass_rate}% | {✅ PASS | ❌ FAIL} |
| Security Issues | 0 | {security_issue_count} | {✅ PASS | ❌ FAIL} |
| Critical NFR Failures | 0 | {critical_nfr_fail_count} | {✅ PASS | ❌ FAIL} |
| Flaky Tests | 0 | {flaky_test_count} | {✅ PASS | ❌ FAIL} |
**P0 Evaluation**: {✅ ALL PASS | ❌ ONE OR MORE FAILED}
---
#### P1 Criteria (Required for PASS, May Accept for CONCERNS)
| Criterion | Threshold | Actual | Status |
| ---------------------- | ------------------------- | -------------------- | -------- | ----------- | -------- |
| P1 Coverage | ≥{min_p1_coverage}% | {p1_coverage}% | {✅ PASS | ⚠️ CONCERNS | ❌ FAIL} |
| P1 Test Pass Rate | ≥{min_p1_pass_rate}% | {p1_pass_rate}% | {✅ PASS | ⚠️ CONCERNS | ❌ FAIL} |
| Overall Test Pass Rate | ≥{min_overall_pass_rate}% | {overall_pass_rate}% | {✅ PASS | ⚠️ CONCERNS | ❌ FAIL} |
| Overall Coverage | ≥{min_coverage}% | {overall_coverage}% | {✅ PASS | ⚠️ CONCERNS | ❌ FAIL} |
**P1 Evaluation**: {✅ ALL PASS | ⚠️ SOME CONCERNS | ❌ FAILED}
---
#### P2/P3 Criteria (Informational, Don't Block)
| Criterion | Actual | Notes |
| ----------------- | --------------- | ------------------------------------------------------------ |
| P2 Test Pass Rate | {p2_pass_rate}% | {allow_p2_failures ? "Tracked, doesn't block" : "Evaluated"} |
| P3 Test Pass Rate | {p3_pass_rate}% | {allow_p3_failures ? "Tracked, doesn't block" : "Evaluated"} |
---
### GATE DECISION: {PASS | CONCERNS | FAIL | WAIVED}
---
### Rationale
{Explain decision based on criteria evaluation}
{Highlight key evidence that drove decision}
{Note any assumptions or caveats}
**Example (PASS):**
> All P0 criteria met with 100% coverage and pass rates across critical tests. All P1 criteria exceeded thresholds with 98% overall pass rate and 92% coverage. No security issues detected. No flaky tests in validation. Feature is ready for production deployment with standard monitoring.
**Example (CONCERNS):**
> All P0 criteria met, ensuring critical user journeys are protected. However, P1 coverage (88%) falls below threshold (90%) due to missing E2E test for AC-5 edge case. Overall pass rate (96%) is excellent. Issues are non-critical and have acceptable workarounds. Risk is low enough to deploy with enhanced monitoring.
**Example (FAIL):**
> CRITICAL BLOCKERS DETECTED:
>
> 1. P0 coverage incomplete (80%) - AC-2 security validation missing
> 2. P0 test failures (75% pass rate) in core search functionality
> 3. Unresolved SQL injection vulnerability in search filter (CRITICAL)
>
> Release MUST BE BLOCKED until P0 issues are resolved. Security vulnerability cannot be waived.
**Example (WAIVED):**
> Original decision was FAIL due to P0 test failure in legacy Excel 2007 export module (affects <1% of users). However, release contains critical GDPR compliance features required by regulatory deadline (Oct 15). Business has approved waiver given:
>
> - Regulatory priority overrides legacy module risk
> - Workaround available (use Excel 2010+)
> - Issue will be fixed in v2.4.1 hotfix (due Oct 20)
> - Enhanced monitoring in place
---
### {Section: Delete if not applicable}
#### Residual Risks (For CONCERNS or WAIVED)
List unresolved P1/P2 issues that don't block release but should be tracked:
1. **{Risk Description}**
- **Priority**: P1 | P2
- **Probability**: Low | Medium | High
- **Impact**: Low | Medium | High
- **Risk Score**: {probability × impact}
- **Mitigation**: {workaround or monitoring plan}
- **Remediation**: {fix in next sprint/release}
**Overall Residual Risk**: {LOW | MEDIUM | HIGH}
---
#### Waiver Details (For WAIVED only)
**Original Decision**: ❌ FAIL
**Reason for Failure**:
- {list_of_blocking_issues}
**Waiver Information**:
- **Waiver Reason**: {business_justification}
- **Waiver Approver**: {name}, {role} (e.g., Jane Doe, VP Engineering)
- **Approval Date**: {YYYY-MM-DD}
- **Waiver Expiry**: {YYYY-MM-DD} (**NOTE**: Does NOT apply to next release)
**Monitoring Plan**:
- {enhanced_monitoring_1}
- {enhanced_monitoring_2}
- {escalation_criteria}
**Remediation Plan**:
- **Fix Target**: {next_release_version} (e.g., v2.4.1 hotfix)
- **Due Date**: {YYYY-MM-DD}
- **Owner**: {team_or_person}
- **Verification**: {how_fix_will_be_verified}
**Business Justification**:
{detailed_explanation_of_why_waiver_is_acceptable}
---
#### Critical Issues (For FAIL or CONCERNS)
Top blockers requiring immediate attention:
| Priority | Issue | Description | Owner | Due Date | Status |
| -------- | ------------- | ------------------- | ------------ | ------------ | ------------------ |
| P0 | {issue_title} | {brief_description} | {owner_name} | {YYYY-MM-DD} | {OPEN/IN_PROGRESS} |
| P0 | {issue_title} | {brief_description} | {owner_name} | {YYYY-MM-DD} | {OPEN/IN_PROGRESS} |
| P1 | {issue_title} | {brief_description} | {owner_name} | {YYYY-MM-DD} | {OPEN/IN_PROGRESS} |
**Blocking Issues Count**: {p0_blocker_count} P0 blockers, {p1_blocker_count} P1 issues
---
### Gate Recommendations
#### For PASS Decision ✅
1. **Proceed to deployment**
- Deploy to staging environment
- Validate with smoke tests
- Monitor key metrics for 24-48 hours
- Deploy to production with standard monitoring
2. **Post-Deployment Monitoring**
- {metric_1_to_monitor}
- {metric_2_to_monitor}
- {alert_thresholds}
3. **Success Criteria**
- {success_criterion_1}
- {success_criterion_2}
---
#### For CONCERNS Decision ⚠️
1. **Deploy with Enhanced Monitoring**
- Deploy to staging with extended validation period
- Enable enhanced logging/monitoring for known risk areas:
- {risk_area_1}
- {risk_area_2}
- Set aggressive alerts for potential issues
- Deploy to production with caution
2. **Create Remediation Backlog**
- Create story: "{fix_title_1}" (Priority: {priority})
- Create story: "{fix_title_2}" (Priority: {priority})
- Target sprint: {next_sprint}
3. **Post-Deployment Actions**
- Monitor {specific_areas} closely for {time_period}
- Weekly status updates on remediation progress
- Re-assess after fixes deployed
---
#### For FAIL Decision ❌
1. **Block Deployment Immediately**
- Do NOT deploy to any environment
- Notify stakeholders of blocking issues
- Escalate to tech lead and PM
2. **Fix Critical Issues**
- Address P0 blockers listed in Critical Issues section
- Owner assignments confirmed
- Due dates agreed upon
- Daily standup on blocker resolution
3. **Re-Run Gate After Fixes**
- Re-run full test suite after fixes
- Re-run `bmad tea *trace` workflow
- Verify decision is PASS before deploying
---
#### For WAIVED Decision 🔓
1. **Deploy with Business Approval**
- Confirm waiver approver has signed off
- Document waiver in release notes
- Notify all stakeholders of waived risks
2. **Aggressive Monitoring**
- {enhanced_monitoring_plan}
- {escalation_procedures}
- Daily checks on waived risk areas
3. **Mandatory Remediation**
- Fix MUST be completed by {due_date}
- Issue CANNOT be waived in next release
- Track remediation progress weekly
- Verify fix in next gate
---
### Next Steps
**Immediate Actions** (next 24-48 hours):
1. {action_1}
2. {action_2}
3. {action_3}
**Follow-up Actions** (next sprint/release):
1. {action_1}
2. {action_2}
3. {action_3}
**Stakeholder Communication**:
- Notify PM: {decision_summary}
- Notify SM: {decision_summary}
- Notify DEV lead: {decision_summary}
---
## Integrated YAML Snippet (CI/CD)
```yaml
traceability_and_gate:
# Phase 1: Traceability
traceability:
story_id: "{STORY_ID}"
date: "{DATE}"
coverage:
overall: {OVERALL_PCT}%
p0: {P0_PCT}%
p1: {P1_PCT}%
p2: {P2_PCT}%
p3: {P3_PCT}%
gaps:
critical: {CRITICAL_COUNT}
high: {HIGH_COUNT}
medium: {MEDIUM_COUNT}
low: {LOW_COUNT}
quality:
passing_tests: {PASSING_COUNT}
total_tests: {TOTAL_TESTS}
blocker_issues: {BLOCKER_COUNT}
warning_issues: {WARNING_COUNT}
recommendations:
- "{RECOMMENDATION_1}"
- "{RECOMMENDATION_2}"
# Phase 2: Gate Decision
gate_decision:
decision: "{PASS | CONCERNS | FAIL | WAIVED}"
gate_type: "{story | epic | release | hotfix}"
decision_mode: "{deterministic | manual}"
criteria:
p0_coverage: {p0_coverage}%
p0_pass_rate: {p0_pass_rate}%
p1_coverage: {p1_coverage}%
p1_pass_rate: {p1_pass_rate}%
overall_pass_rate: {overall_pass_rate}%
overall_coverage: {overall_coverage}%
security_issues: {security_issue_count}
critical_nfrs_fail: {critical_nfr_fail_count}
flaky_tests: {flaky_test_count}
thresholds:
min_p0_coverage: 100
min_p0_pass_rate: 100
min_p1_coverage: {min_p1_coverage}
min_p1_pass_rate: {min_p1_pass_rate}
min_overall_pass_rate: {min_overall_pass_rate}
min_coverage: {min_coverage}
evidence:
test_results: "{CI_run_id | test_report_url}"
traceability: "{trace_file_path}"
nfr_assessment: "{nfr_file_path}"
code_coverage: "{coverage_report_url}"
next_steps: "{brief_summary_of_recommendations}"
waiver: # Only if WAIVED
reason: "{business_justification}"
approver: "{name}, {role}"
expiry: "{YYYY-MM-DD}"
remediation_due: "{YYYY-MM-DD}"
```
---
## Related Artifacts
- **Story File:** {STORY_FILE_PATH}
- **Test Design:** {TEST_DESIGN_PATH} (if available)
- **Tech Spec:** {TECH_SPEC_PATH} (if available)
- **Test Results:** {TEST_RESULTS_PATH}
- **NFR Assessment:** {NFR_FILE_PATH} (if available)
- **Test Files:** {TEST_DIR_PATH}
---
## Sign-Off
**Traceability Assessment:**
**Phase 1 - Traceability Assessment:**
- Overall Coverage: {OVERALL_PCT}%
- P0 Coverage: {P0_PCT}% {P0_STATUS}
@@ -291,16 +650,23 @@ traceability:
- Critical Gaps: {CRITICAL_COUNT}
- High Priority Gaps: {HIGH_COUNT}
**Gate Status:** {STATUS} {STATUS_ICON}
**Phase 2 - Gate Decision:**
- **Decision**: {PASS | CONCERNS | FAIL | WAIVED} {STATUS_ICON}
- **P0 Evaluation**: {✅ ALL PASS | ❌ ONE OR MORE FAILED}
- **P1 Evaluation**: {✅ ALL PASS | ⚠️ SOME CONCERNS | ❌ FAILED}
**Overall Status:** {STATUS} {STATUS_ICON}
**Next Steps:**
- If PASS ✅: Proceed to `*gate` workflow or PR merge
- If WARN ⚠️: Address HIGH priority gaps, re-run `*trace`
- If FAIL ❌: Run `*atdd` to generate missing P0 tests, re-run `*trace`
- If PASS ✅: Proceed to deployment
- If CONCERNS ⚠️: Deploy with monitoring, create remediation backlog
- If FAIL ❌: Block deployment, fix critical issues, re-run workflow
- If WAIVED 🔓: Deploy with business approval and aggressive monitoring
**Generated:** {DATE}
**Workflow:** testarch-trace v4.0
**Workflow:** testarch-trace v4.0 (Enhanced with Gate Decision)
---

View File

@@ -1,6 +1,6 @@
# Test Architect workflow: trace
# Test Architect workflow: trace (enhanced with gate decision)
name: testarch-trace
description: "Generate requirements-to-tests traceability matrix with coverage analysis and gap identification"
description: "Generate requirements-to-tests traceability matrix, analyze coverage, and make quality gate decision (PASS/CONCERNS/FAIL/WAIVED)"
author: "BMad"
# Critical variables from config
@@ -65,6 +65,46 @@ variables:
include_code_coverage: false # Integrate with code coverage reports (Istanbul, NYC)
check_assertions: true # Verify explicit assertions in tests
# PHASE 2: Gate Decision Variables (runs after traceability)
enable_gate_decision: true # Run gate decision after traceability (Phase 2)
# Gate target specification
gate_type: "story" # story | epic | release | hotfix
# story_id, epic_num, release_version inherited from trace context
# Gate decision configuration
decision_mode: "deterministic" # deterministic (rule-based) | manual (team decision)
allow_waivers: true # Allow business-approved waivers for FAIL → WAIVED
require_evidence: true # Require links to test results, reports, etc.
# Input sources for gate (auto-discovered from Phase 1 + external)
# story_file, test_design_file inherited from trace
nfr_file: "" # Path to nfr-assessment.md (optional, recommended for release gates)
test_results: "" # Path to test execution results (CI artifacts, reports)
# Decision criteria thresholds
min_p0_pass_rate: 100 # P0 tests must have 100% pass rate
min_p1_pass_rate: 95 # P1 tests threshold
min_overall_pass_rate: 90 # Overall test pass rate
# min_coverage already defined above (min_overall_coverage: 80)
max_critical_nfrs_fail: 0 # No critical NFRs can fail
max_security_issues: 0 # No unresolved security issues
# Risk tolerance
allow_p2_failures: true # P2 failures don't block release
allow_p3_failures: true # P3 failures don't block release
escalate_p1_failures: true # P1 failures require escalation approval
# Gate output configuration
gate_output_file: "{output_folder}/gate-decision-{gate_type}-{story_id}{epic_num}{release_version}.md"
append_to_history: true # Append to bmm-workflow-status.md gate history
notify_stakeholders: true # Generate notification message for team
# Advanced gate options
check_all_workflows_complete: true # Verify test-design, trace, nfr-assess complete
validate_evidence_freshness: true # Warn if assessments are >7 days old
require_sign_off: false # Require named approver for gate decision
# Output configuration
default_output_file: "{output_folder}/traceability-matrix.md"
@@ -80,9 +120,12 @@ required_tools:
recommended_inputs:
- story: "Story markdown with acceptance criteria (required for BMad mode)"
- test_files: "Test suite for the feature (auto-discovered if not provided)"
- test_design: "Test design with risk/priority assessment (optional)"
- test_design: "Test design with risk/priority assessment (required for Phase 2 gate)"
- tech_spec: "Technical specification (optional)"
- existing_tests: "Current test suite for analysis"
- test_results: "CI/CD test execution results (required for Phase 2 gate)"
- nfr_assess: "Non-functional requirements validation (recommended for release gates)"
- code_coverage: "Code coverage report (optional)"
tags:
- qa
@@ -90,6 +133,9 @@ tags:
- test-architect
- coverage
- requirements
- gate
- decision
- release
execution_hints:
interactive: false # Minimize prompts