mirror of
https://github.com/bmadcode/BMAD-METHOD.git
synced 2025-12-29 16:14:59 +00:00
feat: test design for architecture level (phase 3) (#897)
* feat: test design for architecture level (phase 3) * addressed review comments --------- Co-authored-by: Murat Ozcan <murat@Murats-MacBook-Pro.local> Co-authored-by: Murat Ozcan <murat@mac.lan>
This commit is contained in:
Murat K Ozcan
@@ -204,6 +204,17 @@
|
||||
- Over-engineering indicators
|
||||
</action>
|
||||
|
||||
<action>Check Testability Review (if test-design exists in Phase 3):
|
||||
|
||||
**Note:** test-design is recommended for BMad Method, required for Enterprise Method
|
||||
|
||||
- Check if {output_folder}/test-design-system.md exists
|
||||
- If exists: Review testability assessment (Controllability, Observability, Reliability)
|
||||
- If testability concerns documented: Flag for gate decision
|
||||
- If missing AND track is Enterprise: Flag as CRITICAL gap
|
||||
- If missing AND track is Method: Note as recommendation (not blocker)
|
||||
</action>
|
||||
|
||||
<template-output>gap_risk_analysis</template-output>
|
||||
</step>
|
||||
|
||||
|
||||
@@ -9,24 +9,77 @@
|
||||
|
||||
## Overview
|
||||
|
||||
Plans comprehensive test coverage strategy with risk assessment, priority classification, and execution ordering. This workflow generates a test design document that identifies high-risk areas, maps requirements to test levels, prioritizes scenarios (P0-P3), and provides resource estimates for the testing effort.
|
||||
Plans comprehensive test coverage strategy with risk assessment, priority classification, and execution ordering. This workflow operates in **two modes**:
|
||||
|
||||
- **System-Level Mode (Phase 3)**: Testability review of architecture before solutioning gate check
|
||||
- **Epic-Level Mode (Phase 4)**: Per-epic test planning with risk assessment (current behavior)
|
||||
|
||||
The workflow auto-detects which mode to use based on project phase.
|
||||
|
||||
---
|
||||
|
||||
## Preflight Requirements
|
||||
## Preflight: Detect Mode and Load Context
|
||||
|
||||
**Critical:** Verify these requirements before proceeding. If any fail, HALT and notify the user.
|
||||
**Critical:** Determine mode before proceeding.
|
||||
|
||||
- ✅ Story markdown with acceptance criteria available
|
||||
- ✅ PRD or epic documentation exists for context
|
||||
- ✅ Architecture documents available (optional but recommended)
|
||||
- ✅ Requirements are clear and testable
|
||||
### Mode Detection
|
||||
|
||||
1. **Check for sprint-status.yaml**
|
||||
- If `{output_folder}/bmm-sprint-status.yaml` exists → **Epic-Level Mode** (Phase 4)
|
||||
- If NOT exists → Check workflow status
|
||||
|
||||
2. **Check workflow-status.yaml**
|
||||
- Read `{output_folder}/bmm-workflow-status.yaml`
|
||||
- If `solutioning-gate-check: required` or `solutioning-gate-check: recommended` → **System-Level Mode** (Phase 3)
|
||||
- Otherwise → **Epic-Level Mode** (Phase 4 without sprint status yet)
|
||||
|
||||
3. **Mode-Specific Requirements**
|
||||
|
||||
**System-Level Mode (Phase 3 - Testability Review):**
|
||||
- ✅ Architecture document exists (architecture.md or tech-spec)
|
||||
- ✅ PRD exists with functional and non-functional requirements
|
||||
- ✅ Epics documented (epics.md)
|
||||
- ⚠️ Output: `{output_folder}/test-design-system.md`
|
||||
|
||||
**Epic-Level Mode (Phase 4 - Per-Epic Planning):**
|
||||
- ✅ Story markdown with acceptance criteria available
|
||||
- ✅ PRD or epic documentation exists for context
|
||||
- ✅ Architecture documents available (optional but recommended)
|
||||
- ✅ Requirements are clear and testable
|
||||
- ⚠️ Output: `{output_folder}/test-design-epic-{epic_num}.md`
|
||||
|
||||
**Halt Condition:** If mode cannot be determined or required files missing, HALT and notify user with missing prerequisites.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Load Context and Requirements
|
||||
## Step 1: Load Context (Mode-Aware)
|
||||
|
||||
### Actions
|
||||
**Mode-Specific Loading:**
|
||||
|
||||
### System-Level Mode (Phase 3)
|
||||
|
||||
1. **Read Architecture Documentation**
|
||||
- Load architecture.md or tech-spec (REQUIRED)
|
||||
- Load PRD.md for functional and non-functional requirements
|
||||
- Load epics.md for feature scope
|
||||
- Identify technology stack decisions (frameworks, databases, deployment targets)
|
||||
- Note integration points and external system dependencies
|
||||
- Extract NFR requirements (performance SLOs, security requirements, etc.)
|
||||
|
||||
2. **Load Knowledge Base Fragments (System-Level)**
|
||||
|
||||
**Critical:** Consult `{project-root}/{bmad_folder}/bmm/testarch/tea-index.csv` to load:
|
||||
- `nfr-criteria.md` - NFR validation approach (security, performance, reliability, maintainability)
|
||||
- `test-levels-framework.md` - Test levels strategy guidance
|
||||
- `risk-governance.md` - Testability risk identification
|
||||
- `test-quality.md` - Quality standards and Definition of Done
|
||||
|
||||
3. **Analyze Existing Test Setup (if brownfield)**
|
||||
- Search for existing test directories
|
||||
- Identify current test framework (if any)
|
||||
- Note testability concerns in existing codebase
|
||||
|
||||
### Epic-Level Mode (Phase 4)
|
||||
|
||||
1. **Read Requirements Documentation**
|
||||
- Load PRD.md for high-level product requirements
|
||||
@@ -37,6 +90,7 @@ Plans comprehensive test coverage strategy with risk assessment, priority classi
|
||||
2. **Load Architecture Context**
|
||||
- Read architecture.md for system design
|
||||
- Read tech-spec for implementation details
|
||||
- Read test-design-system.md (if exists from Phase 3)
|
||||
- Identify technical constraints and dependencies
|
||||
- Note integration points and external systems
|
||||
|
||||
@@ -46,7 +100,7 @@ Plans comprehensive test coverage strategy with risk assessment, priority classi
|
||||
- Note areas with insufficient testing
|
||||
- Check for flaky or outdated tests
|
||||
|
||||
4. **Load Knowledge Base Fragments**
|
||||
4. **Load Knowledge Base Fragments (Epic-Level)**
|
||||
|
||||
**Critical:** Consult `{project-root}/{bmad_folder}/bmm/testarch/tea-index.csv` to load:
|
||||
- `risk-governance.md` - Risk classification framework (6 categories: TECH, SEC, PERF, DATA, BUS, OPS), automated scoring, gate decision engine, owner tracking (625 lines, 4 examples)
|
||||
@@ -54,11 +108,118 @@ Plans comprehensive test coverage strategy with risk assessment, priority classi
|
||||
- `test-levels-framework.md` - Test level selection guidance (E2E vs API vs Component vs Unit with decision matrix, characteristics, when to use each, 467 lines, 4 examples)
|
||||
- `test-priorities-matrix.md` - P0-P3 prioritization criteria (automated priority calculation, risk-based mapping, tagging strategy, time budgets, 389 lines, 2 examples)
|
||||
|
||||
**Halt Condition:** If story data or acceptance criteria are missing, check if brownfield exploration is needed. If neither requirements NOR exploration possible, HALT with message: "Test design requires clear requirements, acceptance criteria, or brownfield app URL for exploration"
|
||||
**Halt Condition (Epic-Level only):** If story data or acceptance criteria are missing, check if brownfield exploration is needed. If neither requirements NOR exploration possible, HALT with message: "Epic-level test design requires clear requirements, acceptance criteria, or brownfield app URL for exploration"
|
||||
|
||||
---
|
||||
|
||||
## Step 1.5: Mode Selection (NEW - Phase 2.5)
|
||||
## Step 1.5: System-Level Testability Review (Phase 3 Only)
|
||||
|
||||
**Skip this step if Epic-Level Mode.** This step only executes in System-Level Mode.
|
||||
|
||||
### Actions
|
||||
|
||||
1. **Review Architecture for Testability**
|
||||
|
||||
Evaluate architecture against these criteria:
|
||||
|
||||
**Controllability:**
|
||||
- Can we control system state for testing? (API seeding, factories, database reset)
|
||||
- Are external dependencies mockable? (interfaces, dependency injection)
|
||||
- Can we trigger error conditions? (chaos engineering, fault injection)
|
||||
|
||||
**Observability:**
|
||||
- Can we inspect system state? (logging, metrics, traces)
|
||||
- Are test results deterministic? (no race conditions, clear success/failure)
|
||||
- Can we validate NFRs? (performance metrics, security audit logs)
|
||||
|
||||
**Reliability:**
|
||||
- Are tests isolated? (parallel-safe, stateless, cleanup discipline)
|
||||
- Can we reproduce failures? (deterministic waits, HAR capture, seed data)
|
||||
- Are components loosely coupled? (mockable, testable boundaries)
|
||||
|
||||
2. **Identify Architecturally Significant Requirements (ASRs)**
|
||||
|
||||
From PRD NFRs and architecture decisions, identify quality requirements that:
|
||||
- Drive architecture decisions (e.g., "Must handle 10K concurrent users" → caching architecture)
|
||||
- Pose testability challenges (e.g., "Sub-second response time" → performance test infrastructure)
|
||||
- Require special test environments (e.g., "Multi-region deployment" → regional test instances)
|
||||
|
||||
Score each ASR using risk matrix (probability × impact).
|
||||
|
||||
3. **Define Test Levels Strategy**
|
||||
|
||||
Based on architecture (mobile, web, API, microservices, monolith):
|
||||
- Recommend unit/integration/E2E split (e.g., 70/20/10 for API-heavy, 40/30/30 for UI-heavy)
|
||||
- Identify test environment needs (local, staging, ephemeral, production-like)
|
||||
- Define testing approach per technology (Playwright for web, Maestro for mobile, k6 for performance)
|
||||
|
||||
4. **Assess NFR Testing Approach**
|
||||
|
||||
For each NFR category:
|
||||
- **Security**: Auth/authz tests, OWASP validation, secret handling (Playwright E2E + security tools)
|
||||
- **Performance**: Load/stress/spike testing with k6, SLO/SLA thresholds
|
||||
- **Reliability**: Error handling, retries, circuit breakers, health checks (Playwright + API tests)
|
||||
- **Maintainability**: Coverage targets, code quality gates, observability validation
|
||||
|
||||
5. **Flag Testability Concerns**
|
||||
|
||||
Identify architecture decisions that harm testability:
|
||||
- ❌ Tight coupling (no interfaces, hard dependencies)
|
||||
- ❌ No dependency injection (can't mock external services)
|
||||
- ❌ Hardcoded configurations (can't test different envs)
|
||||
- ❌ Missing observability (can't validate NFRs)
|
||||
- ❌ Stateful designs (can't parallelize tests)
|
||||
|
||||
**Critical:** If testability concerns are blockers (e.g., "Architecture makes performance testing impossible"), document as CONCERNS or FAIL recommendation for gate check.
|
||||
|
||||
6. **Output System-Level Test Design**
|
||||
|
||||
Write to `{output_folder}/test-design-system.md` containing:
|
||||
|
||||
```markdown
|
||||
# System-Level Test Design
|
||||
|
||||
## Testability Assessment
|
||||
|
||||
- Controllability: [PASS/CONCERNS/FAIL with details]
|
||||
- Observability: [PASS/CONCERNS/FAIL with details]
|
||||
- Reliability: [PASS/CONCERNS/FAIL with details]
|
||||
|
||||
## Architecturally Significant Requirements (ASRs)
|
||||
|
||||
[Risk-scored quality requirements]
|
||||
|
||||
## Test Levels Strategy
|
||||
|
||||
- Unit: [X%] - [Rationale]
|
||||
- Integration: [Y%] - [Rationale]
|
||||
- E2E: [Z%] - [Rationale]
|
||||
|
||||
## NFR Testing Approach
|
||||
|
||||
- Security: [Approach with tools]
|
||||
- Performance: [Approach with tools]
|
||||
- Reliability: [Approach with tools]
|
||||
- Maintainability: [Approach with tools]
|
||||
|
||||
## Test Environment Requirements
|
||||
|
||||
[Infrastructure needs based on deployment architecture]
|
||||
|
||||
## Testability Concerns (if any)
|
||||
|
||||
[Blockers or concerns that should inform solutioning gate check]
|
||||
|
||||
## Recommendations for Sprint 0
|
||||
|
||||
[Specific actions for *framework and *ci workflows]
|
||||
```
|
||||
|
||||
**After System-Level Mode:** Skip to Step 4 (Generate Deliverables) - Steps 2-3 are epic-level only.
|
||||
|
||||
---
|
||||
|
||||
## Step 1.6: Exploratory Mode Selection (Epic-Level Only)
|
||||
|
||||
### Actions
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Test Architect workflow: test-design
|
||||
name: testarch-test-design
|
||||
description: "Plan risk mitigation and test coverage strategy before development with risk assessment and prioritization"
|
||||
description: "Dual-mode workflow: (1) System-level testability review in Solutioning phase, or (2) Epic-level test planning in Implementation phase. Auto-detects mode based on project phase."
|
||||
author: "BMad"
|
||||
|
||||
# Critical variables from config
|
||||
@@ -20,8 +20,12 @@ template: "{installed_path}/test-design-template.md"
|
||||
# Variables and inputs
|
||||
variables:
|
||||
design_level: "full" # full, targeted, minimal - scope of design effort
|
||||
mode: "auto-detect" # auto-detect (default), system-level, epic-level
|
||||
|
||||
# Output configuration
|
||||
# Note: Actual output file determined dynamically based on mode detection
|
||||
# - System-Level (Phase 3): {output_folder}/test-design-system.md
|
||||
# - Epic-Level (Phase 4): {output_folder}/test-design-epic-{epic_num}.md
|
||||
default_output_file: "{output_folder}/test-design-epic-{epic_num}.md"
|
||||
|
||||
# Required tools
|
||||
|
||||
@@ -66,20 +66,6 @@ phases:
|
||||
command: "create-design"
|
||||
note: "Recommended - must integrate with existing UX patterns"
|
||||
|
||||
- id: "framework"
|
||||
optional: true
|
||||
agent: "tea"
|
||||
command: "framework"
|
||||
output: "Test framework scaffold (Playwright/Cypress)"
|
||||
note: "Initialize or modernize test framework - critical if brownfield lacks proper test infrastructure"
|
||||
|
||||
- id: "ci"
|
||||
optional: true
|
||||
agent: "tea"
|
||||
command: "ci"
|
||||
output: "CI/CD test pipeline configuration"
|
||||
note: "Establish or enhance CI pipeline with regression testing strategy"
|
||||
|
||||
- phase: 2
|
||||
name: "Solutioning"
|
||||
required: true
|
||||
@@ -91,6 +77,13 @@ phases:
|
||||
output: "Integration architecture with enterprise considerations"
|
||||
note: "Distills brownfield context + adds security/scalability/compliance design"
|
||||
|
||||
- id: "test-design"
|
||||
required: true
|
||||
agent: "tea"
|
||||
command: "test-design"
|
||||
output: "System-level testability review"
|
||||
note: "Enterprise requires testability validation - auto-detects system-level mode"
|
||||
|
||||
- id: "create-security-architecture"
|
||||
optional: true
|
||||
agent: "architect"
|
||||
@@ -106,7 +99,7 @@ phases:
|
||||
note: "Future workflow - optional extended enterprise workflow for CI/CD integration, deployment strategy, monitoring"
|
||||
|
||||
- id: "validate-architecture"
|
||||
optional: true
|
||||
recommended: true
|
||||
agent: "architect"
|
||||
command: "validate-architecture"
|
||||
|
||||
|
||||
@@ -54,20 +54,6 @@ phases:
|
||||
command: "create-design"
|
||||
note: "Highly recommended for enterprise - design system and patterns"
|
||||
|
||||
- id: "framework"
|
||||
optional: true
|
||||
agent: "tea"
|
||||
command: "framework"
|
||||
output: "Test framework scaffold (Playwright/Cypress)"
|
||||
note: "Initialize production-ready test framework - run once per project"
|
||||
|
||||
- id: "ci"
|
||||
optional: true
|
||||
agent: "tea"
|
||||
command: "ci"
|
||||
output: "CI/CD test pipeline configuration"
|
||||
note: "Scaffold CI workflow with selective testing, burn-in, caching"
|
||||
|
||||
- phase: 2
|
||||
name: "Solutioning"
|
||||
required: true
|
||||
@@ -79,6 +65,13 @@ phases:
|
||||
output: "Enterprise-grade system architecture"
|
||||
note: "Includes scalability, multi-tenancy, integration architecture"
|
||||
|
||||
- id: "test-design"
|
||||
required: true
|
||||
agent: "tea"
|
||||
command: "test-design"
|
||||
output: "System-level testability review"
|
||||
note: "Enterprise requires testability validation - auto-detects system-level mode"
|
||||
|
||||
- id: "create-security-architecture"
|
||||
optional: true
|
||||
agent: "architect"
|
||||
@@ -94,7 +87,7 @@ phases:
|
||||
note: "Future workflow - optional extended enterprise workflow for CI/CD, deployment, monitoring"
|
||||
|
||||
- id: "validate-architecture"
|
||||
optional: true
|
||||
recommended: true
|
||||
agent: "architect"
|
||||
command: "validate-architecture"
|
||||
|
||||
@@ -102,7 +95,7 @@ phases:
|
||||
required: true
|
||||
agent: "architect"
|
||||
command: "solutioning-gate-check"
|
||||
note: "Validates all planning artifacts align before implementation"
|
||||
note: "Validates all planning artifacts + testability align before implementation"
|
||||
|
||||
- phase: 3
|
||||
name: "Implementation"
|
||||
|
||||
@@ -75,6 +75,13 @@ phases:
|
||||
output: "Integration architecture - solution design for THIS project"
|
||||
note: "HIGHLY RECOMMENDED: Distills massive brownfield context into focused solution design. Prevents agent confusion."
|
||||
|
||||
- id: "test-design"
|
||||
recommended: true
|
||||
agent: "tea"
|
||||
command: "test-design"
|
||||
output: "System-level testability review"
|
||||
note: "Testability assessment before gate check - auto-detects system-level mode"
|
||||
|
||||
- id: "validate-architecture"
|
||||
optional: true
|
||||
agent: "architect"
|
||||
|
||||
@@ -65,6 +65,13 @@ phases:
|
||||
output: "System architecture document"
|
||||
note: "Complete system design for greenfield projects"
|
||||
|
||||
- id: "test-design"
|
||||
recommended: true
|
||||
agent: "tea"
|
||||
command: "test-design"
|
||||
output: "System-level testability review"
|
||||
note: "Testability assessment before gate check - auto-detects system-level mode"
|
||||
|
||||
- id: "validate-architecture"
|
||||
optional: true
|
||||
agent: "architect"
|
||||
@@ -75,7 +82,7 @@ phases:
|
||||
required: true
|
||||
agent: "architect"
|
||||
command: "solutioning-gate-check"
|
||||
note: "Validates PRD + UX + Architecture cohesion before implementation"
|
||||
note: "Validates PRD + UX + Architecture + Testability cohesion before implementation"
|
||||
|
||||
- phase: 3
|
||||
name: "Implementation"
|
||||
|
||||
Reference in New Issue
Block a user