BMAD-METHOD/docs/document-sharding-guide.md

Document Sharding Guide

Comprehensive guide to BMad Method's document sharding system for managing large planning and architecture documents.

Table of Contents

• What is Document Sharding?
• When to Use Sharding
• How Sharding Works
• Using the Shard-Doc Tool
• Workflow Support
• Best Practices
• Examples

What is Document Sharding?

Document sharding splits large markdown files into smaller, organized files based on level 2 headings (## Heading). This enables:

• Selective Loading - Workflows load only the sections they need
• Reduced Token Usage - Massive efficiency gains for large projects
• Better Organization - Logical section-based file structure
• Maintained Context - Index file preserves document structure

Architecture

Before Sharding:
docs/
└── PRD.md (large 50k token file)

After Sharding:
docs/
└── prd/
    ├── index.md                    # Table of contents with descriptions
    ├── overview.md                 # Section 1
    ├── user-requirements.md        # Section 2
    ├── technical-requirements.md   # Section 3
    └── ...                         # Additional sections

When to Use Sharding

Ideal Candidates

Large Multi-Epic Projects:

• Very large complex PRDs
• Architecture documents with multiple system layers
• Epic files with 4+ epics (especially for Phase 4)
• UX design specs covering multiple subsystems

Token Thresholds:

• Consider sharding: Documents > 20k tokens
• Strongly recommended: Documents > 40k tokens
• Critical for efficiency: Documents > 60k tokens

When NOT to Shard

Small Projects:

• Single epic projects
• Level 0-1 projects (tech-spec only)
• Documents under 10k tokens
• Quick prototypes

Frequently Updated Docs:

• Active work-in-progress documents
• Documents updated daily
• Documents where whole-file context is essential

How Sharding Works

Sharding Process

1. Tool Execution: Run npx @kayvan/markdown-tree-parser source.md destination/ - this is abstracted with the core shard-doc task which is installed as a slash command or manual task rule depending on your tools.
2. Section Extraction: Tool splits by level 2 headings
3. File Creation: Each section becomes a separate file
4. Index Generation: index.md created with structure and descriptions

Workflow Discovery

BMad workflows use a dual discovery system:

1. Try whole document first - Look for document-name.md
2. Check for sharded version - Look for document-name/index.md
3. Priority rule - Whole document takes precedence if both exist

Loading Strategies

Full Load (Phase 1-3 workflows):

If sharded:
  - Read index.md
  - Read ALL section files
  - Treat as single combined document

Selective Load (Phase 4 workflows):

If sharded epics and working on Epic 3:
  - Read epics/index.md
  - Load ONLY epics/epic-3.md
  - Skip all other epic files
  - 90%+ token savings!

Using the Shard-Doc Tool

CLI Command

# Activate bmad-master or analyst agent, then:
/shard-doc

Interactive Process

Agent: Which document would you like to shard?
User: docs/PRD.md

Agent: Default destination: docs/prd/
       Accept default? [y/n]
User: y

Agent: Sharding PRD.md...
       ✓ Created 12 section files
       ✓ Generated index.md
       ✓ Complete!

What Gets Created

index.md structure:

# PRD - Index

## Sections

1. [Overview](./overview.md) - Project vision and objectives
2. [User Requirements](./user-requirements.md) - Feature specifications
3. [Epic 1: Authentication](./epic-1-authentication.md) - User auth system
4. [Epic 2: Dashboard](./epic-2-dashboard.md) - Main dashboard UI
   ...

Individual section files:

• Named from heading text (kebab-case)
• Contains complete section content
• Preserves all markdown formatting
• Can be read independently

Workflow Support

Universal Support

All BMM workflows support both formats:

• ✅ Whole documents
• ✅ Sharded documents
• ✅ Automatic detection
• ✅ Transparent to user

Workflow-Specific Patterns

Phase 1-3 (Full Load)

Workflows load entire sharded documents:

• product-brief - Research, brainstorming docs
• prd - Product brief, research
• gdd - Game brief, research
• create-ux-design - PRD, brief, epics
• tech-spec - Brief, research
• architecture - PRD, epics, UX design
• solutioning-gate-check - All planning docs

Phase 4 (Selective Load)

Workflows load only needed sections:

sprint-planning (Full Load):

• Needs ALL epics to build complete status

epic-tech-context, create-story, story-context, code-review (Selective):

Working on Epic 3, Story 2:
  ✓ Load epics/epic-3.md only
  ✗ Skip epics/epic-1.md, epic-2.md, epic-4.md, etc.

Result: 90%+ token reduction for 10-epic projects!

Input File Patterns

Workflows use standardized patterns:

input_file_patterns:
  prd:
    whole: '{output_folder}/*prd*.md'
    sharded: '{output_folder}/*prd*/index.md'

  epics:
    whole: '{output_folder}/*epic*.md'
    sharded_index: '{output_folder}/*epic*/index.md'
    sharded_single: '{output_folder}/*epic*/epic-{{epic_num}}.md'

Best Practices

Sharding Strategy

Do:

• ✅ Shard after planning phase complete
• ✅ Keep level 2 headings well-organized
• ✅ Use descriptive section names
• ✅ Shard before Phase 4 implementation
• ✅ Keep original file as backup initially

Don't:

• ❌ Shard work-in-progress documents
• ❌ Shard small documents (<20k tokens)
• ❌ Mix sharded and whole versions
• ❌ Manually edit index.md structure

Naming Conventions

Good Section Names:

## Epic 1: User Authentication

## Technical Requirements

## System Architecture

## UX Design Principles

Poor Section Names:

## Section 1

## Part A

## Details

## More Info

File Management

When to Re-shard:

• Significant structural changes to document
• Adding/removing major sections
• After major refactoring

Updating Sharded Docs:

1. Edit individual section files directly
2. OR edit original, delete sharded folder, re-shard
3. Don't manually edit index.md

Examples

Example 1: Large PRD

Scenario: 15-epic project, PRD is 45k tokens

Before Sharding:

Every workflow loads entire 45k token PRD
Epic-tech-context for Epic 3: 45k tokens
Create-story for Epic 3: 45k tokens

After Sharding:

/shard-doc
Source: docs/PRD.md
Destination: docs/prd/

Created:
  prd/index.md
  prd/overview.md (3k tokens)
  prd/epic-1-auth.md (3k tokens)
  prd/epic-2-dashboard.md (3k tokens)
  prd/epic-3-reports.md (3k tokens)
  ...15 epic files

Result:

Epic-tech-context for Epic 3: 3k tokens (93% reduction!)
Create-story for Epic 3: 3k tokens (93% reduction!)

Example 2: Sharding Epics File

Scenario: 8 epics with detailed stories, 35k tokens total

/shard-doc
Source: docs/bmm-epics.md
Destination: docs/epics/

Created:
  epics/index.md
  epics/epic-1.md
  epics/epic-2.md
  ...
  epics/epic-8.md

Efficiency Gain:

Working on Epic 5 stories:
  Old: Load all 8 epics (35k tokens)
  New: Load epic-5.md only (4k tokens)
  Savings: 88% reduction

Example 3: Architecture Document

Scenario: Multi-layer system architecture, 28k tokens

/shard-doc
Source: docs/architecture.md
Destination: docs/architecture/

Created:
  architecture/index.md
  architecture/system-overview.md
  architecture/frontend-architecture.md
  architecture/backend-services.md
  architecture/data-layer.md
  architecture/infrastructure.md
  architecture/security-architecture.md

Benefit: Code-review workflow can reference specific architectural layers without loading entire architecture doc.

Custom Workflow Integration

For Workflow Builders

When creating custom workflows that load large documents:

1. Add input_file_patterns to workflow.yaml:

input_file_patterns:
  your_document:
    whole: '{output_folder}/*your-doc*.md'
    sharded: '{output_folder}/*your-doc*/index.md'

2. Add discovery instructions to instructions.md:

## Document Discovery

1. Search for whole document: _your-doc_.md
2. Check for sharded version: _your-doc_/index.md
3. If sharded: Read index + ALL sections (or specific sections if selective load)
4. Priority: Whole document first

3. Choose loading strategy:

• Full Load: Read all sections when sharded
• Selective Load: Read only relevant sections (requires section identification logic)

Pattern Templates

Full Load Pattern:

<action>Search for document: {output_folder}/*doc-name*.md</action>
<action>If not found, check for sharded: {output_folder}/*doc-name*/index.md</action>
<action if="sharded found">Read index.md to understand structure</action>
<action if="sharded found">Read ALL section files listed in index</action>
<action if="sharded found">Combine content as single document</action>

Selective Load Pattern (with section ID):

<action>Determine section needed (e.g., epic_num = 3)</action>
<action>Check for sharded version: {output_folder}/*doc-name*/index.md</action>
<action if="sharded found">Read ONLY the specific section file needed</action>
<action if="sharded found">Skip all other section files</action>

Troubleshooting

Common Issues

Both whole and sharded exist:

• Workflows will use whole document (priority rule)
• Delete or archive the one you don't want

Index.md out of sync:

• Delete sharded folder
• Re-run shard-doc on original

Workflow can't find document:

• Check file naming matches patterns (*prd*.md, *epic*.md, etc.)
• Verify index.md exists in sharded folder
• Check output_folder path in config

Sections too granular:

• Combine sections in original document
• Use fewer level 2 headings
• Re-shard

Related Documentation

• shard-doc Tool - Tool implementation
• BMM Workflows Guide - Workflow overview
• Workflow Creation Guide - Custom workflow patterns

---

Document sharding is optional but powerful - use it when efficiency matters for large projects!