SuperClaude/docs/memory/README.md

# Memory Directory

This directory contains memory and learning data for the SuperClaude Framework's PM Agent.

## Overview

The PM Agent uses multiple memory systems to learn, improve, and maintain context across sessions:

1. **ReflexionMemory** - Error learning and pattern recognition
2. **Workflow Metrics** - Performance tracking and optimization
3. **Pattern Learning** - Successful implementation patterns

## Files

### reflexion.jsonl (Auto-generated)
**Purpose**: Error learning database
**Format**: [JSON Lines](https://jsonlines.org/)
**Generated by**: ReflexionMemory system (`superclaude/core/pm_init/reflexion_memory.py`)

Stores past errors, root causes, and solutions for instant error resolution.

**Example entry**:
```json
{
  "ts": "2025-10-30T14:23:45+09:00",
  "task": "implement JWT authentication",
  "mistake": "JWT validation failed",
  "evidence": "TypeError: secret undefined",
  "rule": "Check env vars before auth implementation",
  "fix": "Added JWT_SECRET to .env",
  "tests": ["Verify .env vars", "Test JWT signing"],
  "status": "adopted"
}
```

**User Guide**: See [docs/user-guide/memory-system.md](../user-guide/memory-system.md)

### reflexion.jsonl.example
**Purpose**: Sample reflexion entries for reference
**Status**: Template file (15 realistic examples)

Copy this to `reflexion.jsonl` if you want to start with example data, or let the system create it automatically on first error.

### workflow_metrics.jsonl (Auto-generated)
**Purpose**: Task performance tracking
**Format**: JSON Lines
**Generated by**: PM Agent workflow system

Tracks token usage, execution time, and success rates for continuous optimization.

**Example entry**:
```json
{
  "timestamp": "2025-10-17T01:54:21+09:00",
  "session_id": "abc123",
  "task_type": "bug_fix",
  "complexity": "light",
  "workflow_id": "progressive_v3_layer2",
  "layers_used": [0, 1, 2],
  "tokens_used": 650,
  "time_ms": 1800,
  "success": true
}
```

**Schema**: See [WORKFLOW_METRICS_SCHEMA.md](WORKFLOW_METRICS_SCHEMA.md)

### patterns_learned.jsonl (Auto-generated)
**Purpose**: Successful implementation patterns
**Format**: JSON Lines
**Generated by**: PM Agent learning system

Captures reusable patterns from successful implementations.

### Documentation Files

#### WORKFLOW_METRICS_SCHEMA.md
Complete schema definition for workflow metrics data, including field types, descriptions, and examples.

#### pm_context.md
Documentation of the PM Agent's context management system, including progressive loading strategy and token efficiency.

#### token_efficiency_validation.md
Validation results and benchmarks for token efficiency optimizations.

#### last_session.md
Session notes and context from previous work sessions.

#### next_actions.md
Planned improvements and next steps for the memory system.

## File Management

### Automatic Files

These files are **automatically created and managed** by the system:
- `reflexion.jsonl` - Created on first error
- `workflow_metrics.jsonl` - Created on first task
- `patterns_learned.jsonl` - Created when patterns are learned

**Don't manually create these files** - the system handles it.

### When Files Are Missing

If `reflexion.jsonl` doesn't exist:
- ✅ Normal on first run
- ✅ Will be created automatically when first error occurs
- ✅ No action needed

### Backup and Maintenance

**Backup**:
```bash
# Archive old learnings
tar -czf memory-backup-$(date +%Y%m%d).tar.gz docs/memory/*.jsonl
```

**Clean old entries** (if files grow too large):
```bash
# Keep last 100 entries
tail -100 docs/memory/reflexion.jsonl > reflexion.tmp
mv reflexion.tmp docs/memory/reflexion.jsonl
```

**Validate JSON format**:
```bash
# Check all lines are valid JSON
cat docs/memory/reflexion.jsonl | while read line; do
  echo "$line" | jq . >/dev/null || echo "Invalid: $line"
done
```

## Git and Version Control

### What to Commit

✅ **Should be committed**:
- `reflexion.jsonl.example` (template)
- `patterns_learned.jsonl` (shared patterns)
- Documentation files (*.md)

❓ **Optional to commit**:
- `reflexion.jsonl` (team-specific learnings)
- `workflow_metrics.jsonl` (performance data)

**Recommendation**: Add `reflexion.jsonl` to `.gitignore` if learnings are developer-specific.

### Gitignore Configuration

If you want personal memory (not shared with team):
```bash
# Add to .gitignore
echo "docs/memory/reflexion.jsonl" >> .gitignore
echo "docs/memory/workflow_metrics.jsonl" >> .gitignore
```

If you want shared team memory (everyone benefits):
```bash
# Keep files in git (current default)
# All team members learn from each other's mistakes
```

## Privacy and Security

### What's Stored

ReflexionMemory stores:
- ✅ Error messages
- ✅ Task descriptions
- ✅ Solution approaches
- ✅ Timestamps

It does **NOT** store:
- ❌ Passwords or secrets
- ❌ API keys
- ❌ Personal data
- ❌ Production data

### Sensitive Information

If an error message contains sensitive info:
1. The entry will be in `reflexion.jsonl`
2. Manually edit the file to redact sensitive data
3. Keep the learning, remove the secret

**Example**:
```json
// Before (contains secret)
{"evidence": "Auth failed with key abc123xyz"}

// After (redacted)
{"evidence": "Auth failed with invalid API key"}
```

## Performance

### File Sizes

Expected file sizes:
- `reflexion.jsonl`: 1-10 KB per 10 entries (~1MB per 1000 errors)
- `workflow_metrics.jsonl`: 0.5-1 KB per entry
- `patterns_learned.jsonl`: 2-5 KB per pattern

### Search Performance

ReflexionMemory search is fast:
- **<10ms** for files under 1MB
- **<50ms** for files under 10MB
- **<200ms** for files under 100MB

No performance concerns until 10,000+ entries.

## Troubleshooting

### File Permission Errors

If you get `EACCES` errors:
```bash
chmod 644 docs/memory/*.jsonl
```

### Corrupted JSON

If entries are malformed:
```bash
# Find and remove invalid lines
cat reflexion.jsonl | while read line; do
  echo "$line" | jq . >/dev/null 2>&1 && echo "$line"
done > fixed.jsonl
mv fixed.jsonl reflexion.jsonl
```

### Duplicate Entries

If you see duplicate learnings:
```bash
# Show duplicates
cat reflexion.jsonl | jq -r '.mistake' | sort | uniq -c | sort -rn

# Remove duplicates (keeps first occurrence)
cat reflexion.jsonl | jq -s 'unique_by(.mistake)' | jq -c '.[]' > deduplicated.jsonl
mv deduplicated.jsonl reflexion.jsonl
```

## Related Documentation

- **User Guide**: [docs/user-guide/memory-system.md](../user-guide/memory-system.md)
- **Implementation**: `superclaude/core/pm_init/reflexion_memory.py`
- **Research**: [docs/research/reflexion-integration-2025.md](../research/reflexion-integration-2025.md)
- **PM Agent**: [superclaude/agents/pm-agent.md](../../superclaude/agents/pm-agent.md)

## Quick Commands

```bash
# View all learnings
cat docs/memory/reflexion.jsonl | jq

# Count entries
wc -l docs/memory/reflexion.jsonl

# Search for specific topic
grep -i "auth" docs/memory/reflexion.jsonl | jq

# Latest 5 learnings
tail -5 docs/memory/reflexion.jsonl | jq

# Most common mistakes
cat docs/memory/reflexion.jsonl | jq -r '.mistake' | sort | uniq -c | sort -rn | head -10

# Export to readable format
cat docs/memory/reflexion.jsonl | jq > reflexion-readable.json
```

---

**Last Updated**: 2025-10-30
**Maintained by**: SuperClaude Framework Team