feat: Add Deep Research System v4.2.0 (#380)

feat: Add Deep Research System v4.2.0 - Autonomous web research capabilities

## Overview
Comprehensive implementation of Deep Research framework aligned with DR Agent architecture, enabling autonomous, adaptive, and intelligent web research capabilities.

## Key Features

### 🔬 Deep Research Agent
- 15th specialized agent for comprehensive research orchestration
- Adaptive planning strategies: Planning-Only, Intent-Planning, Unified Intent-Planning
- Multi-hop reasoning with genealogy tracking (up to 5 hops)
- Self-reflective mechanisms with confidence scoring (0.0-1.0)
- Case-based learning for cross-session intelligence

### 🎯 New /sc:research Command
- Intelligent web research with depth control (quick/standard/deep/exhaustive)
- Parallel-first execution for optimal performance
- Domain filtering and time-based search options
- Automatic report generation in claudedocs/

### 🔍 Tavily MCP Integration
- 7th MCP server for real-time web search
- News search with time filtering
- Content extraction from search results
- Multi-round searching with iterative refinement
- Free tier available with optional API key

### 🎨 MODE_DeepResearch
- 7th behavioral mode for systematic investigation
- 6-phase workflow: Understand → Plan → TodoWrite → Execute → Track → Validate
- Evidence-based reasoning with citation management
- Parallel operation defaults for efficiency

## Technical Changes

### Framework Updates
- Updated agent count: 14 → 15 agents
- Updated mode count: 6 → 7 modes
- Updated MCP server count: 6 → 7 servers
- Updated command count: 24 → 25 commands

### Configuration
- Added RESEARCH_CONFIG.md for research settings
- Added deep_research_workflows.md with examples
- Standardized file naming conventions (UPPERCASE for Core)
- Removed multi-source investigation features for simplification

### Integration Points
- Enhanced MCP component with remote server support
- Added check_research_prerequisites() in environment.py
- Created verify_research_integration.sh script
- Updated all documentation guides

## Requirements
- TAVILY_API_KEY environment variable (free tier available)
- Node.js and npm for Tavily MCP execution

## Documentation
- Complete user guide integration
- Workflow examples and best practices
- API configuration instructions
- Depth level explanations

🤖 Generated with Claude Code

Co-authored-by: moshe_anconina <moshe_a@ituran.com>
Co-authored-by: Claude <noreply@anthropic.com>

Docs/User-Guide/agents.md

@@ -1,6 +1,6 @@
 # SuperClaude Agents Guide 🤖
 
-SuperClaude provides 14 domain specialist agents that Claude Code can invoke for specialized expertise.
+SuperClaude provides 15 domain specialist agents that Claude Code can invoke for specialized expertise.
 
 
 ## 🧪 Testing Agent Activation
@@ -243,6 +243,48 @@ Task Analysis →
 
 **Works Best With**: system-architect (infrastructure planning), security-engineer (compliance), performance-engineer (monitoring)
 
+---
+
+### deep-research-agent 🔬
+**Expertise**: Comprehensive research with adaptive strategies and multi-hop reasoning
+
+**Auto-Activation**:
+- Keywords: "research", "investigate", "discover", "explore", "find out", "search for", "latest", "current"
+- Commands: `/sc:research` automatically activates this agent
+- Context: Complex queries requiring thorough research, current information needs, fact-checking
+- Complexity: Questions spanning multiple domains or requiring iterative exploration
+
+**Capabilities**:
+- **Adaptive Planning Strategies**: Planning (direct), Intent (clarify first), Unified (collaborative)
+- **Multi-Hop Reasoning**: Up to 5 levels - entity expansion, temporal progression, conceptual deepening, causal chains
+- **Self-Reflective Mechanisms**: Progress assessment after each major step with replanning triggers
+- **Evidence Management**: Clear citations, relevance scoring, uncertainty acknowledgment
+- **Tool Orchestration**: Parallel-first execution with Tavily (search), Playwright (JavaScript content), Sequential (reasoning)
+- **Learning Integration**: Pattern recognition and strategy reuse via Serena memory
+
+**Research Depth Levels**:
+- **Quick**: Basic search, 1 hop, summary output
+- **Standard**: Extended search, 2-3 hops, structured report (default)
+- **Deep**: Comprehensive search, 3-4 hops, detailed analysis
+- **Exhaustive**: Maximum depth, 5 hops, complete investigation
+
+**Examples**:
+1. **Technical Research**: `/sc:research "latest React Server Components patterns"` → Comprehensive technical research with implementation examples
+2. **Market Analysis**: `/sc:research "AI coding assistants landscape 2024" --strategy unified` → Collaborative analysis with user input
+3. **Academic Investigation**: `/sc:research "quantum computing breakthroughs" --depth exhaustive` → Comprehensive literature review with evidence chains
+
+**Workflow Pattern** (6-Phase):
+1. **Understand** (5-10%): Assess query complexity
+2. **Plan** (10-15%): Select strategy and identify parallel opportunities  
+3. **TodoWrite** (5%): Create adaptive task hierarchy (3-15 tasks)
+4. **Execute** (50-60%): Parallel searches and extractions
+5. **Track** (Continuous): Monitor progress and confidence
+6. **Validate** (10-15%): Verify evidence chains
+
+**Output**: Reports saved to `claudedocs/research_[topic]_[timestamp].md`
+
+**Works Best With**: system-architect (technical research), learning-guide (educational research), requirements-analyst (market research)
+
 ### Quality & Analysis Agents 🔍
 
 ### security-engineer 🔒
@@ -618,6 +660,7 @@ After applying agent fixes, test with:
 | **Documentation** | "documentation", "readme", "API docs" | technical-writer |
 | **Learning** | "explain", "tutorial", "beginner", "teaching" | learning-guide |
 | **Requirements** | "requirements", "PRD", "specification" | requirements-analyst |
+| **Research** | "research", "investigate", "latest", "current" | deep-research-agent |
 
 ### Command-Agent Mapping
 
@@ -631,6 +674,7 @@ After applying agent fixes, test with:
 | `/sc:design` | system-architect | Domain architects, requirements-analyst |
 | `/sc:test` | quality-engineer | security-engineer, performance-engineer |
 | `/sc:explain` | learning-guide | technical-writer, domain specialists |
+| `/sc:research` | deep-research-agent | Technical specialists, learning-guide |
 
 ### Effective Agent Combinations
 

Docs/User-Guide/commands.md

@@ -1,6 +1,6 @@
 # SuperClaude Commands Guide
 
-SuperClaude provides 24 commands for Claude Code: `/sc:*` commands for workflows and `@agent-*` for specialists.
+SuperClaude provides 25 commands for Claude Code: `/sc:*` commands for workflows and `@agent-*` for specialists.
 
 ## Command Types
 
@@ -106,7 +106,7 @@ python3 -m SuperClaude install --list-components | grep mcp
 
 - [Essential Commands](#essential-commands) - Start here (8 core commands)
 - [Common Workflows](#common-workflows) - Command combinations that work
-- [Full Command Reference](#full-command-reference) - All 23 commands organized by category
+- [Full Command Reference](#full-command-reference) - All 25 commands organized by category
 - [Troubleshooting](#troubleshooting) - Common issues and solutions
 - [Command Index](#command-index) - Find commands by category
 
@@ -133,6 +133,24 @@ python3 -m SuperClaude install --list-components | grep mcp
 - Discovering available commands: `/sc:help`
 - Getting a quick reminder of command names: `/sc:help`
 
+### `/sc:research` - Deep Research Command
+**Purpose**: Comprehensive web research with adaptive planning and intelligent search  
+**Syntax**: `/sc:research "[query]"` `[--depth quick|standard|deep|exhaustive] [--strategy planning|intent|unified]`
+
+**Use Cases**:
+- Technical research: `/sc:research "latest React 19 features" --depth deep`
+- Market analysis: `/sc:research "AI coding assistant landscape 2024" --strategy unified`
+- Academic investigation: `/sc:research "quantum computing breakthroughs" --depth exhaustive`
+- Current events: `/sc:research "latest AI developments 2024"`
+
+**Key Capabilities**:
+- **6-Phase Workflow**: Understand → Plan → TodoWrite → Execute → Track → Validate
+- **Adaptive Depth**: Quick (basic search), Standard (extended), Deep (comprehensive), Exhaustive (maximum depth)
+- **Planning Strategies**: Planning (direct), Intent (clarify first), Unified (collaborative)
+- **Parallel Execution**: Default parallel searches and extractions
+- **Evidence Management**: Clear citations with relevance scoring
+- **Output Standards**: Reports saved to `claudedocs/research_[topic]_[timestamp].md`
+
 ### `/sc:implement` - Feature Development  
 **Purpose**: Full-stack feature implementation with intelligent specialist routing  
 **Syntax**: `/sc:implement "feature description"` `[--type frontend|backend|fullstack] [--focus security|performance]`  
@@ -277,6 +295,7 @@ python3 -m SuperClaude install --list-components | grep mcp
 | Command | Purpose | Best For |
 |---------|---------|----------|
 | **analyze** | Code assessment | Quality audits, security reviews |
+| **research** | Web research with intelligent search | Technical research, current events, market analysis |
 | **business-panel** | Strategic analysis | Business decisions, competitive assessment |
 | **spec-panel** | Specification review | Requirements validation, architecture analysis |
 | **troubleshoot** | Problem diagnosis | Bug investigation, performance issues |

Docs/User-Guide/mcp-servers.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-MCP (Model Context Protocol) servers extend Claude Code's capabilities through specialized tools. SuperClaude integrates 6 MCP servers and provides Claude with instructions on when to activate them based on your tasks.
+MCP (Model Context Protocol) servers extend Claude Code's capabilities through specialized tools. SuperClaude integrates 7 MCP servers and provides Claude with instructions on when to activate them based on your tasks.
 
 ### 🔍 Reality Check
 - **What MCP servers are**: External Node.js processes that provide additional tools
@@ -17,6 +17,7 @@ MCP (Model Context Protocol) servers extend Claude Code's capabilities through s
 - **playwright**: Browser automation and E2E testing
 - **morphllm-fast-apply**: Pattern-based code transformations
 - **serena**: Semantic code understanding and project memory
+- **tavily**: Web search and real-time information retrieval
 
 ## Quick Start
 
@@ -32,6 +33,7 @@ MCP (Model Context Protocol) servers extend Claude Code's capabilities through s
 | `test`, `e2e`, `browser` | **playwright** |
 | Multi-file edits, refactoring | **morphllm-fast-apply** |
 | Large projects, sessions | **serena** |
+| `/sc:research`, `latest`, `current` | **tavily** |
 
 ## Server Details
 
@@ -119,6 +121,36 @@ export MORPH_API_KEY="your_key_here"
 /sc:refactor "extract UserService" --serena
 ```
 
+### tavily 🔍
+**Purpose**: Web search and real-time information retrieval for research
+**Triggers**: `/sc:research` commands, "latest" information requests, current events, fact-checking
+**Requirements**: Node.js 16+, TAVILY_API_KEY (free tier available at https://app.tavily.com)
+
+```bash
+# Automatic activation
+/sc:research "latest AI developments 2024"
+# → Performs intelligent web research
+
+# Manual activation  
+/sc:analyze "market trends" --tavily
+
+# API key setup (get free key at https://app.tavily.com)
+export TAVILY_API_KEY="tvly-your_api_key_here"
+```
+
+**Capabilities:**
+- **Web Search**: Comprehensive searches with ranking and filtering
+- **News Search**: Time-filtered current events and updates
+- **Content Extraction**: Full-text extraction from search results  
+- **Domain Filtering**: Include/exclude specific domains
+- **Multi-Hop Research**: Iterative searches based on findings (up to 5 hops)
+
+**Research Depth Control:**
+- `--depth quick`: 5-10 sources, basic synthesis
+- `--depth standard`: 10-20 sources, structured report (default)
+- `--depth deep`: 20-40 sources, comprehensive analysis
+- `--depth exhaustive`: 40+ sources, academic-level research
+
 ## Configuration
 
 **MCP Configuration File (`~/.claude.json`):**
@@ -150,6 +182,11 @@ export MORPH_API_KEY="your_key_here"
     "serena": {
       "command": "uvx",
       "args": ["--from", "git+https://github.com/oraios/serena", "serena", "start-mcp-server", "--context", "ide-assistant"]
+    },
+    "tavily": {
+      "command": "npx",
+      "args": ["-y", "@tavily/mcp@latest"],
+      "env": {"TAVILY_API_KEY": "${TAVILY_API_KEY}"}
     }
   }
 }
@@ -211,16 +248,21 @@ export TWENTYFIRST_API_KEY="your_key_here"
 # For Morphllm server (required for bulk transformations)
 export MORPH_API_KEY="your_key_here"
 
+# For Tavily server (required for web search - free tier available)
+export TAVILY_API_KEY="tvly-your_key_here"
+
 # Add to shell profile for persistence
 echo 'export TWENTYFIRST_API_KEY="your_key"' >> ~/.bashrc
 echo 'export MORPH_API_KEY="your_key"' >> ~/.bashrc
+echo 'export TAVILY_API_KEY="your_key"' >> ~/.bashrc
 ```
 
 **Environment Variable Usage:**
 - ✅ `TWENTYFIRST_API_KEY` - Required for Magic MCP server functionality
 - ✅ `MORPH_API_KEY` - Required for Morphllm MCP server functionality  
+- ✅ `TAVILY_API_KEY` - Required for Tavily MCP server functionality (free tier available)
 - ❌ Other env vars in docs - Examples only, not used by framework
-- 📝 Both are paid service API keys, framework works without them
+- 📝 Magic and Morphllm are paid services, Tavily has free tier, framework works without them
 
 ## Server Combinations
 
@@ -238,6 +280,8 @@ echo 'export MORPH_API_KEY="your_key"' >> ~/.bashrc
 - **Web Development**: magic + context7 + playwright  
 - **Enterprise Refactoring**: serena + morphllm + sequential-thinking
 - **Complex Analysis**: sequential-thinking + context7 + serena
+- **Deep Research**: tavily + sequential-thinking + serena + playwright
+- **Current Events**: tavily + context7 + sequential-thinking
 
 ## Integration
 
@@ -245,11 +289,13 @@ echo 'export MORPH_API_KEY="your_key"' >> ~/.bashrc
 - Analysis commands automatically use Sequential + Serena
 - Implementation commands use Magic + Context7
 - Testing commands use Playwright + Sequential
+- Research commands use Tavily + Sequential + Playwright
 
 **With Behavioral Modes:**
 - Brainstorming Mode: Sequential for discovery
 - Task Management: Serena for persistence
 - Orchestration Mode: Optimal server selection
+- Deep Research Mode: Tavily + Sequential + Playwright coordination
 
 **Performance Control:**
 - Automatic resource management based on system load

Docs/User-Guide/modes.md

@@ -9,6 +9,7 @@ Test modes by using `/sc:` commands - they activate automatically based on task
 |------|---------|---------------|---------------|---------------|
 | **🧠 Brainstorming** | Interactive discovery | "brainstorm", "maybe", vague requests | Socratic questions, requirement elicitation | New project planning, unclear requirements |
 | **🔍 Introspection** | Meta-cognitive analysis | Error recovery, "analyze reasoning" | Transparent thinking markers (🤔, 🎯, 💡) | Debugging, learning, optimization |
+| **🔬 Deep Research** | Systematic investigation mindset | `/sc:research`, investigation keywords | 6-phase workflow, evidence-based reasoning | Technical research, current events, market analysis |
 | **📋 Task Management** | Complex coordination | >3 steps, >2 directories | Phase breakdown, memory persistence | Multi-step operations, project management |
 | **🎯 Orchestration** | Intelligent tool selection | Multi-tool ops, high resource usage | Optimal tool routing, parallel execution | Complex analysis, performance optimization |
 | **⚡ Token Efficiency** | Compressed communication | High context usage, `--uc` flag | Symbol systems, estimated 30-50% token reduction | Resource constraints, large operations |
@@ -120,6 +121,60 @@ Introspective Approach:
 
 ---
 
+### 🔬 Deep Research Mode - Systematic Investigation Mindset
+
+**Purpose**: Research mindset for systematic investigation and evidence-based reasoning.
+
+**Auto-Activation Triggers:**
+- `/sc:research` command invocation
+- Research-related keywords: investigate, explore, discover, analyze
+- Questions requiring current information beyond knowledge cutoff
+- Complex research requirements
+- Manual flag: `--research`
+
+**Behavioral Modifications:**
+- **Thinking Style**: Systematic over casual, evidence over assumption, progressive depth exploration
+- **Communication**: Lead with confidence levels, provide inline citations, acknowledge uncertainties
+- **Priority Shifts**: Completeness over speed, accuracy over speculation, verification over assumption
+- **Process Adaptations**: Always create investigation plans, default to parallel operations, maintain evidence chains
+
+**6-Phase Research Workflow:**
+- 📋 **Understand** (5-10%): Assess query complexity and requirements
+- 📝 **Plan** (10-15%): Select strategy (planning/intent/unified) and identify parallelization
+- ✅ **TodoWrite** (5%): Create adaptive task hierarchy (3-15 tasks based on complexity)
+- 🔄 **Execute** (50-60%): Parallel-first searches and smart extraction routing
+- 📊 **Track** (Continuous): Monitor progress and update confidence scores
+- ✓ **Validate** (10-15%): Verify evidence chains and ensure completeness
+
+**Example Experience:**
+```
+Standard Mode: "Here are some search results about quantum computing..."
+Deep Research Mode: 
+"📊 Research Plan: Quantum computing breakthroughs
+ ✓ TodoWrite: Created 8 research tasks
+ 🔄 Executing parallel searches across domains
+ 📈 Confidence: 0.82 across 15 verified sources
+ 📝 Report saved: claudedocs/research_quantum_[timestamp].md"
+```
+
+#### Quality Standards
+- [ ] Minimum 2 sources per claim with inline citations
+- [ ] Confidence scoring (0.0-1.0) for all findings
+- [ ] Parallel execution by default for independent operations
+- [ ] Reports saved to claudedocs/ with proper structure
+- [ ] Clear methodology and evidence presentation
+
+**Verify:** `/sc:research "test topic"` should create TodoWrite and execute systematically  
+**Test:** All research should include confidence scores and citations  
+**Check:** Reports should be saved to claudedocs/ automatically
+
+**Works Best With:**
+- **→ Task Management**: Research planning with TodoWrite integration
+- **→ Orchestration**: Parallel Tavily/Playwright coordination
+- **Manual Override**: Use `--depth` and `--strategy` for fine control
+
+---
+
 ### 📋 Task Management Mode - Complex Coordination
 
 **Purpose**: Hierarchical task organization with session persistence for multi-step operations.