AI Agent Factory with Claude Code Subagents

use-cases/agent-factory-with-subagents/agents/rag_agent/planning/INITIAL.md

@@ -0,0 +1,147 @@
+# Agent Requirements: Semantic Search Agent
+
+## Executive Summary
+A simple yet powerful semantic search agent that leverages PGVector to provide intelligent document retrieval and summarized insights. The agent automatically chooses between semantic and hybrid search while maintaining a clean CLI interface for user interactions.
+
+## Agent Classification
+- **Type**: Tool-Enabled Agent with structured output capabilities
+- **Complexity**: Medium
+- **Priority Features**: 
+  1. Semantic search with embeddings
+  2. Intelligent search type selection
+  3. Search result summarization
+
+## Functional Requirements
+
+### Core Functionality
+1. **Semantic Search Operation**
+   - Execute semantic similarity search using PGVector embeddings
+   - Automatically generate query embeddings using OpenAI text-embedding-3-small (1536 dimensions)
+   - Return top-k relevant document chunks with similarity scores
+   - **Acceptance Criteria**: Successfully retrieve and rank documents by semantic similarity
+
+2. **Hybrid Search with Auto-Selection**
+   - Automatically determine when to use semantic vs hybrid search based on query characteristics
+   - Allow manual override when user explicitly specifies search type
+   - Combine vector similarity with full-text search for enhanced results
+   - **Acceptance Criteria**: Intelligently route queries to optimal search method
+
+3. **Search Result Summarization**
+   - Analyze retrieved chunks and generate concise insights
+   - Synthesize information from multiple sources into coherent summaries
+   - Maintain source attribution for transparency
+   - **Acceptance Criteria**: Provide meaningful summaries with proper source references
+
+### Input/Output Specifications
+- **Input Types**: 
+  - Natural language queries via CLI
+  - Optional search type specification ("semantic", "hybrid", or auto-detect)
+  - Optional result limit (default: 10)
+- **Output Format**: String responses with structured summaries and source citations
+- **Validation Requirements**: Query length validation, result limit bounds (1-50)
+
+## Technical Requirements
+
+### Model Configuration
+- **Primary Model**: openai:gpt-4o-mini (cost-effective for summarization tasks)
+- **Embedding Model**: text-embedding-3-small (1536 dimensions, matches database schema)
+- **Context Window Needs**: ~8K tokens for processing multiple search results
+
+### External Integrations
+1. **PostgreSQL with PGVector**:
+   - Purpose: Semantic similarity search and hybrid search operations
+   - Authentication: DATABASE_URL environment variable
+   - Functions used: `match_chunks()` and `hybrid_search()`
+   - Connection: asyncpg with connection pooling
+
+2. **OpenAI Embeddings API**:
+   - Purpose: Generate query embeddings for semantic search
+   - Authentication: OPENAI_API_KEY environment variable
+   - Model: text-embedding-3-small
+
+### Tool Requirements
+1. **semantic_search**:
+   - Purpose: Execute pure semantic similarity search using embeddings
+   - Parameters: query (str), limit (int, default=10)
+   - Error Handling: Database connection retry, empty result handling
+
+2. **hybrid_search**:
+   - Purpose: Execute combined semantic + keyword search
+   - Parameters: query (str), limit (int, default=10), text_weight (float, default=0.3)
+   - Error Handling: Fallback to semantic search if text search fails
+
+3. **auto_search**:
+   - Purpose: Automatically select search type based on query analysis
+   - Parameters: query (str), limit (int, default=10)
+   - Error Handling: Default to semantic search on classification failure
+
+## Dependencies and Environment
+
+### API Keys and Credentials
+- DATABASE_URL: PostgreSQL connection string with PGVector extension
+- OPENAI_API_KEY: OpenAI API key for embeddings and LLM
+- LLM_MODEL: Model name (default: gpt-4o-mini)
+
+### Python Packages
+- pydantic-ai (core framework)
+- asyncpg (PostgreSQL async driver)
+- python-dotenv (environment variable management)
+- rich (CLI formatting)
+- openai (embeddings API)
+- numpy (embedding vector operations)
+
+### System Requirements
+- Python version: 3.11+
+- PostgreSQL with PGVector extension
+- Memory requirements: ~256MB for embeddings cache
+- Network requirements: Internet access for OpenAI API
+
+## Success Criteria
+1. **Search Accuracy**: Retrieve semantically relevant results with >0.7 similarity threshold
+2. **Response Time**: Complete search and summarization within 3-5 seconds
+3. **Auto-Selection Accuracy**: Correctly choose search type in >80% of cases
+4. **Summary Quality**: Generate coherent summaries that capture key insights from search results
+
+## Security and Compliance
+- **Data Privacy**: Database queries and results handled securely, no data logging
+- **API Key Management**: Environment variables only, no hardcoded credentials
+- **Input Sanitization**: Query length limits, SQL injection prevention via parameterized queries
+- **Audit Logging**: Search queries and result counts logged for performance monitoring
+
+## Testing Requirements
+- **Unit Tests**: Individual tool functions, search type classification logic
+- **Integration Tests**: End-to-end database connectivity and search operations
+- **Performance Tests**: Search response times under different query types and database sizes
+- **Security Tests**: Input validation, SQL injection prevention, API key security
+
+## Constraints and Limitations
+- **Database Schema**: Must work with existing documents/chunks tables and PGVector functions
+- **Embedding Dimensions**: Fixed at 1536 to match existing database schema
+- **Search Result Limit**: Maximum 50 results to prevent performance issues
+- **Query Length**: Maximum 1000 characters to prevent embedding API limits
+
+## Future Enhancements (Optional)
+- Search result caching for frequently asked questions
+- Advanced query preprocessing (entity extraction, query expansion)
+- Multi-language search support
+- Search analytics and result ranking improvements
+- Integration with document ingestion pipeline
+
+## Assumptions Made
+1. **Database Setup**: PGVector extension is properly installed and configured
+2. **Existing Data**: Documents and chunks tables are populated with embedded content
+3. **Search Patterns**: Users will primarily perform knowledge-based queries
+4. **Performance**: Database has appropriate indexes for efficient vector operations
+5. **API Access**: Stable internet connection for OpenAI API calls
+6. **CLI Usage**: Primary interface will be command-line with rich formatting
+
+## Approval Checklist
+- [x] All core requirements defined (semantic search, auto-selection, summarization)
+- [x] External dependencies identified (PostgreSQL/PGVector, OpenAI)
+- [x] Security considerations addressed (env vars, input validation)
+- [x] Testing strategy outlined (unit, integration, performance)
+- [x] Success criteria measurable (accuracy, response time, auto-selection)
+
+---
+Generated: 2025-08-22
+Status: Ready for Component Development