updated the docs

README.md

@@ -1,27 +1,62 @@
 # SecureLens AI — Backend
 
-SecureLens AI Backend is the core security engine that powers the SecureLens AI agent. It performs live security analysis of applications and generates structured risk insights, issue explanations, and remediation guidance.
+SecureLens is an AI security agent that connects directly to your GitHub repositories and autonomously hunts for security vulnerabilities in your source code. It does not scan randomly — it reasons about your codebase the same way a security engineer would: reads the file structure, identifies which files carry the most risk, and focuses its analysis where it actually matters.
+
+The agent integrates with GitHub via a **Personal Access Token (PAT)**, giving it read access to any repository you point it at — public or private. The intelligence layer is powered by the **Google Gemini API** (`gemini-2.0-flash`), which is used for three separate reasoning tasks: prioritising which files to investigate, performing deep security analysis of each file's source code against the OWASP Top 10, and maintaining a contextual chat session so you can ask follow-up questions and request patches for specific issues.
+
+Beyond code scanning, SecureLens also performs infrastructure-level security analysis on live URLs — checking headers, SSL configuration, cookie security, and sensitive path exposure — scoring each target out of 100 and generating an AI-written threat narrative explaining how an attacker could chain the findings together.
 
 ---
 
-## What This Backend Does
+## Documentation
 
-The SecureLens backend acts as the **security brain** of the system.
+Detailed technical documentation lives in the [`docs/`](./docs/) folder:
 
-It:
+| Doc | Description |
+|---|---|
+| [docs/ai-agent.md](./docs/ai-agent.md) | How the AI agent pipeline works — triage, concurrent analysis, chat context |
+| [docs/architecture.md](./docs/architecture.md) | Full system architecture and request flow diagrams |
+| [docs/api-reference.md](./docs/api-reference.md) | Every endpoint, request/response shape, and error codes |
 
-- Scans live applications via URL
-- Checks for security misconfigurations
-- Detects common vulnerabilities (headers, exposure, HTTPS issues, etc.)
-- Assigns a **security score**
-- Categorizes risks into layers
-- Generates **human-readable fix suggestions**
-- Validates URLs and prevents SSRF attacks
-- Rate limits API requests
-- **User authentication** with JWT tokens
-- **Scan history** — saves and retrieves past scan results
+---
 
-This backend is designed to evolve into an **AI-driven autonomous remediation agent**.
+## How It Works
+
+SecureLens runs a three-phase agentic pipeline when you trigger a code scan:
+
+**Phase 1 — Triage**
+The agent fetches the complete file tree from your repository and sends it to Gemini. The model acts as a senior AppSec engineer — it reads the file paths, understands the structure, and selects the files most likely to contain vulnerabilities (auth routes, database queries, API handlers, config files). This is one targeted API call, not a blind scan of everything.
+
+**Phase 2 — Concurrent Analysis**
+The agent fetches the source code for each prioritised file from GitHub and sends them all to Gemini simultaneously for deep security analysis. Each file is checked against the OWASP Top 10 — SQL injection, XSS, hardcoded secrets, IDOR, broken access control, misconfigurations, and more. Results come back with severity ratings, affected line numbers, explanations, and specific code fixes.
+
+**Phase 3 — Context-Aware Chat**
+After the scan, you can open a live conversation with the agent about what it found. The agent has full context of the scan results — it knows which files were checked, what vulnerabilities were found, and at which lines. Ask it to write a patch, explain an issue in plain English, or prioritise what to fix first.
+
+---
+
+## What This Backend Covers
+
+**GitHub Code Scanner**
+- Connects to any GitHub repo via Personal Access Token
+- AI-driven file triage — the agent decides what to read, not a keyword filter
+- Concurrent SAST analysis across prioritised files
+- OWASP Top 10 coverage: SQLi, XSS, IDOR, hardcoded secrets, broken auth, misconfigs
+- Severity-rated findings with line numbers and suggested fixes
+- Persistent scan context for multi-turn AI chat
+
+**Website URL Scanner**
+- 30+ checks across 5 security layers (transport, SSL/TLS, headers, cookies, exposure)
+- Security score (0–100) and letter grade (A–F)
+- AI-enhanced explanations and remediation snippets per finding
+- AI-generated Threat Narrative — attack chain reasoning across multiple findings
+- Scan history saved per user account
+
+**Platform**
+- JWT-based authentication (register, login, token-secured endpoints)
+- Rate limiting and SSRF protection on all scan inputs
+- Full scan history (list, retrieve, delete)
+- Interactive API docs at `/docs`
 
 ---
 
@@ -43,13 +78,14 @@ SecureLens structures vulnerabilities into 5 logical layers with **30+ security
 
 - **Python 3.12+**
 - **FastAPI** — async web framework
-- **httpx** — async HTTP client
-- **SQLAlchemy 2.0** — async ORM (SQLite dev / PostgreSQL production)
-- **Pydantic v2** — data validation & settings
+- **httpx** — async HTTP client (GitHub API + live URL scanning)
+- **SQLAlchemy 2.0** — async ORM
+- **Pydantic v2** — data validation and settings management
+- **google-genai** — Google Gemini AI SDK (code analysis, chat, AI enhancements)
 - **python-jose** — JWT authentication
 - **passlib + bcrypt** — password hashing
 - **SlowAPI** — rate limiting
-- **Docker** — containerized deployment
+- **Docker + PostgreSQL** — containerized deployment
 - **pytest** — testing
 
 ---
@@ -58,6 +94,11 @@ SecureLens structures vulnerabilities into 5 logical layers with **30+ security
 
 ```
 securelens-backend/
+├── docs/
+│   ├── README.md               # Docs index
+│   ├── ai-agent.md             # How the AI agent pipeline works
+│   ├── architecture.md         # Full system architecture
+│   └── api-reference.md        # All endpoints documented
 ├── app/
 │   ├── main.py                 # FastAPI app + middleware + lifespan
 │   ├── config.py               # Pydantic settings (.env)
@@ -67,21 +108,26 @@ securelens-backend/
 │   │   └── scan.py             # ScanResult ORM model
 │   ├── schemas/
 │   │   ├── auth.py             # Auth request/response models
-│   │   └── scan.py             # Scan request/response models
+│   │   ├── scan.py             # Website scan models
+│   │   └── code_scan.py        # Code scan + chat models
 │   ├── routers/
 │   │   ├── auth.py             # Register, login, me
 │   │   ├── health.py           # Health check endpoints
-│   │   ├── scan.py             # Scan endpoint
-│   │   └── history.py          # Scan history endpoints
+│   │   ├── scan.py             # Website scan endpoint
+│   │   ├── history.py          # Scan history endpoints
+│   │   └── code_scan.py        # GitHub repo scan + AI chat
 │   ├── services/
-│   │   ├── scoring.py          # Scoring engine
-│   │   └── scanner/
-│   │       ├── base.py         # Abstract scanner interface
-│   │       ├── transport.py    # Transport & HSTS checks
-│   │       ├── ssl_checker.py  # SSL/TLS certificate analysis
-│   │       ├── headers.py      # Header security checks
-│   │       ├── cookies.py      # Cookie security checks
-│   │       └── exposure.py     # Sensitive path detection
+│   │   ├── ai.py               # Website scanner AI layer (Gemini)
+│   │   ├── scoring.py          # Security scoring engine
+│   │   ├── scanner/            # Website scanner (5 check layers)
+│   │   │   ├── transport.py
+│   │   │   ├── ssl_checker.py
+│   │   │   ├── headers.py
+│   │   │   ├── cookies.py
+│   │   │   └── exposure.py
+│   │   └── code_scanner/       # GitHub repo AI agent
+│   │       ├── orchestrator.py # 3-phase AI pipeline
+│   │       └── github_client.py# GitHub API client
 │   ├── middleware/
 │   │   ├── auth.py             # JWT auth dependencies
 │   │   └── rate_limiter.py     # SlowAPI rate limiting
@@ -89,18 +135,13 @@ securelens-backend/
 │       ├── auth.py             # JWT & password utilities
 │       └── validators.py       # URL validation & SSRF prevention
 ├── tests/
-│   ├── conftest.py             # Test fixtures (in-memory DB)
+│   ├── conftest.py
 │   ├── test_auth.py
-│   ├── test_health.py
 │   ├── test_scan.py
 │   ├── test_history.py
-│   ├── test_validators.py
-│   ├── test_transport.py
-│   ├── test_headers.py
-│   ├── test_ssl_checker.py
-│   ├── test_cookies.py
-│   └── test_scoring.py
-├── main.py                     # Root entry point (backward compat)
+│   └── ...
+├── migrations/                 # Alembic database migrations
+├── main.py                     # Root entry point
 ├── .env.example
 ├── Dockerfile
 ├── docker-compose.yml
@@ -137,7 +178,7 @@ docker compose up --build
 
 ## Configuration
 
-Copy `.env.example` to `.env` and customize:
+Copy `.env.example` to `.env` and set your values:
 
 | Variable              | Default                                    | Description                       |
 | --------------------- | ------------------------------------------ | --------------------------------- |
@@ -154,6 +195,9 @@ Copy `.env.example` to `.env` and customize:
 | `JWT_SECRET`          | (change in production!)                    | Secret key for JWT signing        |
 | `JWT_ALGORITHM`       | HS256                                      | JWT signing algorithm             |
 | `JWT_EXPIRY_MINUTES`  | 1440                                       | Token expiry (default: 24h)       |
+| `GEMINI_API_KEY`      | (required for AI features)                 | Google Gemini API key             |
+
+Get a free Gemini API key at [aistudio.google.com/apikey](https://aistudio.google.com/apikey). Without it, the app still works but all AI-powered features (code scanning, AI chat, AI enhancements) are disabled.
 
 ---
 
@@ -174,7 +218,15 @@ Copy `.env.example` to `.env` and customize:
 | POST   | `/auth/login`     | Login + get token         | No   |
 | GET    | `/auth/me`        | Get current user info     | Yes  |
 
-### Scanning
+### Code Scanner (AI Agent)
+
+| Method | Endpoint                | Description                                      | Auth     |
+| ------ | ----------------------- | ------------------------------------------------ | -------- |
+| POST   | `/code-scan/analyze`    | Run AI agent against a GitHub repo               | No       |
+| POST   | `/code-scan/chat`       | Chat with AI about a specific scan's results     | No       |
+| GET    | `/code-scan/models`     | List available Gemini models for your API key    | No       |
+
+### Website Scanner
 
 | Method | Endpoint         | Description                          | Auth     |
 | ------ | ---------------- | ------------------------------------ | -------- |
@@ -190,6 +242,28 @@ Copy `.env.example` to `.env` and customize:
 
 ### Example Usage
 
+**Scan a GitHub Repo (AI Agent):**
+```bash
+curl -X POST http://127.0.0.1:8000/code-scan/analyze \
+  -H "Content-Type: application/json" \
+  -d '{"repo_url": "https://github.com/user/repo", "github_token": "ghp_xxx", "branch": "main"}'
+```
+
+**Chat with the scan:**
+```bash
+curl -X POST http://127.0.0.1:8000/code-scan/chat \
+  -H "Content-Type: application/json" \
+  -d '{"scan_id": "<scan_id from above>", "message": "Write a patch for the highest severity issue"}'
+```
+
+**Scan a URL:**
+```bash
+curl -X POST http://127.0.0.1:8000/scan \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer YOUR_TOKEN" \
+  -d '{"url": "https://example.com"}'
+```
+
 **Register:**
 ```bash
 curl -X POST http://127.0.0.1:8000/auth/register \
@@ -197,20 +271,6 @@ curl -X POST http://127.0.0.1:8000/auth/register \
   -d '{"email": "user@example.com", "username": "myuser", "password": "securepass123"}'
 ```
 
-**Scan (authenticated):**
-```bash
-curl -X POST http://127.0.0.1:8000/scan \
-  -H "Content-Type: application/json" \
-  -H "Authorization: Bearer YOUR_TOKEN" \
-  -d '{"url": "https://google.com"}'
-```
-
-**View scan history:**
-```bash
-curl http://127.0.0.1:8000/scans \
-  -H "Authorization: Bearer YOUR_TOKEN"
-```
-
 ---
 
 ## Running Tests
@@ -225,12 +285,13 @@ Tests use an in-memory SQLite database — no external DB needed.
 
 ## Future Roadmap
 
-- AI agent for auto-remediation
-- PDF report generation
-- CI/CD pipeline integration
+- **Multi-LLM support** — swap out the AI provider via config. Planned support for OpenAI, Anthropic Claude, Mistral, and self-hosted models (Ollama), so you are not locked into any single API key or vendor
+- **CI/CD integration** — run SecureLens as part of your deployment pipeline (GitHub Actions, GitLab CI, etc.) to automatically scan every pull request or deployment for new vulnerabilities before they hit production
+- Persistent chat history (store code scan results in PostgreSQL)
+- PDF report generation for code scan results
 - DNS security checks (SPF, DKIM, DMARC)
 - Technology fingerprinting
-- JavaScript library CVE detection
+- JavaScript library CVE detection via package.json analysis
 
 ---
 

app/routers/code_scan.py

@@ -76,11 +76,6 @@ async def chat_with_scan(request: CodeChatRequest):
     if not settings.gemini_api_key:
         raise HTTPException(status_code=400, detail="AI Chat is disabled because GEMINI_API_KEY is not configured.")
         
-    from google import genai
-    with open("/app/models.txt", "w") as f:
-        # Just writing a placeholder, list_models is different in new SDK
-        f.write("AVAILABLE MODELS: migrated to new SDK")
-        
     scan_data = scan_store.get(request.scan_id)
     if not scan_data:
         raise HTTPException(status_code=404, detail="Scan ID not found or expired.")

docs/README.md

@@ -0,0 +1,19 @@
+# docs/
+
+This folder contains the technical documentation for the SecureLens backend.
+
+## What's Here
+
+| File | What It Explains |
+|---|---|
+| [ai-agent.md](./ai-agent.md) | How the AI agent works — the three-phase code scanning pipeline, the Gemini integration, the chat system, and the website scanner AI layer |
+| [architecture.md](./architecture.md) | How the full system is structured — all layers from the FastAPI app down to the database, and a step-by-step trace of a real request |
+| [api-reference.md](./api-reference.md) | Every API endpoint, its request/response shape, and error codes |
+
+## Where to Start
+
+If you want to understand **how the AI works** → [ai-agent.md](./ai-agent.md)
+
+If you want to understand **how the system is structured** → [architecture.md](./architecture.md)
+
+If you're looking for a **specific endpoint** → [api-reference.md](./api-reference.md)

docs/ai-agent.md

@@ -0,0 +1,315 @@
+# The SecureLens AI Agent — How It Actually Works
+
+This document explains the AI agent at the heart of SecureLens — what it does, how it thinks, and how all the pieces connect together.
+
+---
+
+## The Big Picture
+
+SecureLens has two separate AI systems, and it's important to understand both:
+
+1. **The Code Scanner Agent** — scans a GitHub repository for security vulnerabilities in your source code (OWASP Top 10, hardcoded secrets, broken access control, etc.)
+
+2. **The Website Scanner AI** — scans a live URL for infrastructure-level security issues (missing headers, SSL misconfigs, exposed paths, etc.)
+
+Both use Google Gemini as their AI engine. This document focuses primarily on the Code Scanner Agent because that's the more complex, agentic system. The website scanner AI is covered at the end.
+
+---
+
+## Part 1 — The Code Scanner Agent
+
+### What Is It?
+
+The Code Scanner Agent is a multi-step AI pipeline that mimics what a real penetration tester does when they look at a codebase.
+
+A real AppSec engineer doesn't read every single file in your project. They look at the file list, immediately know which ones are interesting (auth routes, database models, API handlers), skip the boring stuff (tests, lock files, generated assets), then read those specific files carefully and report what they find.
+
+That's exactly what this agent does. It runs in three phases:
+
+```
+GitHub Repo
+     │
+     ▼
+┌─────────────────────────┐
+│  Phase 1: TRIAGE        │  ← AI decides which files are worth reading
+│  (1 API call)           │
+└─────────────────────────┘
+     │
+     ▼
+┌─────────────────────────┐
+│  Phase 2: ANALYSIS      │  ← AI reads each file, finds vulnerabilities
+│  (up to 5 API calls,    │
+│   running in parallel)  │
+└─────────────────────────┘
+     │
+     ▼
+┌─────────────────────────┐
+│  Phase 3: SUMMARY       │  ← AI writes an executive summary
+│  (1 API call)           │
+└─────────────────────────┘
+     │
+     ▼
+┌─────────────────────────┐
+│  Chat Interface         │  ← User asks follow-up questions
+│  (1 API call per msg)   │
+└─────────────────────────┘
+```
+
+---
+
+### Phase 1 — Triage
+
+**File:** `app/services/code_scanner/orchestrator.py` → `triage_files()`
+
+The agent starts by fetching the complete file tree of the repository from the GitHub API. This gives us a flat list of every file in the repo — all paths, no content yet.
+
+Then it sends that entire list to Gemini with a specific prompt that says: *"You are a Senior Application Security Engineer. From these files, pick the 5 most likely to contain security vulnerabilities. Return them as a JSON array."*
+
+The AI looks at file paths like:
+- `app/api/users/route.ts` — interesting, API route that probably handles user data
+- `middleware.ts` — interesting, probably handles auth
+- `lib/db.js` — interesting, probably has database queries
+- `public/images/logo.png` — not interesting, just a static asset
+- `package-lock.json` — not interesting, auto-generated
+
+It returns a JSON object like:
+
+```json
+{
+  "critical_files": [
+    "app/api/users/route.ts",
+    "middleware.ts",
+    "lib/db.js",
+    "app/users/page.js",
+    "app/layout.tsx"
+  ]
+}
+```
+
+If the AI call fails for any reason (quota, network, etc.), the code falls back to just taking the first 5 files alphabetically. It never crashes — it degrades gracefully.
+
+**Why this matters:** Without triage, you'd have to send every file to the AI for analysis. For a repo with 50 files, that's 50 API calls, 50x the cost, and 50x the chance of hitting rate limits. The triage step collapses that into 1 call, and the AI's reasoning is genuinely smarter than any file extension filter you could write by hand.
+
+---
+
+### Phase 2 — Concurrent File Analysis
+
+**File:** `app/services/code_scanner/orchestrator.py` → `analyze_files()`
+
+Once we have the list of 5 critical files, we fetch the actual source code for each one from GitHub, then send each file individually to Gemini for a deep security review.
+
+The key technical detail here is **concurrency**. We don't do this one file at a time. All 5 files are analyzed at the same time, in parallel:
+
+```python
+results = await asyncio.gather(*(process_file(f) for f in triaged_files))
+```
+
+`asyncio.gather` fires off all 5 analysis tasks simultaneously and waits for all of them to finish. This means a 5-file scan takes roughly the same time as a 1-file scan — the bottleneck is the slowest individual file, not the sum of all of them.
+
+To prevent flooding the API (which would cause 429 rate-limit errors), we wrap the actual API call in a `Semaphore`:
+
+```python
+semaphore = asyncio.Semaphore(5)
+
+async with semaphore:
+    response = await ai_client.aio.models.generate_content(...)
+```
+
+The Semaphore acts like a door with a maximum occupancy. Only 5 tasks can be inside at once. If more arrive, they wait at the door. This gives us parallel execution without overwhelming the API.
+
+**Before fetching file content, we filter out junk:**
+
+```python
+if file_path.endswith('package-lock.json') or file_path.endswith('yarn.lock'):
+    return []
+```
+
+Lock files can be 50,000+ lines long. Sending them to an AI is pointless — they're autogenerated by npm/yarn and contain no code that a developer wrote. We skip them immediately.
+
+We also cap file content at 30,000 characters. If a file is longer than that, we truncate it. This prevents token limit errors on very large files.
+
+**The prompt for each file looks like this:**
+
+```
+Review the following code from the file 'app/users/page.js' for security vulnerabilities.
+Focus on OWASP Top 10: SQLi, XSS, Hardcoded Secrets, IDOR, Misconfigurations, etc.
+
+CODE:
+[file content here]
+
+Return a JSON object with a key 'vulnerabilities' containing a list of objects.
+Each object MUST have: severity, issue, explanation, suggested_fix, line_number.
+```
+
+The response comes back as structured JSON (we explicitly request `application/json` as the response MIME type so we don't have to strip markdown code fences), and we parse it directly into `VulnerabilityIssue` Pydantic models.
+
+---
+
+### Phase 3 — Executive Summary
+
+**File:** `app/services/code_scanner/orchestrator.py` → `generate_summary()`
+
+After all the vulnerabilities are collected, we send them all to Gemini one more time with a different prompt. This time we're asking it to act like a *Senior AppSec Manager* and write a 2-3 paragraph executive summary of the repository's overall security posture.
+
+This is the text that goes in the `summary` field of the API response. It gives a human-readable overview before the detailed issue list.
+
+If there are no vulnerabilities, this phase is skipped entirely and we return a hardcoded message.
+
+---
+
+### The Chat Interface
+
+**File:** `app/routers/code_scan.py` → `chat_with_scan()`
+
+After a scan completes, the results are saved to an in-memory dictionary (`scan_store`) keyed by the `scan_id`. This is a simple Python dict that lives in memory for as long as the server is running.
+
+```python
+scan_store: Dict[str, CodeScanResponse] = {}
+```
+
+When a user sends a chat message, they pass their `scan_id` along with it. The server looks up the full scan result, bundles it into a prompt, and sends it to Gemini:
+
+```
+You are SecureLens AI, an expert application security assistant.
+
+Here is the context of the scan for the repository [repo_url]:
+Summary: [executive summary]
+Vulnerabilities: [full JSON list of vulnerabilities]
+
+User Message: "Can you write a patch for the highest severity issue?"
+
+Answer clearly and professionally. Provide code fixes if requested.
+```
+
+This means the AI is not having a generic conversation — it has the full context of what it actually found in the specific repository being discussed. It knows the file names, the exact vulnerability descriptions, and the suggested fixes. So when you ask "write me a patch", it writes a patch for the specific issue in the specific file it analysed, not generic example code.
+
+**Important limitation:** The `scan_store` is in-memory only. If the server restarts, all scan contexts are lost. This is fine for development and demos. In a production system, you'd store this in the PostgreSQL database so chat sessions persist across deployments.
+
+---
+
+## Part 2 — The Website Scanner AI
+
+**File:** `app/services/ai.py`
+
+This is a separate, simpler AI layer that operates on the results of the infrastructure-level website scanner (the one that checks headers, SSL, cookies, etc.).
+
+It has three functions:
+
+### `enhance_security_issues()`
+
+Takes the raw list of issues detected by the rule-based scanner and asks Gemini to enrich each one. The AI adds:
+- A contextual severity rating (sometimes a "Medium" issue in a specific context is actually Critical)
+- A plain-English, non-technical explanation a non-developer can understand
+- A remediation code snippet (e.g. the exact Nginx config line to add a missing security header)
+
+### `chat_with_scan_context()`
+
+Same concept as the code scan chat, but for website scan results. The user can ask questions about their scan and get AI-powered answers grounded in the specific context of what was found.
+
+### `generate_threat_narrative()`
+
+This is the most creative AI function. Instead of listing issues individually, it asks Gemini to act like a red-teamer and write a "threat narrative" — a story of how an attacker might chain multiple vulnerabilities together to compromise the system.
+
+For example, if a site has a missing `X-Frame-Options` header and also exposes its admin path, the AI might write: *"An attacker could combine the missing clickjacking protection with the exposed admin panel to lure a logged-in administrator into clicking a malicious link, granting the attacker admin access without ever needing credentials."*
+
+This is significantly more actionable than a bullet list of independent issues.
+
+---
+
+## Prompt Engineering — Design Decisions
+
+This section explains why each prompt was designed the way it is. Prompts are the most critical part of an AI agent — small changes in wording produce very different outputs.
+
+---
+
+### Triage Prompt
+
+**Temperature: `0.1`** — Very low. We want the AI to make a deterministic, logical decision about which files are risky. We do not want creative or exploratory answers here. High temperature would cause the AI to sometimes pick random or irrelevant files.
+
+**Response format: `application/json`** — We explicitly request JSON via the `response_mime_type` config parameter. Without this, Gemini wraps the JSON in a markdown code fence (` ```json ... ``` `), which breaks `json.loads()`. Forcing the MIME type eliminates all parsing overhead.
+
+**Role: "Senior Application Security Engineer"** — Giving the model a specific professional role improves the quality of its file prioritisation. A generic prompt like "pick the most important files" produces weaker results than grounding the model in a specific domain expertise. It starts reasoning about auth routes and database queries rather than just file size or alphabetical order.
+
+**Output key: `critical_files`** — We use a named key rather than a raw array so the response is unambiguous to parse. A raw array with no wrapper can be confused with other response formats.
+
+---
+
+### File Analysis Prompt
+
+**Temperature: `0.2`** — Low but slightly higher than triage. Security analysis requires precise, grounded reasoning, but we give it a small amount of flexibility to reason across multiple vulnerability patterns in a single file. Too high (e.g. `0.7`) and the model starts inventing vulnerabilities that aren't there.
+
+**OWASP Top 10 explicit list** — We enumerate specific vulnerability classes (SQLi, XSS, IDOR, Hardcoded Secrets, Misconfigurations) rather than saying "find security issues". Vague prompts produce vague results. Named categories anchor the model to established security standards and produce more consistent severity ratings across different files.
+
+**Structured output schema enforced in the prompt** — We specify the exact keys each vulnerability object must have (`severity`, `issue`, `explanation`, `suggested_fix`, `line_number`). This is important because we parse the output directly into a Pydantic model. If the model returns different key names, parsing silently drops the data. By declaring the schema in plain English inside the prompt, we get consistent field names without needing to post-process the output.
+
+**`line_number` as integer or null** — We explicitly allow null rather than requiring an integer, because some vulnerabilities are architectural (e.g. "no authentication on this endpoint") and don't have a single line to point to. A strict integer requirement would cause the model to hallucinate a line number.
+
+**30,000 character cap** — The `gemini-2.0-flash` context window is large, but sending enormous files increases latency and cost significantly. Most security-critical logic in real files sits in the first few hundred lines. 30,000 characters (~6,000–8,000 lines) covers virtually all real source files while preventing edge cases from blowing up API response times.
+
+**Lock file exclusion** — `package-lock.json` and `yarn.lock` are autogenerated by package managers and can be 50,000+ lines of JSON. They contain no developer-written logic. Sending them to the model wastes tokens and API quota, and the model occasionally returns false positives on dependency version strings (flagging outdated packages as vulnerabilities, which is technically true but unhelpful without context). We skip them with a hard filter before any API call is made.
+
+---
+
+### Summary Prompt
+
+**Temperature: `0.4`** — Middle ground. The summary is a written paragraph, not structured data, so we want some natural language variation. But it's a professional security document, so we don't want it to be overly creative or inconsistent. `0.4` produces readable, varied prose that still stays factual and professional.
+
+**Role: "Senior AppSec Manager"** — Different from the analysis role ("Senior Application Security Engineer"). The manager persona produces higher-level, prioritised language that talks about business risk rather than technical specifics. This is intentional — the summary is meant for someone reading quickly, not a developer who will fix the code.
+
+---
+
+### Chat Prompt
+
+**Temperature: `0.5`** — Higher than the structured tasks because conversation requires natural, responsive language. The model needs to adapt its tone to the user's question — a technical "how does this work?" deserves a different register than "write me a patch".
+
+**Full context injection** — We pass the entire scan result (repo URL, executive summary, full vulnerability JSON) into every chat message. We do not maintain a conversation history array. This is a deliberate tradeoff: a single large context is simpler to implement and avoids the complexity of managing a rolling message window, at the cost of slightly higher token usage per message. For a developer chat session (typically 2–5 messages), this is the right call.
+
+**Role: "SecureLens AI"** — Giving the chat assistant a specific product name and identity ("You are SecureLens AI") improves response consistency and keeps the model on-topic. Without a defined persona, the model can drift into giving generic security advice unrelated to the specific scan results.
+
+---
+
+## The AI Client Setup
+
+Both AI systems use the same `google-genai` SDK client. The client is initialised once at module load time:
+
+```python
+if settings.gemini_api_key:
+    ai_client = genai.Client(api_key=settings.gemini_api_key)
+else:
+    ai_client = None
+```
+
+If no API key is set, `ai_client` is `None`, and every function that uses it has an early return that either returns empty results or a placeholder message. The app never crashes if the AI is unconfigured — it just degrades gracefully.
+
+All API calls use the async interface (`client.aio.models.generate_content`) so they never block the FastAPI event loop.
+
+---
+
+## Model Selection
+
+The current model used for all AI calls is **`gemini-2.0-flash`**.
+
+The choice of model matters a lot for a system like this:
+
+| Model | Speed | Quality | Free Tier Daily Limit |
+|---|---|---|---|
+| `gemini-2.5-flash` | Very fast | Excellent | ~20 requests (very low) |
+| `gemini-2.0-flash` | Fast | Very good | 1,500 requests |
+| `gemini-2.5-pro` | Slower | Best | Very low |
+
+`gemini-2.0-flash` is the practical choice for a system doing concurrent file analysis. It's fast enough that concurrent scans complete in under a minute, good enough to catch real OWASP Top 10 issues, and has a high enough daily free limit for real usage.
+
+To change the model, update the `model_name` attribute in `CodeScanOrchestrator.__init__()` and the hardcoded model string in the chat endpoint. Future versions of SecureLens will make this configurable via an environment variable, supporting any compatible LLM provider.
+
+---
+
+## Error Handling Philosophy
+
+Every AI call in this system is wrapped in a `try/except` block. The system never lets an AI failure bubble up as a 500 error to the user if there's a reasonable fallback available.
+
+- Triage fails → fall back to first 5 files alphabetically
+- File analysis fails → skip that file, continue with others
+- Summary generation fails → return a simple count ("Found N issues")
+- Chat fails → return a user-friendly error message
+
+The only place we raise an HTTP 500 is in the chat endpoint, because there's no meaningful fallback for a failed chat response — the user explicitly asked for an AI reply.

docs/api-reference.md

@@ -0,0 +1,313 @@
+# API Reference — SecureLens Backend
+
+All endpoints run at `http://localhost:8000` in development.
+
+Interactive documentation (Swagger UI) is available at `http://localhost:8000/docs`.
+
+---
+
+## Authentication
+
+SecureLens uses JWT Bearer tokens for authentication. After registering or logging in, you get a token. Pass it in the `Authorization` header for any protected endpoint:
+
+```
+Authorization: Bearer <your_token>
+```
+
+Tokens are valid for 24 hours by default (configurable via `JWT_EXPIRY_MINUTES` in `.env`).
+
+---
+
+## Endpoints
+
+### Health & Status
+
+#### `GET /`
+Returns a simple welcome message. Used to verify the server is up.
+
+**Response:**
+```json
+{ "message": "SecureLens AI Backend is running" }
+```
+
+#### `GET /health`
+Returns server health status and version info.
+
+**Response:**
+```json
+{
+  "status": "healthy",
+  "version": "1.0.0"
+}
+```
+
+---
+
+### Authentication
+
+#### `POST /auth/register`
+Creates a new account and returns a JWT token immediately (no separate login needed).
+
+**Request Body:**
+```json
+{
+  "email": "user@example.com",
+  "username": "myuser",
+  "password": "securepassword123"
+}
+```
+
+**Response:**
+```json
+{
+  "access_token": "eyJhbGciOiJIUzI1...",
+  "token_type": "bearer"
+}
+```
+
+**Errors:**
+- `400` — Email or username already exists
+
+---
+
+#### `POST /auth/login`
+Logs in with email and password, returns a JWT token.
+
+**Request Body:**
+```json
+{
+  "email": "user@example.com",
+  "password": "securepassword123"
+}
+```
+
+**Response:** Same as register.
+
+**Errors:**
+- `401` — Invalid credentials
+
+---
+
+#### `GET /auth/me`
+Returns info about the currently authenticated user. Requires a valid token.
+
+**Response:**
+```json
+{
+  "id": "uuid",
+  "email": "user@example.com",
+  "username": "myuser",
+  "created_at": "2026-04-25T10:00:00Z"
+}
+```
+
+---
+
+### Website Scanner
+
+#### `POST /scan`
+Scans a live URL for infrastructure-level security vulnerabilities. Authentication is optional — if you're logged in, the scan is saved to your history.
+
+**Request Body:**
+```json
+{
+  "url": "https://example.com"
+}
+```
+
+**Response:**
+```json
+{
+  "url": "https://example.com",
+  "score": 62,
+  "grade": "C",
+  "summary": "Found 8 security issues...",
+  "issues": [
+    {
+      "layer": "Server Config",
+      "issue": "Missing Content-Security-Policy header",
+      "severity": "High",
+      "explanation": "Without CSP, the browser cannot restrict...",
+      "remediation_snippet": "add_header Content-Security-Policy \"default-src 'self'\";"
+    }
+  ],
+  "threat_narrative": "An attacker could chain the missing CSP..."
+}
+```
+
+The `score` is 0–100 (higher is better). The `grade` maps to:
+
+| Score | Grade |
+|---|---|
+| 90–100 | A |
+| 75–89 | B |
+| 50–74 | C |
+| 25–49 | D |
+| 0–24 | F |
+
+**Errors:**
+- `400` — Invalid URL or URL validation failed (SSRF protection)
+- `422` — Malformed request body
+
+---
+
+### Scan History
+
+All history endpoints require authentication.
+
+#### `GET /scans`
+Returns a list of all your past website scans.
+
+**Response:**
+```json
+[
+  {
+    "id": "uuid",
+    "url": "https://example.com",
+    "score": 62,
+    "grade": "C",
+    "created_at": "2026-04-25T10:00:00Z"
+  }
+]
+```
+
+---
+
+#### `GET /scans/{id}`
+Returns the full details of a specific scan by ID.
+
+---
+
+#### `DELETE /scans/{id}`
+Deletes a scan from your history.
+
+**Response:** `204 No Content`
+
+**Errors:**
+- `404` — Scan not found or doesn't belong to your account
+
+---
+
+### Code Scanner (AI Agent)
+
+These endpoints power the GitHub repository scanner.
+
+#### `POST /code-scan/analyze`
+Triggers the full AI agent pipeline against a GitHub repository. This is the main endpoint.
+
+**What happens internally:**
+1. Fetches the complete file tree from GitHub
+2. AI selects the 5 most security-critical files (Triage)
+3. AI analyzes each file concurrently for OWASP Top 10 issues (Analysis)
+4. AI writes an executive summary (Summary)
+5. Results are stored in memory under the returned `scan_id`
+
+**Request Body:**
+```json
+{
+  "repo_url": "https://github.com/username/repository",
+  "github_token": "ghp_xxxxxxxxxxxx",
+  "branch": "main"
+}
+```
+
+- `repo_url` — Full GitHub repository URL
+- `github_token` — A GitHub personal access token with `repo` read scope. Needed to access private repos and to avoid GitHub's anonymous rate limits.
+- `branch` — Optional. Defaults to `"main"`.
+
+**Response:**
+```json
+{
+  "scan_id": "4432ad8e-4aa7-40bf-8f37-b0bbb96f6677",
+  "repo_url": "https://github.com/username/repository",
+  "summary": "The repository presents a moderate security risk...",
+  "issues": [
+    {
+      "file_path": "app/users/page.js",
+      "severity": "High",
+      "issue": "Potential Broken Access Control on User List API",
+      "explanation": "The client-side code fetches all users without authentication...",
+      "suggested_fix": "Add auth middleware to the /users API endpoint...",
+      "line_number": 12
+    }
+  ]
+}
+```
+
+**Save the `scan_id`** — you need it to use the chat endpoint.
+
+**Errors:**
+- `500` — AI API error, GitHub API error, or invalid repo URL
+
+---
+
+#### `POST /code-scan/chat`
+Opens a conversational interface with the AI, grounded in the context of a specific scan.
+
+The AI knows exactly what vulnerabilities were found in which files. You can ask it to explain issues, write patches, prioritise fixes, or anything else security-related.
+
+**Request Body:**
+```json
+{
+  "scan_id": "4432ad8e-4aa7-40bf-8f37-b0bbb96f6677",
+  "message": "Can you write a patch for the highest severity issue you found?"
+}
+```
+
+**Response:**
+```json
+{
+  "reply": "The highest severity issue was the Broken Access Control in app/users/page.js. Here's the fix...\n\n```javascript\n// Add this middleware to your API route\n..."
+}
+```
+
+**Errors:**
+- `404` — `scan_id` not found (scan expired or server restarted)
+- `500` — AI API error
+
+**Note:** Scan contexts are stored in memory and are lost when the server restarts. Always run a fresh `/code-scan/analyze` first when starting a new session.
+
+---
+
+#### `GET /code-scan/models`
+Lists all Gemini models available to your API key. Useful for debugging model access issues.
+
+**Response:**
+```json
+{
+  "models": [
+    "models/gemini-2.0-flash",
+    "models/gemini-2.5-flash",
+    "models/gemini-2.5-pro"
+  ]
+}
+```
+
+---
+
+## Rate Limits
+
+The API is rate-limited to **30 requests per minute** by default (configurable via `RATE_LIMIT` in `.env`).
+
+The Gemini AI calls are additionally subject to Google's API rate limits:
+
+| Model | Free Tier RPM | Free Tier RPD |
+|---|---|---|
+| `gemini-2.0-flash` | 15 | 1,500 |
+| `gemini-2.5-flash` | 10 | ~20 |
+
+If you hit the Gemini rate limit, the API returns a `429` from Google which surfaces as a `500` from the SecureLens server. Wait 60 seconds and try again.
+
+---
+
+## Error Codes
+
+| Code | Meaning |
+|---|---|
+| `400` | Bad request — check your input |
+| `401` | Not authenticated — missing or expired token |
+| `403` | Forbidden — you don't have permission for this resource |
+| `404` | Resource not found |
+| `422` | Validation error — malformed request body |
+| `429` | Rate limited — slow down your requests |
+| `500` | Server error — usually an AI API failure |

docs/architecture.md

@@ -0,0 +1,237 @@
+# Architecture Overview
+
+This document explains how the different pieces of SecureLens fit together — what each layer does, why it exists, and how data flows through the system.
+
+---
+
+## High-Level Architecture
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│                        CLIENT                                │
+│  (Next.js Frontend / Swagger UI / curl / API consumer)       │
+└───────────────────────────────┬──────────────────────────────┘
+                                │  HTTP requests
+                                ▼
+┌──────────────────────────────────────────────────────────────┐
+│                    FASTAPI APPLICATION                       │
+│                                                              │
+│  ┌───────────────┐  ┌───────────────┐  ┌─────────────────┐   │
+│  │  Auth Router  │  │  Scan Router  │  │ Code Scan Router│   │
+│  │  /auth/*      │  │  /scan        │  │  /code-scan/*   │   │
+│  └───────┬───────┘  └───────┬───────┘  └────────┬────────┘   │
+│          │                  │                    │           │
+│          ▼                  ▼                    ▼           │
+│  ┌──────────────┐  ┌─────────────────┐  ┌──────────────┐     │
+│  │ Auth Service │  │ Scanner Service │  │  Orchestrator│     │
+│  │ JWT + Users  │  │ 5 check layers  │  │ 3-phase agent│     │
+│  └──────┬───────┘  └───────┬─────────┘  └──────┬───────┘     │
+│         │                  │                    │            │
+└─────────┼──────────────────┼────────────────────┼────────────┘
+          │                  │                    │
+          ▼                  ▼                    ▼
+   ┌─────────────┐   ┌──────────────┐    ┌──────────────┐
+   │ PostgreSQL  │   │ Target URLs  │    │  GitHub API  │
+   │  Database   │   │ (live scans) │    │  + Gemini AI │
+   └─────────────┘   └──────────────┘    └──────────────┘
+```
+
+---
+
+## Application Layers
+
+### 1. FastAPI Application (`app/main.py`)
+
+This is the entry point. It creates the FastAPI app, registers all the routers, sets up CORS, and configures the lifespan (startup/shutdown logic like creating database tables).
+
+FastAPI is async from top to bottom. Every request handler is an `async def` function, which means the server can handle many concurrent requests without blocking on I/O — critical for a system that makes lots of external HTTP calls.
+
+The app listens on port `8000` and serves:
+- A REST API for all functionality
+- An interactive Swagger UI at `/docs`
+- An OpenAPI schema at `/openapi.json`
+
+---
+
+### 2. Routers (`app/routers/`)
+
+Routers are just groups of related endpoints. FastAPI uses them to keep the codebase organised.
+
+| File | What It Handles |
+|---|---|
+| `auth.py` | Register, login, get current user |
+| `scan.py` | Website URL scanning |
+| `history.py` | Reading and deleting past scan results |
+| `code_scan.py` | GitHub repo scanning + AI chat |
+| `health.py` | Health check endpoints |
+
+Routers don't contain business logic. They receive the request, call the appropriate service, and return the result. They're thin by design.
+
+---
+
+### 3. Services (`app/services/`)
+
+Services contain the actual business logic.
+
+#### `scanner/` — The Website Scanner
+
+A collection of five independent checkers, each responsible for one "layer" of security:
+
+- `transport.py` — Checks if the site uses HTTPS and implements HSTS correctly
+- `ssl_checker.py` — Validates the SSL certificate (expiry, chain, TLS version)
+- `headers.py` — Checks for the presence and correct configuration of security headers (CSP, X-Frame-Options, etc.)
+- `cookies.py` — Checks session cookies for HttpOnly, Secure, and SameSite flags
+- `exposure.py` — Probes for exposed sensitive paths like `/admin`, `/.env`, `/phpinfo.php`
+
+Each checker runs independently. The scan router calls all of them, collects their results, passes them to the scoring engine, then sends everything through the AI service for enhancement.
+
+#### `code_scanner/` — The Code Scanner Agent
+
+Contains the three-phase AI pipeline. See [ai-agent.md](./ai-agent.md) for a full explanation.
+
+- `orchestrator.py` — The main pipeline class (Triage → Analysis → Summary)
+- `github_client.py` — Handles all GitHub API communication
+
+#### `ai.py` — Website Scanner AI Layer
+
+Standalone functions that use Gemini to enhance the website scanner's results: `enhance_security_issues()`, `chat_with_scan_context()`, `generate_threat_narrative()`.
+
+#### `scoring.py` — The Scoring Engine
+
+A pure Python function that takes the list of issues from all scanners, applies weights based on severity, and produces a 0–100 score and an A–F letter grade. No AI involved here — it's deterministic and consistent.
+
+---
+
+### 4. Schemas (`app/schemas/`)
+
+Pydantic models that define the shape of every request and response. FastAPI uses these for automatic validation, serialisation, and documentation generation.
+
+If a request body doesn't match the schema, FastAPI returns a `422` automatically without your handler even being called.
+
+Key schemas:
+
+- `auth.py` — `RegisterRequest`, `LoginRequest`, `TokenResponse`, `UserResponse`
+- `scan.py` — `ScanRequest`, `ScanResponse`, `IssueDetail`
+- `code_scan.py` — `CodeScanRequest`, `CodeScanResponse`, `VulnerabilityIssue`, `CodeChatRequest`, `CodeChatResponse`
+
+---
+
+### 5. Models (`app/models/`)
+
+SQLAlchemy ORM models — the Python representation of database tables.
+
+- `user.py` — The `User` table (id, email, username, hashed_password, created_at)
+- `scan.py` — The `ScanResult` table (id, user_id, url, score, grade, full result JSON)
+
+These are what get stored in PostgreSQL. The code scanner's results are *not* stored in the database in the current version — they're kept in an in-memory dict in `code_scan.py`.
+
+---
+
+### 6. Middleware (`app/middleware/`)
+
+- `auth.py` — The `get_current_user` dependency. Any endpoint that requires authentication uses this. It validates the JWT token from the `Authorization` header and returns the user object.
+- `rate_limiter.py` — SlowAPI configuration. Limits the number of requests per IP per minute.
+
+---
+
+### 7. Utils (`app/utils/`)
+
+- `auth.py` — Low-level JWT functions: creating tokens, verifying tokens, hashing passwords, checking passwords
+- `validators.py` — URL validation and SSRF protection. Before scanning any URL, we check it's not a private IP address or localhost, which would let attackers use our scanner to probe internal networks
+
+---
+
+## Data Flow — Code Scan Request
+
+This is exactly what happens when you call `POST /code-scan/analyze`:
+
+```
+1. Request arrives at FastAPI
+      │
+2. Pydantic validates the body → CodeScanRequest(repo_url, github_token, branch)
+      │
+3. Router creates a CodeScanOrchestrator instance
+      │
+4. GitHubClient.get_repo_tree() → fetches all file paths via GitHub Trees API
+      │
+      ├── Makes 1-2 GitHub API calls (uses token for auth)
+      └── Returns: ["app/page.js", "app/users/page.js", "package.json", ...]
+      │
+5. orchestrator.triage_files() → sends file list to Gemini
+      │
+      ├── 1 Gemini API call with all filenames
+      └── Returns: ["app/users/page.js", "middleware.ts", ...] (5 files)
+      │
+6. orchestrator.analyze_files() → fetches and scans each file
+      │
+      ├── GitHubClient.get_file_content() × 5 (concurrent, async)
+      ├── Gemini generate_content() × 5 (concurrent, async, behind Semaphore)
+      └── Returns: [VulnerabilityIssue, VulnerabilityIssue, ...]
+      │
+7. orchestrator.generate_summary() → writes executive summary
+      │
+      ├── 1 Gemini API call with all vulnerability data
+      └── Returns: "The repository presents a moderate risk..."
+      │
+8. Router creates CodeScanResponse with a UUID scan_id
+      │
+9. scan_store[scan_id] = response  (saved in-memory for chat)
+      │
+10. Response returned to client (JSON)
+```
+
+Total external API calls: 2-3 GitHub + 7 Gemini = ~9-10 calls per scan.
+
+---
+
+## Database
+
+We use PostgreSQL in production (via Docker Compose) and SQLite in local development.
+
+The connection is managed by SQLAlchemy's async engine. All database operations use `async with get_db() as session:` — they never block.
+
+Migrations are managed by Alembic. To run migrations:
+
+```bash
+alembic upgrade head
+```
+
+The tables are also auto-created on startup in development mode (the `create_all()` call in `main.py`'s lifespan function).
+
+---
+
+## Environment Configuration
+
+All configuration is driven by the `.env` file. The `config.py` file uses Pydantic's `BaseSettings` to read it:
+
+```python
+class Settings(BaseSettings):
+    gemini_api_key: str | None = None
+    database_url: str = "sqlite+aiosqlite:///./securelens.db"
+    jwt_secret: str = "change-me-in-production"
+    # ...
+```
+
+If a required variable is missing, Pydantic raises an error on startup — not silently at runtime.
+
+See `.env.example` for the full list of options, or the Configuration section in [README.md](../README.md).
+
+---
+
+## Docker Setup
+
+The `docker-compose.yml` runs two services:
+
+```
+backend   ← FastAPI app (port 8000)
+db        ← PostgreSQL (port 5432, internal only)
+```
+
+The backend container reads `DATABASE_URL` from `.env` and connects to the `db` container over the internal Docker network. PostgreSQL data persists in a Docker volume across restarts.
+
+To rebuild from scratch:
+
+```bash
+docker compose down -v   # removes containers AND the data volume
+docker compose up --build
+```