Files
roboco/tests/integration/test_migration_028_seed_self_hosted.py
T
73b7c16211 [3cc1729c] Add self-hosted LLM provider with dynamic model discovery (#128)
* [684dace4] Self-hosted LLM provider: API layer, hooks, UI section, routing mode button, and Mix mode grouping (#124) (#126)

* [684dace4] feat(providers): add self-hosted LLM API types, endpoints, and React Query hooks

- Add ModelProvider.SELF_HOSTED enum value to types/index.ts
- Extend RoutingMode to include 'self_hosted' in lib/api/providers.ts
- Add SelfHostedConfig, SelfHostedTestResult, SelfHostedModel interfaces
- Add SelfHostedConfigPayload for PUT requests
- Add 5 providersApi methods: getSelfHostedConfig, saveSelfHostedConfig,
  testSelfHosted, getSelfHostedModels, refreshSelfHostedModels
- Add 5 React Query hooks: useSelfHostedConfig, useSetSelfHostedConfig,
  useTestSelfHosted, useSelfHostedModels, useRefreshSelfHostedModels
- Cache keys follow existing providerKeys pattern with proper invalidation

* [684dace4] feat(settings): create SelfHostedSection component with full self-hosted LLM UI

- Base URL text input with placeholder showing saved URL when set
- Optional auth token field (type='password') with Eye/EyeOff toggle button
- Save button that calls useSetSelfHostedConfig mutation
- Test Connection button disabled until a URL is saved; shows inline
  green 'Connected — N models' badge on success or red error badge on fail
- Three empty states: no URL configured (CTA), error state (last-checked +
  Retry), connected with 0 models (pull-guidance)
- Model list with auto-discovered chip, Refresh Models button, and
  Last refreshed relative timestamp when test_status === 'connected'
- Token field shows masked placeholder when has_auth_token is true
  (consistent with Ollama Cloud key field pattern)

* [684dace4] feat(settings): add Self-Hosted mode button, model picker, and Mix mode provider grouping

- Wire SelfHostedSection into AIRoutingCard with testResult state tracking
- Expand routing mode grid from 3 to 4 buttons (2×2 on mobile, 4-col on md+)
- 4th 'Self-Hosted' mode button disabled until test_status === 'connected'
- Self-hosted model picker appears below mode grid when mode === 'self_hosted'
- flipToSelfHosted handler sends mode='self_hosted' with optional default_model
- Mix mode per-agent dropdown now groups entries under SelectGroup/SelectLabel
  headings: Anthropic, Ollama Cloud, Self-Hosted with colored ProviderBadge pill
- saveMix validates self-hosted model selection requires a successful test
- ProviderBadge helper renders blue/violet/purple pills for each provider type
- pnpm typecheck and pnpm lint pass with zero errors

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [2897ce90] Implement self-hosted LLM provider API, routing, and discovery (#125) (#127)

* [2897ce90] feat(provider): add self-hosted LLM provider API, routing, and discovery

- Add migration 027 to seed Self-Hosted (Ollama) LOCAL provider row
- Add probe_ollama_tags() helper for Ollama /api/tags connectivity checks
- Extend ModelRoutingService: derive_mode returns 'self_hosted' for LOCAL
  GLOBAL assignments; apply_mode handles 'self_hosted' mode; upsert_assignment
  routes non-catalog model names to LOCAL provider; resolve_for_agent falls
  back to Anthropic when self-hosted server is unreachable
- Add PUT /api/providers/self-hosted, POST /api/providers/self-hosted/test,
  GET /api/providers/self-hosted/models endpoints
- Extend ApplyModeRequest and ModeResponse literals with 'self_hosted'
- Add SelfHostedConfigRequest, SelfHostedConfigResponse, SelfHostedTestResponse schemas

* [2897ce90] test(provider): add integration tests for self-hosted routing and route endpoints

- Add llm_setup_with_local fixture that seeds LOCAL provider row
- Test derive_mode returns 'self_hosted' for single GLOBAL LOCAL assignment
- Test apply_mode('self_hosted') clears prior assignments, enables LOCAL, inserts GLOBAL
- Test apply_mode('self_hosted') requires default_model argument
- Test upsert_assignment routes non-catalog model names to LOCAL provider
- Test mix mode accepts self-hosted model names without ValueError
- Test resolve_for_agent returns base_url when LOCAL server is reachable
- Test resolve_for_agent falls back to Anthropic when LOCAL server is unreachable
- Test upsert_assignment raises ValueError when model unknown and no LOCAL provider
- Add app_client_with_local fixture for route tests
- Test PUT /self-hosted saves base_url and enables provider
- Test PUT /self-hosted stores encrypted token when auth_token provided
- Test PUT /self-hosted returns 404 when LOCAL provider not seeded
- Test POST /self-hosted/test returns {ok:true,model_count:N} when reachable
- Test POST /self-hosted/test returns {ok:false,error} (never 500) when unreachable
- Test GET /self-hosted/models returns model name list
- Test GET /self-hosted/models returns 404 when not configured
- Test GET /self-hosted/models returns 503 when server unreachable
- Rename migration from 027 to 028 to rebase on 027_system_settings

* [2897ce90] chore(migration): remove superseded 027 migration, fix formatter changes to provider schemas

---------

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [042462df] feat(providers): align self-hosted types, hooks, and UI to backend contract (#129) (#131)

- SelfHostedConfig now has {base_url: string, has_token: boolean, enabled: boolean}
- SelfHostedTestResult now has {ok: boolean, model_count: number | null, error: string | null}
- Remove SelfHostedTestStatus type and refreshSelfHostedModels POST API function
- Remove SELF_HOSTED from ModelProvider enum (LOCAL covers self-hosted semantics)
- useRefreshSelfHostedModels now invalidates GET cache instead of calling POST
- isSelfHostedConnected derived from testResult?.ok === true
- Self-hosted model picker uses value='__clear__' sentinel (no empty-string SelectItem)
- self-hosted-section.tsx reads result.ok/result.error and config?.has_token
- pnpm typecheck passes with zero errors

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [f66d6d4d] Fix self-hosted API S1-S4/L1-L5: routes, schemas, services, migration 028, and tests (#130) (#132)

* [f66d6d4d] fix(provider): self-hosted API S1-S4/L1-L5 - routes, schemas, services, migration 028, and tests

AC1: Add GET /providers/self-hosted returning {base_url, has_token, enabled}
AC2: GET /self-hosted/models now returns list[SelfHostedModelEntry] with model_name and display_name
AC3: probe_ollama_tags generic except logs exception server-side and returns hardcoded generic string
AC4: upsert_assignment calls ProviderService.update_provider(enabled=True) when routing to LOCAL
AC5: derive_mode return annotation is Literal[...] — type:ignore comments removed
AC6: All migration refs in routes/services say 028 (not 027)
AC7: Migration 028 downgrade() deletes model_assignments before provider_configs
AC8: PUT /self-hosted only passes enabled=True when data.base_url is non-empty
AC9: ModelProvider.LOCAL docstring updated to describe self-hosted Ollama provider
AC10: Direct unit tests for probe_ollama_tags (5 cases) in tests/unit/llm/
AC11: Contract tests added/updated for GET /providers/self-hosted, models, and test endpoints
AC12: test_migration_028_seed_self_hosted.py with upgrade and FK-safe downgrade tests
AC13: test_apply_mode_ollama_without_provider_returns_404 asserts exactly HTTPStatus.NOT_FOUND
AC14: ruff and mypy pass with zero errors

* [f66d6d4d] fix(tests): add AC4 test proving LOCAL.enabled transitions False->True in upsert_assignment

The existing tests (test_upsert_assignment_routes_unknown_model_to_local and
test_mix_mode_with_self_hosted_models) both use llm_setup_with_local which seeds
LOCAL with enabled=True, making the AC4 assertion vacuous.

New test test_upsert_assignment_enables_local_when_disabled:
- Creates LOCAL ProviderConfigTable row with enabled=False
- Asserts pre-condition: local.enabled is False
- Calls upsert_assignment with a non-catalog model name ('non-catalog-model:7b')
- Refreshes LOCAL row via db_session.refresh(local)
- Asserts row.provider.type == ModelProvider.LOCAL and local.enabled is True

This proves the state transition from False->True, not merely that the
already-enabled state is preserved. ruff and mypy still pass with zero errors.

---------

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [7cd6ae6e] fix(providers): type SelfHostedConfig.base_url as string | null to match backend contract (#133) (#136)

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>

* [46ee9104] test(migration_028): replace upgrade test with self-seeding contract test (#134) (#135)

Remove test_migration_028_upgrade_local_row_inserted which relied on alembic
upgrade head having run (and thus the Self-Hosted Ollama row being present).

Replace it with test_migration_028_upgrade_insert_contract that:
- Executes the exact INSERT SQL from migration 028 upgrade() directly
- Asserts name='Self-Hosted (Ollama)', type='local', enabled=False
- Runs the INSERT a second time and asserts exactly one row (ON CONFLICT
  DO NOTHING idempotency)

The downgrade test is left byte-for-byte unchanged.

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* [f0d19f30] test(provider): add DELETE-before-seed isolation and app_client_with_ollama fixture (#137) (#138)

- Add ModelAssignmentTable import to test_provider_routes.py
- Fix app_client_with_local: execute DELETE on ModelAssignmentTable then
  DELETE on ProviderConfigTable (FK-safe order) and flush before seeding
- Add new app_client_with_ollama fixture with same isolation pattern,
  seeding only ANTHROPIC + OLLAMA_CLOUD rows
- Update 7 tests to use app_client_with_ollama instead of app_client:
  test_get_catalog, test_get_ollama_key_status, test_set_ollama_key,
  test_get_current_mode, test_apply_mode_anthropic_clears_assignments,
  test_apply_mode_unknown_returns_4xx, test_apply_mode_mix_without_per_agent_returns_400

Fixes order-dependent failures in test_get_self_hosted_models_not_configured_returns_404:
routes call db.commit() which persists rows across test sessions; without
DELETE-before-seed, stale LOCAL provider rows with base_url set from prior
runs cause the test to see 503 instead of 404.

Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>

* refactor(llm): split resolve_for_agent and apply_mode to clear xenon rank C

resolve_for_agent and apply_mode were cyclomatic rank C, failing the
xenon gate (--max-absolute B). Extract behavior-preserving helpers:

- resolve_for_agent -> _resolve_assignment (precedence ladder),
  _route_from_resolved / _local_route_or_none / _decrypt_route_or_none
  (None signals fall-through to legacy), _legacy_route.
- apply_mode -> _apply_anthropic / _apply_ollama / _apply_self_hosted /
  _apply_mix dispatched from a thin if/elif.

No behavior change. Also correct the stale 'default: Kimi K2.6' docstring
(OLLAMA_DEFAULT_MODEL is minimax-m3:cloud).

---------

Co-authored-by: Frontend Developer 1 <fe-dev-1@agents.roboco.dev>
Co-authored-by: Backend Developer 1 <be-dev-1@agents.roboco.dev>
Co-authored-by: Renn F <rennf93@users.noreply.github.com>
2026-06-13 08:38:31 +02:00

169 lines
5.6 KiB
Python

"""Migration 028 tests — seed_self_hosted_provider.
Verifies the post-upgrade state and exercises the downgrade SQL ordering
to prove the FK-safe delete sequence works.
NOT a real alembic round-trip — the suite builds the test DB via
Base.metadata.create_all (see conftest). Migration 028's upgrade()/downgrade()
bodies are reviewed here; the tests guard the resulting DB-level contract.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from uuid import uuid4
import pytest
from roboco.db.tables import ModelAssignmentTable, ProviderConfigTable
from roboco.models.base import AssignmentScope, ModelProvider
from sqlalchemy import text
if TYPE_CHECKING:
from sqlalchemy.ext.asyncio import AsyncSession
@pytest.mark.asyncio
async def test_migration_028_upgrade_insert_contract(
db_session: AsyncSession,
) -> None:
"""The upgrade INSERT SQL seeds the correct LOCAL provider row and is idempotent.
Executes the INSERT ... ON CONFLICT DO NOTHING SQL from migration 028's
upgrade() directly in the test session, verifying:
- name='Self-Hosted (Ollama)', type='local', enabled=False on the row.
- Running the same INSERT a second time leaves exactly one row (idempotency).
"""
_insert_sql = text(
"""
INSERT INTO provider_configs
(id, name, type, base_url, auth_token_encrypted, enabled, created_at)
VALUES
(
gen_random_uuid(),
'Self-Hosted (Ollama)',
'local',
NULL,
NULL,
false,
now()
)
ON CONFLICT (name) DO NOTHING
"""
)
# --- First run: the row should be inserted.
await db_session.execute(_insert_sql)
await db_session.flush()
# Verify the field contract on the newly-inserted row.
result = await db_session.execute(
text(
"SELECT name, type, enabled "
"FROM provider_configs "
"WHERE name = 'Self-Hosted (Ollama)'"
)
)
rows = list(result)
assert len(rows) == 1
name, ptype, enabled = rows[0]
assert name == "Self-Hosted (Ollama)"
assert ptype == "local"
assert enabled is False # starts disabled; user configures via PUT /self-hosted
# --- Second run: ON CONFLICT DO NOTHING must not create a duplicate.
await db_session.execute(_insert_sql)
await db_session.flush()
result = await db_session.execute(
text("SELECT id FROM provider_configs WHERE name = 'Self-Hosted (Ollama)'")
)
assert len(list(result)) == 1, (
"Expected exactly one 'Self-Hosted (Ollama)' row after two INSERT "
"executions; ON CONFLICT DO NOTHING must prevent duplicates."
)
@pytest.mark.asyncio
async def test_migration_028_downgrade_deletes_assignments_before_config(
db_session: AsyncSession,
) -> None:
"""Downgrade SQL deletes model_assignments before provider_configs.
Simulates the downgrade() logic from migration 028:
1. DELETE FROM model_assignments WHERE provider_config_id IN (SELECT id ...)
2. DELETE FROM provider_configs WHERE name = 'Self-Hosted (Ollama)'
A FK RESTRICT constraint on model_assignments.provider_config_id means that
executing step 2 before step 1 would raise an IntegrityError. This test
proves that doing them in the correct order succeeds without violation.
"""
# --- Arrange: insert a fresh LOCAL provider row and a referencing assignment.
suffix = uuid4().hex[:8]
local = ProviderConfigTable(
name=f"Self-Hosted (Ollama)-test-{suffix}",
type=ModelProvider.LOCAL,
enabled=False,
)
db_session.add(local)
await db_session.flush()
assignment = ModelAssignmentTable(
scope=AssignmentScope.AGENT_SLUG,
scope_value=f"test-agent-{suffix}",
provider_config_id=local.id,
model_name="llama3.1:8b",
)
db_session.add(assignment)
await db_session.flush()
# Verify both rows exist before we run the downgrade SQL.
result = await db_session.execute(
text("SELECT id FROM provider_configs WHERE name = :name").bindparams(
name=local.name
)
)
assert result.scalar_one_or_none() is not None
result = await db_session.execute(
text("SELECT id FROM model_assignments WHERE scope_value = :sv").bindparams(
sv=assignment.scope_value
)
)
assert result.scalar_one_or_none() is not None
# --- Act: execute downgrade SQL in the correct FK-safe order.
# Step 1: delete referencing model_assignments first.
await db_session.execute(
text(
"DELETE FROM model_assignments "
"WHERE provider_config_id IN ("
" SELECT id FROM provider_configs WHERE name = :name"
")"
).bindparams(name=local.name)
)
# Step 2: now safe to delete the provider row.
await db_session.execute(
text("DELETE FROM provider_configs WHERE name = :name").bindparams(
name=local.name
)
)
# --- Assert: both rows are gone, no IntegrityError was raised.
result = await db_session.execute(
text("SELECT id FROM provider_configs WHERE name = :name").bindparams(
name=local.name
)
)
assert result.scalar_one_or_none() is None, (
"provider_configs row should be deleted by downgrade"
)
result = await db_session.execute(
text("SELECT id FROM model_assignments WHERE scope_value = :sv").bindparams(
sv=assignment.scope_value
)
)
assert result.scalar_one_or_none() is None, (
"model_assignments row should be deleted before provider_configs"
)