[chore] ci-watch/dep-update dedupe: normalize git_url + treat empty-string workflow as default (#148 #1267)

The per-repo open-task dedupe filtered ProjectTable.git_url == git_url
(exact), while the orchestrator collapses its poll set by repo_key
(lower / strip trailing '/' / drop '.git'). Two projects whose git_url
differs only by those accidentals (a monorepo's cell-projects, or a
re-registered canonical project) defeated the one-open-task-per-repo
invariant and opened duplicate fix / dep-update tasks. Extract
roboco.utils.converters.repo_key as the single source and match the
dedupe query on its SQL mirror (regexp_replace(rtrim(lower(...)))).

The ci_watch (git_url, workflow) dedupe used func.coalesce(ci_watch_workflow,
default), but SQL COALESCE only substitutes for NULL — a project saved with
ci_watch_workflow='' (reachable via panel/API) yielded coalesce('', default)
= '' != default, so the DB diverged from the engine/orchestrator (which
collapse '' to the default via Python truthiness) and opened a duplicate
fix task every red cycle. Wrap with func.nullif(..., '') so an empty string
collapses to the default too.

Tests: a ''-workflow + NULL-workflow project on one repo dedupe to one task;
git_url accidentals (.git suffix / trailing slash) dedupe across both
ci_watch and dep_update. The orchestrator _repo_key now delegates to repo_key.
This commit is contained in:
Renn F
2026-06-30 19:19:31 +02:00
parent 6907103073
commit d34bc1a7a5
5 changed files with 100 additions and 8 deletions
@@ -209,3 +209,40 @@ async def test_default_workflow_null_rows_deduped(
src = _FakeSource([_breach("mono3-a"), _breach("mono3-b")])
created = await get_ci_watch_engine(db_session, source=src).run_cycle([p1, p2])
assert len(created) == 1
# ---------------------------------------------------------------------------
# #148: an empty-string ci_watch_workflow (saved via panel/API, not NULL) must
# collapse to the default workflow for dedupe — SQL COALESCE alone treats '' as
# a real value, so the DB query diverged from the engine's Python truthiness and
# opened a duplicate fix task every red cycle.
# #1267: git_url accidentals (case / .git suffix / trailing slash) must collapse
# to one repo for dedupe, mirroring the orchestrator's poll-set repo_key.
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_empty_string_workflow_deduped(db_session: AsyncSession) -> None:
"""#148: a ''-workflow project and a NULL-workflow project on one repo both
use the default workflow -> deduped to one task (NULLIF treats '' as NULL)."""
git = "https://github.com/x/empty-wf.git"
p1 = await _seed_project(db_session, "empty-wf-a", git, workflow="")
p2 = await _seed_project(db_session, "empty-wf-b", git) # ci_watch_workflow=None
src = _FakeSource([_breach("empty-wf-a"), _breach("empty-wf-b")])
created = await get_ci_watch_engine(db_session, source=src).run_cycle([p1, p2])
assert len(created) == 1
@pytest.mark.asyncio
async def test_git_url_accidentals_deduped(db_session: AsyncSession) -> None:
"""#1267: two projects whose git_url differs only by a ``.git`` suffix (and
trailing slash) are the same repo -> one fix task, not two."""
p1 = await _seed_project(
db_session, "acc-a", "https://github.com/x/acc.git", workflow="wf.yml"
)
p2 = await _seed_project(
db_session, "acc-b", "https://github.com/x/acc/", workflow="wf.yml"
)
src = _FakeSource([_breach("acc-a"), _breach("acc-b")])
created = await get_ci_watch_engine(db_session, source=src).run_cycle([p1, p2])
assert len(created) == 1
@@ -130,3 +130,19 @@ async def test_git_url_scoping(db_session: AsyncSession) -> None:
scoped = await svc.list_open_dep_update_tasks(git_url="https://github.com/x/a.git")
assert len(scoped) == 1
assert scoped[0].project_id == proj_a.id
@pytest.mark.asyncio
async def test_git_url_accidentals_scoping(db_session: AsyncSession) -> None:
"""#1267: a dep_update task open on ``.../a.git`` is found when scoping by a
git_url that differs only by a ``.git`` suffix / trailing slash — the dedupe
key is the normalized repo, not the exact string."""
proj_a = await _seed_project(db_session, "https://github.com/x/a.git")
await _make_task(db_session, proj_a)
svc = get_task_service(db_session)
# Same repo, accidental variants — each scope finds the one open task.
for variant in ("https://github.com/x/a", "https://github.com/x/a.git/"):
scoped = await svc.list_open_dep_update_tasks(git_url=variant)
assert len(scoped) == 1
assert scoped[0].project_id == proj_a.id