Files
open-design/docs/testing/workspace-funding-sandbox.md

20 KiB
Raw Permalink Blame History

Real local Workspace funding regression

This explicit sandbox lane preserves the VIEW-015 browser transitions observed on OD 39b9966cd0fec6f5f0d1a3697ce4fcc6683de1af and AMR 885d2f8229d725768803db8b18058ae0f97e3bdb. The original Home red evidence is retained; the new cases have no skip, expected-failure, or relaxed draft oracle. Tests must run on the selected new source before claiming its result.

This is separate from the normal hermetic UI pool. It requires an explicitly leased disposable loopback AMR API/Link/Web environment with real PG/Redis, an explicit compiled Vela CLI, and a controlled upstream provider. It does not create or stop AMR, change pricing, recharge funds, or implement task concurrency. It uses the repository's suite entrypoint and shared tools-dev lifecycle for one independent OD namespace/data root per test. Daemon data paths follow the root AGENTS.md data directory contract. It does not borrow normal developer data.

Run from e2e, after the normal workspace dependency/build preparation, through the mandatory owner runner. The evidence directory must be a fresh, nonexistent path under an existing private evidence parent:

OD_FUNDING_FIXTURE="$PRIVATE_FIXTURE" OD_FUNDING_RUN_ID="$SANDBOX_RUN_ID" \
OD_FUNDING_EVIDENCE_DIR="$NEW_PRIVATE_EVIDENCE" \
  pnpm exec tsx lib/playwright/funding-runner.ts

The config uses one worker and no retry; the three cases are independent and can be selected individually with OD_FUNDING_CASE=project, home, or dp. Direct Playwright execution fails before OD startup. Missing, expired, mismatched, or non-private fixtures fail explicitly. Do not insert this lane into CI without a disposable fixture provisioner. The normal config excludes this file, so ordinary CI does not require private sandbox inputs.

The external JSON fixture is mode 0600 and is never committed:

{
  "version": 1,
  "isolation": {
    "runId": "<exclusive run ID, same as OD_FUNDING_RUN_ID>",
    "expiresAt": "<future ISO timestamp>",
    "workspaceNamePrefix": "<unique disposable workspace name prefix>",
    "exclusive": true
  },
  "upstream": {
    "apiUrl": "<loopback API origin>",
    "linkUrl": "<loopback Link origin>",
    "webUrl": "<loopback console origin>",
    "amrSha": "<complete source SHA>"
  },
  "owner": {
    "sourceRoot": "<absolute canonical AMR checkout>",
    "sandboxName": "<exact named sandbox>",
    "ownerEnvPath": "<absolute 0600 server-owner capability JSON>",
    "runtimeReceiptPath": "<absolute 0600 server-owner runtime receipt JSON>",
    "helperSha256": "<canonical e2e/helper/e2e_ownership.py SHA256>"
  },
  "runtime": {
    "velaBin": "<absolute explicitly supplied executable>",
    "velaSha256": "<SHA256>",
    "openCodeBin": "<absolute explicitly supplied executable>",
    "openCodeSha256": "<SHA256>",
    "modelId": "<model routed to the controlled sandbox provider>"
  },
  "actor": {
    "controlKey": "<sandbox control key>",
    "userId": "<sandbox user>",
    "email": "<sandbox email>"
  },
  "spaces": {
    "funded": { "workspaceId": "<A>", "ownerMemberId": "<A owner>", "runtimeKey": "<A scoped key>" },
    "empty": { "workspaceId": "<B>", "ownerMemberId": "<B owner>", "runtimeKey": "<B scoped key>" },
    "dp": { "workspaceId": "<D>", "ownerMemberId": "<D owner>", "runtimeKey": "<D scoped key>" }
  }
}

The environment owner provisions three distinct Go spaces for the same Owner: A has $100; B has $0 and both DP windows exhausted; D has $0 and both DP windows available with modelCovered === true for the chosen model. Legitimate advisory fundingCheck: unknown remains allowed for A/D. The runner verifies exact IDs/Owner memberships, name prefix, real preflight verdicts, and executable hashes before the browser workflow. exclusive and future timestamps alone are insufficient: the server owner holds the real canonical target lock throughout the browser command.

The capability JSON contains only E2E_DATABASE_URL, VELA_E2E_OWNER_TOKEN, VELA_E2E_OWNER_LABEL, VELA_E2E_OWNER_KEY, and VELA_SANDBOX_NAME. The server's runtime receipt contains name, database, targetKey, sourceRoot, amrSha, urls: {api, link, web}, redisDb, guardianPid, ISO deadlineUtc, helperPath, and helperSha256. The runner checks the actual canonical sandbox/cli.py status <name> --json, exact Git HEAD, clean tracked source, both canonical control files' Git bytes, endpoints, database, Redis DB, and inherited target digest. It then executes the real canonical e2e/helper/e2e_ownership.py verify-inherited immediately before and in finally after the entire browser command. Each OD case also checks before startup and after cleanup. No new lock is acquired. Failed pre-verification starts no child; failed post-verification rejects an otherwise green child as ownership-unverified. The guardian must retain the whole-command budget plus cleanup reserve. The runner cannot take over or stop the server owner's AMR environment.

Only safe identity/exit receipts are curated; database URLs and tokens never appear there. Provisioning/provider isolation and the subsequent AMR all-key/ ledger audit remain the environment owner's job. Unit checks of the guard are not evidence that a real live lease was held during any browser execution.

For Project and Home, only the delivery time of genuine OD billing responses is controlled using route.fetch; their body bytes are unchanged and hashed. B's actual response is completed in the browser and its insufficient dialog is visible before A's older response is delivered. Both phases assert the original page, exact draft, and zero new run; editability is required after dismissing the modal. Background editability while the modal is open is diagnostic only. Project creation and queue presence are diagnostic records, not additional QA requirements. The Project case creates its own local project and establishes its Recent entry before testing the real workspace switch.

The positive D case requires one correctly scoped HTTP 202 submission followed by the actual persisted run reaching succeeded. It retains the complete run events before cleanup and records whether end_turn appears, without imposing that exact event serialization as a product requirement. Only an unfinished test-created run is canceled during cleanup. Provider admission, actual CLI completion and the canonical usage/DP-window/zero-cash reconciliation still require the environment owner's independent readonly audit. These three cases cover VIEW-015 slices; they do not mark the whole QA case passed. Project and Home each map to step 1; D covers the browser/runtime portion of step 2 and requires the separate accounting evidence before whole-case review. Attachment preservation is not exercised in this response-order scenario. The absence of POST /api/runs proves no new OD run was submitted through this page; a separate AMR/provider/ledger audit is needed to rule out model requests through every other path. The source cannot turn HTTP 202 into successful DP use.

Playwright traces and runtime scratch are local diagnostics and can contain sandbox credentials. Review/redact them before any publication. Curated JSON attachments omit control/runtime keys and user email. All created runtime processes are stopped through the shared lifecycle in finally, after page close.

The concurrency-runner.ts entrypoint borrows the same canonical server-owner lease as the funding lane. It requires a fresh evidence directory and starts only the repository browser spec plus the fixture's immutable server-owned test coordinator. It does not start, stop, or repair AMR, change pricing, recharge funds, or introduce application task concurrency.

The private funding fixture adds concurrency.space with the exact Owner actor, concurrency.member with a different real membership/key, holdModelId, and coordinator fields helperPath, helperSha256, helperSourceSha, and helperRepoPath. The copied coordinator must match both its declared SHA256 and the exact Git blob in the server repository. The runtime checkout HEAD and the canonical receipt must still match the fixture's AMR source; the coordinator's pure-test commit is separately recorded rather than silently changing that HEAD.

From e2e/, supply an existing exclusively leased private fixture and a new output path, then run:

OD_FUNDING_FIXTURE=<absolute-private-fixture> \
OD_FUNDING_RUN_ID=<exact-fixture-run-id> \
OD_FUNDING_EVIDENCE_DIR=<new-absolute-output-directory> \
F04_SURFACE=home \
corepack pnpm exec tsx lib/playwright/concurrency-runner.ts

Omit F04_SURFACE to execute both Home and Project. Missing capability files, expired canonical ownership, reused output paths, different principals, or a coordinator source mismatch reject the command before browser startup. The coordinator receives only the explicit fixture/output paths, run ID, a fresh attempt UUID, selected surfaces, and fixture digest. Every hold/release marker binds protocol od-link-concurrency-v1, run ID, attempt ID, tag, workspace, surface, and fixture SHA256. Each surface retains separate immutable server-held-receipt.safe.json and server-released-receipt.safe.json files. The coordinator atomically writes each receipt before its matching marker and only releases request tags created for this exact attempt, even on failure; it must never release all provider requests. Each marker hashes its receipt; the UI verifies that receipt and requires 10 held requests or 0 remaining. An old ready: true file cannot release the browser.

The browser continues to require no run POST on refusal, original draft and real attachment retention, no automatic submit after Link capacity release, and a successful explicitly submitted run. The failed Project counterexample remains an assertion failure. A successful subprocess, helper unit test, or delivered commit alone is not whole F04/RUN-003 acceptance; the complete QA role/tier, visual, and timing requirements still apply. Preserve original command failures and record the actual tested product/test commits on every real execution.

The private concurrency.serverControl additionally binds a dedicated controlled provider origin, private PID/port state, source file/SHA256, a 0600 plain-text control-token file, and the canonical sandbox Redis container/prefix. Provider control calls require that token; model calls still go through actual Link with the Owner/Member runtime keys. No control token is recorded in receipts or argv. The browser verifies the authoritative workspace snapshot is the intended Go space before using the Go10 capacity witness; legacy summary fields are not used to fabricate that precondition.

For F04 step 2's explicit single-call recovery check, add F04_RELEASE_ONE=1 and F04_SURFACE=home to the same canonical concurrency runner command. This keeps nine real Link requests held while checking draft and attachment retention, no automatic submission, and a successful explicit manual send. The separate hashed server-one-released-receipt.safe.json proves the intermediate count of nine; the final released receipt proves cleanup to zero. It does not cover the remaining tier/surface matrix, visual requirements, or RUN-003 timing targets.

OD_FUNDING_CASE=view020 runs the real member/admin insufficient-funds notice layout case through the same canonical owner runner. Run it once for each of separate admin and member version2 fixtures; a single role does not pass the whole QA case. Each fixture includes noticeLayout with the actual long Owner profile and the path/digest of its environment owner's private read-only profile proof. The proof binds the current AMR SHA, empty workspace, Owner identity/name, and actual actor membership/role. No Owner name is injected into the UI.

The matrix uses empty and existing-artifact projects, English and Simplified Chinese, the established 1280×800 baseline and a 375×812 narrow window. It changes the real stored locale preference and reloads the project, resizes the open notice, and closes/reopens it. It covers visible/actionable contact-Owner copy and close controls, stacking above project content, absence of purchase actions, and no task start on dismissal. It does not impose F04's image/benefits checks, all19locales, name interpolation, or initial-send admission as new VIEW-020 criteria. Both roles' screenshots, traces, profile/authorization evidence and server-side no-usage review are required before whole-case acceptance.

VIEW-021: actual Free gift and retained funds

OD_FUNDING_CASE=view021 uses the same mandatory canonical runner and a private version2 fixture with freeWallet.retained and freeWallet.giftOnly. These are actual default personal spaces and separate real Owner sessions; they are not team-header aliases. Each wallet includes a hashed private original-signup cookie file and server-proven original gift/retained funds. The client keeps this phase's simplified single balance card.

The lane checks the actual canonical phase1 console and client before/after original-gift expiry:40.30→40.00 and0.30→0.00, Free onlyFlash and no new topup actions. Both before sets are frozen before the repository-owned expiry helper can run. The helper is bound by source/rawhash, canonical lease, fixture hash, new run/attempt IDs, both exact workspace IDs and original gift IDs. For a qualified natural-expiry check use freeWallet.expiry.mode=natural_wait: prepare the original gifts' common near-future expiry before opening either UI, preserve their three-day lifetime, then make no further funding-row changes. Before evidence must show the client projection carries that same future expiry. The helper waits for actual database time to cross it and refreshes the production projection. Its receipt proves the original boundary, clock samples before and after, and no funding-row writes after the before evidence. The retained cash, global time, AMR lifecycle and model prices stay outside this operation. The older controlled-age mode remains a diagnostic; advancing a date after clients have cached a later expiry does not prove natural-crossing behavior.

freeWallet.consoleUi records the actual canonical served feature flag and source proof. Only visible seats-console-page, exact scoped snapshot and wallet DOM can satisfy console acceptance. A legacy console is explicitly blocked configuration evidence, never phase1 UI acceptance. The command's exit0 can mean a qualified client/API portion only; VIEW021-exact-scope-results and the independent server/root review decide the full QA case. Original failures remain frozen.

The fixed product HEAD is still mandatory. The single reviewed canonical seats-UI opt-in infrastructure change may be declared as owner.canonicalToolsOverlay: source3dfcdc6fb4f940f2955cfa4835315d4ee9f83c82 and exact raw/blob SHA256s for sandbox/cli.py and sandbox/app/docker-compose.yml. The verifier requires exactly those two tracked differences, checks all other tracked product files unchanged and keeps the ownership helper equal to the fixed HEAD. This is not a general dirty-checkout allowance. Missing overlay metadata retains the original fully clean source guard.

A separate OD_FUNDING_CASE=view021-after continuation accepts the original receipt path and SHA256 through OD_FREE_WALLET_EXPIRY_RECEIPT and OD_FREE_WALLET_EXPIRY_RECEIPT_SHA256. It verifies the same fixture/run/workspace binding, opens fresh real Owner clients and consoles, and records both after amounts without repeating before or executing expiry. It cannot erase a prior failed execution or turn a diagnostic into whole-case acceptance.

WS-005: concurrent tasks retain their original workspace

OD_FUNDING_CASE=ws005 selects the explicit canonical two-space lane. Its private scopeSwitch Given binds the same real actor to A with 10 USD and B with 20 USD, their original member identities, an original balance proof and a repository-owned controlled upstream. A real HTTP DeviceAuthorization login provides one cli_device runtime/control key pair for the whole flow, linked by actual key hashes, approved authorization, and both active memberships in the same private proof. Workspace switching never replaces that credential; a workspace-bound source=user API key cannot stand in for an account login. The standard Go and Free fixtures keep their existing assertions; this scenario does not infer membership tier from the Coding Plan DTO.

The actual browser starts A, observes its main request held, navigates through Home and the real workspace switcher to B, then submits B. The dedicated upstream releases A only after this attempt's B main request actually arrives. Auxiliary requests cannot satisfy that phase boundary. Existing prices and allowlists stay fixed; every actual main and auxiliary charge must be included in the task totals of 2 and 3 USD without assuming one auxiliary invocation.

The test records both persisted terminal runs, A 8 / B 17 authoritative balances, the visible selected B row and current B context. Independent server review must match every usage and unique ledger to the original run, workspace and initiator. The environment owner remains the only fixture/route writer. A shared auxiliary route may be borrowed only in the explicitly coordinated window, with before/after snapshots and normal restoration; fault experiments on the same environment must not overlap that route change.

Use corepack pnpm --filter @open-design/e2e exec tsx lib/playwright/free-concurrency-runner.ts with the existing private OD_FUNDING_FIXTURE, OD_FUNDING_RUN_ID, and a fresh absolute OD_FUNDING_EVIDENCE_DIR. The runtime-only fixture carries actual Owner/Observer DeviceAuthorization pairs, W1/W2, authoritative Free=3 policy provenance, immutable repository helpers, and an already-owned canonical sandbox. The command borrows its lease; it never allocates or starts AMR.

The independent observer uses the real authenticated workspace SSE and the actual quota realtime health, while the initiating Home surface is blocked before a project/run POST. Three actual Link model requests occupy the shared pool. The owner coordinator releases exactly one, the browser observes the original QA's real one-minute idle period, and manual submission rereads concurrency and funding. W2 runs while W1 is full. A real invitation/acceptance adds a W1 member, and a request by that member still encounters Link's shared limit. Provider usage is controlled and positive; model prices and catalog permissions stay unchanged. Per-phase immutable receipts, screenshots, HTTP observations, real terminal runs, financial snapshots, and owned cleanup remain in the evidence output. Offline guards do not establish QA acceptance.

The same canonical free-concurrency-runner.ts command selects a separate RUN-002 browser witness only when the private fixture explicitly carries freeConcurrency.manualRetry.caseKey = "server:RUN-002" and its immutable repository coordinator. It does not transfer a RUN013 passing result: the RUN013 advisory block alone is insufficient for RUN002's actual Link refusal precondition.

This lane reads the genuine two-of-three advisory response, holds only its delivery until a third actual Link request arrives, and delivers the unchanged bytes. The real project submission must then fail at Link, preserve its input, and show the concurrency dialog. Exactly one held request completes; no task restarts during a real minute. Manual submission rereads funds and concurrency, starts one successful run and keeps every charge in its original scope. Financial audit uses a fresh per-attempt baseline, preserving prior attempts' funds and request history. UI signals and immutable receipt filenames occupy disjoint namespaces. Original failures remain frozen; upgrades are the other allowed trigger in QA, rather than a mandatory extra matrix for this release branch.