agent-skills/skills/test-real-entry-paths/SKILL.md

name, description

name	description
test-real-entry-paths	Use when designing integration, end-to-end, release, or regression tests that must prove shipped behavior through the project's actual composition and published runtime entry point rather than a hand-assembled test path.

Test Real Entry Paths

A green unit suite can coexist with a completely broken product when tests bypass loading, composition, packaging, or the external world. Identify the path users actually execute for this project and test that path.

1. Name the shipped path before choosing a test

Ask:

What exact artifact and entry point does a user, editor, server, job runner, package consumer, or operating system invoke after release?

Examples include a built CLI executable, published package export, application bootstrap, plugin/config loader, worker entry, installed extension, container command, HTTP route through production middleware, or migration binary. Do not assume source imports or a specific stack.

Write the chain:

published artifact → loader/configuration → composition/DI → public action → external state

The test must traverse every link implicated by the change.

2. Identify bypasses in existing tests

Flag tests that:

• import an internal function instead of the published entry;
• mount a component manually instead of using the shipping loader;
• run source through a development transpiler while users execute built output;
• replace real downstream tools with hand-written stand-ins;
• assert only a returned message or agent self-report;
• never exercise module exports, configuration resolution, registration, or process exit.

These tests may remain useful unit tests. They do not satisfy real-entry coverage.

3. Mock only true nondeterminism

Keep real:

• application composition and dependency registration;
• loaders and configuration parsing;
• the tool/executor under test;
• persistence and serialization;
• public transport and protocol assembly;
• built artifact and process boundary where shipped.

Mock only a genuinely nondeterministic, costly, or unavailable boundary: LLM response, remote network, wall clock, random source, or third-party outage. Put the mock at that outer boundary, not between internal components.

Bad: replace the filesystem tool with { success: true }.

Good: script the model’s tool call, run the real filesystem tool in a temporary workspace, then inspect the file externally.

4. Assert the world, not the actor’s story

If the action should create a file, read it from a separate handle. If it should run a command, execute or inspect the resulting process state. If it should publish a record, query through an independent consumer. If it should leave unrelated files untouched, compare them byte-for-byte.

Weak: output contains “created report.txt”
Strong: report.txt exists, has exact expected bytes, and sibling.txt is unchanged

Agent/model output is untrusted evidence. Logs and exit codes are supporting evidence, not substitutes for world state when the world is observable.

5. Exercise built and non-index entries

Build the product using the release build. Invoke the built entry under the same runtime users receive, without source-only path maps, hot reloaders, or test transpilers.

Include non-index runtime siblings when they ship independently: workers, subprocess helpers, migration binaries, generated clients, or singleton-bearing modules used across bundles. Test missing configuration and load failures produce a non-zero exit or documented error rather than being swallowed.

This catches module-resolution, export-shape, duplicated-singleton, settle-race, and packaging omissions that source tests mask.

6. Own resources and teardown

Each test creates its own workspace, server, process, credentials fixture, and application instance. Register cleanup immediately in afterEach/finally, not after the first successful action. Await quiescent disposal so retries and failures do not leak processes or ports.

Shared helpers belong in non-test modules; importing one test file from another can register its suite twice.

7. Prove the regression guard goes red

For every regression test:

1. run it against the fixed code; observe green;
2. locally restore the exact defective condition or equivalent mutation;
3. run the focused test; observe the intended assertion fail for the intended reason;
4. reapply the fix;
5. run the focused test again; observe green;
6. run the broader relevant suite.

If the test remains green under the regression, it is not a guard. Strengthen its entry path or assertion. Do not commit the deliberate regression.

8. Layer evidence deliberately

Use the smallest set that covers distinct risks:

• unit test: branch and contract detail;
• real-composition test: loader, configuration, registration, and collaboration;
• built-artifact smoke: packaging and runtime entry;
• external-world assertion: actual effect;
• live-service smoke: provider compatibility when a key/service is available.

Line coverage proves execution, not shipped behavior. A real-service test does not replace deterministic keyless coverage; each answers a different question.

9. Handoff checklist

• actual published/runtime entry point is named;
• test crosses the real loader/composition path;
• only nondeterministic outer boundaries are mocked;
• assertion observes independent external state;
• built artifact is exercised where release changes can matter;
• failure/missing-config behavior is asserted;
• resources are disposed even on timeout or retry;
• deliberate regression makes the guard red;
• fix restored and focused plus broader checks are green.

Report the exact entry chain, mocked boundaries, external observations, and red-capability evidence.