feat: 10 new skills scouted from spec-kit and deepseek-harness
Sourced via Codex scan of github/spec-kit and deepseek-ai/deepseek-harness (scan reports in granja/_temp/codex-logs/), then authored by Codex against this repo's exact SKILL.md format/density, calibrated against bastille-jail-provisioning/writing-implementation-plans/tdd. Spot-checked two directly (write-feature-specification, harden-async-lifecycle-code) -- concrete, code-example-backed procedures, not generic advice. From spec-kit: write-feature-specification, clarify-feature-specification, audit-requirements-quality, analyze-spec-plan-task-consistency, converge-implementation-to-spec. From deepseek-harness: harden-async-lifecycle-code, test-real-entry-paths, snapshot-agent-behavior, maintain-decision-records, remove-reasoning-transcript-prose. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
---
|
||||
name: test-real-entry-paths
|
||||
description: Use when designing integration, end-to-end, release, or regression tests that must prove shipped behavior through the project's actual composition and published runtime entry point rather than a hand-assembled test path.
|
||||
---
|
||||
|
||||
# Test Real Entry Paths
|
||||
|
||||
A green unit suite can coexist with a completely broken product when tests bypass loading, composition, packaging, or the external world. Identify the path users actually execute for this project and test that path.
|
||||
|
||||
## 1. Name the shipped path before choosing a test
|
||||
|
||||
Ask:
|
||||
|
||||
```text
|
||||
What exact artifact and entry point does a user, editor, server, job runner, package consumer, or operating system invoke after release?
|
||||
```
|
||||
|
||||
Examples include a built CLI executable, published package export, application bootstrap, plugin/config loader, worker entry, installed extension, container command, HTTP route through production middleware, or migration binary. Do not assume source imports or a specific stack.
|
||||
|
||||
Write the chain:
|
||||
|
||||
```text
|
||||
published artifact → loader/configuration → composition/DI → public action → external state
|
||||
```
|
||||
|
||||
The test must traverse every link implicated by the change.
|
||||
|
||||
## 2. Identify bypasses in existing tests
|
||||
|
||||
Flag tests that:
|
||||
|
||||
- import an internal function instead of the published entry;
|
||||
- mount a component manually instead of using the shipping loader;
|
||||
- run source through a development transpiler while users execute built output;
|
||||
- replace real downstream tools with hand-written stand-ins;
|
||||
- assert only a returned message or agent self-report;
|
||||
- never exercise module exports, configuration resolution, registration, or process exit.
|
||||
|
||||
These tests may remain useful unit tests. They do not satisfy real-entry coverage.
|
||||
|
||||
## 3. Mock only true nondeterminism
|
||||
|
||||
Keep real:
|
||||
|
||||
- application composition and dependency registration;
|
||||
- loaders and configuration parsing;
|
||||
- the tool/executor under test;
|
||||
- persistence and serialization;
|
||||
- public transport and protocol assembly;
|
||||
- built artifact and process boundary where shipped.
|
||||
|
||||
Mock only a genuinely nondeterministic, costly, or unavailable boundary: LLM response, remote network, wall clock, random source, or third-party outage. Put the mock at that outer boundary, not between internal components.
|
||||
|
||||
Bad: replace the filesystem tool with `{ success: true }`.
|
||||
|
||||
Good: script the model’s tool call, run the real filesystem tool in a temporary workspace, then inspect the file externally.
|
||||
|
||||
## 4. Assert the world, not the actor’s story
|
||||
|
||||
If the action should create a file, read it from a separate handle. If it should run a command, execute or inspect the resulting process state. If it should publish a record, query through an independent consumer. If it should leave unrelated files untouched, compare them byte-for-byte.
|
||||
|
||||
```text
|
||||
Weak: output contains “created report.txt”
|
||||
Strong: report.txt exists, has exact expected bytes, and sibling.txt is unchanged
|
||||
```
|
||||
|
||||
Agent/model output is untrusted evidence. Logs and exit codes are supporting evidence, not substitutes for world state when the world is observable.
|
||||
|
||||
## 5. Exercise built and non-index entries
|
||||
|
||||
Build the product using the release build. Invoke the built entry under the same runtime users receive, without source-only path maps, hot reloaders, or test transpilers.
|
||||
|
||||
Include non-index runtime siblings when they ship independently: workers, subprocess helpers, migration binaries, generated clients, or singleton-bearing modules used across bundles. Test missing configuration and load failures produce a non-zero exit or documented error rather than being swallowed.
|
||||
|
||||
This catches module-resolution, export-shape, duplicated-singleton, settle-race, and packaging omissions that source tests mask.
|
||||
|
||||
## 6. Own resources and teardown
|
||||
|
||||
Each test creates its own workspace, server, process, credentials fixture, and application instance. Register cleanup immediately in `afterEach`/`finally`, not after the first successful action. Await quiescent disposal so retries and failures do not leak processes or ports.
|
||||
|
||||
Shared helpers belong in non-test modules; importing one test file from another can register its suite twice.
|
||||
|
||||
## 7. Prove the regression guard goes red
|
||||
|
||||
For every regression test:
|
||||
|
||||
1. run it against the fixed code; observe green;
|
||||
2. locally restore the exact defective condition or equivalent mutation;
|
||||
3. run the focused test; observe the intended assertion fail for the intended reason;
|
||||
4. reapply the fix;
|
||||
5. run the focused test again; observe green;
|
||||
6. run the broader relevant suite.
|
||||
|
||||
If the test remains green under the regression, it is not a guard. Strengthen its entry path or assertion. Do not commit the deliberate regression.
|
||||
|
||||
## 8. Layer evidence deliberately
|
||||
|
||||
Use the smallest set that covers distinct risks:
|
||||
|
||||
- unit test: branch and contract detail;
|
||||
- real-composition test: loader, configuration, registration, and collaboration;
|
||||
- built-artifact smoke: packaging and runtime entry;
|
||||
- external-world assertion: actual effect;
|
||||
- live-service smoke: provider compatibility when a key/service is available.
|
||||
|
||||
Line coverage proves execution, not shipped behavior. A real-service test does not replace deterministic keyless coverage; each answers a different question.
|
||||
|
||||
## 9. Handoff checklist
|
||||
|
||||
- actual published/runtime entry point is named;
|
||||
- test crosses the real loader/composition path;
|
||||
- only nondeterministic outer boundaries are mocked;
|
||||
- assertion observes independent external state;
|
||||
- built artifact is exercised where release changes can matter;
|
||||
- failure/missing-config behavior is asserted;
|
||||
- resources are disposed even on timeout or retry;
|
||||
- deliberate regression makes the guard red;
|
||||
- fix restored and focused plus broader checks are green.
|
||||
|
||||
Report the exact entry chain, mocked boundaries, external observations, and red-capability evidence.
|
||||
Reference in New Issue
Block a user