feat(ecc2): add bounded harness evaluation loop

This commit is contained in:
Affaan Mustafa
2026-08-05 15:43:42 -04:00
parent f1fec0e539
commit 23bca4f3b7
13 changed files with 928 additions and 32 deletions
+2 -2
View File
@@ -509,9 +509,9 @@ Kimi Code discovers the installed `.kimi/AGENTS.md` instructions and `.kimi/skil
### Itô compute CLI bridge
`ecc ito` delegates to the separately installed canonical Itô client; ECC does not maintain a second API client or browser handoff. The available operations are `ecc ito auth`, `ecc ito find`, `ecc ito status`, and the separately gated `ecc ito evals`. The matching MCP tools remain `ito_auth`, `ito_find`, and `ito_status`; node qualification is CLI-only.
`ecc ito` delegates to the separately installed canonical Itô client; ECC does not maintain a second API client. `ecc ito auth` delegates to the canonical CLI's device authorization, opens the Itô verification page by default, and persists a device token in macOS Keychain; pass `--no-browser` to suppress the page handoff. ECC itself does no browser automation. The available operations are `ecc ito auth`, `ecc ito find`, `ecc ito status`, and the separately gated `ecc ito evals`. The matching MCP tools remain `ito_auth`, `ito_find`, and `ito_status`; node qualification is CLI-only.
The `ito-compute-cli` package is currently unpublished. Build it locally from the Itô runtime repo (private while the desk hardens; design partners get access) under `cli/ito-compute-cli`, run `npm ci` and `npm run check`, then set `ECC_ITO_CLI_EXECUTABLE` to that build's absolute `dist/bin/ito.js` path. Inject `ITO_API_KEY` from 1Password or the launching environment. ECC does not discover this credential-bearing client through `PATH`. See the [`ito-compute` skill](skills/ito-compute/SKILL.md) for the full RFQ authority and MCP setup contract.
The `ito-compute-cli` package is currently unpublished. Build it locally from the Itô runtime repo (private while the desk hardens; design partners get access) under `cli/ito-compute-cli`, run `npm ci` and `npm run check`, then set `ECC_ITO_CLI_EXECUTABLE` to that build's absolute `dist/bin/ito.js` path. Device authorization is the default; keep legacy `ITO_API_KEY` only with explicit `ITO_AUTH_MODE=legacy`. ECC does not discover this credential-bearing client through `PATH`. See the [`ito-compute` skill](skills/ito-compute/SKILL.md) for the full RFQ authority and MCP setup contract.
`find` submits a live authenticated RFQ. It does not reserve capacity. `evals` requires both `ITO_ENABLE_SIXTYTWO_LIVE=1` and `--live-sixtytwo`, a separately installed `sixtytwo-cli==0.3.33`, an explicit node list, and an existing absolute configuration directory. It cannot rent, launch, recover, repair, or purchase. ECC exposes no quote lock, purchase, workload, or inference path, and it never replaces a missing client or failed live call with a local result.
</details>
+5 -4
View File
@@ -1,9 +1,10 @@
# Evaluator RAG Prototype
ECC 2.0 needs a self-improving harness loop that can learn from real work
without blindly mutating a user's Claude, Codex, OpenCode, dmux, Zed, or
terminal setup. This prototype defines the smallest read-only artifact set for
that loop.
ECC 2.0 needs an evidence-driven harness evaluation loop that can compare
operator-supplied candidates from real work without implying model learning or
blindly mutating a user's Claude, Codex, OpenCode, dmux, Zed, or terminal
setup. This prototype defines the smallest read-only artifact set for that
loop.
The fixture set lives in
[`examples/evaluator-rag-prototype/`](../../examples/evaluator-rag-prototype/).
+13 -7
View File
@@ -36,8 +36,11 @@ The canonical MCP server exposes only `ito_auth`, `ito_find`, and `ito_status`.
ECC includes an opt-in configuration template pointing to the local built MCP
entry. It does not enable the server by default.
The former browser/manual-copy command is retired. `ecc ito` performs no
browser navigation and stores no economic state.
The former browser/manual-copy command is retired. `ecc ito auth` delegates to
the canonical CLI's device authorization, which opens the Itô verification page
by default and persists a device token in macOS Keychain. `--no-browser`
suppresses that page handoff. ECC itself performs no browser automation and
stores no economic state.
## Local install
@@ -53,19 +56,22 @@ Set `ECC_ITO_CLI_EXECUTABLE` to the explicit absolute built entry:
/absolute/path/to/ito-cloud-runtime/cli/ito-compute-cli/dist/bin/ito.js
ECC does not resolve the credential-bearing client through `PATH`; this avoids
forwarding `ITO_API_KEY` to an unrelated executable with the same name.
forwarding authentication material to an unrelated executable with the same
name.
For MCP, configure `node` with:
/absolute/path/to/ito-cloud-runtime/cli/ito-compute-cli/dist/bin/ito-mcp.js
Inject `ITO_API_KEY` with 1Password or the launching environment. ECC forwards
only `ITO_API_KEY`, optional Itô endpoint overrides, and the minimum process
environment. It does not inspect or log the key.
Device authorization is the default. ECC forwards only the required device
authorization settings, optional Itô endpoint overrides, and the minimum
process environment. Legacy `ITO_API_KEY` is forwarded only with explicit
`ITO_AUTH_MODE=legacy`; ECC does not inspect or log it.
## Authority and economics
- `auth` validates the configured Itô API key.
- `auth` starts canonical device authorization, with `--no-browser` available
when the operator does not want the CLI to open the verification page.
- `find` reads live inventory and submits a live authenticated RFQ. An operator
or agent must gather every hard topology/economic constraint and obtain
explicit buyer authority before invoking it.
+15
View File
@@ -70,6 +70,21 @@ cargo run -- resume <session-id>
cargo run -- daemon
```
## Bounded Harness Evaluation
ECC2 now has an operator-driven configuration registry and promotion gate. Candidate JSON is canonicalized and addressed by its SHA-256 digest, with immutable trace/evidence references. Evaluation uses the same explicit unique seeds for candidate and active baseline through a pluggable Rust trait. The CLI exposes only a deterministic local recorded-measurements evaluator; it makes no network or process calls.
```bash
cargo run -- harness-eval record --config candidate.json --trace-ref trace://run-1 --evidence-ref evidence://review-1
cargo run -- harness-eval activate-initial <sha256> --evidence-ref evidence://baseline-approval
cargo run -- harness-eval run --candidate <sha256> --baseline <sha256> --seed 1 --seed 2 --measurements measurements.json --evidence-ref evidence://evaluation-1 --min-samples 2 --min-mean-delta 0.05 --min-win-rate 0.5
cargo run -- harness-eval audit
```
`measurements.json` contains `{"evaluator":"recorded-v1","scores":{"<candidate>":{"1":0.9},"<baseline>":{"1":0.7}},"health":{"<candidate>":true}}` (with every requested seed present). Promotion requires minimum paired samples, arithmetic-mean delta, and per-seed win rate. SQLite transactions update the active pointer and append audit evidence atomically; a failed or errored candidate-keyed recorded health assertion restores the prior pointer and records rollback evidence. Database triggers reject update/deletion of candidate, evaluation, and audit rows.
Limitations: this performs one bounded deterministic comparison. It does not autonomously rewrite prompts or `ecc2.toml`, train/fine-tune a model, implement or claim reinforcement learning, call a network service, or run shell-command evaluators. It does not alter running sessions. Evidence references and scores are operator assertions, not authenticated truth. Arithmetic gates do not establish statistical significance. The active pointer is registry state only; it is not automatic deployment into a harness runtime.
## Validate
```bash
+402
View File
@@ -0,0 +1,402 @@
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
use std::collections::BTreeMap;
#[test]
fn candidate_id_is_content_addressed_over_canonical_json() {
let first = CandidateSpec::new(
json!({"model": "fixed", "limits": {"steps": 3, "tools": ["read"]}}),
vec!["trace://one".into()],
vec!["evidence://one".into()],
)
.unwrap();
let second = CandidateSpec::new(
json!({"limits": {"tools": ["read"], "steps": 3}, "model": "fixed"}),
vec!["trace://two".into()],
vec!["evidence://two".into()],
)
.unwrap();
assert_eq!(first.id, second.id);
assert_eq!(first.canonical_config, second.canonical_config);
}
#[test]
fn policy_requires_explicit_unique_seeds_and_minimum_samples() {
let policy = PromotionPolicy {
min_samples: 3,
min_mean_delta: 0.05,
min_win_rate: 2.0 / 3.0,
};
let duplicate = vec![
paired(7, 1.0, 0.0),
paired(7, 1.0, 0.0),
paired(9, 1.0, 0.0),
];
assert!(policy.compare(&duplicate).is_err());
let too_few = vec![paired(7, 1.0, 0.0), paired(8, 1.0, 0.0)];
let decision = policy.compare(&too_few).unwrap();
assert!(!decision.passed);
assert!(decision
.failures
.iter()
.any(|failure| failure.contains("minimum sample")));
}
#[test]
fn thresholds_are_deterministic_and_all_must_pass() {
let policy = PromotionPolicy {
min_samples: 3,
min_mean_delta: 0.1,
min_win_rate: 0.75,
};
let samples = vec![
paired(1, 0.9, 0.7),
paired(2, 0.8, 0.7),
paired(3, 0.6, 0.7),
paired(4, 0.8, 0.7),
];
let first = policy.compare(&samples).unwrap();
let second = policy.compare(&samples).unwrap();
assert_eq!(first, second);
assert!(!first.passed);
assert_eq!(first.win_rate, 0.75);
assert!(first
.failures
.iter()
.any(|failure| failure.contains("mean delta")));
}
#[test]
fn evaluator_is_called_for_each_explicit_seed_in_order() {
let mut evaluator = RecordedEvaluator::new(
BTreeMap::from([
(("candidate".into(), 4), 0.9),
(("baseline".into(), 4), 0.5),
(("candidate".into(), 2), 0.8),
(("baseline".into(), 2), 0.6),
]),
true,
);
let samples = evaluate_paired(&mut evaluator, "candidate", "baseline", &[4, 2]).unwrap();
assert_eq!(samples, vec![paired(4, 0.9, 0.5), paired(2, 0.8, 0.6)]);
assert_eq!(
evaluator.calls(),
&[
("candidate".into(), 4),
("baseline".into(), 4),
("candidate".into(), 2),
("baseline".into(), 2)
]
);
}
fn paired(seed: u64, candidate_score: f64, baseline_score: f64) -> PairedSample {
PairedSample {
seed,
candidate_score,
baseline_score,
}
}
}
use anyhow::{bail, Context, Result};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, BTreeSet};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct CandidateSpec {
pub id: String,
pub canonical_config: String,
pub trace_refs: Vec<String>,
pub evidence_refs: Vec<String>,
}
impl CandidateSpec {
pub fn new(config: Value, trace_refs: Vec<String>, evidence_refs: Vec<String>) -> Result<Self> {
validate_refs("trace", &trace_refs)?;
validate_refs("evidence", &evidence_refs)?;
let canonical_config = serde_json::to_string(&canonicalize(config))?;
if canonical_config.len() > 1024 * 1024 {
bail!("candidate configuration exceeds 1 MiB");
}
let digest = Sha256::digest(canonical_config.as_bytes());
let id = digest.iter().map(|byte| format!("{byte:02x}")).collect();
Ok(Self {
id,
canonical_config,
trace_refs,
evidence_refs,
})
}
pub fn verify_integrity(&self) -> Result<()> {
let value: Value = serde_json::from_str(&self.canonical_config)?;
let rebuilt = Self::new(value, self.trace_refs.clone(), self.evidence_refs.clone())?;
if rebuilt.id != self.id || rebuilt.canonical_config != self.canonical_config {
bail!("candidate content address or canonical configuration is invalid");
}
Ok(())
}
}
fn validate_refs(kind: &str, refs: &[String]) -> Result<()> {
if refs.is_empty() || refs.iter().any(|reference| reference.trim().is_empty()) {
bail!("at least one non-empty {kind} reference is required");
}
if refs.len() > 100 || refs.iter().any(|reference| reference.len() > 4096) {
bail!("{kind} references exceed bounded limits");
}
Ok(())
}
fn canonicalize(value: Value) -> Value {
match value {
Value::Object(entries) => Value::Object(
entries
.into_iter()
.map(|(key, value)| (key, canonicalize(value)))
.collect::<BTreeMap<_, _>>()
.into_iter()
.collect(),
),
Value::Array(values) => Value::Array(values.into_iter().map(canonicalize).collect()),
other => other,
}
}
pub trait Evaluator {
fn name(&self) -> &str;
fn evaluate(&mut self, candidate_id: &str, seed: u64) -> Result<f64>;
fn health_check(&mut self, candidate_id: &str) -> Result<bool>;
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RecordedEvidence {
pub evaluator: String,
pub scores: BTreeMap<String, BTreeMap<u64, f64>>,
pub health: BTreeMap<String, bool>,
}
pub struct RecordedEvaluator {
name: String,
scores: BTreeMap<(String, u64), f64>,
health_ok: bool,
health_candidate: Option<String>,
calls: Vec<(String, u64)>,
}
impl RecordedEvaluator {
#[cfg(test)]
pub fn new(scores: BTreeMap<(String, u64), f64>, health_ok: bool) -> Self {
Self {
name: "recorded-v1".into(),
scores,
health_ok,
health_candidate: None,
calls: Vec::new(),
}
}
pub fn from_evidence(evidence: RecordedEvidence) -> Result<Self> {
if evidence.evaluator != "recorded-v1" {
bail!("CLI evidence evaluator must be recorded-v1");
}
let score_count = evidence.scores.values().map(BTreeMap::len).sum::<usize>();
if score_count > 20_000 || evidence.scores.keys().any(|id| id.len() != 64) {
bail!("recorded evidence exceeds bounded score or candidate limits");
}
if evidence.health.len() != 1 {
bail!("exactly one candidate-keyed health assertion is required");
}
let (health_candidate, health_ok) = evidence
.health
.into_iter()
.next()
.context("candidate-keyed health evidence is required")?;
let scores = evidence
.scores
.into_iter()
.flat_map(|(id, values)| {
values
.into_iter()
.map(move |(seed, score)| ((id.clone(), seed), score))
})
.collect();
Ok(Self {
name: evidence.evaluator,
scores,
health_ok,
health_candidate: Some(health_candidate),
calls: Vec::new(),
})
}
#[cfg(test)]
pub fn calls(&self) -> &[(String, u64)] {
&self.calls
}
}
impl Evaluator for RecordedEvaluator {
fn name(&self) -> &str {
&self.name
}
fn evaluate(&mut self, candidate_id: &str, seed: u64) -> Result<f64> {
self.calls.push((candidate_id.to_string(), seed));
let score = *self
.scores
.get(&(candidate_id.to_string(), seed))
.with_context(|| format!("missing recorded score for {candidate_id} seed {seed}"))?;
if !score.is_finite() || !(0.0..=1.0).contains(&score) {
bail!("score must be finite and between 0 and 1");
}
Ok(score)
}
fn health_check(&mut self, candidate_id: &str) -> Result<bool> {
if self
.health_candidate
.as_deref()
.is_some_and(|expected| expected != candidate_id)
{
bail!("health evidence does not match promoted candidate");
}
Ok(self.health_ok)
}
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct PairedSample {
pub seed: u64,
pub candidate_score: f64,
pub baseline_score: f64,
}
pub fn evaluate_paired(
evaluator: &mut dyn Evaluator,
candidate_id: &str,
baseline_id: &str,
seeds: &[u64],
) -> Result<Vec<PairedSample>> {
if seeds.is_empty() {
bail!("at least one explicit seed is required");
}
if seeds.len() > 10_000 {
bail!("seed count exceeds 10000");
}
if seeds.iter().copied().collect::<BTreeSet<_>>().len() != seeds.len() {
bail!("seeds must be unique");
}
seeds
.iter()
.map(|seed| {
Ok(PairedSample {
seed: *seed,
candidate_score: evaluator.evaluate(candidate_id, *seed)?,
baseline_score: evaluator.evaluate(baseline_id, *seed)?,
})
})
.collect()
}
#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
pub struct PromotionPolicy {
pub min_samples: usize,
pub min_mean_delta: f64,
pub min_win_rate: f64,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Comparison {
pub passed: bool,
pub sample_count: usize,
pub candidate_mean: f64,
pub baseline_mean: f64,
pub mean_delta: f64,
pub win_rate: f64,
pub failures: Vec<String>,
}
impl PromotionPolicy {
pub fn validate(self) -> Result<()> {
if self.min_samples == 0 {
bail!("minimum samples must be positive");
}
if !self.min_mean_delta.is_finite() {
bail!("minimum mean delta must be finite");
}
if !self.min_win_rate.is_finite() || !(0.0..=1.0).contains(&self.min_win_rate) {
bail!("minimum win rate must be between 0 and 1");
}
Ok(())
}
pub fn compare(self, samples: &[PairedSample]) -> Result<Comparison> {
self.validate()?;
if samples.is_empty() {
bail!("samples cannot be empty");
}
if samples
.iter()
.map(|sample| sample.seed)
.collect::<BTreeSet<_>>()
.len()
!= samples.len()
{
bail!("sample seeds must be unique");
}
if samples.iter().any(|s| {
!s.candidate_score.is_finite()
|| !s.baseline_score.is_finite()
|| !(0.0..=1.0).contains(&s.candidate_score)
|| !(0.0..=1.0).contains(&s.baseline_score)
}) {
bail!("scores must be finite and between 0 and 1");
}
let count = samples.len();
let candidate_mean = samples.iter().map(|s| s.candidate_score).sum::<f64>() / count as f64;
let baseline_mean = samples.iter().map(|s| s.baseline_score).sum::<f64>() / count as f64;
let mean_delta = candidate_mean - baseline_mean;
let win_rate = samples
.iter()
.filter(|s| s.candidate_score > s.baseline_score)
.count() as f64
/ count as f64;
let mut failures = Vec::new();
if count < self.min_samples {
failures.push(format!(
"minimum sample count is {}, got {count}",
self.min_samples
));
}
if mean_delta < self.min_mean_delta {
failures.push(format!(
"mean delta {mean_delta:.6} is below {:.6}",
self.min_mean_delta
));
}
if win_rate < self.min_win_rate {
failures.push(format!(
"win rate {win_rate:.6} is below {:.6}",
self.min_win_rate
));
}
Ok(Comparison {
passed: failures.is_empty(),
sample_count: count,
candidate_mean,
baseline_mean,
mean_delta,
win_rate,
failures,
})
}
}
+149
View File
@@ -1,5 +1,6 @@
mod comms;
mod config;
mod harness_eval;
mod notifications;
mod observability;
mod session;
@@ -108,6 +109,11 @@ impl OptionalWorktreePolicyArgs {
#[derive(clap::Subcommand, Debug)]
enum Commands {
/// Run bounded, deterministic harness configuration evaluations
HarnessEval {
#[command(subcommand)]
command: HarnessEvalCommands,
},
/// Launch the TUI dashboard
Dashboard,
/// Start a new agent session
@@ -437,6 +443,34 @@ enum Commands {
},
}
#[derive(clap::Subcommand, Debug)]
enum HarnessEvalCommands {
/// Record an immutable content-addressed candidate from a local JSON file
Record {
#[arg(long)] config: PathBuf,
#[arg(long = "trace-ref", required = true)] trace_refs: Vec<String>,
#[arg(long = "evidence-ref", required = true)] evidence_refs: Vec<String>,
},
/// Set the first baseline; subsequent changes require evaluation
ActivateInitial {
candidate_id: String,
#[arg(long)] evidence_ref: String,
},
/// Evaluate paired scores and conditionally promote with a health gate
Run {
#[arg(long)] candidate: String,
#[arg(long)] baseline: String,
#[arg(long = "seed", required = true)] seeds: Vec<u64>,
#[arg(long)] measurements: PathBuf,
#[arg(long)] evidence_ref: String,
#[arg(long)] min_samples: usize,
#[arg(long)] min_mean_delta: f64,
#[arg(long)] min_win_rate: f64,
},
/// Show append-only promotion audit entries
Audit,
}
#[derive(clap::Subcommand, Debug)]
enum MessageCommands {
/// Send a structured message between sessions
@@ -1345,6 +1379,37 @@ struct DotenvMemoryEntry {
details: BTreeMap<String, String>,
}
fn read_bounded_file(path: &Path, max_bytes: u64, label: &str) -> Result<Vec<u8>> {
let mut options = File::options();
options.read(true);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
options.custom_flags(libc::O_NONBLOCK);
}
let file = options
.open(path)
.with_context(|| format!("Failed to open {}", path.display()))?;
let metadata = file
.metadata()
.with_context(|| format!("Failed to inspect {}", path.display()))?;
if !metadata.is_file() {
anyhow::bail!("{label} must be a regular file");
}
let read_limit = max_bytes
.checked_add(1)
.context("bounded input byte limit is too large")?;
let mut content = Vec::new();
file.take(read_limit)
.read_to_end(&mut content)
.with_context(|| format!("Failed to read {}", path.display()))?;
if content.len() as u64 > max_bytes {
anyhow::bail!("{label} exceeds the {max_bytes}-byte limit");
}
Ok(content)
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt()
@@ -1357,6 +1422,33 @@ async fn main() -> Result<()> {
let db = session::store::StateStore::open(&cfg.db_path)?;
match cli.command {
Some(Commands::HarnessEval { command }) => match command {
HarnessEvalCommands::Record { config, trace_refs, evidence_refs } => {
let value: serde_json::Value = serde_json::from_slice(&read_bounded_file(&config, 1_048_576, "candidate configuration")?)
.with_context(|| format!("Invalid JSON in {}", config.display()))?;
let candidate = harness_eval::CandidateSpec::new(value, trace_refs, evidence_refs)?;
db.record_harness_candidate(&candidate)?;
println!("{}", candidate.id);
}
HarnessEvalCommands::ActivateInitial { candidate_id, evidence_ref } => {
db.activate_initial_harness(&candidate_id, &evidence_ref)?;
println!("Activated initial baseline: {candidate_id}");
}
HarnessEvalCommands::Run { candidate, baseline, seeds, measurements, evidence_ref, min_samples, min_mean_delta, min_win_rate } => {
use harness_eval::Evaluator;
let evidence: harness_eval::RecordedEvidence = serde_json::from_slice(&read_bounded_file(&measurements, 8_388_608, "recorded measurements")?)
.with_context(|| format!("Invalid recorded evidence in {}", measurements.display()))?;
let mut evaluator = harness_eval::RecordedEvaluator::from_evidence(evidence)?;
let evaluator_name = evaluator.name().to_string();
let samples = harness_eval::evaluate_paired(&mut evaluator, &candidate, &baseline, &seeds)?;
let policy = harness_eval::PromotionPolicy { min_samples, min_mean_delta, min_win_rate };
let outcome = db.evaluate_promote_and_health_check(&candidate, &baseline, &evaluator_name, &samples, policy, &evidence_ref, |id| evaluator.health_check(id))?;
println!("{}", serde_json::to_string_pretty(&outcome)?);
}
HarnessEvalCommands::Audit => {
println!("{}", serde_json::to_string_pretty(&db.harness_audit_entries()?)?);
}
},
Some(Commands::Dashboard) | None => {
tui::app::run(db, cfg).await?;
}
@@ -8533,6 +8625,63 @@ mod tests {
assert!(!policy.resolve(&cfg));
}
#[test]
fn harness_eval_cli_requires_explicit_bounded_inputs() {
let cli = Cli::try_parse_from([
"ecc", "harness-eval", "run", "--candidate", "candidate", "--baseline", "baseline",
"--seed", "1", "--seed", "2", "--measurements", "scores.json", "--evidence-ref",
"evidence://run", "--min-samples", "2", "--min-mean-delta", "0.1", "--min-win-rate", "0.5",
]).expect("valid harness evaluation command");
match cli.command {
Some(Commands::HarnessEval { command: HarnessEvalCommands::Run { seeds, min_samples, .. } }) => {
assert_eq!(seeds, vec![1, 2]);
assert_eq!(min_samples, 2);
}
other => panic!("unexpected command: {other:?}"),
}
assert!(Cli::try_parse_from(["ecc", "harness-eval", "run", "--candidate", "c", "--baseline", "b"]).is_err());
}
#[test]
fn harness_eval_bounded_input_rejects_content_over_limit() -> Result<()> {
let tempdir = TestDir::new("harness-eval-oversized-input")?;
let input = tempdir.path().join("measurements.json");
fs::write(&input, b"12345")?;
let error = read_bounded_file(&input, 4, "recorded measurements")
.expect_err("input larger than the byte limit must fail");
assert_eq!(
error.to_string(),
"recorded measurements exceeds the 4-byte limit"
);
Ok(())
}
#[cfg(unix)]
#[test]
fn harness_eval_bounded_input_rejects_non_regular_file() -> Result<()> {
use std::ffi::CString;
use std::os::unix::ffi::OsStrExt;
let tempdir = TestDir::new("harness-eval-non-regular-input")?;
let input = tempdir.path().join("measurements.fifo");
let input_c = CString::new(input.as_os_str().as_bytes())?;
// SAFETY: `input_c` is a valid, NUL-terminated path and the mode is valid.
let result = unsafe { libc::mkfifo(input_c.as_ptr(), 0o600) };
if result != 0 {
return Err(std::io::Error::last_os_error().into());
}
let error = read_bounded_file(&input, 4, "recorded measurements")
.expect_err("non-regular input must fail");
assert_eq!(
error.to_string(),
"recorded measurements must be a regular file"
);
Ok(())
}
#[test]
fn worktree_policy_explicit_flags_override_config_setting() {
let mut cfg = Config::default();
+245
View File
@@ -10,6 +10,7 @@ use std::time::Duration;
use crate::comms;
use crate::config::Config;
use crate::harness_eval::{CandidateSpec, PairedSample, PromotionPolicy};
use crate::observability::{ToolCallEvent, ToolLogEntry, ToolLogPage};
use super::output::{OutputLine, OutputStream, OUTPUT_BUFFER_LIMIT};
@@ -27,6 +28,25 @@ pub struct StateStore {
conn: Connection,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct HarnessAuditEntry {
pub id: i64,
pub event_type: String,
pub candidate_id: String,
pub prior_candidate_id: Option<String>,
pub evaluation_id: Option<i64>,
pub evidence_ref: String,
pub created_at: String,
}
#[derive(Debug, Clone, PartialEq, Serialize)]
pub struct HarnessPromotionOutcome {
pub evaluation_id: Option<i64>,
pub promoted: bool,
pub rolled_back: bool,
pub failures: Vec<String>,
}
const DEFAULT_CONTEXT_GRAPH_OBSERVATION_RETENTION: usize = 12;
#[derive(Debug, Clone)]
@@ -403,6 +423,45 @@ impl StateStore {
last_auto_prune_active_skipped INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS harness_candidates (
id TEXT PRIMARY KEY,
canonical_config_json TEXT NOT NULL,
trace_refs_json TEXT NOT NULL,
evidence_refs_json TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS harness_evaluations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
candidate_id TEXT NOT NULL REFERENCES harness_candidates(id),
baseline_id TEXT NOT NULL REFERENCES harness_candidates(id),
evaluator TEXT NOT NULL,
samples_json TEXT NOT NULL,
policy_json TEXT NOT NULL,
comparison_json TEXT NOT NULL,
evidence_ref TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS active_harness_config (
slot TEXT PRIMARY KEY CHECK(slot = 'default'),
candidate_id TEXT NOT NULL REFERENCES harness_candidates(id),
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS harness_eval_audit (
id INTEGER PRIMARY KEY AUTOINCREMENT,
event_type TEXT NOT NULL,
candidate_id TEXT NOT NULL REFERENCES harness_candidates(id),
prior_candidate_id TEXT REFERENCES harness_candidates(id),
evaluation_id INTEGER REFERENCES harness_evaluations(id),
evidence_ref TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE TRIGGER IF NOT EXISTS harness_candidates_no_update BEFORE UPDATE ON harness_candidates BEGIN SELECT RAISE(ABORT, 'harness candidates are immutable'); END;
CREATE TRIGGER IF NOT EXISTS harness_candidates_no_delete BEFORE DELETE ON harness_candidates BEGIN SELECT RAISE(ABORT, 'harness candidates are immutable'); END;
CREATE TRIGGER IF NOT EXISTS harness_evaluations_no_update BEFORE UPDATE ON harness_evaluations BEGIN SELECT RAISE(ABORT, 'harness evaluations are immutable'); END;
CREATE TRIGGER IF NOT EXISTS harness_evaluations_no_delete BEFORE DELETE ON harness_evaluations BEGIN SELECT RAISE(ABORT, 'harness evaluations are immutable'); END;
CREATE TRIGGER IF NOT EXISTS harness_eval_audit_no_update BEFORE UPDATE ON harness_eval_audit BEGIN SELECT RAISE(ABORT, 'harness audit is immutable'); END;
CREATE TRIGGER IF NOT EXISTS harness_eval_audit_no_delete BEFORE DELETE ON harness_eval_audit BEGIN SELECT RAISE(ABORT, 'harness audit is immutable'); END;
CREATE INDEX IF NOT EXISTS idx_sessions_state ON sessions(state);
CREATE INDEX IF NOT EXISTS idx_tool_log_session ON tool_log(session_id);
CREATE INDEX IF NOT EXISTS idx_messages_to ON messages(to_session, read);
@@ -5067,6 +5126,92 @@ fn overlap_state_priority(state: &SessionState) -> u8 {
}
}
impl StateStore {
pub fn record_harness_candidate(&self, candidate: &CandidateSpec) -> Result<()> {
candidate.verify_integrity()?;
let existing: Option<(String, String, String)> = self.conn.query_row(
"SELECT canonical_config_json, trace_refs_json, evidence_refs_json FROM harness_candidates WHERE id = ?1",
[&candidate.id],
|row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)),
).optional()?;
let trace_json = serde_json::to_string(&candidate.trace_refs)?;
let evidence_json = serde_json::to_string(&candidate.evidence_refs)?;
if let Some(existing) = existing {
if existing != (candidate.canonical_config.clone(), trace_json, evidence_json) {
anyhow::bail!("candidate id already exists with different immutable content or references");
}
return Ok(());
}
self.conn.execute(
"INSERT INTO harness_candidates (id, canonical_config_json, trace_refs_json, evidence_refs_json, created_at) VALUES (?1, ?2, ?3, ?4, ?5)",
rusqlite::params![candidate.id, candidate.canonical_config, trace_json, evidence_json, chrono::Utc::now().to_rfc3339()],
)?;
Ok(())
}
pub fn activate_initial_harness(&self, candidate_id: &str, evidence_ref: &str) -> Result<()> {
if candidate_id.len() != 64 || evidence_ref.trim().is_empty() || evidence_ref.len() > 4096 { anyhow::bail!("valid candidate id and bounded activation evidence reference are required"); }
let tx = self.conn.unchecked_transaction()?;
if tx.query_row("SELECT candidate_id FROM active_harness_config WHERE slot = 'default'", [], |row| row.get::<_, String>(0)).optional()?.is_some() {
anyhow::bail!("an active harness configuration already exists");
}
let now = chrono::Utc::now().to_rfc3339();
tx.execute("INSERT INTO active_harness_config (slot, candidate_id, updated_at) VALUES ('default', ?1, ?2)", rusqlite::params![candidate_id, now])?;
tx.execute("INSERT INTO harness_eval_audit (event_type, candidate_id, evidence_ref, created_at) VALUES ('initial_activation', ?1, ?2, ?3)", rusqlite::params![candidate_id, evidence_ref, now])?;
tx.commit()?;
Ok(())
}
#[cfg(test)]
pub fn active_harness_id(&self) -> Result<Option<String>> {
Ok(self.conn.query_row("SELECT candidate_id FROM active_harness_config WHERE slot = 'default'", [], |row| row.get(0)).optional()?)
}
#[allow(clippy::too_many_arguments)]
pub fn evaluate_promote_and_health_check<F>(&self, candidate_id: &str, baseline_id: &str, evaluator: &str, samples: &[PairedSample], policy: PromotionPolicy, evidence_ref: &str, health_check: F) -> Result<HarnessPromotionOutcome>
where F: FnOnce(&str) -> Result<bool> {
if candidate_id.len() != 64 || baseline_id.len() != 64 || evaluator != "recorded-v1" || evidence_ref.trim().is_empty() || evidence_ref.len() > 4096 { anyhow::bail!("valid candidate ids, recorded-v1 evaluator, and bounded evidence reference are required"); }
let comparison = policy.compare(samples)?;
let tx = self.conn.unchecked_transaction()?;
let active: String = tx.query_row("SELECT candidate_id FROM active_harness_config WHERE slot = 'default'", [], |row| row.get(0)).context("no active baseline configuration")?;
if active != baseline_id { anyhow::bail!("baseline is not the active harness configuration"); }
let now = chrono::Utc::now().to_rfc3339();
tx.execute("INSERT INTO harness_evaluations (candidate_id, baseline_id, evaluator, samples_json, policy_json, comparison_json, evidence_ref, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)", rusqlite::params![candidate_id, baseline_id, evaluator, serde_json::to_string(samples)?, serde_json::to_string(&policy)?, serde_json::to_string(&comparison)?, evidence_ref, now])?;
let evaluation_id = tx.last_insert_rowid();
if !comparison.passed {
tx.execute("INSERT INTO harness_eval_audit (event_type, candidate_id, prior_candidate_id, evaluation_id, evidence_ref, created_at) VALUES ('promotion_rejected', ?1, ?2, ?3, ?4, ?5)", rusqlite::params![candidate_id, baseline_id, evaluation_id, evidence_ref, now])?;
tx.commit()?;
return Ok(HarnessPromotionOutcome { evaluation_id: Some(evaluation_id), promoted: false, rolled_back: false, failures: comparison.failures });
}
let changed = tx.execute("UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3", rusqlite::params![candidate_id, now, baseline_id])?;
if changed != 1 { anyhow::bail!("atomic promotion compare-and-swap failed"); }
let health_result = health_check(candidate_id);
let healthy = matches!(health_result, Ok(true));
let event_type = match &health_result { Ok(true) => "promoted", Ok(false) => "promotion_rolled_back", Err(_) => "health_check_error_rolled_back" };
if !healthy {
let restored = tx.execute("UPDATE active_harness_config SET candidate_id = ?1, updated_at = ?2 WHERE slot = 'default' AND candidate_id = ?3", rusqlite::params![baseline_id, now, candidate_id])?;
if restored != 1 { anyhow::bail!("atomic rollback compare-and-swap failed"); }
}
tx.execute("INSERT INTO harness_eval_audit (event_type, candidate_id, prior_candidate_id, evaluation_id, evidence_ref, created_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6)", rusqlite::params![event_type, candidate_id, baseline_id, evaluation_id, evidence_ref, now])?;
tx.commit()?;
let failures = match health_result {
Ok(true) => Vec::new(),
Ok(false) => vec!["post-promotion health check returned false".to_string()],
Err(error) => vec![format!("health check error: {error:#}")],
};
Ok(HarnessPromotionOutcome { evaluation_id: Some(evaluation_id), promoted: healthy, rolled_back: !healthy, failures })
}
pub fn harness_audit_entries(&self) -> Result<Vec<HarnessAuditEntry>> {
let mut statement = self.conn.prepare("SELECT id, event_type, candidate_id, prior_candidate_id, evaluation_id, evidence_ref, created_at FROM harness_eval_audit ORDER BY id")?;
let entries = statement.query_map([], |row| Ok(HarnessAuditEntry { id: row.get(0)?, event_type: row.get(1)?, candidate_id: row.get(2)?, prior_candidate_id: row.get(3)?, evaluation_id: row.get(4)?, evidence_ref: row.get(5)?, created_at: row.get(6)? }))?.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(entries)
}
#[cfg(test)]
fn connection_for_test(&self) -> &Connection { &self.conn }
}
#[cfg(test)]
mod tests {
use super::*;
@@ -7110,4 +7255,104 @@ mod tests {
Ok(())
}
#[test]
fn harness_eval_store_promotes_and_rolls_back_with_immutable_audit() -> Result<()> {
use crate::harness_eval::{CandidateSpec, PairedSample, PromotionPolicy};
use serde_json::json;
let tempdir = TestDir::new("store-harness-eval")?;
let db = StateStore::open(&tempdir.path().join("state.db"))?;
let baseline = CandidateSpec::new(json!({"prompt": "baseline"}), vec!["trace://b".into()], vec!["evidence://b".into()])?;
let candidate = CandidateSpec::new(json!({"prompt": "candidate"}), vec!["trace://c".into()], vec!["evidence://c".into()])?;
db.record_harness_candidate(&baseline)?;
db.record_harness_candidate(&candidate)?;
db.activate_initial_harness(&baseline.id, "evidence://bootstrap")?;
let samples = vec![
PairedSample { seed: 1, candidate_score: 0.9, baseline_score: 0.5 },
PairedSample { seed: 2, candidate_score: 0.8, baseline_score: 0.5 },
];
let policy = PromotionPolicy { min_samples: 2, min_mean_delta: 0.1, min_win_rate: 1.0 };
let outcome = db.evaluate_promote_and_health_check(&candidate.id, &baseline.id, "recorded-v1", &samples, policy, "evidence://run", |_| Ok(false))?;
assert!(outcome.rolled_back);
assert_eq!(
outcome.failures,
vec!["post-promotion health check returned false"]
);
assert_eq!(db.active_harness_id()?.as_deref(), Some(baseline.id.as_str()));
let audit = db.harness_audit_entries()?;
assert_eq!(audit.iter().map(|entry| entry.event_type.as_str()).collect::<Vec<_>>(), vec!["initial_activation", "promotion_rolled_back"]);
assert!(db.connection_for_test().execute("UPDATE harness_eval_audit SET event_type = 'tampered'", []).is_err());
assert!(db.connection_for_test().execute("DELETE FROM harness_candidates", []).is_err());
Ok(())
}
#[test]
fn harness_eval_health_callback_error_is_reported_and_rolled_back() -> Result<()> {
use crate::harness_eval::{CandidateSpec, PairedSample, PromotionPolicy};
use serde_json::json;
let tempdir = TestDir::new("store-harness-health-error")?;
let db = StateStore::open(&tempdir.path().join("state.db"))?;
let baseline = CandidateSpec::new(json!({"v": 1}), vec!["trace://b".into()], vec!["evidence://b".into()])?;
let candidate = CandidateSpec::new(json!({"v": 2}), vec!["trace://c".into()], vec!["evidence://c".into()])?;
db.record_harness_candidate(&baseline)?;
db.record_harness_candidate(&candidate)?;
db.activate_initial_harness(&baseline.id, "evidence://bootstrap")?;
let outcome = db.evaluate_promote_and_health_check(
&candidate.id,
&baseline.id,
"recorded-v1",
&[PairedSample { seed: 1, candidate_score: 0.9, baseline_score: 0.5 }],
PromotionPolicy { min_samples: 1, min_mean_delta: 0.4, min_win_rate: 1.0 },
"evidence://run",
|_| anyhow::bail!("probe unavailable"),
)?;
assert!(outcome.rolled_back);
assert_eq!(outcome.failures, vec!["health check error: probe unavailable"]);
assert_eq!(db.active_harness_id()?.as_deref(), Some(baseline.id.as_str()));
assert_eq!(db.harness_audit_entries()?.last().unwrap().event_type, "health_check_error_rolled_back");
Ok(())
}
#[test]
fn harness_eval_failed_gate_never_changes_active_configuration() -> Result<()> {
use crate::harness_eval::{CandidateSpec, PairedSample, PromotionPolicy};
use serde_json::json;
let tempdir = TestDir::new("store-harness-gate")?;
let db = StateStore::open(&tempdir.path().join("state.db"))?;
let baseline = CandidateSpec::new(json!({"v": 1}), vec!["trace://b".into()], vec!["evidence://b".into()])?;
let candidate = CandidateSpec::new(json!({"v": 2}), vec!["trace://c".into()], vec!["evidence://c".into()])?;
db.record_harness_candidate(&baseline)?;
db.record_harness_candidate(&candidate)?;
db.activate_initial_harness(&baseline.id, "evidence://bootstrap")?;
let outcome = db.evaluate_promote_and_health_check(&candidate.id, &baseline.id, "recorded-v1", &[PairedSample { seed: 1, candidate_score: 0.6, baseline_score: 0.5 }], PromotionPolicy { min_samples: 2, min_mean_delta: 0.0, min_win_rate: 0.0 }, "evidence://run", |_| Ok(true))?;
assert!(!outcome.promoted);
assert_eq!(db.active_harness_id()?.as_deref(), Some(baseline.id.as_str()));
assert_eq!(db.harness_audit_entries()?.last().unwrap().event_type, "promotion_rejected");
Ok(())
}
#[test]
fn harness_eval_successful_promotion_is_persisted() -> Result<()> {
use crate::harness_eval::{CandidateSpec, PairedSample, PromotionPolicy};
use serde_json::json;
let tempdir = TestDir::new("store-harness-success")?;
let db_path = tempdir.path().join("state.db");
let db = StateStore::open(&db_path)?;
let baseline = CandidateSpec::new(json!({"v": 1}), vec!["trace://b".into()], vec!["evidence://b".into()])?;
let candidate = CandidateSpec::new(json!({"v": 2}), vec!["trace://c".into()], vec!["evidence://c".into()])?;
db.record_harness_candidate(&baseline)?;
db.record_harness_candidate(&candidate)?;
db.activate_initial_harness(&baseline.id, "evidence://bootstrap")?;
let outcome = db.evaluate_promote_and_health_check(&candidate.id, &baseline.id, "recorded-v1", &[PairedSample { seed: 1, candidate_score: 0.9, baseline_score: 0.5 }], PromotionPolicy { min_samples: 1, min_mean_delta: 0.4, min_win_rate: 1.0 }, "evidence://run", |_| Ok(true))?;
assert!(outcome.promoted);
drop(db);
let reopened = StateStore::open(&db_path)?;
assert_eq!(reopened.active_harness_id()?.as_deref(), Some(candidate.id.as_str()));
assert_eq!(reopened.harness_audit_entries()?.last().unwrap().event_type, "promoted");
Ok(())
}
}
+1 -1
View File
@@ -8,7 +8,7 @@
"ito-compute": {
"command": "node",
"args": ["/absolute/path/to/ito-cloud-runtime/cli/ito-compute-cli/dist/bin/ito-mcp.js"],
"description": "Opt-in local Itô compute MCP. The canonical package is unpublished and must be built from Ito-Markets/ito-cloud-runtime/cli/ito-compute-cli. Exposes only ito_auth, ito_find, and ito_status; inject ITO_API_KEY from the launching environment."
"description": "Opt-in local Itô compute MCP. The canonical package is unpublished and must be built from Ito-Markets/ito-cloud-runtime/cli/ito-compute-cli. Exposes only ito_auth, ito_find, and ito_status. Authentication delegates to canonical CLI device authorization, which opens the Itô verification page by default and persists a device token in macOS Keychain; ecc ito auth --no-browser suppresses the CLI page handoff. ECC itself performs no browser automation. Legacy ITO_API_KEY requires explicit ITO_AUTH_MODE=legacy."
},
"jira": {
"command": "uvx",
+9 -6
View File
@@ -28,15 +28,18 @@ function showHelp() {
ECC × Itô local CLI bridge
Usage:
ecc ito auth
ecc ito auth [--no-browser]
ecc ito find <all required RFQ options>
ecc ito status
ecc ito evals --cluster <id> --live-sixtytwo --nodes <list> --config-dir <dir>
ecc ito <auth|find|status|evals> --json
The bridge invokes the separately installed canonical Itô CLI and returns its
real stdout, stderr, and exit code unchanged. It performs no browser navigation
and adds no lock, workload, inference, or purchase path.
real stdout, stderr, and exit code unchanged. "ecc ito auth" delegates to the
canonical CLI's device authorization. It opens the Itô verification page by default
and persists its device token in macOS Keychain. Pass --no-browser to
suppress that handoff. ECC itself performs no browser automation and adds no
lock, workload, inference, or purchase path.
Important:
- "find" reads live inventory and submits an authenticated RFQ.
@@ -67,9 +70,9 @@ The same package's MCP server exposes only:
Configure the MCP command as "node" with this absolute argument:
/absolute/path/to/ito-cloud-runtime/${CANONICAL_PACKAGE_PATH}/dist/bin/ito-mcp.js
For auth, find, and status, inject ITO_API_KEY into the child process from
1Password or the launching environment. Never put the key in arguments,
tracked files, or chat.
Device authorization is the default. Legacy ITO_API_KEY authentication is
forwarded only with explicit ITO_AUTH_MODE=legacy. Never put a key or token in
arguments, tracked files, or chat.
Live node qualification requires ITO_ENABLE_SIXTYTWO_LIVE=1,
--live-sixtytwo, an explicit node list, and an existing absolute config
+4
View File
@@ -27,6 +27,9 @@ const ITO_RUNTIME_ENVIRONMENT_KEYS = Object.freeze([
"ITO_API_KEY",
"ITO_API_URL",
"ITO_INVENTORY_URL",
"ITO_AUTH_MODE",
"ITO_ALLOW_FILE_TOKEN",
"ITO_TOKEN_FILE",
]);
const ITO_EVAL_ENVIRONMENT_KEYS = Object.freeze([
@@ -61,6 +64,7 @@ function createSafeItoEnvironment(source = process.env, options = {}) {
if (options.includeItoRuntime) {
for (const key of ITO_RUNTIME_ENVIRONMENT_KEYS) {
if (key === "ITO_API_KEY" && source.ITO_AUTH_MODE !== "legacy") continue;
copyDefined(source, safe, key);
}
}
+10 -5
View File
@@ -8,8 +8,8 @@ metadata:
# Itô Compute
Use the canonical Itô compute CLI or MCP server. ECC does not implement a
parallel client, browser handoff, local simulation, reservation, workload
runner, or inference server.
parallel client, local simulation, reservation, workload runner, or inference
server. ECC itself does no browser automation.
## Install the canonical local package
@@ -30,12 +30,17 @@ Set `ECC_ITO_CLI_EXECUTABLE` to the explicit absolute built entry:
```
ECC never discovers this credential-bearing client through `PATH`.
Inject `ITO_API_KEY` through 1Password or the launching process environment.
Never put it in arguments, tracked files, MCP results, logs, or chat.
Device authorization is the default. Keep legacy `ITO_API_KEY` only with
explicit `ITO_AUTH_MODE=legacy`; never put it in arguments, tracked files, MCP
results, logs, or chat.
## CLI workflow
1. Run `ecc ito auth` before the first operation.
1. Run `ecc ito auth` before the first operation. ECC delegates this to the
canonical CLI's device authorization, which opens the Itô verification page
by default and persists a device token in macOS Keychain. Use
`ecc ito auth --no-browser` to suppress the page handoff. ECC itself does no
browser automation.
2. Before `ecc ito find`, obtain explicit buyer authority to submit an RFQ.
- Require `gpu`, `count`, whole `days`, `max-rate`, `nodes`,
`gpus-per-node`, `storage-tb`, `start-window`, `form-factor`,
+8
View File
@@ -57,6 +57,11 @@ function main() {
assert.match(skill, /ECC_ITO_CLI_EXECUTABLE/);
assert.match(skill, /explicit absolute built entry/);
assert.match(skill, /never discovers[^\n]*through `PATH`/);
assert.match(skill, /opens the Itô verification page\s+by default/i);
assert.match(skill, /--no-browser/);
assert.match(skill, /macOS Keychain/i);
assert.match(skill, /ITO_AUTH_MODE=legacy/);
assert.match(skill, /ECC (?:itself )?(?:does|performs) no browser automation/i);
assert.match(skill, /ITO_ENABLE_SIXTYTWO_LIVE/);
assert.match(skill, /sixtytwo-cli==0\.3\.33/);
assert.match(skill, /explicit node/i);
@@ -106,6 +111,9 @@ function main() {
assert.doesNotMatch(JSON.stringify(server), /npx|ito_lock|ito_run|paper|simulat/i);
assert.match(server.description, /ito_auth, ito_find, and ito_status/);
assert.match(server.description, /unpublished/i);
assert.match(server.description, /device authorization/i);
assert.match(server.description, /macOS Keychain/i);
assert.match(server.description, /no browser automation/i);
}],
];
+65 -7
View File
@@ -21,6 +21,7 @@ const {
const {
createSafeItoInvocationEnvironment,
getInvocationCommand,
ITO_RUNTIME_ENVIRONMENT_KEYS,
} = require("../../scripts/lib/ito-environment");
function runCli(args, environment = {}) {
@@ -103,6 +104,18 @@ function main() {
}
}
}],
["forwards the canonical auth browser opt-out without performing browser automation", () => {
const probe = makeItoProbe();
try {
const result = runCli(["ito", "auth", "--no-browser"], {
ECC_ITO_CLI_EXECUTABLE: probe.executable,
});
assert.strictEqual(result.status, 0, result.stderr);
assert.deepStrictEqual(readInvocation(probe).argv, ["auth", "--no-browser"]);
} finally {
fs.rmSync(probe.directory, { recursive: true, force: true });
}
}],
["normalizes JSON and forwards every RFQ constraint without interpretation", () => {
const probe = makeItoProbe();
try {
@@ -135,12 +148,15 @@ function main() {
fs.rmSync(probe.directory, { recursive: true, force: true });
}
}],
["passes only the required Itô runtime settings across the process boundary", () => {
["passes only the required device-auth runtime settings across the process boundary", () => {
const probe = makeItoProbe();
try {
const result = runCli(["ito", "auth"], {
ECC_ITO_CLI_EXECUTABLE: probe.executable,
ITO_API_KEY: "ito_test_key",
ITO_API_KEY: "must-not-cross-without-legacy-mode",
ITO_AUTH_MODE: "device",
ITO_ALLOW_FILE_TOKEN: "1",
ITO_TOKEN_FILE: "/tmp/ito-device-token",
ITO_API_URL: "https://compute.example.test",
ITO_INVENTORY_URL: "https://edge.example.test",
AWS_SECRET_ACCESS_KEY: "must-not-cross",
@@ -149,7 +165,10 @@ function main() {
});
assert.strictEqual(result.status, 0, result.stderr);
const childEnvironment = readInvocation(probe).env;
assert.strictEqual(childEnvironment.ITO_API_KEY, "ito_test_key");
assert.strictEqual(childEnvironment.ITO_API_KEY, undefined);
assert.strictEqual(childEnvironment.ITO_AUTH_MODE, "device");
assert.strictEqual(childEnvironment.ITO_ALLOW_FILE_TOKEN, "1");
assert.strictEqual(childEnvironment.ITO_TOKEN_FILE, "/tmp/ito-device-token");
assert.strictEqual(childEnvironment.ITO_API_URL, "https://compute.example.test");
assert.strictEqual(childEnvironment.ITO_INVENTORY_URL, "https://edge.example.test");
assert.strictEqual(childEnvironment.AWS_SECRET_ACCESS_KEY, undefined);
@@ -160,6 +179,26 @@ function main() {
fs.rmSync(probe.directory, { recursive: true, force: true });
}
}],
["forwards the legacy API key only with explicit legacy auth mode", () => {
for (const [mode, expectedKey] of [
[undefined, undefined],
["device", undefined],
["legacy", "ito_test_key"],
]) {
const probe = makeItoProbe();
try {
const result = runCli(["ito", "status"], {
ECC_ITO_CLI_EXECUTABLE: probe.executable,
ITO_API_KEY: "ito_test_key",
...(mode ? { ITO_AUTH_MODE: mode } : {}),
});
assert.strictEqual(result.status, 0, result.stderr);
assert.strictEqual(readInvocation(probe).env.ITO_API_KEY, expectedKey);
} finally {
fs.rmSync(probe.directory, { recursive: true, force: true });
}
}
}],
["isolates live node qualification from Itô and unrelated credentials", () => {
const probe = makeItoProbe();
try {
@@ -176,6 +215,9 @@ function main() {
], {
ECC_ITO_CLI_EXECUTABLE: probe.executable,
ITO_API_KEY: "must-not-cross-into-node-qualification",
ITO_AUTH_MODE: "legacy",
ITO_ALLOW_FILE_TOKEN: "1",
ITO_TOKEN_FILE: "/tmp/must-not-cross-token-file",
ITO_API_URL: "https://compute.example.test",
ITO_INVENTORY_URL: "https://edge.example.test",
ITO_ENABLE_SIXTYTWO_LIVE: "1",
@@ -201,6 +243,9 @@ function main() {
assert.strictEqual(invocation.env.SIXTYTWO_TOKEN, "sixtytwo-legacy-test-token");
assert.strictEqual(invocation.env.SSH_AUTH_SOCK, "/tmp/ecc-test-agent.sock");
assert.strictEqual(invocation.env.ITO_API_KEY, undefined);
assert.strictEqual(invocation.env.ITO_AUTH_MODE, undefined);
assert.strictEqual(invocation.env.ITO_ALLOW_FILE_TOKEN, undefined);
assert.strictEqual(invocation.env.ITO_TOKEN_FILE, undefined);
assert.strictEqual(invocation.env.ITO_API_URL, undefined);
assert.strictEqual(invocation.env.ITO_INVENTORY_URL, undefined);
assert.strictEqual(invocation.env.ITO_CLI_DEMO, undefined);
@@ -310,6 +355,14 @@ function main() {
}
}],
["classifies Itō child environments once and fails closed on unknown prefixes", () => {
assert.deepStrictEqual(ITO_RUNTIME_ENVIRONMENT_KEYS, [
"ITO_API_KEY",
"ITO_API_URL",
"ITO_INVENTORY_URL",
"ITO_AUTH_MODE",
"ITO_ALLOW_FILE_TOKEN",
"ITO_TOKEN_FILE",
]);
const safe = createSafeItoInvocationEnvironment(
{
PATH: process.env.PATH,
@@ -338,7 +391,7 @@ function main() {
);
}],
["rejects unsupported browser, paper, and execution operations before spawning", () => {
for (const command of ["rent", "lock", "run", "inference", "mcp"]) {
for (const command of ["rent", "lock", "purchase", "run", "inference", "mcp"]) {
const probe = makeItoProbe();
try {
const result = runCli(["ito", command], {
@@ -510,14 +563,14 @@ function main() {
fs.rmSync(probe.directory, { recursive: true, force: true });
}
}],
["help exposes the truthful CLI and MCP surface without a browser path", () => {
["help exposes canonical device auth without claiming ECC browser automation", () => {
const probe = makeItoProbe();
try {
const result = runCli(["ito", "--help"], {
ECC_ITO_CLI_EXECUTABLE: probe.executable,
});
assert.strictEqual(result.status, 0, result.stderr);
assert.match(result.stdout, /ecc ito auth/);
assert.match(result.stdout, /ecc ito auth \[--no-browser\]/);
assert.match(result.stdout, /ecc ito find/);
assert.match(result.stdout, /ecc ito status/);
assert.match(result.stdout, /ecc ito evals/);
@@ -528,9 +581,14 @@ function main() {
assert.match(result.stdout, new RegExp(CANONICAL_PACKAGE.replaceAll("/", "\\/")));
assert.match(result.stdout, /unpublished/i);
assert.match(result.stdout, /never discovers[^\n]*through PATH/i);
assert.match(result.stdout, /device authorization/i);
assert.match(result.stdout, /opens the Itô verification page by default/i);
assert.match(result.stdout, /macOS Keychain/i);
assert.match(result.stdout, /ECC itself performs no browser automation/i);
assert.match(result.stdout, /ITO_AUTH_MODE=legacy/);
assert.doesNotMatch(
result.stdout,
/manual copy|open(?:s)? (?:a )?browser|ito_lock|ito_run|npm link|paper|simulat/i
/manual copy|ito_lock|ito_run|npm link|paper|simulat/i
);
assert.ok(!fs.existsSync(probe.log));
} finally {