diff --git a/README.md b/README.md index 190a36e..873f823 100644 --- a/README.md +++ b/README.md @@ -43,13 +43,15 @@ No setup at all also works: `npx @only-cli/oc` runs without a global install, an ``` oc open fetch and render a page with numbered actions +oc do follow the numbered link [n] from the last page oc raw distilled markdown of the whole page -oc do activate a numbered element (v0.2) oc fill type into a numbered input (v0.2) oc submit [n] submit a form (v0.2) ``` -Flags: `--budget ` (default 500), `--json`, `--html` (raw as cleaned HTML instead of markdown), `--verbose`/`-v` (metrics on stderr: tokens saved vs the page HTML, HTTP status and client identity, timing, transfer size, memory; alias `--stats`, or export `OC_VERBOSE=1`). Metrics are off by default because they cost tokens too; agents should pass `--verbose` only when running verbosely. +Flags: `--budget ` (default 500), `--json`, `--html` (raw as cleaned HTML instead of markdown), `--session ` (separate page state per name), `--verbose`/`-v` (metrics on stderr: tokens saved vs the page HTML, HTTP status and client identity, timing, transfer size, memory; alias `--stats`, or export `OC_VERBOSE=1`). Metrics are off by default because they cost tokens too; agents should pass `--verbose` only when running verbosely. + +`oc open` remembers the elements it numbered, so `oc do 3` follows `[3]` without the agent ever handling a URL. That state is a small JSON file per session under `~/.only-cli` (override the directory with `OC_HOME`); there is no daemon and no background browser. Links that search engines wrap in a tracking redirect resolve to the real destination, so `do` on a result works like a click. ## Supported websites @@ -67,7 +69,7 @@ only-cli works on any mostly-static website with no per-site setup: news sites, The engine also renders Atom and RSS feeds as regular pages. That is how Stack Overflow works: the site serves every HTML page a Cloudflare challenge, but publishes full question and answer bodies under `/feeds`, so `oc open stackoverflow.com/feeds/question/11227809` returns the question and its top answers in about 500 tokens. The same trick applies to any site that gates its pages but leaves its feeds open. -Not supported yet: pages that only render with JavaScript (a headless fallback is planned for v0.3), sites behind logins (sessions land in v0.2), and sites with hard bot challenges that do not expose feeds. Adding a site shortcut is a small JSON file; see [CONTRIBUTING.md](CONTRIBUTING.md). +Not supported yet: pages that only render with JavaScript (a headless fallback is planned for v0.3), sites behind logins (page state is saved, cookies are not, so logins land in v0.2), and sites with hard bot challenges that do not expose feeds. Adding a site shortcut is a small JSON file; see [CONTRIBUTING.md](CONTRIBUTING.md). ## Benchmarks @@ -111,13 +113,13 @@ jina-reader ################## 855,243 tokens 30 tur playwright-mcp ################################## 1,575,695 tokens 48 turns ``` -Each session gets a skill documenting its tool, so every condition runs at its best. Two results are worth stating plainly. oc and lynx were the only tools that returned real content on every task: Reddit served Jina Reader and Playwright MCP a 403, so both "answered" that task by reporting the block, and raw curl burned its full turn budget there and on the GitHub search, roughly 400k tokens each, and returned nothing. But lynx, not oc, took the token and cost columns this round, and the reason is a missing feature. The compact view leaves link URLs out to save tokens, and `oc do ` does not ship until v0.2, so an agent following a link has to re-fetch the page as `oc open --json` or `oc raw` just to learn where `[15]` points, while `lynx -dump` prints a references list for free. That tax is most of the gap on the multi-step tasks. Activatable numbered actions are the fix and the point of v0.2. +Each session gets a skill documenting its tool, so every condition runs at its best. Two results are worth stating plainly. oc and lynx were the only tools that returned real content on every task: Reddit served Jina Reader and Playwright MCP a 403, so both "answered" that task by reporting the block, and raw curl burned its full turn budget there and on the GitHub search, roughly 400k tokens each, and returned nothing. But lynx, not oc, took the token and cost columns this round, and the reason was a missing feature. The compact view leaves link URLs out to save tokens, and the version under test had no way to follow one, so an agent had to re-fetch the page as `oc open --json` or `oc raw` just to learn where `[15]` points, while `lynx -dump` prints a references list for free. That tax is most of the gap on the multi-step tasks. `oc do ` has since landed and closes it: following a Hacker News story into its comments now costs about 3k characters instead of the roughly 23k the re-fetch route spent. The next benchmark run will say whether that is enough to take the column. The same six tasks run through OpenAI's Codex CLI (`codex exec`) as well, where oc comes out ahead: of the two tools that returned real content on all six tasks, oc spent 287,862 tokens and lynx 408,548, and Playwright MCP spent 699,810 on the Reddit thread alone. Codex's per-session overhead is much smaller than Claude Code's, so the two agents are compared within their own tables rather than against each other. The benchmark repo has per-task numbers, per-tier breakdowns, methodology, and instructions for adding other tools and models. ## Status -Early. v0.1 covers static pages, budget-aware rendering, and offline tests. Sessions and actions land in v0.2, a lazy headless fallback for script-heavy pages in v0.3. The design principles and how to contribute are in [CONTRIBUTING.md](CONTRIBUTING.md). +Early. v0.1 covers static pages, budget-aware rendering, and offline tests. Sessions and `oc do ` are in, the rest of the actions (`fill`, `submit`, `read`, `find`, `back`) land in v0.2, and a lazy headless fallback for script-heavy pages in v0.3. The design principles and how to contribute are in [CONTRIBUTING.md](CONTRIBUTING.md). Known limits, honestly: no JavaScript rendering yet, no sites behind logins yet, and pages behind hard bot challenges may still refuse the tool. diff --git a/skills/only-cli/SKILL.md b/skills/only-cli/SKILL.md index c222fd9..f473a0a 100644 --- a/skills/only-cli/SKILL.md +++ b/skills/only-cli/SKILL.md @@ -11,6 +11,7 @@ No install needed, run it with npx: ``` npx @only-cli/oc open compact view with numbered elements +npx @only-cli/oc do follow numbered link [n] from the last page npx @only-cli/oc raw whole page as markdown (add --html for cleaned HTML) ``` @@ -21,9 +22,24 @@ npx @only-cli/oc raw whole page as markdown (add --html for cleaned H - The `actions:` line at the bottom lists valid next commands. - `... N more blocks over budget` means content was cut to stay cheap. Rerun with `--budget 1500` if you need more, or use `raw` for everything. -## Following links (current version) +## Following links -`do ` and the other action commands land in v0.2. Until then: run `raw ` to see each link's href in markdown form, then `open` that URL directly. +Use `do `. The compact view leaves link URLs out because they cost tokens and you do not need them, so to open `[15] 41 comments` run `oc do 15`. It renders the new page exactly like `open` does, and the numbers then refer to that new page. + +``` +oc open news.ycombinator.com -> [15] 41 comments +oc do 15 -> the comment thread, renumbered +``` + +Notes that save a round trip: + +- Numbers come from the most recent render, so re-read the newest output before choosing one. Any command that renders a page renumbers. +- Handles hidden behind a `[6-9] 4 similar links` marker still work, even though their text was collapsed. +- Search result links resolve to the destination, not the search engine's tracking redirect. +- `do` on an input or a button says so; typing and submitting are not available yet. +- `--session ` keeps separate page state, for working on two sites at once. + +Reach for `raw ` when you need the whole page text, not to hunt for a URL. ## Flags diff --git a/src/act.js b/src/act.js index 64c7154..685d533 100644 --- a/src/act.js +++ b/src/act.js @@ -1,18 +1,46 @@ /** - * Actions against a live session: do, fill, submit, read, find, back, next. - * All of this ships in v0.2 together with session state. The signatures - * exist now so cli.js wires up once and the command surface stays stable. + * Actions against a saved session: do now, fill, submit, read, find, back, + * and next still ahead. Errors here are read by agents, so every one of them + * names the command to run next. */ +import { DEFAULT_SESSION, loadSession } from './session.js'; + export class NotImplemented extends Error { constructor(command) { - super(`'oc ${command}' lands in v0.2. Until then use 'oc open' and 'oc raw'.`); + super(`'oc ${command}' is not available yet. Until then use 'oc open' and 'oc raw'.`); } } -/** @param {number} n */ -export function activate(n) { - throw new NotImplemented('do'); +/** + * Resolve a numbered handle from the last render into something to open. + * Returns the target URL; the caller fetches and renders it exactly as + * `oc open` would, so `do` and `open` always agree on what a page looks like. + * @param {number} n + * @param {{session?: string}} [opts] + * @returns {{url: string, text: string}} + */ +export function activate(n, { session = DEFAULT_SESSION } = {}) { + if (!Number.isInteger(n) || n < 1) { + throw new Error('usage: oc do , where is a number from the last page'); + } + const state = loadSession(session); + if (!state) { + throw new Error("nothing open in this session yet, run 'oc open ' first"); + } + const handle = state.handles?.[n]; + if (!handle) { + const nums = Object.keys(state.handles ?? {}).map(Number); + const range = nums.length ? `1-${Math.max(...nums)}` : 'none'; + throw new Error(`no [${n}] on ${state.url} (handles ${range}), run 'oc open ' again to renumber`); + } + if (handle.type === 'input') { + throw new Error(`[${n}] is an input (${handle.name ?? 'text'}), typing needs 'oc fill', which is not available yet`); + } + if (handle.type === 'button' || !handle.href) { + throw new Error(`[${n}] has no link to follow, it is a ${handle.type} the page handles itself`); + } + return { url: handle.href, text: handle.text }; } /** @param {number} n @param {string} text */ diff --git a/src/cli.js b/src/cli.js index a8b2761..4a4f8b8 100644 --- a/src/cli.js +++ b/src/cli.js @@ -4,6 +4,7 @@ import { fetchPage } from './fetch.js'; import { distill, toMarkdown, toHTML } from './distill.js'; import { render, estimateTokens } from './render.js'; import * as act from './act.js'; +import { DEFAULT_SESSION, loadSession, saveSession, sessionFromPage } from './session.js'; const HELP = `only-cli: the web as a compact terminal, built for AI agents. @@ -11,7 +12,7 @@ usage: oc [args] [flags] open fetch and render a page with numbered actions raw distilled markdown of the whole page - do activate a numbered element (v0.2) + do follow the numbered link [n] from the last page fill type into a numbered input (v0.2) submit [n] submit a form (v0.2) read [n] full text of one region (v0.2) @@ -27,7 +28,20 @@ flags: status and client identity, timing, transfer size, and memory. --stats is an alias; OC_VERBOSE=1 turns it on globally. Off by default because metrics cost tokens too. - --session named session (v0.2)`; + --session keep separate page state under a name (default: default) + +'oc open' remembers the numbered elements it printed, so 'oc do 3' follows +link [3] without you ever handling its URL. State lives in ~/.only-cli +(override with OC_HOME).`; + +// A rendered page has to be remembered or its [3] means nothing to the next +// command. Saving state must never break a render, so a home directory that +// cannot be written costs the agent `do` and nothing else. +const remember = (page, name) => { + try { + saveSession(name, sessionFromPage(page, loadSession(name))); + } catch {} +}; // What browsing costs without this tool is the raw page HTML in context. const savings = (out, raw) => @@ -57,10 +71,17 @@ async function main() { return; } + const sessionName = values.session || DEFAULT_SESSION; + switch (command) { case 'open': + case 'do': case 'raw': { - const url = args[0]; + // `do` is `open` with the URL looked up from the last render instead of + // typed, so both commands share one fetch, render, and save path. + const url = command === 'do' + ? act.activate(Number(args[0]), { session: sessionName }).url + : args[0]; if (!url) throw new Error(`usage: oc ${command} `); const budget = values.budget ? Number(values.budget) : 500; if (!Number.isFinite(budget) || budget <= 0) throw new Error('--budget must be a positive number'); @@ -74,7 +95,9 @@ async function main() { + `${Math.round(html.length / 1024)}KB transferred, ${Math.round(rss / 1048576)}MB memory`; }; if (values.json) { - console.log(JSON.stringify(distill(html, finalUrl))); + const page = distill(html, finalUrl); + remember(page, sessionName); + console.log(JSON.stringify(page)); if (verbose) console.error(resources()); return; } @@ -86,6 +109,7 @@ async function main() { return; } const page = distill(html, finalUrl); + remember(page, sessionName); const { text, stats } = render(page, { budget }); console.log(text); if (verbose) { @@ -93,7 +117,6 @@ async function main() { } return; } - case 'do': return act.activate(Number(args[0])); case 'fill': return act.fill(Number(args[0]), args.slice(1).join(' ')); case 'submit': return act.submit(args[0] ? Number(args[0]) : undefined); case 'read': return act.read(args[0] ? Number(args[0]) : undefined); diff --git a/src/session.js b/src/session.js index 3471afc..2f48e90 100644 --- a/src/session.js +++ b/src/session.js @@ -1,16 +1,109 @@ /** - * Sessions are plain JSON files on disk, one per name: current URL, cookies, - * the last distilled page so actions can resolve handles, and history. No - * daemon, no background process. Read/write lands in v0.2 with act.js. + * Sessions are plain JSON files on disk, one per name: the current URL, the + * numbered handles from the last render so actions can resolve them, and a + * short history. No daemon, no background process, no cookies yet. + * + * The file exists so `oc do ` can follow a link the compact view never + * printed the URL of. Hiding URLs is what makes `oc open` cheap; this is what + * makes hiding them free. */ import { homedir } from 'node:os'; import { join } from 'node:path'; +import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'; -export const SESSION_DIR = join(homedir(), '.only-cli', 'sessions'); +export const DEFAULT_SESSION = 'default'; + +// OC_HOME relocates the whole state directory, for sandboxes, CI, and tests. +export const sessionDir = () => join(process.env.OC_HOME ?? join(homedir(), '.only-cli'), 'sessions'); /** * @param {string} name * @returns {string} */ -export const sessionPath = (name) => join(SESSION_DIR, `${name}.json`); +export const sessionPath = (name) => join(sessionDir(), `${name}.json`); + +// Search engines and link aggregators wrap outbound links in a tracking +// redirector whose landing page is a script, not content, so following one +// verbatim renders nothing. The target is sitting in the query string. +const REDIRECT_PATH = /^\/(l|l\.php|url|out|redirect|away|link)\/?$/i; +const REDIRECT_PARAMS = ['uddg', 'url', 'u', 'q', 'target', 'to', 'dest']; + +/** + * @param {URL} url + * @returns {string|null} the wrapped destination, or null if this is a normal link + */ +function unwrapRedirect(url) { + if (!REDIRECT_PATH.test(url.pathname)) return null; + for (const param of REDIRECT_PARAMS) { + const value = url.searchParams.get(param); + if (value && /^https?:\/\//i.test(value)) return value; + } + return null; +} + +/** + * Absolute URL for a handle, or null when the link is not followable + * (javascript: handlers, malformed hrefs). + * @param {string} href + * @param {string} base + * @returns {string|null} + */ +export function resolveHref(href, base) { + if (!href || /^(javascript|about):/i.test(href)) return null; + try { + const url = new URL(href, base || undefined); + return unwrapRedirect(url) ?? url.href; + } catch { + return null; + } +} + +const HISTORY_LIMIT = 20; + +/** + * Session state for a freshly rendered page. Every numbered block is kept, + * including the ones the budget skipped, because the handles an agent wants + * are often the ones that did not fit. + * @param {import('./distill.js').Page} page + * @param {{history?: string[]}} [previous] + */ +export function sessionFromPage(page, previous) { + /** @type {Record} */ + const handles = {}; + for (const block of page.blocks) { + if (block.n == null) continue; + const url = block.href ? resolveHref(block.href, page.url) : null; + handles[block.n] = { + type: block.type, + text: block.text, + ...(url ? { href: url } : {}), + ...(block.name ? { name: block.name } : {}), + }; + } + const history = [...(previous?.history ?? []), page.url].slice(-HISTORY_LIMIT); + return { url: page.url, title: page.title, savedAt: new Date().toISOString(), handles, history }; +} + +/** + * @param {string} name + * @param {object} state + */ +export function saveSession(name, state) { + mkdirSync(sessionDir(), { recursive: true }); + writeFileSync(sessionPath(name), JSON.stringify(state)); +} + +/** + * Missing or unreadable state is not an error: it means nothing is open yet, + * and the caller says so in a sentence that names the next command. + * @param {string} name + * @returns {any|null} + */ +export function loadSession(name) { + try { + return JSON.parse(readFileSync(sessionPath(name), 'utf8')); + } catch { + return null; + } +} diff --git a/tests/act.test.js b/tests/act.test.js new file mode 100644 index 0000000..146c7c0 --- /dev/null +++ b/tests/act.test.js @@ -0,0 +1,73 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync, mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +// Every session helper reads OC_HOME when it runs, so pointing it at a temp +// directory here keeps the suite offline and out of the real home directory. +process.env.OC_HOME = mkdtempSync(join(tmpdir(), 'oc-test-')); + +const { distill } = await import('../src/distill.js'); +const { activate } = await import('../src/act.js'); +const { sessionFromPage, saveSession, loadSession, resolveHref } = await import('../src/session.js'); + +const html = readFileSync(new URL('./pages/news.html', import.meta.url), 'utf8'); +const page = () => distill(html, 'https://example.test/news'); +const open = (name = 'default') => saveSession(name, sessionFromPage(page(), loadSession(name))); + +test('a rendered page is remembered with absolute URLs for every handle', () => { + open(); + const state = loadSession('default'); + assert.equal(state.url, 'https://example.test/news'); + assert.equal(state.handles[1].href, 'https://example.test/item?id=1'); + const numbered = page().blocks.filter((b) => b.n != null); + assert.equal(Object.keys(state.handles).length, numbered.length, 'every numbered block must be resolvable'); +}); + +test('do follows the link behind a number without the agent seeing a URL', () => { + open(); + assert.deepEqual(activate(1), { url: 'https://example.test/item?id=1', text: 'Show HN: I built a tiny CSV toolkit' }); +}); + +test('search result redirectors resolve to the page they wrap', () => { + const wrapped = 'https://duckduckgo.com/l/?uddg=https%3A%2F%2Fdoc.rust-lang.org%2Fbook%2F&rut=abc'; + assert.equal(resolveHref(wrapped, 'https://html.duckduckgo.com/html/'), 'https://doc.rust-lang.org/book/'); + assert.equal(resolveHref('https://www.google.com/url?q=https://example.test/a', ''), 'https://example.test/a'); + // A normal link that merely has a url-shaped query parameter is left alone. + assert.equal( + resolveHref('/search?url=https://example.test/a', 'https://example.test/'), + 'https://example.test/search?url=https://example.test/a', + ); +}); + +test('links a browser cannot follow are not offered as handles', () => { + assert.equal(resolveHref('javascript:void(0)', 'https://example.test/'), null); + assert.equal(resolveHref('', 'https://example.test/'), null); +}); + +test('every failure names the command that fixes it', () => { + open(); + assert.throws(() => activate(9999), /handles 1-\d+.*oc open/s); + assert.throws(() => activate(0), /usage: oc do /); + assert.throws(() => activate(1, { session: 'never-opened' }), /oc open ' first/); + + const input = page().blocks.find((b) => b.type === 'input'); + assert.throws(() => activate(input.n), /is an input.*oc fill/s); + const button = page().blocks.find((b) => b.type === 'button'); + assert.throws(() => activate(button.n), /no link to follow/); +}); + +test('named sessions keep separate page state', () => { + open('work'); + saveSession('other', { url: 'https://example.test/other', handles: {} }); + assert.equal(activate(1, { session: 'work' }).url, 'https://example.test/item?id=1'); + assert.throws(() => activate(1, { session: 'other' }), /no \[1\]/); +}); + +test('history grows with each page and stays bounded', () => { + let state = null; + for (let i = 0; i < 25; i++) state = sessionFromPage(distill(html, `https://example.test/p${i}`), state); + assert.equal(state.history.length, 20); + assert.equal(state.history.at(-1), 'https://example.test/p24'); +});