From dcc0531ef32ce9a2a0616d1cae41de343e003374 Mon Sep 17 00:00:00 2001 From: only-cli Date: Tue, 18 Aug 2026 08:55:52 -0400 Subject: [PATCH] only-cli v0.1: turn websites into a compact CLI for AI agents Generic distillation engine (no per-site adapters): fetch via impers impersonating Chrome with a firefox-fingerprint retry, distill to an interaction tree, render under a hard token budget with numbered action handles. Raw mode emits markdown via turndown or cleaned HTML. Per-site CLI definitions for HN, Reddit, Bing, DuckDuckGo. Offline test suite, agent skill, OIDC publish workflow. --- .github/workflows/publish.yml | 29 ++++++ .gitignore | 4 + CONTRIBUTING.md | 48 ++++++++++ README.md | 66 +++++++++++++ clis/bing.com.json | 7 ++ clis/duckduckgo.com.json | 7 ++ clis/news.ycombinator.com.json | 9 ++ clis/reddit.com.json | 9 ++ package.json | 44 +++++++++ skills/only-cli/SKILL.md | 37 ++++++++ src/act.js | 45 +++++++++ src/cli.js | 112 ++++++++++++++++++++++ src/distill.js | 167 +++++++++++++++++++++++++++++++++ src/fetch.js | 68 ++++++++++++++ src/render.js | 115 +++++++++++++++++++++++ src/session.js | 16 ++++ tests/distill.test.js | 79 ++++++++++++++++ tests/pages/news.html | 32 +++++++ 18 files changed, 894 insertions(+) create mode 100644 .github/workflows/publish.yml create mode 100644 .gitignore create mode 100644 CONTRIBUTING.md create mode 100644 README.md create mode 100644 clis/bing.com.json create mode 100644 clis/duckduckgo.com.json create mode 100644 clis/news.ycombinator.com.json create mode 100644 clis/reddit.com.json create mode 100644 package.json create mode 100644 skills/only-cli/SKILL.md create mode 100644 src/act.js create mode 100644 src/cli.js create mode 100644 src/distill.js create mode 100644 src/fetch.js create mode 100644 src/render.js create mode 100644 src/session.js create mode 100644 tests/distill.test.js create mode 100644 tests/pages/news.html diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml new file mode 100644 index 0000000..1caa28b --- /dev/null +++ b/.github/workflows/publish.yml @@ -0,0 +1,29 @@ +# Publishes to npm via OIDC trusted publishing: no token, no OTP prompt. +# One-time setup on npmjs.com after the package exists: package settings, +# Trusted Publisher, GitHub Actions, repository only-cli/oc, workflow +# file publish.yml. From then on, publishing a GitHub release ships to npm. +name: publish + +on: + release: + types: [published] + workflow_dispatch: + +permissions: + contents: read + id-token: write + +jobs: + publish: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 24 + registry-url: https://registry.npmjs.org + # Trusted publishing needs npm 11.5.1 or newer. + - run: npm install -g npm@latest + - run: npm ci + - run: npm test + - run: npm publish --access public diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b507c80 --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +node_modules/ +package-lock.json +*.log +.idea/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..ae2d58f --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,48 @@ +# Contributing to only-cli + +Thanks for wanting to help. This is a small project with strong opinions, so this guide is short but strict. Read [PROMPT.md](PROMPT.md) before anything else: it holds the design principles, the pipeline, and the roadmap, and every PR is judged against it. + +## Setup + +``` +git clone https://github.com/only-cli/oc +cd oc +npm install +npm test +``` + +Node 20+. Tests run fully offline against saved fixtures in `tests/pages/`, so a plane is a fine place to work on this. + +## What makes a PR easy to accept + +- **It respects the token budget.** Everything this tool prints gets read by a paying model. If your change adds output, show the before and after of `--stats` on a fixture page. A default render that crosses 500 tokens needs a very good reason; past 2,000 it is a bug. +- **It adds no dependencies.** The runtime dependency count (three) is a feature. If you truly need a new one, justify it in one line in the PR description and expect the default answer to be no. Standard library first, always. +- **It keeps output deterministic.** Same page, same command, same output. There is a test for this; do not weaken it. +- **It comes with an offline test.** New behavior gets a fixture in `tests/pages/` and a test in `tests/`. No network calls in tests, ever. +- **It fails loud and cheap.** A feature that cannot handle a page should say so in one line and exit nonzero, never dump raw HTML as a fallback. + +## Code style + +Plain JavaScript, ESM, JSDoc types, no build step. Match the code around you. Comments explain constraints and trade-offs, not what the next line does; if a comment restates the code, delete it. Small functions, few files: if you are adding a new file to `src/`, pause and check whether the logic belongs in one of the six that exist. + +## Writing style + +All prose in this repo (docs, comments, commit messages, error text, CLI help) follows the rules in PROMPT.md: write like a human, be precise like a developer, and leave a trail like a contributor. Be honest about limitations. Do not use em dashes anywhere; use commas, colons, or separate sentences. + +Commit messages explain why, not just what. "Cap link text at 200 chars, long titles were eating half the budget" tells the next person everything. + +## Adding a site definition + +Per-site CLIs live in `clis/`, one JSON file per domain, following the spec format section in PROMPT.md. Keep it under 50 lines. If the site has a public JSON API, point the commands at that instead of the HTML pages. If your definition needs logic, it is trying to become an adapter, and the answer is to improve the generic engine instead. + +## Reporting bugs + +Open an issue with the exact command, the output you got, and the output you expected. If the page is public, include the URL. If distillation mangled a page, a saved copy of the HTML as a fixture is the most useful thing you can attach. + +## Releasing (maintainer) + +Publishing a GitHub release runs `.github/workflows/publish.yml`, which tests and publishes to npm through OIDC trusted publishing: no npm token stored anywhere, no one-time password, and npm attaches provenance automatically. The trusted publisher link (npm package settings, GitHub Actions, repo `only-cli/oc`, workflow `publish.yml`) has to be configured once on npmjs.com after the first manual publish, since npm only lets you attach a trusted publisher to a package that already exists. + +## Maintainer + +[only-cli](https://github.com/only-cli) diff --git a/README.md b/README.md new file mode 100644 index 0000000..c7a6dd9 --- /dev/null +++ b/README.md @@ -0,0 +1,66 @@ +# only-cli + +Turn most website into a command line interface, so AI agents like Claude Code, Codex, and Antigravity can browse without burning tokens on raw HTML or screenshots. + +A typical page is tens of thousands of tokens of markup. The signal on it fits in a few hundred. only-cli fetches the page, distills it into a compact text view with numbered actions, and lets an agent drive the site by number: + +``` +$ oc open news.ycombinator.com +# Hacker News +[1] Show HN: I built a tiny CSV toolkit +[2] 312 comments +... +actions: do | raw + +$ oc do 1 +``` + +No per-site adapters required, no browser extension, no daemon. One generic distillation engine, three runtime dependencies, and a hard token budget on everything it prints. + +## Install + +``` +npm install -g only-cli +``` + +Requires Node 20+. Requests go through [impers](https://github.com/lexiforest/impers) impersonating Chrome; if impers is unavailable the tool falls back to native fetch. + +## For AI agents + +The fastest setup is one line in your agent's instructions file (CLAUDE.md, AGENTS.md, or equivalent): + +> When you need content from a web page, run `npx only-cli open ` instead of fetching raw HTML. Run `npx only-cli --help` once to learn the commands. + +Claude Code users can install the skill instead: copy `skills/only-cli/` into your project's `.claude/skills/` directory (or `~/.claude/skills/` to enable it everywhere). A skill costs almost no tokens until the agent actually invokes it, which fits how this whole project thinks. The same skill also installs through the [skills.sh](https://skills.sh) directory into Claude Code, Cursor, Codex, Copilot, and others: + +``` +npx skills add only-cli/oc +``` + +No setup at all also works: `npx only-cli` runs without a global install, and the tool teaches its own command surface through `--help`, the `actions:` line at the bottom of every render, and error messages that name the next command to run. + +## Commands + +``` +oc open fetch and render a page with numbered actions +oc raw distilled markdown of the whole page +oc do activate a numbered element (v0.2) +oc fill type into a numbered input (v0.2) +oc submit [n] submit a form (v0.2) +``` + +Flags: `--budget ` (default 500), `--json`, `--html` (raw as cleaned HTML instead of markdown), `--verbose`/`-v` (metrics on stderr: tokens saved vs the page HTML, HTTP status and client identity, timing, transfer size, memory; alias `--stats`, or export `OC_VERBOSE=1`). Metrics are off by default because they cost tokens too; agents should pass `--verbose` only when running verbosely. + +## Status + +Early. v0.1 covers static pages, budget-aware rendering, and offline tests. Sessions and actions land in v0.2, a lazy headless fallback for script-heavy pages in v0.3. The roadmap and all design constraints live in [PROMPT.md](PROMPT.md); how to contribute is in [CONTRIBUTING.md](CONTRIBUTING.md). + +Known limits, honestly: no JavaScript rendering yet, no sites behind logins yet, and pages behind hard bot challenges may still refuse the tool. + +## Contributors + +- [only-cli](https://github.com/only-cli), creator and maintainer + +## License + +MIT diff --git a/clis/bing.com.json b/clis/bing.com.json new file mode 100644 index 0000000..d2e0fe8 --- /dev/null +++ b/clis/bing.com.json @@ -0,0 +1,7 @@ +{ + "domain": "bing.com", + "commands": { + "search": { "open": "https://www.bing.com/search?q={query}", "args": ["query"] }, + "news": { "open": "https://www.bing.com/news/search?q={query}", "args": ["query"] } + } +} diff --git a/clis/duckduckgo.com.json b/clis/duckduckgo.com.json new file mode 100644 index 0000000..46fc60d --- /dev/null +++ b/clis/duckduckgo.com.json @@ -0,0 +1,7 @@ +{ + "domain": "duckduckgo.com", + "commands": { + "search": { "open": "https://html.duckduckgo.com/html/?q={query}", "args": ["query"] }, + "lite": { "open": "https://lite.duckduckgo.com/lite/?q={query}", "args": ["query"] } + } +} diff --git a/clis/news.ycombinator.com.json b/clis/news.ycombinator.com.json new file mode 100644 index 0000000..9b4cc1a --- /dev/null +++ b/clis/news.ycombinator.com.json @@ -0,0 +1,9 @@ +{ + "domain": "news.ycombinator.com", + "commands": { + "top": { "open": "https://news.ycombinator.com" }, + "new": { "open": "https://news.ycombinator.com/newest" }, + "item": { "open": "https://news.ycombinator.com/item?id={id}", "args": ["id"] }, + "user": { "open": "https://news.ycombinator.com/user?id={name}", "args": ["name"] } + } +} diff --git a/clis/reddit.com.json b/clis/reddit.com.json new file mode 100644 index 0000000..5e6047b --- /dev/null +++ b/clis/reddit.com.json @@ -0,0 +1,9 @@ +{ + "domain": "reddit.com", + "commands": { + "sub": { "open": "https://old.reddit.com/r/{sub}", "args": ["sub"] }, + "post": { "open": "https://old.reddit.com/comments/{id}", "args": ["id"] }, + "user": { "open": "https://old.reddit.com/user/{name}", "args": ["name"] }, + "search": { "open": "https://old.reddit.com/search?q={query}", "args": ["query"] } + } +} diff --git a/package.json b/package.json new file mode 100644 index 0000000..af0895c --- /dev/null +++ b/package.json @@ -0,0 +1,44 @@ +{ + "name": "only-cli", + "version": "0.1.0", + "description": "Turn websites into a compact CLI so AI agents can browse without burning tokens.", + "type": "module", + "bin": { + "oc": "src/cli.js" + }, + "files": [ + "src", + "clis", + "skills" + ], + "engines": { + "node": ">=20" + }, + "scripts": { + "test": "node --test" + }, + "keywords": [ + "cli", + "ai-agents", + "web", + "scraping", + "tokens" + ], + "author": "only-cli (https://github.com/only-cli)", + "license": "MIT", + "repository": { + "type": "git", + "url": "git+https://github.com/only-cli/oc.git" + }, + "homepage": "https://github.com/only-cli/oc#readme", + "bugs": { + "url": "https://github.com/only-cli/oc/issues" + }, + "dependencies": { + "linkedom": "^0.18.12", + "turndown": "^7.2.4" + }, + "optionalDependencies": { + "impers": "^0.1.0" + } +} diff --git a/skills/only-cli/SKILL.md b/skills/only-cli/SKILL.md new file mode 100644 index 0000000..3a38aa6 --- /dev/null +++ b/skills/only-cli/SKILL.md @@ -0,0 +1,37 @@ +--- +name: only-cli +description: Browse websites from the terminal in a few hundred tokens. Use when you need content from a web page, want to check a link, or would otherwise fetch raw HTML or reach for a browser. +--- + +# only-cli + +Turns a web page into a compact terminal view instead of a raw HTML dump. A typical page renders in under 500 tokens. + +No install needed, run it with npx: + +``` +npx only-cli open compact view with numbered elements +npx only-cli raw whole page as markdown (add --html for cleaned HTML) +``` + +## Reading the output + +- The first line is the page title, then headings, text, and interactive elements in page order. +- `[n]` marks a link, button, or input. +- The `actions:` line at the bottom lists valid next commands. +- `... N more blocks over budget` means content was cut to stay cheap. Rerun with `--budget 1500` if you need more, or use `raw` for everything. + +## Following links (current version) + +`do ` and the other action commands land in v0.2. Until then: run `raw ` to see each link's href in markdown form, then `open` that URL directly. + +## Flags + +- `--budget ` raise or lower the render budget (default 500) +- `--json` machine-stable JSON of the distilled page +- `--html` with raw: cleaned HTML instead of markdown, if markup suits your task better +- `--verbose` (`-v`, alias `--stats`) metrics on stderr: tokens saved vs the page HTML, HTTP status and which client identity got the page, fetch and processing time, bytes transferred, memory use. Only pass this when you are running in verbose mode or diagnosing a problem; the metrics line costs tokens like everything else. Users can export `OC_VERBOSE=1` to turn it on globally. + +## When not to use it + +Pages that require login or heavy client-side JavaScript are not supported yet. If a page comes back empty or blocked, say so and fall back to another method rather than retrying. diff --git a/src/act.js b/src/act.js new file mode 100644 index 0000000..69e2a79 --- /dev/null +++ b/src/act.js @@ -0,0 +1,45 @@ +/** + * Actions against a live session: do, fill, submit, read, find, back, next. + * All of this ships in v0.2 together with session state (see PROMPT.md + * milestones). The signatures exist now so cli.js wires up once and the + * command surface stays stable. + */ + +export class NotImplemented extends Error { + constructor(command) { + super(`'oc ${command}' lands in v0.2, see PROMPT.md milestones. Until then use 'oc open' and 'oc raw'.`); + } +} + +/** @param {number} n */ +export function activate(n) { + throw new NotImplemented('do'); +} + +/** @param {number} n @param {string} text */ +export function fill(n, text) { + throw new NotImplemented('fill'); +} + +/** @param {number} [n] */ +export function submit(n) { + throw new NotImplemented('submit'); +} + +/** @param {number} [n] */ +export function read(n) { + throw new NotImplemented('read'); +} + +/** @param {string} query */ +export function find(query) { + throw new NotImplemented('find'); +} + +export function back() { + throw new NotImplemented('back'); +} + +export function next() { + throw new NotImplemented('next'); +} diff --git a/src/cli.js b/src/cli.js new file mode 100644 index 0000000..a8b2761 --- /dev/null +++ b/src/cli.js @@ -0,0 +1,112 @@ +#!/usr/bin/env node +import { parseArgs } from 'node:util'; +import { fetchPage } from './fetch.js'; +import { distill, toMarkdown, toHTML } from './distill.js'; +import { render, estimateTokens } from './render.js'; +import * as act from './act.js'; + +const HELP = `only-cli: the web as a compact terminal, built for AI agents. + +usage: oc [args] [flags] + + open fetch and render a page with numbered actions + raw distilled markdown of the whole page + do activate a numbered element (v0.2) + fill type into a numbered input (v0.2) + submit [n] submit a form (v0.2) + read [n] full text of one region (v0.2) + find search visible text on the current page (v0.2) + back | next history and pagination (v0.2) + session ls|rm manage saved sessions (v0.2) + +flags: + --budget tighten or loosen the render budget (default 500) + --json machine-stable JSON output + --html raw only: cleaned HTML instead of markdown + --verbose, -v metrics on stderr: tokens saved vs the page HTML, HTTP + status and client identity, timing, transfer size, and + memory. --stats is an alias; OC_VERBOSE=1 turns it on + globally. Off by default because metrics cost tokens too. + --session named session (v0.2)`; + +// What browsing costs without this tool is the raw page HTML in context. +const savings = (out, raw) => + `~${out} tokens vs ~${raw} for the page HTML (${Math.max(0, 100 - Math.round((out / Math.max(raw, 1)) * 100))}% saved)`; + +async function main() { + const { values, positionals } = parseArgs({ + allowPositionals: true, + options: { + json: { type: 'boolean', default: false }, + html: { type: 'boolean', default: false }, + stats: { type: 'boolean', default: false }, + verbose: { type: 'boolean', short: 'v', default: false }, + budget: { type: 'string' }, + session: { type: 'string' }, + help: { type: 'boolean', short: 'h', default: false }, + }, + }); + + // Metrics cost tokens too, so they are opt-in: agents pass --verbose only + // when their own verbose mode is on, or the user exports OC_VERBOSE=1. + const verbose = values.stats || values.verbose || process.env.OC_VERBOSE === '1'; + + const [command, ...args] = positionals; + if (values.help || !command) { + console.log(HELP); + return; + } + + switch (command) { + case 'open': + case 'raw': { + const url = args[0]; + if (!url) throw new Error(`usage: oc ${command} `); + const budget = values.budget ? Number(values.budget) : 500; + if (!Number.isFinite(budget) || budget <= 0) throw new Error('--budget must be a positive number'); + const t0 = performance.now(); + const { url: finalUrl, html, status, via } = await fetchPage(url); + const fetchMs = performance.now() - t0; + const resources = () => { + const processMs = performance.now() - t0 - fetchMs; + const rss = process.memoryUsage().rss; + return `HTTP ${status} via ${via}, fetch ${Math.round(fetchMs)}ms, process ${Math.round(processMs)}ms, ` + + `${Math.round(html.length / 1024)}KB transferred, ${Math.round(rss / 1048576)}MB memory`; + }; + if (values.json) { + console.log(JSON.stringify(distill(html, finalUrl))); + if (verbose) console.error(resources()); + return; + } + const htmlTokens = estimateTokens(html); + if (command === 'raw') { + const out = values.html ? toHTML(html) : toMarkdown(html); + console.log(out); + if (verbose) console.error(`${savings(estimateTokens(out), htmlTokens)}; ${resources()}`); + return; + } + const page = distill(html, finalUrl); + const { text, stats } = render(page, { budget }); + console.log(text); + if (verbose) { + console.error(`~${stats.tokens} tokens, ${stats.rendered}/${stats.blocks} blocks rendered, ${savings(stats.tokens, htmlTokens)}; ${resources()}`); + } + return; + } + case 'do': return act.activate(Number(args[0])); + case 'fill': return act.fill(Number(args[0]), args.slice(1).join(' ')); + case 'submit': return act.submit(args[0] ? Number(args[0]) : undefined); + case 'read': return act.read(args[0] ? Number(args[0]) : undefined); + case 'find': return act.find(args.join(' ')); + case 'back': return act.back(); + case 'next': return act.next(); + case 'session': throw new act.NotImplemented('session'); + default: + throw new Error(`unknown command '${command}', run oc --help`); + } +} + +main().catch((err) => { + console.error(`oc: ${err.message}`); + process.exit(1); +}); diff --git a/src/distill.js b/src/distill.js new file mode 100644 index 0000000..d5b4ffd --- /dev/null +++ b/src/distill.js @@ -0,0 +1,167 @@ +import { parseHTML } from 'linkedom'; +import TurndownService from 'turndown'; + +/** + * @typedef {Object} Block + * @property {'heading'|'text'|'link'|'input'|'button'} type + * @property {string} text + * @property {number} [n] - action handle, only on interactive blocks + * @property {number} [level] - heading level 1..6 + * @property {string} [href] - links only + * @property {string} [name] - inputs only + * + * @typedef {Object} Page + * @property {string} url + * @property {string} title + * @property {Block[]} blocks + */ + +// Dropped wholesale, subtree included. Nav and footer stay in v0.1: on many +// sites they carry the only working links, and the budget in render.js is +// what keeps them from costing anything. +const DROP = new Set([ + 'script', 'style', 'noscript', 'template', 'svg', 'iframe', + 'link', 'meta', 'head', 'canvas', 'video', 'audio', 'object', +]); + +const clean = (s) => s.replace(/\s+/g, ' ').trim(); + +/** + * Reduce raw HTML to an interaction tree: readable text plus numbered + * interactive elements, in document order. Handles are assigned during a + * single deterministic walk, so the same page always yields the same numbers. + * @param {string} html + * @param {string} url + * @returns {Page} + */ +export function distill(html, url = '') { + const { document } = parseHTML(html); + const title = clean(document.querySelector('title')?.textContent ?? ''); + /** @type {Block[]} */ + const blocks = []; + let handle = 0; + + const hidden = (el) => + el.getAttribute('hidden') !== null || + el.getAttribute('aria-hidden') === 'true' || + /display:\s*none/.test(el.getAttribute('style') ?? ''); + + const walk = (node) => { + if (node.nodeType === 3) { + const text = clean(node.textContent); + if (text) blocks.push({ type: 'text', text }); + return; + } + if (node.nodeType !== 1) return; + const tag = node.localName; + if (DROP.has(tag) || hidden(node)) return; + + if (/^h[1-6]$/.test(tag)) { + const text = clean(node.textContent); + if (text) blocks.push({ type: 'heading', level: Number(tag[1]), text }); + return; + } + if (tag === 'a' && node.getAttribute('href')) { + const text = clean(node.textContent); + if (text) { + blocks.push({ type: 'link', n: ++handle, text, href: node.getAttribute('href') }); + } + return; + } + if (tag === 'input' || tag === 'textarea' || tag === 'select') { + const kind = node.getAttribute('type') ?? 'text'; + if (kind === 'hidden') return; + if (kind === 'submit' || kind === 'button') { + blocks.push({ type: 'button', n: ++handle, text: node.getAttribute('value') ?? 'submit' }); + return; + } + const name = node.getAttribute('name') ?? node.getAttribute('placeholder') ?? tag; + blocks.push({ type: 'input', n: ++handle, text: kind, name }); + return; + } + if (tag === 'button') { + const text = clean(node.textContent) || 'button'; + blocks.push({ type: 'button', n: ++handle, text }); + return; + } + for (const child of node.childNodes) walk(child); + }; + + const body = bodyOf(document); + if (body) walk(body); + return { url, title, blocks: mergeText(blocks) }; +} + +/** + * linkedom's document.body getter comes back empty on some real pages (Bing) + * while querySelector finds the populated element, so always resolve the body + * this way. + * @returns {any} + */ +const bodyOf = (document) => document.querySelector('body') ?? document.documentElement; + +/** + * Shared cleanup for the raw modes: parse, then delete the same noise + * distill() skips, so neither raw output ever leaks scripts, styles, or + * hidden content. + * @param {string} html + */ +function cleanDocument(html) { + const { document } = parseHTML(html); + for (const tag of DROP) { + for (const el of [...document.querySelectorAll(tag)]) el.remove(); + } + for (const el of [...document.querySelectorAll('[hidden], [aria-hidden="true"], input[type="hidden"]')]) el.remove(); + for (const el of [...document.querySelectorAll('[style]')]) { + if (/display:\s*none/.test(el.getAttribute('style') ?? '')) el.remove(); + } + return document; +} + +/** + * Whole-page markdown for `oc raw`, produced by turndown so lists, emphasis, + * links, and code blocks come out as real markdown instead of flat lines. + * @param {string} html + * @returns {string} + */ +export function toMarkdown(html) { + const document = cleanDocument(html); + const title = clean(document.querySelector('title')?.textContent ?? ''); + const turndown = new TurndownService({ headingStyle: 'atx', codeBlockStyle: 'fenced' }); + const el = bodyOf(document); + const body = el ? turndown.turndown(el.innerHTML).trim() : ''; + return title && !body.startsWith(`# ${title}`) ? `# ${title}\n\n${body}` : body; +} + +/** + * Whole-page cleaned HTML for `oc raw --html`, for agents that would rather + * work with markup than markdown. Same noise removal, no other rewriting. + * @param {string} html + * @returns {string} + */ +export function toHTML(html) { + const document = cleanDocument(html); + const el = bodyOf(document); + return el ? el.innerHTML.trim() : ''; +} + +/** + * Adjacent text nodes arrive fragmented (one per inline element boundary). + * Merging them is what turns DOM noise into readable lines. + * @param {Block[]} blocks + * @returns {Block[]} + */ +function mergeText(blocks) { + /** @type {Block[]} */ + const out = []; + for (const b of blocks) { + const prev = out[out.length - 1]; + if (b.type === 'text' && prev?.type === 'text') { + prev.text = `${prev.text} ${b.text}`; + } else { + out.push(b); + } + } + // Single stray characters (list bullets, separators) cost tokens and say nothing. + return out.filter((b) => b.type !== 'text' || b.text.length > 1); +} diff --git a/src/fetch.js b/src/fetch.js new file mode 100644 index 0000000..6262e3c --- /dev/null +++ b/src/fetch.js @@ -0,0 +1,68 @@ +/** + * HTTP layer. impers (libcurl-impersonate) presents a real browser TLS and + * HTTP/2 fingerprint so ordinary public pages load the way they would in + * Chrome. It is loaded lazily and native fetch is the silent fallback, so a + * bare `npm install --omit=optional` still gives a working tool. The headless + * fallback for script-gated pages lands in v0.3 and must stay lazy too: + * never import a browser here. + */ + +// The fetch fallback can't fake a TLS fingerprint like impers does, but it +// should at least send the same Chrome identity in its headers. +const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36'; + +/** @type {Promise | null} */ +let impersPromise = null; +const loadImpers = () => { + impersPromise ??= import('impers').catch(() => null); + return impersPromise; +}; + +/** + * Fetch a page. + * @param {string} url - with or without a scheme, https is assumed + * @returns {Promise<{url: string, html: string, status: number, via: string}>} + * final URL after redirects, the body, the HTTP status, and which client + * identity got the page (impers:chrome, impers:firefox, or fetch) + */ +export async function fetchPage(url) { + const target = /^https?:\/\//i.test(url) ? url : `https://${url}`; + const impers = await loadImpers(); + return impers ? viaImpers(impers, target) : viaFetch(target); +} + +async function viaImpers(impers, target) { + // Some sites (Reddit) 403 the chrome fingerprint but accept firefox, so a + // blocked first attempt gets one cheap retry with a second identity. + let via = 'impers:chrome'; + let res = await impers.get(target, { impersonate: 'chrome' }); + // impers mirrors the curl_cffi response shape, not the WHATWG one. + let status = res.status ?? res.statusCode ?? 0; + if (status >= 400) { + via = 'impers:firefox'; + res = await impers.get(target, { impersonate: 'firefox' }); + status = res.status ?? res.statusCode ?? 0; + } + if (status >= 400) throw new Error(`fetch failed: ${status} for ${target}`); + const html = typeof res.text === 'function' ? await res.text() : String(res.text ?? res.body ?? ''); + return { url: res.url ?? target, html, status, via }; +} + +async function viaFetch(target) { + const res = await fetch(target, { + redirect: 'follow', + headers: { + 'user-agent': UA, + accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', + 'accept-language': 'en-US,en;q=0.9', + }, + }); + if (!res.ok) { + throw new Error(`fetch failed: ${res.status} ${res.statusText} for ${target}`); + } + const type = res.headers.get('content-type') ?? ''; + if (type && !type.includes('html')) { + throw new Error(`not an HTML page (${type.split(';')[0]}), nothing to distill`); + } + return { url: res.url, html: await res.text(), status: res.status, via: 'fetch' }; +} diff --git a/src/render.js b/src/render.js new file mode 100644 index 0000000..2cab808 --- /dev/null +++ b/src/render.js @@ -0,0 +1,115 @@ +/** + * Rendering is where the token budget is enforced. Everything printed here + * gets read by a paying model, so the default view is dense and anything + * skipped says so in one line. + */ + +const TEXT_CAP = 200; + +/** + * Rough but stable token estimate. Close enough for budgets; the point is + * that it never changes between runs, not that it matches any one tokenizer. + * @param {string} s + * @returns {number} + */ +export const estimateTokens = (s) => Math.ceil(s.length / 4); + +/** + * Budget-aware compact view of a distilled page. + * @param {import('./distill.js').Page} page + * @param {{budget?: number}} [opts] + * @returns {{text: string, stats: {tokens: number, blocks: number, rendered: number}}} + */ +export function render(page, { budget = 500 } = {}) { + const lines = page.title ? [`# ${page.title}`] : []; + let spent = estimateTokens(lines.join('\n')); + let skipped = 0; + let hasLinks = false; + let hasInputs = false; + + for (const block of collapseRuns(page.blocks)) { + // Rule from PROMPT.md: never print the same content twice. Pages often + // repeat the title as their first heading. + if (block.type === 'heading' && block.text === page.title) continue; + const line = formatBlock(block); + if (!line) continue; + const cost = estimateTokens(line) + 1; + if (spent + cost > budget) { + skipped++; + continue; + } + spent += cost; + lines.push(line); + if (block.type === 'link' || block.type === 'button') hasLinks = true; + if (block.type === 'input') hasInputs = true; + } + + if (skipped) lines.push(`... ${skipped} more blocks over budget, raise --budget or use oc raw`); + const actions = [ + hasLinks && 'do ', + hasInputs && 'fill ', + hasInputs && 'submit', + 'raw ', + ].filter(Boolean); + lines.push(`actions: ${actions.join(' | ')}`); + + const text = lines.join('\n'); + return { + text, + stats: { tokens: estimateTokens(text), blocks: page.blocks.length, rendered: page.blocks.length - skipped }, + }; +} + +/** + * The "collapse repeated siblings" rule from PROMPT.md. Long runs of short + * links are almost always nav chrome (subreddit bars, tag clouds, footers) + * and would otherwise eat the whole budget before the content starts. Handles + * are assigned in distill, so the hidden links keep their numbers and the + * marker names the range. + * @param {import('./distill.js').Block[]} blocks + * @returns {import('./distill.js').Block[]} + */ +function collapseRuns(blocks) { + const SHORT = 20; + const RUN = 8; + const KEEP = 5; + const out = []; + let i = 0; + while (i < blocks.length) { + let j = i; + while (j < blocks.length && blocks[j].type === 'link' && blocks[j].text.length <= SHORT) j++; + const run = j - i; + if (run > RUN) { + out.push(...blocks.slice(i, i + KEEP)); + const first = blocks[i + KEEP]; + const last = blocks[j - 1]; + out.push({ type: 'text', text: `[${first.n}-${last.n}] ${run - KEEP} similar links, expand with oc raw` }); + i = j; + } else { + out.push(blocks[i]); + i++; + } + } + return out; +} + +/** + * @param {import('./distill.js').Block} b + * @returns {string} + */ +function formatBlock(b) { + switch (b.type) { + case 'heading': + return `${'#'.repeat(Math.min(b.level ?? 2, 3))} ${b.text}`; + case 'link': + return `[${b.n}] ${truncate(b.text)}`; + case 'button': + return `[${b.n}] button "${truncate(b.text)}"`; + case 'input': + return `[${b.n}] input ${b.name} (${b.text})`; + default: + return truncate(b.text); + } +} + +const truncate = (s) => (s.length > TEXT_CAP ? `${s.slice(0, TEXT_CAP)} ...` : s); diff --git a/src/session.js b/src/session.js new file mode 100644 index 0000000..3471afc --- /dev/null +++ b/src/session.js @@ -0,0 +1,16 @@ +/** + * Sessions are plain JSON files on disk, one per name: current URL, cookies, + * the last distilled page so actions can resolve handles, and history. No + * daemon, no background process. Read/write lands in v0.2 with act.js. + */ + +import { homedir } from 'node:os'; +import { join } from 'node:path'; + +export const SESSION_DIR = join(homedir(), '.only-cli', 'sessions'); + +/** + * @param {string} name + * @returns {string} + */ +export const sessionPath = (name) => join(SESSION_DIR, `${name}.json`); diff --git a/tests/distill.test.js b/tests/distill.test.js new file mode 100644 index 0000000..7e1ebf2 --- /dev/null +++ b/tests/distill.test.js @@ -0,0 +1,79 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { distill, toMarkdown, toHTML } from '../src/distill.js'; +import { render, estimateTokens } from '../src/render.js'; + +const html = readFileSync(new URL('./pages/news.html', import.meta.url), 'utf8'); +const page = () => distill(html, 'https://example.test/news'); + +test('noise never reaches the output, compact or raw', () => { + for (const out of [render(page(), { budget: 5000 }).text, toMarkdown(html), toHTML(html)]) { + assert.ok(!out.includes('tracker'), 'script content leaked'); + assert.ok(!out.includes('font-family'), 'style content leaked'); + assert.ok(!out.includes('cookies'), 'display:none content leaked'); + assert.ok(!out.includes('hidden drawer'), 'hidden attribute content leaked'); + assert.ok(!out.includes('csrf'), 'hidden input leaked'); + } +}); + +test('raw mode emits real markdown with hrefs an agent can follow', () => { + const md = toMarkdown(html); + assert.ok(md.startsWith('# Fixture News')); + assert.ok(md.includes('[Show HN: I built a tiny CSV toolkit](/item?id=1)'), 'link markdown missing'); +}); + +test('raw html mode keeps markup', () => { + const out = toHTML(html); + assert.ok(out.includes(''), 'anchor tag missing'); + assert.ok(!out.includes(' { + const p = page(); + const links = p.blocks.filter((b) => b.type === 'link'); + assert.equal(links[0].n, 1); + assert.equal(links[0].text, 'Show HN: I built a tiny CSV toolkit'); + const input = p.blocks.find((b) => b.type === 'input'); + assert.equal(input.name, 'q'); + const button = p.blocks.find((b) => b.type === 'button'); + assert.equal(button.text, 'Search'); +}); + +test('same page yields the same output', () => { + assert.equal(render(page()).text, render(page()).text); +}); + +test('render respects the token budget', () => { + const { text, stats } = render(page(), { budget: 100 }); + assert.ok(stats.tokens <= 120, `render cost ~${stats.tokens} tokens against a budget of 100`); + assert.ok(text.includes('over budget'), 'skipped blocks must be announced'); +}); + +test('default render of a normal page fits the 500 token target', () => { + const { stats } = render(page()); + assert.ok(stats.tokens <= 500, `render cost ~${stats.tokens} tokens`); +}); + +test('long text is truncated with a marker', () => { + const { text } = render(page(), { budget: 2000 }); + assert.ok(text.includes(' ...'), 'expected a truncation marker'); + assert.ok(!text.includes('safely does'), 'text cap was not applied'); +}); + +test('title becomes the page heading', () => { + assert.ok(render(page()).text.startsWith('# Fixture News')); +}); + +test('token estimate is stable and roughly chars over four', () => { + assert.equal(estimateTokens('abcdefgh'), 2); +}); + +test('long runs of short links collapse into a range marker', () => { + const nav = Array.from({ length: 15 }, (_, i) => `sub${i}`).join(' '); + const navHtml = `T${nav}

actual content

`; + const { text } = render(distill(navHtml, 'https://x.test'), { budget: 2000 }); + assert.ok(text.includes('[6-15] 10 similar links'), `run not collapsed:\n${text}`); + assert.ok(text.includes('actual content'), 'content after the run was lost'); + assert.ok(!text.includes('sub9'), 'collapsed link still rendered'); +}); diff --git a/tests/pages/news.html b/tests/pages/news.html new file mode 100644 index 0000000..fef8d2c --- /dev/null +++ b/tests/pages/news.html @@ -0,0 +1,32 @@ + + + + Fixture News + + + + + + + +

Fixture News

+
    +
  1. Show HN: I built a tiny CSV toolkit 312 points 87 comments
    2. +
  2. Postgres 18 released 540 points 203 comments
    3. +
  3. Why terminals still win 128 points 45 comments
    4. +
+

About

+

+ This paragraph exists to test truncation. It repeats a long filler sentence several times so the + renderer has something to cap. The quick brown fox jumps over the lazy dog and keeps jumping + because the fixture needs enough characters to cross the two hundred character text cap used by + the compact renderer, which this sentence now safely does. +

+
+ + + +
+ newest + +