External/oc
1
0
Fork 0
mirror of https://github.com/only-cli/oc.git synced 2026-09-15 10:40:56 +02:00
Code Issues Packages Projects Releases Wiki Activity

add oc next and oc read <n> so a long page costs a screenful, not a refetch

Browse Source 
The compact view was all or nothing: an agent that needed more than the 500
token budget had only oc raw, ten to twenty times the price. Now open saves
the distilled page, next continues it where the view stopped, and read <n>
prints one region in full. Headings and text blocks long enough to be cut are
numbered so they can be addressed, and the marker prices what it left behind.

On one Reddit thread: open 475 tokens, next 455, read 88, raw 9,670.
This commit is contained in:
only-cli
2026-08-18 16:11:46 -04:00
parent 79ddeebc2e
commit 0f7a38d44a
11 changed files with 449 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+25 -5
View File
@@ -10,7 +10,7 @@ $ oc open news.ycombinator.com
[1] Show HN: I built a tiny CSV toolkit
[2] 312 comments
...
actions: do <n> | raw <url>
actions: do <n> | read <n> | next | raw
$ oc do 1
```
@@ -44,14 +44,34 @@ No setup at all also works: `npx @only-cli/oc` runs without a global install, an
```
oc open <url> fetch and render a page with numbered actions
oc do <n> follow the numbered link [n] from the last page
oc raw <url> distilled markdown of the whole page
oc next the next budget worth of the page already open
oc read <n> full text of the region at [n], up to 2000 tokens
oc raw [url] distilled markdown of the whole page
oc fill <n> <text> type into a numbered input (v0.2)
oc submit [n] submit a form (v0.2)
```
Flags: `--budget <tokens>` (default 500), `--json`, `--html` (raw as cleaned HTML instead of markdown), `--session <name>` (separate page state per name), `--verbose`/`-v` (metrics on stderr: tokens saved vs the page HTML, HTTP status and client identity, timing, transfer size, memory; alias `--stats`, or export `OC_VERBOSE=1`). Metrics are off by default because they cost tokens too; agents should pass `--verbose` only when running verbosely.
Flags: `--budget <tokens>` (default 500, 2000 for `read`), `--json`, `--html` (raw as cleaned HTML instead of markdown), `--session <name>` (separate page state per name), `--verbose`/`-v` (metrics on stderr: tokens saved vs the page HTML, HTTP status and client identity, timing, transfer size, memory; alias `--stats`, or export `OC_VERBOSE=1`). Metrics are off by default because they cost tokens too; agents should pass `--verbose` only when running verbosely.
`oc open` remembers the elements it numbered, so `oc do 3` follows `[3]` without the agent ever handling a URL. That state is a small JSON file per session under `~/.only-cli` (override the directory with `OC_HOME`); there is no daemon and no background browser. Links that search engines wrap in a tracking redirect resolve to the real destination, so `do` on a result works like a click.
`oc open` remembers the page it rendered, so `oc do 3` follows `[3]` without the agent ever handling a URL. That state is a JSON file per session under `~/.only-cli` (override the directory with `OC_HOME`); there is no daemon and no background browser. Links that search engines wrap in a tracking redirect resolve to the real destination, so `do` on a result works like a click.
### Reading past the budget
A 500 token view of a long page is cheap on the first read and expensive on the second, if the only way to see more is the whole page again. So the view says what it left behind, and there are two ways to collect it that are not the whole page:
```
$ oc open https://old.reddit.com/r/linuxquestions/comments/xpznb1/best_terminal_web_browser/
...
[80] Kind of a weird question, but does anyone here use a terminal browser to ... +312 chars
... 570 more blocks (~4,362 tokens): 'oc next' for the next ~500, 'oc raw' for all
actions: do <n> | read <n> | next | raw
$ oc next # the next ~455 tokens, continuing exactly where the view stopped
$ oc read 80 # that one comment in full, 88 tokens
$ oc raw # the whole thread, 9,670 tokens
```
Neither `next` nor `read` fetches anything: `open` saves the distilled page, so continuing to read it costs one file read. `read <n>` takes one region, which is the block at `[n]` with the couple of blocks that lead into it, or the whole section when `[n]` is a heading. Headings and text blocks long enough to be cut get numbers for exactly this reason, and `... +312 chars` on a line is how much of that block the view did not print.
## Supported websites
@@ -119,7 +139,7 @@ The same six tasks run through OpenAI's Codex CLI (`codex exec`) as well, where
## Status
Early. v0.1 covers static pages, budget-aware rendering, and offline tests. Sessions and `oc do <n>` are in, the rest of the actions (`fill`, `submit`, `read`, `find`, `back`) land in v0.2, and a lazy headless fallback for script-heavy pages in v0.3. The design principles and how to contribute are in [CONTRIBUTING.md](CONTRIBUTING.md).
Early. v0.1 covers static pages, budget-aware rendering, and offline tests. Sessions, `oc do <n>`, `oc next`, and `oc read <n>` are in, the rest of the actions (`fill`, `submit`, `find`, `back`) land in v0.2, and a lazy headless fallback for script-heavy pages in v0.3. The design principles and how to contribute are in [CONTRIBUTING.md](CONTRIBUTING.md).
Known limits, honestly: no JavaScript rendering yet, no sites behind logins yet, and pages behind hard bot challenges may still refuse the tool.
llms.txt
+2 -2
View File
@@ -5,8 +5,8 @@
Key facts:
- Install: `npm install -g @only-cli/oc`, or zero-install with `npx @only-cli/oc`
- Commands: `oc open <url>` (compact view with numbered actions), `oc raw <url>` (whole page as markdown, `--html` for cleaned HTML), `oc --help` for the full surface
- Default output budget is 500 tokens per page; `--budget <n>` adjusts it
- Commands: `oc open <url>` (compact view with numbered actions), `oc do <n>` (follow numbered link [n]), `oc next` (the next screenful of the page already open), `oc read <n>` (one region in full), `oc raw [url]` (whole page as markdown, `--html` for cleaned HTML), `oc --help` for the full surface
- Default output budget is 500 tokens per page; `--budget <n>` adjusts it, and `next` or `read <n>` collect what the budget cut without refetching the page
- Benchmarked at roughly 45x fewer tokens than reading raw HTML, with per-task numbers at https://github.com/only-cli/benchmarks
- Works on any mostly-static website; tuned shortcuts ship for Hacker News, Reddit, GitHub, LinkedIn (public guest views), DuckDuckGo, and Bing
- Requests impersonate Chrome, so pages that block plain scripts often still work
package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@only-cli/oc",
"version": "0.2.0-beta.1",
"version": "0.2.0-beta.2",
"description": "Turn websites into a compact CLI so AI agents can browse without burning tokens.",
"type": "module",
"bin": {
skills/only-cli/SKILL.md
+24 -4
View File
@@ -12,15 +12,34 @@ No install needed, run it with npx:
```
npx @only-cli/oc open <url> compact view with numbered elements
npx @only-cli/oc do <n> follow numbered link [n] from the last page
npx @only-cli/oc raw <url> whole page as markdown (add --html for cleaned HTML)
npx @only-cli/oc next the next ~500 tokens of the page already open
npx @only-cli/oc read <n> full text of the region at [n]
npx @only-cli/oc raw [url] whole page as markdown (add --html for cleaned HTML)
```
## Reading the output
- The first line is the page title, then headings, text, and interactive elements in page order.
- `[n]` marks a link, button, or input.
- `[n]` marks a link, button, input, heading, or a text block long enough to be cut.
- `... +820 chars` at the end of a line means that block was cut there. `read <n>` prints it whole.
- `... 164 more blocks (~7,100 tokens)` means the page ran past the budget. That is the price of the rest, so you can decide before paying.
- The `actions:` line at the bottom lists valid next commands.
- `... N more blocks over budget` means content was cut to stay cheap. Rerun with `--budget 1500` if you need more, or use `raw` for everything.
## Reading more of a page
Three ways to go past the first view, cheapest first. Pick by what you need, not by habit.
- `oc next` prints the next budget worth of the same page and remembers where it stopped, so calling it again continues. This is the right answer for "the answer is further down": a comment thread, a long article, a list that ran past the cut.
- `oc read <n>` prints one region in full: the block at `[n]` with a little context, or the whole section when `[n]` is a heading. Use it when the view shows you exactly the block you want and it was cut.
- `oc raw` (no URL needed once a page is open) prints everything. It costs an order of magnitude more, so use it when you genuinely need the whole page.
Measured on one Reddit thread: `open` 475 tokens, each `next` about 455, one `read` 88, `raw` 9,670. Neither `next` nor `read` fetches anything; they work from the page `open` already saved.
```
oc open https://old.reddit.com/r/linuxquestions/comments/xpznb1/best_terminal_web_browser/
oc next -> the comments, 455 tokens at a time
oc read 180 -> that one comment in full, 88 tokens
```
## Following links
@@ -37,13 +56,14 @@ Notes that save a round trip:
- Handles hidden behind a `[6-9] 4 similar links` marker still work, even though their text was collapsed.
- Search result links resolve to the destination, not the search engine's tracking redirect.
- `do` on an input or a button says so; typing and submitting are not available yet.
- `do` on a heading or a text block says to use `read <n>` instead.
- `--session <name>` keeps separate page state, for working on two sites at once.
Reach for `raw <url>` when you need the whole page text, not to hunt for a URL.
## Flags
- `--budget <tokens>` raise or lower the render budget (default 500)
- `--budget <tokens>` raise or lower the render budget (default 500, 2000 for `read`)
- `--json` machine-stable JSON of the distilled page
- `--html` with raw: cleaned HTML instead of markdown, if markup suits your task better
- `--verbose` (`-v`, alias `--stats`) metrics on stderr: tokens saved vs the page HTML, HTTP status and which client identity got the page, fetch and processing time, bytes transferred, memory use. Only pass this when you are running in verbose mode or diagnosing a problem; the metrics line costs tokens like everything else. Users can export `OC_VERBOSE=1` to turn it on globally.
src/act.js
+127 -18
View File
@@ -1,10 +1,15 @@
/**
* Actions against a saved session: do now, fill, submit, read, find, back,
* and next still ahead. Errors here are read by agents, so every one of them
* Actions against a saved session: do, read, and next now, fill, submit, find,
* and back still ahead. Errors here are read by agents, so every one of them
* names the command to run next.
*
* Nothing in this file touches the network. The page an agent is working on is
* already on disk from the last `oc open`, so continuing to read it costs one
* file read and whatever tokens the caller asked for.
*/
import { DEFAULT_SESSION, loadSession } from './session.js';
import { DEFAULT_SESSION, handleFor, handleNumbers, loadSession, saveSession } from './session.js';
import { estimateTokens, formatBlock, render } from './render.js';
export class NotImplemented extends Error {
constructor(command) {
@@ -12,6 +17,31 @@ export class NotImplemented extends Error {
}
}
/**
* @param {string} session
* @returns {any}
*/
function requireSession(session) {
const state = loadSession(session);
if (!state) {
throw new Error("nothing open in this session yet, run 'oc open <url>' first");
}
return state;
}
/**
* `read` and `next` work from the saved blocks, which older sessions do not
* have. One `oc open` fixes it, so say that instead of failing blankly.
* @param {string} session
*/
function requireBlocks(session) {
const state = requireSession(session);
if (!Array.isArray(state.blocks)) {
throw new Error(`this session was saved by an older oc, run 'oc open ${state.url}' again`);
}
return state;
}
/**
* Resolve a numbered handle from the last render into something to open.
* Returns the target URL; the caller fetches and renders it exactly as
@@ -24,16 +54,16 @@ export function activate(n, { session = DEFAULT_SESSION } = {}) {
if (!Number.isInteger(n) || n < 1) {
throw new Error('usage: oc do <n>, where <n> is a number from the last page');
}
const state = loadSession(session);
if (!state) {
throw new Error("nothing open in this session yet, run 'oc open <url>' first");
}
const handle = state.handles?.[n];
const state = requireSession(session);
const handle = handleFor(state, n);
if (!handle) {
const nums = Object.keys(state.handles ?? {}).map(Number);
const nums = handleNumbers(state);
const range = nums.length ? `1-${Math.max(...nums)}` : 'none';
throw new Error(`no [${n}] on ${state.url} (handles ${range}), run 'oc open <url>' again to renumber`);
}
if (handle.type === 'text' || handle.type === 'heading') {
throw new Error(`[${n}] is ${handle.type}, not a link, use 'oc read ${n}' for the full text there`);
}
if (handle.type === 'input') {
throw new Error(`[${n}] is an input (${handle.name ?? 'text'}), typing needs 'oc fill', which is not available yet`);
}
@@ -43,6 +73,94 @@ export function activate(n, { session = DEFAULT_SESSION } = {}) {
return { url: handle.href, text: handle.text };
}
// How many blocks of run-up `read` prints before the one it was asked for, so
// a comment or paragraph arrives with the byline that introduces it, and how
// many it prints after. The trailing window only applies when the target is
// not a heading: a heading owns its whole section, anything else is one thing
// the agent asked to see in full and a little of what follows it.
const LEAD = 2;
const TRAIL = 6;
/**
* Full text of one region of the current page: the block at [n], the couple of
* blocks that lead into it, and either the rest of its section when [n] is a
* heading or a short run after it when it is not. This is the middle setting
* between the 500 token view and the whole page.
* @param {number} n
* @param {{session?: string, budget?: number}} [opts]
* @returns {string}
*/
export function read(n, { session = DEFAULT_SESSION, budget = 2000 } = {}) {
if (!Number.isInteger(n) || n < 1) {
throw new Error('usage: oc read <n>, where <n> is a number from the last page');
}
const state = requireBlocks(session);
const blocks = state.blocks;
const at = blocks.findIndex((b) => b.n === n);
if (at < 0) {
const nums = handleNumbers(state);
const range = nums.length ? `1-${Math.max(...nums)}` : 'none';
throw new Error(`no [${n}] on ${state.url} (handles ${range}), run 'oc open <url>' again to renumber`);
}
const isHeading = blocks[at].type === 'heading';
let start = isHeading ? at : Math.max(0, at - LEAD);
// Never open with the tail of the section before this one.
for (let i = at - 1; i > start; i--) {
if (blocks[i].type === 'heading') {
start = i;
break;
}
}
// A heading owns everything down to the next heading of its level or above;
// anything else runs to the next heading of any level.
const stopLevel = blocks[start].type === 'heading' ? (blocks[start].level ?? 2) : 6;
const end = isHeading ? blocks.length : Math.min(blocks.length, at + TRAIL + 1);
const lines = [];
let spent = 0;
let i = start;
for (; i < end; i++) {
const block = blocks[i];
if (i > start && block.type === 'heading' && (block.level ?? 2) <= stopLevel) break;
const line = formatBlock(block, { full: true });
if (!line) continue;
const cost = estimateTokens(line) + 1;
if (spent + cost > budget && lines.length) break;
spent += cost;
lines.push(line);
}
const stoppedEarly = i < end && !(blocks[i].type === 'heading' && (blocks[i].level ?? 2) <= stopLevel);
if (stoppedEarly) {
const resume = blocks.slice(i).find((b) => b.n != null)?.n;
const how = resume ? `continue with 'oc read ${resume}'` : "use 'oc raw' for the rest";
lines.push(`... region cut at ~${budget} tokens, ${how} or raise --budget`);
}
return lines.join('\n');
}
/**
* The next budget worth of the page the session already holds. `oc open` says
* how many blocks it left behind; this is how an agent takes them a screenful
* at a time instead of paying for the whole page to get one more paragraph.
* @param {{session?: string, budget?: number}} [opts]
* @returns {string}
*/
export function next({ session = DEFAULT_SESSION, budget = 500 } = {}) {
const state = requireBlocks(session);
const from = state.cursor;
if (from == null) {
return `end of ${state.url}, nothing left to render. 'oc open <url>' to reload it, 'oc raw' for the whole page.`;
}
const page = { url: state.url, title: state.title, blocks: state.blocks };
const { text, stats } = render(page, { budget, from });
try {
saveSession(session, { ...state, cursor: stats.next });
} catch {}
return text;
}
/** @param {number} n @param {string} text */
export function fill(n, text) {
throw new NotImplemented('fill');
@@ -53,11 +171,6 @@ export function submit(n) {
throw new NotImplemented('submit');
}
/** @param {number} [n] */
export function read(n) {
throw new NotImplemented('read');
}
/** @param {string} query */
export function find(query) {
throw new NotImplemented('find');
@@ -66,7 +179,3 @@ export function find(query) {
export function back() {
throw new NotImplemented('back');
}
export function next() {
throw new NotImplemented('next');
}
src/cli.js
+26 -16
View File
@@ -11,17 +11,19 @@ const HELP = `only-cli: the web as a compact terminal, built for AI agents.
usage: oc <command> [args] [flags]
open <url> fetch and render a page with numbered actions
raw <url> distilled markdown of the whole page
next the next budget worth of the page already open
read <n> full text of the region at [n], up to 2000 tokens
raw [url] distilled markdown of the whole page
do <n> follow the numbered link [n] from the last page
fill <n> <text> type into a numbered input (v0.2)
submit [n] submit a form (v0.2)
read [n] full text of one region (v0.2)
find <query> search visible text on the current page (v0.2)
back | next history and pagination (v0.2)
back return to the previous page (v0.2)
session ls|rm manage saved sessions (v0.2)
flags:
--budget <tokens> tighten or loosen the render budget (default 500)
--budget <tokens> tighten or loosen the render budget (default 500,
2000 for read)
--json machine-stable JSON output
--html raw only: cleaned HTML instead of markdown
--verbose, -v metrics on stderr: tokens saved vs the page HTML, HTTP
@@ -30,16 +32,17 @@ flags:
globally. Off by default because metrics cost tokens too.
--session <name> keep separate page state under a name (default: default)
'oc open' remembers the numbered elements it printed, so 'oc do 3' follows
link [3] without you ever handling its URL. State lives in ~/.only-cli
'oc open' remembers the page it printed, so 'oc do 3' follows link [3] without
you ever handling its URL, and 'oc next' or 'oc read 12' picks up what the
budget left behind without fetching it again. State lives in ~/.only-cli
(override with OC_HOME).`;
// A rendered page has to be remembered or its [3] means nothing to the next
// command. Saving state must never break a render, so a home directory that
// cannot be written costs the agent `do` and nothing else.
const remember = (page, name) => {
// cannot be written costs the agent `do`, `read`, and `next`, and nothing else.
const remember = (page, name, cursor) => {
try {
saveSession(name, sessionFromPage(page, loadSession(name)));
saveSession(name, sessionFromPage(page, loadSession(name), { cursor }));
} catch {}
};
@@ -72,19 +75,26 @@ async function main() {
}
const sessionName = values.session || DEFAULT_SESSION;
// Zero means "whatever this command's default is", which differs: the
// compact view targets 500 tokens, read targets 2000.
const asked = values.budget ? Number(values.budget) : 0;
if (values.budget && (!Number.isFinite(asked) || asked <= 0)) {
throw new Error('--budget must be a positive number');
}
switch (command) {
case 'open':
case 'do':
case 'raw': {
// `do` is `open` with the URL looked up from the last render instead of
// typed, so both commands share one fetch, render, and save path.
// typed, so both commands share one fetch, render, and save path. `raw`
// with no URL means the page already open, which is what the compact
// view's footer offers when it has cut something.
const url = command === 'do'
? act.activate(Number(args[0]), { session: sessionName }).url
: args[0];
: args[0] ?? (command === 'raw' ? loadSession(sessionName)?.url : undefined);
if (!url) throw new Error(`usage: oc ${command} <url>`);
const budget = values.budget ? Number(values.budget) : 500;
if (!Number.isFinite(budget) || budget <= 0) throw new Error('--budget must be a positive number');
const budget = asked || 500;
const t0 = performance.now();
const { url: finalUrl, html, status, via } = await fetchPage(url);
const fetchMs = performance.now() - t0;
@@ -109,20 +119,20 @@ async function main() {
return;
}
const page = distill(html, finalUrl);
remember(page, sessionName);
const { text, stats } = render(page, { budget });
remember(page, sessionName, stats.next);
console.log(text);
if (verbose) {
console.error(`~${stats.tokens} tokens, ${stats.rendered}/${stats.blocks} blocks rendered, ${savings(stats.tokens, htmlTokens)}; ${resources()}`);
}
return;
}
case 'read': return console.log(act.read(Number(args[0]), { session: sessionName, budget: asked || 2000 }));
case 'next': return console.log(act.next({ session: sessionName, budget: asked || 500 }));
case 'fill': return act.fill(Number(args[0]), args.slice(1).join(' '));
case 'submit': return act.submit(args[0] ? Number(args[0]) : undefined);
case 'read': return act.read(args[0] ? Number(args[0]) : undefined);
case 'find': return act.find(args.join(' '));
case 'back': return act.back();
case 'next': return act.next();
case 'session': throw new act.NotImplemented('session');
default:
throw new Error(`unknown command '${command}', run oc --help`);
src/distill.js
+29 -9
View File
@@ -5,7 +5,7 @@ import TurndownService from 'turndown';
* @typedef {Object} Block
* @property {'heading'|'text'|'link'|'input'|'button'} type
* @property {string} text
* @property {number} [n] - action handle, only on interactive blocks
* @property {number} [n] - action handle
* @property {number} [level] - heading level 1..6
* @property {string} [href] - links only
* @property {string} [name] - inputs only
@@ -16,6 +16,10 @@ import TurndownService from 'turndown';
* @property {Block[]} blocks
*/
// Where the compact view cuts a text block. It lives here because numbering
// depends on it: a block long enough to be cut is a block that needs a handle.
export const TEXT_CAP = 200;
// Dropped wholesale, subtree included. Nav and footer stay in v0.1: on many
// sites they carry the only working links, and the budget in render.js is
// what keeps them from costing anything.
@@ -28,8 +32,9 @@ const clean = (s) => s.replace(/\s+/g, ' ').trim();
/**
* Reduce raw HTML to an interaction tree: readable text plus numbered
* interactive elements, in document order. Handles are assigned during a
* single deterministic walk, so the same page always yields the same numbers.
* elements, in document order. The walk is deterministic and numbering is a
* second pass over its result, so the same page always yields the same
* numbers.
* @param {string} html
* @param {string} url
* @returns {Page}
@@ -39,7 +44,6 @@ export function distill(html, url = '') {
const title = clean(document.querySelector('title')?.textContent ?? '');
/** @type {Block[]} */
const blocks = [];
let handle = 0;
const hidden = (el) =>
el.getAttribute('hidden') !== null ||
@@ -64,7 +68,7 @@ export function distill(html, url = '') {
if (tag === 'a' && node.getAttribute('href')) {
const text = clean(node.textContent);
if (text) {
blocks.push({ type: 'link', n: ++handle, text, href: node.getAttribute('href') });
blocks.push({ type: 'link', text, href: node.getAttribute('href') });
}
return;
}
@@ -72,16 +76,16 @@ export function distill(html, url = '') {
const kind = node.getAttribute('type') ?? 'text';
if (kind === 'hidden') return;
if (kind === 'submit' || kind === 'button') {
blocks.push({ type: 'button', n: ++handle, text: node.getAttribute('value') ?? 'submit' });
blocks.push({ type: 'button', text: node.getAttribute('value') ?? 'submit' });
return;
}
const name = node.getAttribute('name') ?? node.getAttribute('placeholder') ?? tag;
blocks.push({ type: 'input', n: ++handle, text: kind, name });
blocks.push({ type: 'input', text: kind, name });
return;
}
if (tag === 'button') {
const text = clean(node.textContent) || 'button';
blocks.push({ type: 'button', n: ++handle, text });
blocks.push({ type: 'button', text });
return;
}
for (const child of node.childNodes) walk(child);
@@ -89,7 +93,23 @@ export function distill(html, url = '') {
const body = bodyOf(document);
if (body) walk(body);
return { url, title, blocks: mergeText(blocks) };
return { url, title, blocks: number(mergeText(blocks)) };
}
/**
* Assign handles in document order. Interactive elements get one because they
* can be acted on, headings and long text blocks because they can be read:
* a text block over the cap is printed cut, and its number is what makes the
* rest of it reachable with `oc read <n>` instead of a second whole-page fetch.
* @param {Block[]} blocks
* @returns {Block[]}
*/
function number(blocks) {
let handle = 0;
for (const block of blocks) {
if (block.type !== 'text' || block.text.length > TEXT_CAP) block.n = ++handle;
}
return blocks;
}
/**
src/render.js
+54 -24
View File
@@ -1,10 +1,10 @@
/**
* Rendering is where the token budget is enforced. Everything printed here
* gets read by a paying model, so the default view is dense and anything
* skipped says so in one line.
* gets read by a paying model, so the default view is dense, anything cut says
* how much was cut, and the footer names the cheapest command that gets it.
*/
const TEXT_CAP = 200;
import { TEXT_CAP } from './distill.js';
/**
* Rough but stable token estimate. Close enough for budgets; the point is
@@ -14,49 +14,73 @@ const TEXT_CAP = 200;
*/
export const estimateTokens = (s) => Math.ceil(s.length / 4);
const num = (v) => v.toLocaleString('en-US');
/**
* Budget-aware compact view of a distilled page.
* Budget-aware compact view of a distilled page. `from` is a position in the
* collapsed block list, which is how `oc next` resumes a page where the last
* render stopped.
* @param {import('./distill.js').Page} page
* @param {{budget?: number}} [opts]
* @returns {{text: string, stats: {tokens: number, blocks: number, rendered: number}}}
* @param {{budget?: number, from?: number}} [opts]
* @returns {{text: string, stats: {tokens: number, blocks: number, rendered: number, next: number|null, left: number, leftTokens: number}}}
*/
export function render(page, { budget = 500 } = {}) {
const lines = page.title ? [`# ${page.title}`] : [];
export function render(page, { budget = 500, from = 0 } = {}) {
const blocks = collapseRuns(page.blocks);
const head = page.title ? [from > 0 ? `# ${page.title} (continued)` : `# ${page.title}`] : [];
const lines = [...head];
let spent = estimateTokens(lines.join('\n'));
let skipped = 0;
let hasLinks = false;
let hasInputs = false;
let i = Math.max(0, from);
for (const block of collapseRuns(page.blocks)) {
for (; i < blocks.length; i++) {
const block = blocks[i];
// Never print the same content twice. Pages often repeat the title as
// their first heading.
if (block.type === 'heading' && block.text === page.title) continue;
const line = formatBlock(block);
if (!line) continue;
const cost = estimateTokens(line) + 1;
if (spent + cost > budget) {
skipped++;
continue;
}
// Stop at the first block that does not fit instead of skipping past it:
// what is left has to stay one contiguous run for `oc next` to continue.
// The exception is a first block bigger than the whole budget, which is
// printed anyway so the cursor always moves.
if (spent + cost > budget && lines.length > head.length) break;
spent += cost;
lines.push(line);
if (block.type === 'link' || block.type === 'button') hasLinks = true;
if (block.type === 'input') hasInputs = true;
}
if (skipped) lines.push(`... ${skipped} more blocks over budget, raise --budget or use oc raw`);
const rest = blocks.slice(i);
const leftTokens = rest.reduce((sum, b) => {
const line = formatBlock(b);
return line ? sum + estimateTokens(line) + 1 : sum;
}, 0);
if (rest.length) {
lines.push(`... ${num(rest.length)} more blocks (~${num(leftTokens)} tokens): 'oc next' for the next ~${num(budget)}, 'oc raw' for all`);
}
const actions = [
hasLinks && 'do <n>',
hasInputs && 'fill <n> <text>',
hasInputs && 'submit',
'raw <url>',
'read <n>',
rest.length && 'next',
'raw',
].filter(Boolean);
lines.push(`actions: ${actions.join(' | ')}`);
const text = lines.join('\n');
return {
text,
stats: { tokens: estimateTokens(text), blocks: page.blocks.length, rendered: page.blocks.length - skipped },
stats: {
tokens: estimateTokens(text),
blocks: blocks.length,
rendered: i - Math.max(0, from),
next: rest.length ? i : null,
left: rest.length,
leftTokens,
},
};
}
@@ -94,22 +118,28 @@ function collapseRuns(blocks) {
}
/**
* One line per block. `full` keeps the whole text, which is what `oc read`
* prints; the compact view cuts at TEXT_CAP and says how many characters went
* with the cut, so the agent can price the rest before asking for it.
* @param {import('./distill.js').Block} b
* @param {{full?: boolean}} [opts]
* @returns {string}
*/
function formatBlock(b) {
export function formatBlock(b, { full = false } = {}) {
const tag = b.n == null ? '' : `[${b.n}] `;
switch (b.type) {
case 'heading':
return `${'#'.repeat(Math.min(b.level ?? 2, 3))} ${b.text}`;
return `${'#'.repeat(Math.min(b.level ?? 2, 3))} ${tag}${b.text}`;
case 'link':
return `[${b.n}] ${truncate(b.text)}`;
return `${tag}${full ? b.text : truncate(b.text)}`;
case 'button':
return `[${b.n}] button "${truncate(b.text)}"`;
return `${tag}button "${full ? b.text : truncate(b.text)}"`;
case 'input':
return `[${b.n}] input ${b.name} (${b.text})`;
return `${tag}input ${b.name} (${b.text})`;
default:
return truncate(b.text);
return full ? `${tag}${b.text}` : `${tag}${truncate(b.text)}`;
}
}
const truncate = (s) => (s.length > TEXT_CAP ? `${s.slice(0, TEXT_CAP)} ...` : s);
const truncate = (s) =>
s.length > TEXT_CAP ? `${s.slice(0, TEXT_CAP)} ... +${num(s.length - TEXT_CAP)} chars` : s;
src/session.js
+60 -15
View File
@@ -1,11 +1,12 @@
/**
* Sessions are plain JSON files on disk, one per name: the current URL, the
* numbered handles from the last render so actions can resolve them, and a
* short history. No daemon, no background process, no cookies yet.
* distilled blocks of the page it holds, how far the last render got through
* them, and a short history. No daemon, no background process, no cookies yet.
*
* The file exists so `oc do <n>` can follow a link the compact view never
* printed the URL of. Hiding URLs is what makes `oc open` cheap; this is what
* makes hiding them free.
* makes hiding them free. Keeping the blocks costs disk, not tokens, and it is
* what lets `oc read` and `oc next` answer without fetching the page again.
*/
import { homedir } from 'node:os';
@@ -61,28 +62,72 @@ export function resolveHref(href, base) {
const HISTORY_LIMIT = 20;
// A ceiling on what one page may leave on disk. Nothing real comes close;
// it is here so a runaway page cannot fill a home directory.
const SNAPSHOT_CHARS = 500_000;
/**
* Session state for a freshly rendered page. Every numbered block is kept,
* including the ones the budget skipped, because the handles an agent wants
* are often the ones that did not fit.
* Session state for a freshly rendered page. Every block is kept, including
* the ones the budget stopped short of, because the part an agent wants next
* is by definition the part that did not fit.
* @param {import('./distill.js').Page} page
* @param {{history?: string[]}} [previous]
* @param {{cursor?: number|null}} [opts] - where the render stopped, null when it finished the page
*/
export function sessionFromPage(page, previous) {
/** @type {Record<string, {type: string, text: string, href?: string, name?: string}>} */
const handles = {};
export function sessionFromPage(page, previous, { cursor = 0 } = {}) {
/** @type {import('./distill.js').Block[]} */
const blocks = [];
let chars = 0;
let dropped = 0;
for (const block of page.blocks) {
if (block.n == null) continue;
const url = block.href ? resolveHref(block.href, page.url) : null;
handles[block.n] = {
chars += block.text.length;
if (chars > SNAPSHOT_CHARS) {
dropped++;
continue;
}
const href = block.href ? resolveHref(block.href, page.url) : null;
blocks.push({
type: block.type,
text: block.text,
...(url ? { href: url } : {}),
...(block.n == null ? {} : { n: block.n }),
...(block.level == null ? {} : { level: block.level }),
...(href ? { href } : {}),
...(block.name ? { name: block.name } : {}),
};
});
}
const history = [...(previous?.history ?? []), page.url].slice(-HISTORY_LIMIT);
return { url: page.url, title: page.title, savedAt: new Date().toISOString(), handles, history };
return {
url: page.url,
title: page.title,
savedAt: new Date().toISOString(),
blocks,
cursor,
...(dropped ? { dropped } : {}),
history,
};
}
/**
* Handle lookup for a saved page. Sessions written by earlier versions hold a
* handles map instead of blocks, so `oc do` keeps working across an upgrade
* even though `oc read` and `oc next` need the page reopened.
* @param {any} state
* @param {number} n
*/
export function handleFor(state, n) {
if (state?.blocks) return state.blocks.find((b) => b.n === n) ?? null;
return state?.handles?.[n] ?? null;
}
/**
* The numbers a saved page offers, for error messages that tell an agent what
* it could have asked for.
* @param {any} state
* @returns {number[]}
*/
export function handleNumbers(state) {
if (state?.blocks) return state.blocks.filter((b) => b.n != null).map((b) => b.n);
return Object.keys(state?.handles ?? {}).map(Number);
}
/**
tests/act.test.js
+64 -9
View File
@@ -9,25 +9,77 @@ import { join } from 'node:path';
process.env.OC_HOME = mkdtempSync(join(tmpdir(), 'oc-test-'));
const { distill } = await import('../src/distill.js');
const { activate } = await import('../src/act.js');
const { activate, read, next } = await import('../src/act.js');
const { sessionFromPage, saveSession, loadSession, resolveHref } = await import('../src/session.js');
const { render } = await import('../src/render.js');
const html = readFileSync(new URL('./pages/news.html', import.meta.url), 'utf8');
const page = () => distill(html, 'https://example.test/news');
const open = (name = 'default') => saveSession(name, sessionFromPage(page(), loadSession(name)));
const open = (name = 'default', budget = 500) => {
const p = page();
saveSession(name, sessionFromPage(p, loadSession(name), { cursor: render(p, { budget }).stats.next }));
};
test('a rendered page is remembered with absolute URLs for every handle', () => {
open();
const state = loadSession('default');
assert.equal(state.url, 'https://example.test/news');
assert.equal(state.handles[1].href, 'https://example.test/item?id=1');
const numbered = page().blocks.filter((b) => b.n != null);
assert.equal(Object.keys(state.handles).length, numbered.length, 'every numbered block must be resolvable');
assert.equal(state.blocks.find((b) => b.n === 2).href, 'https://example.test/item?id=1');
assert.equal(state.blocks.length, page().blocks.length, 'the whole page must survive for read and next');
});
test('do follows the link behind a number without the agent seeing a URL', () => {
open();
assert.deepEqual(activate(1), { url: 'https://example.test/item?id=1', text: 'Show HN: I built a tiny CSV toolkit' });
assert.deepEqual(activate(2), { url: 'https://example.test/item?id=1', text: 'Show HN: I built a tiny CSV toolkit' });
});
test('do still works on a session saved by an older version', () => {
saveSession('legacy', { url: 'https://example.test/old', handles: { 1: { type: 'link', text: 'a', href: 'https://example.test/a' } } });
assert.equal(activate(1, { session: 'legacy' }).url, 'https://example.test/a');
assert.throws(() => next({ session: 'legacy' }), /older oc, run 'oc open https:\/\/example.test\/old'/);
});
test('read prints the region at a number in full, uncut', () => {
open();
const out = read(9);
assert.ok(out.includes('safely does'), 'read must not stop at the compact cap');
assert.ok(!out.includes('+144 chars'), 'read must not print a cut marker for text it printed whole');
assert.ok(out.includes('## [8] About'), 'the heading above the block gives it context');
assert.ok(!out.includes('Postgres 18 released'), 'the section before it is not part of the region');
});
test('read of a heading takes the section under it', () => {
open();
const out = read(8);
assert.ok(out.startsWith('## [8] About'));
assert.ok(out.includes('safely does'));
});
test('a region too big for the budget says where to pick it up', () => {
open();
const out = read(8, { budget: 20 });
assert.match(out, /region cut at ~20 tokens, continue with 'oc read \d+'/);
});
test('read and next explain themselves when the number or the page is missing', () => {
open();
assert.throws(() => read(9999), /no \[9999\].*oc open/s);
assert.throws(() => read(0), /usage: oc read <n>/);
assert.throws(() => read(1, { session: 'never-opened' }), /oc open <url>' first/);
});
test('next continues where the budget stopped, then says the page is done', () => {
open('paged', 100);
const first = next({ session: 'paged', budget: 100 });
assert.ok(first.startsWith('# Fixture News (continued)'));
assert.ok(!first.includes('Show HN'), 'next must not reprint what open already charged for');
let out = first;
for (let i = 0; i < 10 && loadSession('paged').cursor != null; i++) {
out = next({ session: 'paged', budget: 100 });
}
assert.equal(loadSession('paged').cursor, null, 'paging must reach the end of the page');
assert.ok(out.includes('newest'), 'the last block of the page must come out eventually');
assert.match(next({ session: 'paged' }), /end of https:\/\/example.test\/news/);
});
test('search result redirectors resolve to the page they wrap', () => {
@@ -56,13 +108,16 @@ test('every failure names the command that fixes it', () => {
assert.throws(() => activate(input.n), /is an input.*oc fill/s);
const button = page().blocks.find((b) => b.type === 'button');
assert.throws(() => activate(button.n), /no link to follow/);
// The numbers that are not links point at the command that does use them.
assert.throws(() => activate(1), /is heading.*oc read 1/s);
assert.throws(() => activate(9), /is text.*oc read 9/s);
});
test('named sessions keep separate page state', () => {
open('work');
saveSession('other', { url: 'https://example.test/other', handles: {} });
assert.equal(activate(1, { session: 'work' }).url, 'https://example.test/item?id=1');
assert.throws(() => activate(1, { session: 'other' }), /no \[1\]/);
saveSession('other', { url: 'https://example.test/other', blocks: [], cursor: null });
assert.equal(activate(2, { session: 'work' }).url, 'https://example.test/item?id=1');
assert.throws(() => activate(2, { session: 'other' }), /no \[2\]/);
});
test('history grows with each page and stays bounded', () => {
tests/distill.test.js
+37 -4
View File
@@ -1,7 +1,7 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { distill, toMarkdown, toHTML, feedToHTML } from '../src/distill.js';
import { distill, toMarkdown, toHTML, feedToHTML, TEXT_CAP } from '../src/distill.js';
import { render, estimateTokens } from '../src/render.js';
const html = readFileSync(new URL('./pages/news.html', import.meta.url), 'utf8');
@@ -30,15 +30,26 @@ test('raw html mode keeps markup', () => {
assert.ok(!out.includes('<script'), 'script tag survived');
});
test('interactive elements get numbered handles in document order', () => {
test('elements get numbered handles in document order', () => {
const p = page();
const links = p.blocks.filter((b) => b.type === 'link');
assert.equal(links[0].n, 1);
assert.equal(links[0].text, 'Show HN: I built a tiny CSV toolkit');
assert.equal(links[0].n, 2, 'the page heading takes [1]');
const input = p.blocks.find((b) => b.type === 'input');
assert.equal(input.name, 'q');
const button = p.blocks.find((b) => b.type === 'button');
assert.equal(button.text, 'Search');
// Numbers rise once, in document order, and never repeat.
const nums = p.blocks.filter((b) => b.n != null).map((b) => b.n);
assert.deepEqual(nums, nums.map((_, i) => i + 1));
});
test('a text block long enough to be cut is numbered, a short one is not', () => {
const p = page();
const long = p.blocks.find((b) => b.type === 'text' && b.text.length > TEXT_CAP);
assert.ok(long.n, 'a cut block with no number cannot be read back');
const short = p.blocks.find((b) => b.type === 'text' && b.text === '312 points');
assert.equal(short.n, undefined);
});
test('same page yields the same output', () => {
@@ -48,7 +59,29 @@ test('same page yields the same output', () => {
test('render respects the token budget', () => {
const { text, stats } = render(page(), { budget: 100 });
assert.ok(stats.tokens <= 120, `render cost ~${stats.tokens} tokens against a budget of 100`);
assert.ok(text.includes('over budget'), 'skipped blocks must be announced');
assert.match(text, /\.\.\. \d+ more blocks \(~\d+ tokens\)/, 'what was cut must be priced');
assert.ok(text.includes("'oc next'"), 'the cheapest way to the rest must be named');
assert.ok(text.includes('| next |'), 'next belongs in the actions of a cut page');
});
test('what a render stops at is where the next one starts', () => {
const p = page();
const first = render(p, { budget: 100 });
assert.equal(typeof first.stats.next, 'number');
const rest = render(p, { budget: 500, from: first.stats.next });
assert.ok(rest.text.startsWith('# Fixture News (continued)'));
assert.equal(rest.stats.next, null, 'the second render finishes the page');
// Nothing is printed twice and nothing is lost between the two.
assert.ok(!rest.text.includes('Postgres 18 released'));
assert.ok(first.text.includes('Postgres 18 released'));
assert.ok(rest.text.includes('input q'));
});
test('a page render never stalls on a block bigger than the budget', () => {
const { text, stats } = render(page(), { budget: 1 });
assert.ok(stats.next > 0, 'one block must always go out or next can never advance');
assert.ok(text.includes('Show HN'), 'the block that did not fit is printed anyway');
assert.ok(text.includes('more blocks'));
});
test('default render of a normal page fits the 500 token target', () => {