External/oc
1
0
Fork 0
mirror of https://github.com/only-cli/oc.git synced 2026-09-15 10:40:56 +02:00
Code Issues Packages Projects Releases Wiki Activity

Lead the render with the page's main content

Browse Source 
The budget was being spent on whatever came first in the document, which on
most pages is menus. On the Reddit thread the benchmark uses, all 500 tokens
went to nav, sidebar, and moderator boxes, so an agent that opened the page to
read the discussion had to escalate to oc raw to see a single comment.

distill now finds the content (main, role=main, a single article, else the
densest run of prose) and emits it first, with the rest of the page after it
under a divider. Nothing is dropped, so do <n> still reaches every link.

Link labels that repeat down a page go with it: a thread stamps permalink,
save, and report onto every comment, which cost more than the comments. The
count and three examples are printed in place, and oc raw still has them.

Same thread, before and after: 660 blocks to 254, whole compact page 4,184
tokens to 2,712, and the first view now holds comments instead of a sidebar.
Pages small enough to print whole keep document order.
This commit is contained in:
only-cli
2026-08-18 17:20:55 -04:00
parent 6aaa8f1975
commit 0435b386ae
7 changed files with 321 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+26 -6
View File
@@ -56,6 +56,26 @@ Flags: `--budget <tokens>` (default 500, 2000 for `read`), `--json`, `--html` (r
`oc open` remembers the page it rendered, so `oc do 3` follows `[3]` without the agent ever handling a URL. That state is a JSON file per session under `~/.only-cli` (override the directory with `OC_HOME`); there is no daemon and no background browser. Links that search engines wrap in a tracking redirect resolve to the real destination, so `do` on a result works like a click.
### What goes first
A budget only helps if it is spent on the part of the page someone asked for. Most pages open with menus, so the render puts the main content first and everything else after it:
```
# Best Terminal Web Browser : linuxquestions
[3] Best Terminal Web Browser
IYO What's The Best Terminal Web Browser?
[17] Xendarq
46 points 3 years ago
You mean there's a terminal web browser other than lynx??! Who knew!
...
--- 348 repeated links hidden (save, report, [-]), 'oc raw' has them ---
--- rest of page: navigation, sidebar, footer ---
```
The content is found by following the page's own markup (`main`, `role="main"`, a single `article`) and falling back to the densest run of prose when a page says nothing. Nothing is dropped: nav and sidebar links still carry numbers `oc do` can follow, they just stop being what the budget buys. Link labels that repeat down a page are the exception, because a forum thread stamps permalink, save, and report onto every comment, and on a long thread those cost more than the comments do. They are removed and the count says so.
Pages small enough to print whole are left in document order, since nothing is competing for the budget there.
### Reading past the budget
A 500 token view of a long page is cheap on the first read and expensive on the second, if the only way to see more is the whole page again. So the view says what it left behind, and there are three ways to collect it that are not the whole page:
@@ -63,14 +83,14 @@ A 500 token view of a long page is cheap on the first read and expensive on the
```
$ oc open https://old.reddit.com/r/linuxquestions/comments/xpznb1/best_terminal_web_browser/
...
[80] Kind of a weird question, but does anyone here use a terminal browser to ... +312 chars
... 570 more blocks (~4,362 tokens): 'oc next' for the next ~500, 'oc raw' for all
[26] Lynx because it's the only one I'm aware of. These days the web is ... +53 chars
... 168 more blocks (~2,478 tokens): 'oc next' for the next ~500, 'oc raw' for all
actions: do <n> | read <n> | next | raw
$ oc find w3m # 7 matches with the number to read each by, 115 tokens
$ oc read 245 # that one comment in full, 88 tokens
$ oc next # the next ~455 tokens, continuing exactly where the view stopped
$ oc raw # the whole thread, 9,670 tokens
$ oc find w3m # 7 matches with the number to read each by, 142 tokens
$ oc read 23 # that comment in full, 143 tokens
$ oc next # the next ~450 tokens, continuing exactly where the view stopped
$ oc raw # the whole thread, 9,636 tokens
```
None of the three fetches anything: `open` saves the distilled page, so working through it afterwards costs one file read. `find` matches as a phrase, case insensitive, and falls back to the words separately when the phrase is absent. `read <n>` takes one region, which is the block at `[n]` with the couple of blocks that lead into it, or the whole section when `[n]` is a heading. Headings and text blocks long enough to be cut get numbers for exactly this reason, and `... +312 chars` on a line is how much of that block the view did not print.
llms.txt
+1
View File
@@ -7,6 +7,7 @@ Key facts:
- Install: `npm install -g @only-cli/oc`, or zero-install with `npx @only-cli/oc`
- Commands: `oc open <url>` (compact view with numbered actions), `oc do <n>` (follow numbered link [n]), `oc find <query>` (where a string appears on the page already open), `oc read <n>` (one region in full), `oc next` (the next screenful), `oc raw [url]` (whole page as markdown, `--html` for cleaned HTML), `oc --help` for the full surface
- Default output budget is 500 tokens per page; `--budget <n>` adjusts it, and `find`, `read <n>`, or `next` collect what the budget cut without refetching the page
- The render leads with the page's main content and puts navigation, sidebar, and footer after it, so the budget is spent on what was asked for rather than on menus
- Benchmarked at roughly 45x fewer tokens than reading raw HTML, with per-task numbers at https://github.com/only-cli/benchmarks
- Works on any mostly-static website; tuned shortcuts ship for Hacker News, Reddit, GitHub, LinkedIn (public guest views), DuckDuckGo, and Bing
- Requests impersonate Chrome, so pages that block plain scripts often still work
skills/only-cli/SKILL.md
+5 -5
View File
@@ -20,7 +20,7 @@ npx @only-cli/oc raw [url] whole page as markdown (add --html for cleaned H
## Reading the output
- The first line is the page title, then headings, text, and interactive elements in page order.
- The first line is the page title, then the page's main content: the article, the comment thread, the results. Navigation, sidebar, and footer come after it, under a `--- rest of page ---` line, still numbered and still followable with `do <n>`.
- `[n]` marks a link, button, input, heading, or a text block long enough to be cut.
- `... +820 chars` at the end of a line means that block was cut there. `read <n>` prints it whole.
- `... 164 more blocks (~7,100 tokens)` means the page ran past the budget. That is the price of the rest, so you can decide before paying.
@@ -35,13 +35,13 @@ Four ways to go past the first view, cheapest first. Pick by what you need, not
- `oc next` prints the next budget worth of the same page and remembers where it stopped, so calling it again continues. Use it when you are reading rather than looking something up.
- `oc raw` (no URL needed once a page is open) prints everything. It costs an order of magnitude more, so use it when you genuinely need the whole page.
Measured on one Reddit thread: `open` 475 tokens, one `find` 115, one `read` 88, each `next` about 455, `raw` 9,670. None of them fetches anything; they all work from the page `open` already saved.
Measured on one Reddit thread: `open` 436 tokens, one `find` 142, one `read` 143, each `next` about 450, `raw` 9,636. None of them fetches anything; they all work from the page `open` already saved.
```
oc open https://old.reddit.com/r/linuxquestions/comments/xpznb1/best_terminal_web_browser/
oc find w3m -> 7 matches with their numbers, 115 tokens
oc read 245 -> that comment in full, 88 tokens
oc next -> keep reading, 455 tokens at a time
oc find w3m -> 7 matches with their numbers, 142 tokens
oc read 23 -> that comment in full, 143 tokens
oc next -> keep reading, 450 tokens at a time
```
`find` matches the query as a phrase, case insensitive, and falls back to matching the words separately when the phrase is not there. It says how many matches it held back if they did not fit the budget.
src/distill.js
+139 -3
View File
@@ -3,7 +3,7 @@ import TurndownService from 'turndown';
/**
* @typedef {Object} Block
* @property {'heading'|'text'|'link'|'input'|'button'} type
* @property {'heading'|'text'|'link'|'input'|'button'|'divider'} type
* @property {string} text
* @property {number} [n] - action handle
* @property {number} [level] - heading level 1..6
@@ -28,6 +28,34 @@ const DROP = new Set([
'link', 'meta', 'head', 'canvas', 'video', 'audio', 'object',
]);
// Elements that are furniture by definition. The main content is never one of
// them, so the search below refuses to descend into them.
const FURNITURE = new Set(['nav', 'header', 'footer', 'aside']);
// Selectors a page uses to say where its content is. One match is a claim
// worth believing; several <article> elements mean a listing, where the list
// is the content and taking the first would throw the rest away.
const MAIN_SELECTORS = ['main', '[role="main"]', 'article'];
// Reordering only matters on a page that cannot be printed whole: if the
// budget covers everything, what leads is a question with no consequences.
// Below this much text the page is left in document order.
const MIN_PAGE = 3000;
// A candidate has to hold this share of the page's prose to be believed.
const MIN_SHARE = 0.25;
// How much of its parent's prose a child must hold before the search moves
// down into it, and how much of the page's it must keep to be worth choosing.
const DESCEND_SHARE = 0.6;
const KEEP_SHARE = 0.5;
const MAX_DEPTH = 12;
// A short link label repeated this many times down a page is per-item
// furniture (permalink, embed, save, reply, hide) rather than content. On a
// forum thread there is one set per comment, which costs more than the
// comments do.
const REPEAT_LIMIT = 5;
const REPEAT_MAX_LEN = 25;
const clean = (s) => s.replace(/\s+/g, ' ').trim();
/**
@@ -50,6 +78,9 @@ export function distill(html, url = '') {
el.getAttribute('aria-hidden') === 'true' ||
/display:\s*none/.test(el.getAttribute('style') ?? '');
/** Subtree already emitted, skipped when the rest of the page is walked. */
let done = null;
const walk = (node) => {
if (node.nodeType === 3) {
const text = clean(node.textContent);
@@ -57,6 +88,7 @@ export function distill(html, url = '') {
return;
}
if (node.nodeType !== 1) return;
if (node === done) return;
const tag = node.localName;
if (DROP.has(tag) || hidden(node)) return;
@@ -92,8 +124,111 @@ export function distill(html, url = '') {
};
const body = bodyOf(document);
if (body) walk(body);
return { url, title, blocks: number(mergeText(blocks)) };
const main = body ? mainOf(document, body) : null;
if (main) {
// Content first, everything else after it. The budget in render.js is
// spent top down, so whatever leads the list is what an agent gets for
// its first 500 tokens, and on most pages the top of the document is
// menus. Nothing is dropped: the chrome carries links `oc do` needs, it
// just stops being the thing that gets read.
walk(main);
done = main;
const contentBlocks = blocks.length;
if (body) walk(body);
if (blocks.length > contentBlocks) {
blocks.splice(contentBlocks, 0, { type: 'divider', text: '--- rest of page: navigation, sidebar, footer ---' });
}
} else if (body) {
walk(body);
}
return { url, title, blocks: number(mergeText(dropRepeats(blocks))) };
}
/**
* Remove link labels that repeat down the page. They are the per-item controls
* a template stamps onto every row, and on a long thread they outweigh the
* content. What went is stated in place, because a view that quietly drops
* things is a view an agent cannot trust.
* @param {Block[]} blocks
* @returns {Block[]}
*/
function dropRepeats(blocks) {
/** @type {Map<string, number>} */
const counts = new Map();
for (const b of blocks) {
if (b.type !== 'link' || b.text.length > REPEAT_MAX_LEN) continue;
const key = b.text.toLowerCase();
counts.set(key, (counts.get(key) ?? 0) + 1);
}
const repeated = new Set([...counts].filter(([, n]) => n >= REPEAT_LIMIT).map(([k]) => k));
if (!repeated.size) return blocks;
const kept = blocks.filter((b) => !(b.type === 'link' && repeated.has(b.text.toLowerCase())));
const gone = blocks.length - kept.length;
const names = [...repeated].slice(0, 3).join(', ');
const note = { type: 'divider', text: `--- ${gone} repeated links hidden (${names}), 'oc raw' has them ---` };
// The note belongs with the content it was cut from, not after the chrome.
const boundary = kept.findIndex((b) => b.type === 'divider');
kept.splice(boundary === -1 ? kept.length : boundary, 0, note);
return kept;
}
/**
* Find the element holding the page's main content, or null when the page is
* too small or too flat for the question to have an answer.
* @param {any} document
* @param {any} body
* @returns {any}
*/
function mainOf(document, body) {
if (clean(body.textContent ?? '').length < MIN_PAGE) return null;
const total = prose(body);
for (const selector of MAIN_SELECTORS) {
const found = [...document.querySelectorAll(selector)];
if (found.length !== 1) continue;
const el = found[0];
if (el !== body && prose(el) >= total * MIN_SHARE) return el;
}
return densest(body, total);
}
/**
* Walk down the tree while one child holds most of the prose of the element
* above it. That is what separates a content column from the page around it
* without knowing anything about the site.
* @param {any} body
* @param {number} total
* @returns {any}
*/
function densest(body, total) {
let node = body;
for (let depth = 0; depth < MAX_DEPTH; depth++) {
let best = null;
let bestProse = 0;
for (const child of node.children ?? []) {
if (FURNITURE.has(child.localName) || DROP.has(child.localName)) continue;
const len = prose(child);
if (len > bestProse) {
best = child;
bestProse = len;
}
}
if (!best || bestProse < prose(node) * DESCEND_SHARE || bestProse < total * KEEP_SHARE) break;
node = best;
}
return node === body ? null : node;
}
/**
* Text length minus link text. Link text is what navigation is made of, so
* subtracting it is what keeps a menu from outscoring an article.
* @param {any} el
* @returns {number}
*/
function prose(el) {
const text = clean(el.textContent ?? '').length;
let links = 0;
for (const a of el.querySelectorAll?.('a') ?? []) links += clean(a.textContent ?? '').length;
return text - links;
}
/**
@@ -107,6 +242,7 @@ export function distill(html, url = '') {
function number(blocks) {
let handle = 0;
for (const block of blocks) {
if (block.type === 'divider') continue;
if (block.type !== 'text' || block.text.length > TEXT_CAP) block.n = ++handle;
}
return blocks;
src/render.js
+2
View File
@@ -136,6 +136,8 @@ export function formatBlock(b, { full = false } = {}) {
return `${tag}button "${full ? b.text : truncate(b.text)}"`;
case 'input':
return `${tag}input ${b.name} (${b.text})`;
case 'divider':
return b.text;
default:
return full ? `${tag}${b.text}` : `${tag}${truncate(b.text)}`;
}
tests/distill.test.js
+40
View File
@@ -7,6 +7,8 @@ import { render, estimateTokens } from '../src/render.js';
const html = readFileSync(new URL('./pages/news.html', import.meta.url), 'utf8');
const page = () => distill(html, 'https://example.test/news');
const feed = readFileSync(new URL('./pages/feed.xml', import.meta.url), 'utf8');
const forum = readFileSync(new URL('./pages/forum.html', import.meta.url), 'utf8');
const thread = () => distill(forum, 'https://example.test/t/1');
test('noise never reaches the output, compact or raw', () => {
for (const out of [render(page(), { budget: 5000 }).text, toMarkdown(html), toHTML(html)]) {
@@ -137,6 +139,44 @@ test('rss with cdata bodies converts too, ordinary html does not', () => {
assert.equal(feedToHTML(html), null, 'ordinary html misread as a feed');
});
test('the content leads the page and the chrome follows it', () => {
const { text } = render(thread(), { budget: 500 });
const lines = text.split('\n');
// The budget is spent top down, so what matters is that comments are inside
// the first view at all: on this page they used to start below it.
assert.ok(text.includes('Lynx is the one I keep coming back to'), `content missed the first view:\n${text}`);
assert.ok(!text.includes('section 7'), 'nav still printed ahead of the content');
const blocks = thread().blocks;
const divider = blocks.findIndex((b) => b.type === 'divider' && b.text.includes('rest of page'));
const nav = blocks.findIndex((b) => b.text === 'section 7');
assert.ok(divider > 0 && nav > divider, 'the nav was not moved below the content');
assert.ok(lines[1].startsWith('# '), 'the first line under the title is not the content heading');
});
test('nothing is dropped when the content is moved up, only reordered', () => {
const blocks = thread().blocks;
const text = blocks.map((b) => b.text).join(' ');
assert.ok(text.includes('section 13'), 'a nav link went missing');
assert.ok(text.includes('terms'), 'a footer link went missing');
assert.ok(text.includes('This sidebar exists on every page'), 'sidebar text went missing');
});
test('per item links that repeat down a page are dropped, and say so', () => {
const blocks = thread().blocks;
assert.ok(!blocks.some((b) => b.type === 'link' && b.text === 'permalink'), 'per comment chrome survived');
assert.ok(blocks.some((b) => b.type === 'link' && b.text === 'commenter3'), 'a unique link was dropped with them');
const note = blocks.find((b) => b.type === 'divider' && b.text.includes('repeated links hidden'));
assert.ok(note, 'links vanished with nothing said about it');
assert.ok(note.text.includes("'oc raw' has them"), 'the note does not say how to get them back');
assert.ok(toMarkdown(forum).includes('permalink'), 'raw lost them too, so the note lies');
});
test('a page that fits the budget is left in document order', () => {
const blocks = page().blocks;
assert.ok(!blocks.some((b) => b.type === 'divider'), 'a small page was reordered for no reason');
assert.equal(blocks[0].text, 'Fixture News');
});
test('long runs of short links collapse into a range marker', () => {
const nav = Array.from({ length: 15 }, (_, i) => `<a href="/s/${i}">sub${i}</a>`).join(' ');
const navHtml = `<html><head><title>T</title></head><body>${nav}<p>actual content</p></body></html>`;
tests/pages/forum.html
+108
View File
@@ -0,0 +1,108 @@
<!doctype html>
<html>
<head>
<title>Best terminal web browser : fixtureforum</title>
</head>
<body>
<nav>
<a href="/section/0">section 0</a>
<a href="/section/1">section 1</a>
<a href="/section/2">section 2</a>
<a href="/section/3">section 3</a>
<a href="/section/4">section 4</a>
<a href="/section/5">section 5</a>
<a href="/section/6">section 6</a>
<a href="/section/7">section 7</a>
<a href="/section/8">section 8</a>
<a href="/section/9">section 9</a>
<a href="/section/10">section 10</a>
<a href="/section/11">section 11</a>
<a href="/section/12">section 12</a>
<a href="/section/13">section 13</a>
</nav>
<header><a href="/login">log in</a> <a href="/signup">sign up</a></header>
<div id="sidebar">
<h2>About this forum</h2>
<p>This sidebar exists on every page of the forum. It explains the rules, links to the moderators, and repeats the same three hundred characters of boilerplate on every single thread, which is exactly the kind of text that should never be the first thing an agent pays to read when it opened a discussion to find out what the discussion says.</p>
<a href="/rules">rules</a>
<a href="/moderators">moderators</a>
</div>
<div id="content" role="main">
<h1>Best terminal web browser</h1>
<p>Which terminal web browser do people here actually use day to day, and why that one over the others?</p>
<div class="comment">
<a href="/user/commenter1">commenter1</a>
<span>37 points, 3 years ago</span>
<p>Lynx is the one I keep coming back to. It is fast, it never surprises me, and it renders documentation pages better than anything else I have tried in a terminal. The keybindings take an afternoon to learn and then they are yours forever, which is the opposite of how most software works.</p>
<a href="/c/1">permalink</a>
<a href="/c/1/reply">reply</a>
<a href="/c/1/save">save</a>
<a href="/c/1/report">report</a>
</div>
<div class="comment">
<a href="/user/commenter2">commenter2</a>
<span>34 points, 3 years ago</span>
<p>I switched to w3m two years ago and never went back. Inline images in the terminal sound like a gimmick until you read a bug report with a screenshot in it and the screenshot is simply there, in the same window, without a browser tab in sight.</p>
<a href="/c/2">permalink</a>
<a href="/c/2/reply">reply</a>
<a href="/c/2/save">save</a>
<a href="/c/2/report">report</a>
</div>
<div class="comment">
<a href="/user/commenter3">commenter3</a>
<span>31 points, 3 years ago</span>
<p>Nobody has mentioned links yet, which surprises me. It handles tables far better than lynx does, and the mouse support in the framebuffer version is genuinely useful when a page insists on being a grid.</p>
<a href="/c/3">permalink</a>
<a href="/c/3/reply">reply</a>
<a href="/c/3/save">save</a>
<a href="/c/3/report">report</a>
</div>
<div class="comment">
<a href="/user/commenter4">commenter4</a>
<span>28 points, 3 years ago</span>
<p>For reading news I use w3m with a readability filter in front of it. The filter strips the navigation and the cookie banners before w3m ever sees them, so a news article arrives as an article instead of as a menu with an article somewhere inside it.</p>
<a href="/c/4">permalink</a>
<a href="/c/4/reply">reply</a>
<a href="/c/4/save">save</a>
<a href="/c/4/report">report</a>
</div>
<div class="comment">
<a href="/user/commenter5">commenter5</a>
<span>25 points, 3 years ago</span>
<p>elinks deserves a mention for its tabs and its bookmark handling. It is the closest thing to a graphical browser that still runs over ssh, and the configuration file is plain and documented rather than a maze of hidden preferences.</p>
<a href="/c/5">permalink</a>
<a href="/c/5/reply">reply</a>
<a href="/c/5/save">save</a>
<a href="/c/5/report">report</a>
</div>
<div class="comment">
<a href="/user/commenter6">commenter6</a>
<span>22 points, 3 years ago</span>
<p>The honest answer is that it depends on the page. You will use links for sites with tables, w3m for anything with images, and lynx when you want the fastest possible path from a url to the text on it. Keeping all three costs a few megabytes.</p>
<a href="/c/6">permalink</a>
<a href="/c/6/reply">reply</a>
<a href="/c/6/save">save</a>
<a href="/c/6/report">report</a>
</div>
<div class="comment">
<a href="/user/commenter7">commenter7</a>
<span>19 points, 3 years ago</span>
<p>I run lynx inside tmux on a small server and read my mail queue reports with it. No javascript means no surprises, and a page that needs javascript to show me a number was never going to be a page I wanted to read anyway.</p>
<a href="/c/7">permalink</a>
<a href="/c/7/reply">reply</a>
<a href="/c/7/save">save</a>
<a href="/c/7/report">report</a>
</div>
<div class="comment">
<a href="/user/commenter8">commenter8</a>
<span>16 points, 3 years ago</span>
<p>w3m with the readable extension is my default. The extension is what makes the difference: without it you get the whole page, navigation and all, and with it you get the part a person would actually want to read.</p>
<a href="/c/8">permalink</a>
<a href="/c/8/reply">reply</a>
<a href="/c/8/save">save</a>
<a href="/c/8/report">report</a>
</div>
</div>
<footer><a href="/tos">terms</a> <a href="/privacy">privacy</a></footer>
</body>
</html>