A syntax highlighter gives every token of a command its own element, so
`s3://bucket/` reached the walk as `s3`, `:`, `//`, `bucket`, `/`, and the
rule that reassembles text fragments only glues the ones sharing a parent.
The rest were space-joined. The AWS CLI reference handed over
aws s3 cp test . txt s3 : // amzn - s3 - demo - bucket / test2 . txt
and Node's fs docs handed over `console . log` and `fd ?. close ()`. Any
command or snippet an agent took from a docs page was wrong, and nothing in
the output said so. Highlighting is not an edge case: of 172 pre elements on
the AWS CLI reference, the Rust book, the Node API docs and the Python
library docs, 159 are split this way.
A pre or code subtree is now read as one string. That costs nothing to
follow, because not one of those 172 blocks contains a link, and it also
fixes inline code, which was inserting a space into `Byte ( u8 only)`.
Two things fall out of it.
Page furniture had to stop riding along. Node puts a language label beside a
copy button inside every code block, so the subtree's text ended in
`javascriptcopy`. A control is chrome, and so is the block-level element
holding it, which is how the label leaves with the button it sits beside.
The test stays on block-level wrappers because a highlighter's own elements
are inline, so a stray control can never take a line of code out with it.
Code blocks keep their lines. Collapsing them was survivable while the code
was already wrecked; once it reads correctly, `// read it back` in front of
the statements that followed it is worse, because the output now looks
trustworthy. Blank runs and the indentation the whole block shares carry no
meaning and still go. A cut lands on a line end for the same reason it
already lands on a sentence end, and a snippet stays one line, because that
is what find's index promises.
Measured over five real pages, the compact view moves by -15, -7, +208, -45
and 0 characters, about 35 tokens in total, all of the growth being Python's
pretty-printed output getting its indentation back.
A tool call inside an agent session costs 23,000 to 33,000 tokens of
overhead whatever it prints, so the page-view win only reaches the
session total if answering a task takes fewer commands. Three places
were charging a command to say what the next command should be, each
found by capturing the command stream of a real agent run rather than
by reading the code.
A search result title is a link. Every engine puts it in an anchor
filling an <h2>, and the walk took the heading's text and returned,
dropping the href, so `do` on the most obvious number on a results page
printed the title back. The agent then spent a second command finding
the number that navigates. The href now rides along when the anchor is
the whole heading, which is the test documentation fails on purpose:
every heading in the Rust book and on an AWS CLI reference page carries
a permalink to its own id, and following one would refetch the page the
agent is already reading.
`find` pointed at its answer. With a single match it printed the block
and a number, and the agent's next command was always the `read` on
that number, so it now prints the region. With several matches it
showed a 200 character snippet of each even when the budget had room
for them whole, so it spends that room, on the same terms `FINISH`
already documents for a page that nearly fits.
A truncated block ended mid sentence. Asked for the first sentence of a
page, an agent was handed it complete, followed by a marker saying 302
characters were cut, and spent a command on `read` to find out whether
the sentence went on. The cut now falls on the last sentence that
finished inside the cap, and measured across five real pages it costs
nothing: four came out within three characters of before.
The package-lock name field catches up with the scoped package name,
which npm rewrites on any install.
Three faults in the JSON renderer, all of which the npm registry's package
endpoint hits at once, where the render came out as a truncated blob titled
after the package's two maintainers.
mainArray took the longest array of objects at the top level, so `maintainers`
became the subject and the package itself was pushed into the metadata line. A
root carrying its own name is the resource, and an array hanging off it
describes that resource rather than standing in for it. Conventional container
keys are checked first, so a named collection is still read as a collection.
The metadata line capped nothing. One long scalar there, a readme in npm's
case, cost more than the rest of the page put together; a summary line has to
stay a line, so a long one becomes its own block.
An oversized markup field distilled into more blocks than the item it hangs
off had fields. Under BODY_CAP a body is still rendered in place with its links
followable, which is what the Stack Exchange withbody shape wants; over it, one
numbered line, with `oc raw` still holding the whole thing.
Testing the 0.3.0-beta.1 build against live URLs turned up a gap the beta
notes claimed was closed: only the native-fetch path checked the content
type, and impers is the default whenever the optional dependency installs.
So 'oc open' on a PNG rendered eight kilobytes of mojibake as a page, with
numbered blocks, an actions footer, and a straight face.
The check now lives in one exported assertReadableType that both transports
call, so a refusal cannot depend on which client happened to get the page.
While the gate was being written down it also grew a correct allow list.
The old one matched the substring html, xml, or json anywhere in the header,
which let application/vnd.ms-htmlhelp through and, worse, refused text/plain:
a robots.txt or an llms.txt is exactly the kind of small text file an agent
asks for, and the fetch path was answering that it was not a page. Readable
now means any text/* type plus the application/* types that are really text,
including the +json and +xml families a feed answers with. A missing header
stays readable, since small servers omit it and the page behind it is fine.
Tested offline against the header strings themselves rather than the network.
Closes#3.
An API answer is a page: jsonToHTML turns a JSON body into one article per
item, and everything downstream (numbering, budget, do, read, next, raw)
treats it as an ordinary document. No per-site logic and no new dependency.
The compact view is the hard part, since a search response carries far more
fields than fit in 500 tokens. So the renderer scores each field by how much
it varies across items against how wide it prints, penalises fields flattened
out of a sub-object (owner.reputation describes the asker, not the answer),
and spends about 60 characters per item on the winners. What every item
shares is stated once at the bottom instead of repeated, empty fields are
named rather than printed, and what was cut says so and points at oc raw,
which keeps every field.
On the Stack Exchange search endpoint that is 30 results in ~960 tokens
against ~5,500 for the raw body, with each title a link and question_id
visible.
Also here:
- clis/stackoverflow.com.json gains search <query>, which is what #3 was
blocking. Results carry question_id, and the question feed reads one in
full, so search now completes without touching the challenged HTML page.
- fetch: the native-fetch path rejected anything that was not HTML or XML.
It now accepts JSON, which also makes the two transports render one URL
the same way, since the impers path never checked the type at all.
- raw threads the URL through so its view of an API response can be titled
and, unlike the compact view, keeps every field.
Deliberately not done, from the notes on the issue: pagination in the
actions line, and API metadata on stderr. There is no stderr channel at the
distill seam, so response-level fields (has_more, quota_remaining) render as
one footer line instead. A columns hint in the clis specs and a --json
passthrough both looked like the wrong trade: the first needs per-site
tuning for something the scoring already handles, the second would break the
machine-stable Page contract.
PR #5's guard pattern-matched the URL's hostname against a regex, which
both under- and over-blocked: IPv4-mapped IPv6 loopback ([::ffff:127.0.0.1]),
0.0.0.0, and any DNS name that merely resolves to a private address all
sailed through, while a legitimate public hostname like 10.example.com was
wrongly rejected because it starts with "10.". It also only checked the
original URL, so a public URL that redirects to an internal address was
never re-validated.
This replaces the regex with net.isIP + dns.lookup: IP literals are checked
directly (including decoding IPv4-mapped/-compatible IPv6), and DNS names
are resolved first so every address they point to is validated before
connecting. The same check now reruns on every redirect hop for both the
impers and native-fetch transports. Resolving before connecting doesn't
pin the address for the actual connection (neither impers nor fetch expose
that here), so a name that re-resolves differently between this check and
the real connect remains a known, documented residual gap.
Watch pages need client JS to become interactive, but the title,
description, view count, and caption tracks already ship inline in
the initial HTML as ytInitialPlayerResponse, so this reads that
directly instead of waiting on the v0.3 headless fallback. Each
caption track becomes a numbered link, and oc do on it fetches the
timedtext transcript, collapsed into one block so it pages through
oc next/oc read like any other long document instead of costing one
block per caption line.
Adds youtubeToHTML and transcriptToHTML alongside feedToHTML in the
distiller, a youtube.com.json shortcut, and offline tests against a
fixture watch page.
Fixes#6
Three changes with one motive: a second command costs an agent more than
the lines it saves. 'oc do' on a heading or text block now prints the read
instead of refusing, since refusing spends a whole turn naming the command
that should have run. A page that would finish within about four times the
budget is printed whole rather than cut, because the cut moves tokens into
a second command instead of saving them. And social timelines stopped
rendering as one fused paragraph: linkedom splits text nodes around
apostrophes, so fragments are merged back by parent node and edge
whitespace, block elements now end lines, and repeated button labels are
trimmed sooner than links because a button label is never the content.
With that, x.com profiles and posts read without a login, so a six line
cli definition ships for the two x.com pages that work.
The budget was being spent on whatever came first in the document, which on
most pages is menus. On the Reddit thread the benchmark uses, all 500 tokens
went to nav, sidebar, and moderator boxes, so an agent that opened the page to
read the discussion had to escalate to oc raw to see a single comment.
distill now finds the content (main, role=main, a single article, else the
densest run of prose) and emits it first, with the rest of the page after it
under a divider. Nothing is dropped, so do <n> still reaches every link.
Link labels that repeat down a page go with it: a thread stamps permalink,
save, and report onto every comment, which cost more than the comments. The
count and three examples are printed in place, and oc raw still has them.
Same thread, before and after: 660 blocks to 254, whole compact page 4,184
tokens to 2,712, and the first view now holds comments instead of a sidebar.
Pages small enough to print whole keep document order.
find searches the distilled page the session already holds, prints one line
per match with the number to read it by, and costs no fetch. It matches the
query as a phrase, case insensitive, and falls back to matching the words
separately when the phrase is not there.
On the reddit thread from the benchmark: 'oc find w3m' is 115 tokens against
9,670 for oc raw, and it lands on the numbers to read.
The compact view was all or nothing: an agent that needed more than the 500
token budget had only oc raw, ten to twenty times the price. Now open saves
the distilled page, next continues it where the view stopped, and read <n>
prints one region in full. Headings and text blocks long enough to be cut are
numbered so they can be addressed, and the marker prices what it left behind.
On one Reddit thread: open 475 tokens, next 455, read 88, raw 9,670.
The compact view hides link URLs because printing them is most of what
makes a page expensive, which left an agent re-fetching the same page as
--json or raw just to learn where [15] pointed. oc open now saves the
handles it numbered to a small JSON file per session under ~/.only-cli
(OC_HOME relocates it), and oc do <n> resolves one and renders the target
exactly as open would. Search engine tracking redirects are unwrapped so a
result link opens the destination instead of a script page.
Errors name the command that fixes them, since agents read them: an
out-of-range number reports the valid range, an input says to use fill, a
button says the page handles it itself.
Two hops on Hacker News cost about 3k characters this way, against roughly
23k for the re-fetch route.
The feed fixture covers Atom entries (escaped bodies, self-closed
categories, bylines) and an RSS item with a CDATA body. Raw markdown of
a feed exposed an old quirk: cleanDocument removed the head before
toMarkdown read the title, so any page whose body lacked a matching h1
lost its title in raw mode. cleanDocument now captures the title first
and returns both.
Generic distillation engine (no per-site adapters): fetch via impers
impersonating Chrome with a firefox-fingerprint retry, distill to an
interaction tree, render under a hard token budget with numbered action
handles. Raw mode emits markdown via turndown or cleaned HTML. Per-site
CLI definitions for HN, Reddit, Bing, DuckDuckGo. Offline test suite,
agent skill, OIDC publish workflow.
only-cli