mirror of
https://github.com/only-cli/oc.git
synced 2026-09-15 10:40:56 +02:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b1cbf2302d | ||
|
|
5ed47af59b | ||
|
|
962356cb91 | ||
|
|
5b30fbfbc1 | ||
|
|
431dad3abd | ||
|
|
c344e639b6 | ||
|
|
3b44bc944b | ||
|
|
21f9955e15 | ||
|
|
e7a36a67f0 | ||
|
|
0ea9f4cbda | ||
|
|
ff5f376063 | ||
|
|
c5426c246b | ||
|
|
e046d308c5 | ||
|
|
3ea03d341c |
@@ -6,7 +6,7 @@
|
||||
"name": "only-cli",
|
||||
"source": { "source": "github", "repo": "only-cli/oc" },
|
||||
"description": "Browse websites from the terminal in a few hundred tokens",
|
||||
"version": "0.5.1",
|
||||
"version": "0.5.3",
|
||||
"homepage": "https://github.com/only-cli/oc",
|
||||
"license": "MIT"
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"name": "only-cli",
|
||||
"description": "Browse websites from the terminal in a few hundred tokens",
|
||||
"version": "0.5.1"
|
||||
"version": "0.5.3"
|
||||
}
|
||||
|
||||
@@ -3,6 +3,31 @@
|
||||
Notable changes per release. Releases before 0.4.0 are listed at
|
||||
[github.com/only-cli/oc/releases](https://github.com/only-cli/oc/releases).
|
||||
|
||||
## 0.5.3
|
||||
|
||||
### Changed
|
||||
|
||||
- Requests to reddit.com present the Firefox fingerprint first and fall back
|
||||
to Chrome, the reverse of every other site. Reddit's edge answers the Chrome
|
||||
fingerprint with a 403 or a 429 while letting Firefox through, and since it
|
||||
allows anonymous readers about ten requests a minute per address, the wasted
|
||||
Chrome attempt was costing a real share of that budget on every read (#52).
|
||||
|
||||
## 0.5.2
|
||||
|
||||
### Changed
|
||||
|
||||
- `oc reddit` reads the Atom feeds on www.reddit.com instead of old.reddit.com
|
||||
pages. Reddit has sent every logged-out old.reddit.com request to a login
|
||||
page since 30 June 2026, and the `.json` views on www.reddit.com have
|
||||
answered 403 to anything without an OAuth token since 30 May, whatever the
|
||||
User-Agent or TLS fingerprint. The feeds still answer, so `sub`, `post`,
|
||||
`user`, and `search` point at them, and `new <name>` and `top <name>` join
|
||||
the verbs. A subreddit renders in about 480 tokens and a thread with 22
|
||||
comments in about 1,000. The feeds carry no scores or comment counts, and
|
||||
anonymous reddit.com allows roughly ten requests a minute per address, so a
|
||||
burst of shortcuts ends in a 429 that takes minutes to clear. (#52)
|
||||
|
||||
## 0.5.1
|
||||
|
||||
### Added
|
||||
|
||||
@@ -81,7 +81,7 @@ Works on any mostly-static site with no per-site setup: news sites, blogs, docum
|
||||
| website | command | shortcuts |
|
||||
| --- | --- | --- |
|
||||
| Hacker News | `oc hn` | `top`, `new`, `item <id>`, `user <name>` |
|
||||
| Reddit | `oc reddit` (via old.reddit.com) | `sub <name>`, `post <id>`, `user <name>`, `search <query>` |
|
||||
| Reddit | `oc reddit` (via the Atom feeds on www.reddit.com) | `sub <name>`, `new <name>`, `top <name>`, `post <id>`, `user <name>`, `search <query>` |
|
||||
| GitHub | `oc gh` | `repo <owner> <name>`, `user <name>`, `search <query>`, `trending`, `issues <owner> <name>` |
|
||||
| X | `oc x` | `user <name>`, `post <id>` |
|
||||
| LinkedIn | `oc linkedin` | `profile <name>`, `company <name>`, `jobs <query>` (public guest views) |
|
||||
@@ -107,7 +107,7 @@ Works on any mostly-static site with no per-site setup: news sites, blogs, docum
|
||||
|
||||
A shortcut only ever resolves to a URL and then takes the same path `oc open` does, so it changes nothing about what a page costs or how it reads. The last argument takes every word after it, so `oc ddg search claude code cli` and `oc aws search s3 lifecycle rules` need no quoting, and a path argument keeps its slashes, so `oc learn doc azure/aks/what-is-aks` reaches that page.
|
||||
|
||||
A few of these (X, Stack Overflow, YouTube, Microsoft Learn search) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question <id>` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. AWS, Google Cloud, Rust, Java, TypeScript, PHP, and cppreference render docs search client-side, or as a page too bare for oc to read, so their `search` goes through DuckDuckGo with a baked-in `site:` filter instead; Go needs no such fallback, because pkg.go.dev renders its search results on the server and `oc go search` simply opens them. Python's docs are built with Sphinx, which publishes the site's full-text search index as one static file, so `oc py search` fetches that index (cached on disk for a day), ranks it locally, and prints a numbered result list; a query that names a symbol exactly, like `json.dumps`, links straight to its anchor. The same backend will work for any Sphinx site, including most Read the Docs projects. MDN also renders its search client-side, but the page gets its results from a public JSON endpoint, so `oc mdn search` asks that endpoint directly and prints the site's own ranking; that `api` shape in a site definition works for any site whose search answers as JSON. Node.js ships no search endpoint at all, but publishes its whole API reference as one static JSON file, so `oc node search` ranks that file locally the same way the Sphinx backend does, under the same day cache, and every module, class, method, property, and event heading links to its own anchor. Ruby's docs are built with RDoc, which also ships its search index as one static file, so `oc ruby search` ranks every class, method, and guide page locally the same way. PHP's manual has a lookup endpoint that sends an exact function name straight to its page, which is what `oc php fn` rides. Not supported yet: pages that only render with JavaScript and sites with hard bot challenges that expose no feed. Sites that genuinely require your account can be reached with `oc login` (bring your own cookies).
|
||||
A few of these (X, Reddit, Stack Overflow, YouTube, Microsoft Learn search) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question <id>` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. Reddit goes through the Atom feeds on www.reddit.com: old.reddit.com has sent logged-out readers to a login page since June 2026 and the `.json` views answer 403 without an OAuth token, so a reddit.com page URL handed to `oc open` still meets that wall, where `oc reddit post <id>`, or the same URL with `/.rss` on the end, reads the post and its comments. oc asks reddit.com with its Firefox fingerprint first, because Reddit's edge refuses the Chrome one more often than not. The feeds carry titles, authors, dates, and bodies but no scores or comment counts, and anonymous reddit.com allows roughly ten requests a minute per address, so a burst of Reddit shortcuts ends in a 429 that takes minutes to clear. AWS, Google Cloud, Rust, Java, TypeScript, PHP, and cppreference render docs search client-side, or as a page too bare for oc to read, so their `search` goes through DuckDuckGo with a baked-in `site:` filter instead; Go needs no such fallback, because pkg.go.dev renders its search results on the server and `oc go search` simply opens them. Python's docs are built with Sphinx, which publishes the site's full-text search index as one static file, so `oc py search` fetches that index (cached on disk for a day), ranks it locally, and prints a numbered result list; a query that names a symbol exactly, like `json.dumps`, links straight to its anchor. The same backend will work for any Sphinx site, including most Read the Docs projects. MDN also renders its search client-side, but the page gets its results from a public JSON endpoint, so `oc mdn search` asks that endpoint directly and prints the site's own ranking; that `api` shape in a site definition works for any site whose search answers as JSON. Node.js ships no search endpoint at all, but publishes its whole API reference as one static JSON file, so `oc node search` ranks that file locally the same way the Sphinx backend does, under the same day cache, and every module, class, method, property, and event heading links to its own anchor. Ruby's docs are built with RDoc, which also ships its search index as one static file, so `oc ruby search` ranks every class, method, and guide page locally the same way. PHP's manual has a lookup endpoint that sends an exact function name straight to its page, which is what `oc php fn` rides. Not supported yet: pages that only render with JavaScript and sites with hard bot challenges that expose no feed. Sites that genuinely require your account can be reached with `oc login` (bring your own cookies).
|
||||
|
||||
Want a website on that list? Open a pull request, or an issue naming the site; see [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
@@ -168,12 +168,13 @@ Private and internal addresses are refused whether or not a proxy is set. With a
|
||||
|
||||
## Benchmarks
|
||||
|
||||
Full methodology, per-task rows, and the Codex runs live in [only-cli/benchmarks](https://github.com/only-cli/benchmarks). Where things stand (oc 0.5.0, August 2026, live sites):
|
||||
Full methodology, per-task rows, and the Codex runs live in [only-cli/benchmarks](https://github.com/only-cli/benchmarks). Where things stand (oc 0.5.1, September 2026, live sites):
|
||||
|
||||
- **142x fewer tokens than raw HTML** across 15 real pages: 10,936 against 1,552,491. 14x fewer than Jina Reader, 49x fewer than Playwright MCP's accessibility snapshot.
|
||||
- **The only reader that returned real content on every page.** Reddit blocked curl, Jina Reader and Playwright; Jina also failed LinkedIn and Yahoo Finance; DuckDuckGo blocked lynx. oc's Chrome impersonation read all fifteen.
|
||||
- **Half the cost of Claude Code's built-in `WebSearch`** on Wikipedia lookups: $0.27 against $0.52 for five questions, both 5/5 correct, on 29x less fresh input.
|
||||
- **23% cheaper than `WebFetch` and 35% cheaper than `WebSearch`** on eleven language docs lookups, at equal or better accuracy.
|
||||
- **125x fewer tokens than raw HTML** across the twelve real pages both could read: 8,519 against 1,064,474. 15x fewer than Jina Reader, 59x fewer than Playwright MCP's accessibility snapshot.
|
||||
- **Real content on every page it could reach, and an honest failure on the two it could not.** Reddit now sends logged-out readers to a login wall, and every other tool returned that wall, or a 403 block page, as a success. The suite hands every tool the old.reddit.com page URLs, which still end at that wall; since 0.5.2 the `oc reddit` shortcuts read the same threads through Reddit's Atom feeds, a route the suite does not measure yet. Yahoo Finance refuses plain fetch outright and DuckDuckGo still blocks lynx; oc's Chrome impersonation read both.
|
||||
- **Half the cost of Claude Code's built-in `WebSearch`** on Wikipedia lookups: $0.23 against $0.45 for five questions, both 5/5 correct, on 25x less fresh input.
|
||||
- **21% cheaper than `WebFetch` and 34% cheaper than `WebSearch`** on eleven language docs lookups, at equal or better accuracy.
|
||||
- **10% cheaper than `WebFetch` and 49% cheaper than `WebSearch`** on twelve dependency lookups across GitHub, npm, PyPI, RubyGems, crates.io, Docker Hub, Stack Overflow and an RFC, 12/12 correct with no tuned shortcut for most of those sites.
|
||||
|
||||
The tables behind those numbers:
|
||||
|
||||
@@ -181,27 +182,30 @@ The tables behind those numbers:
|
||||
|
||||
| method | tokens for 15 pages | notes |
|
||||
| --- | ---: | --- |
|
||||
| `oc open` | 10,936 | the only reader that returned real content on every page |
|
||||
| Jina Reader | 148,479 | both Reddit results are block pages; failed LinkedIn and Yahoo Finance outright |
|
||||
| Playwright MCP | 531,335 | accessibility snapshots; both Reddit snapshots are block pages |
|
||||
| raw HTML fetch | 1,552,491 | the stock quote page alone is 399,881 tokens |
|
||||
| `oc open` | 8,971 | real content on 13 of 15 pages; the two Reddit pages are behind a login wall, and oc is the only reader that reported that instead of returning the wall |
|
||||
| Jina Reader | 105,198 | both Reddit results are block pages; failed LinkedIn and the Node.js `fs` page outright |
|
||||
| Playwright MCP | 531,303 | accessibility snapshots; both Reddit snapshots are the login wall |
|
||||
| raw HTML fetch | 1,240,669 | both Reddit results are the login wall, 88,000 tokens of it each; Yahoo Finance refused the connection |
|
||||
|
||||
oc's budget keeps every page near 500 tokens however much it weighs: the Yahoo Finance quote is 399,881 tokens raw and 456 through oc, Node's `fs` reference 273,820 against 475.
|
||||
oc's budget keeps every page near 500 tokens however much it weighs: the YouTube watch page is 345,487 tokens raw and 688 through oc, Node's `fs` reference 275,425 against 479. On the twelve pages both could read, raw HTML costs 125x what oc does: 1,064,474 against 8,519.
|
||||
|
||||
**Whole tasks against the agent's built-in web tools.** Read cost is one thing, what an agent actually spends is another, so a second set of suites runs full lookups end to end in Claude Code (`claude-sonnet-5`), one tool per run, and grades every answer. Five Wikipedia lookups and eleven language documentation lookups:
|
||||
**Whole tasks against the agent's built-in web tools.** Read cost is one thing, what an agent actually spends is another, so a second set of suites runs full lookups end to end in Claude Code (`claude-sonnet-5`), one tool per run, and grades every answer. Five Wikipedia lookups, eleven language documentation lookups, and twelve lookups on the pages around a dependency, where oc has shortcuts only for GitHub and Stack Overflow and renders the rest generically:
|
||||
|
||||
| suite | tool | correct | input tokens | cost | avg time |
|
||||
| --- | --- | ---: | ---: | ---: | ---: |
|
||||
| Wikipedia | `oc wiki` | 5/5 | 5,535 | $0.27 | 11s |
|
||||
| | built-in `WebFetch` | 5/5 | 128,792 | $0.37 | 14s |
|
||||
| | built-in `WebSearch` | 5/5 | 160,431 | $0.52 | 22s |
|
||||
| Language docs | `oc docs` | 11/11 | 12,965 | $0.57 | 9s |
|
||||
| | built-in `WebFetch` | 10/11 | 203,489 | $0.74 | 11s |
|
||||
| | built-in `WebSearch` | 11/11 | 209,782 | $0.89 | 15s |
|
||||
| Wikipedia | `oc wiki` | 5/5 | 5,535 | $0.23 | 8s |
|
||||
| | built-in `WebFetch` | 5/5 | 129,257 | $0.35 | 12s |
|
||||
| | built-in `WebSearch` | 5/5 | 136,982 | $0.45 | 16s |
|
||||
| Language docs | `oc docs` | 11/11 | 12,967 | $0.56 | 8s |
|
||||
| | built-in `WebFetch` | 10/11 | 203,497 | $0.71 | 12s |
|
||||
| | built-in `WebSearch` | 11/11 | 215,833 | $0.85 | 14s |
|
||||
| Dependency research | `oc open` | 12/12 | 13,261 | $0.63 | 10s |
|
||||
| | built-in `WebFetch` | 10/12 | 173,779 | $0.70 | 11s |
|
||||
| | built-in `WebSearch` | 12/12 | 326,088 | $1.22 | 19s |
|
||||
|
||||
Input tokens are the fresh context each tool put in front of the model, which is the number the page size drives; totals including cache reads sit closer together because the agent's own prompt dominates them. oc stays flat at roughly 1,100 to 1,200 tokens per task, while `WebFetch` pays for whatever the page weighs, from 5.7x more on a short Wikipedia stub to 35x more on the German Berlin article. `WebFetch`'s one wrong answer is an access result: cppreference returns 403 to it, while oc's Chrome impersonation reads the same page. `WebSearch` was given only the question, never the URL, which is the honest way to use it and part of why it costs the most.
|
||||
Input tokens are the fresh context each tool put in front of the model, which is the number the page size drives; totals including cache reads sit closer together because the agent's own prompt dominates them. oc stays flat at roughly 1,100 to 1,200 tokens per task, while `WebFetch` pays for whatever the page weighs, from 5.9x more on a short Wikipedia stub to 35x more on the German Berlin article. `WebFetch`'s three misses are access results: cppreference, npm, and Stack Overflow all refuse it, while oc's Chrome impersonation reads the same pages. The dependency suite is also where oc's generic renderer pays for hard pages: the JavaScript-only crates.io entry and a support table whose row spans several blocks each cost it eight turns, and on the two tasks that start a link away the built-in tools were cheaper. `WebSearch` was given only the question, never the URL, which is the honest way to use it and part of why it costs the most.
|
||||
|
||||
The same suites through Codex (`gpt-5.6-sol`) split. On Wikipedia, `oc wiki` was cheaper and also right where Codex's own search quoted a stale Berlin population. On the docs lookups Codex's search won by 16%: those facts are already in its snippets, and it answered most tasks in two turns without opening a page.
|
||||
The same suites through Codex (`gpt-5.6-sol`, on the 0.4.0 and 0.5.0 runs) split. On Wikipedia, `oc wiki` was cheaper and also right where Codex's own search quoted a stale Berlin population. On the docs lookups Codex's search won by 16%: those facts are already in its snippets, and it answered most tasks in two turns without opening a page.
|
||||
|
||||
## Status
|
||||
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
{
|
||||
"domain": "reddit.com",
|
||||
"commands": {
|
||||
"sub": { "open": "https://old.reddit.com/r/{name}", "args": ["name"] },
|
||||
"post": { "open": "https://old.reddit.com/comments/{id}", "args": ["id"] },
|
||||
"user": { "open": "https://old.reddit.com/user/{name}", "args": ["name"] },
|
||||
"search": { "open": "https://old.reddit.com/search?q={query}", "args": ["query"] }
|
||||
"sub": { "open": "https://www.reddit.com/r/{name}/.rss", "args": ["name"] },
|
||||
"new": { "open": "https://www.reddit.com/r/{name}/new/.rss", "args": ["name"] },
|
||||
"top": { "open": "https://www.reddit.com/r/{name}/top/.rss?t=week", "args": ["name"] },
|
||||
"post": { "open": "https://www.reddit.com/comments/{id}/.rss", "args": ["id"] },
|
||||
"user": { "open": "https://www.reddit.com/user/{name}/.rss", "args": ["name"] },
|
||||
"search": { "open": "https://www.reddit.com/search.rss?q={query}", "args": ["query"] }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -10,7 +10,7 @@ Key facts:
|
||||
- The budget is a target rather than a hard cap: a page that would finish within about four times it is printed whole, because a second command costs the agent far more than the lines the cut would have saved
|
||||
- The render leads with the page's main content and puts navigation, sidebar, and footer after it, so the budget is spent on what was asked for rather than on menus
|
||||
- Benchmarked at roughly 140x fewer tokens than reading raw HTML across fifteen live pages, with per-task numbers at https://github.com/only-cli/benchmarks
|
||||
- Works on any mostly-static website; tuned shortcuts ship for Hacker News, Reddit, GitHub, X, LinkedIn (public guest views), DuckDuckGo, Bing, Stack Overflow (via its Atom feeds and the Stack Exchange API), Yahoo Finance (quotes, history, markets), Wikipedia (articles, search, and other language editions), the AWS, Google Cloud, and Microsoft Learn documentation sites (guides, CLI reference, and search), and the language documentation for Python, JavaScript (MDN), Node.js, Ruby, Go, Rust, Java, PHP, TypeScript, C and C++ (cppreference), and .NET (the Microsoft Learn API browser)
|
||||
- Works on any mostly-static website; tuned shortcuts ship for Hacker News, Reddit (via its Atom feeds, since old.reddit.com and the `.json` views need a login now), GitHub, X, LinkedIn (public guest views), DuckDuckGo, Bing, Stack Overflow (via its Atom feeds and the Stack Exchange API), Yahoo Finance (quotes, history, markets), Wikipedia (articles, search, and other language editions), the AWS, Google Cloud, and Microsoft Learn documentation sites (guides, CLI reference, and search), and the language documentation for Python, JavaScript (MDN), Node.js, Ruby, Go, Rust, Java, PHP, TypeScript, C and C++ (cppreference), and .NET (the Microsoft Learn API browser)
|
||||
- JSON APIs render like pages: an endpoint that answers with JSON becomes one numbered item per record, with the fields that differ between items kept and the ones every item shares stated once, so a search endpoint reads like a results page for a few hundred tokens
|
||||
- A page that comes back with no readable text (JavaScript-only, a consent wall, a bot challenge) prints one line on stderr and exits 2, rather than reporting an empty render as a success. `--json` carries the same verdict as an `empty` field, so a caller can tell "nothing on this page" from "oc could not read this page" and fall back to a browser only when it is worth it
|
||||
- A shortcut is `oc <site> <verb> [args]`: `oc hn top`, `oc reddit sub ClaudeAI`, `oc gh repo only-cli oc`, `oc ddg search claude code cli`, `oc learn doc azure/aks/what-is-aks`, `oc py library json`. Name the site by its short name, bare name, or domain (`oc hn`, `oc ycombinator`, `oc news.ycombinator.com`), the last argument takes every word after it so a query needs no quoting, and `oc sites` lists every site with its verbs. A shortcut resolves to a URL and then behaves exactly like `oc open <url>`
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@only-cli/oc",
|
||||
"version": "0.5.1",
|
||||
"version": "0.5.3",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@only-cli/oc",
|
||||
"version": "0.5.1",
|
||||
"version": "0.5.3",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"impers": "0.1.1",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@only-cli/oc",
|
||||
"version": "0.5.1",
|
||||
"version": "0.5.3",
|
||||
"description": "Turn websites into a compact CLI so AI agents can browse without burning tokens.",
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
@@ -8,15 +8,15 @@ description: Token-efficient web browsing and web content extraction for AI agen
|
||||
Renders a web page as a compact, numbered terminal view instead of raw HTML. A typical page is under 500 tokens.
|
||||
|
||||
```
|
||||
npx --yes @only-cli/oc@0.5.1 open <url> compact view, numbered elements
|
||||
npx --yes @only-cli/oc@0.5.1 do <n> follow link [n], or read it if [n] is text
|
||||
npx --yes @only-cli/oc@0.5.1 find <query> where a string appears, or that place itself
|
||||
npx --yes @only-cli/oc@0.5.3 open <url> compact view, numbered elements
|
||||
npx --yes @only-cli/oc@0.5.3 do <n> follow link [n], or read it if [n] is text
|
||||
npx --yes @only-cli/oc@0.5.3 find <query> where a string appears, or that place itself
|
||||
when only one matches
|
||||
npx --yes @only-cli/oc@0.5.1 next next ~500 tokens of the page already open
|
||||
npx --yes @only-cli/oc@0.5.1 read <n> full text of region [n]
|
||||
npx --yes @only-cli/oc@0.5.1 raw [url] whole page as markdown (--html for cleaned HTML)
|
||||
npx --yes @only-cli/oc@0.5.1 login seed cookies (--cookie, --domain, --expires)
|
||||
npx --yes @only-cli/oc@0.5.1 logout [session] forget a session: cookies and saved page
|
||||
npx --yes @only-cli/oc@0.5.3 next next ~500 tokens of the page already open
|
||||
npx --yes @only-cli/oc@0.5.3 read <n> full text of region [n]
|
||||
npx --yes @only-cli/oc@0.5.3 raw [url] whole page as markdown (--html for cleaned HTML)
|
||||
npx --yes @only-cli/oc@0.5.3 login seed cookies (--cookie, --domain, --expires)
|
||||
npx --yes @only-cli/oc@0.5.3 logout [session] forget a session: cookies and saved page
|
||||
```
|
||||
|
||||
None of these except `open`/`do`/`raw <url>` fetch anything; they replay the page `open` already saved.
|
||||
@@ -32,7 +32,7 @@ oc ddg search claude code oc so question 231767 oc learn doc azur
|
||||
oc py library json oc mdn js Array/map oc node api fs
|
||||
```
|
||||
|
||||
Sites: `hn`, `reddit`, `gh`, `x`, `linkedin`, `ddg`, `bing`, `so`, `yahoo`, `yt`, `aws`, `gcp`, `learn`, `wiki`, `py`, `mdn`, `node`, `ruby`, `go`, `rust`, `java`, `php`, `cpp`, `ts`. Name one by short name, bare name, or domain (`oc hn`, `oc ycombinator`, `oc news.ycombinator.com`). The last argument takes every word after it, so a query or title needs no quoting. `oc sites` lists every site with its verbs, which is cheaper than guessing one.
|
||||
Sites: `hn`, `reddit`, `gh`, `x`, `linkedin`, `ddg`, `bing`, `so`, `yahoo`, `yt`, `aws`, `gcp`, `learn`, `wiki`, `py`, `mdn`, `node`, `ruby`, `go`, `rust`, `java`, `php`, `cpp`, `ts`. Name one by short name, bare name, or domain (`oc hn`, `oc ycombinator`, `oc news.ycombinator.com`). The last argument takes every word after it, so a query or title needs no quoting. `oc sites` lists every site with its verbs, which is cheaper than guessing one. Reddit reads through its Atom feeds: a reddit.com page URL given to `open` meets a login wall or a 403, so use `oc reddit post <id>` or add `/.rss` to the URL, and keep Reddit calls under about ten a minute or the site answers 429.
|
||||
|
||||
Prefer a shortcut over a hand-built URL when one exists for the site, and prefer `oc wiki article <title>` over a search when you already know the article's name.
|
||||
|
||||
|
||||
+31
-6
@@ -555,6 +555,29 @@ function captureSetCookie(jar, url, res) {
|
||||
jar.storeFromResponse(url, getSetCookieHeaders(res));
|
||||
}
|
||||
|
||||
// Hosts whose edge answers the chrome fingerprint with a 403 or a 429 while
|
||||
// letting firefox through. reddit.com started doing this in 2026 (#52), so
|
||||
// starting with chrome there would turn every read into two requests against
|
||||
// a per-address rate limit of about ten a minute. Subdomains inherit the
|
||||
// entry.
|
||||
const FIREFOX_FIRST_HOSTS = ['reddit.com'];
|
||||
|
||||
/**
|
||||
* The order in which impers identities are tried for a URL.
|
||||
* @param {string} target
|
||||
* @returns {['chrome', 'firefox'] | ['firefox', 'chrome']}
|
||||
*/
|
||||
export function identityOrder(target) {
|
||||
let host = '';
|
||||
try {
|
||||
host = new URL(target).hostname.toLowerCase();
|
||||
} catch {
|
||||
return ['chrome', 'firefox'];
|
||||
}
|
||||
const firefoxFirst = FIREFOX_FIRST_HOSTS.some((h) => host === h || host.endsWith(`.${h}`));
|
||||
return firefoxFirst ? ['firefox', 'chrome'] : ['chrome', 'firefox'];
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch a page through impers, downgrading identity when one is refused.
|
||||
* Exported so the downgrade chain can be proven against a fake impers; the
|
||||
@@ -565,14 +588,15 @@ function captureSetCookie(jar, url, res) {
|
||||
* @returns {Promise<{url: string, html: string, status: number, via: string}>}
|
||||
*/
|
||||
export async function viaImpers(impers, target, jar) {
|
||||
// Some sites (Reddit) 403 the chrome fingerprint but accept firefox, so a
|
||||
// blocked first attempt gets one cheap retry with a second identity. An
|
||||
// A blocked first attempt gets one cheap retry with the other identity. An
|
||||
// ImpersonateError is the same story one layer down: impers resolves the
|
||||
// 'chrome' alias to its newest fingerprint, but the native library it loads
|
||||
// can be an older system copy of libcurl-impersonate that predates that
|
||||
// fingerprint and refuses it before any request leaves. Firefox aliases to
|
||||
// an older target that such a library usually still knows, and when both
|
||||
// identities are refused the plain fetch transport still gets the page.
|
||||
// Hosts that are known to refuse chrome outright start with firefox, so the
|
||||
// usual case there costs one request instead of a 403 and a retry.
|
||||
const asking = (impersonate) => (url) =>
|
||||
impers.get(url, {
|
||||
impersonate,
|
||||
@@ -590,11 +614,12 @@ export async function viaImpers(impers, target, jar) {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
let via = 'impers:chrome';
|
||||
let got = await attempt('chrome');
|
||||
const [first, second] = identityOrder(target);
|
||||
let via = `impers:${first}`;
|
||||
let got = await attempt(first);
|
||||
if (!got || got.status >= 400) {
|
||||
via = 'impers:firefox';
|
||||
got = (await attempt('firefox')) ?? got;
|
||||
via = `impers:${second}`;
|
||||
got = (await attempt(second)) ?? got;
|
||||
}
|
||||
if (!got) return viaFetch(target, jar);
|
||||
const { res, status } = got;
|
||||
|
||||
@@ -145,6 +145,30 @@ test('atom feeds render as pages: entries become headings, bodies unescape', ()
|
||||
assert.ok(!text.includes('<'), 'entry body left escaped');
|
||||
});
|
||||
|
||||
test('a reddit post feed renders as the post followed by its comments', () => {
|
||||
// Reddit closed old.reddit.com and its .json views to logged-out readers in
|
||||
// 2026; the Atom feeds on www.reddit.com are what oc reddit rides now. A
|
||||
// post feed is one entry for the post and one per comment, each comment
|
||||
// titled "/u/name on <post title>", so the whole thread reads as one page.
|
||||
const xml = readFileSync(new URL('./pages/reddit_post.xml', import.meta.url), 'utf8');
|
||||
const p = distill(xml, 'https://www.reddit.com/comments/1fixture/.rss');
|
||||
assert.equal(p.title, 'Why does the budget flag round up? : reddit.com');
|
||||
const headings = p.blocks.filter((b) => b.type === 'heading').map((b) => b.text);
|
||||
assert.deepEqual(headings, [
|
||||
'Why does the budget flag round up?',
|
||||
'/u/first_reply on Why does the budget flag round up?',
|
||||
'/u/second_reply on Why does the budget flag round up?',
|
||||
]);
|
||||
const text = p.blocks.map((b) => b.text).join(' ');
|
||||
assert.ok(text.includes('Is that on purpose?'), 'post body missing');
|
||||
assert.ok(text.includes('by /u/fixture_poster, 2026-09-01'), 'post byline missing');
|
||||
assert.ok(text.includes('One extra tool call costs more'), 'comment body missing');
|
||||
assert.ok(!text.includes('SC_OFF'), 'reddit markup comments leaked into the text');
|
||||
const rendered = render(p).text;
|
||||
assert.ok(rendered.includes('## [1] Why does the budget flag round up?'), 'post is not the first numbered heading');
|
||||
assert.ok(estimateTokens(rendered) < 500, `three-entry thread should fit the default budget, got ${estimateTokens(rendered)}`);
|
||||
});
|
||||
|
||||
test('feed entry code blocks survive raw markdown', () => {
|
||||
const md = toMarkdown(feed);
|
||||
assert.ok(md.startsWith('# Why is the sky blue? - Fixture Overflow'));
|
||||
|
||||
+28
-1
@@ -5,7 +5,7 @@ import https from 'node:https';
|
||||
import net from 'node:net';
|
||||
import tls from 'node:tls';
|
||||
|
||||
const { fetchPage, followRedirects, resolveProxy, proxyGet, viaImpers } = await import('../src/fetch.js');
|
||||
const { fetchPage, followRedirects, identityOrder, resolveProxy, proxyGet, viaImpers } = await import('../src/fetch.js');
|
||||
|
||||
const BLOCKED_MESSAGE = 'blocked: private or internal URL';
|
||||
|
||||
@@ -791,6 +791,33 @@ test('when both identities are refused the page still arrives via plain fetch',
|
||||
}
|
||||
});
|
||||
|
||||
test('reddit.com is asked with the firefox fingerprint first', () => withoutProxyEnv(async () => {
|
||||
// Reddit's edge answers the chrome fingerprint with a 403 or a 429 while
|
||||
// letting firefox through (#52), and it rate-limits anonymous readers per
|
||||
// address, so a wasted chrome attempt there is a real cost, not a retry.
|
||||
assert.deepEqual(identityOrder('https://www.reddit.com/r/ClaudeAI/.rss'), ['firefox', 'chrome']);
|
||||
assert.deepEqual(identityOrder('https://old.reddit.com/r/ClaudeAI/'), ['firefox', 'chrome']);
|
||||
assert.deepEqual(identityOrder('https://reddit.com/'), ['firefox', 'chrome']);
|
||||
assert.deepEqual(identityOrder('https://notreddit.com/'), ['chrome', 'firefox']);
|
||||
assert.deepEqual(identityOrder('https://reddit.com.example/'), ['chrome', 'firefox']);
|
||||
assert.deepEqual(identityOrder('https://news.ycombinator.com/'), ['chrome', 'firefox']);
|
||||
assert.deepEqual(identityOrder('not a url'), ['chrome', 'firefox']);
|
||||
|
||||
const impers = fakeImpers([]);
|
||||
const page = await viaImpers(impers, 'https://www.reddit.com/r/ClaudeAI/.rss');
|
||||
assert.deepEqual(impers.identities, ['firefox']);
|
||||
assert.equal(page.via, 'impers:firefox');
|
||||
assert.equal(page.status, 200);
|
||||
}));
|
||||
|
||||
test('a refused firefox fingerprint on reddit.com falls back to chrome', () => withoutProxyEnv(async () => {
|
||||
const impers = fakeImpers(['firefox']);
|
||||
const page = await viaImpers(impers, 'https://www.reddit.com/r/ClaudeAI/.rss');
|
||||
assert.deepEqual(impers.identities, ['firefox', 'chrome']);
|
||||
assert.equal(page.via, 'impers:chrome');
|
||||
assert.equal(page.status, 200);
|
||||
}));
|
||||
|
||||
test('only an ImpersonateError downgrades; other impers failures propagate', () => withoutProxyEnv(async () => {
|
||||
const impers = {
|
||||
get: () => Promise.reject(new Error('connection reset')),
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?><feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/"><category term=" reddit.com" label="r/ reddit.com"/><updated>2026-09-04T12:39:19+00:00</updated><id>/comments/1fixture/.rss</id><link rel="self" href="https://www.reddit.com/comments/1fixture/.rss" type="application/atom+xml" /><link rel="alternate" href="https://www.reddit.com/comments/1fixture/" type="text/html" /><title>Why does the budget flag round up? : reddit.com</title><entry><author><name>/u/fixture_poster</name><uri>https://www.reddit.com/user/fixture_poster</uri></author><category term="FixtureSub" label="r/FixtureSub"/><content type="html"><!-- SC_OFF --><div class="md"><p>A page that runs a little over the budget prints whole instead of being cut. Is that on purpose?</p> </div><!-- SC_ON --> &#32; submitted by &#32; <a href="https://www.reddit.com/user/fixture_poster"> /u/fixture_poster </a> &#32; to &#32; <a href="https://www.reddit.com/r/FixtureSub/"> r/FixtureSub </a> <br/> <span><a href="https://www.reddit.com/r/FixtureSub/comments/1fixture/why_does_the_budget_flag_round_up/">[link]</a></span> &#32; <span><a href="https://www.reddit.com/r/FixtureSub/comments/1fixture/why_does_the_budget_flag_round_up/">[comments]</a></span></content><id>t3_1fixture</id><link href="https://www.reddit.com/r/FixtureSub/comments/1fixture/why_does_the_budget_flag_round_up/" /><updated>2026-09-01T11:18:08+00:00</updated><published>2026-09-01T11:18:08+00:00</published><title>Why does the budget flag round up?</title></entry><entry><author><name>/u/first_reply</name><uri>https://www.reddit.com/user/first_reply</uri></author><category term="FixtureSub" label="r/FixtureSub" /><content type="html"><!-- SC_OFF --><div class="md"><p>Yes. One extra tool call costs more than the tokens it would save.</p> </div><!-- SC_ON --></content><id>t1_c0000001</id><link href="https://www.reddit.com/r/FixtureSub/comments/1fixture/why_does_the_budget_flag_round_up/c0000001/"/><updated>2026-09-01T13:50:03+00:00</updated><title>/u/first_reply on Why does the budget flag round up?</title></entry><entry><author><name>/u/second_reply</name><uri>https://www.reddit.com/user/second_reply</uri></author><category term="FixtureSub" label="r/FixtureSub" /><content type="html"><!-- SC_OFF --><div class="md"><p>The README calls it a target rather than a hard cap.</p> </div><!-- SC_ON --></content><id>t1_c0000002</id><link href="https://www.reddit.com/r/FixtureSub/comments/1fixture/why_does_the_budget_flag_round_up/c0000002/"/><updated>2026-09-01T14:02:11+00:00</updated><title>/u/second_reply on Why does the budget flag round up?</title></entry></feed>
|
||||
@@ -42,6 +42,19 @@ test('a real site with a missing or unknown verb names the verbs it has', () =>
|
||||
/not a reddit\.com shortcut.*sub <name>/s);
|
||||
});
|
||||
|
||||
test('reddit verbs reach the www.reddit.com atom feeds, not old.reddit.com', () => {
|
||||
// old.reddit.com sends every logged-out request to a login page and the
|
||||
// .json views on www answer 403, so the feeds are the only public reading.
|
||||
assert.equal(resolveSite('reddit', ['sub', 'ClaudeAI']).url, 'https://www.reddit.com/r/ClaudeAI/.rss');
|
||||
assert.equal(resolveSite('reddit', ['post', '1w48zcr']).url, 'https://www.reddit.com/comments/1w48zcr/.rss');
|
||||
assert.equal(resolveSite('reddit', ['search', 'claude code']).url, 'https://www.reddit.com/search.rss?q=claude%20code');
|
||||
for (const verb of Object.keys(sites().get('reddit').commands)) {
|
||||
const { url } = resolveSite('reddit', [verb, 'x']);
|
||||
assert.ok(url.startsWith('https://www.reddit.com/'), `${verb} left www: ${url}`);
|
||||
assert.ok(/\.rss(\?|$)/.test(url), `${verb} is not a feed: ${url}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('a shortcut called with too few args says what it needs', () => {
|
||||
assert.throws(() => resolveSite('gh', ['repo', 'only-cli']), /usage: oc gh repo <owner> <name>/);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user