mirror of
https://github.com/only-cli/oc.git
synced 2026-09-15 10:40:56 +02:00
merge main: authenticated sessions via per-session cookie jars
This commit is contained in:
only-cli
@@ -2,3 +2,6 @@ node_modules
|
||||
*.log
|
||||
.idea/
|
||||
.env
|
||||
# Cookie sidecars and page snapshots hold credentials; never commit them.
|
||||
sessions/
|
||||
*.cookies.json
|
||||
|
||||
@@ -97,10 +97,32 @@ oc <site> <verb> ... site shortcut: 'oc hn top', 'oc reddit sub ClaudeAI'
|
||||
oc sites the site shortcuts that ship with oc
|
||||
oc fill <n> <text> type into a numbered input (planned)
|
||||
oc submit [n] submit a form (planned)
|
||||
oc login seed cookies for a session (--cookie, --domain)
|
||||
oc logout [session] forget a session: cookies and saved page
|
||||
```
|
||||
|
||||
Flags: `--budget <tokens>` (default 500), `--json`, `--html` (raw as cleaned HTML), `--session <name>`, `--verbose`/`-v` (metrics on stderr, or export `OC_VERBOSE=1`).
|
||||
|
||||
### Authenticated sessions
|
||||
|
||||
Pages behind a login need cookies. Seed them once per session, then browse normally:
|
||||
|
||||
```bash
|
||||
printf %s "session=...; auth=..." | oc login --cookie - --domain example.com --expires 2h --session work
|
||||
oc open https://example.com/dashboard --session work
|
||||
oc logout work
|
||||
```
|
||||
|
||||
Prefer `--cookie -`, which reads the header from stdin. The flag also takes the header inline (`--cookie "session=..."`), but an argument is a live credential in `ps` for as long as `oc` runs and in your shell history afterwards.
|
||||
|
||||
Copy the `Cookie` header from your browser's devtools (Application → Cookies, or the Network tab on a request); a leading `Cookie:` is stripped for you. `--domain` is the site hostname those cookies belong to, and it has to be a real hostname: a bare TLD like `com` is refused, because the match is a suffix match and those cookies would go to every `.com` host the session ever fetched. Cookie names and values are checked at login too, so a stray control character fails there rather than deep inside the HTTP client.
|
||||
|
||||
Seeded cookies are https-only. They almost always come from an https browser session, so `oc` marks them secure and never sends them over plain `http` — including on a hop an `https` page redirects into, where you never typed the downgrade. A site that really is http-only needs `--allow-http` at login. Cookies a site sets over https are pinned the same way.
|
||||
|
||||
Cookies live in a separate sidecar file (`<session>.cookies.json`) under `~/.only-cli/sessions/`, mode `0600`, not in the page-state JSON and never in `--json` output. The default lifetime is one hour (`--expires 1h`), and a jar holds at most 50 cookies so a page cannot bloat it. When cookies expire or the site returns a login page, `oc` says so plainly (exit 2) instead of distilling the login form as content.
|
||||
|
||||
`oc logout` forgets the whole session, not just its cookies: a page saved under that name can hold the distilled text of something only the login could reach, so the snapshot goes with the jar.
|
||||
|
||||
`oc open` remembers the page it rendered in a JSON file per session under `~/.only-cli` (override with `OC_HOME`), so `oc do 3` follows `[3]` without the agent ever handling a URL. A result title on a search page is a link, so `oc do` on it opens the result rather than repeating the title. Pages longer than the budget say what they left out; `oc find`, `oc read <n>`, and `oc next` read the rest without refetching the page, and a `find` with a single match prints that region instead of the number to read it with. The budget is a target rather than a hard cap: a page that would only run a little long is printed whole rather than cut, since one extra tool call costs far more than the tokens it would have saved.
|
||||
|
||||
When a page comes back with no readable text (JavaScript-only, a consent wall, a bot challenge), `oc` says so in one line on stderr and exits 2 instead of printing a title and calling it a render. That is a different exit code from every other failure, and `--json` carries the same verdict as an `empty` field, so an agent can tell "this page has nothing on it" from "oc could not read this page" and pay for a browser only when it is worth it.
|
||||
@@ -138,7 +160,7 @@ Works on any mostly-static site with no per-site setup: news sites, blogs, docum
|
||||
|
||||
A shortcut only ever resolves to a URL and then takes the same path `oc open` does, so it changes nothing about what a page costs or how it reads. The last argument takes every word after it, so `oc ddg search claude code cli` and `oc aws search s3 lifecycle rules` need no quoting, and a path argument keeps its slashes, so `oc learn doc azure/aks/what-is-aks` reaches that page.
|
||||
|
||||
A few of these (X, Stack Overflow, YouTube, Microsoft Learn search) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question <id>` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. AWS, Google Cloud, Rust, Java, TypeScript, PHP, and cppreference render docs search client-side, or as a page too bare for oc to read, so their `search` goes through DuckDuckGo with a baked-in `site:` filter instead; Go needs no such fallback, because pkg.go.dev renders its search results on the server and `oc go search` simply opens them. Python's docs are built with Sphinx, which publishes the site's full-text search index as one static file, so `oc py search` fetches that index (cached on disk for a day), ranks it locally, and prints a numbered result list; a query that names a symbol exactly, like `json.dumps`, links straight to its anchor. The same backend will work for any Sphinx site, including most Read the Docs projects. MDN also renders its search client-side, but the page gets its results from a public JSON endpoint, so `oc mdn search` asks that endpoint directly and prints the site's own ranking; that `api` shape in a site definition works for any site whose search answers as JSON. Node.js ships no search endpoint at all, but publishes its whole API reference as one static JSON file, so `oc node search` ranks that file locally the same way the Sphinx backend does, under the same day cache, and every module, class, method, property, and event heading links to its own anchor. Ruby's docs are built with RDoc, which also ships its search index as one static file, so `oc ruby search` ranks every class, method, and guide page locally the same way. PHP's manual has a lookup endpoint that sends an exact function name straight to its page, which is what `oc php fn` rides. Not supported yet: pages that only render with JavaScript, sites behind logins, and sites with hard bot challenges that expose no feed.
|
||||
A few of these (X, Stack Overflow, YouTube, Microsoft Learn search) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question <id>` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. AWS, Google Cloud, Rust, Java, TypeScript, PHP, and cppreference render docs search client-side, or as a page too bare for oc to read, so their `search` goes through DuckDuckGo with a baked-in `site:` filter instead; Go needs no such fallback, because pkg.go.dev renders its search results on the server and `oc go search` simply opens them. Python's docs are built with Sphinx, which publishes the site's full-text search index as one static file, so `oc py search` fetches that index (cached on disk for a day), ranks it locally, and prints a numbered result list; a query that names a symbol exactly, like `json.dumps`, links straight to its anchor. The same backend will work for any Sphinx site, including most Read the Docs projects. MDN also renders its search client-side, but the page gets its results from a public JSON endpoint, so `oc mdn search` asks that endpoint directly and prints the site's own ranking; that `api` shape in a site definition works for any site whose search answers as JSON. Node.js ships no search endpoint at all, but publishes its whole API reference as one static JSON file, so `oc node search` ranks that file locally the same way the Sphinx backend does, under the same day cache, and every module, class, method, property, and event heading links to its own anchor. Ruby's docs are built with RDoc, which also ships its search index as one static file, so `oc ruby search` ranks every class, method, and guide page locally the same way. PHP's manual has a lookup endpoint that sends an exact function name straight to its page, which is what `oc php fn` rides. Not supported yet: pages that only render with JavaScript and sites with hard bot challenges that expose no feed. Sites that genuinely require your account can be reached with `oc login` (bring your own cookies).
|
||||
|
||||
Want a website on that list? Open a pull request, or an issue naming the site; see [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
@@ -175,9 +197,9 @@ part of why it costs the most.
|
||||
|
||||
## Status
|
||||
|
||||
Early. Reading works and is covered by offline tests: static pages, XML feeds, JSON APIs, budget-aware rendering, sessions, and the numbered actions `do`, `find`, `read`, `next`, and `raw`. Writing does not: `fill`, `submit`, and `back` report that they are not implemented rather than pretending, and a lazy headless fallback for script-heavy pages comes after them.
|
||||
Early. Reading works and is covered by offline tests: static pages, XML feeds, JSON APIs, budget-aware rendering, sessions, authenticated cookie jars, and the numbered actions `do`, `find`, `read`, `next`, and `raw`. Writing does not: `fill`, `submit`, and `back` report that they are not implemented rather than pretending, and a lazy headless fallback for script-heavy pages comes after them.
|
||||
|
||||
Known limits, honestly: no JavaScript rendering yet, no sites behind logins yet, and pages behind hard bot challenges may still refuse the tool.
|
||||
Known limits, honestly: no JavaScript rendering yet, and pages behind hard bot challenges may still refuse the tool.
|
||||
|
||||
## Contributors
|
||||
|
||||
|
||||
@@ -18,7 +18,8 @@ Key facts:
|
||||
- Outbound fetches honor `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` (and their lowercase forms), so oc works in a sandbox whose only route to the network is a proxy. An https target is tunneled with CONNECT and its certificate is still verified, credentials in the proxy URL reach the proxy and nothing else, and private or locally unresolvable targets stay refused. `ALL_PROXY` is not read
|
||||
- Requests impersonate Chrome, so pages that block plain scripts often still work
|
||||
- Agent skill included: `npx skills add https://github.com/only-cli/oc --skill web-browsing-cli` ([skills.sh](https://www.skills.sh/only-cli/oc/web-browsing-cli))
|
||||
- No JavaScript rendering yet and no login sessions yet (both on the roadmap)
|
||||
- Authenticated pages: `printf %s "..." | oc login --cookie - --domain example.com [--expires 1h] [--session name]` seeds a timeboxed cookie jar; cookies are sent on every fetch for that session and live in a separate file from page state. `--cookie -` reads the header from stdin, which keeps the credential out of `ps` and shell history; `--domain` must be a real hostname, not a bare TLD. Seeded cookies are https-only unless `--allow-http` says the site is not, so a redirect that downgrades to `http` drops them. `oc logout` forgets that session's cookies and its saved page
|
||||
- No JavaScript rendering yet (on the roadmap)
|
||||
|
||||
## Docs
|
||||
|
||||
|
||||
@@ -15,6 +15,8 @@ npx --yes @only-cli/oc@0.4.0 find <query> where a string appears, or that plac
|
||||
npx --yes @only-cli/oc@0.4.0 next next ~500 tokens of the page already open
|
||||
npx --yes @only-cli/oc@0.4.0 read <n> full text of region [n]
|
||||
npx --yes @only-cli/oc@0.4.0 raw [url] whole page as markdown (--html for cleaned HTML)
|
||||
npx --yes @only-cli/oc@0.4.0 login seed cookies (--cookie, --domain, --expires)
|
||||
npx --yes @only-cli/oc@0.4.0 logout [session] forget a session: cookies and saved page
|
||||
```
|
||||
|
||||
None of these except `open`/`do`/`raw <url>` fetch anything; they replay the page `open` already saved.
|
||||
@@ -74,9 +76,23 @@ Prefer a shortcut over a hand-built URL when one exists for the site, and prefer
|
||||
|
||||
`HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` are honored automatically: no flag, no setup. An error starting `proxy` is the network between the machine and the site, not the page. `blocked: private or internal URL` means the target is private, or does not resolve while a proxy is set. Neither succeeds on retry: report it rather than trying other URLs.
|
||||
|
||||
## Authenticated pages
|
||||
|
||||
Sites that need your account: seed cookies once, then browse normally.
|
||||
|
||||
```bash
|
||||
printf %s "session=...; auth=..." | oc login --cookie - --domain example.com --expires 2h --session work
|
||||
oc open https://example.com/dashboard --session work
|
||||
oc logout work
|
||||
```
|
||||
|
||||
Pass `--cookie -` and pipe the header in, as above: an inline `--cookie "session=..."` puts a live credential in `ps` and in shell history. Copy the header from browser devtools. `--domain` must be a real hostname — a bare TLD like `com` is refused, since the cookies would then go to every `.com` host the session fetched.
|
||||
|
||||
Default lifetime is 1h. Seeded cookies are https-only: they are never sent over plain `http`, including on a redirect that downgrades, unless you seeded them with `--allow-http`. When cookies expire or the site returns a login page, `oc` says so (exit 2) instead of rendering the login form as content. Cookies live in a separate file from page state and are never included in `--json` output. `oc logout` drops that session's saved page along with its cookies.
|
||||
|
||||
## When not to use it
|
||||
|
||||
Pages needing login or heavy client-side JS aren't supported yet. A page with no readable text (JavaScript-only, a consent wall, a bot challenge) prints one line on stderr and exits 2, which is distinct from the exit 1 every other failure uses, so exit 2 means "oc cannot read this one" rather than "this page is empty". Take it at its word: say so and fall back to another tool rather than retrying the same URL.
|
||||
Pages needing heavy client-side JS aren't supported yet. A page with no readable text (JavaScript-only, a consent wall, a bot challenge) prints one line on stderr and exits 2, which is distinct from the exit 1 every other failure uses, so exit 2 means "oc cannot read this one" rather than "this page is empty". Take it at its word: say so and fall back to another tool rather than retrying the same URL.
|
||||
|
||||
## Untrusted content
|
||||
|
||||
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
/**
|
||||
* Detect when a fetched page is a login gate rather than the content the
|
||||
* caller expected. Runs before contentFailure so a thin login form is not
|
||||
* partially rendered as if it were real content.
|
||||
*/
|
||||
|
||||
const LOGIN_PATH = /\/(?:login|signin|sign-in|auth|oauth|sso)(?:\/|$|\?)/i;
|
||||
const LOGIN_TITLE = /\b(?:log\s*in|sign\s*in|authenticate)\b/i;
|
||||
const LOGIN_BUTTON = /\b(?:log\s*in|sign\s*in|continue|submit)\b/i;
|
||||
|
||||
/**
|
||||
* @param {string} url
|
||||
* @returns {string}
|
||||
*/
|
||||
export function sessionExpiredMessage(url) {
|
||||
return `session expired or cookies are no longer valid for ${url}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {import('./distill.js').Page} page
|
||||
* @param {string} url
|
||||
* @param {{ hadAuth?: boolean }} [opts]
|
||||
* @returns {string|null}
|
||||
*/
|
||||
export function authFailure(page, url, { hadAuth = false } = {}) {
|
||||
const hasPassword = page.blocks.some((b) => b.type === 'input' && b.text === 'password');
|
||||
if (!hasPassword) return null;
|
||||
|
||||
let pathname = '';
|
||||
try {
|
||||
pathname = new URL(url).pathname;
|
||||
} catch {
|
||||
// malformed URL: rely on other signals only
|
||||
}
|
||||
|
||||
const loginUrl = LOGIN_PATH.test(pathname);
|
||||
const loginTitle = LOGIN_TITLE.test(page.title ?? '');
|
||||
const loginButton = page.blocks.some((b) =>
|
||||
(b.type === 'button' || b.type === 'input') && LOGIN_BUTTON.test(b.text ?? ''),
|
||||
);
|
||||
|
||||
if (!loginUrl && !loginTitle && !loginButton) return null;
|
||||
|
||||
if (hadAuth) return sessionExpiredMessage(url);
|
||||
return 'this page requires login; run \'printf %s "session=..." | oc login --cookie - --domain example.com\'';
|
||||
}
|
||||
+121
-7
@@ -1,5 +1,6 @@
|
||||
#!/usr/bin/env node
|
||||
import { parseArgs } from 'node:util';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { fetchPage } from './fetch.js';
|
||||
import { distill, toMarkdown, toHTML } from './distill.js';
|
||||
import { render, estimateTokens, contentTokens, contentFailure, MIN_CONTENT } from './render.js';
|
||||
@@ -9,7 +10,19 @@ import { nodeSearch } from './nodedocs.js';
|
||||
import { rdocSearch } from './rdoc.js';
|
||||
import { apiSearch } from './apisearch.js';
|
||||
import * as act from './act.js';
|
||||
import { DEFAULT_SESSION, loadSession, saveSession, sessionFromPage } from './session.js';
|
||||
import { DEFAULT_SESSION, assertSafeName, clearSession, loadSession, saveSession, sessionFromPage } from './session.js';
|
||||
import { authFailure, sessionExpiredMessage } from './auth.js';
|
||||
import {
|
||||
loadCookieJar,
|
||||
saveCookieJar,
|
||||
clearCookieJar,
|
||||
createJarHandle,
|
||||
loginCookieJar,
|
||||
parseExpires,
|
||||
withheldForScheme,
|
||||
DEFAULT_EXPIRES_MS,
|
||||
JAR_EXPIRED,
|
||||
} from './cookies.js';
|
||||
|
||||
const HELP = `only-cli: the web as a compact terminal, built for AI agents.
|
||||
|
||||
@@ -27,6 +40,8 @@ usage: oc <command> [args] [flags]
|
||||
fill <n> <text> type into a numbered input (planned)
|
||||
submit [n] submit a form (planned)
|
||||
back return to the previous page (planned)
|
||||
login seed cookies for a session (--cookie, --domain)
|
||||
logout [session] forget a session: its cookies and its saved page
|
||||
session ls|rm manage saved sessions (planned)
|
||||
|
||||
flags:
|
||||
@@ -41,6 +56,20 @@ flags:
|
||||
memory. --stats is an alias; OC_VERBOSE=1 turns it on
|
||||
globally. Off by default because metrics cost tokens too.
|
||||
--session <name> keep separate page state under a name (default: default)
|
||||
--cookie <header> login only: the Cookie header to seed. '-' reads it from
|
||||
stdin, which is the form to prefer: an argv secret is
|
||||
visible in ps and kept in shell history
|
||||
--domain <host> login only: the hostname those cookies belong to
|
||||
--expires <dur> login only: how long the session lasts (default 1h)
|
||||
--allow-http login only: let these cookies travel over plain http
|
||||
|
||||
Authenticated pages: run 'printf %s "session=..." | oc login --cookie - --domain
|
||||
example.com' to seed cookies for a session (default lifetime 1h, override with
|
||||
--expires 2h). Cookies live in a separate file from page state and are sent on
|
||||
every fetch for that session. They are marked secure, so they go over https
|
||||
only and a redirect down to http drops them; pass --allow-http at login if a
|
||||
site really is http-only. 'oc logout' forgets the session early: cookies and
|
||||
saved page both.
|
||||
|
||||
A page that comes back with no readable text (JavaScript-only, a consent wall,
|
||||
a bot challenge) says so in one line on stderr and exits 2, so a caller can
|
||||
@@ -82,10 +111,36 @@ const noContent = (url, detail, hint = "; 'oc raw' has the page's markdown if th
|
||||
process.exitCode = NO_CONTENT_EXIT;
|
||||
};
|
||||
|
||||
const LOGIN_USAGE = 'usage: printf %s "session=..." | oc login --cookie - --domain example.com'
|
||||
+ ' [--expires 1h] [--session name] [--allow-http]';
|
||||
|
||||
const stripCookiePrefix = (value) => String(value).trim().replace(/^cookie\s*:\s*/i, '');
|
||||
|
||||
// The Cookie header a browser hands over is a live credential, and an argv
|
||||
// secret is readable in ps for as long as oc runs and kept in shell history
|
||||
// afterwards, so '-' reads it from stdin instead. The flag form stays, because
|
||||
// it is what an agent already has in hand, but the docs lead with the pipe.
|
||||
// Devtools' "copy as cURL"-style output carries the header name, and a strict
|
||||
// cookie-name check would only report that as a puzzling parse error, so a
|
||||
// leading 'Cookie:' is dropped here rather than refused.
|
||||
const cookieHeaderArg = (value) => {
|
||||
if (value !== '-') return stripCookiePrefix(value);
|
||||
if (process.stdin.isTTY) throw new Error(`--cookie - reads the header from stdin, ${LOGIN_USAGE}`);
|
||||
let raw;
|
||||
try {
|
||||
raw = readFileSync(0, 'utf8');
|
||||
} catch (err) {
|
||||
throw new Error(`could not read the cookie header from stdin (${err.message})`);
|
||||
}
|
||||
const header = stripCookiePrefix(raw);
|
||||
if (!header) throw new Error(`nothing on stdin to read a cookie header from, ${LOGIN_USAGE}`);
|
||||
return header;
|
||||
};
|
||||
|
||||
// Anything else in the first position is tried as a site shortcut before it is
|
||||
// called unknown, so a new clis/ definition needs no change here.
|
||||
const COMMANDS = new Set([
|
||||
'open', 'do', 'raw', 'read', 'next', 'find', 'fill', 'submit', 'back', 'session', 'sites',
|
||||
'open', 'do', 'raw', 'read', 'next', 'find', 'fill', 'submit', 'back', 'login', 'logout', 'session', 'sites',
|
||||
]);
|
||||
|
||||
async function main() {
|
||||
@@ -98,6 +153,10 @@ async function main() {
|
||||
verbose: { type: 'boolean', short: 'v', default: false },
|
||||
budget: { type: 'string' },
|
||||
session: { type: 'string' },
|
||||
cookie: { type: 'string' },
|
||||
domain: { type: 'string' },
|
||||
expires: { type: 'string' },
|
||||
'allow-http': { type: 'boolean', default: false },
|
||||
help: { type: 'boolean', short: 'h', default: false },
|
||||
},
|
||||
});
|
||||
@@ -129,7 +188,7 @@ async function main() {
|
||||
}
|
||||
}
|
||||
|
||||
const sessionName = values.session || DEFAULT_SESSION;
|
||||
const sessionName = assertSafeName(values.session || DEFAULT_SESSION);
|
||||
// Zero means "whatever this command's default is", which differs: the
|
||||
// compact view targets 500 tokens, read targets 2000.
|
||||
const asked = values.budget ? Number(values.budget) : 0;
|
||||
@@ -137,6 +196,26 @@ async function main() {
|
||||
throw new Error('--budget must be a positive number');
|
||||
}
|
||||
|
||||
if (command === 'login') {
|
||||
if (!values.cookie) throw new Error(LOGIN_USAGE);
|
||||
if (!values.domain) throw new Error('--domain is required (the site hostname your cookies belong to)');
|
||||
const expiresMs = values.expires ? parseExpires(values.expires) : DEFAULT_EXPIRES_MS;
|
||||
loginCookieJar(sessionName, cookieHeaderArg(values.cookie), values.domain, {
|
||||
expiresMs,
|
||||
allowHttp: values['allow-http'],
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (command === 'logout') {
|
||||
const name = args[0] ? assertSafeName(args[0]) : sessionName;
|
||||
clearCookieJar(name);
|
||||
// The page saved under this name can be the distilled text of a page only
|
||||
// the cookies could reach, so logout drops it too.
|
||||
clearSession(name);
|
||||
return;
|
||||
}
|
||||
|
||||
switch (command) {
|
||||
case 'open':
|
||||
case 'do':
|
||||
@@ -159,8 +238,24 @@ async function main() {
|
||||
}
|
||||
if (!url) throw new Error(`usage: oc ${command} <url>`);
|
||||
const budget = asked || 500;
|
||||
const jarData = loadCookieJar(sessionName);
|
||||
// Clock-expired jars are wiped on load; say so in the same voice as a
|
||||
// login-page redirect rather than fetching with no cookies and guessing.
|
||||
if (jarData === JAR_EXPIRED) {
|
||||
noContent(url, sessionExpiredMessage(url));
|
||||
return;
|
||||
}
|
||||
const hadAuth = jarData != null;
|
||||
// Secure cookies are withheld from a plain-http request. Say so, or the
|
||||
// fetch comes back a login page and nothing explains why.
|
||||
if (jarData && withheldForScheme(jarData, url)) {
|
||||
console.error(`oc: warning: session '${sessionName}' holds https-only cookies, so they are not sent to ${url}`
|
||||
+ '; seed them with --allow-http if this site really is http-only');
|
||||
}
|
||||
const jar = jarData ? createJarHandle(sessionName, jarData) : null;
|
||||
const t0 = performance.now();
|
||||
const { url: finalUrl, html, status, via } = await fetchPage(url);
|
||||
const { url: finalUrl, html, status, via } = await fetchPage(url, { jar: jar ?? undefined });
|
||||
if (jar) saveCookieJar(sessionName, jar.toJSON());
|
||||
const fetchMs = performance.now() - t0;
|
||||
const resources = () => {
|
||||
const processMs = performance.now() - t0 - fetchMs;
|
||||
@@ -171,16 +266,29 @@ async function main() {
|
||||
const htmlTokens = estimateTokens(html);
|
||||
if (values.json) {
|
||||
const page = distill(html, finalUrl);
|
||||
remember(page, sessionName);
|
||||
const failure = contentFailure(contentTokens(page), htmlTokens);
|
||||
const auth = authFailure(page, finalUrl, { hadAuth });
|
||||
const failure = auth ?? contentFailure(contentTokens(page), htmlTokens);
|
||||
// Always present, so a caller can branch on the field rather than on
|
||||
// whether a field it was hoping for turned up.
|
||||
console.log(JSON.stringify({ ...page, empty: failure != null }));
|
||||
if (verbose) console.error(resources());
|
||||
if (auth) {
|
||||
if (jar) clearCookieJar(sessionName);
|
||||
noContent(finalUrl, auth);
|
||||
return;
|
||||
}
|
||||
remember(page, sessionName);
|
||||
if (failure) noContent(finalUrl, failure);
|
||||
return;
|
||||
}
|
||||
if (command === 'raw') {
|
||||
const page = distill(html, finalUrl);
|
||||
const auth = authFailure(page, finalUrl, { hadAuth });
|
||||
if (auth) {
|
||||
if (jar) clearCookieJar(sessionName);
|
||||
noContent(finalUrl, auth);
|
||||
return;
|
||||
}
|
||||
const out = values.html ? toHTML(html, finalUrl) : toMarkdown(html, finalUrl);
|
||||
const outTokens = estimateTokens(out);
|
||||
console.log(out);
|
||||
@@ -202,7 +310,13 @@ async function main() {
|
||||
return;
|
||||
}
|
||||
const page = distill(html, finalUrl);
|
||||
const failure = contentFailure(contentTokens(page), htmlTokens);
|
||||
const auth = authFailure(page, finalUrl, { hadAuth });
|
||||
const failure = auth ?? contentFailure(contentTokens(page), htmlTokens);
|
||||
if (auth) {
|
||||
if (jar) clearCookieJar(sessionName);
|
||||
noContent(finalUrl, auth);
|
||||
return;
|
||||
}
|
||||
const { text, stats } = render(page, { budget });
|
||||
remember(page, sessionName, stats.next);
|
||||
console.log(text);
|
||||
|
||||
+530
@@ -0,0 +1,530 @@
|
||||
/**
|
||||
* Per-session cookie jar, stored in a sidecar file next to the page-state
|
||||
* JSON. Credentials never live in the session snapshot itself.
|
||||
*/
|
||||
|
||||
import net from 'node:net';
|
||||
import { join } from 'node:path';
|
||||
import { mkdirSync, readFileSync, writeFileSync, unlinkSync, readdirSync, chmodSync } from 'node:fs';
|
||||
|
||||
import { sessionDir, assertSafeName } from './session.js';
|
||||
|
||||
const DEFAULT_EXPIRES_MS = 60 * 60 * 1000; // 1h
|
||||
export { DEFAULT_EXPIRES_MS };
|
||||
const FILE_MODE = 0o600;
|
||||
|
||||
// A jar is one login's worth of cookies, not a browser profile. The cap is
|
||||
// what stops a hostile page from growing the sidecar without bound through
|
||||
// Set-Cookie, and 4KB per cookie is the ceiling browsers already enforce.
|
||||
export const MAX_COOKIES = 50;
|
||||
export const MAX_COOKIE_BYTES = 4096;
|
||||
|
||||
// RFC 6265 cookie-name is an RFC 7230 token. Real cookie names are always one.
|
||||
const COOKIE_NAME = /^[!#$%&'*+\-.^_`|~0-9A-Za-z]+$/;
|
||||
// RFC 6265's cookie-value is stricter than this - no space, comma, quote, or
|
||||
// backslash - but real browser cookies carry all four, so a strict rule would
|
||||
// reject headers a user correctly copied out of devtools. "Printable ASCII, no
|
||||
// semicolon" keeps those and still rejects CR, LF, NUL, and every other
|
||||
// control character, which is all a header-injection attempt has to work with.
|
||||
const COOKIE_VALUE = /^[\x20-\x3A\x3C-\x7E]*$/;
|
||||
|
||||
// Hostnames only: no scheme, port, path, userinfo, or IPv6 literal.
|
||||
const HOSTNAME = /^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)*$/;
|
||||
|
||||
/** Returned by loadCookieJar when a sidecar existed but its session ceiling had passed. */
|
||||
export const JAR_EXPIRED = Object.freeze({ expired: true });
|
||||
|
||||
let purged = false;
|
||||
|
||||
/**
|
||||
* A user-supplied string as it can safely appear in an error message: control
|
||||
* characters escaped so a CR cannot rewrite the line, and clipped so a huge
|
||||
* value does not become the error.
|
||||
* @param {unknown} value
|
||||
* @returns {string}
|
||||
*/
|
||||
function clip(value) {
|
||||
const s = String(value).replace(/[\x00-\x1f\x7f]/g, (c) => `\\x${c.charCodeAt(0).toString(16).padStart(2, '0')}`);
|
||||
return s.length > 40 ? `${s.slice(0, 40)}...` : s;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} name
|
||||
* @returns {string}
|
||||
*/
|
||||
export function cookieJarPath(name) {
|
||||
return join(sessionDir(), `${assertSafeName(name)}.cookies.json`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse --expires values like 1h, 30m, 2d into milliseconds.
|
||||
* @param {string} value
|
||||
* @returns {number}
|
||||
*/
|
||||
export function parseExpires(value) {
|
||||
const m = String(value).trim().match(/^(\d+(?:\.\d+)?)(h|m|d|s)?$/i);
|
||||
if (!m) throw new Error(`invalid --expires '${value}', use a duration like 1h, 30m, or 2d`);
|
||||
const n = Number(m[1]);
|
||||
const unit = (m[2] ?? 'h').toLowerCase();
|
||||
const mult = unit === 'd' ? 86_400_000 : unit === 'h' ? 3_600_000 : unit === 'm' ? 60_000 : 1000;
|
||||
return n * mult;
|
||||
}
|
||||
|
||||
/**
|
||||
* The hostname a jar is scoped to.
|
||||
*
|
||||
* domainMatches is a suffix match, so the value here decides how far the
|
||||
* cookies reach: '--domain com' would hand them to every .com host the session
|
||||
* ever fetches. --domain is trusted input, but the caller is often an agent and
|
||||
* the failure mode is silent credential spray, so a bare name is refused. The
|
||||
* rule needs no Public Suffix List: at least one dot, unless the value is an IP
|
||||
* literal or localhost. It does not catch multi-label public suffixes
|
||||
* ('--domain co.uk' still passes); a PSL is the only thing that would, and it
|
||||
* is a dependency this project will not take.
|
||||
* @param {string} domain
|
||||
* @returns {string} the lowercased, dot-stripped hostname
|
||||
*/
|
||||
export function normalizeDomain(domain) {
|
||||
const host = String(domain ?? '').trim().toLowerCase().replace(/^\./, '').replace(/\.$/, '');
|
||||
if (!host || !HOSTNAME.test(host)) {
|
||||
throw new Error(`--domain must be a hostname like example.com (got '${clip(domain)}')`);
|
||||
}
|
||||
if (net.isIP(host) || host === 'localhost') return host;
|
||||
if (!host.includes('.')) {
|
||||
throw new Error(
|
||||
`--domain '${host}' is a bare name, so these cookies would be sent to every host under it; `
|
||||
+ 'use the full hostname they belong to, like example.com',
|
||||
);
|
||||
}
|
||||
return host;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} name
|
||||
* @param {string} value
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function isValidCookie(name, value) {
|
||||
return COOKIE_NAME.test(name)
|
||||
&& COOKIE_VALUE.test(value)
|
||||
&& Buffer.byteLength(name) + Buffer.byteLength(value) <= MAX_COOKIE_BYTES;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fail at `oc login` rather than deep inside a transport. A CR or LF in a
|
||||
* seeded cookie surfaces later as node's own header-validation error, which
|
||||
* says nothing about which cookie is wrong, and whatever curl does with it
|
||||
* through impers is a separate question this closes off for both transports.
|
||||
* @param {string} name
|
||||
* @param {string} value
|
||||
*/
|
||||
function assertValidCookie(name, value) {
|
||||
if (!COOKIE_NAME.test(name)) {
|
||||
throw new Error(`invalid cookie name '${clip(name)}', names are letters, digits, and !#$%&'*+-.^_\`|~`);
|
||||
}
|
||||
if (!COOKIE_VALUE.test(value)) {
|
||||
throw new Error(
|
||||
`invalid value for cookie '${clip(name)}', cookie values cannot hold control characters or non-ASCII bytes`,
|
||||
);
|
||||
}
|
||||
if (Buffer.byteLength(name) + Buffer.byteLength(value) > MAX_COOKIE_BYTES) {
|
||||
throw new Error(`cookie '${clip(name)}' is over the ${MAX_COOKIE_BYTES}-byte limit`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @typedef {Object} Cookie
|
||||
* @property {string} name
|
||||
* @property {string} value
|
||||
* @property {string} domain
|
||||
* @property {string} path
|
||||
* @property {boolean} [secure]
|
||||
* @property {boolean} [httpOnly]
|
||||
* @property {string} [expires] - ISO timestamp
|
||||
*/
|
||||
|
||||
/**
|
||||
* @typedef {Object} CookieJar
|
||||
* @property {string} expiresAt - ISO session ceiling
|
||||
* @property {Cookie[]} cookies
|
||||
*/
|
||||
|
||||
/**
|
||||
* Seed a jar from a Cookie request header string.
|
||||
*
|
||||
* Seeded cookies are marked secure unless the caller opts out: they were
|
||||
* almost certainly copied out of an https browser session, and cookieHeaderFor
|
||||
* withholds a secure cookie from a plain-http request, so the default is that
|
||||
* they never travel in cleartext - including on an https page that 302s to
|
||||
* http, where the user never typed the downgrade.
|
||||
* @param {string} header
|
||||
* @param {string} domain
|
||||
* @param {{ expiresMs?: number, allowHttp?: boolean }} [opts]
|
||||
* @returns {CookieJar}
|
||||
*/
|
||||
export function jarFromCookieHeader(header, domain, { expiresMs = DEFAULT_EXPIRES_MS, allowHttp = false } = {}) {
|
||||
const host = normalizeDomain(domain);
|
||||
/** @type {Cookie[]} */
|
||||
const cookies = [];
|
||||
for (const part of String(header ?? '').split(';')) {
|
||||
const trimmed = part.trim();
|
||||
if (!trimmed) continue;
|
||||
const eq = trimmed.indexOf('=');
|
||||
if (eq <= 0) continue;
|
||||
const name = trimmed.slice(0, eq).trim();
|
||||
const value = trimmed.slice(eq + 1).trim();
|
||||
if (!name) continue;
|
||||
assertValidCookie(name, value);
|
||||
cookies.push({ name, value, domain: host, path: '/', ...(allowHttp ? {} : { secure: true }) });
|
||||
}
|
||||
if (!cookies.length) throw new Error('no cookies found in --cookie string');
|
||||
if (cookies.length > MAX_COOKIES) {
|
||||
throw new Error(`--cookie holds ${cookies.length} cookies, more than the ${MAX_COOKIES} a session keeps`);
|
||||
}
|
||||
return {
|
||||
expiresAt: new Date(Date.now() + expiresMs).toISOString(),
|
||||
cookies,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {CookieJar} jar
|
||||
* @returns {boolean}
|
||||
*/
|
||||
export function isSessionExpired(jar) {
|
||||
return Date.now() >= Date.parse(jar.expiresAt);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {Cookie} cookie
|
||||
* @param {string} sessionCeiling
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function isCookieExpired(cookie, sessionCeiling) {
|
||||
const ceiling = Date.parse(sessionCeiling);
|
||||
if (Date.now() >= ceiling) return true;
|
||||
if (!cookie.expires) return false;
|
||||
const exp = Date.parse(cookie.expires);
|
||||
if (Number.isNaN(exp)) return false;
|
||||
return exp <= Date.now() || exp > ceiling;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {CookieJar} jar
|
||||
* @returns {CookieJar}
|
||||
*/
|
||||
function pruneExpiredCookies(jar) {
|
||||
return {
|
||||
...jar,
|
||||
cookies: jar.cookies.filter((c) => !isCookieExpired(c, jar.expiresAt)),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Read this session's jar before the process-wide purge can erase the evidence
|
||||
* of expiry; then sweep other stale sidecars.
|
||||
* @param {string} name
|
||||
* @returns {CookieJar | typeof JAR_EXPIRED | null}
|
||||
*/
|
||||
export function loadCookieJar(name) {
|
||||
let result = null;
|
||||
try {
|
||||
const jar = /** @type {CookieJar} */ (JSON.parse(readFileSync(cookieJarPath(name), 'utf8')));
|
||||
if (!jar?.expiresAt || !Array.isArray(jar.cookies)) {
|
||||
result = null;
|
||||
} else if (isSessionExpired(jar)) {
|
||||
clearCookieJar(name);
|
||||
result = JAR_EXPIRED;
|
||||
} else {
|
||||
const pruned = pruneExpiredCookies(jar);
|
||||
if (!pruned.cookies.length) {
|
||||
clearCookieJar(name);
|
||||
result = JAR_EXPIRED;
|
||||
} else {
|
||||
result = pruned;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
result = null;
|
||||
}
|
||||
ensurePurged();
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} name
|
||||
* @param {CookieJar} jar
|
||||
*/
|
||||
export function saveCookieJar(name, jar) {
|
||||
mkdirSync(sessionDir(), { recursive: true });
|
||||
// writeFileSync only sets the mode on create, so an existing sidecar has its
|
||||
// owner-only mode reasserted on every save.
|
||||
const path = cookieJarPath(name);
|
||||
writeFileSync(path, JSON.stringify(pruneExpiredCookies(jar)), { mode: FILE_MODE });
|
||||
chmodSync(path, FILE_MODE);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} name
|
||||
*/
|
||||
export function clearCookieJar(name) {
|
||||
try {
|
||||
unlinkSync(cookieJarPath(name));
|
||||
} catch {
|
||||
// missing file is fine
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete expired sidecar jars under OC_HOME/sessions.
|
||||
*/
|
||||
export function purgeExpiredJars() {
|
||||
let dir;
|
||||
try {
|
||||
dir = sessionDir();
|
||||
readdirSync(dir);
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
for (const file of readdirSync(dir)) {
|
||||
if (!file.endsWith('.cookies.json')) continue;
|
||||
const name = file.slice(0, -'.cookies.json'.length);
|
||||
try {
|
||||
const jar = /** @type {CookieJar} */ (JSON.parse(readFileSync(join(dir, file), 'utf8')));
|
||||
if (isSessionExpired(jar)) clearCookieJar(name);
|
||||
} catch {
|
||||
try { unlinkSync(join(dir, file)); } catch {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function ensurePurged() {
|
||||
if (purged) return;
|
||||
purged = true;
|
||||
purgeExpiredJars();
|
||||
}
|
||||
|
||||
/** Reset purge-once guard (tests). */
|
||||
export function _resetPurgeGuard() {
|
||||
purged = false;
|
||||
}
|
||||
|
||||
/**
|
||||
* RFC 6265 domain matching (host-only and domain cookies).
|
||||
* @param {Cookie} cookie
|
||||
* @param {string} host
|
||||
*/
|
||||
function domainMatches(cookie, host) {
|
||||
const d = cookie.domain.toLowerCase().replace(/^\./, '');
|
||||
return host === d || host.endsWith(`.${d}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {Cookie} cookie
|
||||
* @param {string} path
|
||||
*/
|
||||
function pathMatches(cookie, path) {
|
||||
const p = cookie.path || '/';
|
||||
if (path === p) return true;
|
||||
if (!path.startsWith(p)) return false;
|
||||
return p.endsWith('/') || path[p.length] === '/';
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {Cookie} cookie
|
||||
* @param {CookieJar} jar
|
||||
* @param {string} host
|
||||
* @param {string} path
|
||||
*/
|
||||
function scopeMatches(cookie, jar, host, path) {
|
||||
return !isCookieExpired(cookie, jar.expiresAt) && domainMatches(cookie, host) && pathMatches(cookie, path);
|
||||
}
|
||||
|
||||
/**
|
||||
* Cookies to send for a request URL.
|
||||
* @param {CookieJar} jar
|
||||
* @param {string} urlStr
|
||||
* @returns {string | undefined}
|
||||
*/
|
||||
export function cookieHeaderFor(jar, urlStr) {
|
||||
let url;
|
||||
try {
|
||||
url = new URL(urlStr);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
const host = url.hostname.toLowerCase();
|
||||
const path = url.pathname || '/';
|
||||
const secure = url.protocol === 'https:';
|
||||
const active = jar.cookies.filter((c) => {
|
||||
if (c.secure && !secure) return false;
|
||||
return scopeMatches(c, jar, host, path);
|
||||
});
|
||||
if (!active.length) return undefined;
|
||||
return active.map((c) => `${c.name}=${c.value}`).join('; ');
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether this URL would have received cookies but for its scheme. The CLI
|
||||
* uses it to name --allow-http, rather than fetching without credentials and
|
||||
* leaving the caller to wonder why an authenticated page came back a login
|
||||
* form.
|
||||
* @param {CookieJar} jar
|
||||
* @param {string} urlStr
|
||||
* @returns {boolean}
|
||||
*/
|
||||
export function withheldForScheme(jar, urlStr) {
|
||||
let url;
|
||||
try {
|
||||
url = new URL(urlStr);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
if (url.protocol !== 'http:') return false;
|
||||
const host = url.hostname.toLowerCase();
|
||||
const path = url.pathname || '/';
|
||||
return jar.cookies.some((c) => c.secure && scopeMatches(c, jar, host, path));
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse one Set-Cookie header value.
|
||||
* @param {string} header
|
||||
* @param {string} requestUrl
|
||||
* @returns {Cookie | null}
|
||||
*/
|
||||
export function parseSetCookie(header, requestUrl) {
|
||||
const parts = header.split(';').map((p) => p.trim()).filter(Boolean);
|
||||
if (!parts.length) return null;
|
||||
const eq = parts[0].indexOf('=');
|
||||
if (eq <= 0) return null;
|
||||
const name = parts[0].slice(0, eq).trim();
|
||||
const value = parts[0].slice(eq + 1).trim();
|
||||
if (!name) return null;
|
||||
// A response is untrusted input, and whatever it sets here is echoed back in
|
||||
// the Cookie header of the next request, so it faces the same rule a seeded
|
||||
// cookie does. Dropped silently: a page setting a junk cookie is the page's
|
||||
// problem, not a reason to fail the render.
|
||||
if (!isValidCookie(name, value)) return null;
|
||||
|
||||
const url = new URL(requestUrl);
|
||||
/** @type {Cookie} */
|
||||
const cookie = {
|
||||
name,
|
||||
value,
|
||||
domain: url.hostname.toLowerCase(),
|
||||
path: '/',
|
||||
// A cookie learned over https is pinned secure whether or not the response
|
||||
// said so, so a later hop to http - a redirect, or a link the agent
|
||||
// follows - cannot carry it in cleartext.
|
||||
...(url.protocol === 'https:' ? { secure: true } : {}),
|
||||
};
|
||||
|
||||
for (const attr of parts.slice(1)) {
|
||||
const sep = attr.indexOf('=');
|
||||
const key = (sep === -1 ? attr : attr.slice(0, sep)).trim().toLowerCase();
|
||||
const val = sep === -1 ? '' : attr.slice(sep + 1).trim();
|
||||
// The Domain attribute is deliberately ignored: cookies learned from a
|
||||
// response are pinned host-only to the host that set them. Honoring Domain
|
||||
// safely needs the Public Suffix List (a site could otherwise scope a
|
||||
// cookie to '.com' and have it sent to every site under it), and a PSL is a
|
||||
// dependency this project will not take. User-seeded cookies still scope by
|
||||
// the --domain they pass, which normalizeDomain holds to the same floor.
|
||||
if (key === 'path') {
|
||||
cookie.path = val || '/';
|
||||
} else if (key === 'secure') {
|
||||
cookie.secure = true;
|
||||
} else if (key === 'httponly') {
|
||||
cookie.httpOnly = true;
|
||||
} else if (key === 'max-age') {
|
||||
const age = Number(val);
|
||||
if (Number.isFinite(age)) {
|
||||
cookie.expires = new Date(Date.now() + age * 1000).toISOString();
|
||||
}
|
||||
} else if (key === 'expires') {
|
||||
const exp = Date.parse(val);
|
||||
if (!Number.isNaN(exp)) cookie.expires = new Date(exp).toISOString();
|
||||
}
|
||||
}
|
||||
return cookie;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract Set-Cookie header values from a fetch/impers response.
|
||||
* @param {any} res
|
||||
* @returns {string[]}
|
||||
*/
|
||||
export function getSetCookieHeaders(res) {
|
||||
if (typeof res.headers?.getSetCookie === 'function') {
|
||||
const list = res.headers.getSetCookie();
|
||||
if (Array.isArray(list) && list.length) return list;
|
||||
}
|
||||
const raw = res.headers?.get?.('set-cookie');
|
||||
if (!raw) return [];
|
||||
// Undici joins with ", " but Expires contains commas: split only on ", " followed by a token=
|
||||
return raw.split(/,\s(?=[\w!#$%&'*+\-.^`|~]+=)/);
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply Set-Cookie headers to the jar for a response URL.
|
||||
* @param {CookieJar} jar
|
||||
* @param {string} url
|
||||
* @param {string[]} setCookieHeaders
|
||||
* @returns {CookieJar}
|
||||
*/
|
||||
export function storeFromResponse(jar, url, setCookieHeaders) {
|
||||
if (!setCookieHeaders.length) return jar;
|
||||
let cookies = [...jar.cookies];
|
||||
for (const header of setCookieHeaders) {
|
||||
const parsed = parseSetCookie(header, url);
|
||||
if (!parsed) continue;
|
||||
// Max-Age=0 or Expires in the past deletes the cookie
|
||||
if (parsed.expires && Date.parse(parsed.expires) <= Date.now()) {
|
||||
cookies = cookies.filter((c) => !(c.name === parsed.name && domainMatches(c, parsed.domain)));
|
||||
continue;
|
||||
}
|
||||
cookies = cookies.filter((c) => !(c.name === parsed.name && domainMatches(c, parsed.domain)));
|
||||
// Replacing a cookie the jar already holds is always allowed; growing past
|
||||
// the cap is not, so a page cannot bloat the sidecar with fresh names. The
|
||||
// cookies already there - the seeded login among them - are what survive.
|
||||
if (cookies.length >= MAX_COOKIES) continue;
|
||||
if (parsed.expires) {
|
||||
const ceiling = Date.parse(jar.expiresAt);
|
||||
const exp = Date.parse(parsed.expires);
|
||||
if (exp > ceiling) parsed.expires = jar.expiresAt;
|
||||
}
|
||||
cookies.push(parsed);
|
||||
}
|
||||
return { ...jar, cookies };
|
||||
}
|
||||
|
||||
/**
|
||||
* Mutable jar wrapper for fetch to update in place.
|
||||
* @param {string} sessionName
|
||||
* @param {CookieJar} data
|
||||
*/
|
||||
export function createJarHandle(sessionName, data) {
|
||||
let jar = data;
|
||||
return {
|
||||
sessionName,
|
||||
cookieHeaderFor(url) {
|
||||
return cookieHeaderFor(jar, url);
|
||||
},
|
||||
storeFromResponse(url, headers) {
|
||||
jar = storeFromResponse(jar, url, headers);
|
||||
},
|
||||
toJSON() {
|
||||
return jar;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} name
|
||||
* @param {string} header
|
||||
* @param {string} domain
|
||||
* @param {{ expiresMs?: number, allowHttp?: boolean }} [opts]
|
||||
*/
|
||||
export function loginCookieJar(name, header, domain, opts) {
|
||||
const jar = jarFromCookieHeader(header, domain, opts);
|
||||
saveCookieJar(name, jar);
|
||||
}
|
||||
+51
-13
@@ -13,6 +13,8 @@ import https from 'node:https';
|
||||
import net from 'node:net';
|
||||
import tls from 'node:tls';
|
||||
|
||||
import { getSetCookieHeaders } from './cookies.js';
|
||||
|
||||
// The fetch fallback can't fake a TLS fingerprint like impers does, but it
|
||||
// should at least send the same Chrome identity in its headers.
|
||||
const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36';
|
||||
@@ -330,6 +332,14 @@ function wrapNodeResponse(res, url) {
|
||||
if (v == null) return null;
|
||||
return Array.isArray(v) ? v.join(', ') : v;
|
||||
},
|
||||
// Node keeps Set-Cookie as an array of raw values. Expose it unjoined so
|
||||
// the cookie jar reads each header intact: a comma in an Expires date
|
||||
// makes the joined form ambiguous to split back apart.
|
||||
getSetCookie() {
|
||||
const v = res.headers['set-cookie'];
|
||||
if (v == null) return [];
|
||||
return Array.isArray(v) ? v : [v];
|
||||
},
|
||||
};
|
||||
const text = () => new Promise((resolve, reject) => {
|
||||
const chunks = [];
|
||||
@@ -430,7 +440,9 @@ function httpsViaConnect(target, proxy, headers, tlsOpts = {}) {
|
||||
// tls.connect already opened the tunnel. https.request would wrap TLS
|
||||
// again, and the origin would see a second ClientHello as garbage.
|
||||
// SNI is a hostname; an IP literal is only used for the cert check.
|
||||
const hostname = target.hostname;
|
||||
// URL.hostname keeps the brackets around an IPv6 literal ("[::1]"), which
|
||||
// tls.connect would treat as a DNS name; strip them like assertSafeTarget.
|
||||
const hostname = target.hostname.replace(/^\[|\]$/g, '');
|
||||
const tlsSocket = tls.connect({
|
||||
socket,
|
||||
host: hostname,
|
||||
@@ -479,15 +491,16 @@ export function proxyGet(url, proxy, headers = {}, tlsOpts = {}) {
|
||||
/**
|
||||
* Fetch a page.
|
||||
* @param {string} url - with or without a scheme, https is assumed
|
||||
* @param {{ jar?: { cookieHeaderFor(url: string): string|undefined, storeFromResponse(url: string, headers: string[]): void } }} [opts]
|
||||
* @returns {Promise<{url: string, html: string, status: number, via: string}>}
|
||||
* final URL after redirects, the body, the HTTP status, and which client
|
||||
* identity got the page (impers:chrome, impers:firefox, or fetch)
|
||||
*/
|
||||
export async function fetchPage(url) {
|
||||
export async function fetchPage(url, { jar } = {}) {
|
||||
const target = /^https?:\/\//i.test(url) ? url : `https://${url}`;
|
||||
await assertSafeTarget(target);
|
||||
const impers = await loadImpers();
|
||||
return impers ? viaImpers(impers, target) : viaFetch(target);
|
||||
return impers ? viaImpers(impers, target, jar) : viaFetch(target, jar);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -504,11 +517,12 @@ export async function fetchPage(url) {
|
||||
* @param {string} start
|
||||
* @returns {Promise<{res: any, url: string}>} the first non-redirect response
|
||||
*/
|
||||
export async function followRedirects(get, start) {
|
||||
export async function followRedirects(get, start, { onResponse } = {}) {
|
||||
let current = start;
|
||||
for (let i = 0; ; i++) {
|
||||
if (i > MAX_REDIRECTS) throw new Error(`too many redirects for ${start}`);
|
||||
const res = await get(current);
|
||||
onResponse?.(current, res);
|
||||
const status = res.status ?? res.statusCode ?? 0;
|
||||
const location = res.headers.get('location');
|
||||
if (status >= 300 && status < 400 && location) {
|
||||
@@ -526,17 +540,38 @@ const FETCH_HEADERS = {
|
||||
'accept-language': 'en-US,en;q=0.9',
|
||||
};
|
||||
|
||||
async function viaImpers(impers, target) {
|
||||
function mergeHeaders(base, extra) {
|
||||
return extra ? { ...base, ...extra } : base;
|
||||
}
|
||||
|
||||
function jarHeaders(jar, url, base) {
|
||||
if (!jar) return base;
|
||||
const cookie = jar.cookieHeaderFor(url);
|
||||
return cookie ? mergeHeaders(base, { cookie }) : base;
|
||||
}
|
||||
|
||||
function captureSetCookie(jar, url, res) {
|
||||
if (!jar) return;
|
||||
jar.storeFromResponse(url, getSetCookieHeaders(res));
|
||||
}
|
||||
|
||||
async function viaImpers(impers, target, jar) {
|
||||
// Some sites (Reddit) 403 the chrome fingerprint but accept firefox, so a
|
||||
// blocked first attempt gets one cheap retry with a second identity.
|
||||
const asking = (impersonate) => (url) =>
|
||||
impers.get(url, { impersonate, allowRedirects: false, proxy: resolveProxy(url) ?? '' });
|
||||
impers.get(url, {
|
||||
impersonate,
|
||||
allowRedirects: false,
|
||||
proxy: resolveProxy(url) ?? '',
|
||||
headers: jarHeaders(jar, url, {}),
|
||||
});
|
||||
const onResponse = (url, res) => captureSetCookie(jar, url, res);
|
||||
let via = 'impers:chrome';
|
||||
let { res } = await followRedirects(asking('chrome'), target);
|
||||
let { res } = await followRedirects(asking('chrome'), target, { onResponse });
|
||||
let status = res.status ?? res.statusCode ?? 0;
|
||||
if (status >= 400) {
|
||||
via = 'impers:firefox';
|
||||
({ res } = await followRedirects(asking('firefox'), target));
|
||||
({ res } = await followRedirects(asking('firefox'), target, { onResponse }));
|
||||
status = res.status ?? res.statusCode ?? 0;
|
||||
}
|
||||
if (status >= 400) throw new Error(`fetch failed: ${status} for ${target}`);
|
||||
@@ -550,13 +585,16 @@ async function viaImpers(impers, target) {
|
||||
return { url: res.url ?? target, html, status, via };
|
||||
}
|
||||
|
||||
async function viaFetch(target) {
|
||||
const { res, url: current } = await followRedirects((url) => {
|
||||
async function viaFetch(target, jar) {
|
||||
const get = (url) => {
|
||||
const proxy = resolveProxy(url);
|
||||
const headers = jarHeaders(jar, url, FETCH_HEADERS);
|
||||
return proxy
|
||||
? proxyGet(url, proxy, FETCH_HEADERS)
|
||||
: fetch(url, { redirect: 'manual', headers: FETCH_HEADERS });
|
||||
}, target);
|
||||
? proxyGet(url, proxy, headers)
|
||||
: fetch(url, { redirect: 'manual', headers });
|
||||
};
|
||||
const onResponse = (url, res) => captureSetCookie(jar, url, res);
|
||||
const { res, url: current } = await followRedirects(get, target, { onResponse });
|
||||
if (!res.ok) {
|
||||
throw new Error(`fetch failed: ${res.status} ${res.statusText} for ${current}`);
|
||||
}
|
||||
|
||||
+43
-4
@@ -1,7 +1,8 @@
|
||||
/**
|
||||
* Sessions are plain JSON files on disk, one per name: the current URL, the
|
||||
* distilled blocks of the page it holds, how far the last render got through
|
||||
* them, and a short history. No daemon, no background process, no cookies yet.
|
||||
* them, and a short history. No daemon, no background process; cookies live in
|
||||
* a separate sidecar file (see cookies.js).
|
||||
*
|
||||
* The file exists so `oc do <n>` can follow a link the compact view never
|
||||
* printed the URL of. Hiding URLs is what makes `oc open` cheap; this is what
|
||||
@@ -11,18 +12,35 @@
|
||||
|
||||
import { homedir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { mkdirSync, readFileSync, writeFileSync, chmodSync, unlinkSync } from 'node:fs';
|
||||
|
||||
export const DEFAULT_SESSION = 'default';
|
||||
|
||||
// OC_HOME relocates the whole state directory, for sandboxes, CI, and tests.
|
||||
export const sessionDir = () => join(process.env.OC_HOME ?? join(homedir(), '.only-cli'), 'sessions');
|
||||
|
||||
// A session name is interpolated straight into a filename, and the cookie
|
||||
// sidecar it names now holds real credentials, so a name that is a path
|
||||
// (absolute, or with a separator or '..') could write or delete a file outside
|
||||
// the store. Names are user-facing labels, so this charset loses nothing real.
|
||||
const SAFE_NAME = /^[A-Za-z0-9._-]+$/;
|
||||
|
||||
/**
|
||||
* @param {string} name
|
||||
* @returns {string} the same name, once it is known to be a safe filename
|
||||
*/
|
||||
export function assertSafeName(name) {
|
||||
if (typeof name !== 'string' || name === '.' || name === '..' || !SAFE_NAME.test(name)) {
|
||||
throw new Error(`invalid session name '${name}', use letters, numbers, '.', '-', or '_'`);
|
||||
}
|
||||
return name;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} name
|
||||
* @returns {string}
|
||||
*/
|
||||
export const sessionPath = (name) => join(sessionDir(), `${name}.json`);
|
||||
export const sessionPath = (name) => join(sessionDir(), `${assertSafeName(name)}.json`);
|
||||
|
||||
// Search engines and link aggregators wrap outbound links in a tracking
|
||||
// redirector whose landing page is a script, not content, so following one
|
||||
@@ -136,7 +154,28 @@ export function handleNumbers(state) {
|
||||
*/
|
||||
export function saveSession(name, state) {
|
||||
mkdirSync(sessionDir(), { recursive: true });
|
||||
writeFileSync(sessionPath(name), JSON.stringify(state));
|
||||
// A snapshot of an authenticated page holds that page's text, so it gets the
|
||||
// same owner-only mode as the cookie sidecar. writeFileSync only sets the
|
||||
// mode on create, so a snapshot left world-readable by an older version is
|
||||
// tightened explicitly on the next save.
|
||||
const path = sessionPath(name);
|
||||
writeFileSync(path, JSON.stringify(state), { mode: 0o600 });
|
||||
chmodSync(path, 0o600);
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop a saved page. `oc logout` calls this alongside clearing the cookie jar:
|
||||
* a snapshot taken under a login holds that page's text, so leaving it behind
|
||||
* would make logout mean "the cookies are gone" rather than "nothing of this
|
||||
* login remains".
|
||||
* @param {string} name
|
||||
*/
|
||||
export function clearSession(name) {
|
||||
try {
|
||||
unlinkSync(sessionPath(name));
|
||||
} catch {
|
||||
// nothing saved under that name is fine
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import http from 'node:http';
|
||||
import { readFileSync } from 'node:fs';
|
||||
|
||||
const { distill, toMarkdown } = await import('../src/distill.js');
|
||||
const { authFailure } = await import('../src/auth.js');
|
||||
const { fetchPage } = await import('../src/fetch.js');
|
||||
|
||||
const loginHtml = readFileSync(new URL('./pages/login.html', import.meta.url), 'utf8');
|
||||
|
||||
const navWithLoginLink = `<html><head><title>News</title></head><body>
|
||||
<nav><a href="/login">Log in</a></nav>
|
||||
<article>${'<p>Real story content here.</p>'.repeat(20)}</article>
|
||||
</body></html>`;
|
||||
|
||||
const PROXY_ENV_KEYS = ['HTTP_PROXY', 'HTTPS_PROXY', 'NO_PROXY', 'http_proxy', 'https_proxy', 'no_proxy'];
|
||||
|
||||
function listen(server) {
|
||||
return new Promise((resolve) => {
|
||||
server.listen(0, '127.0.0.1', () => resolve(server.address().port));
|
||||
});
|
||||
}
|
||||
|
||||
function withoutProxyEnv(run) {
|
||||
const prev = Object.fromEntries(PROXY_ENV_KEYS.map((k) => [k, process.env[k]]));
|
||||
for (const k of PROXY_ENV_KEYS) delete process.env[k];
|
||||
return run().finally(() => {
|
||||
for (const k of PROXY_ENV_KEYS) {
|
||||
if (prev[k] === undefined) delete process.env[k];
|
||||
else process.env[k] = prev[k];
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
test('authFailure detects a login page with password input and supporting signals', () => {
|
||||
const page = distill(loginHtml, 'https://example.com/login');
|
||||
assert.match(authFailure(page, 'https://example.com/login'), /requires login/);
|
||||
});
|
||||
|
||||
test('authFailure reports expired session when auth was sent', () => {
|
||||
const page = distill(loginHtml, 'https://example.com/login');
|
||||
assert.match(
|
||||
authFailure(page, 'https://example.com/login', { hadAuth: true }),
|
||||
/session expired or cookies are no longer valid/,
|
||||
);
|
||||
});
|
||||
|
||||
test('authFailure ignores nav login links without a password field', () => {
|
||||
const page = distill(navWithLoginLink, 'https://example.com/news');
|
||||
assert.equal(authFailure(page, 'https://example.com/news'), null);
|
||||
});
|
||||
|
||||
test('authFailure ignores a password field without login context', () => {
|
||||
const html = `<html><head><title>Account settings</title></head><body>
|
||||
<p>Change your password below.</p>
|
||||
<input type="password" name="new">
|
||||
<button>Save</button>
|
||||
</body></html>`;
|
||||
const page = distill(html, 'https://example.com/settings');
|
||||
assert.equal(authFailure(page, 'https://example.com/settings'), null);
|
||||
});
|
||||
|
||||
test('fetch through a proxy detects a login page end to end (offline)', async () => {
|
||||
await withoutProxyEnv(async () => {
|
||||
const proxy = http.createServer((req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end(loginHtml);
|
||||
});
|
||||
const port = await listen(proxy);
|
||||
process.env.HTTP_PROXY = `http://127.0.0.1:${port}`;
|
||||
try {
|
||||
const { html, url } = await fetchPage('http://1.1.1.1/login');
|
||||
const page = distill(html, url);
|
||||
assert.match(authFailure(page, url), /requires login/);
|
||||
assert.match(authFailure(page, url, { hadAuth: true }), /session expired or cookies are no longer valid/);
|
||||
} finally {
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
test('auth failure gates raw output before markdown is emitted', async () => {
|
||||
await withoutProxyEnv(async () => {
|
||||
const proxy = http.createServer((req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end(loginHtml);
|
||||
});
|
||||
const port = await listen(proxy);
|
||||
process.env.HTTP_PROXY = `http://127.0.0.1:${port}`;
|
||||
try {
|
||||
const { html, url } = await fetchPage('http://1.1.1.1/login');
|
||||
const page = distill(html, url);
|
||||
assert.ok(authFailure(page, url));
|
||||
assert.match(toMarkdown(html, url), /Sign in/);
|
||||
} finally {
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,277 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import http from 'node:http';
|
||||
import { mkdtempSync, readFileSync, writeFileSync, mkdirSync, existsSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
|
||||
const OC_HOME = mkdtempSync(join(tmpdir(), 'oc-cli-auth-'));
|
||||
process.env.OC_HOME = OC_HOME;
|
||||
|
||||
const bin = new URL('../src/cli.js', import.meta.url).pathname;
|
||||
const loginHtml = readFileSync(new URL('./pages/login.html', import.meta.url), 'utf8');
|
||||
const dashHtml = `<html><head><title>Dashboard</title></head><body>
|
||||
<h1>Welcome back</h1>
|
||||
${'<p>Secret project notes for the signed-in user.</p>'.repeat(20)}
|
||||
</body></html>`;
|
||||
|
||||
const PROXY_ENV_KEYS = ['HTTP_PROXY', 'HTTPS_PROXY', 'NO_PROXY', 'http_proxy', 'https_proxy', 'no_proxy'];
|
||||
|
||||
function childEnv(envExtra = {}) {
|
||||
const env = { ...process.env, OC_HOME, ...envExtra };
|
||||
for (const k of PROXY_ENV_KEYS) {
|
||||
if (!(k in envExtra)) delete env[k];
|
||||
}
|
||||
return env;
|
||||
}
|
||||
|
||||
// Sync run for cases that never touch the network (login, logout, expired jar).
|
||||
function oc(args, envExtra = {}) {
|
||||
return spawnSync(process.execPath, [bin, ...args], { encoding: 'utf8', env: childEnv(envExtra) });
|
||||
}
|
||||
|
||||
// Async run for cases that fetch through an in-process mock proxy: spawnSync
|
||||
// would block the event loop the proxy server runs on and deadlock the test.
|
||||
function ocAsync(args, envExtra = {}) {
|
||||
return new Promise((resolve) => {
|
||||
const child = spawn(process.execPath, [bin, ...args], { env: childEnv(envExtra) });
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
child.stdout.on('data', (d) => { stdout += d; });
|
||||
child.stderr.on('data', (d) => { stderr += d; });
|
||||
child.on('close', (status) => resolve({ status, stdout, stderr }));
|
||||
});
|
||||
}
|
||||
|
||||
function listen(server) {
|
||||
return new Promise((resolve) => {
|
||||
server.listen(0, '127.0.0.1', () => resolve(server.address().port));
|
||||
});
|
||||
}
|
||||
|
||||
test('login saves a sidecar jar and logout removes it', () => {
|
||||
let r = oc(['login', '--cookie', 'sid=abc', '--domain', 'example.com', '--session', 'work']);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
const jarPath = join(OC_HOME, 'sessions', 'work.cookies.json');
|
||||
const saved = JSON.parse(readFileSync(jarPath, 'utf8'));
|
||||
assert.equal(saved.cookies[0].value, 'abc');
|
||||
|
||||
r = oc(['logout', 'work']);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.throws(() => readFileSync(jarPath), /ENOENT/);
|
||||
});
|
||||
|
||||
test('open with an expired jar reports session expired and clears it', () => {
|
||||
const sessionsDir = join(OC_HOME, 'sessions');
|
||||
mkdirSync(sessionsDir, { recursive: true });
|
||||
const jarPath = join(sessionsDir, 'expired.cookies.json');
|
||||
writeFileSync(jarPath, JSON.stringify({
|
||||
expiresAt: new Date(Date.now() - 1000).toISOString(),
|
||||
cookies: [{ name: 'sid', value: 'old', domain: 'example.com', path: '/' }],
|
||||
}));
|
||||
const r = oc(['open', 'example.com', '--session', 'expired']);
|
||||
assert.equal(r.status, 2, r.stderr);
|
||||
assert.match(r.stderr, /session expired or cookies are no longer valid/);
|
||||
assert.equal(r.stdout.trim(), '');
|
||||
assert.throws(() => readFileSync(jarPath), /ENOENT/);
|
||||
});
|
||||
|
||||
test('login requires --domain', () => {
|
||||
const r = oc(['login', '--cookie', 'sid=abc']);
|
||||
assert.notEqual(r.status, 0);
|
||||
assert.match(r.stderr, /--domain is required/);
|
||||
});
|
||||
|
||||
test('a session name that is a path is refused before any file is written', () => {
|
||||
for (const bad of ['../../.ssh/id_rsa', '/tmp/leak', 'a/b']) {
|
||||
const r = oc(['login', '--cookie', 'sid=abc', '--domain', 'example.com', '--session', bad]);
|
||||
assert.notEqual(r.status, 0, `expected failure for ${bad}`);
|
||||
assert.match(r.stderr, /invalid session name/);
|
||||
}
|
||||
});
|
||||
|
||||
test('open sends the jar cookies and renders authenticated content', async () => {
|
||||
const proxy = http.createServer((req, res) => {
|
||||
const cookie = req.headers.cookie || '';
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end(cookie.includes('sid=secret') ? dashHtml : loginHtml);
|
||||
});
|
||||
const port = await listen(proxy);
|
||||
const proxyUrl = `http://127.0.0.1:${port}`;
|
||||
try {
|
||||
// --allow-http because this mock speaks plain http; without it the cookie
|
||||
// is withheld, which is the case the next test covers.
|
||||
let r = oc(['login', '--cookie', 'sid=secret', '--domain', '1.1.1.1', '--session', 'authed', '--allow-http']);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
|
||||
r = await ocAsync(['open', 'http://1.1.1.1/dashboard', '--session', 'authed'], { HTTP_PROXY: proxyUrl });
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.match(r.stdout, /Welcome back/);
|
||||
assert.doesNotMatch(r.stderr, /requires login|session expired/);
|
||||
} finally {
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('a seeded cookie is not sent over plain http unless the user asked for it', async () => {
|
||||
const proxy = http.createServer((req, res) => {
|
||||
const cookie = req.headers.cookie || '';
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end(cookie.includes('sid=secret') ? dashHtml : loginHtml);
|
||||
});
|
||||
const port = await listen(proxy);
|
||||
try {
|
||||
let r = oc(['login', '--cookie', 'sid=secret', '--domain', '1.1.1.1', '--session', 'httponly']);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
const saved = JSON.parse(readFileSync(join(OC_HOME, 'sessions', 'httponly.cookies.json'), 'utf8'));
|
||||
assert.equal(saved.cookies[0].secure, true);
|
||||
|
||||
r = await ocAsync(['open', 'http://1.1.1.1/dashboard', '--session', 'httponly'], {
|
||||
HTTP_PROXY: `http://127.0.0.1:${port}`,
|
||||
});
|
||||
// The page came back a login form because the credential stayed home, and
|
||||
// the warning names the flag that would have sent it.
|
||||
assert.match(r.stderr, /https-only cookies.*--allow-http/s);
|
||||
assert.doesNotMatch(r.stdout, /Welcome back/);
|
||||
} finally {
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('an https page that redirects to http does not carry the cookie down with it', async () => {
|
||||
const seen = [];
|
||||
const proxy = http.createServer((req, res) => {
|
||||
seen.push(req.headers.cookie || '');
|
||||
if (req.url.endsWith('/start')) {
|
||||
res.writeHead(302, { location: 'http://1.1.1.1/landed' });
|
||||
res.end();
|
||||
return;
|
||||
}
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end(dashHtml);
|
||||
});
|
||||
const port = await listen(proxy);
|
||||
try {
|
||||
// Seeded over http so the first hop is reachable through the mock proxy,
|
||||
// then pinned secure by hand: the jar is what an https login leaves behind.
|
||||
let r = oc(['login', '--cookie', 'sid=secret', '--domain', '1.1.1.1', '--session', 'hop', '--allow-http']);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
const jarPath = join(OC_HOME, 'sessions', 'hop.cookies.json');
|
||||
const jar = JSON.parse(readFileSync(jarPath, 'utf8'));
|
||||
jar.cookies[0].secure = true;
|
||||
writeFileSync(jarPath, JSON.stringify(jar));
|
||||
|
||||
r = await ocAsync(['open', 'http://1.1.1.1/start', '--session', 'hop'], {
|
||||
HTTP_PROXY: `http://127.0.0.1:${port}`,
|
||||
});
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.ok(seen.length >= 2, `expected a redirect hop, saw ${seen.length} requests`);
|
||||
for (const cookie of seen) assert.doesNotMatch(cookie, /sid=secret/);
|
||||
} finally {
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('--cookie - reads the header from stdin instead of argv', () => {
|
||||
const r = spawnSync(process.execPath, [bin, 'login', '--cookie', '-', '--domain', 'example.com', '--session', 'piped'], {
|
||||
encoding: 'utf8',
|
||||
env: childEnv(),
|
||||
input: 'Cookie: sid=from-stdin; auth=xyz\n',
|
||||
});
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
const saved = JSON.parse(readFileSync(join(OC_HOME, 'sessions', 'piped.cookies.json'), 'utf8'));
|
||||
assert.deepEqual(saved.cookies.map((c) => `${c.name}=${c.value}`), ['sid=from-stdin', 'auth=xyz']);
|
||||
});
|
||||
|
||||
test('--cookie - with nothing piped in says what to pipe', () => {
|
||||
const r = spawnSync(process.execPath, [bin, 'login', '--cookie', '-', '--domain', 'example.com'], {
|
||||
encoding: 'utf8',
|
||||
env: childEnv(),
|
||||
input: ' \n',
|
||||
});
|
||||
assert.notEqual(r.status, 0);
|
||||
assert.match(r.stderr, /nothing on stdin/);
|
||||
});
|
||||
|
||||
test('login refuses a bare TLD and a cookie carrying a control character', () => {
|
||||
let r = oc(['login', '--cookie', 'sid=abc', '--domain', 'com', '--session', 'tld']);
|
||||
assert.notEqual(r.status, 0);
|
||||
assert.match(r.stderr, /bare name/);
|
||||
assert.ok(!existsSync(join(OC_HOME, 'sessions', 'tld.cookies.json')));
|
||||
|
||||
r = oc(['login', '--cookie', 'sid=a\r\nX-Injected: 1', '--domain', 'example.com', '--session', 'crlf']);
|
||||
assert.notEqual(r.status, 0);
|
||||
assert.match(r.stderr, /invalid value for cookie 'sid'/);
|
||||
assert.ok(!existsSync(join(OC_HOME, 'sessions', 'crlf.cookies.json')));
|
||||
});
|
||||
|
||||
test('logout drops the saved page along with the cookies', () => {
|
||||
let r = oc(['login', '--cookie', 'sid=abc', '--domain', 'example.com', '--session', 'clean']);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
const jarPath = join(OC_HOME, 'sessions', 'clean.cookies.json');
|
||||
const pagePath = join(OC_HOME, 'sessions', 'clean.json');
|
||||
writeFileSync(pagePath, JSON.stringify({
|
||||
url: 'https://example.com/dashboard',
|
||||
title: 'Dashboard',
|
||||
savedAt: new Date().toISOString(),
|
||||
blocks: [{ type: 'text', text: 'Secret project notes for the signed-in user.' }],
|
||||
cursor: null,
|
||||
history: [],
|
||||
}));
|
||||
|
||||
r = oc(['logout', 'clean']);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.ok(!existsSync(jarPath));
|
||||
assert.ok(!existsSync(pagePath));
|
||||
});
|
||||
|
||||
test('open without cookies detects a login page and fails loud', async () => {
|
||||
const proxy = http.createServer((req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end(loginHtml);
|
||||
});
|
||||
const port = await listen(proxy);
|
||||
try {
|
||||
const r = await ocAsync(['open', 'http://1.1.1.1/login', '--session', 'anon'], {
|
||||
HTTP_PROXY: `http://127.0.0.1:${port}`,
|
||||
});
|
||||
assert.equal(r.status, 2, r.stderr);
|
||||
assert.match(r.stderr, /requires login/);
|
||||
assert.equal(r.stdout.trim(), '');
|
||||
} finally {
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('json auth failure does not overwrite saved page state', async () => {
|
||||
const sessionsDir = join(OC_HOME, 'sessions');
|
||||
mkdirSync(sessionsDir, { recursive: true });
|
||||
const sessionPath = join(sessionsDir, 'keep.json');
|
||||
writeFileSync(sessionPath, JSON.stringify({
|
||||
url: 'http://1.1.1.1/dashboard',
|
||||
title: 'Dashboard',
|
||||
savedAt: new Date().toISOString(),
|
||||
blocks: [{ type: 'heading', text: 'Welcome back', n: 1, level: 1 }],
|
||||
cursor: null,
|
||||
history: ['http://1.1.1.1/dashboard'],
|
||||
}));
|
||||
|
||||
const proxy = http.createServer((req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end(loginHtml);
|
||||
});
|
||||
const port = await listen(proxy);
|
||||
try {
|
||||
const r = await ocAsync(['open', 'http://1.1.1.1/login', '--json', '--session', 'keep'], {
|
||||
HTTP_PROXY: `http://127.0.0.1:${port}`,
|
||||
});
|
||||
assert.equal(r.status, 2, r.stderr);
|
||||
assert.match(r.stderr, /requires login/);
|
||||
const saved = JSON.parse(readFileSync(sessionPath, 'utf8'));
|
||||
assert.equal(saved.title, 'Dashboard');
|
||||
assert.ok(existsSync(sessionPath));
|
||||
} finally {
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,220 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtempSync, readFileSync, writeFileSync, statSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
process.env.OC_HOME = mkdtempSync(join(tmpdir(), 'oc-cookie-test-'));
|
||||
|
||||
const {
|
||||
jarFromCookieHeader,
|
||||
parseExpires,
|
||||
cookieHeaderFor,
|
||||
parseSetCookie,
|
||||
storeFromResponse,
|
||||
saveCookieJar,
|
||||
loadCookieJar,
|
||||
clearCookieJar,
|
||||
purgeExpiredJars,
|
||||
isSessionExpired,
|
||||
cookieJarPath,
|
||||
normalizeDomain,
|
||||
withheldForScheme,
|
||||
MAX_COOKIES,
|
||||
MAX_COOKIE_BYTES,
|
||||
JAR_EXPIRED,
|
||||
_resetPurgeGuard,
|
||||
} = await import('../src/cookies.js');
|
||||
|
||||
test('parseExpires accepts common durations', () => {
|
||||
assert.equal(parseExpires('1h'), 3_600_000);
|
||||
assert.equal(parseExpires('30m'), 1_800_000);
|
||||
assert.equal(parseExpires('2d'), 172_800_000);
|
||||
});
|
||||
|
||||
test('jarFromCookieHeader parses a Cookie header for a domain', () => {
|
||||
const jar = jarFromCookieHeader('session=abc; auth=xyz', 'Example.COM');
|
||||
assert.equal(jar.cookies.length, 2);
|
||||
assert.equal(jar.cookies[0].name, 'session');
|
||||
assert.equal(jar.cookies[0].value, 'abc');
|
||||
assert.equal(jar.cookies[0].domain, 'example.com');
|
||||
assert.ok(Date.parse(jar.expiresAt) > Date.now());
|
||||
});
|
||||
|
||||
test('a seeded cookie is https-only by default and travels over http only on request', () => {
|
||||
const secure = jarFromCookieHeader('session=abc', 'example.com');
|
||||
assert.equal(secure.cookies[0].secure, true);
|
||||
// The credential came out of an https browser session, so plain http never
|
||||
// sees it unless the user says the site is http-only.
|
||||
assert.equal(cookieHeaderFor(secure, 'http://example.com/'), undefined);
|
||||
assert.equal(cookieHeaderFor(secure, 'https://example.com/'), 'session=abc');
|
||||
assert.ok(withheldForScheme(secure, 'http://example.com/'));
|
||||
assert.ok(!withheldForScheme(secure, 'https://example.com/'));
|
||||
|
||||
const opted = jarFromCookieHeader('session=abc', 'example.com', { allowHttp: true });
|
||||
assert.equal(opted.cookies[0].secure, undefined);
|
||||
assert.equal(cookieHeaderFor(opted, 'http://example.com/'), 'session=abc');
|
||||
assert.ok(!withheldForScheme(opted, 'http://example.com/'));
|
||||
});
|
||||
|
||||
test('a cookie learned over https is pinned secure even without the attribute', () => {
|
||||
const jar = { expiresAt: new Date(Date.now() + 3_600_000).toISOString(), cookies: [] };
|
||||
const next = storeFromResponse(jar, 'https://example.com/', ['sid=x; Path=/']);
|
||||
assert.equal(next.cookies[0].secure, true);
|
||||
// Which is what keeps it off the wire when a later hop drops to http.
|
||||
assert.equal(cookieHeaderFor(next, 'http://example.com/'), undefined);
|
||||
|
||||
// A cookie a site set over http was never secret to begin with; it is left alone.
|
||||
const plain = storeFromResponse(jar, 'http://example.com/', ['sid=x; Path=/']);
|
||||
assert.equal(plain.cookies[0].secure, undefined);
|
||||
});
|
||||
|
||||
test('normalizeDomain refuses a bare TLD but keeps IPs and localhost', () => {
|
||||
assert.equal(normalizeDomain('.Example.COM.'), 'example.com');
|
||||
assert.equal(normalizeDomain('1.1.1.1'), '1.1.1.1');
|
||||
assert.equal(normalizeDomain('localhost'), 'localhost');
|
||||
// A suffix match on a bare TLD would hand the cookie to every host under it.
|
||||
for (const bad of ['com', 'co', 'localdomain']) {
|
||||
assert.throws(() => normalizeDomain(bad), /bare name/, `expected '${bad}' to be refused`);
|
||||
}
|
||||
for (const bad of ['', '.', '/', 'example.com:8443', 'http://example.com', 'example.com/x', 'ex ample.com', '-x.com']) {
|
||||
assert.throws(() => normalizeDomain(bad), /must be a hostname/, `expected '${bad}' to be refused`);
|
||||
}
|
||||
});
|
||||
|
||||
test('a jar seeded with a bare TLD never reaches every host under it', () => {
|
||||
assert.throws(() => jarFromCookieHeader('sid=secret', 'com'), /bare name/);
|
||||
});
|
||||
|
||||
test('jarFromCookieHeader rejects control characters in a name or value', () => {
|
||||
assert.throws(() => jarFromCookieHeader('sid=a\r\nX-Injected: 1', 'example.com'), /invalid value for cookie 'sid'/);
|
||||
assert.throws(() => jarFromCookieHeader('sid=a\u0000b', 'example.com'), /invalid value for cookie/);
|
||||
assert.throws(() => jarFromCookieHeader('sid=caf\u00e9', 'example.com'), /invalid value for cookie/);
|
||||
assert.throws(() => jarFromCookieHeader('bad name=x', 'example.com'), /invalid cookie name/);
|
||||
assert.throws(() => jarFromCookieHeader('sid=x'.padEnd(MAX_COOKIE_BYTES + 8, 'y'), 'example.com'), /over the .* limit/);
|
||||
// A CR in the error message would let the rejected value rewrite the line.
|
||||
assert.throws(() => jarFromCookieHeader('sid=a\rb', 'example.com'), (err) => !/[\r\n]/.test(err.message));
|
||||
// Values a browser really hands over - base64 padding, commas, quotes - still pass.
|
||||
const ok = jarFromCookieHeader('sid="a,b+c/d=="; _ga=GA1.2.3', 'example.com');
|
||||
assert.equal(ok.cookies.length, 2);
|
||||
});
|
||||
|
||||
test('a hostile response cannot grow the jar past its cap', () => {
|
||||
const jar = jarFromCookieHeader('sid=secret', 'example.com');
|
||||
const headers = Array.from({ length: MAX_COOKIES * 3 }, (_, i) => `junk${i}=x; Path=/`);
|
||||
const next = storeFromResponse(jar, 'https://example.com/', headers);
|
||||
assert.equal(next.cookies.length, MAX_COOKIES);
|
||||
// The seeded login is what survives; the overflow is what is refused.
|
||||
assert.ok(next.cookies.some((c) => c.name === 'sid' && c.value === 'secret'));
|
||||
// A full jar still takes an update to a cookie it already holds.
|
||||
const rotated = storeFromResponse(next, 'https://example.com/', ['junk0=rotated; Path=/']);
|
||||
assert.equal(rotated.cookies.length, MAX_COOKIES);
|
||||
assert.equal(rotated.cookies.find((c) => c.name === 'junk0').value, 'rotated');
|
||||
});
|
||||
|
||||
test('a response cannot smuggle a control character into the next request', () => {
|
||||
const jar = { expiresAt: new Date(Date.now() + 3_600_000).toISOString(), cookies: [] };
|
||||
const next = storeFromResponse(jar, 'https://example.com/', ['sid=a\r\nX-Injected: 1; Path=/']);
|
||||
assert.equal(next.cookies.length, 0);
|
||||
assert.equal(parseSetCookie('sid=a\r\nb', 'https://example.com/'), null);
|
||||
});
|
||||
|
||||
test('cookieHeaderFor matches domain and path', () => {
|
||||
const jar = {
|
||||
expiresAt: new Date(Date.now() + 3_600_000).toISOString(),
|
||||
cookies: [
|
||||
{ name: 'a', value: '1', domain: 'example.com', path: '/' },
|
||||
{ name: 'b', value: '2', domain: 'other.com', path: '/' },
|
||||
{ name: 'c', value: '3', domain: 'example.com', path: '/app', secure: true },
|
||||
],
|
||||
};
|
||||
assert.equal(cookieHeaderFor(jar, 'https://example.com/app/home'), 'a=1; c=3');
|
||||
assert.equal(cookieHeaderFor(jar, 'http://example.com/app/home'), 'a=1');
|
||||
assert.equal(cookieHeaderFor(jar, 'https://other.com/'), 'b=2');
|
||||
assert.equal(cookieHeaderFor(jar, 'https://example.com/other'), 'a=1');
|
||||
});
|
||||
|
||||
test('parseSetCookie reads attributes and pins the cookie host-only', () => {
|
||||
const c = parseSetCookie('sid=val; Path=/app; Domain=.example.com; Secure; HttpOnly; Max-Age=3600',
|
||||
'https://www.example.com/login');
|
||||
assert.equal(c.name, 'sid');
|
||||
assert.equal(c.value, 'val');
|
||||
// Domain is ignored: the cookie is scoped to the host that set it, not the
|
||||
// wider domain the response asked for.
|
||||
assert.equal(c.domain, 'www.example.com');
|
||||
assert.equal(c.path, '/app');
|
||||
assert.ok(c.secure);
|
||||
assert.ok(c.httpOnly);
|
||||
assert.ok(c.expires);
|
||||
});
|
||||
|
||||
test('a response cannot widen a cookie to a public suffix and reach other sites', () => {
|
||||
const jar = { expiresAt: new Date(Date.now() + 3_600_000).toISOString(), cookies: [] };
|
||||
// A page fetched under the jar tries to plant a '.com'-scoped cookie.
|
||||
const next = storeFromResponse(jar, 'https://evil.example/', ['sid=x; Domain=.com; Path=/']);
|
||||
assert.equal(next.cookies[0].domain, 'evil.example');
|
||||
// It is never sent to an unrelated site that merely shares the suffix.
|
||||
assert.equal(cookieHeaderFor(next, 'https://bank.com/'), undefined);
|
||||
assert.equal(cookieHeaderFor(next, 'https://evil.example/'), 'sid=x');
|
||||
});
|
||||
|
||||
test('storeFromResponse replaces cookies with the same name and domain', () => {
|
||||
const jar = {
|
||||
expiresAt: new Date(Date.now() + 3_600_000).toISOString(),
|
||||
cookies: [{ name: 'sid', value: 'old', domain: 'example.com', path: '/' }],
|
||||
};
|
||||
const next = storeFromResponse(jar, 'https://example.com/', ['sid=new; Path=/; Domain=example.com']);
|
||||
assert.equal(next.cookies.length, 1);
|
||||
assert.equal(next.cookies[0].value, 'new');
|
||||
});
|
||||
|
||||
test('session ceiling caps per-cookie expiry from Set-Cookie', () => {
|
||||
const ceiling = new Date(Date.now() + 3_600_000).toISOString();
|
||||
const jar = { expiresAt: ceiling, cookies: [] };
|
||||
const next = storeFromResponse(jar, 'https://example.com/', [
|
||||
'sid=x; Max-Age=86400; Domain=example.com; Path=/',
|
||||
]);
|
||||
assert.equal(next.cookies[0].expires, ceiling);
|
||||
});
|
||||
|
||||
test('saveCookieJar writes with mode 0600 and loadCookieJar reads back', () => {
|
||||
clearCookieJar('work');
|
||||
const jar = jarFromCookieHeader('token=secret', 'example.com', { expiresMs: 3_600_000 });
|
||||
saveCookieJar('work', jar);
|
||||
const mode = statSync(cookieJarPath('work')).mode & 0o777;
|
||||
assert.equal(mode, 0o600);
|
||||
const loaded = loadCookieJar('work');
|
||||
assert.equal(loaded.cookies[0].value, 'secret');
|
||||
});
|
||||
|
||||
test('loadCookieJar returns JAR_EXPIRED and clears an expired jar', () => {
|
||||
clearCookieJar('expired');
|
||||
saveCookieJar('expired', {
|
||||
expiresAt: new Date(Date.now() - 1000).toISOString(),
|
||||
cookies: [{ name: 'a', value: 'b', domain: 'example.com', path: '/' }],
|
||||
});
|
||||
_resetPurgeGuard();
|
||||
assert.equal(loadCookieJar('expired'), JAR_EXPIRED);
|
||||
assert.throws(() => readFileSync(cookieJarPath('expired')), /ENOENT/);
|
||||
});
|
||||
|
||||
test('purgeExpiredJars removes stale sidecar files', () => {
|
||||
clearCookieJar('old');
|
||||
clearCookieJar('fresh');
|
||||
writeFileSync(cookieJarPath('old'), JSON.stringify({
|
||||
expiresAt: new Date(Date.now() - 1000).toISOString(),
|
||||
cookies: [{ name: 'a', value: 'b', domain: 'example.com', path: '/' }],
|
||||
}));
|
||||
saveCookieJar('fresh', jarFromCookieHeader('x=1', 'example.com'));
|
||||
_resetPurgeGuard();
|
||||
purgeExpiredJars();
|
||||
assert.throws(() => readFileSync(cookieJarPath('old')), /ENOENT/);
|
||||
assert.ok(loadCookieJar('fresh'));
|
||||
});
|
||||
|
||||
test('isSessionExpired respects the session ceiling', () => {
|
||||
const jar = { expiresAt: new Date(Date.now() + 1000).toISOString(), cookies: [] };
|
||||
assert.ok(!isSessionExpired(jar));
|
||||
jar.expiresAt = new Date(Date.now() - 1000).toISOString();
|
||||
assert.ok(isSessionExpired(jar));
|
||||
});
|
||||
@@ -549,8 +549,10 @@ test('a link-list page counts as content even with no prose on it', () => {
|
||||
|
||||
test('every fixture page reads as content, none as a failed render', () => {
|
||||
// Feeds, a JSON API, and a YouTube watch page are all thin by design, which
|
||||
// is exactly where this check must not cry wolf.
|
||||
// is exactly where this check must not cry wolf. login.html is an auth-gate
|
||||
// fixture, not a page that should distill as content.
|
||||
for (const name of readdirSync(PAGES)) {
|
||||
if (name === 'login.html') continue;
|
||||
const raw = readFileSync(PAGES + name, 'utf8');
|
||||
const page = distill(raw, `https://api.example.test/2.3/search/advanced?site=fixture&f=${name}`);
|
||||
assert.equal(contentFailure(contentTokens(page), estimateTokens(raw)), null, name);
|
||||
|
||||
@@ -3,6 +3,7 @@ import assert from 'node:assert/strict';
|
||||
import http from 'node:http';
|
||||
import https from 'node:https';
|
||||
import net from 'node:net';
|
||||
import tls from 'node:tls';
|
||||
|
||||
const { fetchPage, followRedirects, resolveProxy, proxyGet } = await import('../src/fetch.js');
|
||||
|
||||
@@ -498,6 +499,90 @@ test('proxyGet returns the origin body through an HTTPS CONNECT tunnel', async (
|
||||
}
|
||||
});
|
||||
|
||||
// A separate cert whose SAN covers the IPv6 loopback ::1, so the tunneled
|
||||
// TLS handshake to an IPv6 literal can be validated offline.
|
||||
const LOCAL_CERT_V6 = `-----BEGIN CERTIFICATE-----
|
||||
MIIBsjCCAVigAwIBAgIUYhesMP2mQCQ6S4SrcGJshDK0g9owCgYIKoZIzj0EAwIw
|
||||
FzEVMBMGA1UEAwwMb2MtaXB2Ni10ZXN0MB4XDTI2MDgyNDE4NTM1MVoXDTM2MDgy
|
||||
MTE4NTM1MVowFzEVMBMGA1UEAwwMb2MtaXB2Ni10ZXN0MFkwEwYHKoZIzj0CAQYI
|
||||
KoZIzj0DAQcDQgAE26JWljo6HQCqheYsEL/xViNZpq+6NPKBlEjlvXf/WtJa2mAl
|
||||
qELRtfWYJeRS+0ogeMNjYXTYME2WKHL3il88cqOBgTB/MB0GA1UdDgQWBBSx4h35
|
||||
MCnlyDhcx7ATZiWyJ/IYEzAfBgNVHSMEGDAWgBSx4h35MCnlyDhcx7ATZiWyJ/IY
|
||||
EzAPBgNVHRMBAf8EBTADAQH/MCwGA1UdEQQlMCOHEAAAAAAAAAAAAAAAAAAAAAGH
|
||||
BH8AAAGCCWxvY2FsaG9zdDAKBggqhkjOPQQDAgNIADBFAiAFJGgQcNAAXI5HWj02
|
||||
NYBPF1nTo3BfOoT/PY5pUsSjuAIhAP9oPp1R2+ckC9sXTOL8n1vw2qVElGDLuvES
|
||||
Hi24p3Qi
|
||||
-----END CERTIFICATE-----`;
|
||||
|
||||
const LOCAL_KEY_V6 = `-----BEGIN PRIVATE KEY-----
|
||||
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgjZ74TmqrsdfKIelm
|
||||
KQFHEBF+5zD8lk8lDLuPgvz2dNGhRANCAATbolaWOjodAKqF5iwQv/FWI1mmr7o0
|
||||
8oGUSOW9d/9a0lraYCWoQtG19Zgl5FL7SiB4w2NhdNgwTZYocveKXzxy
|
||||
-----END PRIVATE KEY-----`;
|
||||
|
||||
test('an IPv6 literal target tunnels through a proxy with its brackets stripped', async () => {
|
||||
// URL.hostname keeps the brackets ("[::1]"); before the fix they reached
|
||||
// tls.connect as a DNS name and the handshake never happened, so what this
|
||||
// test is really about is the host oc hands to the identity check.
|
||||
//
|
||||
// It asserts that host directly rather than letting the handshake stand in
|
||||
// for it: node 24.19 stopped matching IPv6 addresses in a certificate's SAN
|
||||
// (IPv4 still matches), so the default check now rejects an ::1 origin on
|
||||
// grounds that have nothing to do with oc, and 24.8 accepts it. Chain
|
||||
// verification against `ca` stays on; only the hostname step is ours.
|
||||
const origin = https.createServer({ cert: LOCAL_CERT_V6, key: LOCAL_KEY_V6 }, (req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end('<html><title>v6 tunnel</title></html>');
|
||||
});
|
||||
await new Promise((r) => origin.listen(0, '::1', r));
|
||||
const originPort = origin.address().port;
|
||||
|
||||
const proxy = http.createServer();
|
||||
proxy.on('connect', (req, socket) => {
|
||||
const host = req.url.replace(/:\d+$/, '').replace(/^\[|\]$/g, '');
|
||||
const port = Number(req.url.slice(req.url.lastIndexOf(':') + 1));
|
||||
const dest = net.connect(port, host, () => {
|
||||
socket.write('HTTP/1.1 200 Connection Established\r\n\r\n');
|
||||
dest.pipe(socket);
|
||||
socket.pipe(dest);
|
||||
});
|
||||
dest.on('error', () => socket.destroy());
|
||||
});
|
||||
const proxyPort = await listen(proxy);
|
||||
|
||||
const realConnect = tls.connect;
|
||||
let identity = null;
|
||||
let servername = 'unset';
|
||||
tls.connect = (opts, onSecure) => {
|
||||
servername = opts.servername;
|
||||
return realConnect({
|
||||
...opts,
|
||||
checkServerIdentity: (host) => {
|
||||
identity = host;
|
||||
return undefined;
|
||||
},
|
||||
}, onSecure);
|
||||
};
|
||||
|
||||
try {
|
||||
const res = await proxyGet(
|
||||
`https://[::1]:${originPort}/page`,
|
||||
`http://127.0.0.1:${proxyPort}`,
|
||||
{ 'user-agent': 'oc-test' },
|
||||
{ ca: LOCAL_CERT_V6 },
|
||||
);
|
||||
// The bare address, and no SNI: an IP literal is not a server name.
|
||||
assert.equal(identity, '::1');
|
||||
assert.equal(servername, undefined);
|
||||
assert.equal(res.status, 200);
|
||||
assert.equal(await res.text(), '<html><title>v6 tunnel</title></html>');
|
||||
} finally {
|
||||
tls.connect = realConnect;
|
||||
origin.close();
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('followRedirects still blocks a private hop when the transport is a proxy', async () => {
|
||||
// The page 302s to loopback. The proxy is also loopback, which is allowed;
|
||||
// the hop is not. Blocked before the second request goes out.
|
||||
@@ -531,6 +616,80 @@ test('proxyGet refuses a non-HTTP proxy scheme', () => {
|
||||
);
|
||||
});
|
||||
|
||||
test('followRedirects sends jar cookies on every hop and stores Set-Cookie', () => withoutProxyEnv(async () => {
|
||||
const { jarFromCookieHeader, createJarHandle } = await import('../src/cookies.js');
|
||||
const seed = jarFromCookieHeader('sid=abc', 'public.example');
|
||||
const jar = createJarHandle('test', seed);
|
||||
const seen = [];
|
||||
const get = (url) => {
|
||||
seen.push({ url, cookie: jar.cookieHeaderFor(url) });
|
||||
const setCookie = url.includes('/two')
|
||||
? ['fresh=1; Path=/; Domain=public.example']
|
||||
: [];
|
||||
return Promise.resolve({
|
||||
status: url.includes('/two') ? 200 : 302,
|
||||
headers: {
|
||||
get(name) {
|
||||
if (name === 'location' && !url.includes('/two')) return 'https://public.example/two';
|
||||
if (name === 'set-cookie') return setCookie.join(', ');
|
||||
return null;
|
||||
},
|
||||
getSetCookie() { return setCookie; },
|
||||
},
|
||||
});
|
||||
};
|
||||
const { getSetCookieHeaders } = await import('../src/cookies.js');
|
||||
await followRedirects(get, 'https://public.example/one', {
|
||||
onResponse: (url, res) => jar.storeFromResponse(url, getSetCookieHeaders(res)),
|
||||
});
|
||||
assert.equal(seen.length, 2);
|
||||
assert.equal(seen[0].cookie, 'sid=abc');
|
||||
assert.equal(seen[1].cookie, 'sid=abc');
|
||||
assert.ok(jar.toJSON().cookies.some((c) => c.name === 'fresh'));
|
||||
}));
|
||||
|
||||
test('proxyGet forwards a cookie header from the jar', async () => {
|
||||
const seen = [];
|
||||
const proxy = http.createServer((req, res) => {
|
||||
seen.push({ cookie: req.headers.cookie });
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end('<html><title>ok</title></html>');
|
||||
});
|
||||
const port = await listen(proxy);
|
||||
try {
|
||||
await proxyGet('http://example.test/page', `http://127.0.0.1:${port}`, { cookie: 'a=1; b=2' });
|
||||
assert.equal(seen[0].cookie, 'a=1; b=2');
|
||||
} finally {
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('a proxied response exposes each Set-Cookie intact, even with a comma in Expires', async () => {
|
||||
const { getSetCookieHeaders } = await import('../src/cookies.js');
|
||||
const proxy = http.createServer((req, res) => {
|
||||
// Two separate Set-Cookie headers, one carrying a comma inside Expires:
|
||||
// joining them into one string would make them impossible to split back.
|
||||
res.writeHead(200, {
|
||||
'content-type': 'text/html',
|
||||
'set-cookie': [
|
||||
'sid=abc; Path=/; Expires=Wed, 21 Oct 2026 07:28:00 GMT',
|
||||
'theme=dark; Path=/',
|
||||
],
|
||||
});
|
||||
res.end('<html><title>ok</title></html>');
|
||||
});
|
||||
const port = await listen(proxy);
|
||||
try {
|
||||
const res = await proxyGet('http://example.test/page', `http://127.0.0.1:${port}`);
|
||||
const headers = getSetCookieHeaders(res);
|
||||
assert.equal(headers.length, 2);
|
||||
assert.match(headers[0], /^sid=abc;/);
|
||||
assert.match(headers[1], /^theme=dark;/);
|
||||
} finally {
|
||||
proxy.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('a body over the cap is refused, from the header or from the bytes', async () => {
|
||||
const { assertBodySize, readBody, MAX_BODY } = await import('../src/fetch.js');
|
||||
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
<html><head><title>Sign in</title></head><body>
|
||||
<form action="/login">
|
||||
<input type="email" name="email" placeholder="Email">
|
||||
<input type="password" name="password">
|
||||
<button type="submit">Sign in</button>
|
||||
</form>
|
||||
</body></html>
|
||||
Reference in New Issue
Block a user