mirror of
https://github.com/only-cli/oc.git
synced 2026-09-15 10:40:56 +02:00
feat: render JSON API responses as pages
Closes #3. An API answer is a page: jsonToHTML turns a JSON body into one article per item, and everything downstream (numbering, budget, do, read, next, raw) treats it as an ordinary document. No per-site logic and no new dependency. The compact view is the hard part, since a search response carries far more fields than fit in 500 tokens. So the renderer scores each field by how much it varies across items against how wide it prints, penalises fields flattened out of a sub-object (owner.reputation describes the asker, not the answer), and spends about 60 characters per item on the winners. What every item shares is stated once at the bottom instead of repeated, empty fields are named rather than printed, and what was cut says so and points at oc raw, which keeps every field. On the Stack Exchange search endpoint that is 30 results in ~960 tokens against ~5,500 for the raw body, with each title a link and question_id visible. Also here: - clis/stackoverflow.com.json gains search <query>, which is what #3 was blocking. Results carry question_id, and the question feed reads one in full, so search now completes without touching the challenged HTML page. - fetch: the native-fetch path rejected anything that was not HTML or XML. It now accepts JSON, which also makes the two transports render one URL the same way, since the impers path never checked the type at all. - raw threads the URL through so its view of an API response can be titled and, unlike the compact view, keeps every field. Deliberately not done, from the notes on the issue: pagination in the actions line, and API metadata on stderr. There is no stderr channel at the distill seam, so response-level fields (has_more, quota_remaining) render as one footer line instead. A columns hint in the clis specs and a --json passthrough both looked like the wrong trade: the first needs per-site tuning for something the scoring already handles, the second would break the machine-stable Page contract.
This commit is contained in:
only-cli
@@ -73,7 +73,7 @@ Flags: `--budget <tokens>` (default 500), `--json`, `--html` (raw as cleaned HTM
|
||||
|
||||
## Supported websites
|
||||
|
||||
Works on any mostly-static site with no per-site setup: news sites, blogs, documentation, forums, search engines. On top of that, `clis/` ships tuned shortcuts for:
|
||||
Works on any mostly-static site with no per-site setup: news sites, blogs, documentation, forums, search engines. A JSON API is a page here too: `oc open` on an endpoint that answers with JSON renders one numbered item per record, keeps the fields that actually differ between items, and says once what every item shares. On top of that, `clis/` ships tuned shortcuts for:
|
||||
|
||||
| website | domain | shortcuts |
|
||||
| --- | --- | --- |
|
||||
@@ -84,11 +84,11 @@ Works on any mostly-static site with no per-site setup: news sites, blogs, docum
|
||||
| LinkedIn | linkedin.com | `profile <name>`, `company <name>`, `jobs <query>` (public guest views) |
|
||||
| DuckDuckGo | duckduckgo.com | `search <query>`, `lite <query>` |
|
||||
| Bing | bing.com | `search <query>`, `news <query>` |
|
||||
| Stack Overflow | stackoverflow.com (via Atom feeds) | `question <id>`, `tag <name>`, `user <id>`, `recent` |
|
||||
| Stack Overflow | stackoverflow.com (via Atom feeds and the Stack Exchange API) | `search <query>`, `question <id>`, `tag <name>`, `user <id>`, `recent` |
|
||||
| Yahoo Finance | finance.yahoo.com | `quote <symbol>`, `news <symbol>`, `history <symbol>`, `lookup <query>`, `markets`, `gainers`, `losers`, `trending` |
|
||||
| YouTube | youtube.com | `video <id>`, `channel <name>` |
|
||||
|
||||
A few of these (X, Stack Overflow, YouTube) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, or inline data the page already ships without a login. Not supported yet: pages that only render with JavaScript, sites behind logins, and sites with hard bot challenges that expose no feed.
|
||||
A few of these (X, Stack Overflow, YouTube) read pages that look login-gated or JS-only from the outside, by finding the server-rendered HTML, feed, inline data, or public API the page already ships without a login. Stack Overflow search goes through the Stack Exchange API, and each result prints its `question_id`: read one with the `question <id>` feed rather than following its link, since the question page itself answers a bot challenge instead of the question. Not supported yet: pages that only render with JavaScript, sites behind logins, and sites with hard bot challenges that expose no feed.
|
||||
|
||||
Want a website on that list? Open a pull request, or an issue naming the site — see [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
{
|
||||
"domain": "stackoverflow.com",
|
||||
"commands": {
|
||||
"search": { "open": "https://api.stackexchange.com/2.3/search/advanced?order=desc&sort=relevance&site=stackoverflow&q={query}", "args": ["query"] },
|
||||
"question": { "open": "https://stackoverflow.com/feeds/question/{id}", "args": ["id"] },
|
||||
"tag": { "open": "https://stackoverflow.com/feeds/tag?tagnames={name}&sort=newest", "args": ["name"] },
|
||||
"user": { "open": "https://stackoverflow.com/feeds/user/{id}", "args": ["id"] },
|
||||
|
||||
@@ -10,7 +10,8 @@ Key facts:
|
||||
- The budget is a target rather than a hard cap: a page that would finish within about four times it is printed whole, because a second command costs the agent far more than the lines the cut would have saved
|
||||
- The render leads with the page's main content and puts navigation, sidebar, and footer after it, so the budget is spent on what was asked for rather than on menus
|
||||
- Benchmarked at roughly 45x fewer tokens than reading raw HTML, with per-task numbers at https://github.com/only-cli/benchmarks
|
||||
- Works on any mostly-static website; tuned shortcuts ship for Hacker News, Reddit, GitHub, X, LinkedIn (public guest views), DuckDuckGo, Bing, Stack Overflow (via its Atom feeds), and Yahoo Finance (quotes, history, markets)
|
||||
- Works on any mostly-static website; tuned shortcuts ship for Hacker News, Reddit, GitHub, X, LinkedIn (public guest views), DuckDuckGo, Bing, Stack Overflow (via its Atom feeds and the Stack Exchange API), and Yahoo Finance (quotes, history, markets)
|
||||
- JSON APIs render like pages: an endpoint that answers with JSON becomes one numbered item per record, with the fields that differ between items kept and the ones every item shares stated once, so a search endpoint reads like a results page for a few hundred tokens
|
||||
- X profiles and individual posts read without a login (about 390 and 260 tokens); X search, explore, and hashtag pages do not, and oc reports the block instead of guessing
|
||||
- Requests impersonate Chrome, so pages that block plain scripts often still work
|
||||
- Agent skill included: `npx skills add https://github.com/only-cli/oc --skill web-browsing-cli` ([skills.sh](https://www.skills.sh/only-cli/oc/web-browsing-cli))
|
||||
|
||||
+1
-1
@@ -130,7 +130,7 @@ async function main() {
|
||||
}
|
||||
const htmlTokens = estimateTokens(html);
|
||||
if (command === 'raw') {
|
||||
const out = values.html ? toHTML(html) : toMarkdown(html);
|
||||
const out = values.html ? toHTML(html, finalUrl) : toMarkdown(html, finalUrl);
|
||||
console.log(out);
|
||||
if (verbose) console.error(`${savings(estimateTokens(out), htmlTokens)}; ${resources()}`);
|
||||
return;
|
||||
|
||||
+375
-7
@@ -70,6 +70,18 @@ const REPEAT_MAX_LEN = 25;
|
||||
|
||||
const clean = (s) => s.replace(/\s+/g, ' ').trim();
|
||||
|
||||
/**
|
||||
* Everything that is not a page, made into one. The compact view and both raw
|
||||
* modes go through here, so a format is never readable in one of them and a
|
||||
* blob in the other. Each converter recognises its own input and returns null
|
||||
* otherwise, and HTML falls through untouched.
|
||||
* @param {string} text
|
||||
* @param {string} url
|
||||
* @returns {string}
|
||||
*/
|
||||
const asHTML = (text, url = '', opts = {}) =>
|
||||
jsonToHTML(text, url, opts) ?? youtubeToHTML(text) ?? transcriptToHTML(text) ?? feedToHTML(text) ?? text;
|
||||
|
||||
/**
|
||||
* Reduce raw HTML to an interaction tree: readable text plus numbered
|
||||
* elements, in document order. The walk is deterministic and numbering is a
|
||||
@@ -80,7 +92,7 @@ const clean = (s) => s.replace(/\s+/g, ' ').trim();
|
||||
* @returns {Page}
|
||||
*/
|
||||
export function distill(html, url = '') {
|
||||
const { document } = parseHTML(youtubeToHTML(html) ?? transcriptToHTML(html) ?? feedToHTML(html) ?? html);
|
||||
const { document } = parseHTML(asHTML(html, url));
|
||||
const title = clean(document.querySelector('title')?.textContent ?? '');
|
||||
/** @type {Block[]} */
|
||||
const blocks = [];
|
||||
@@ -302,8 +314,10 @@ const bodyOf = (document) => document.querySelector('body') ?? document.document
|
||||
* hidden content.
|
||||
* @param {string} html
|
||||
*/
|
||||
function cleanDocument(html) {
|
||||
const { document } = parseHTML(youtubeToHTML(html) ?? transcriptToHTML(html) ?? feedToHTML(html) ?? html);
|
||||
function cleanDocument(html, url = '') {
|
||||
// Raw is the mode an agent reaches for when the compact view left something
|
||||
// out, so it is the one place a JSON response keeps every field.
|
||||
const { document } = parseHTML(asHTML(html, url, { full: true }));
|
||||
// Read the title before the sweep below removes the head with it.
|
||||
const title = clean(document.querySelector('title')?.textContent ?? '');
|
||||
for (const tag of DROP) {
|
||||
@@ -320,10 +334,12 @@ function cleanDocument(html) {
|
||||
* Whole-page markdown for `oc raw`, produced by turndown so lists, emphasis,
|
||||
* links, and code blocks come out as real markdown instead of flat lines.
|
||||
* @param {string} html
|
||||
* @param {string} url - only read when the body turns out to be JSON, whose
|
||||
* title has to come from the endpoint because the payload has none
|
||||
* @returns {string}
|
||||
*/
|
||||
export function toMarkdown(html) {
|
||||
const { document, title } = cleanDocument(html);
|
||||
export function toMarkdown(html, url = '') {
|
||||
const { document, title } = cleanDocument(html, url);
|
||||
const turndown = new TurndownService({ headingStyle: 'atx', codeBlockStyle: 'fenced' });
|
||||
const el = bodyOf(document);
|
||||
const body = el ? turndown.turndown(el.innerHTML).trim() : '';
|
||||
@@ -334,10 +350,11 @@ export function toMarkdown(html) {
|
||||
* Whole-page cleaned HTML for `oc raw --html`, for agents that would rather
|
||||
* work with markup than markdown. Same noise removal, no other rewriting.
|
||||
* @param {string} html
|
||||
* @param {string} url - see toMarkdown
|
||||
* @returns {string}
|
||||
*/
|
||||
export function toHTML(html) {
|
||||
const { document } = cleanDocument(html);
|
||||
export function toHTML(html, url = '') {
|
||||
const { document } = cleanDocument(html, url);
|
||||
const el = bodyOf(document);
|
||||
return el ? el.innerHTML.trim() : '';
|
||||
}
|
||||
@@ -489,6 +506,357 @@ export function transcriptToHTML(text) {
|
||||
return `<html><head><title>Transcript</title></head><body>\n<p>${escHTML(lines.join(' '))}</p>\n</body></html>`;
|
||||
}
|
||||
|
||||
// Keys an API is likely to give the human-readable name of an item, in the
|
||||
// order they win when an item carries several of them.
|
||||
const TITLE_KEYS = [
|
||||
'title', 'name', 'headline', 'subject', 'label', 'display_name',
|
||||
'full_name', 'summary', 'question', 'message',
|
||||
];
|
||||
|
||||
// Keys holding the item's own page. A URL under any other name is still found,
|
||||
// by looking at values rather than names, but these win when several qualify.
|
||||
const LINK_KEYS = ['link', 'url', 'html_url', 'web_url', 'permalink', 'href'];
|
||||
|
||||
// A title has to fit on a line to be one. Anything longer is a body that
|
||||
// happens to live under a title-ish key, and belongs in a block of its own.
|
||||
const TITLE_MAX = 300;
|
||||
|
||||
// How many constant fields the footer names before it stops counting them out.
|
||||
const CONST_LISTED = 8;
|
||||
|
||||
// Characters of field text an item may spend in the compact view. A response
|
||||
// carries far more fields than an agent asked for: one Stack Exchange result
|
||||
// brings eight about the asker alone, which is the whole budget spent on who
|
||||
// rather than what. Thirty items make this thirty times over, so it buys two
|
||||
// or three fields, not a record. `oc raw` still has all of them.
|
||||
const FIELD_BUDGET = 60;
|
||||
|
||||
// What a field from a flattened sub-object scores against one of the item's
|
||||
// own. `owner.user_id` varies perfectly and costs little, which is enough to
|
||||
// win on width and variance alone, but it identifies somebody attached to the
|
||||
// result rather than the result, and no agent searched for it.
|
||||
const NESTED_PENALTY = 0.25;
|
||||
|
||||
// How many names the footer lists when it says which fields it left out.
|
||||
const DROPPED_LISTED = 5;
|
||||
|
||||
// Seconds and milliseconds since the epoch, bounded either side so an ordinary
|
||||
// count (a score, a byte size) is never mistaken for a date.
|
||||
const EPOCH_S = [1e9, 4e9];
|
||||
const EPOCH_MS = [1e12, 4e12];
|
||||
const DATE_KEY = /(^|_)(date|at|time|timestamp|created|updated|published|modified)$/i;
|
||||
|
||||
// Named entities worth knowing without a table: the five XML ones plus the
|
||||
// space. Everything else arrives numeric.
|
||||
const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ' };
|
||||
|
||||
/**
|
||||
* Undo one layer of HTML escaping in a string value. APIs that back an HTML
|
||||
* site tend to escape the text they return: Stack Exchange answers with
|
||||
* `Is "==" slower`, and re-escaping that on the way into a document
|
||||
* would print the entity instead of the quote it stands for.
|
||||
* @param {string} s
|
||||
* @returns {string}
|
||||
*/
|
||||
const decodeEntities = (s) =>
|
||||
s.replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (whole, body) => {
|
||||
if (body[0] === '#') {
|
||||
const code = body[1] === 'x' || body[1] === 'X'
|
||||
? parseInt(body.slice(2), 16)
|
||||
: Number(body.slice(1));
|
||||
return Number.isInteger(code) && code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : whole;
|
||||
}
|
||||
return ENTITIES[body.toLowerCase()] ?? whole;
|
||||
});
|
||||
|
||||
const isPlain = (v) => v !== null && typeof v === 'object' && !Array.isArray(v);
|
||||
const isURL = (v) => typeof v === 'string' && /^https?:\/\/\S+$/.test(v);
|
||||
const looksHTML = (v) => typeof v === 'string' && /<\/?(p|div|pre|code|br|ul|ol|li|h[1-6]|blockquote|table|img|a|em|strong)\b[^>]*>/i.test(v);
|
||||
|
||||
/**
|
||||
* One level of flattening, so `owner: {display_name}` becomes an
|
||||
* `owner.display_name` field. Deeper than that an object stops being a set of
|
||||
* fields and starts being a document, which no line-per-item view can hold.
|
||||
* @param {Record<string, any>} item
|
||||
* @returns {Map<string, any>}
|
||||
*/
|
||||
function flattenItem(item) {
|
||||
/** @type {Map<string, any>} */
|
||||
const out = new Map();
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
if (!isPlain(value)) {
|
||||
out.set(key, value);
|
||||
continue;
|
||||
}
|
||||
for (const [inner, deep] of Object.entries(value)) {
|
||||
if (deep !== null && typeof deep === 'object') continue;
|
||||
out.set(`${key}.${inner}`, deep);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* One field as one string. Epoch integers under a date-ish key become ISO
|
||||
* dates, because an agent that has to convert one pays a turn for it. Arrays
|
||||
* of scalars (tags, labels) join; arrays of objects are counted, since
|
||||
* spelling them out is what the flattening above already refused to do.
|
||||
* @param {string} key
|
||||
* @param {any} value
|
||||
* @returns {string}
|
||||
*/
|
||||
function renderValue(key, value) {
|
||||
if (value === null || value === undefined) return '';
|
||||
if (Array.isArray(value)) {
|
||||
if (!value.length) return '';
|
||||
if (value.every((v) => v === null || typeof v !== 'object')) return value.join(', ');
|
||||
return `[${value.length} items]`;
|
||||
}
|
||||
if (typeof value === 'object') return '';
|
||||
if (typeof value === 'number' && Number.isInteger(value) && DATE_KEY.test(key)) {
|
||||
const ms = value >= EPOCH_S[0] && value < EPOCH_S[1] ? value * 1000
|
||||
: value >= EPOCH_MS[0] && value < EPOCH_MS[1] ? value
|
||||
: null;
|
||||
if (ms !== null) return new Date(ms).toISOString().slice(0, 19).replace('T', ' ');
|
||||
}
|
||||
return typeof value === 'string' ? decodeEntities(value) : String(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Order fields by how much they say per character, and keep taking them while
|
||||
* an item can still afford one. Variance is what carries the information: a
|
||||
* field reading the same on every row has already been lifted out as a
|
||||
* constant, and one reading differently every time is why the response was
|
||||
* fetched. Dividing by width is what stops a long field (a profile image URL,
|
||||
* a licence string) from crowding out three short ones that matter more.
|
||||
* @param {Array<Map<string, string>|null>} rows
|
||||
* @param {Set<string>} skip - fields already spoken for as title, link, or constant
|
||||
* @returns {{kept: Set<string>, dropped: string[]}}
|
||||
*/
|
||||
function chooseFields(rows, skip) {
|
||||
const present = rows.filter(Boolean);
|
||||
const keys = [];
|
||||
for (const row of present) {
|
||||
for (const key of row.keys()) if (!skip.has(key) && !keys.includes(key)) keys.push(key);
|
||||
}
|
||||
const scored = [];
|
||||
for (const key of keys) {
|
||||
const values = present.map((row) => row.get(key) ?? '').filter((v) => v !== '');
|
||||
if (!values.length) continue;
|
||||
const width = values.reduce((sum, v) => sum + v.length + key.length + 3, 0) / values.length;
|
||||
const variance = new Set(values).size / values.length;
|
||||
const penalty = key.includes('.') ? NESTED_PENALTY : 1;
|
||||
scored.push({ key, width, score: (variance / width) * penalty });
|
||||
}
|
||||
scored.sort((a, b) => b.score - a.score);
|
||||
const kept = new Set();
|
||||
let spent = 0;
|
||||
for (const field of scored) {
|
||||
// The first field is taken whatever it costs, so an item of one long field
|
||||
// still renders something rather than nothing.
|
||||
if (spent + field.width > FIELD_BUDGET && kept.size) continue;
|
||||
kept.add(field.key);
|
||||
spent += field.width;
|
||||
}
|
||||
return { kept, dropped: scored.filter((f) => !kept.has(f.key)).map((f) => f.key) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick the array the response is actually about: the root when it is one,
|
||||
* otherwise the longest array of objects at the top level, which is where
|
||||
* `items`, `data`, `results`, and `hits` all live. Everything beside it is
|
||||
* metadata about the request rather than content.
|
||||
* @param {any} data
|
||||
* @returns {{items: any[], meta: Record<string, any>}}
|
||||
*/
|
||||
function mainArray(data) {
|
||||
if (Array.isArray(data)) return { items: data, meta: {} };
|
||||
let key = '';
|
||||
/** @type {any[] | null} */
|
||||
let items = null;
|
||||
for (const [k, v] of Object.entries(data)) {
|
||||
if (!Array.isArray(v) || !v.length) continue;
|
||||
if (!v.some(isPlain)) continue;
|
||||
if (!items || v.length > items.length) {
|
||||
items = v;
|
||||
key = k;
|
||||
}
|
||||
}
|
||||
// A response with no array is a single resource, which renders as one item
|
||||
// rather than as a special case.
|
||||
if (!items) return { items: [data], meta: {} };
|
||||
const meta = { ...data };
|
||||
delete meta[key];
|
||||
return { items, meta };
|
||||
}
|
||||
|
||||
/**
|
||||
* Fields whose rendered value is the same on every item. In a list of thirty
|
||||
* results they are thirty copies of one fact, so they come out of the rows and
|
||||
* get stated once at the foot of the page: the saving is the point of the
|
||||
* exercise, and dropping them silently would be a lie about what the API said.
|
||||
* @param {Array<Map<string, string>|null>} rows
|
||||
* @returns {Map<string, string>}
|
||||
*/
|
||||
function constantFields(rows) {
|
||||
const present = rows.filter(Boolean);
|
||||
/** @type {Map<string, string>} */
|
||||
const out = new Map();
|
||||
if (present.length < 2) return out;
|
||||
for (const [key, value] of present[0]) {
|
||||
if (present.every((row) => row.get(key) === value)) out.set(key, value);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* A JSON body is not a page, so nothing downstream can read one: the HTML
|
||||
* parser turns it into a single unreadable text node and the budget truncates
|
||||
* the blob. This turns a response into the same shape feedToHTML produces, one
|
||||
* article per item, so numbering, the budget, `oc do`, and raw markdown all
|
||||
* work on an API exactly as they do on a page.
|
||||
*
|
||||
* The view is a line per item rather than a table: two or three fields carry
|
||||
* the signal in most responses, and turndown cannot write a markdown table
|
||||
* without the GFM plugin, which is a dependency this project does not want.
|
||||
* Returns null for anything that is not JSON.
|
||||
* @param {string} text
|
||||
* @param {string} url
|
||||
* @param {{full?: boolean}} [opts] - full keeps every field, which is what the
|
||||
* raw modes are for; the compact view keeps the ones that earn their tokens
|
||||
* @returns {string | null}
|
||||
*/
|
||||
export function jsonToHTML(text, url = '', { full = false } = {}) {
|
||||
if (!/^\s*[[{]/.test(text.slice(0, 200))) return null;
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(text);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (!data || typeof data !== 'object') return null;
|
||||
|
||||
const { items, meta } = mainArray(data);
|
||||
const flats = items.map((item) => (isPlain(item) ? flattenItem(item) : null));
|
||||
const rows = flats.map((flat) => {
|
||||
if (!flat) return null;
|
||||
/** @type {Map<string, string>} */
|
||||
const row = new Map();
|
||||
for (const [key, value] of flat) row.set(key, renderValue(key, value));
|
||||
return row;
|
||||
});
|
||||
|
||||
// One item's field names stand for all of them: a ragged response still gets
|
||||
// a consistent title and link column, and a field missing from an item is
|
||||
// simply absent from its line.
|
||||
const sample = flats.find(Boolean) ?? new Map();
|
||||
const titled = (k) => {
|
||||
const v = sample.get(k);
|
||||
return typeof v === 'string' && v.trim() && v.length <= TITLE_MAX && !isURL(v) && !looksHTML(v);
|
||||
};
|
||||
const titleKey = TITLE_KEYS.find(titled) ?? [...sample.keys()].find(titled);
|
||||
const linkKey = LINK_KEYS.find((k) => isURL(sample.get(k))) ?? [...sample.keys()].find((k) => isURL(sample.get(k)));
|
||||
|
||||
const constants = constantFields(rows);
|
||||
const spoken = new Set([titleKey, linkKey, ...constants.keys()].filter(Boolean));
|
||||
const { kept, dropped } = full
|
||||
? { kept: null, dropped: [] }
|
||||
: chooseFields(rows, spoken);
|
||||
const parts = [];
|
||||
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
const flat = flats[i];
|
||||
if (!flat) {
|
||||
const line = renderValue('', items[i]);
|
||||
if (line) parts.push(`<p>${escHTML(line)}</p>`);
|
||||
continue;
|
||||
}
|
||||
const row = rows[i];
|
||||
const title = titleKey ? row.get(titleKey) : '';
|
||||
const link = linkKey && isURL(flat.get(linkKey)) ? flat.get(linkKey) : '';
|
||||
const inline = [];
|
||||
const long = [];
|
||||
const bodies = [];
|
||||
for (const [key, value] of flat) {
|
||||
if (spoken.has(key)) continue;
|
||||
// A field carrying markup is a document, and is kept whatever it scored:
|
||||
// it renders as its own block, so it never competes for the field line.
|
||||
if (kept && !kept.has(key) && !looksHTML(value)) continue;
|
||||
// A field carrying HTML is a page in itself: `filter=withbody` on the
|
||||
// Stack Exchange API puts a whole question in one. It goes through the
|
||||
// distiller like any other markup instead of into a cell.
|
||||
if (looksHTML(value)) {
|
||||
bodies.push(String(value));
|
||||
continue;
|
||||
}
|
||||
const rendered = row.get(key);
|
||||
if (!rendered) continue;
|
||||
if (rendered.length > TEXT_CAP) long.push(`<p>${escHTML(`${key}: ${rendered}`)}</p>`);
|
||||
else inline.push(`${key}: ${rendered}`);
|
||||
}
|
||||
|
||||
parts.push('<article>');
|
||||
if (title && link) parts.push(`<p><a href="${escHTML(link)}">${escHTML(title)}</a></p>`);
|
||||
else if (title) parts.push(`<p>${escHTML(title)}</p>`);
|
||||
else if (link) parts.push(`<p><a href="${escHTML(link)}">open</a></p>`);
|
||||
if (inline.length) parts.push(`<p>${escHTML(inline.join(' | '))}</p>`);
|
||||
parts.push(...long);
|
||||
for (const body of bodies) parts.push(`<div>${body}</div>`);
|
||||
parts.push('</article>');
|
||||
}
|
||||
|
||||
// What the rows no longer carry, said once. Empty-everywhere fields are
|
||||
// named but not valued, because their value is the fact that there isn't one.
|
||||
const shown = [...constants].filter(([, v]) => v !== '');
|
||||
const empty = [...constants].filter(([, v]) => v === '').map(([k]) => k);
|
||||
const clipped = (list, render) => {
|
||||
const head = list.slice(0, CONST_LISTED).map(render).join(', ');
|
||||
return list.length > CONST_LISTED ? `${head}, +${list.length - CONST_LISTED} more` : head;
|
||||
};
|
||||
if (shown.length) {
|
||||
parts.push(`<p>same on every item: ${escHTML(clipped(shown, ([k, v]) => `${k}=${v.length > 60 ? `${v.slice(0, 60)}...` : v}`))}</p>`);
|
||||
}
|
||||
if (empty.length) parts.push(`<p>empty on every item: ${escHTML(clipped(empty, (k) => k))}</p>`);
|
||||
if (dropped.length) {
|
||||
const names = dropped.slice(0, DROPPED_LISTED).join(', ');
|
||||
const more = dropped.length > DROPPED_LISTED ? `, +${dropped.length - DROPPED_LISTED} more` : '';
|
||||
parts.push(`<p>${dropped.length} fields per item not shown (${escHTML(names + more)}), 'oc raw' has them</p>`);
|
||||
}
|
||||
|
||||
const metaBits = [];
|
||||
for (const [key, value] of Object.entries(meta)) {
|
||||
if (value === null || typeof value === 'object') continue;
|
||||
const rendered = renderValue(key, value);
|
||||
if (rendered) metaBits.push(`${key}=${rendered}`);
|
||||
}
|
||||
if (metaBits.length) parts.push(`<p>response: ${escHTML(metaBits.join(', '))}</p>`);
|
||||
|
||||
const count = `${items.length} ${items.length === 1 ? 'item' : 'items'}`;
|
||||
return `<html><head><title>${escHTML(jsonTitle(url, count))}</title></head><body>\n${parts.join('\n')}\n</body></html>`;
|
||||
}
|
||||
|
||||
/**
|
||||
* An API response has no title of its own, so the endpoint becomes one. The
|
||||
* query parameter is worth the tokens it costs: it is the only part of a
|
||||
* search URL that says what the page is, and without it every search a session
|
||||
* runs is titled the same.
|
||||
* @param {string} url
|
||||
* @param {string} count
|
||||
* @returns {string}
|
||||
*/
|
||||
function jsonTitle(url, count) {
|
||||
try {
|
||||
const u = new URL(url);
|
||||
const query = ['q', 'query', 'search', 'terms', 'keywords', 'text']
|
||||
.map((k) => u.searchParams.get(k))
|
||||
.find((v) => v);
|
||||
const base = `${u.host}${u.pathname}`.replace(/\/+$/, '');
|
||||
return query ? `${base}: "${query}" (${count})` : `${base} (${count})`;
|
||||
} catch {
|
||||
return `JSON (${count})`;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Adjacent text nodes arrive fragmented (one per inline element boundary).
|
||||
* Merging them is what turns DOM noise into readable lines.
|
||||
|
||||
+5
-2
@@ -197,9 +197,12 @@ async function viaFetch(target) {
|
||||
if (!res.ok) {
|
||||
throw new Error(`fetch failed: ${res.status} ${res.statusText} for ${current}`);
|
||||
}
|
||||
// JSON is a page here too: an API answer distills into one article per item.
|
||||
// The impers path never checked the type at all, so this is also what keeps
|
||||
// the two transports rendering the same URL the same way.
|
||||
const type = res.headers.get('content-type') ?? '';
|
||||
if (type && !type.includes('html') && !type.includes('xml')) {
|
||||
throw new Error(`not an HTML page (${type.split(';')[0]}), nothing to distill`);
|
||||
if (type && !/html|xml|json/.test(type)) {
|
||||
throw new Error(`not a page oc can read (${type.split(';')[0]}), it renders HTML, XML feeds, and JSON`);
|
||||
}
|
||||
return { url: res.url || current, html: await res.text(), status: res.status, via: 'fetch' };
|
||||
}
|
||||
|
||||
+72
-1
@@ -1,7 +1,7 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { distill, toMarkdown, toHTML, feedToHTML, youtubeToHTML, transcriptToHTML, TEXT_CAP } from '../src/distill.js';
|
||||
import { distill, toMarkdown, toHTML, feedToHTML, jsonToHTML, youtubeToHTML, transcriptToHTML, TEXT_CAP } from '../src/distill.js';
|
||||
import { render, estimateTokens } from '../src/render.js';
|
||||
|
||||
const html = readFileSync(new URL('./pages/news.html', import.meta.url), 'utf8');
|
||||
@@ -11,6 +11,12 @@ const forum = readFileSync(new URL('./pages/forum.html', import.meta.url), 'utf8
|
||||
const thread = () => distill(forum, 'https://example.test/t/1');
|
||||
const timeline = readFileSync(new URL('./pages/social.html', import.meta.url), 'utf8');
|
||||
const social = () => distill(timeline, 'https://social.test/fixture');
|
||||
const api = readFileSync(new URL('./pages/api.json', import.meta.url), 'utf8');
|
||||
const API_URL = 'https://api.example.test/2.3/search/advanced?site=fixture&q=sky+blue';
|
||||
const results = () => distill(api, API_URL);
|
||||
// Turndown escapes the underscores in field names, which is correct markdown
|
||||
// and only noise to assert against.
|
||||
const rawApi = () => toMarkdown(api, API_URL).replace(/\\_/g, '_');
|
||||
|
||||
test('noise never reaches the output, compact or raw', () => {
|
||||
for (const out of [render(page(), { budget: 5000 }).text, toMarkdown(html), toHTML(html)]) {
|
||||
@@ -260,6 +266,71 @@ test('separate posts stay separate blocks', () => {
|
||||
assert.ok(!ncurses.includes('1987 manual'), `two posts merged into one block:\n${ncurses}`);
|
||||
});
|
||||
|
||||
test('a json api response renders as items, each title a link to its page', () => {
|
||||
const p = results();
|
||||
assert.equal(p.title, 'api.example.test/2.3/search/advanced: "sky blue" (3 items)');
|
||||
const links = p.blocks.filter((b) => b.type === 'link');
|
||||
assert.equal(links[0].text, 'Why is the sky blue, and why does "blue" scatter most?', 'title left html-escaped');
|
||||
assert.equal(links[0].href, 'https://example.test/questions/42/why-is-the-sky-blue');
|
||||
assert.ok(links[0].n, 'a result with no handle cannot be followed');
|
||||
const text = p.blocks.map((b) => b.text).join('\n');
|
||||
assert.ok(/score: 512/.test(text), 'the fields that vary went missing');
|
||||
});
|
||||
|
||||
test('fields identical on every item are stated once, not thirty times', () => {
|
||||
const { text } = render(results(), { budget: 2000 });
|
||||
assert.equal(text.match(/content_license/g).length, 1, 'a constant field repeated per item');
|
||||
assert.ok(text.includes('same on every item: is_answered=true, content_license=CC BY-SA 4.0'));
|
||||
assert.ok(text.includes('empty on every item: closed_date'), 'a null-everywhere field vanished silently');
|
||||
});
|
||||
|
||||
test('the compact view drops low-value fields and says which, raw keeps them all', () => {
|
||||
const { text } = render(results(), { budget: 2000 });
|
||||
assert.ok(!text.includes('profile_image'), 'an image url outscored the fields worth reading');
|
||||
assert.ok(/\d+ fields per item not shown \(/.test(text), 'fields went missing with nothing said about it');
|
||||
const md = rawApi();
|
||||
assert.ok(md.includes('profile_image'), 'raw mode is the escape hatch and has to hold everything');
|
||||
assert.ok(md.includes('owner.display_name: Ray Leigh'), 'nested objects flatten one level');
|
||||
});
|
||||
|
||||
test('request metadata sits in the footer instead of on every row', () => {
|
||||
const { text } = render(results(), { budget: 2000 });
|
||||
assert.ok(text.includes('response: has_more=true, quota_max=300, quota_remaining=297'));
|
||||
assert.equal(text.match(/quota_remaining/g).length, 1);
|
||||
});
|
||||
|
||||
test('epoch timestamps under a date key render as dates', () => {
|
||||
const md = rawApi();
|
||||
assert.ok(md.includes('creation_date: 2012-06-27 13:51:36'), `epoch left raw:\n${md.slice(0, 400)}`);
|
||||
// A number that is not a date must stay the number it is.
|
||||
assert.ok(md.includes('view_count: 91234'), 'a plain count was mangled into a date');
|
||||
});
|
||||
|
||||
test('an html field in json goes through the distiller, not into a cell', () => {
|
||||
const md = rawApi();
|
||||
assert.ok(md.includes('```\nwavelength < 450nm'), 'code block in a json body field was flattened');
|
||||
assert.ok(md.includes('[the derivation](https://example.test/scattering)'), 'link inside a json body field lost');
|
||||
const p = results();
|
||||
assert.ok(p.blocks.some((b) => b.type === 'link' && b.text === 'the derivation'), 'body link is not followable');
|
||||
});
|
||||
|
||||
test('json shapes other than a wrapped array still render', () => {
|
||||
const bare = distill('[{"name":"one","url":"https://example.test/1"},{"name":"two","url":"https://example.test/2"}]', 'https://x.test/a.json');
|
||||
assert.equal(bare.blocks.filter((b) => b.type === 'link').length, 2, 'a root array lost its items');
|
||||
const single = distill('{"title":"Just one","score":3}', 'https://x.test/one.json');
|
||||
assert.ok(single.blocks.some((b) => b.text === 'Just one'), 'an object with no array rendered nothing');
|
||||
assert.ok(single.blocks.some((b) => b.text?.includes('score: 3')), 'a single resource lost its fields');
|
||||
});
|
||||
|
||||
test('only json is read as json', () => {
|
||||
assert.equal(jsonToHTML(html), null, 'an html page was parsed as json');
|
||||
assert.equal(jsonToHTML(feed), null, 'a feed was parsed as json');
|
||||
assert.equal(jsonToHTML('{"broken": '), null, 'invalid json did not fall through to the html path');
|
||||
assert.equal(jsonToHTML('"a string"'), null, 'a bare scalar has no items to render');
|
||||
// The feed path must still win for xml, and html must reach the html parser.
|
||||
assert.ok(distill(feed, 'https://x.test/f').title.includes('Fixture Overflow'));
|
||||
});
|
||||
|
||||
test('long runs of short links collapse into a range marker', () => {
|
||||
const nav = Array.from({ length: 15 }, (_, i) => `<a href="/s/${i}">sub${i}</a>`).join(' ');
|
||||
const navHtml = `<html><head><title>T</title></head><body>${nav}<p>actual content</p></body></html>`;
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"tags": ["physics", "optics"],
|
||||
"owner": {
|
||||
"account_id": 1,
|
||||
"reputation": 5120,
|
||||
"user_id": 11,
|
||||
"display_name": "Ray Leigh",
|
||||
"profile_image": "https://example.test/img/1.png?s=256",
|
||||
"link": "https://example.test/users/11/ray-leigh"
|
||||
},
|
||||
"is_answered": true,
|
||||
"closed_date": null,
|
||||
"view_count": 91234,
|
||||
"answer_count": 4,
|
||||
"score": 512,
|
||||
"creation_date": 1340805096,
|
||||
"question_id": 42,
|
||||
"content_license": "CC BY-SA 4.0",
|
||||
"link": "https://example.test/questions/42/why-is-the-sky-blue",
|
||||
"title": "Why is the sky blue, and why does "blue" scatter most?",
|
||||
"body": "<p>Looking up on a clear day the sky is blue, yet sunlight is white.</p>\n<pre><code>wavelength < 450nm\n</code></pre>\n<p>What scatters the shorter wavelengths? See <a href=\"https://example.test/scattering\">the derivation</a>.</p>"
|
||||
},
|
||||
{
|
||||
"tags": ["optics"],
|
||||
"owner": {
|
||||
"account_id": 2,
|
||||
"reputation": 87,
|
||||
"user_id": 12,
|
||||
"display_name": "Tyndall",
|
||||
"profile_image": "https://example.test/img/2.png?s=256",
|
||||
"link": "https://example.test/users/12/tyndall"
|
||||
},
|
||||
"is_answered": true,
|
||||
"closed_date": null,
|
||||
"view_count": 300,
|
||||
"answer_count": 1,
|
||||
"score": 7,
|
||||
"creation_date": 1340808000,
|
||||
"question_id": 43,
|
||||
"content_license": "CC BY-SA 4.0",
|
||||
"link": "https://example.test/questions/43/what-is-tyndall-scattering",
|
||||
"title": "What is Tyndall scattering?"
|
||||
},
|
||||
{
|
||||
"tags": ["physics", "atmosphere"],
|
||||
"owner": {
|
||||
"account_id": 3,
|
||||
"reputation": 1904,
|
||||
"user_id": 13,
|
||||
"display_name": "Mie",
|
||||
"profile_image": "https://example.test/img/3.png?s=256",
|
||||
"link": "https://example.test/users/13/mie"
|
||||
},
|
||||
"is_answered": true,
|
||||
"closed_date": null,
|
||||
"view_count": 1580,
|
||||
"answer_count": 2,
|
||||
"score": 33,
|
||||
"creation_date": 1340900000,
|
||||
"question_id": 44,
|
||||
"content_license": "CC BY-SA 4.0",
|
||||
"link": "https://example.test/questions/44/why-are-sunsets-red",
|
||||
"title": "Why are sunsets red?"
|
||||
}
|
||||
],
|
||||
"has_more": true,
|
||||
"quota_max": 300,
|
||||
"quota_remaining": 297
|
||||
}
|
||||
Reference in New Issue
Block a user