mirror of
https://github.com/snapotter-hq/SnapOtter.git
synced 2026-08-03 07:46:42 +02:00
feat(i18n): 21-language pipeline, landing/docs/API wiring, landing+API translations
Shared Claude Code translation pipeline (scripts/i18n, no API key) plus Astro/VitePress/Scalar i18n wiring. Landing and API reference translated into all 20 languages; docs i18n wiring + English source anchors. The translated docs markdown (apps/docs/<locale>/**, 3,620 files) follows in a companion PR because it exceeds GitHub's per-PR CI file limit.
This commit is contained in:
SnapOtter
@@ -0,0 +1,144 @@
|
||||
<!-- scripts/i18n/README.md -->
|
||||
# Web-surfaces translation pipeline
|
||||
|
||||
Translates the three public web surfaces (landing site, docs, API reference) into
|
||||
the 20 non-English languages the main app already supports, gated on content
|
||||
hashes so only changed English source is re-translated. This is build-time
|
||||
tooling. It is not shipped in any app runtime and never runs on an end user's
|
||||
machine.
|
||||
|
||||
The pipeline core, the surface adapters, and the CLI live in this directory. The
|
||||
adapter registry (`adapters/registry.mjs`) is the single source of truth for the
|
||||
four surfaces (`landing-ui`, `landing-seo`, `docs`, `api`); both `translate.mjs`
|
||||
and `check-parity.mjs` import it, so the surface list is defined once.
|
||||
|
||||
## The engine: a Claude Code batch handoff (no API key)
|
||||
|
||||
Translation is a three-step handoff, not a headless API call. There is no
|
||||
`ANTHROPIC_API_KEY` in the default path and no CI job that spends tokens.
|
||||
|
||||
1. **Export.** `--export` collects every pending unit (missing, or whose English
|
||||
source moved under a machine translation), masks the parts a translator must
|
||||
not touch (code fences, inline code, link URLs, `{placeholders}`), and writes
|
||||
one file per surface and locale:
|
||||
`.i18n-batches/<surface>.<locale>.pending.json`. Each row is `{ id, masked }`.
|
||||
2. **Translate.** A human, or a Claude Code session, translates each pending file
|
||||
into `.i18n-batches/<surface>.<locale>.done.json`, a flat
|
||||
`{ [id]: translatedMaskedText }` map. Every `⸤I18N…⸥` mask token must survive
|
||||
verbatim; the import step validates this and rejects a unit that drops or
|
||||
reorders one.
|
||||
3. **Import.** `--import` reads the done files, restores each mask token to its
|
||||
original code/URL/placeholder, validates structure, and writes the result
|
||||
through the surface adapters (landing JSON + `.meta.json`, per-locale docs
|
||||
markdown, `openapi.<locale>.yaml`). Each written unit is stamped `machine`
|
||||
with the source hash it was translated against.
|
||||
|
||||
An optional `--engine=api` path exists for self-hosters who bring their own key
|
||||
and want a headless run, but the shipped, CI-safe default is the batch handoff
|
||||
above.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# 1. Export pending batches (all surfaces, all locales).
|
||||
pnpm i18n:translate --export
|
||||
|
||||
# Scope by surface and/or locale (comma lists).
|
||||
pnpm i18n:translate --export --surface=docs --locale=de,fr
|
||||
|
||||
# 2. Translate each .pending.json into a .done.json (Claude Code session or human).
|
||||
|
||||
# 3. Import the done batches back through the adapters.
|
||||
pnpm i18n:translate --import --surface=docs --locale=de,fr
|
||||
|
||||
# Report the change set without exporting or writing anything.
|
||||
pnpm i18n:translate --dry-run
|
||||
|
||||
# Show usage and the currently registered surfaces.
|
||||
pnpm i18n:translate --help
|
||||
|
||||
# Fail if any registered locale is missing a unit or its source hash is stale.
|
||||
pnpm i18n:check
|
||||
```
|
||||
|
||||
If no adapter is registered, both commands are no-ops: `i18n:translate` prints
|
||||
"No adapters registered yet" and `i18n:check` passes.
|
||||
|
||||
## Chunking a large surface (docs)
|
||||
|
||||
The docs surface is by far the biggest: full parity is on the order of thousands
|
||||
of markdown files. A single `.pending.json` can be large, so translate it in
|
||||
parallel:
|
||||
|
||||
- Split the pending file's rows into chunks by a rough character budget (a few
|
||||
thousand chars per chunk keeps each agent's context small and its output
|
||||
reviewable), one Claude Code agent per chunk.
|
||||
- Each agent translates only its slice, preserving every `⸤I18N…⸥` token.
|
||||
- Merge the per-chunk outputs into one `.done.json` for that locale, keyed by
|
||||
`id`, then run `--import`.
|
||||
|
||||
Chunk the shared English source once and reuse the slices across locales so every
|
||||
language sees the same unit boundaries. Because import is idempotent and
|
||||
hash-gated, a partial merge is safe: re-running import after adding more chunks
|
||||
only writes the newly present ids.
|
||||
|
||||
## How gating works
|
||||
|
||||
Each translation unit's English source is hashed (`lib/hash.mjs`, a 12-char
|
||||
sha256 with CRLF normalized to LF). The hash is stored inline next to the
|
||||
translation by the adapter, not in a central manifest. Per unit, on export:
|
||||
|
||||
- Missing translation: include it in the pending batch.
|
||||
- Stored hash matches the current English hash: skip it.
|
||||
- Hash differs and the unit is machine-translated: include it (re-translate).
|
||||
- Hash differs and the unit was human-refined: mark it `stale` for review and
|
||||
keep the human text. The pipeline never overwrites human work.
|
||||
|
||||
## Resumability
|
||||
|
||||
Import checkpoints every 20 units per locale, so an interrupted run resumes from
|
||||
the last completed chunk with no duplicated work: a re-run skips everything whose
|
||||
hash already matches. Exporting again is likewise safe: an already-in-sync unit
|
||||
is not re-exported, so the pending files shrink to only what still needs work.
|
||||
|
||||
## Parity check (local / optional gate)
|
||||
|
||||
`pnpm i18n:check` validates every registered adapter against the current English
|
||||
source and exits non-zero if any locale is missing a unit or carries a stale
|
||||
source hash. It is the build-time mirror of the app's TypeScript key-parity
|
||||
guarantee, and it imports the same `adapters/registry.mjs` the CLI does.
|
||||
|
||||
Run it before committing regenerated translations. It is documented as a local
|
||||
and optional gate rather than wired as an always-on required status check: on
|
||||
this repo, a required context that never reports a conclusion deadlocks
|
||||
docs-only and landing-only PRs (see the header of `.github/workflows/ci.yml`).
|
||||
The first surface plan that needs a hard gate can add a non-required
|
||||
`pull_request` job that runs `pnpm i18n:check`.
|
||||
|
||||
## Deliberately no translation CI job
|
||||
|
||||
There is intentionally no GitHub Actions workflow that runs the translation
|
||||
engine. The engine is a Claude Code batch handoff (export, translate, import),
|
||||
not a headless API call, so there is nothing for a scheduled or dispatched job to
|
||||
run unattended. Translation is done on demand from a local session; the only CI
|
||||
touchpoint is the parity check above. A note for anyone tempted to wire one up
|
||||
later: never pass a list of surfaces or locales through a workflow `inputs` /
|
||||
`args` field and split it in the job. Workflow inputs arrive as a single string,
|
||||
which silently breaks list handling. Hardcode the surface/locale set in the
|
||||
workflow and hard-cap it instead.
|
||||
|
||||
## Accepted decisions
|
||||
|
||||
Two decisions from the design spec, recorded here as revisit-points, not one-way
|
||||
doors:
|
||||
|
||||
1. **Manual, on-demand translation cadence.** We translate by hand (export ->
|
||||
Claude Code session -> import) when content changes warrant it, rather than on
|
||||
every push. This keeps the first-pass effort controlled and avoids an
|
||||
always-on job. Moving to a scheduled or push-triggered cadence later is a
|
||||
process change, not a rewrite.
|
||||
2. **Full docs coverage was chosen.** We translate the entire docs surface into
|
||||
all 20 languages rather than restricting docs to a subset of high-traffic
|
||||
languages. That grows VitePress build time and repo size and adds a pagefind
|
||||
index per locale. We accept this for the first pass and watch Cloudflare Pages
|
||||
build times after docs lands.
|
||||
@@ -0,0 +1,26 @@
|
||||
<!-- scripts/i18n/adapter-contract.md -->
|
||||
# i18n adapter contract
|
||||
|
||||
Each surface (landing UI, landing SEO, docs, API spec) implements one adapter object:
|
||||
|
||||
```js
|
||||
export const adapter = {
|
||||
name: "docs", // unique surface key used on the CLI
|
||||
async extract() { /* return Unit[] */ },
|
||||
async load(locale) { /* return Map<id, StoredEntry> */ },
|
||||
async write(locale, entries) { /* persist Map<id, StoredEntry> */ },
|
||||
};
|
||||
```
|
||||
|
||||
Unit: `{ id: string, sourceText: string, kind: "markdown"|"text"|"html", context?: string }`
|
||||
- `id` must be stable across runs (a file path, a data key). It keys the stored translation.
|
||||
|
||||
StoredEntry (written and read back by the adapter, hashes stored inline per the spec):
|
||||
`{ text: string, sourceHash: string, provenance: "machine"|"human", outputHash: string, stale?: boolean }`
|
||||
|
||||
Rules the adapter owns:
|
||||
- `load` reads whatever inline representation the surface uses (frontmatter for `.md`,
|
||||
`_sourceHash`/`_meta` for JSON) and reconstructs `StoredEntry`.
|
||||
- `write` persists it back in that same representation. `stale: true` entries keep their
|
||||
existing `text` and should surface a review marker in the file (e.g. a frontmatter flag).
|
||||
- Adapters never call the model. They only extract and persist. `core.mjs` owns gating.
|
||||
@@ -0,0 +1,147 @@
|
||||
// scripts/i18n/adapters/api-spec.mjs
|
||||
import { existsSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import yaml from "js-yaml";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
// The real spec lives next to the API source. Tests pass a `dir` override.
|
||||
const DEFAULT_DIR = join(__dirname, "../../../apps/api/src");
|
||||
const HTTP_METHODS = ["get", "post", "put", "patch", "delete", "head", "options"];
|
||||
|
||||
// structuredClone is global in Node 22+, used to deep-copy the parsed spec.
|
||||
const clone = (value) => structuredClone(value);
|
||||
|
||||
/**
|
||||
* Read and parse the English OpenAPI document.
|
||||
* @param {string} dir
|
||||
* @returns {any}
|
||||
*/
|
||||
function loadEnglishSpec(dir) {
|
||||
const raw = readFileSync(join(dir, "openapi.yaml"), "utf8");
|
||||
return yaml.load(raw);
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk a parsed spec and yield [id, text] for every translatable prose field,
|
||||
* in a deterministic order. This is the single source of the id contract used by
|
||||
* extract, load, and write.
|
||||
* @param {any} spec
|
||||
* @returns {Array<[string, string]>}
|
||||
*/
|
||||
export function proseFields(spec) {
|
||||
const out = [];
|
||||
if (typeof spec?.info?.description === "string") {
|
||||
out.push(["info.description", spec.info.description]);
|
||||
}
|
||||
for (const tag of spec?.tags ?? []) {
|
||||
if (tag && typeof tag.name === "string" && typeof tag.description === "string") {
|
||||
out.push([`tags.${tag.name}.description`, tag.description]);
|
||||
}
|
||||
}
|
||||
for (const [path, methods] of Object.entries(spec?.paths ?? {})) {
|
||||
for (const method of HTTP_METHODS) {
|
||||
const op = methods?.[method];
|
||||
if (!op) continue;
|
||||
if (typeof op.summary === "string") {
|
||||
out.push([`paths.${path}.${method}.summary`, op.summary]);
|
||||
}
|
||||
if (typeof op.description === "string") {
|
||||
out.push([`paths.${path}.${method}.description`, op.description]);
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Set a prose field on a cloned spec by the same id `proseFields` produced.
|
||||
* @param {any} spec
|
||||
* @param {string} id
|
||||
* @param {string} value
|
||||
*/
|
||||
export function setProseField(spec, id, value) {
|
||||
if (id === "info.description") {
|
||||
spec.info.description = value;
|
||||
return;
|
||||
}
|
||||
const tagMatch = id.match(/^tags\.(.+)\.description$/);
|
||||
if (tagMatch) {
|
||||
const tag = (spec.tags ?? []).find((t) => t?.name === tagMatch[1]);
|
||||
if (tag) tag.description = value;
|
||||
return;
|
||||
}
|
||||
// paths.<path>.<method>.<field> where <path> may itself contain dots.
|
||||
const pathMatch = id.match(/^paths\.(.+)\.([a-z]+)\.(summary|description)$/);
|
||||
if (pathMatch) {
|
||||
const [, path, method, field] = pathMatch;
|
||||
const op = spec.paths?.[path]?.[method];
|
||||
if (op) op[field] = value;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the api-spec adapter.
|
||||
* @param {{ dir?: string }} [opts]
|
||||
*/
|
||||
export function makeApiSpecAdapter({ dir = DEFAULT_DIR } = {}) {
|
||||
return {
|
||||
name: "api",
|
||||
async extract() {
|
||||
const spec = loadEnglishSpec(dir);
|
||||
return proseFields(spec).map(([id, sourceText]) => ({ id, sourceText, kind: "text" }));
|
||||
},
|
||||
|
||||
async write(locale, entries) {
|
||||
const english = loadEnglishSpec(dir);
|
||||
const localized = clone(english);
|
||||
const stamp = {};
|
||||
|
||||
// Replace only the prose fields that have a translation; anything missing
|
||||
// keeps its English text so the document is always complete and valid.
|
||||
for (const [id] of proseFields(english)) {
|
||||
const entry = entries.get(id);
|
||||
if (!entry) continue;
|
||||
setProseField(localized, id, entry.text);
|
||||
stamp[id] = {
|
||||
sourceHash: entry.sourceHash,
|
||||
provenance: entry.provenance,
|
||||
outputHash: entry.outputHash,
|
||||
...(entry.stale ? { stale: true } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
localized["x-i18n"] = {
|
||||
locale,
|
||||
generator: "scripts/i18n/adapters/api-spec.mjs",
|
||||
entries: stamp,
|
||||
};
|
||||
|
||||
const out = yaml.dump(localized, { lineWidth: -1, noRefs: true });
|
||||
writeFileSync(join(dir, `openapi.${locale}.yaml`), out, "utf8");
|
||||
},
|
||||
|
||||
async load(locale) {
|
||||
const file = join(dir, `openapi.${locale}.yaml`);
|
||||
const result = new Map();
|
||||
if (!existsSync(file)) return result;
|
||||
const spec = yaml.load(readFileSync(file, "utf8"));
|
||||
const stamp = spec?.["x-i18n"]?.entries ?? {};
|
||||
for (const [id, text] of proseFields(spec)) {
|
||||
const meta = stamp[id];
|
||||
if (!meta) continue; // untranslated fallback field, not a stored entry
|
||||
result.set(id, {
|
||||
text,
|
||||
sourceHash: meta.sourceHash,
|
||||
provenance: meta.provenance === "human" ? "human" : "machine",
|
||||
outputHash: meta.outputHash,
|
||||
...(meta.stale ? { stale: true } : {}),
|
||||
});
|
||||
}
|
||||
return result;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const adapter = makeApiSpecAdapter();
|
||||
@@ -0,0 +1,338 @@
|
||||
// scripts/i18n/adapters/docs-md.mjs
|
||||
import { mkdir, readdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { dirname, join, relative, sep } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { localeCodes } from "../lib/shared-i18n.mjs";
|
||||
import { slugify } from "../lib/slugify.mjs";
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
// scripts/i18n/adapters -> repo root -> apps/docs
|
||||
const DEFAULT_ROOT = join(HERE, "..", "..", "..", "apps", "docs");
|
||||
|
||||
const LOCALE_DIRS = new Set(localeCodes().filter((c) => c !== "en"));
|
||||
const SKIP_DIRS = new Set(["node_modules", ".vitepress", "public", ...LOCALE_DIRS]);
|
||||
|
||||
// Docs-dialect masking tokens (distinct delimiters from the shared mask lib so
|
||||
// the two layers never collide).
|
||||
const DTOKEN = (i) => `⟦DOCS${i}⟧`;
|
||||
const DTOKEN_RE = /⟦DOCS\d+⟧/g;
|
||||
|
||||
/**
|
||||
* List every root markdown file (relative POSIX path), excluding locale subtrees
|
||||
* and non-content dirs.
|
||||
* @param {string} root
|
||||
* @returns {Promise<string[]>}
|
||||
*/
|
||||
async function listRootMarkdown(root) {
|
||||
const out = [];
|
||||
async function walk(dir) {
|
||||
const entries = await readdir(dir, { withFileTypes: true });
|
||||
for (const ent of entries) {
|
||||
const abs = join(dir, ent.name);
|
||||
if (ent.isDirectory()) {
|
||||
if (SKIP_DIRS.has(ent.name)) continue;
|
||||
await walk(abs);
|
||||
} else if (ent.name.endsWith(".md")) {
|
||||
out.push(relative(root, abs).split(sep).join("/"));
|
||||
}
|
||||
}
|
||||
}
|
||||
await walk(root);
|
||||
return out.sort();
|
||||
}
|
||||
|
||||
/**
|
||||
* Split frontmatter from body. Returns { fm, body } where fm is the raw
|
||||
* frontmatter block WITHOUT the fences (or null), and body is everything after.
|
||||
* @param {string} text
|
||||
* @returns {{ fm: string|null, body: string }}
|
||||
*/
|
||||
function splitFrontmatter(text) {
|
||||
const m = text.match(/^---\n([\s\S]*?)\n---\n?/);
|
||||
if (!m) return { fm: null, body: text };
|
||||
return { fm: m[1], body: text.slice(m[0].length) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a single scalar value from a raw frontmatter block, e.g. key: value.
|
||||
* @param {string|null} fm
|
||||
* @param {string} key
|
||||
* @returns {string|undefined}
|
||||
*/
|
||||
function fmGet(fm, key) {
|
||||
if (!fm) return undefined;
|
||||
const m = fm.match(new RegExp(`^${key}:\\s*(.*)$`, "m"));
|
||||
if (!m) return undefined;
|
||||
return m[1].trim().replace(/^["']|["']$/g, "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Split a body into fenced-code segments and prose segments so heading/anchor and
|
||||
* docs-mask logic never touch lines inside ``` fences.
|
||||
* @param {string} body
|
||||
* @returns {Array<{ code: boolean, text: string }>}
|
||||
*/
|
||||
function segmentFences(body) {
|
||||
const parts = [];
|
||||
const re = /(^|\n)(```|~~~)[\s\S]*?\n\2/g;
|
||||
let last = 0;
|
||||
let m;
|
||||
while ((m = re.exec(body)) !== null) {
|
||||
if (m.index > last) parts.push({ code: false, text: body.slice(last, m.index) });
|
||||
parts.push({ code: true, text: m[0] });
|
||||
last = m.index + m[0].length;
|
||||
}
|
||||
if (last < body.length) parts.push({ code: false, text: body.slice(last) });
|
||||
return parts;
|
||||
}
|
||||
|
||||
/**
|
||||
* Inject `{#slug}` into ATX headings that lack an explicit anchor. Idempotent.
|
||||
* Only runs over non-code segments.
|
||||
* @param {string} body
|
||||
* @returns {string}
|
||||
*/
|
||||
function injectAnchors(body) {
|
||||
// De-duplicate slugs exactly like markdown-it-anchor: a base slug already used
|
||||
// in the file gets "-1", "-2", ... appended (start index 1). Explicit {#id}
|
||||
// anchors already in the source claim their slot too, so injected slugs never
|
||||
// collide with them. Seen-set spans every prose segment, matching VitePress.
|
||||
const seen = new Set();
|
||||
const unique = (base) => {
|
||||
let slug = base;
|
||||
let i = 1;
|
||||
while (seen.has(slug)) {
|
||||
slug = `${base}-${i}`;
|
||||
i += 1;
|
||||
}
|
||||
seen.add(slug);
|
||||
return slug;
|
||||
};
|
||||
return segmentFences(body)
|
||||
.map((seg) => {
|
||||
if (seg.code) return seg.text;
|
||||
return seg.text.replace(/^(#{1,6})[ \t]+(.+?)[ \t]*$/gm, (line, hashes, title) => {
|
||||
const explicit = title.match(/\{#([^}]+)\}\s*$/);
|
||||
if (explicit) {
|
||||
seen.add(explicit[1]); // pre-existing anchor claims its slug
|
||||
return line;
|
||||
}
|
||||
const clean = title.replace(/[ \t]+#*$/, ""); // drop optional closing ATX hashes
|
||||
return `${hashes} ${clean} {#${unique(slugify(clean))}}`;
|
||||
});
|
||||
})
|
||||
.join("");
|
||||
}
|
||||
|
||||
/**
|
||||
* Docs-dialect pre-mask over non-code segments: hide `:::` container markers
|
||||
* (keep the title label), `[[toc]]`, and explicit `{#anchor}` slugs so the model
|
||||
* never rewrites them. Returns masked text plus the token table.
|
||||
* @param {string} body
|
||||
* @returns {{ masked: string, tokens: string[] }}
|
||||
*/
|
||||
function maskDocs(body) {
|
||||
const tokens = [];
|
||||
const push = (v) => {
|
||||
tokens.push(v);
|
||||
return DTOKEN(tokens.length - 1);
|
||||
};
|
||||
const masked = segmentFences(body)
|
||||
.map((seg) => {
|
||||
if (seg.code) return seg.text;
|
||||
let out = seg.text;
|
||||
// ::: type (mask the "::: type " marker, leave the title text after it)
|
||||
out = out.replace(/^(:{3,})[ \t]*([a-zA-Z-]+)[ \t]*/gm, (_m, colons, type) =>
|
||||
push(`${colons} ${type} `),
|
||||
);
|
||||
// bare closing :::
|
||||
out = out.replace(/^:{3,}[ \t]*$/gm, (m) => push(m));
|
||||
// [[toc]]
|
||||
out = out.replace(/\[\[toc\]\]/gi, (m) => push(m));
|
||||
// explicit {#anchor}
|
||||
out = out.replace(/\{#[^}\n]+\}/g, (m) => push(m));
|
||||
return out;
|
||||
})
|
||||
.join("");
|
||||
return { masked, tokens };
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore docs-dialect tokens.
|
||||
* @param {string} masked
|
||||
* @param {string[]} tokens
|
||||
* @returns {string}
|
||||
*/
|
||||
function restoreDocs(masked, tokens) {
|
||||
return String(masked).replace(DTOKEN_RE, (t) => {
|
||||
const m = t.match(/DOCS(\d+)/);
|
||||
const i = m ? Number(m[1]) : Number.NaN;
|
||||
return tokens[i] ?? t;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Rewrite internal absolute links (/guide/x, /tools/y, /api/z) to /<locale>/...
|
||||
* Leaves external URLs, already-prefixed links, and root asset paths alone.
|
||||
* @param {string} body
|
||||
* @param {string} locale
|
||||
* @returns {string}
|
||||
*/
|
||||
function rewriteLinks(body, locale) {
|
||||
return body.replace(/(\]\()(\/[^)\s]*)(\))/g, (_m, open, url, close) => {
|
||||
if (url.startsWith(`/${locale}/`)) return `${open}${url}${close}`;
|
||||
// Do not prefix asset paths VitePress serves from root public/.
|
||||
if (/^\/(fonts|screenshots|logo|favicon|apple-touch|og-|llms)/.test(url)) {
|
||||
return `${open}${url}${close}`;
|
||||
}
|
||||
return `${open}/${locale}${url}${close}`;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert or replace scalar frontmatter keys, preserving existing ones.
|
||||
* @param {string} text
|
||||
* @param {Record<string,string>} fields
|
||||
* @returns {string}
|
||||
*/
|
||||
function upsertFrontmatter(text, fields) {
|
||||
const { fm, body } = splitFrontmatter(text);
|
||||
const lines = fm != null ? fm.split("\n") : [];
|
||||
const seen = new Set();
|
||||
const next = lines.map((line) => {
|
||||
const m = line.match(/^([A-Za-z0-9_]+):/);
|
||||
if (m && fields[m[1]] !== undefined) {
|
||||
seen.add(m[1]);
|
||||
return `${m[1]}: ${fields[m[1]]}`;
|
||||
}
|
||||
return line;
|
||||
});
|
||||
for (const [k, v] of Object.entries(fields)) {
|
||||
if (!seen.has(k)) next.push(`${k}: ${v}`);
|
||||
}
|
||||
return `---\n${next.join("\n")}\n---\n${body}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Double-quote bare `description`/`title` frontmatter values so a translated
|
||||
* value containing a colon, `#`, or other YAML-significant character does not
|
||||
* break frontmatter parsing (VitePress loads it as YAML).
|
||||
* @param {string} text
|
||||
* @returns {string}
|
||||
*/
|
||||
export function quoteFrontmatterScalars(text) {
|
||||
const { fm, body } = splitFrontmatter(text);
|
||||
if (fm == null) return text;
|
||||
const next = fm.split("\n").map((line) => {
|
||||
const m = line.match(/^(description|title):[ \t]*(.*)$/);
|
||||
if (!m) return line;
|
||||
const val = m[2];
|
||||
// Leave empty, already-quoted, or block-scalar values alone.
|
||||
if (val === "" || /^["'|>]/.test(val)) return line;
|
||||
const escaped = val.replace(/\\/g, "\\\\").replace(/"/g, '\\"');
|
||||
return `${m[1]}: "${escaped}"`;
|
||||
});
|
||||
return `---\n${next.join("\n")}\n---\n${body}`;
|
||||
}
|
||||
|
||||
/** Recursively list markdown under a locale dir, relative + POSIX. */
|
||||
async function listLocaleMarkdown(dir) {
|
||||
const out = [];
|
||||
async function walk(d) {
|
||||
const entries = await readdir(d, { withFileTypes: true });
|
||||
for (const ent of entries) {
|
||||
const abs = join(d, ent.name);
|
||||
if (ent.isDirectory()) await walk(abs);
|
||||
else if (ent.name.endsWith(".md")) out.push(relative(dir, abs).split(sep).join("/"));
|
||||
}
|
||||
}
|
||||
await walk(dir);
|
||||
return out.sort();
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the docs adapter. `root` defaults to apps/docs; tests override it.
|
||||
* @param {{ root?: string }} [opts]
|
||||
*/
|
||||
export function createDocsAdapter({ root = DEFAULT_ROOT } = {}) {
|
||||
// id -> docs token table, populated by extract() and consumed by write() in the
|
||||
// same run so the shared translator never needs to know about docs tokens.
|
||||
const tokenTables = new Map();
|
||||
|
||||
return {
|
||||
name: "docs",
|
||||
|
||||
async extract() {
|
||||
const files = await listRootMarkdown(root);
|
||||
const units = [];
|
||||
for (const id of files) {
|
||||
const abs = join(root, id);
|
||||
const original = await readFile(abs, "utf8");
|
||||
const { fm, body } = splitFrontmatter(original);
|
||||
const anchored = injectAnchors(body);
|
||||
const rebuilt = fm != null ? `---\n${fm}\n---\n${anchored}` : anchored;
|
||||
// Persist anchored English source in place (idempotent) so the running
|
||||
// site and every locale share the same stable slugs.
|
||||
if (rebuilt !== original) await writeFile(abs, rebuilt, "utf8");
|
||||
const { masked, tokens } = maskDocs(anchored);
|
||||
tokenTables.set(id, tokens);
|
||||
const sourceText = fm != null ? `---\n${fm}\n---\n${masked}` : masked;
|
||||
units.push({ id, sourceText, kind: "markdown" });
|
||||
}
|
||||
return units;
|
||||
},
|
||||
|
||||
async load(locale) {
|
||||
const dir = join(root, locale);
|
||||
const map = new Map();
|
||||
let files = [];
|
||||
try {
|
||||
files = await listLocaleMarkdown(dir);
|
||||
} catch {
|
||||
return map; // no locale subtree yet
|
||||
}
|
||||
for (const rel of files) {
|
||||
const text = await readFile(join(dir, rel), "utf8");
|
||||
const { fm } = splitFrontmatter(text);
|
||||
map.set(rel, {
|
||||
text,
|
||||
sourceHash: fmGet(fm, "i18n_source_hash") ?? "",
|
||||
provenance: fmGet(fm, "i18n_provenance") === "human" ? "human" : "machine",
|
||||
outputHash: fmGet(fm, "i18n_output_hash") ?? "",
|
||||
stale: fmGet(fm, "i18n_stale") === "true",
|
||||
});
|
||||
}
|
||||
return map;
|
||||
},
|
||||
|
||||
async write(locale, entries) {
|
||||
for (const [id, entry] of entries) {
|
||||
const abs = join(root, locale, id);
|
||||
await mkdir(dirname(abs), { recursive: true });
|
||||
const table = tokenTables.get(id) ?? [];
|
||||
const undone = restoreDocs(entry.text, table);
|
||||
const linked = quoteFrontmatterScalars(rewriteLinks(undone, locale));
|
||||
const withMeta = upsertFrontmatter(linked, {
|
||||
i18n_source_hash: entry.sourceHash,
|
||||
i18n_provenance: entry.provenance,
|
||||
i18n_output_hash: entry.outputHash,
|
||||
...(entry.stale ? { i18n_stale: "true" } : {}),
|
||||
});
|
||||
await writeFile(abs, withMeta, "utf8");
|
||||
}
|
||||
},
|
||||
|
||||
// Adapter extra (not part of the 3-method contract): write the English body
|
||||
// under the locale path, flagged so a real translation replaces it later.
|
||||
async writeFallback(locale, id) {
|
||||
const abs = join(root, locale, id);
|
||||
const original = await readFile(join(root, id), "utf8");
|
||||
await mkdir(dirname(abs), { recursive: true });
|
||||
const linked = rewriteLinks(original, locale);
|
||||
const flagged = upsertFrontmatter(linked, { i18n_fallback: "true" });
|
||||
await writeFile(abs, flagged, "utf8");
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const adapter = createDocsAdapter();
|
||||
@@ -0,0 +1,114 @@
|
||||
// scripts/i18n/adapters/landing-seo.mjs
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const DEFAULT_OUT_DIR = join(__dirname, "../../../apps/landing/src/data/i18n");
|
||||
|
||||
// Only the alternatives pages are localized. Tool-detail pages are English-only
|
||||
// (thin-content SEO risk), so their tool-seo prose is never rendered in a locale
|
||||
// and is intentionally not extracted here.
|
||||
const ALT_STRING_FIELDS = ["pageTitle", "h1", "metaDescription", "intro", "breadth"];
|
||||
|
||||
function pushIfString(units, id, value) {
|
||||
if (typeof value === "string" && value.trim().length > 0) {
|
||||
units.push({ id, sourceText: value, kind: "text" });
|
||||
}
|
||||
}
|
||||
|
||||
function extractAlternatives(units, alternatives) {
|
||||
for (const alt of alternatives) {
|
||||
const base = `alt:${alt.slug}`;
|
||||
for (const field of ALT_STRING_FIELDS) pushIfString(units, `${base}:${field}`, alt[field]);
|
||||
const rows = Array.isArray(alt.rows) ? alt.rows : [];
|
||||
rows.forEach((row, i) => {
|
||||
pushIfString(units, `${base}:rows.${i}.feature`, row.feature);
|
||||
pushIfString(units, `${base}:rows.${i}.snapotter`, row.snapotter);
|
||||
pushIfString(units, `${base}:rows.${i}.competitor`, row.competitor);
|
||||
});
|
||||
const faqs = Array.isArray(alt.faqs) ? alt.faqs : [];
|
||||
faqs.forEach((faq, i) => {
|
||||
pushIfString(units, `${base}:faqs.${i}.q`, faq.q);
|
||||
pushIfString(units, `${base}:faqs.${i}.a`, faq.a);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async function readJson(path) {
|
||||
try {
|
||||
return JSON.parse(await readFile(path, "utf8"));
|
||||
} catch (err) {
|
||||
if (err && err.code === "ENOENT") return null;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
function altFileFor(dir, locale) {
|
||||
return join(dir, `alternatives.${locale}.json`);
|
||||
}
|
||||
|
||||
function toStored(record) {
|
||||
const out = new Map();
|
||||
for (const [id, e] of Object.entries(record ?? {})) {
|
||||
out.set(id, {
|
||||
text: e.text ?? "",
|
||||
sourceHash: e._sourceHash ?? "",
|
||||
provenance: e.provenance ?? "machine",
|
||||
outputHash: e.outputHash ?? "",
|
||||
stale: Boolean(e.stale),
|
||||
});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function toRecord(entries) {
|
||||
const record = {};
|
||||
for (const id of [...entries.keys()].sort()) {
|
||||
const e = entries.get(id);
|
||||
record[id] = {
|
||||
text: e.text,
|
||||
_sourceHash: e.sourceHash,
|
||||
provenance: e.provenance,
|
||||
outputHash: e.outputHash,
|
||||
stale: Boolean(e.stale),
|
||||
};
|
||||
}
|
||||
return record;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the landing SEO data adapter (alternatives pages only).
|
||||
* @param {{
|
||||
* dir?: string,
|
||||
* alternatives?: any[],
|
||||
* }} [opts]
|
||||
*/
|
||||
export function makeLandingSeoAdapter(opts = {}) {
|
||||
const dir = opts.dir ?? DEFAULT_OUT_DIR;
|
||||
return {
|
||||
name: "landing-seo",
|
||||
|
||||
async extract() {
|
||||
const alternatives =
|
||||
opts.alternatives ??
|
||||
(await import("../../../apps/landing/src/data/alternatives.ts")).ALTERNATIVES;
|
||||
const units = [];
|
||||
extractAlternatives(units, alternatives);
|
||||
return units;
|
||||
},
|
||||
|
||||
async load(locale) {
|
||||
const alt = await readJson(altFileFor(dir, locale));
|
||||
return toStored(alt);
|
||||
},
|
||||
|
||||
async write(locale, entries) {
|
||||
const altRecord = toRecord(entries);
|
||||
await mkdir(dir, { recursive: true });
|
||||
await writeFile(altFileFor(dir, locale), `${JSON.stringify(altRecord, null, 2)}\n`);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const adapter = makeLandingSeoAdapter();
|
||||
@@ -0,0 +1,75 @@
|
||||
// scripts/i18n/adapters/landing-ui.mjs
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const DEFAULT_DIR = join(__dirname, "../../../apps/landing/src/i18n");
|
||||
|
||||
async function readJson(path) {
|
||||
try {
|
||||
return JSON.parse(await readFile(path, "utf8"));
|
||||
} catch (err) {
|
||||
if (err && err.code === "ENOENT") return null;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the landing UI catalog adapter.
|
||||
* Runtime catalog: `<locale>.json` (flat { key: text }).
|
||||
* Sidecar metadata: `<locale>.meta.json` ({ key: { sourceHash, provenance, outputHash, stale } }).
|
||||
* @param {{ dir?: string }} [opts]
|
||||
*/
|
||||
export function makeLandingUiAdapter({ dir = DEFAULT_DIR } = {}) {
|
||||
const enPath = join(dir, "en.json");
|
||||
return {
|
||||
name: "landing-ui",
|
||||
|
||||
async extract() {
|
||||
const en = (await readJson(enPath)) ?? {};
|
||||
return Object.entries(en).map(([id, sourceText]) => ({
|
||||
id,
|
||||
sourceText: String(sourceText),
|
||||
kind: "text",
|
||||
}));
|
||||
},
|
||||
|
||||
async load(locale) {
|
||||
const catalog = (await readJson(join(dir, `${locale}.json`))) ?? {};
|
||||
const meta = (await readJson(join(dir, `${locale}.meta.json`))) ?? {};
|
||||
const out = new Map();
|
||||
for (const [id, text] of Object.entries(catalog)) {
|
||||
const m = meta[id] ?? {};
|
||||
out.set(id, {
|
||||
text: String(text),
|
||||
sourceHash: m.sourceHash ?? "",
|
||||
provenance: m.provenance ?? "machine",
|
||||
outputHash: m.outputHash ?? "",
|
||||
stale: Boolean(m.stale),
|
||||
});
|
||||
}
|
||||
return out;
|
||||
},
|
||||
|
||||
async write(locale, entries) {
|
||||
const catalog = {};
|
||||
const meta = {};
|
||||
// Stable key order for reviewable diffs.
|
||||
for (const id of [...entries.keys()].sort()) {
|
||||
const e = entries.get(id);
|
||||
catalog[id] = e.text;
|
||||
meta[id] = {
|
||||
sourceHash: e.sourceHash,
|
||||
provenance: e.provenance,
|
||||
outputHash: e.outputHash,
|
||||
stale: Boolean(e.stale),
|
||||
};
|
||||
}
|
||||
await writeFile(join(dir, `${locale}.json`), `${JSON.stringify(catalog, null, 2)}\n`);
|
||||
await writeFile(join(dir, `${locale}.meta.json`), `${JSON.stringify(meta, null, 2)}\n`);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const adapter = makeLandingUiAdapter();
|
||||
@@ -0,0 +1,23 @@
|
||||
// scripts/i18n/adapters/registry.mjs
|
||||
// Single source of truth for surface adapters. Both translate.mjs and
|
||||
// check-parity.mjs import this so the surface list is defined once.
|
||||
// Surface plans (02-04) uncomment/add their entry as each adapter lands.
|
||||
export const ADAPTERS = {
|
||||
"landing-ui": () => import("./landing-ui.mjs"),
|
||||
"landing-seo": () => import("./landing-seo.mjs"),
|
||||
docs: () => import("./docs-md.mjs"),
|
||||
api: () => import("./api-spec.mjs"),
|
||||
};
|
||||
|
||||
/**
|
||||
* @param {string} spec "all" or a comma list of surface keys
|
||||
* @returns {string[]} known surface keys
|
||||
*/
|
||||
export function resolveSurfaces(spec) {
|
||||
const keys = Object.keys(ADAPTERS);
|
||||
if (spec === "all") return keys;
|
||||
return spec
|
||||
.split(",")
|
||||
.map((s) => s.trim())
|
||||
.filter((k) => keys.includes(k));
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
// scripts/i18n/check-parity.mjs
|
||||
// Parity/staleness gate for the web-surfaces translation pipeline. For every
|
||||
// registered adapter and every non-en locale, it replays extract() + load() and
|
||||
// fails if any unit is missing a translation or its stored sourceHash no longer
|
||||
// matches the current English source (stale). This is the build-time mirror of
|
||||
// the app's TypeScript key-parity guarantee.
|
||||
//
|
||||
// It imports the SAME adapter registry the CLI uses (adapters/registry.mjs), so
|
||||
// the surface list is defined once and never drifts from translate.mjs.
|
||||
import { ADAPTERS } from "./adapters/registry.mjs";
|
||||
import { hash } from "./lib/hash.mjs";
|
||||
import { localeCodes } from "./lib/shared-i18n.mjs";
|
||||
|
||||
/**
|
||||
* Validate one adapter's translations against the current English source.
|
||||
* A problem is: a locale missing a unit, or a stored sourceHash that no longer
|
||||
* matches the current English source hash (stale). Human-refined units still
|
||||
* count as stale here: staleness means "needs review", which is exactly what a
|
||||
* parity check should flag. `en` is always the source and never a target.
|
||||
*
|
||||
* @param {{ name: string, extract: () => Promise<any[]>, load: (l: string) => Promise<Map<string, any>> }} adapter
|
||||
* @param {string[]} locales
|
||||
* @returns {Promise<{ ok: boolean, problems: string[] }>}
|
||||
*/
|
||||
export async function checkAdapter(adapter, locales) {
|
||||
const units = await adapter.extract();
|
||||
const sourceHashes = new Map(units.map((u) => [u.id, hash(u.sourceText)]));
|
||||
const problems = [];
|
||||
|
||||
for (const locale of locales) {
|
||||
if (locale === "en") continue;
|
||||
const stored = await adapter.load(locale);
|
||||
for (const [id, srcHash] of sourceHashes) {
|
||||
const entry = stored.get(id);
|
||||
if (!entry) {
|
||||
problems.push(`[${adapter.name}] ${locale}: missing translation for unit "${id}"`);
|
||||
continue;
|
||||
}
|
||||
if (entry.sourceHash !== srcHash) {
|
||||
problems.push(
|
||||
`[${adapter.name}] ${locale}: stale translation for unit "${id}" ` +
|
||||
`(stored ${entry.sourceHash || "<none>"}, source ${srcHash})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { ok: problems.length === 0, problems };
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the parity check across every registered adapter for every non-en locale.
|
||||
* @param {{ log?: (msg: string) => void }} [opts]
|
||||
* @returns {Promise<{ ok: boolean, problems: string[] }>}
|
||||
*/
|
||||
export async function runParityCheck({ log = () => {} } = {}) {
|
||||
const surfaces = Object.keys(ADAPTERS);
|
||||
if (surfaces.length === 0) {
|
||||
log("No adapters registered yet. Parity check is a no-op until a surface adapter lands.");
|
||||
return { ok: true, problems: [] };
|
||||
}
|
||||
|
||||
const locales = localeCodes().filter((c) => c !== "en");
|
||||
const problems = [];
|
||||
for (const surface of surfaces) {
|
||||
const mod = await ADAPTERS[surface]();
|
||||
const report = await checkAdapter(mod.adapter, locales);
|
||||
problems.push(...report.problems);
|
||||
log(`[${surface}] ${report.ok ? "OK" : `${report.problems.length} problem(s)`}`);
|
||||
}
|
||||
return { ok: problems.length === 0, problems };
|
||||
}
|
||||
|
||||
// Only run the CLI body when invoked directly, not when imported by tests.
|
||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||
runParityCheck({ log: (m) => process.stdout.write(`${m}\n`) })
|
||||
.then((report) => {
|
||||
if (!report.ok) {
|
||||
process.stderr.write("\nTranslation parity check FAILED:\n");
|
||||
for (const p of report.problems) process.stderr.write(` - ${p}\n`);
|
||||
process.stderr.write(
|
||||
"\nExport the affected surface/locale (pnpm i18n:translate --export), translate the\n" +
|
||||
"pending batch, then import it (--import) and commit the regenerated files.\n",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
process.stdout.write("Translation parity check passed.\n");
|
||||
})
|
||||
.catch((err) => {
|
||||
console.error(err);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
// scripts/i18n/core.mjs
|
||||
import { hash } from "./lib/hash.mjs";
|
||||
import { validate } from "./lib/validate.mjs";
|
||||
|
||||
const CHUNK = 20; // units per write, for resumability
|
||||
|
||||
/**
|
||||
* @typedef {Object} StoredEntry
|
||||
* @property {string} text
|
||||
* @property {string} sourceHash
|
||||
* @property {"machine"|"human"} provenance
|
||||
* @property {string} outputHash hash of the text the pipeline last wrote
|
||||
* @property {boolean} [stale] source moved under a human translation
|
||||
*/
|
||||
|
||||
/**
|
||||
* Classify units for a locale against stored translations.
|
||||
* @param {any[]} units
|
||||
* @param {Map<string, StoredEntry>} stored
|
||||
* @returns {{ pending: Array<{unit: any, srcHash: string}>, merged: Map<string, StoredEntry>, stats: { skipped: number, stale: number } }}
|
||||
*/
|
||||
export function collectPending(units, stored) {
|
||||
const merged = new Map(stored);
|
||||
const pending = [];
|
||||
const stats = { skipped: 0, stale: 0 };
|
||||
for (const unit of units) {
|
||||
const srcHash = hash(unit.sourceText);
|
||||
const prev = stored.get(unit.id);
|
||||
if (!prev) {
|
||||
pending.push({ unit, srcHash });
|
||||
continue;
|
||||
}
|
||||
const provenance = hash(prev.text) === prev.outputHash ? prev.provenance : "human";
|
||||
if (prev.sourceHash === srcHash) {
|
||||
merged.set(unit.id, { ...prev, provenance, stale: false });
|
||||
stats.skipped++;
|
||||
continue;
|
||||
}
|
||||
if (provenance === "human") {
|
||||
merged.set(unit.id, { ...prev, provenance, stale: true });
|
||||
stats.stale++;
|
||||
continue;
|
||||
}
|
||||
pending.push({ unit, srcHash });
|
||||
}
|
||||
return { pending, merged, stats };
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate every unit for every locale, gating on source hashes.
|
||||
* @param {{
|
||||
* adapter: { name: string, extract: () => Promise<any[]>, load: (l: string) => Promise<Map<string, StoredEntry>>, write: (l: string, e: Map<string, StoredEntry>) => Promise<void> },
|
||||
* locales: string[],
|
||||
* translate: (units: any[], locale: string) => Promise<Map<string, string>>,
|
||||
* log?: (msg: string) => void,
|
||||
* }} args
|
||||
* @returns {Promise<Record<string, { translated: number, skipped: number, stale: number, failed: number }>>}
|
||||
*/
|
||||
export async function runTranslation({ adapter, locales, translate, log = () => {} }) {
|
||||
const units = await adapter.extract();
|
||||
const summary = {};
|
||||
|
||||
for (const locale of locales) {
|
||||
if (locale === "en") continue; // English is the source
|
||||
const stored = await adapter.load(locale);
|
||||
const { pending: todo, merged, stats: base } = collectPending(units, stored);
|
||||
const stats = { translated: 0, skipped: base.skipped, stale: base.stale, failed: 0 };
|
||||
|
||||
for (let i = 0; i < todo.length; i += CHUNK) {
|
||||
const chunk = todo.slice(i, i + CHUNK);
|
||||
const result = await translate(
|
||||
chunk.map((t) => t.unit),
|
||||
locale,
|
||||
);
|
||||
for (const { unit, srcHash } of chunk) {
|
||||
const text = result.get(unit.id);
|
||||
const check =
|
||||
text == null ? { ok: false, errors: ["no output"] } : validate(unit.sourceText, text);
|
||||
if (!check.ok) {
|
||||
stats.failed++;
|
||||
log(`[${locale}] FAIL ${unit.id}: ${check.errors.join("; ")}`);
|
||||
continue;
|
||||
}
|
||||
merged.set(unit.id, {
|
||||
text,
|
||||
sourceHash: srcHash,
|
||||
provenance: "machine",
|
||||
outputHash: hash(text),
|
||||
stale: false,
|
||||
});
|
||||
stats.translated++;
|
||||
}
|
||||
await adapter.write(locale, merged); // checkpoint each chunk -> resumable
|
||||
log(`[${locale}] ${Math.min(i + CHUNK, todo.length)}/${todo.length}`);
|
||||
}
|
||||
|
||||
await adapter.write(locale, merged);
|
||||
summary[locale] = stats;
|
||||
}
|
||||
|
||||
return summary;
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
// scripts/i18n/lib/batch.mjs
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { mask, restore } from "./mask.mjs";
|
||||
|
||||
function pendingPath(dir, surface, locale) {
|
||||
return join(dir, `${surface}.${locale}.pending.json`);
|
||||
}
|
||||
function donePath(dir, surface, locale) {
|
||||
return join(dir, `${surface}.${locale}.done.json`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Write the masked source of each pending unit for subagents to translate.
|
||||
* @param {string} dir
|
||||
* @param {string} surface
|
||||
* @param {string} locale
|
||||
* @param {Array<{unit: {id: string, sourceText: string}}>} pending
|
||||
* @returns {string} the file path written
|
||||
*/
|
||||
export function writePending(dir, surface, locale, pending) {
|
||||
const rows = pending.map(({ unit }) => ({ id: unit.id, masked: mask(unit.sourceText).masked }));
|
||||
const path = pendingPath(dir, surface, locale);
|
||||
writeFileSync(path, `${JSON.stringify(rows, null, 2)}\n`);
|
||||
return path;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a subagent-produced done file: { [id]: translatedMaskedText }.
|
||||
* @returns {Map<string, string>}
|
||||
*/
|
||||
export function readDone(dir, surface, locale) {
|
||||
const obj = JSON.parse(readFileSync(donePath(dir, surface, locale), "utf8"));
|
||||
return new Map(Object.entries(obj));
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a translate(units, locale) that reads translations from a done map,
|
||||
* re-masking each unit's source to recover its tokens and restoring them.
|
||||
* Returns null for any id absent from the done map (core records it as failed).
|
||||
* @param {Map<string, string>} doneMap
|
||||
*/
|
||||
export function batchResultTranslator(doneMap) {
|
||||
return async function translate(units, _locale) {
|
||||
const out = new Map();
|
||||
for (const unit of units) {
|
||||
const raw = doneMap.get(unit.id);
|
||||
if (raw == null) {
|
||||
out.set(unit.id, null);
|
||||
continue;
|
||||
}
|
||||
const { tokens } = mask(unit.sourceText);
|
||||
out.set(unit.id, restore(raw, tokens));
|
||||
}
|
||||
return out;
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
// scripts/i18n/lib/claude.mjs
|
||||
import { buildSystemPrompt } from "./glossary.mjs";
|
||||
import { mask, restore } from "./mask.mjs";
|
||||
|
||||
const API_URL = "https://api.anthropic.com/v1/messages";
|
||||
const MODEL = process.env.I18N_MODEL || "claude-sonnet-5"; // set I18N_MODEL=claude-opus-4-8 for max quality
|
||||
const MAX_RETRIES = 5;
|
||||
|
||||
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
/**
|
||||
* Default network sender: one Messages API call, returns the assistant text.
|
||||
* @param {{ system: string, text: string }} args
|
||||
* @returns {Promise<string>}
|
||||
*/
|
||||
async function defaultSend({ system, text }) {
|
||||
const key = process.env.ANTHROPIC_API_KEY;
|
||||
if (!key) throw new Error("ANTHROPIC_API_KEY is not set");
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
const res = await fetch(API_URL, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"content-type": "application/json",
|
||||
"x-api-key": key,
|
||||
"anthropic-version": "2023-06-01",
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model: MODEL,
|
||||
max_tokens: 8192,
|
||||
system,
|
||||
messages: [{ role: "user", content: text }],
|
||||
}),
|
||||
});
|
||||
if (res.status === 429 || res.status >= 500) {
|
||||
if (attempt >= MAX_RETRIES) throw new Error(`Messages API failed: ${res.status}`);
|
||||
await sleep(1000 * 2 ** attempt);
|
||||
continue;
|
||||
}
|
||||
if (!res.ok) throw new Error(`Messages API failed: ${res.status} ${await res.text()}`);
|
||||
const json = await res.json();
|
||||
return (json.content || []).map((c) => c.text || "").join("");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a translate function: (units, locale) => Promise<Map<id, translatedText>>.
|
||||
* `send` is injectable for tests. Each unit is masked before sending and restored after.
|
||||
* @param {{ send?: (args: { system: string, text: string }) => Promise<string> }} [opts]
|
||||
*/
|
||||
export function makeTranslator({ send = defaultSend } = {}) {
|
||||
return async function translate(units, locale) {
|
||||
const system = buildSystemPrompt(locale);
|
||||
const out = new Map();
|
||||
for (const unit of units) {
|
||||
const { masked, tokens } = mask(unit.sourceText);
|
||||
const raw = await send({ system, text: masked });
|
||||
out.set(unit.id, restore(raw, tokens));
|
||||
}
|
||||
return out;
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
// scripts/i18n/lib/glossary.mjs
|
||||
import { SUPPORTED_LOCALES } from "../../../packages/shared/src/i18n/index.ts";
|
||||
|
||||
/** Terms that must stay verbatim in every language. */
|
||||
export const DO_NOT_TRANSLATE = [
|
||||
"SnapOtter",
|
||||
"API",
|
||||
"REST",
|
||||
"JSON",
|
||||
"YAML",
|
||||
"URL",
|
||||
"HTTP",
|
||||
"JPEG",
|
||||
"PNG",
|
||||
"WebP",
|
||||
"AVIF",
|
||||
"HEIC",
|
||||
"GIF",
|
||||
"SVG",
|
||||
"EXIF",
|
||||
"PDF",
|
||||
"FFmpeg",
|
||||
"Docker",
|
||||
"Compose",
|
||||
"Postgres",
|
||||
"Redis",
|
||||
"OIDC",
|
||||
"SSO",
|
||||
"SAML",
|
||||
"SCIM",
|
||||
];
|
||||
|
||||
/**
|
||||
* @param {string} locale
|
||||
* @returns {string} the English display name of the locale
|
||||
*/
|
||||
function localeName(locale) {
|
||||
const entry = SUPPORTED_LOCALES.find((l) => l.code === locale);
|
||||
if (!entry) throw new Error(`Unknown locale: ${locale}`);
|
||||
return entry.name;
|
||||
}
|
||||
|
||||
/**
|
||||
* System prompt for translating one batch into `locale`.
|
||||
* @param {string} locale
|
||||
* @returns {string}
|
||||
*/
|
||||
export function buildSystemPrompt(locale) {
|
||||
const name = localeName(locale);
|
||||
return [
|
||||
`You are a professional software-localization translator. Translate the user's content into ${name}.`,
|
||||
"",
|
||||
"Rules:",
|
||||
`- Translate prose only. Preserve all markdown structure, whitespace, and line breaks exactly.`,
|
||||
`- Never translate, reorder, or remove any ⸤I18N…⸥ marker. Copy each one verbatim in place.`,
|
||||
`- Do not translate these terms; keep them exactly as written: ${DO_NOT_TRANSLATE.join(", ")}.`,
|
||||
`- Keep code, URLs, file paths, and CLI flags unchanged.`,
|
||||
`- Preserve the meaning and tone. Do not add notes, explanations, or extra text.`,
|
||||
`- Output only the translation. No preamble, no closing remarks.`,
|
||||
].join("\n");
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
// scripts/i18n/lib/hash.mjs
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
/**
|
||||
* Stable 12-char hex hash of a source string.
|
||||
* CRLF is normalized to LF so line-ending changes never trigger re-translation.
|
||||
* @param {string} text
|
||||
* @returns {string}
|
||||
*/
|
||||
export function hash(text) {
|
||||
const normalized = String(text).replace(/\r\n/g, "\n");
|
||||
return createHash("sha256").update(normalized, "utf8").digest("hex").slice(0, 12);
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
// scripts/i18n/lib/mask.mjs
|
||||
|
||||
// Ordered so multi-char structures are matched before their sub-parts.
|
||||
// Each pattern captures the exact substring to hide from the translator.
|
||||
const FENCE = /(^|\n)(```|~~~)[\s\S]*?\n\2/g; // fenced code blocks
|
||||
const INLINE = /`[^`\n]+`/g; // inline code spans
|
||||
// Link/image URL: mask the URL, keep the [text]. The URL may itself contain a
|
||||
// single level of balanced parentheses (e.g. wikipedia .../Foo_(bar)); match
|
||||
// runs of non-paren chars and optional (balanced) groups, then the closing paren.
|
||||
const LINK_URL = /(!?\[[^\]]*\]\()((?:[^()]|\([^()]*\))*)(\))/g;
|
||||
// {{double}} first so it is captured whole, then {single} interpolation.
|
||||
const PLACEHOLDER = /\{\{[^{}\n]+\}\}|\{[^{}\n]+\}/g;
|
||||
|
||||
const TOKEN = (i) => `⸤I18N${i}⸥`; // uncommon delimiters, restored 1:1
|
||||
const TOKEN_RE = /⸤I18N\d+⸥/g;
|
||||
|
||||
/**
|
||||
* Replace protected substrings with opaque tokens.
|
||||
* @param {string} text
|
||||
* @returns {{ masked: string, tokens: string[] }}
|
||||
*/
|
||||
export function mask(text) {
|
||||
const tokens = [];
|
||||
const push = (value) => {
|
||||
tokens.push(value);
|
||||
return TOKEN(tokens.length - 1);
|
||||
};
|
||||
let out = String(text);
|
||||
out = out.replace(FENCE, (m) => push(m));
|
||||
out = out.replace(INLINE, (m) => push(m));
|
||||
out = out.replace(LINK_URL, (_m, open, url, close) => `${open}${push(url)}${close}`);
|
||||
out = out.replace(PLACEHOLDER, (m) => push(m));
|
||||
return { masked: out, tokens };
|
||||
}
|
||||
|
||||
/**
|
||||
* Reinsert masked substrings by token index.
|
||||
* @param {string} masked
|
||||
* @param {string[]} tokens
|
||||
* @returns {string}
|
||||
*/
|
||||
export function restore(masked, tokens) {
|
||||
return String(masked).replace(TOKEN_RE, (t) => {
|
||||
const m = t.match(/I18N(\d+)/);
|
||||
const i = m ? Number(m[1]) : Number.NaN;
|
||||
return tokens[i] ?? t;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Count structural elements, used by the validator to prove nothing was lost.
|
||||
* @param {string} text
|
||||
* @returns {{ fences: number, inlineCode: number, links: number, placeholders: number }}
|
||||
*/
|
||||
export function countStructures(text) {
|
||||
const s = String(text);
|
||||
const count = (re) => (s.match(re) || []).length;
|
||||
return {
|
||||
fences: count(FENCE),
|
||||
inlineCode: count(INLINE),
|
||||
links: count(LINK_URL),
|
||||
placeholders: count(PLACEHOLDER),
|
||||
};
|
||||
}
|
||||
|
||||
export { TOKEN_RE };
|
||||
@@ -0,0 +1,30 @@
|
||||
// scripts/i18n/lib/shared-i18n.mjs
|
||||
import { SUPPORTED_LOCALES } from "../../../packages/shared/src/i18n/index.ts";
|
||||
|
||||
/** @returns {string[]} all supported locale codes */
|
||||
export function localeCodes() {
|
||||
return SUPPORTED_LOCALES.map((l) => l.code);
|
||||
}
|
||||
|
||||
/**
|
||||
* The exported const name inside a locale file is the code camel-cased:
|
||||
* "de" -> "de", "pt-BR" -> "ptBR", "zh-CN" -> "zhCN".
|
||||
* @param {string} code
|
||||
* @returns {string}
|
||||
*/
|
||||
function exportName(code) {
|
||||
return code.replace(/-([a-z])/gi, (_m, c) => c.toUpperCase());
|
||||
}
|
||||
|
||||
/**
|
||||
* Load the translated `tools` namespace for a locale:
|
||||
* { [toolId]: { name, description } }.
|
||||
* @param {string} code
|
||||
* @returns {Promise<Record<string, { name: string, description: string }>>}
|
||||
*/
|
||||
export async function loadToolStrings(code) {
|
||||
const mod = await import(`../../../packages/shared/src/i18n/${code}.ts`);
|
||||
const dict = mod[exportName(code)];
|
||||
if (!dict?.tools) throw new Error(`No tools namespace in locale ${code}`);
|
||||
return dict.tools;
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
// scripts/i18n/lib/slugify.mjs
|
||||
|
||||
// Exact port of the slugify VitePress uses (mdit-vue @mdit-vue/shared). Keeping
|
||||
// this identical means the {#anchor} we inject equals the auto-slug VitePress
|
||||
// would produce, so pre-existing #deep-links keep resolving after we add
|
||||
// explicit anchors. Regexes copied verbatim from vitepress/dist/node, with
|
||||
// explicit unicode escapes for the control and combining ranges.
|
||||
// biome-ignore lint/suspicious/noControlCharactersInRegex: byte-exact port of mdit-vue slugify; the control-char range is required for VitePress anchor parity.
|
||||
const rControl = /[\u0000-\u001f]/g;
|
||||
const rSpecial = /[\s~`!@#$%^&*()\-_+=[\]{}|\\;:"'“”‘’<>,.?/]+/g;
|
||||
const rCombining = /[\u0300-\u036f]/g;
|
||||
|
||||
/**
|
||||
* @param {string} str
|
||||
* @returns {string}
|
||||
*/
|
||||
export function slugify(str) {
|
||||
return String(str)
|
||||
.normalize("NFKD")
|
||||
.replace(rCombining, "")
|
||||
.replace(rControl, "")
|
||||
.replace(rSpecial, "-")
|
||||
.replace(/-{2,}/g, "-")
|
||||
.replace(/^-+|-+$/g, "")
|
||||
.replace(/^(\d)/, "_$1")
|
||||
.toLowerCase();
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
// scripts/i18n/lib/validate.mjs
|
||||
import { countStructures, TOKEN_RE } from "./mask.mjs";
|
||||
|
||||
/**
|
||||
* Assert that the translated text preserved the source's markdown structure.
|
||||
* Runs on the FINAL (restored) translation, so no mask tokens should remain.
|
||||
* @param {string} source
|
||||
* @param {string} translated
|
||||
* @returns {{ ok: boolean, errors: string[] }}
|
||||
*/
|
||||
export function validate(source, translated) {
|
||||
const errors = [];
|
||||
const a = countStructures(source);
|
||||
const b = countStructures(translated);
|
||||
for (const key of /** @type {const} */ (["fences", "inlineCode", "links", "placeholders"])) {
|
||||
if (a[key] !== b[key]) errors.push(`${key} count changed: ${a[key]} -> ${b[key]}`);
|
||||
}
|
||||
TOKEN_RE.lastIndex = 0;
|
||||
if (TOKEN_RE.test(translated)) errors.push("unrestored mask token remained in output");
|
||||
return { ok: errors.length === 0, errors };
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
// scripts/i18n/translate.mjs
|
||||
import { mkdirSync } from "node:fs";
|
||||
import { ADAPTERS, resolveSurfaces } from "./adapters/registry.mjs";
|
||||
import { collectPending, runTranslation } from "./core.mjs";
|
||||
import { batchResultTranslator, readDone, writePending } from "./lib/batch.mjs";
|
||||
import { localeCodes } from "./lib/shared-i18n.mjs";
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = {
|
||||
surface: "all",
|
||||
locale: "all",
|
||||
engine: "claude-code",
|
||||
mode: "export",
|
||||
batchDir: ".i18n-batches",
|
||||
help: false,
|
||||
};
|
||||
for (const a of argv) {
|
||||
if (a === "--help" || a === "-h") args.help = true;
|
||||
else if (a === "--export") args.mode = "export";
|
||||
else if (a === "--import") args.mode = "import";
|
||||
else if (a === "--dry-run") args.mode = "dry-run";
|
||||
else if (a.startsWith("--engine=")) args.engine = a.slice("--engine=".length);
|
||||
else if (a.startsWith("--surface=")) args.surface = a.slice("--surface=".length);
|
||||
else if (a.startsWith("--locale=")) args.locale = a.slice("--locale=".length);
|
||||
else if (a.startsWith("--batch-dir=")) args.batchDir = a.slice("--batch-dir=".length);
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
function resolveLocales(spec) {
|
||||
const all = localeCodes().filter((c) => c !== "en");
|
||||
if (spec === "all") return all;
|
||||
return spec
|
||||
.split(",")
|
||||
.map((s) => s.trim())
|
||||
.filter((c) => all.includes(c));
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (args.help) {
|
||||
console.log(
|
||||
[
|
||||
"Usage: tsx scripts/i18n/translate.mjs --export|--import|--dry-run [options]",
|
||||
" --surface=all|<name,..> default all",
|
||||
" --locale=all|<code,..> default all (minus en)",
|
||||
" --engine=claude-code|api default claude-code (no key). api needs ANTHROPIC_API_KEY.",
|
||||
" --batch-dir=<path> default .i18n-batches",
|
||||
"",
|
||||
"Claude Code flow: --export writes pending masked strings; a Claude Code session",
|
||||
"translates each <surface>.<locale>.pending.json into <surface>.<locale>.done.json",
|
||||
"(keeping every mask token); --import restores, validates, and writes via adapters.",
|
||||
`Surfaces: ${Object.keys(ADAPTERS).join(", ") || "(none registered yet)"}`,
|
||||
].join("\n"),
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const surfaces = resolveSurfaces(args.surface);
|
||||
const locales = resolveLocales(args.locale);
|
||||
if (surfaces.length === 0) {
|
||||
console.log("No adapters registered yet. Nothing to do.");
|
||||
return;
|
||||
}
|
||||
mkdirSync(args.batchDir, { recursive: true });
|
||||
|
||||
for (const surface of surfaces) {
|
||||
const mod = await ADAPTERS[surface]();
|
||||
const adapter = mod.adapter;
|
||||
|
||||
if (args.mode === "export") {
|
||||
const units = await adapter.extract();
|
||||
for (const locale of locales) {
|
||||
const stored = await adapter.load(locale);
|
||||
const { pending } = collectPending(units, stored);
|
||||
if (pending.length === 0) continue;
|
||||
const path = writePending(args.batchDir, surface, locale, pending);
|
||||
console.log(`export ${surface} ${locale}: ${pending.length} -> ${path}`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (args.mode === "import") {
|
||||
for (const locale of locales) {
|
||||
let doneMap;
|
||||
try {
|
||||
doneMap = readDone(args.batchDir, surface, locale);
|
||||
} catch {
|
||||
continue; // no done file for this locale yet
|
||||
}
|
||||
const summary = await runTranslation({
|
||||
adapter,
|
||||
locales: [locale],
|
||||
translate: batchResultTranslator(doneMap),
|
||||
log: (m) => process.stdout.write(`${m}\n`),
|
||||
});
|
||||
console.log(`import ${surface} ${locale}:`, JSON.stringify(summary[locale]));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// dry-run: stub translator, no batch files, proves wiring
|
||||
const summary = await runTranslation({
|
||||
adapter,
|
||||
locales,
|
||||
translate: async (units, locale) =>
|
||||
new Map(units.map((u) => [u.id, `${locale}:${u.sourceText}`])),
|
||||
});
|
||||
console.log(`dry-run ${surface}:`, JSON.stringify(summary, null, 2));
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(err);
|
||||
process.exit(1);
|
||||
});
|
||||
Reference in New Issue
Block a user