docs.beardrive.ai shipped no robots.txt, so nothing on that host named the
sitemap — a crawler arriving at the subdomain had to guess the URL or be
handed it in Search Console. And every entry was a bare <loc>: no freshness
signal at all, on the site whose whole value is being current.
Each URL now carries the commit date of the markdown behind it, read from one
`git log` for the whole tree. The build refuses to guess: in a shallow clone
(the default for CI checkouts) git can only attribute every file to the single
commit it has, so lastmod is omitted entirely rather than claiming the site
changed wholesale on every deploy — Google discounts a sitemap that does that,
which would cost more than the absent dates. Hosts that want the dates need
full history; README says so.
Declaring @astrojs/sitemap explicitly replaces the copy Starlight adds for
itself rather than duplicating it — that's the supported way to reach these
options, and Starlight's own version only configures i18n, which this
single-language site doesn't use.
Adds `npm run check:sitemap <origin>`: robots.txt -> index -> every advertised
URL returns 200, the checks Search Console runs, against a local preview or
against production. Run against production today it fails on the missing
Sitemap: line, and reports the deployed site is several commits behind the
repo — /concepts/permissions/ and /reference/migration/ are live 404s.
Adds web/docs — the public product documentation, built with Astro +
Starlight and deployed on its own rather than embedded in the binary.
Why a standalone site in the OSS repo, rather than a section of the cloud
landing page:
- Docs change far more often than the binary does. Embedding them would
mean a Go rebuild and redeploy to fix a typo, and would ship a Pagefind
search index inside every self-hoster's install.
- Self-hosting instructions and the CLI reference document the OSS
project, so "edit this page" should resolve to something an outside
contributor can open a PR against.
- Design tokens get easier, not harder: scripts/tokens.mjs generates
src/styles/tokens.gen.css from the @theme block in the hub frontend's
tw.css, so there is one source of truth and nothing to police. (The
cloud landing sits across a module edge and has to keep a *copy*,
guarded by its own check-tokens.mjs.) custom.css maps Starlight's
--sl-color-* onto those tokens and invents no colors of its own.
Content is seeded from README.md, docs/self-hosting.md, and the plugin
skill. Guides deliberately cover agent workflows — connecting an agent,
the two-file AGENTS.md orientation pattern, artifacts and links, read
heat, scoping the folder — rather than re-teaching the CLI, which lives
in Reference.
starlight-llms-txt emits /llms.txt at build. Convention wants that at the
root domain, so beardrive.ai/llms.txt should point here; that redirect
belongs to the cloud landing and is the one cross-repo coordination point
this split introduces.
Claude-Session: https://claude.ai/code/session_018GcqsM6prjdv9rUrhVVEiC
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
