beardrive/web/docs/README.md

# docs.beardrive.ai

The public product documentation: CLI, sync model, self-hosting. Astro +
[Starlight](https://starlight.astro.build), static output, Pagefind search.

```sh
npm install
npm run dev       # http://localhost:4321
npm run build     # -> dist/
npm run preview

npm run check:sitemap http://localhost:4321        # after npm run preview
npm run check:sitemap https://docs.beardrive.ai    # or against production
```

## The sitemap

`/sitemap-index.xml` -> `/sitemap-0.xml`, generated by `@astrojs/sitemap`.
Starlight would add that integration itself; `astro.config.mjs` declares it
explicitly instead, which replaces Starlight's default rather than duplicating
it and is what makes the `lastmod` option reachable.

Each URL carries a `<lastmod>` taken from the **commit date of the markdown
behind it** — see "Checkout depth" below for the one way that goes wrong. The
redirect stubs are correctly absent: Astro emits them as `noindex` meta-refresh
pages, and `@astrojs/sitemap` leaves them out.

`public/robots.txt` exists to carry the `Sitemap:` line: robots.txt is the one
file a crawler fetches without being told, and the landing page's robots.txt is
on a different host so it cannot point here.

**Verify that file survives the deploy.** This host is behind Cloudflare, which
was serving a managed robots.txt of its own (the content-signals policy block)
back when the origin had none — a body with no `Sitemap:` and, in fact, no
directives at all. Whether an origin robots.txt now replaces that or gets merged
with it is Cloudflare's call, not this repo's, so after deploying check the
served file rather than the built one:

```sh
curl -s https://docs.beardrive.ai/robots.txt | grep -i sitemap
```

`check:sitemap` is the same set of checks Search Console runs — follow
robots.txt to the index, parse it, confirm every advertised URL returns 200 —
and takes any origin, so it works against a local preview and against
production. It is worth running against production after a deploy: a sitemap
that has gone stale or started advertising 404s looks completely fine until
something crawls it, which is weeks later.

## Why this is a standalone site

Unlike the hub frontend (`internal/webapp/static`) and the cloud landing page
(`cloud/internal/landing/dist`), this is **not** embedded into a Go binary:

- Docs change far more often than the binary. Embedding them would mean a Go
  rebuild and redeploy to fix a typo.
- A Pagefind search index has no business shipping inside every self-hoster's
  install.

It lives in the OSS repo because that is what it documents. "Edit this page"
resolves to something an outside contributor can open a PR against.

## Design tokens

`scripts/tokens.mjs` reads the `@theme` block in
`internal/webapp/frontend/src/tw.css` — the source of truth for the BearDrive
palette — and emits `src/styles/tokens.gen.css`. That file is generated,
gitignored, and regenerated by `npm run dev` and `npm run build`, so the palette
cannot drift. `src/styles/custom.css` maps Starlight's `--sl-color-*` variables
onto it and invents no colors of its own.

(The cloud landing page can't do this — it sits in a different Go module and
keeps a *copy* of the tokens, policed by its own `check-tokens.mjs`.)

## Adding a page

Drop a `.md` file under `src/content/docs/<section>/` with `title` and
`description` frontmatter, then add its slug to the `sidebar` in
`astro.config.mjs`. Sidebar order is explicit, not alphabetical.

Write `description` for every page: it is the meta description, the search
result snippet, and what `llms.txt` shows.

## llms.txt

`starlight-llms-txt` generates `/llms.txt`, `/llms-small.txt`, and
`/llms-full.txt` at build time.

Convention puts `llms.txt` at the **root** domain, not a docs subdomain — so
`beardrive.ai/llms.txt` should redirect or proxy to `docs.beardrive.ai/llms.txt`.
That redirect lives in the cloud landing page and is the one cross-repo
coordination point this split introduces.

## Structure

The sidebar order in `astro.config.mjs` **is** the recommended path, and the
recommended path is agent-first:

- **Start here** — what it is, set up with your agent, first hour. No `brew
  install` appears in this group.
- **Working with agents** — the workflows the product exists for.
- **Manual setup (optional)** — the CLI route: install, set up by hand, skills
  and hooks in detail. Same destination, more steps; one click away, never on
  the critical path.
- **Self-hosting**, **Reference**, **Concepts** — unchanged in intent.
- **More** — off-site links (use cases, blog, GitHub), last because the order
  is the recommended path and nothing that leaves the docs belongs above them.

Keep new onboarding content out of Manual. If a page teaches someone how to get
started, it belongs in Start here and should say what to ask an agent, not what
to type.

Job-shaped "use case" pages — written for someone deciding, not someone
building — live on the marketing site (`beardrive.ai/use-cases`), which owns
that audience and already had its own copy of all of them. The docs used to
carry a parallel set; they were deleted and each URL redirects to its
counterpart there.

The header logo links to `beardrive.ai`, not the docs index. Starlight always
points it at the docs root and exposes no option for it, so that link lives in
`src/components/SiteTitle.astro` — Starlight's own component with one changed
`href`, registered through `components:` in `astro.config.mjs`. If a Starlight
upgrade restyles the header, that file is the first thing to re-diff against
`node_modules/@astrojs/starlight/components/SiteTitle.astro`.

## Deploying

Static output in `dist/`. Any static host works; build command `npm run build`,
output directory `dist`, project root `web/docs`.

Today that host is **Cloudflare Pages**, project `beardrive-docs`
(`docs.beardrive.ai`), deployed by `.github/workflows/docs.yml` on every push to
`main` that touches `web/docs/**` or the token source. It needs two repository
secrets: `CLOUDFLARE_API_TOKEN` (an account token with *Cloudflare Pages: Edit*)
and `CLOUDFLARE_ACCOUNT_ID`.

Not the Pages **Git integration**, deliberately: it clones shallow, which costs
every `<lastmod>` (see "Checkout depth"), and it would build the docs on commits
that cannot change them. The Pages project is a direct-upload project — nothing
deploys it but this workflow. That is also the failure mode to watch for: when
deploys were manual, they simply stopped, and the site sat three weeks stale
while every check that only runs *during* a deploy stayed green.

### Redirects

The docs were reorganized around the agent-first path, so three old URLs moved:

| Old | New |
|---|---|
| `/start/install` | `/manual/install/` |
| `/start/quickstart` | `/manual/setup-by-hand/` |
| `/guides/connect-an-agent` | `/start/setup/` |

`astro.config.mjs` declares these, which in a static build emits **meta-refresh
pages** — fine for humans, weak for search engines. Real 301s belong in the host.

**Firebase Hosting** (simplest static option on GCP — CDN, TLS, and custom
domains included):

```json
{
  "hosting": {
    "public": "dist",
    "ignore": ["firebase.json", "**/.*"],
    "redirects": [
      { "source": "/start/install", "destination": "/manual/install/", "type": 301 },
      { "source": "/start/quickstart", "destination": "/manual/setup-by-hand/", "type": 301 },
      { "source": "/guides/connect-an-agent", "destination": "/start/setup/", "type": 301 }
    ]
  }
}
```

**Cloud Storage behind an external Application Load Balancer:** put the rules in
the URL map, which redirects before the bucket is ever reached.

```sh
gcloud compute url-maps edit docs-url-map   # pathMatchers[].pathRules[]:
#   - paths: ["/start/install"]
#     urlRedirect:
#       pathRedirect: "/manual/install/"
#       redirectResponseCode: MOVED_PERMANENTLY_DEFAULT
#       stripQuery: false
```

Whichever host wins, keep the Astro `redirects` block as well: it is the
portable fallback, and it keeps local `npm run preview` honest.

Note that the build reads a file **outside** `web/docs` (the token source), so
the host must check out the whole repository rather than just this
subdirectory.

### Checkout depth

Check out with **full history**, not a shallow clone. The sitemap's `<lastmod>`
for each page is the commit date of the markdown behind it, so a depth-1
checkout — the default for `actions/checkout` and for most build hosts — has
nothing to read the dates from.

```yaml
- uses: actions/checkout@v4
  with:
    fetch-depth: 0 # sitemap <lastmod> comes from commit dates
```

Getting this wrong degrades rather than breaks: `astro.config.mjs` detects the
shallow clone and emits **no** `lastmod` at all, because the alternative is
stamping all 25 pages with the one commit a shallow clone has, and a sitemap
that claims the whole site changed on every deploy is one Google learns to
ignore. So the symptom is a silently less useful sitemap — check for `<lastmod>`
in the deployed `sitemap-0.xml` after changing hosts.