fix(docs): sitemap gains lastmod, and robots.txt points at it (#140)

docs.beardrive.ai shipped no robots.txt, so nothing on that host named the
sitemap — a crawler arriving at the subdomain had to guess the URL or be
handed it in Search Console. And every entry was a bare <loc>: no freshness
signal at all, on the site whose whole value is being current.

Each URL now carries the commit date of the markdown behind it, read from one
`git log` for the whole tree. The build refuses to guess: in a shallow clone
(the default for CI checkouts) git can only attribute every file to the single
commit it has, so lastmod is omitted entirely rather than claiming the site
changed wholesale on every deploy — Google discounts a sitemap that does that,
which would cost more than the absent dates. Hosts that want the dates need
full history; README says so.

Declaring @astrojs/sitemap explicitly replaces the copy Starlight adds for
itself rather than duplicating it — that's the supported way to reach these
options, and Starlight's own version only configures i18n, which this
single-language site doesn't use.

Adds `npm run check:sitemap <origin>`: robots.txt -> index -> every advertised
URL returns 200, the checks Search Console runs, against a local preview or
against production. Run against production today it fails on the missing
Sitemap: line, and reports the deployed site is several commits behind the
repo — /concepts/permissions/ and /reference/migration/ are live 404s.
This commit is contained in:
Snow Lee (Sungwon)
2026-08-10 23:27:47 +09:00
committed by GitHub
parent d362b1dbf9
commit 70cf9818ce
5 changed files with 264 additions and 1 deletions
+22
View File
@@ -0,0 +1,22 @@
# docs.beardrive.ai
#
# The Sitemap line is why this file exists. robots.txt is the one place a
# crawler looks without being told, and until this existed the docs sitemap
# was reachable only by guessing its URL or by submitting it in Search
# Console — the landing page's sitemap is on a different host and can't
# point here.
#
# Everything is allowed, including every AI crawler. That is the same posture
# as the landing's robots.txt, which spells each agent out one by one; this
# file doesn't need to, and the difference is not an oversight. That file
# carries Disallow rules for the hub's own routes (/api/, /auth/, /s/ — the
# app shares its origin), and a named user-agent stops reading the `*` group
# entirely, so each one has to be re-granted access explicitly. Nothing on
# this host is private, so nothing is disallowed, so one group covers all of
# them. Add a Disallow here and that stops being true — restore the
# enumeration if you ever do.
User-agent: *
Allow: /
Sitemap: https://docs.beardrive.ai/sitemap-index.xml