mirror of
https://github.com/runbear-io/beardrive.git
synced 2026-08-25 08:08:08 +02:00
fix(docs): sitemap gains lastmod, and robots.txt points at it (#140)
docs.beardrive.ai shipped no robots.txt, so nothing on that host named the sitemap — a crawler arriving at the subdomain had to guess the URL or be handed it in Search Console. And every entry was a bare <loc>: no freshness signal at all, on the site whose whole value is being current. Each URL now carries the commit date of the markdown behind it, read from one `git log` for the whole tree. The build refuses to guess: in a shallow clone (the default for CI checkouts) git can only attribute every file to the single commit it has, so lastmod is omitted entirely rather than claiming the site changed wholesale on every deploy — Google discounts a sitemap that does that, which would cost more than the absent dates. Hosts that want the dates need full history; README says so. Declaring @astrojs/sitemap explicitly replaces the copy Starlight adds for itself rather than duplicating it — that's the supported way to reach these options, and Starlight's own version only configures i18n, which this single-language site doesn't use. Adds `npm run check:sitemap <origin>`: robots.txt -> index -> every advertised URL returns 200, the checks Search Console runs, against a local preview or against production. Run against production today it fails on the missing Sitemap: line, and reports the deployed site is several commits behind the repo — /concepts/permissions/ and /reference/migration/ are live 404s.
This commit is contained in:
@@ -0,0 +1,22 @@
|
||||
# docs.beardrive.ai
|
||||
#
|
||||
# The Sitemap line is why this file exists. robots.txt is the one place a
|
||||
# crawler looks without being told, and until this existed the docs sitemap
|
||||
# was reachable only by guessing its URL or by submitting it in Search
|
||||
# Console — the landing page's sitemap is on a different host and can't
|
||||
# point here.
|
||||
#
|
||||
# Everything is allowed, including every AI crawler. That is the same posture
|
||||
# as the landing's robots.txt, which spells each agent out one by one; this
|
||||
# file doesn't need to, and the difference is not an oversight. That file
|
||||
# carries Disallow rules for the hub's own routes (/api/, /auth/, /s/ — the
|
||||
# app shares its origin), and a named user-agent stops reading the `*` group
|
||||
# entirely, so each one has to be re-granted access explicitly. Nothing on
|
||||
# this host is private, so nothing is disallowed, so one group covers all of
|
||||
# them. Add a Disallow here and that stops being true — restore the
|
||||
# enumeration if you ever do.
|
||||
|
||||
User-agent: *
|
||||
Allow: /
|
||||
|
||||
Sitemap: https://docs.beardrive.ai/sitemap-index.xml
|
||||
Reference in New Issue
Block a user