From 28dc882c66251604bbde0dee6604898a845e7036 Mon Sep 17 00:00:00 2001 From: Snow Lee Date: Sat, 11 Jul 2026 14:41:07 -0700 Subject: [PATCH 1/3] =?UTF-8?q?feat(web):=20read=20heatmap=20phase=201=20?= =?UTF-8?q?=E2=80=94=20ledger,=20heat=20API,=20folder=20heat=20dots?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Read telemetry per docs/design/read-heatmap.md: a ReadLedger over a new batch-oriented MetaStore ReadRepo (file reads.json + SQL read_stats) aggregates viewer and share reads into daily per-actor buckets, debounced to visits, folded into all-time rows past retention. GET /api/p//heat serves per-path counts (human/agent/share, distinct readers, last read) — never identities. /store sync traffic and history blob views are not reads. The viewer shows heat dots and read counts on folder listings and the file meta line. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01P5cxPQdSGJnjXCYY9GeWXt --- cmd/bdrive/web.go | 27 ++ docs/design/read-heatmap.md | 237 ++++++++++++++ internal/webapp/db.go | 11 + internal/webapp/db_conformance_test.go | 28 +- internal/webapp/db_file.go | 72 +++++ internal/webapp/db_sql.go | 64 ++++ internal/webapp/reads.go | 411 +++++++++++++++++++++++++ internal/webapp/reads_test.go | 236 ++++++++++++++ internal/webapp/server.go | 22 +- internal/webapp/shares.go | 3 + internal/webapp/static/app.js | 68 ++++ internal/webapp/static/style.css | 6 + 12 files changed, 1179 insertions(+), 6 deletions(-) create mode 100644 docs/design/read-heatmap.md create mode 100644 internal/webapp/reads.go create mode 100644 internal/webapp/reads_test.go diff --git a/cmd/bdrive/web.go b/cmd/bdrive/web.go index b80ba20..bf20215 100644 --- a/cmd/bdrive/web.go +++ b/cmd/bdrive/web.go @@ -57,6 +57,13 @@ type webConfig struct { Driver string `json:"driver,omitempty"` // file (default) | sqlite | postgres DSN string `json:"dsn,omitempty"` // sqlite file path, or a Postgres/Supabase URL } `json:"database,omitempty"` + // Reads tunes read telemetry (the heat API): aggregate view counts per + // file, split human/agent/share. On by default in hub mode; counts only, + // no reader identities in any API. + Reads *struct { + Enabled *bool `json:"enabled,omitempty"` // default true + RetentionDays int `json:"retention_days,omitempty"` // default 400; older days fold into all-time + } `json:"reads,omitempty"` } func loadWebConfig(path string) (webConfig, error) { @@ -340,6 +347,26 @@ credentials); otherwise it is relayed through this server.`, return fmt.Errorf("open share registry: %w", err) } srv.Shares = shares + readsOn, retention := true, 0 + if cfg.Reads != nil { + if cfg.Reads.Enabled != nil { + readsOn = *cfg.Reads.Enabled + } + retention = cfg.Reads.RetentionDays + } + if readsOn { + var reads *webapp.ReadLedger + if meta != nil { + reads, err = webapp.NewReadLedger(meta.Reads(), retention) + } else { + reads, err = webapp.OpenReadLedger(filepath.Join(filepath.Dir(projectsDB), "reads.json"), retention) + } + if err != nil { + return fmt.Errorf("open read ledger: %w", err) + } + defer reads.Close() + srv.Reads = reads + } if meta != nil { display += " (db: " + cfg.Database.Driver + ")" } else { diff --git a/docs/design/read-heatmap.md b/docs/design/read-heatmap.md new file mode 100644 index 0000000..9a0788a --- /dev/null +++ b/docs/design/read-heatmap.md @@ -0,0 +1,237 @@ +# Read heatmap — design + +Status: proposed (2026-07-11) · Owner: snow · Prior art: session-linked notes (shipped), history API + +## Problem + +BearDrive knows everything about *writes* (journals: who, when, which session) +and nothing about *reads*. Admins curating a shared knowledge folder can't see +which files the team actually consumes, so they can't tell a load-bearing +document from dead weight. The killer view is the **read×write matrix**: +heavily-read + long-unwritten is the danger zone (stale knowledge people still +rely on); unread + unwritten is archive material. + +Two properties make this more than Confluence-style view counts: + +1. **Write provenance is already perfect** (journals), so read data alone + completes the matrix — competitors have reads but weak write history. +2. **Agent reads are attributable.** The agent hooks + session notes shipped in + PR #12 mean we can distinguish *human* reads from *agent* reads — the agent + hot-path is effectively the team's context window, and nobody else can see it. + +## Non-goals + +- **`/store/*` sync traffic is never a read.** Devices replicating a volume is + replication, not consumption. Only deliberate consumption counts. +- No per-user browsing profiles in any API response. Aggregate counts only. +- No general web analytics (referrers, dwell time, scroll depth). +- Volume mode (`DirSource`, auth-free viewer) is out of scope — like history, + this is a hub feature. + +## What counts as a read + +| Source | Kind | Actor (internal only) | Where recorded | +|---|---|---|---| +| Viewer `GET file` / `render` / `download` | `human` | account email | `serveBlob`, `handleRender` | +| Share link hit `GET /s/` | `share` | share token | `handleShared` | +| Agent tool read (Read / read_file / …) | `agent` | device id | reported by client, phase 3 | +| History version view (`/blob`) | — not counted | | spelunking ≠ consumption | +| `/store/*` | — never | | replication | + +Details: + +- **Record before the ETag check.** A 304 render is still a person reading the + file; skipping it would undercount exactly the hottest (most-cached) pages. +- **Debounce to visits, not requests.** An in-memory `(project, path, actor)` + seen-map with a 10-minute window collapses reload storms and the + render-then-raw double fetch into one read. The map is pruned on the flush + tick. Embedded assets fetched during a markdown render do count as reads of + those assets ("this diagram is viewed a lot" is signal, not noise). + +## Data model + +One new MetaStore repo. Rows are daily aggregation buckets, not events — the +ledger never stores an event log, so there is nothing sensitive to leak and +nothing that grows per-request. + +```go +// ReadStat is one aggregation bucket: reads of one path by one actor on one +// day. Day=="" is the all-time fold (see retention). Actor is an opaque +// internal id (account email / device id / share token) used only to count +// distinct readers; it never appears in an API response. +type ReadStat struct { + Project string `json:"project"` + Path string `json:"path"` + Day string `json:"day"` // "2026-07-11" UTC, or "" for all-time + Kind string `json:"kind"` // human | agent | share + Actor string `json:"actor"` + Count int64 `json:"count"` + Last time.Time `json:"last"` +} + +// ReadRepo persists read buckets. Unlike the other repos this one is batch- +// oriented: reads are telemetry and flushes carry many dirty buckets at once — +// one file rewrite / one SQL transaction per flush, not per bucket. +type ReadRepo interface { + Load() ([]ReadStat, error) + PutBatch(stats []ReadStat) error // upsert by (project,path,day,kind,actor) + DeleteBatch(keys []ReadStatKey) error // used by retention fold +} +``` + +- `MetaStore` gains `Reads() ReadRepo`; file backend adds `reads.json` + (same load-all / rewrite-atomically discipline, 0o755 dir mode — no secrets), + SQL backend adds one table with PK `(project, path, day, kind, actor)` and + the usual idempotent `CREATE TABLE IF NOT EXISTS` migration. +- `db_conformance_test.go` gets ReadRepo cases like every other repo. + +### Cardinality & retention + +Worst realistic case (1k files × 30 actors × 400 days, every actor reading +every file daily) is implausible; actual rows ≈ files-actually-read × active +actors × active days. Retention keeps it bounded regardless: + +- Config `reads.retention_days` (default **400** — enough for a year-over-year + view). On the daily compaction pass (at load + once per day on the flush + loop), buckets older than the horizon are **folded into the all-time row** + `(project, path, "", kind, actor)` and deleted. All-time totals survive + forever; per-day resolution ages out. + +## Server plumbing + +New `ReadLedger` service (`webapp/reads.go`), same shape as `DeviceRegistry`: +in-memory state over a repo, with write throttling. + +```go +type ReadLedger struct { + repo ReadRepo + mu sync.Mutex + byKey map[readKey]ReadStat // loaded at open, bumped in memory + dirty map[readKey]struct{} // flushed every 30s and on Close + seen map[visitKey]time.Time // debounce window +} +func (l *ReadLedger) Record(project, path, kind, actor string) +func (l *ReadLedger) Heat(project, prefix string, since time.Time) map[string]HeatEntry +func (l *ReadLedger) Close() error // final flush +``` + +- `Server.Reads *ReadLedger` — nil means the feature is off (mirrors + `Devices`/`Shares`). `Record` on a nil ledger is a no-op, so call sites + stay unconditional. +- **Recording sites**: `serveBlob` (covers `file` + `download`), + `handleRender`, and `handleShared`. The `proj()` resolver already knows the + project id; it stashes it in the request context so `recordRead(r, path, + kind)` can pick it up without changing every handler signature. +- **Never on the failure path**: record only after the path resolved and + membership passed (`proj()` runs `projectAllowed` first — a 403 records + nothing). +- Flush errors degrade silently (log once), same "never break the request" + posture as sync: telemetry must never 500 a page view. + +## API + +One endpoint, membership-gated like every per-project route: + +``` +GET /api/p//heat?prefix=&days=30 +→ { "since": "2026-06-11", "entries": { + "wiki/onboarding.md": { "human": 42, "agent": 17, "share": 3, + "readers": 6, "last_read": "2026-07-10T…" }, + … } } +``` + +- `days=0` → all-time (daily buckets + the all-time fold). +- `readers` = distinct human actors in the window. Actor identities never + leave the server; this is the only trace of them. +- **No writes-join on the server**: the frontend already has per-path last + write time from `tree`, so the read×write matrix is a client-side join. + One source of truth for write recency, no new endpoint. +- `/api/config` gains `"reads": {"enabled": true|false}` so the frontend knows + whether to fetch heat at all. + +## Frontend + +**Phase 1 — ambient heat (all members).** +- Folder listing rows (`renderFolderListing`): a heat dot before the meta text, + intensity 0–4 on a log scale of 30-day reads (any-kind), plus `· 42 reads` + in `dl-meta`. Folders show the sum of their subtree. +- File view meta line gains `· 42 reads/30d`. +- One `heat?days=30` fetch per project, cached alongside the tree and + refreshed with it. + +**Phase 2 — Insights (admins + org owners).** +- An "Insights" panel per project: SVG quadrant scatter, **x = days since last + write (log), y = reads in window (log)**, one point per file. Quadrants: + hot+fresh (healthy), **hot+stale (danger zone — fix these first)**, + cold+fresh (new, unproven), cold+stale (archive candidates). +- Below it, the danger-zone list ranked by `reads × staleness`, each row + linking into the file with its history. +- A human/agent toggle (or stacked dot colors) — "what do the agents live on" + is a different question from "what do people open". +- Dependency-free vanilla JS/SVG like the rest of the frontend. + +## Agent reads (phase 3) + +Agents read files from the *local synced folder* — the hub never sees those +reads. The hooks pipeline shipped for session notes closes the gap: + +1. **Hook**: each platform gets one more matcher — `Read` (Claude Code), + `read_file|read_many_files` (Gemini), `read_file` (Hermes); Codex reads are + mostly shell commands, so coverage there is best-effort. The hook runs + `bdrive read-log`, which reads the hook JSON on stdin, extracts + `tool_input.file_path`, and — iff the path is inside the mount and not + ignored — appends `{path, time}` to a spool (`reads.jsonl`, `O_APPEND`, + capped at ~10k lines) in the volume store. No network in the hook path. +2. **Flush**: the sync cycle (daemon remote tick and one-shot `bdrive sync`) + drains the spool best-effort to `POST /api/p//reads` — a new + device-token-authenticated endpoint that records each entry as + `kind=agent, actor=`. Offline → the spool just waits; a failed + flush never fails the cycle. `remote/http.go` gains the client call as an + optional capability interface (the `PutSigner` pattern) so `file://` + test remotes simply don't report. +3. Session attribution rides along free: the persisted session note is active + during the same turn, so the hub can tag agent read batches with the same + session id the writes carry (kept server-side; surfaced later if wanted). + +## Privacy & config + +```json +"reads": { "enabled": true, "retention_days": 400 } +``` + +- Default **on** (aggregate-only data; a hub admin can disable it). +- Exposed data is only ever: per-path counts by kind, distinct-reader count, + last-read time. Who-read-what is not queryable through any API, including + admin APIs — the actor column exists solely to make `readers` honest. +- Heat is member-visible (it helps everyone find the good docs); the Insights + panel is admin/org-owner only, matching the saved intent. + +## Testing + +- **Unit** (`reads_test.go`): debounce window, retention fold (daily → all-time), + distinct-reader counting, nil-ledger no-ops, flush-on-close. +- **Conformance**: ReadRepo ops across file/sqlite/postgres backends. +- **Handler**: render/file/download/share record with the right kind; `/blob` + and `/store/*` do not; non-member 403 records nothing; `heat` respects + prefix + window; heat 404s in volume mode. +- **Client** (phase 3): spool append from real hook JSON shapes (reuse the + fixtures from `agenthooks_test.go`), flush drains + survives offline, + cycle never fails on report errors. + +## Phasing + +1. **Ledger + heat** — ReadRepo (file/SQL), ReadLedger, recording sites, + `/heat`, config block, folder heat dots + file read counts. Ships alone; + human/share data starts accruing immediately. +2. **Insights quadrant** — admin panel, danger-zone list. Pure frontend + the + existing endpoints. +3. **Agent reads** — `bdrive read-log`, hook matchers, spool, `POST /reads`, + human/agent split in the UI. Depends on 1. + +Phase 1 is the prerequisite for everything and is deliberately boring: one +repo, one service, three record calls, one endpoint, two UI touches. + +Phase 3 is the point of the feature, not tail work: human view counts are a +commodity (every Confluence app has them); *agent* read visibility is the +part nobody else can build. Phases are ordered by dependency, not value — +ship 1 and 3 before polishing 2 if time is short. diff --git a/internal/webapp/db.go b/internal/webapp/db.go index bb2f2cc..fa75ca3 100644 --- a/internal/webapp/db.go +++ b/internal/webapp/db.go @@ -18,6 +18,7 @@ type MetaStore interface { Orgs() OrgRepo Shares() ShareRepo Devices() DeviceRepo + Reads() ReadRepo Close() error } @@ -56,3 +57,13 @@ type DeviceRepo interface { Load() ([]DeviceInfo, error) Put(d DeviceInfo) error } + +// ReadRepo persists read-telemetry buckets (see ReadStat). Unlike the other +// repos it is batch-oriented: reads are telemetry, and the ledger flushes many +// dirty buckets at once — one file rewrite / one SQL transaction per flush, +// not one write per bucket. +type ReadRepo interface { + Load() ([]ReadStat, error) + PutBatch(stats []ReadStat) error // upsert by (project, path, day, kind, actor) + DeleteBatch(keys []ReadStatKey) error +} diff --git a/internal/webapp/db_conformance_test.go b/internal/webapp/db_conformance_test.go index 75942b2..8b0548a 100644 --- a/internal/webapp/db_conformance_test.go +++ b/internal/webapp/db_conformance_test.go @@ -55,7 +55,7 @@ func metaBackends(t *testing.T) []metaBackend { t.Fatalf("postgres reset: %v", err) } defer db.Close() - db.Exec(`DROP TABLE IF EXISTS accounts, tokens, auth_policy, projects, orgs, org_members, invites, shares, devices`) + db.Exec(`DROP TABLE IF EXISTS accounts, tokens, auth_policy, projects, orgs, org_members, invites, shares, devices, read_stats`) }, open: func(t *testing.T) MetaStore { s, err := OpenSQLStore("pgx", dsn) @@ -176,6 +176,17 @@ func TestMetaStoreConformance(t *testing.T) { } devices.Observe(DeviceInfo{ID: "d1", Name: "laptop", OS: "mac", User: "dev@x.io", IP: "1.2.3.4"}) + reads, err := NewReadLedger(st.Reads(), 0) + if err != nil { + t.Fatal(err) + } + reads.Record(p1.ID, "handbook.md", ReadKindHuman, "dev@x.io") + reads.Record(p1.ID, "handbook.md", ReadKindHuman, "boss@x.io") + reads.Record(p1.ID, "wiki/deep.md", ReadKindAgent, "d1") + if err := reads.Close(); err != nil { + t.Fatal(err) + } + if err := st.Close(); err != nil { t.Fatal(err) } @@ -236,6 +247,21 @@ func TestMetaStoreConformance(t *testing.T) { if !ok || d.Name != "laptop" || d.IP != "1.2.3.4" { t.Fatalf("device lost across reload: %+v", d) } + + reads2, err := NewReadLedger(st2.Reads(), 0) + if err != nil { + t.Fatal(err) + } + heat := reads2.Heat(p1.ID, "", time.Time{}) + if e := heat["handbook.md"]; e.Human != 2 || e.Readers != 2 { + t.Fatalf("read buckets lost across reload: %+v", e) + } + if e := heat["wiki/deep.md"]; e.Agent != 1 || e.Readers != 0 { + t.Fatalf("agent read bucket lost across reload: %+v", e) + } + if sub := reads2.Heat(p1.ID, "wiki", time.Time{}); len(sub) != 1 { + t.Fatalf("prefix heat = %+v, want only wiki/deep.md", sub) + } }) } } diff --git a/internal/webapp/db_file.go b/internal/webapp/db_file.go index 8e8e4b1..d2cae67 100644 --- a/internal/webapp/db_file.go +++ b/internal/webapp/db_file.go @@ -61,6 +61,7 @@ type fileMetaStore struct { orgs *fileOrgRepo shares *fileShareRepo devices *fileDeviceRepo + reads *fileReadRepo } // OpenFileStore builds the file backend over dir, using the historical @@ -72,6 +73,7 @@ func OpenFileStore(dir string) (MetaStore, error) { orgs: newFileOrgRepo(filepath.Join(dir, "orgs.json")), shares: newFileShareRepo(filepath.Join(dir, "shares.json")), devices: newFileDeviceRepo(filepath.Join(dir, "devices.json")), + reads: newFileReadRepo(filepath.Join(dir, "reads.json")), }, nil } @@ -80,6 +82,7 @@ func (s *fileMetaStore) Projects() ProjectRepo { return s.projects } func (s *fileMetaStore) Orgs() OrgRepo { return s.orgs } func (s *fileMetaStore) Shares() ShareRepo { return s.shares } func (s *fileMetaStore) Devices() DeviceRepo { return s.devices } +func (s *fileMetaStore) Reads() ReadRepo { return s.reads } func (s *fileMetaStore) Close() error { return nil } // ---- accounts (auth.json: users + tokens + policy) ---- @@ -419,3 +422,72 @@ func (r *fileDeviceRepo) Put(d DeviceInfo) error { r.byID[d.ID] = d return r.write() } + +// ---- reads (reads.json) ---- + +type fileReadRepo struct { + path string + mu sync.Mutex + byKey map[ReadStatKey]ReadStat +} + +func newFileReadRepo(path string) *fileReadRepo { + return &fileReadRepo{path: path, byKey: map[ReadStatKey]ReadStat{}} +} + +func (r *fileReadRepo) Load() ([]ReadStat, error) { + r.mu.Lock() + defer r.mu.Unlock() + var f struct { + Reads []ReadStat `json:"reads"` + } + if _, err := readJSONFile(r.path, &f); err != nil { + return nil, err + } + r.byKey = map[ReadStatKey]ReadStat{} + for _, st := range f.Reads { + r.byKey[st.key()] = st + } + return f.Reads, nil +} + +func (r *fileReadRepo) write() error { + var f struct { + Reads []ReadStat `json:"reads"` + } + f.Reads = make([]ReadStat, 0, len(r.byKey)) + for _, st := range r.byKey { + f.Reads = append(f.Reads, st) + } + sort.Slice(f.Reads, func(i, j int) bool { + a, b := f.Reads[i], f.Reads[j] + if a.Path != b.Path { + return a.Path < b.Path + } + return a.Day < b.Day + }) + data, err := json.Marshal(f) // telemetry: compact beats pretty + if err != nil { + return err + } + // 0700 dir: buckets carry actor emails, like auth.json carries accounts. + return writeFileAtomic(r.path, append(data, '\n'), 0o700) +} + +func (r *fileReadRepo) PutBatch(stats []ReadStat) error { + r.mu.Lock() + defer r.mu.Unlock() + for _, st := range stats { + r.byKey[st.key()] = st + } + return r.write() +} + +func (r *fileReadRepo) DeleteBatch(keys []ReadStatKey) error { + r.mu.Lock() + defer r.mu.Unlock() + for _, k := range keys { + delete(r.byKey, k) + } + return r.write() +} diff --git a/internal/webapp/db_sql.go b/internal/webapp/db_sql.go index ed4118a..100d64f 100644 --- a/internal/webapp/db_sql.go +++ b/internal/webapp/db_sql.go @@ -33,6 +33,7 @@ type sqlMetaStore struct { orgs *sqlOrgRepo shares *sqlShareRepo devices *sqlDeviceRepo + reads *sqlReadRepo } // OpenSQLStore opens (and migrates) a SQL metadata store. driver is "sqlite" @@ -66,6 +67,7 @@ func OpenSQLStore(driver, dsn string) (MetaStore, error) { s.orgs = &sqlOrgRepo{s} s.shares = &sqlShareRepo{s} s.devices = &sqlDeviceRepo{s} + s.reads = &sqlReadRepo{s} return s, nil } @@ -74,6 +76,7 @@ func (s *sqlMetaStore) Projects() ProjectRepo { return s.projects } func (s *sqlMetaStore) Orgs() OrgRepo { return s.orgs } func (s *sqlMetaStore) Shares() ShareRepo { return s.shares } func (s *sqlMetaStore) Devices() DeviceRepo { return s.devices } +func (s *sqlMetaStore) Reads() ReadRepo { return s.reads } func (s *sqlMetaStore) Close() error { return s.db.Close() } // q rebinds ?-placeholders to $1,$2,… for Postgres; SQLite keeps ?. @@ -153,6 +156,11 @@ func (s *sqlMetaStore) migrate() error { `CREATE TABLE IF NOT EXISTS devices ( id TEXT PRIMARY KEY, name TEXT NOT NULL DEFAULT '', os TEXT NOT NULL DEFAULT '', user_email TEXT NOT NULL DEFAULT '', ip TEXT NOT NULL DEFAULT '', last_seen TEXT NOT NULL DEFAULT '')`, + `CREATE TABLE IF NOT EXISTS read_stats ( + project TEXT NOT NULL, path TEXT NOT NULL, day TEXT NOT NULL DEFAULT '', + kind TEXT NOT NULL, actor TEXT NOT NULL, + count INTEGER NOT NULL DEFAULT 0, last TEXT NOT NULL DEFAULT '', + PRIMARY KEY (project, path, day, kind, actor))`, } for _, st := range stmts { if _, err := s.db.Exec(st); err != nil { @@ -462,3 +470,59 @@ func (r *sqlDeviceRepo) Put(d DeviceInfo) error { ip=excluded.ip, last_seen=excluded.last_seen`, d.ID, d.Name, d.OS, d.User, d.IP, tenc(d.LastSeen)) } + +// ---- reads ---- + +type sqlReadRepo struct{ s *sqlMetaStore } + +func (r *sqlReadRepo) Load() ([]ReadStat, error) { + rows, err := r.s.db.Query(`SELECT project, path, day, kind, actor, count, last FROM read_stats`) + if err != nil { + return nil, err + } + defer rows.Close() + var out []ReadStat + for rows.Next() { + var st ReadStat + var last string + if err := rows.Scan(&st.Project, &st.Path, &st.Day, &st.Kind, &st.Actor, &st.Count, &last); err != nil { + return nil, err + } + st.Last = tdec(last) + out = append(out, st) + } + return out, rows.Err() +} + +func (r *sqlReadRepo) PutBatch(stats []ReadStat) error { + tx, err := r.s.db.Begin() + if err != nil { + return err + } + defer tx.Rollback() + for _, st := range stats { + if _, err := tx.Exec(r.s.q(`INSERT INTO read_stats (project,path,day,kind,actor,count,last) + VALUES (?,?,?,?,?,?,?) + ON CONFLICT(project,path,day,kind,actor) DO UPDATE SET count=excluded.count, last=excluded.last`), + st.Project, st.Path, st.Day, st.Kind, st.Actor, st.Count, tenc(st.Last)); err != nil { + return err + } + } + return tx.Commit() +} + +func (r *sqlReadRepo) DeleteBatch(keys []ReadStatKey) error { + tx, err := r.s.db.Begin() + if err != nil { + return err + } + defer tx.Rollback() + for _, k := range keys { + if _, err := tx.Exec(r.s.q(`DELETE FROM read_stats + WHERE project = ? AND path = ? AND day = ? AND kind = ? AND actor = ?`), + k.Project, k.Path, k.Day, k.Kind, k.Actor); err != nil { + return err + } + } + return tx.Commit() +} diff --git a/internal/webapp/reads.go b/internal/webapp/reads.go new file mode 100644 index 0000000..d0b5d13 --- /dev/null +++ b/internal/webapp/reads.go @@ -0,0 +1,411 @@ +package webapp + +import ( + "context" + "encoding/json" + "fmt" + "io" + "log" + "net/http" + "strconv" + "strings" + "sync" + "time" +) + +// Read telemetry: who consumes what, aggregated. Together with the write +// history the journals already carry, this completes the read×write matrix — +// heavily-read + long-unwritten is the danger zone an admin should fix first +// (see docs/design/read-heatmap.md). +// +// What counts as a read: viewer file/render/download hits (human), share-link +// hits (share), and agent tool reads reported by syncing devices (agent). +// /store/* sync traffic is replication, not reading, and is never counted; +// history /blob views are spelunking, not consumption, and aren't either. +// +// Privacy: rows are daily aggregation buckets, never an event log. The actor +// column (account email / device id / share token) exists only to count +// distinct readers and never appears in an API response. + +// Read kinds. +const ( + ReadKindHuman = "human" + ReadKindAgent = "agent" + ReadKindShare = "share" +) + +// ReadStat is one aggregation bucket: reads of one path by one actor on one +// day. Day == "" is the all-time fold that survives retention. +type ReadStat struct { + Project string `json:"project"` + Path string `json:"path"` + Day string `json:"day"` // "2006-01-02" UTC, or "" for all-time + Kind string `json:"kind"` + Actor string `json:"actor"` + Count int64 `json:"count"` + Last time.Time `json:"last"` +} + +// ReadStatKey identifies one bucket. +type ReadStatKey struct { + Project, Path, Day, Kind, Actor string +} + +func (s ReadStat) key() ReadStatKey { + return ReadStatKey{s.Project, s.Path, s.Day, s.Kind, s.Actor} +} + +// HeatEntry is the per-path aggregate the heat API returns. Counts only — +// never identities. +type HeatEntry struct { + Human int64 `json:"human,omitempty"` + Agent int64 `json:"agent,omitempty"` + Share int64 `json:"share,omitempty"` + Readers int `json:"readers,omitempty"` // distinct human readers + LastRead time.Time `json:"last_read,omitzero"` +} + +const ( + // readDebounce collapses request storms (reloads, render-then-raw double + // fetches) into visits: repeat reads of a path by the same actor within + // the window don't count again. + readDebounce = 10 * time.Minute + // readFlushEvery throttles persistence; dirty buckets ride in memory + // between flushes, so a crash loses at most this much telemetry. + readFlushEvery = 30 * time.Second + // DefaultReadRetentionDays is how long daily buckets keep per-day + // resolution before folding into the all-time row. + DefaultReadRetentionDays = 400 +) + +// ReadLedger is the in-memory read-telemetry service over a ReadRepo, in the +// mold of DeviceRegistry: reads stay in memory, writes are throttled. There is +// no background goroutine — flushes piggyback on Record calls, and telemetry +// failures never surface to the request that triggered them. +type ReadLedger struct { + repo ReadRepo + retention time.Duration + + mu sync.Mutex + byKey map[ReadStatKey]ReadStat + dirty map[ReadStatKey]bool + pendingDel []ReadStatKey // retention deletions awaiting a successful flush + seen map[ReadStatKey]time.Time // debounce; Day field unused ("") + lastFlush time.Time + warned bool +} + +// NewReadLedger loads the ledger and immediately folds buckets older than the +// retention horizon into their all-time rows. retentionDays <= 0 means the +// default. +func NewReadLedger(repo ReadRepo, retentionDays int) (*ReadLedger, error) { + if retentionDays <= 0 { + retentionDays = DefaultReadRetentionDays + } + l := &ReadLedger{ + repo: repo, + retention: time.Duration(retentionDays) * 24 * time.Hour, + byKey: map[ReadStatKey]ReadStat{}, + dirty: map[ReadStatKey]bool{}, + seen: map[ReadStatKey]time.Time{}, + lastFlush: time.Now(), + } + stats, err := repo.Load() + if err != nil { + return nil, err + } + for _, st := range stats { + l.byKey[st.key()] = st + } + // Fold anything past the retention horizon right away. A failed persist + // is not a boot failure — the fold stays dirty and later flushes retry. + l.mu.Lock() + l.compactLocked() + if err := l.persistLocked(); err != nil { + log.Printf("beardrive: read telemetry compact failed (will retry): %v", err) + } + l.mu.Unlock() + return l, nil +} + +// OpenReadLedger loads the file-backed ledger at path. +func OpenReadLedger(path string, retentionDays int) (*ReadLedger, error) { + return NewReadLedger(newFileReadRepo(path), retentionDays) +} + +// Record counts one read. Nil-safe and never fails: telemetry must not break +// the page view (or sync cycle) that triggered it. +func (l *ReadLedger) Record(project, path, kind, actor string) { + if l == nil || project == "" || path == "" { + return + } + now := time.Now().UTC() + l.mu.Lock() + defer l.mu.Unlock() + visit := ReadStatKey{Project: project, Path: path, Kind: kind, Actor: actor} + if t, ok := l.seen[visit]; ok && now.Sub(t) < readDebounce { + return + } + l.seen[visit] = now + key := visit + key.Day = now.Format("2006-01-02") + st := l.byKey[key] + st.Project, st.Path, st.Day, st.Kind, st.Actor = project, path, key.Day, kind, actor + st.Count++ + st.Last = now + l.byKey[key] = st + l.dirty[key] = true + if now.Sub(l.lastFlush) >= readFlushEvery { + l.flushLocked() + } +} + +// Heat aggregates reads per path for one project. since bounds the window +// (zero = all time, including retention folds); prefix "" means the whole +// project, otherwise paths under "/". +func (l *ReadLedger) Heat(project, prefix string, since time.Time) map[string]HeatEntry { + if l == nil { + return nil + } + sinceDay := "" + if !since.IsZero() { + sinceDay = since.UTC().Format("2006-01-02") + } + prefix = strings.TrimSuffix(prefix, "/") + out := map[string]HeatEntry{} + humans := map[string]map[string]bool{} // path → distinct human actors + l.mu.Lock() + defer l.mu.Unlock() + for key, st := range l.byKey { + if key.Project != project { + continue + } + if prefix != "" && !strings.HasPrefix(key.Path, prefix+"/") { + continue + } + if key.Day == "" { + if sinceDay != "" { + continue // all-time fold is older than any windowed query + } + } else if key.Day < sinceDay { + continue + } + e := out[key.Path] + switch key.Kind { + case ReadKindHuman: + e.Human += st.Count + set := humans[key.Path] + if set == nil { + set = map[string]bool{} + humans[key.Path] = set + } + set[key.Actor] = true + case ReadKindAgent: + e.Agent += st.Count + case ReadKindShare: + e.Share += st.Count + } + if st.Last.After(e.LastRead) { + e.LastRead = st.Last + } + out[key.Path] = e + } + for p, set := range humans { + e := out[p] + e.Readers = len(set) + out[p] = e + } + return out +} + +// Close flushes any pending buckets. +func (l *ReadLedger) Close() error { + if l == nil { + return nil + } + l.mu.Lock() + defer l.mu.Unlock() + l.flushLocked() + if n := len(l.dirty); n > 0 { + return fmt.Errorf("read ledger: flush failed, %d buckets pending", n) + } + return nil +} + +// flushLocked persists dirty buckets (and, once a day, retention folds), +// pruning the debounce map along the way. Failures keep the buckets dirty for +// the next attempt and log once — telemetry never breaks a request. +func (l *ReadLedger) flushLocked() { + now := time.Now() + l.lastFlush = now + for k, t := range l.seen { + if now.Sub(t) >= readDebounce { + delete(l.seen, k) + } + } + l.compactLocked() + if err := l.persistLocked(); err != nil { + if !l.warned { + l.warned = true + log.Printf("beardrive: read telemetry flush failed (will retry): %v", err) + } + } else { + l.warned = false + } +} + +// compactLocked folds daily buckets older than the retention horizon into +// their all-time rows, queueing the daily rows for deletion. Callers hold mu. +func (l *ReadLedger) compactLocked() { + horizon := time.Now().UTC().Add(-l.retention).Format("2006-01-02") + for key, st := range l.byKey { + if key.Day == "" || key.Day >= horizon { + continue + } + fold := key + fold.Day = "" + agg := l.byKey[fold] + agg.Project, agg.Path, agg.Kind, agg.Actor = st.Project, st.Path, st.Kind, st.Actor + agg.Day = "" + agg.Count += st.Count + if st.Last.After(agg.Last) { + agg.Last = st.Last + } + l.byKey[fold] = agg + l.dirty[fold] = true + delete(l.byKey, key) + delete(l.dirty, key) + l.pendingDel = append(l.pendingDel, key) + } +} + +// persistLocked writes queued deletions and dirty buckets through the repo. +// Callers hold mu. Both queues survive a failure so the next flush retries — +// dropping a deletion would resurrect folded rows on the next load and +// double-count them. +func (l *ReadLedger) persistLocked() error { + if len(l.pendingDel) > 0 { + if err := l.repo.DeleteBatch(l.pendingDel); err != nil { + return err + } + l.pendingDel = nil + } + if len(l.dirty) == 0 { + return nil + } + batch := make([]ReadStat, 0, len(l.dirty)) + for key := range l.dirty { + batch = append(batch, l.byKey[key]) + } + if err := l.repo.PutBatch(batch); err != nil { + return err + } + l.dirty = map[ReadStatKey]bool{} + return nil +} + +// ---- server integration ---- + +// ctxProjectKey carries the resolved project id from the proj() route +// resolver to handlers that record reads. +type ctxProjectKey struct{} + +func withProjectID(r *http.Request, id string) *http.Request { + return r.WithContext(context.WithValue(r.Context(), ctxProjectKey{}, id)) +} + +func projectID(r *http.Request) string { + id, _ := r.Context().Value(ctxProjectKey{}).(string) + return id +} + +// recordRead counts a human read of path for the request's project. No-op +// outside hub mode (no project id) or when read tracking is off. +func (s *Server) recordRead(r *http.Request, path string) { + if s.Reads == nil { + return + } + project := projectID(r) + if project == "" { + return + } + actor := s.requestUser(r).Email + if actor == "" { + actor = "anonymous" + } + s.Reads.Record(project, path, ReadKindHuman, actor) +} + +// handleHeat serves per-path read aggregates: ?prefix= bounds to a folder, +// ?days= bounds the window (default 30, 0 = all time). Counts only — actor +// identities never leave the server. +func (s *Server) handleHeat(v *volume, w http.ResponseWriter, r *http.Request) { + if s.Reads == nil { + http.Error(w, "read tracking is not enabled on this server", http.StatusNotFound) + return + } + _ = v + q := r.URL.Query() + days := 30 + if raw := q.Get("days"); raw != "" { + var err error + if days, err = strconv.Atoi(raw); err != nil || days < 0 { + http.Error(w, "invalid days", http.StatusBadRequest) + return + } + } + var since time.Time + if days > 0 { + since = time.Now().UTC().AddDate(0, 0, -days) + } + entries := s.Reads.Heat(projectID(r), q.Get("prefix"), since) + out := map[string]any{"entries": entries} + if !since.IsZero() { + out["since"] = since.Format("2006-01-02") + } + writeJSON(w, out) +} + +// handleReadReport ingests agent reads from a syncing device: the client's +// read spool, drained best-effort at sync time. Requires a device identity — +// the device id is the actor, so reads count as agent traffic. +func (s *Server) handleReadReport(v *volume, w http.ResponseWriter, r *http.Request) { + if s.Reads == nil { + http.Error(w, "read tracking is not enabled on this server", http.StatusNotFound) + return + } + _ = v + device := r.Header.Get("X-Bdrive-Device") + if device == "" { + http.Error(w, "agent read reports need a device identity", http.StatusBadRequest) + return + } + var req struct { + Reads []struct { + Path string `json:"path"` + // Time is accepted for forward compatibility but buckets use + // server time: client clocks are unreliable and late flushes are + // telemetry noise, not data loss. + Time time.Time `json:"time,omitzero"` + } `json:"reads"` + } + if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&req); err != nil { + http.Error(w, "bad request: "+err.Error(), http.StatusBadRequest) + return + } + if len(req.Reads) > 4096 { + http.Error(w, "too many reads in one report", http.StatusBadRequest) + return + } + s.observeDevice(r) + project := projectID(r) + n := 0 + for _, e := range req.Reads { + if e.Path == "" || strings.Contains(e.Path, "..") { + continue + } + s.Reads.Record(project, e.Path, ReadKindAgent, device) + n++ + } + writeJSON(w, map[string]any{"accepted": n}) +} diff --git a/internal/webapp/reads_test.go b/internal/webapp/reads_test.go new file mode 100644 index 0000000..ed4d6b8 --- /dev/null +++ b/internal/webapp/reads_test.go @@ -0,0 +1,236 @@ +package webapp + +import ( + "bytes" + "encoding/json" + "net/http" + "net/http/httptest" + "path/filepath" + "testing" + "time" +) + +// doHdr is do() with request headers (device identity on read reports). +func doHdr(t *testing.T, h http.Handler, method, url string, body any, hdr map[string]string) *httptest.ResponseRecorder { + t.Helper() + data, err := json.Marshal(body) + if err != nil { + t.Fatal(err) + } + req := httptest.NewRequest(method, url, bytes.NewReader(data)) + for k, v := range hdr { + req.Header.Set(k, v) + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + return rec +} + +func openTestLedger(t *testing.T, retentionDays int) (*ReadLedger, *fileReadRepo) { + t.Helper() + repo := newFileReadRepo(filepath.Join(t.TempDir(), "reads.json")) + l, err := NewReadLedger(repo, retentionDays) + if err != nil { + t.Fatal(err) + } + return l, repo +} + +func TestReadLedgerDebounce(t *testing.T) { + l, _ := openTestLedger(t, 0) + // A reload storm and the render-then-raw double fetch are one visit… + l.Record("p-1", "a.md", ReadKindHuman, "alice@x.io") + l.Record("p-1", "a.md", ReadKindHuman, "alice@x.io") + l.Record("p-1", "a.md", ReadKindHuman, "alice@x.io") + // …but a different actor, kind, or path counts on its own. + l.Record("p-1", "a.md", ReadKindHuman, "bob@x.io") + l.Record("p-1", "a.md", ReadKindAgent, "alice@x.io") + l.Record("p-1", "b.md", ReadKindHuman, "alice@x.io") + heat := l.Heat("p-1", "", time.Time{}) + if e := heat["a.md"]; e.Human != 2 || e.Agent != 1 || e.Readers != 2 { + t.Fatalf("a.md = %+v, want human 2, agent 1, readers 2", e) + } + if e := heat["b.md"]; e.Human != 1 || e.Readers != 1 { + t.Fatalf("b.md = %+v", e) + } + if heat["a.md"].LastRead.IsZero() { + t.Fatal("last_read not set") + } +} + +func TestReadLedgerWindow(t *testing.T) { + l, repo := openTestLedger(t, 0) + l.Record("p-1", "a.md", ReadKindHuman, "alice@x.io") + if err := l.Close(); err != nil { + t.Fatal(err) + } + // An old bucket inside retention: counted all-time, outside a 7-day window. + old := ReadStat{Project: "p-1", Path: "a.md", Day: "2026-01-01", Kind: ReadKindHuman, + Actor: "carol@x.io", Count: 5, Last: time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)} + if err := repo.PutBatch([]ReadStat{old}); err != nil { + t.Fatal(err) + } + l2, err := NewReadLedger(repo, 0) + if err != nil { + t.Fatal(err) + } + if e := l2.Heat("p-1", "", time.Time{})["a.md"]; e.Human != 6 || e.Readers != 2 { + t.Fatalf("all-time = %+v, want human 6, readers 2", e) + } + week := time.Now().UTC().AddDate(0, 0, -7) + if e := l2.Heat("p-1", "", week)["a.md"]; e.Human != 1 || e.Readers != 1 { + t.Fatalf("windowed = %+v, want only today's read", e) + } +} + +func TestReadLedgerRetentionFold(t *testing.T) { + repo := newFileReadRepo(filepath.Join(t.TempDir(), "reads.json")) + seed := []ReadStat{ + {Project: "p-1", Path: "a.md", Day: "2020-01-01", Kind: ReadKindHuman, Actor: "alice@x.io", Count: 3, + Last: time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)}, + {Project: "p-1", Path: "a.md", Day: "2020-01-02", Kind: ReadKindHuman, Actor: "alice@x.io", Count: 2, + Last: time.Date(2020, 1, 2, 0, 0, 0, 0, time.UTC)}, + {Project: "p-1", Path: "a.md", Day: time.Now().UTC().Format("2006-01-02"), Kind: ReadKindHuman, + Actor: "alice@x.io", Count: 1, Last: time.Now().UTC()}, + } + if err := repo.PutBatch(seed); err != nil { + t.Fatal(err) + } + l, err := NewReadLedger(repo, 30) + if err != nil { + t.Fatal(err) + } + // All-time totals survive the fold; per-day resolution ages out. + if e := l.Heat("p-1", "", time.Time{})["a.md"]; e.Human != 6 || e.Readers != 1 { + t.Fatalf("after fold = %+v, want human 6, readers 1", e) + } + if err := l.Close(); err != nil { + t.Fatal(err) + } + rows, err := repo.Load() + if err != nil { + t.Fatal(err) + } + var folds, dailies int + for _, st := range rows { + if st.Day == "" { + folds++ + if st.Count != 5 { + t.Fatalf("fold count = %d, want 5", st.Count) + } + } else { + dailies++ + } + } + if folds != 1 || dailies != 1 { + t.Fatalf("rows after fold: %d folds, %d dailies; want 1 and 1 (%+v)", folds, dailies, rows) + } + // Reloading must not double-count: the fold replaced the old rows. + l2, err := NewReadLedger(repo, 30) + if err != nil { + t.Fatal(err) + } + if e := l2.Heat("p-1", "", time.Time{})["a.md"]; e.Human != 6 { + t.Fatalf("after reload = %+v, want human still 6", e) + } +} + +func TestReadLedgerNil(t *testing.T) { + var l *ReadLedger + l.Record("p-1", "a.md", ReadKindHuman, "x") // must not panic + if l.Heat("p-1", "", time.Time{}) != nil { + t.Fatal("nil ledger heat should be nil") + } + if err := l.Close(); err != nil { + t.Fatal(err) + } +} + +// TestHeatAPI drives the recording sites and the heat/report endpoints +// through the real handler: renders and downloads count (debounced), the +// store sync proxy and history blob views never count, and device-reported +// agent reads land as agent traffic. +func TestHeatAPI(t *testing.T) { + srv, p, root := newHub(t, false, nil) + f := newFakeRemoteAt(t, filepath.Join(root, p.ID)) + f.putAs("dev1", "alice@x.io", "Alice", "wiki/plan.md", "# plan") + f.putAs("dev1", "alice@x.io", "Alice", "top.md", "top") + var err error + srv.Reads, err = OpenReadLedger(filepath.Join(t.TempDir(), "reads.json"), 0) + if err != nil { + t.Fatal(err) + } + h := srv.Handler() + base := "/api/p/" + p.ID + "/" + + // render + raw fetch of the same file = one visit (debounced) + for _, u := range []string{base + "render?path=wiki/plan.md", base + "file?path=wiki/plan.md"} { + if rec := do(t, h, "GET", u, nil); rec.Code != 200 { + t.Fatalf("GET %s: %d %s", u, rec.Code, rec.Body) + } + } + // store proxy traffic is replication, not reading + do(t, h, "GET", base+"store/list?prefix=journal/", nil) + do(t, h, "GET", base+"store/object?key=journal/dev1.jsonl", nil) + + rec := do(t, h, "GET", base+"heat", nil) + if rec.Code != 200 { + t.Fatalf("heat: %d %s", rec.Code, rec.Body) + } + var out struct { + Entries map[string]HeatEntry `json:"entries"` + Since string `json:"since"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil { + t.Fatal(err) + } + if e := out.Entries["wiki/plan.md"]; e.Human != 1 || e.Readers != 1 { + t.Fatalf("plan.md heat = %+v, want one human visit", e) + } + if len(out.Entries) != 1 || out.Since == "" { + t.Fatalf("heat = %+v; store traffic must not count", out) + } + + // an agent device reports its local tool reads + report := map[string]any{ + "reads": []map[string]string{{"path": "wiki/plan.md"}, {"path": "top.md"}, {"path": "../evil"}}, + } + rec = doHdr(t, h, "POST", base+"reads", report, map[string]string{"X-Bdrive-Device": "dev1"}) + if rec.Code != 200 { + t.Fatalf("report: %d %s", rec.Code, rec.Body) + } + var acc struct { + Accepted int `json:"accepted"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &acc); err != nil { + t.Fatal(err) + } + if acc.Accepted != 2 { + t.Fatalf("accepted = %d, want 2 (traversal path dropped)", acc.Accepted) + } + // …and without a device identity the report is rejected + if rec := doHdr(t, h, "POST", base+"reads", report, nil); rec.Code != 400 { + t.Fatalf("device-less report: %d, want 400", rec.Code) + } + + rec = do(t, h, "GET", base+"heat?prefix=wiki", nil) + if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil { + t.Fatal(err) + } + if e := out.Entries["wiki/plan.md"]; e.Human != 1 || e.Agent != 1 { + t.Fatalf("plan.md after report = %+v, want human 1 + agent 1", e) + } + if _, ok := out.Entries["top.md"]; ok { + t.Fatal("prefix filter leaked top.md") + } + + if rec := do(t, h, "GET", base+"heat?days=x", nil); rec.Code != 400 { + t.Fatalf("bad days: %d, want 400", rec.Code) + } + + // a hub without read tracking 404s cleanly + srv.Reads = nil + if rec := do(t, h, "GET", base+"heat", nil); rec.Code != 404 { + t.Fatalf("disabled heat: %d, want 404", rec.Code) + } +} diff --git a/internal/webapp/server.go b/internal/webapp/server.go index c9b316d..59c10a5 100644 --- a/internal/webapp/server.go +++ b/internal/webapp/server.go @@ -71,6 +71,9 @@ type Server struct { Devices *DeviceRegistry // Shares, when set, enables public share links (/s/). Shares *ShareDB + // Reads, when set, aggregates read telemetry (viewer, share, and agent + // reads) for the heat API. Nil means read tracking is off. + Reads *ReadLedger // Orgs, when set, walls projects off by organization membership. Orgs *OrgDB // Quota, when set, enforces plan limits (managed deployments). Nil @@ -305,7 +308,9 @@ func (s *Server) Handler() http.Handler { http.Error(w, "you are not a member of this project's organization", http.StatusForbidden) return } - h(v, w, r) + // Read recording (and anything else downstream) finds the project + // id in the context; membership has already passed at this point. + h(v, w, withProjectID(r, id)) } } @@ -348,6 +353,8 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc("GET /api/p/{project}/history", proj(s.handleHistory)) mux.HandleFunc("GET /api/p/{project}/blob", proj(s.handleBlob)) + mux.HandleFunc("GET /api/p/{project}/heat", proj(s.handleHeat)) + mux.HandleFunc("POST /api/p/{project}/reads", proj(s.handleReadReport)) mux.HandleFunc("POST /api/p/{project}/shares", proj(s.handleShareCreate)) mux.HandleFunc("GET /api/p/{project}/shares", proj(s.handleShareList)) mux.HandleFunc("DELETE /api/shares/{token}", s.handleShareRevoke) @@ -429,7 +436,8 @@ func (s *Server) handleConfig(w http.ResponseWriter, r *http.Request) { "upload": map[string]any{ "enabled": s.Upload.Enabled, }, - "auth": auth, + "auth": auth, + "reads": map[string]any{"enabled": s.Reads != nil}, } if me.Email != "" { out["me"] = map[string]string{"email": me.Email, "name": me.Name} @@ -607,12 +615,15 @@ func lookup(v *volume, r *http.Request) (string, FileInfo, int, error) { return p, fi, 0, nil } -func serveBlob(v *volume, w http.ResponseWriter, r *http.Request, attach bool) { +func (s *Server) serveBlob(v *volume, w http.ResponseWriter, r *http.Request, attach bool) { p, fi, code, err := lookup(v, r) if err != nil { http.Error(w, err.Error(), code) return } + // Count the read before the ETag check: a 304 render is still a person + // reading the file, and skipping it would undercount the hottest pages. + s.recordRead(r, p) etag := `"` + fi.Blob + `"` if r.Header.Get("If-None-Match") == etag { w.WriteHeader(http.StatusNotModified) @@ -634,11 +645,11 @@ func serveBlob(v *volume, w http.ResponseWriter, r *http.Request, attach bool) { } func (s *Server) handleFile(v *volume, w http.ResponseWriter, r *http.Request) { - serveBlob(v, w, r, false) + s.serveBlob(v, w, r, false) } func (s *Server) handleDownload(v *volume, w http.ResponseWriter, r *http.Request) { - serveBlob(v, w, r, true) + s.serveBlob(v, w, r, true) } func (s *Server) handleRender(v *volume, w http.ResponseWriter, r *http.Request) { @@ -647,6 +658,7 @@ func (s *Server) handleRender(v *volume, w http.ResponseWriter, r *http.Request) http.Error(w, err.Error(), code) return } + s.recordRead(r, p) rc, err := v.source.Open(r.Context(), p, fi) if err != nil { http.Error(w, fmt.Sprintf("fetch content: %v", err), http.StatusBadGateway) diff --git a/internal/webapp/shares.go b/internal/webapp/shares.go index 2d76959..c7b44ea 100644 --- a/internal/webapp/shares.go +++ b/internal/webapp/shares.go @@ -243,6 +243,9 @@ func (s *Server) handleShared(w http.ResponseWriter, r *http.Request) { http.Error(w, "the shared file no longer exists", http.StatusNotFound) return } + // A share hit is external consumption. Actor is token+IP: one audience + // member reloading is debounced to a visit, distinct visitors still count. + s.Reads.Record(sh.Project, sh.Path, ReadKindShare, sh.Token+"/"+clientIP(r)) // Sandbox everything under /s/: shared content executes in an opaque // origin (scripts allowed — charts in reports — but no cookies, no diff --git a/internal/webapp/static/app.js b/internal/webapp/static/app.js index 58c9797..8b9dbaf 100644 --- a/internal/webapp/static/app.js +++ b/internal/webapp/static/app.js @@ -21,6 +21,8 @@ let currentProject = null; // hub mode: the selected project let apiBase = "/api/"; // volume-scoped endpoint prefix let orgs = []; // hub mode: the orgs this account belongs to let joinedOrgId = null; // org just joined via an invite this page-load +let heatMap = null; // hub: path → 30-day read counts, from the heat API +let heatAt = 0; // when heatMap was last fetched (ms) const fileURL = (p) => apiBase + "file?path=" + encodeURIComponent(p); @@ -148,6 +150,8 @@ function selectProject(p, path) { currentProject = p; expanded = new Set(); // fresh collapse state for the new project's tree treeFirstLoad = true; + heatMap = null; // the old project's heat means nothing here + heatAt = 0; apiBase = "/api/p/" + p.id + "/"; $("vault-name").textContent = p.name; document.title = p.name + " — BearDrive"; @@ -694,6 +698,60 @@ async function refreshTree() { if (currentPath && dirIndex.has(currentPath) && $("content").querySelector(".dirlist")) { renderFolderListing(currentPath); } + refreshHeat(); +} + +/* ---- read heat ---- + 30-day read counts per path from the heat API (hub only). Counts only — + the server never says who read what. Fetched lazily alongside the tree. */ +async function refreshHeat(force) { + if (!(serverConfig.mode === "hub" && currentProject)) return; + if (!(serverConfig.reads && serverConfig.reads.enabled)) return; + if (!force && Date.now() - heatAt < 60000) return; + heatAt = Date.now(); // set before the fetch so failures don't hammer + let out; + try { + out = await getJSON(apiBase + "heat?days=30"); + } catch { return; } // keep the last good heat + heatMap = out.entries || {}; + if (currentPath && dirIndex.has(currentPath) && $("content").querySelector(".dirlist")) { + renderFolderListing(currentPath); + } +} + +/* Heat for one listing entry: a file's own bucket, or the subtree sum for a + folder. Null when there is nothing to show. */ +function heatFor(path, isDir) { + if (!heatMap) return null; + if (!isDir) return heatMap[path] || null; + const agg = { human: 0, agent: 0, share: 0 }; + for (const [p, e] of Object.entries(heatMap)) { + if (!p.startsWith(path + "/")) continue; + agg.human += e.human || 0; + agg.agent += e.agent || 0; + agg.share += e.share || 0; + } + return agg.human || agg.agent || agg.share ? agg : null; +} + +function heatTotal(e) { return (e.human || 0) + (e.agent || 0) + (e.share || 0); } + +function heatText(e) { + const total = heatTotal(e); + if (!total) return ""; + let s = total + (total === 1 ? " read" : " reads"); + if (e.agent) s += " (" + e.agent + " agent)"; + return s; +} + +/* Dot intensity 1–4, log-ish steps: 1–2, 3–9, 10–29, 30+ reads. */ +function heatLevel(e) { + const total = heatTotal(e); + if (!total) return 0; + if (total < 3) return 1; + if (total < 10) return 2; + if (total < 30) return 3; + return 4; } function renderChildren(children) { @@ -865,6 +923,8 @@ function renderFolderListing(p) { const counts = []; if (dirs) counts.push(dirs + (dirs === 1 ? " folder" : " folders")); if (files) counts.push(files + (files === 1 ? " file" : " files")); + const folderHeat = heatFor(p, true); + if (folderHeat) counts.push(heatText(folderHeat) + " in 30 days"); el(wrap, "p", "dl-sub", counts.join(" · ") || "Empty folder"); if (!kids.length) { el(wrap, "div", "dl-empty", "Nothing in this folder yet."); @@ -888,6 +948,12 @@ function renderFolderListing(p) { meta = [c.size ? humanSize(c.size) : "", c.time ? new Date(c.time).toLocaleDateString() : ""] .filter(Boolean).join(" · "); } + const he = heatFor(c.path, !!c.dir); + if (he) { + const dot = el(row, "span", "heatdot lvl" + heatLevel(he)); + dot.title = heatText(he) + " in 30 days"; + meta = heatText(he) + (meta ? " · " + meta : ""); + } el(row, "span", "dl-meta", meta); row.onclick = () => openPath(c.path); row.onkeydown = (e) => { if (e.key === "Enter" || e.key === " ") { e.preventDefault(); row.click(); } }; @@ -964,6 +1030,8 @@ function showMeta(doc) { const parts = []; if (doc.author) parts.push(doc.author + (doc.device ? " on " + doc.device : "")); if (doc.time) parts.push(new Date(doc.time).toLocaleString()); + const he = heatMap && heatMap[doc.path]; + if (he && heatTotal(he)) parts.push(heatText(he) + " / 30d"); $("meta").textContent = parts.join(" · "); } diff --git a/internal/webapp/static/style.css b/internal/webapp/static/style.css index 5e7eee5..c0f5d94 100644 --- a/internal/webapp/static/style.css +++ b/internal/webapp/static/style.css @@ -269,6 +269,12 @@ button, input, a.btn { font-family: inherit; } .dl-row:hover .ticon { color: var(--text-faint); } .dl-name { flex: 1; min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-size: 13.5px; color: var(--text); } .dl-meta { flex: none; font-size: 12px; color: var(--text-faint); font-variant-numeric: tabular-nums; } +/* read-heat dot: intensity grows with 30-day reads (heatLevel in app.js) */ +.heatdot { flex: none; width: 7px; height: 7px; border-radius: 50%; background: var(--accent); } +.heatdot.lvl1 { opacity: .3; } +.heatdot.lvl2 { opacity: .55; } +.heatdot.lvl3 { opacity: .8; } +.heatdot.lvl4 { opacity: 1; box-shadow: 0 0 6px rgba(245, 166, 35, .55); } .dl-empty { padding: 24px 14px; color: var(--text-faint); font-size: 13px; border: 1px dashed var(--border); border-radius: var(--r-card); text-align: center; } .dl-h3 { margin: 28px 0 8px; font-size: 10.5px; text-transform: uppercase; letter-spacing: .07em; color: var(--text-ghost); font-weight: 600; } .dl-hlist { border: 1px solid var(--border); border-radius: var(--r-card); background: var(--bg-side); overflow: hidden; max-width: none; } From 2ce714ac4d6b8f1ef2203a14bdea8b1b6f1d7b6d Mon Sep 17 00:00:00 2001 From: Snow Lee Date: Sat, 11 Jul 2026 14:49:02 -0700 Subject: [PATCH 2/3] =?UTF-8?q?feat(sync):=20agent=20read=20reporting=20?= =?UTF-8?q?=E2=80=94=20read-log,=20spool,=20hub=20report,=20hook=20matcher?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3 of the read heatmap: `bdrive read-log` parses any platform's hook event JSON from stdin and queues in-project file reads (mount-relative, ignore/include-filtered) in a per-volume spool — no network on the hook path. The sync cycle drains the spool best-effort to the hub's new POST /api/p//reads via the remote ReadReporter capability (https backend only); a failed report retries next cycle and never fails or offlines the cycle. `bdrive hooks install` now registers a third hook per platform on its read-tool matcher (claude Read, codex read_file best-effort, gemini read_file|read_many_files, hermes read_file), each idempotent on its own marker so sync-only configs upgrade in place. Agent reads land as agent traffic in the heat view, actor = device id. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01P5cxPQdSGJnjXCYY9GeWXt --- cmd/bdrive/main.go | 1 + cmd/bdrive/readlog.go | 131 +++++++++++++++++++++++++ cmd/bdrive/readlog_test.go | 93 ++++++++++++++++++ internal/agenthooks/agenthooks.go | 89 ++++++++++++----- internal/agenthooks/agenthooks_test.go | 92 ++++++++++++++++- internal/remote/http.go | 24 +++++ internal/remote/remote.go | 14 +++ internal/store/reads.go | 107 ++++++++++++++++++++ internal/store/reads_test.go | 111 +++++++++++++++++++++ internal/syncer/ignore.go | 8 ++ internal/syncer/reads_flow_test.go | 101 +++++++++++++++++++ internal/syncer/syncer.go | 15 +++ 12 files changed, 758 insertions(+), 28 deletions(-) create mode 100644 cmd/bdrive/readlog.go create mode 100644 cmd/bdrive/readlog_test.go create mode 100644 internal/store/reads.go create mode 100644 internal/store/reads_test.go create mode 100644 internal/syncer/reads_flow_test.go diff --git a/cmd/bdrive/main.go b/cmd/bdrive/main.go index 52d5c65..3624ada 100644 --- a/cmd/bdrive/main.go +++ b/cmd/bdrive/main.go @@ -36,6 +36,7 @@ everything keeps working offline; changes sync when the remote is reachable.`, shareCmd(), stopCmd(), syncCmd(), + readLogCmd(), hooksCmd(), statusCmd(), logCmd(), diff --git a/cmd/bdrive/readlog.go b/cmd/bdrive/readlog.go new file mode 100644 index 0000000..b1ca6c8 --- /dev/null +++ b/cmd/bdrive/readlog.go @@ -0,0 +1,131 @@ +package main + +import ( + "encoding/json" + "io" + "path/filepath" + "strings" + + "github.com/spf13/cobra" + + "github.com/runbear-io/beardrive/internal/config" + "github.com/runbear-io/beardrive/internal/store" + "github.com/runbear-io/beardrive/internal/syncer" +) + +// read-log is the agent read hook's command: it runs after every file-read +// tool call, so it must be a fast, silent no-op in every case that isn't +// "a synced file was just read" — no network, no locking, one appended line. +func readLogCmd() *cobra.Command { + return &cobra.Command{ + Use: "read-log [folder]", + Short: "Record agent file reads from a hook event (JSON on stdin)", + Long: `Record which project files an agent just read, from the hook event JSON +piped on stdin (any platform's PostToolUse-style payload: file paths are +found wherever they appear in the event). + +Reads are queued locally in the volume store and drained to the hub on the +next sync, where they show up as agent traffic in the project's read heat. +Registered automatically by "bdrive hooks install"; there is rarely a reason +to run it by hand.`, + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + folder, err := absFolder(args) + if err != nil { + return nil + } + proj, found, err := config.ResolveMount(folder) + if err != nil || !found { + return nil // not a beardrive project: fast no-op + } + data, _ := io.ReadAll(io.LimitReader(cmd.InOrStdin(), 1<<20)) + paths := extractEventPaths(data) + if len(paths) == 0 { + return nil + } + filter, err := syncer.LoadFilter(folder, proj.Include) + if err != nil { + return nil + } + vdir, err := config.VolumeDir(proj.ID) + if err != nil { + return nil + } + st, err := store.Open(vdir) + if err != nil { + return nil + } + for _, p := range paths { + abs := p + if !filepath.IsAbs(abs) { + abs = filepath.Join(folder, p) + } + rel, err := filepath.Rel(folder, abs) + if err != nil || rel == "." || strings.HasPrefix(rel, "..") { + continue // outside the mount + } + rel = filepath.ToSlash(rel) + if filter.Skip(rel) { + continue // not part of the project (ignore/include rules) + } + st.LogRead(rel) // best-effort; the hook must never fail the turn + } + return nil + }, + } +} + +// Keys that carry file paths in the hook payloads of the supported agent +// platforms (Claude Code tool_input.file_path, Gemini/Hermes read_file +// path/absolute_path, multi-read paths arrays). +var ( + eventPathKeys = map[string]bool{"file_path": true, "absolute_path": true, "path": true, "notebook_path": true} + eventPathListKeys = map[string]bool{"paths": true, "file_paths": true} +) + +// extractEventPaths pulls candidate file paths out of an arbitrary hook +// event JSON. The hook only fires on read-tool matchers, so any path-shaped +// field is a read; non-project paths are filtered by the caller. +func extractEventPaths(data []byte) []string { + var root any + if json.Unmarshal(data, &root) != nil { + return nil + } + seen := map[string]bool{} + var out []string + add := func(s string) { + if s != "" && !seen[s] { + seen[s] = true + out = append(out, s) + } + } + var walk func(v any) + walk = func(v any) { + switch t := v.(type) { + case map[string]any: + for k, val := range t { + switch { + case eventPathKeys[k]: + if s, ok := val.(string); ok { + add(s) + } + case eventPathListKeys[k]: + if arr, ok := val.([]any); ok { + for _, it := range arr { + if s, ok := it.(string); ok { + add(s) + } + } + } + } + walk(val) + } + case []any: + for _, it := range t { + walk(it) + } + } + } + walk(root) + return out +} diff --git a/cmd/bdrive/readlog_test.go b/cmd/bdrive/readlog_test.go new file mode 100644 index 0000000..ff0c040 --- /dev/null +++ b/cmd/bdrive/readlog_test.go @@ -0,0 +1,93 @@ +package main + +import ( + "bytes" + "path/filepath" + "strings" + "testing" + + "github.com/runbear-io/beardrive/internal/config" + "github.com/runbear-io/beardrive/internal/store" +) + +// Real hook payload shapes from the supported platforms — read-log must find +// the file paths wherever each platform puts them. +func TestExtractEventPaths(t *testing.T) { + cases := map[string]struct { + payload string + want []string + }{ + "claude read": { + `{"session_id":"abc","hook_event_name":"PostToolUse","tool_name":"Read", + "tool_input":{"file_path":"/proj/wiki/a.md"},"tool_response":{"type":"text"}}`, + []string{"/proj/wiki/a.md"}, + }, + "gemini read_many_files": { + `{"session_id":"g1","tool":{"name":"read_many_files","args":{"paths":["wiki/a.md","wiki/b.md"]}}}`, + []string{"wiki/a.md", "wiki/b.md"}, + }, + "gemini read_file absolute": { + `{"session_id":"g2","tool":{"name":"read_file","args":{"absolute_path":"/proj/notes.md"}}}`, + []string{"/proj/notes.md"}, + }, + "hermes read_file": { + `{"hook_event_name":"post_tool_call","tool_name":"read_file","tool_args":{"path":"docs/x.md"}}`, + []string{"docs/x.md"}, + }, + "duplicates collapse": { + `{"tool_input":{"file_path":"a.md"},"extra":{"file_path":"a.md"}}`, + []string{"a.md"}, + }, + "no paths": { + `{"session_id":"abc","prompt":"hello"}`, + nil, + }, + "not json": { + `plain text`, + nil, + }, + } + for name, c := range cases { + got := extractEventPaths([]byte(c.payload)) + if strings.Join(got, ",") != strings.Join(c.want, ",") { + t.Errorf("%s: paths = %v, want %v", name, got, c.want) + } + } +} + +// End to end through the cobra command: a claude-style event lands in the +// mount's read spool, filtered to project-relative synced paths. +func TestReadLogCommand(t *testing.T) { + t.Setenv("BDRIVE_HOME", t.TempDir()) + folder := t.TempDir() + folder, _ = filepath.EvalSymlinks(folder) + proj, err := config.SaveProject(folder, config.Project{Volume: "wiki"}) + if err != nil { + t.Fatal(err) + } + + c := readLogCmd() + c.SetIn(bytes.NewReader([]byte(`{"session_id":"abc","tool_name":"Read", + "tool_input":{"file_path":"` + folder + `/wiki/a.md"}, + "other":{"file_path":"/somewhere/else/entirely.md"}}`))) + c.SetArgs([]string{folder}) + if err := c.Execute(); err != nil { + t.Fatal(err) + } + + vdir, err := config.VolumeDir(proj.ID) + if err != nil { + t.Fatal(err) + } + st, err := store.Open(vdir) + if err != nil { + t.Fatal(err) + } + evs, err := st.PendingReads() + if err != nil { + t.Fatal(err) + } + if len(evs) != 1 || evs[0].Path != "wiki/a.md" { + t.Fatalf("spool = %+v, want just the in-project read, mount-relative", evs) + } +} diff --git a/internal/agenthooks/agenthooks.go b/internal/agenthooks/agenthooks.go index f6200bd..2cf08a9 100644 --- a/internal/agenthooks/agenthooks.go +++ b/internal/agenthooks/agenthooks.go @@ -13,7 +13,10 @@ // // The hook syncs the project and stamps changes with " session " // (see `bdrive sync --note`), so hub history links every change to the agent -// session that made it. Hooks are fast no-ops outside bdrive projects. +// session that made it. A third hook on each platform's read-tool matcher +// runs `bdrive read-log`, queueing agent file reads for the hub's read +// heatmap (drained on the next sync — the hook itself never touches the +// network). Hooks are fast no-ops outside bdrive projects. package agenthooks import ( @@ -28,8 +31,14 @@ import ( "github.com/runbear-io/beardrive/internal/store" ) -// marker identifies our hooks inside a config, for idempotency and status. -const marker = "bdrive sync" +// Markers identify our hooks inside a config, for idempotency and status. +// The sync and read hooks are separate groups (different matchers), so each +// carries its own marker — re-running install on a config that predates the +// read hook adds just the missing group. +const ( + marker = "bdrive sync" + readMarker = "bdrive read-log" +) // Agent names, in the order they are reported. var Agents = []string{"claude", "codex", "gemini", "hermes"} @@ -53,6 +62,15 @@ func hookCommand(label string) string { `else bdrive sync . >/dev/null 2>&1 || true; fi'` } +// readHookCommand queues agent file reads for the hub's read heatmap: +// `bdrive read-log` parses the hook's stdin JSON itself and only appends to +// a local spool, so this stays cheap enough to run on every read-tool call. +func readHookCommand() string { + return `sh -c '` + + `cd "${CLAUDE_PROJECT_DIR:-.}" && [ -d .bdrive ] && command -v bdrive >/dev/null || exit 0; ` + + `bdrive read-log . >/dev/null 2>&1 || true'` +} + type platform struct { label string // session-note label projectDir string // presence of this dir (project or home) = detected @@ -67,15 +85,17 @@ var platforms = map[string]platform{ projectDir: ".claude", install: func(folder string) (string, bool, error) { return mergeJSONHooks(filepath.Join(folder, ".claude", "settings.json"), - "UserPromptSubmit", "PostToolUse", "Write|Edit|MultiEdit", "claude-code", 30, true) + "UserPromptSubmit", "PostToolUse", "Write|Edit|MultiEdit", "Read", "claude-code", 30, true) }, }, "codex": { label: "codex", projectDir: ".codex", install: func(folder string) (string, bool, error) { + // Codex reads mostly happen through shell commands, so the + // read_file matcher is best-effort coverage. return mergeJSONHooks(filepath.Join(folder, ".codex", "hooks.json"), - "UserPromptSubmit", "PostToolUse", "apply_patch", "codex", 30, false) + "UserPromptSubmit", "PostToolUse", "apply_patch", "read_file", "codex", 30, false) }, note: "run /hooks inside Codex once to trust the project's .codex layer", }, @@ -85,7 +105,7 @@ var platforms = map[string]platform{ install: func(folder string) (string, bool, error) { // Gemini uses its own event names and millisecond timeouts. return mergeJSONHooks(filepath.Join(folder, ".gemini", "settings.json"), - "BeforeAgent", "AfterTool", "write_file|replace|edit", "gemini", 30000, false) + "BeforeAgent", "AfterTool", "write_file|replace|edit", "read_file|read_many_files", "gemini", 30000, false) }, }, "hermes": { @@ -159,10 +179,12 @@ func Install(folder string, agents []string) ([]Result, error) { return out, nil } -// mergeJSONHooks adds the pull + push hook pair to a Claude-style hooks JSON -// file (Claude, Codex, and Gemini all use this shape: hooks. is an -// array of {matcher?, hooks: [{type: "command", ...}]} groups). -func mergeJSONHooks(path, pullEvent, pushEvent, pushMatcher, label string, timeout int, async bool) (string, bool, error) { +// mergeJSONHooks adds the pull + push + read hook trio to a Claude-style +// hooks JSON file (Claude, Codex, and Gemini all use this shape: +// hooks. is an array of {matcher?, hooks: [{type: "command", ...}]} +// groups). Push and read share the tool-use event under different matchers, +// each idempotent on its own marker. +func mergeJSONHooks(path, pullEvent, pushEvent, pushMatcher, readMatcher, label string, timeout int, async bool) (string, bool, error) { root := map[string]any{} if data, err := os.ReadFile(path); err == nil { if err := json.Unmarshal(data, &root); err != nil { @@ -182,18 +204,29 @@ func mergeJSONHooks(path, pullEvent, pushEvent, pushMatcher, label string, timeo "statusMessage": "beardrive: pulling latest files", }}} pushHook := map[string]any{"type": "command", "command": cmd, "timeout": timeout} + readHook := map[string]any{"type": "command", "command": readHookCommand(), "timeout": timeout} if async { pushHook["async"] = true + readHook["async"] = true } push := map[string]any{"matcher": pushMatcher, "hooks": []any{pushHook}} + read := map[string]any{"matcher": readMatcher, "hooks": []any{readHook}} changed := false - for event, group := range map[string]any{pullEvent: pull, pushEvent: push} { - arr, _ := hooks[event].([]any) - if containsMarker(arr) { + for _, g := range []struct { + event string + group map[string]any + marker string + }{ + {pullEvent, pull, marker}, + {pushEvent, push, marker}, + {pushEvent, read, readMarker}, + } { + arr, _ := hooks[g.event].([]any) + if containsMarker(arr, g.marker) { continue } - hooks[event] = append(arr, group) + hooks[g.event] = append(arr, g.group) changed = true } if !changed { @@ -222,17 +255,22 @@ func installHermes(string) (string, bool, error) { root["hooks"] = hooks } cmd := hookCommand("hermes") - groups := map[string]any{ - "pre_llm_call": map[string]any{"command": cmd, "timeout": 30}, - "post_tool_call": map[string]any{"matcher": "write_file|patch", "command": cmd, "timeout": 30}, + groups := []struct { + event string + group map[string]any + marker string + }{ + {"pre_llm_call", map[string]any{"command": cmd, "timeout": 30}, marker}, + {"post_tool_call", map[string]any{"matcher": "write_file|patch", "command": cmd, "timeout": 30}, marker}, + {"post_tool_call", map[string]any{"matcher": "read_file", "command": readHookCommand(), "timeout": 30}, readMarker}, } changed := false - for event, group := range groups { - arr, _ := hooks[event].([]any) - if containsMarker(arr) { + for _, g := range groups { + arr, _ := hooks[g.event].([]any) + if containsMarker(arr, g.marker) { continue } - hooks[event] = append(arr, group) + hooks[g.event] = append(arr, g.group) changed = true } if !changed { @@ -243,11 +281,12 @@ func installHermes(string) (string, bool, error) { }) } -// containsMarker reports whether a hook array already holds one of ours. -// Serializing sidesteps walking every platform's nesting by hand. -func containsMarker(v any) bool { +// containsMarker reports whether a hook array already holds the hook the +// marker identifies. Serializing sidesteps walking every platform's nesting +// by hand. +func containsMarker(v any, m string) bool { data, err := json.Marshal(v) - return err == nil && strings.Contains(string(data), marker) + return err == nil && strings.Contains(string(data), m) } func writeConfig(path string, marshal func() ([]byte, error)) error { diff --git a/internal/agenthooks/agenthooks_test.go b/internal/agenthooks/agenthooks_test.go index 30b1a54..d66d82a 100644 --- a/internal/agenthooks/agenthooks_test.go +++ b/internal/agenthooks/agenthooks_test.go @@ -75,6 +75,10 @@ func TestInstallJSONPlatforms(t *testing.T) { if !strings.Contains(string(raw), `"async":true`) { t.Fatal("claude push hook should be async") } + // …and the read hook, on its own matcher. + if !strings.Contains(string(raw), "bdrive read-log") || !strings.Contains(string(raw), `"matcher":"Read"`) { + t.Fatalf("claude read hook missing: %s", raw) + } // Codex: same schema, its own label and matcher, no async field. cx, _ := json.Marshal(readJSON(t, filepath.Join(folder, ".codex", "hooks.json"))) @@ -84,10 +88,13 @@ func TestInstallJSONPlatforms(t *testing.T) { if strings.Contains(string(cx), "async") { t.Fatal("codex should not get the claude-only async field") } + if !strings.Contains(string(cx), "bdrive read-log") || !strings.Contains(string(cx), `"matcher":"read_file"`) { + t.Fatalf("codex read hook missing: %s", cx) + } // Gemini: its own event names and ms timeout. gm, _ := json.Marshal(readJSON(t, filepath.Join(folder, ".gemini", "settings.json"))) - for _, want := range []string{"BeforeAgent", "AfterTool", "gemini session $s", "30000"} { + for _, want := range []string{"BeforeAgent", "AfterTool", "gemini session $s", "30000", "bdrive read-log", "read_file|read_many_files"} { if !strings.Contains(string(gm), want) { t.Fatalf("gemini hooks missing %q: %s", want, gm) } @@ -114,8 +121,8 @@ func TestInstallIdempotentAndPreserving(t *testing.T) { if _, ok := cfg["permissions"]; !ok { t.Fatal("merge dropped unrelated settings keys") } - if got := len(cfg["hooks"].(map[string]any)["PostToolUse"].([]any)); got != 2 { - t.Fatalf("PostToolUse groups = %d, want user's + ours", got) + if got := len(cfg["hooks"].(map[string]any)["PostToolUse"].([]any)); got != 3 { + t.Fatalf("PostToolUse groups = %d, want user's + our push + our read", got) } // Second install: no change, byte-identical file. @@ -133,6 +140,38 @@ func TestInstallIdempotentAndPreserving(t *testing.T) { } } +// A config from before the read hook existed (sync hooks only) gains just +// the read group on re-install — the sync hooks are not duplicated. +func TestInstallUpgradesSyncOnlyConfig(t *testing.T) { + t.Setenv("HOME", t.TempDir()) + folder := t.TempDir() + old := `{"hooks":{ + "UserPromptSubmit":[{"hooks":[{"type":"command","command":"sh -c 'bdrive sync .'"}]}], + "PostToolUse":[{"matcher":"Write|Edit|MultiEdit","hooks":[{"type":"command","command":"sh -c 'bdrive sync .'"}]}]}}` + os.MkdirAll(filepath.Join(folder, ".claude"), 0o755) + os.WriteFile(filepath.Join(folder, ".claude", "settings.json"), []byte(old), 0o644) + + results, err := Install(folder, []string{"claude"}) + if err != nil { + t.Fatal(err) + } + if !results[0].Changed { + t.Fatal("upgrade install reported unchanged") + } + cfg := readJSON(t, filepath.Join(folder, ".claude", "settings.json")) + hooks := cfg["hooks"].(map[string]any) + if got := len(hooks["UserPromptSubmit"].([]any)); got != 1 { + t.Fatalf("UserPromptSubmit groups = %d, want the old one only", got) + } + if got := len(hooks["PostToolUse"].([]any)); got != 2 { + t.Fatalf("PostToolUse groups = %d, want old push + new read", got) + } + raw, _ := json.Marshal(cfg) + if !strings.Contains(string(raw), "bdrive read-log") { + t.Fatal("upgrade did not add the read hook") + } +} + func TestInstallHermesYAML(t *testing.T) { home := t.TempDir() t.Setenv("HOME", home) @@ -162,9 +201,15 @@ func TestInstallHermesYAML(t *testing.T) { t.Fatalf("hermes missing %s", ev) } } + if got := len(hooks["post_tool_call"].([]any)); got != 2 { + t.Fatalf("hermes post_tool_call groups = %d, want push + read", got) + } if !strings.Contains(string(data), "hermes session $s") { t.Fatal("hermes hook lacks its session-note label") } + if !strings.Contains(string(data), "bdrive read-log") { + t.Fatal("hermes read hook missing") + } // Idempotent. results, _ = Install(t.TempDir(), []string{"hermes"}) @@ -231,6 +276,47 @@ func TestHookCommandExtraction(t *testing.T) { } } +// The read hook must hand the event JSON through to `bdrive read-log` +// untouched — the binary does the parsing, the shell only guards. +func TestReadHookCommand(t *testing.T) { + if _, err := os.Stat("/bin/sh"); err != nil { + t.Skip("no /bin/sh") + } + dir := t.TempDir() + os.MkdirAll(filepath.Join(dir, ".bdrive"), 0o755) + bin := filepath.Join(dir, "bin") + os.MkdirAll(bin, 0o755) + fake := "#!/bin/sh\necho \"$@\" > \"" + dir + "/args.txt\"\ncat > \"" + dir + "/stdin.txt\"\n" + os.WriteFile(filepath.Join(bin, "bdrive"), []byte(fake), 0o755) + + payload := `{"session_id":"abc","tool_name":"Read","tool_input":{"file_path":"/x/wiki/a.md"}}` + sh := "cd " + dir + " && PATH=" + bin + ":$PATH " + readHookCommand() + if err := runShell(t, sh, payload); err != nil { + t.Fatal(err) + } + args, err := os.ReadFile(filepath.Join(dir, "args.txt")) + if err != nil { + t.Fatalf("read hook never called bdrive: %v", err) + } + if !strings.Contains(string(args), "read-log .") { + t.Fatalf("bdrive argv = %q, want read-log .", args) + } + stdin, _ := os.ReadFile(filepath.Join(dir, "stdin.txt")) + if string(stdin) != payload { + t.Fatalf("event JSON not passed through: %q", stdin) + } + + // Outside a bdrive project the hook exits before invoking anything. + os.Remove(filepath.Join(dir, "args.txt")) + os.RemoveAll(filepath.Join(dir, ".bdrive")) + if err := runShell(t, sh, payload); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(dir, "args.txt")); !os.IsNotExist(err) { + t.Fatal("hook invoked bdrive outside a project") + } +} + func runShell(t *testing.T, script, stdin string) error { t.Helper() cmd := exec.Command("/bin/sh", "-c", script) diff --git a/internal/remote/http.go b/internal/remote/http.go index ed80d2b..9504598 100644 --- a/internal/remote/http.go +++ b/internal/remote/http.go @@ -248,4 +248,28 @@ func (b *httpBackend) putViaServer(ctx context.Context, key string, r io.Reader, return nil } +// ReportReads sends the device's queued agent reads to the hub's read +// ledger, where they count as agent traffic (actor = this device). +func (b *httpBackend) ReportReads(ctx context.Context, reads []ReadEvent) error { + body, err := json.Marshal(map[string]any{"reads": reads}) + if err != nil { + return err + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, + b.base+"/api/p/"+b.project+"/reads", bytes.NewReader(body)) + if err != nil { + return err + } + req.Header.Set("Content-Type", "application/json") + resp, err := b.do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return httpError(resp) + } + return nil +} + func (b *httpBackend) Close() error { return nil } diff --git a/internal/remote/remote.go b/internal/remote/remote.go index 4d68152..59d44d2 100644 --- a/internal/remote/remote.go +++ b/internal/remote/remote.go @@ -52,6 +52,20 @@ type Backend interface { Close() error } +// ReadEvent is one agent file read reported to the hub for its read heatmap. +type ReadEvent struct { + Path string `json:"path"` + Time time.Time `json:"time,omitzero"` +} + +// ReadReporter is the optional read-telemetry capability, in the PutSigner +// mold: backends that sync through a hub report the device's agent reads so +// the heat view can split human from agent traffic. Object-store backends +// simply don't implement it — there is no hub to tell. +type ReadReporter interface { + ReportReads(ctx context.Context, reads []ReadEvent) error +} + // Open creates a backend from a remote URL. func Open(ctx context.Context, raw string) (Backend, error) { u, err := url.Parse(raw) diff --git a/internal/store/reads.go b/internal/store/reads.go new file mode 100644 index 0000000..f62e3cd --- /dev/null +++ b/internal/store/reads.go @@ -0,0 +1,107 @@ +package store + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "time" +) + +// The read spool queues agent tool reads observed by `bdrive read-log` (the +// agent read hook) until a sync cycle drains it to the hub, where they count +// as agent traffic in the read heatmap. Hooks only append locally — no +// network on the hook path — and the flush is best-effort: offline, the +// spool just waits. + +// ReadEvent is one observed read of a synced file (mount-relative path). +type ReadEvent struct { + Path string `json:"path"` + Time time.Time `json:"time"` +} + +// readSpoolMax caps the spool: past it new events are dropped rather than +// letting an unreachable hub grow telemetry without bound. +const readSpoolMax = 1 << 20 + +// readReportMax bounds one drained batch to what the hub accepts per report. +const readReportMax = 4096 + +func (s *Store) readSpoolPath() string { return filepath.Join(s.dir, "reads.jsonl") } +func (s *Store) readFlushPath() string { return filepath.Join(s.dir, "reads-flushing.jsonl") } + +// LogRead appends one read event to the spool. Single-line O_APPEND writes +// keep concurrent hook invocations from interleaving. +func (s *Store) LogRead(rel string) error { + if fi, err := os.Stat(s.readSpoolPath()); err == nil && fi.Size() > readSpoolMax { + return nil // spool full: drop, never grow unbounded + } + line, err := json.Marshal(ReadEvent{Path: rel, Time: time.Now().UTC()}) + if err != nil { + return err + } + f, err := os.OpenFile(s.readSpoolPath(), os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644) + if err != nil { + return err + } + defer f.Close() + _, err = f.Write(append(line, '\n')) + return err +} + +// PendingReads returns the queued batch awaiting report, deduplicated by path +// (latest time wins). The spool is rotated aside first, so events logged +// after this call land in a fresh spool; the batch survives until +// ClearPendingReads — a failed report is simply retried next cycle. +func (s *Store) PendingReads() ([]ReadEvent, error) { + if _, err := os.Stat(s.readFlushPath()); os.IsNotExist(err) { + if err := os.Rename(s.readSpoolPath(), s.readFlushPath()); err != nil { + if os.IsNotExist(err) { + return nil, nil // nothing queued + } + return nil, err + } + } + data, err := os.ReadFile(s.readFlushPath()) + if err != nil { + if os.IsNotExist(err) { + return nil, nil + } + return nil, err + } + latest := map[string]time.Time{} + var order []string + for _, line := range bytes.Split(data, []byte("\n")) { + if len(bytes.TrimSpace(line)) == 0 { + continue + } + var e ReadEvent + if json.Unmarshal(line, &e) != nil || e.Path == "" { + continue // torn or corrupt line; drop it + } + if _, ok := latest[e.Path]; !ok { + order = append(order, e.Path) + } + if e.Time.After(latest[e.Path]) { + latest[e.Path] = e.Time + } + } + if len(order) > readReportMax { + order = order[len(order)-readReportMax:] + } + out := make([]ReadEvent, 0, len(order)) + for _, p := range order { + out = append(out, ReadEvent{Path: p, Time: latest[p]}) + } + return out, nil +} + +// ClearPendingReads drops the batch PendingReads returned, after a +// successful report. +func (s *Store) ClearPendingReads() error { + err := os.Remove(s.readFlushPath()) + if os.IsNotExist(err) { + return nil + } + return err +} diff --git a/internal/store/reads_test.go b/internal/store/reads_test.go new file mode 100644 index 0000000..ab52bea --- /dev/null +++ b/internal/store/reads_test.go @@ -0,0 +1,111 @@ +package store + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func openTestStore(t *testing.T) *Store { + t.Helper() + s, err := Open(filepath.Join(t.TempDir(), "volume")) + if err != nil { + t.Fatal(err) + } + return s +} + +func TestReadSpool(t *testing.T) { + s := openTestStore(t) + + // Nothing queued: no batch, no error. + if evs, err := s.PendingReads(); err != nil || len(evs) != 0 { + t.Fatalf("empty spool = %v, %v", evs, err) + } + + // Repeat reads of one path dedupe to its latest event. + for i := 0; i < 3; i++ { + if err := s.LogRead("wiki/a.md"); err != nil { + t.Fatal(err) + } + } + if err := s.LogRead("b.md"); err != nil { + t.Fatal(err) + } + evs, err := s.PendingReads() + if err != nil { + t.Fatal(err) + } + if len(evs) != 2 || evs[0].Path != "wiki/a.md" || evs[1].Path != "b.md" { + t.Fatalf("batch = %+v, want deduped a.md + b.md", evs) + } + if evs[0].Time.IsZero() { + t.Fatal("events must carry their read time") + } + + // The batch survives until cleared — a failed report just retries — and + // reads logged meanwhile land in a fresh spool behind it. + if err := s.LogRead("c.md"); err != nil { + t.Fatal(err) + } + again, err := s.PendingReads() + if err != nil { + t.Fatal(err) + } + if len(again) != 2 || again[0].Path != "wiki/a.md" { + t.Fatalf("retry batch = %+v, want the same uncleared batch", again) + } + if err := s.ClearPendingReads(); err != nil { + t.Fatal(err) + } + next, err := s.PendingReads() + if err != nil { + t.Fatal(err) + } + if len(next) != 1 || next[0].Path != "c.md" { + t.Fatalf("post-clear batch = %+v, want just c.md", next) + } + s.ClearPendingReads() + if evs, _ := s.PendingReads(); len(evs) != 0 { + t.Fatalf("drained spool still returned %+v", evs) + } +} + +func TestReadSpoolSurvivesCorruptLines(t *testing.T) { + s := openTestStore(t) + s.LogRead("good.md") + f, err := os.OpenFile(s.readSpoolPath(), os.O_WRONLY|os.O_APPEND, 0o644) + if err != nil { + t.Fatal(err) + } + f.WriteString(`{"path": "torn`) // a torn write + f.Close() + s.LogRead("also-good.md") + evs, err := s.PendingReads() + if err != nil { + t.Fatal(err) + } + // The torn line joins the next event's line; both are dropped, but the + // batch itself survives. + if len(evs) == 0 || evs[0].Path != "good.md" { + t.Fatalf("batch = %+v, want good.md to survive the torn line", evs) + } +} + +func TestReadSpoolCap(t *testing.T) { + s := openTestStore(t) + long := strings.Repeat("d", 1024) + for i := 0; i < 1100; i++ { // ~1.1 MB of events + if err := s.LogRead(long + "/" + string(rune('a'+i%26)) + ".md"); err != nil { + t.Fatal(err) + } + } + fi, err := os.Stat(s.readSpoolPath()) + if err != nil { + t.Fatal(err) + } + if fi.Size() > readSpoolMax+4096 { + t.Fatalf("spool grew past its cap: %d bytes", fi.Size()) + } +} diff --git a/internal/syncer/ignore.go b/internal/syncer/ignore.go index b142605..8e9adec 100644 --- a/internal/syncer/ignore.go +++ b/internal/syncer/ignore.go @@ -53,6 +53,14 @@ type pattern struct { negate bool } +// LoadFilter builds the filter for a folder from its .bdriveignore (if any) +// plus the include list from the .bdrive settings file — the exact rules the +// sync cycle applies, for callers outside the cycle (e.g. `bdrive read-log` +// deciding whether an agent-read path is part of the project). +func LoadFilter(folder string, include []string) (*Filter, error) { + return loadFilter(folder, include) +} + // loadFilter builds the filter for a folder from its .bdriveignore (if any) // plus the include list from the .bdrive settings file. func loadFilter(folder string, include []string) (*Filter, error) { diff --git a/internal/syncer/reads_flow_test.go b/internal/syncer/reads_flow_test.go new file mode 100644 index 0000000..69122b9 --- /dev/null +++ b/internal/syncer/reads_flow_test.go @@ -0,0 +1,101 @@ +package syncer + +import ( + "context" + "fmt" + "sync" + "testing" + + "github.com/runbear-io/beardrive/internal/remote" +) + +// readReportingRemote wraps a backend with the hub's ReadReporter capability, +// standing in for the https:// backend in the multi-device harness. +type readReportingRemote struct { + remote.Backend + + mu sync.Mutex + fail bool + reports [][]remote.ReadEvent +} + +func (r *readReportingRemote) ReportReads(_ context.Context, reads []remote.ReadEvent) error { + r.mu.Lock() + defer r.mu.Unlock() + if r.fail { + return fmt.Errorf("hub unreachable") + } + cp := make([]remote.ReadEvent, len(reads)) + copy(cp, reads) + r.reports = append(r.reports, cp) + return nil +} + +func (r *readReportingRemote) setFail(v bool) { + r.mu.Lock() + defer r.mu.Unlock() + r.fail = v +} + +func (r *readReportingRemote) all() [][]remote.ReadEvent { + r.mu.Lock() + defer r.mu.Unlock() + return r.reports +} + +// TestAgentReadReporting drives the read spool through real sync cycles: the +// queued reads flush to a reporting hub (deduped), survive an unreachable hub +// and retry, and never disturb the sync result itself. +func TestAgentReadReporting(t *testing.T) { + hub := &readReportingRemote{Backend: sharedRemote(t)} + a := newDevice(t, "deva", hub) + write(t, a.Folder, "wiki/a.md", "content") + + // The agent read a.md twice and b.md once before this cycle. + a.Store.LogRead("wiki/a.md") + a.Store.LogRead("wiki/a.md") + a.Store.LogRead("b.md") + res := cycle(t, a) + if !res.Pushed { + t.Fatal("cycle should have pushed") + } + reports := hub.all() + if len(reports) != 1 || len(reports[0]) != 2 { + t.Fatalf("reports = %+v, want one deduped batch of 2", reports) + } + if reports[0][0].Path != "wiki/a.md" || reports[0][1].Path != "b.md" { + t.Fatalf("batch = %+v", reports[0]) + } + // Drained: an idle cycle reports nothing. + cycle(t, a) + if len(hub.all()) != 1 { + t.Fatal("empty spool still produced a report") + } + + // Hub down: the cycle still succeeds and the batch stays queued. + hub.setFail(true) + a.Store.LogRead("wiki/a.md") + if res := cycle(t, a); res.Offline { + t.Fatal("a failed read report must not mark the cycle offline") + } + if len(hub.all()) != 1 { + t.Fatal("failed report should not have landed") + } + // Hub back: the next cycle retries the same batch. + hub.setFail(false) + cycle(t, a) + reports = hub.all() + if len(reports) != 2 || len(reports[1]) != 1 || reports[1][0].Path != "wiki/a.md" { + t.Fatalf("retry reports = %+v", reports) + } + + // A backend without the capability (plain object store) is untouched by + // queued reads: the cycle runs, the spool just keeps waiting. + b := newDevice(t, "devb", sharedRemote(t)) + write(t, b.Folder, "x.md", "x") + b.Store.LogRead("x.md") + cycle(t, b) + if evs, err := b.Store.PendingReads(); err != nil || len(evs) != 1 { + t.Fatalf("spool on a hubless device = %v, %v; want the read still queued", evs, err) + } +} diff --git a/internal/syncer/syncer.go b/internal/syncer/syncer.go index 7f53e8e..869a588 100644 --- a/internal/syncer/syncer.go +++ b/internal/syncer/syncer.go @@ -199,6 +199,21 @@ func (s *Session) Cycle(ctx context.Context) (*Result, error) { } } + // 6. Drain the agent read spool to the hub (read heatmap telemetry). + // Strictly best-effort: a failed report keeps the batch queued for the + // next cycle and never fails — or even marks offline — this one. + if rr, ok := s.Backend.(remote.ReadReporter); ok && !res.Offline { + if evs, err := s.Store.PendingReads(); err == nil && len(evs) > 0 { + reads := make([]remote.ReadEvent, len(evs)) + for i, e := range evs { + reads[i] = remote.ReadEvent{Path: e.Path, Time: e.Time} + } + if rr.ReportReads(ctx, reads) == nil { + s.Store.ClearPendingReads() + } + } + } + if err := s.Store.SaveCache(s.mountID(), cache); err != nil { return nil, err } From e3ca821dc4297165bd8b7b0aa30c7e7c1f91fc54 Mon Sep 17 00:00:00 2001 From: Snow Lee Date: Sat, 11 Jul 2026 14:54:32 -0700 Subject: [PATCH 3/3] feat(web): Insights quadrant + read-heat docs (phase 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Admin/org-owner Insights view (⋯ menu): dependency-free SVG scatter of every file by 30-day reads × days since last change, log scales, with the hot-but-stale danger quadrant shaded and a ranked fix-these-first list; lens toggle for all/human/agent reads. The Claude Code plugin gains a PostToolUse(Read) hook so plugin users feed agent-read telemetry without project-level hook registration. Docs synced: README, SKILL.md, plugin install/init commands, CLAUDE.md, design doc marked implemented. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01P5cxPQdSGJnjXCYY9GeWXt --- CLAUDE.md | 2 +- README.md | 17 +++- docs/design/read-heatmap.md | 2 +- internal/webapp/static/app.js | 136 ++++++++++++++++++++++++++++++- internal/webapp/static/style.css | 19 +++++ plugin/commands/init.md | 10 ++- plugin/commands/install.md | 6 +- plugin/hooks/hooks.json | 12 +++ plugin/scripts/beardrive-read.sh | 10 +++ plugin/skills/beardrive/SKILL.md | 32 ++++++-- 10 files changed, 229 insertions(+), 17 deletions(-) create mode 100755 plugin/scripts/beardrive-read.sh diff --git a/CLAUDE.md b/CLAUDE.md index f2e71ab..c73bf36 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,7 +39,7 @@ Package roles (`internal/`): - **`syncer`** — the heart: `Session.Cycle()` runs one pass: scan → commit local ops → pull peer journals → preserve conflict copies → materialize merged state → push blobs + own journal. Read the package doc comment in `syncer.go` first. `ignore.go` holds the path filter (`.bdriveignore` rules + the `.bdrive` include list), applied symmetrically in scan and materialize; a newly filtered path is dropped from the cache *without* a delete op so opting out locally never deletes remotely. - **`daemon`** — per-mount background loop (detached process, `daemon.pid`/`daemon.log` in the mount's volume dir). Scans every `--scan-interval` (3s), talks to the remote every `--remote-interval` (10s) or immediately after local edits. Re-reads `.bdrive/config.json` each tick; if it vanishes (folder moved/renamed/deleted) the daemon **exits cleanly without propagating deletes** — the next bdrive command at the new location resumes it (self-heal on next touch). - **`config`** — global state under `$BDRIVE_HOME` (default `~/.bdrive`): device identity (`device.json`), settings (`settings.json`: default server + device token + signed-in account), and the mount registry (`mounts.json`, keyed by **stable mount id**, holding only each mount's last-known path). The per-folder `.bdrive/` directory (`project.go`) holds `config.json` with the mount id + volume/remote/include; **nothing is keyed by the folder path**, so renames/moves are free — `ResolveMount` self-heals the registry path, and the volume store lives at `~/.bdrive/volumes//`. `.bdrive/` is never synced and holds no credentials. -- **`webapp`** — the `bdrive web` server, in two modes. Single-volume: `Source` is a `DirSource` (plain folder from disk) or `RemoteSource` (folds journals into a file tree with per-file provenance). Hub: `Root` + `Projects` host many projects on one storage root, each under `//` via `remote.Prefixed`; `ProjectDB` (`projects.go`) is a file-backed registry (JSON, loaded at open, rewritten atomically per change) with create-or-join-by-name semantics, name-scoped per organization. Orgs (`orgs.go`, file-backed `orgs.json`) wall projects by membership (email → owner|member): every per-project route — viewer APIs, uploads, history, shares management, the `/store/*` sync proxy — 403s for non-members, `/api/projects` lists only your orgs' projects, owners mint expiring multi-use invite links (`/join/`), and a pre-org hub migrates all projects into a "default" org (all existing accounts join, oldest owns) at startup. `QuotaProvider` (`quota.go`) is the plan-enforcement seam mirroring `AuthProvider` — CheckWrite/RecordUsage on every write path, CheckSeat on invite redemption; OSS ships only `UnlimitedQuota`, managed deployments swap the provider. Renders markdown (goldmark + Obsidian `[[wikilinks]]`). With `--upload` it accepts writes: browser uploads (`upload.go` — direct-to-storage via presigned URLs when the backend implements `remote.PutSigner`, relayed otherwise; ops journaled under the server's own device) and the per-project `/api/p//store/*` proxy (`store.go`) that whole devices sync through — the `https://` remote backend (`remote/http.go`) is its client; journals are never presigned, only immutable blobs. Frontend is dependency-free vanilla JS embedded via `go:embed static`; it learns everything from `/api/config` (+ `/api/projects` in hub mode) and never sees storage info or credentials. It uses native History-API path routing (`//` in hub mode, `/` in volume mode, `/join/` for invites — no `#`, slashes stay literal); `Server.frontend` serves `index.html` as the SPA fallback for any non-asset, non-API/auth/share route so deep links and refreshes resolve, and all client API/asset URLs are root-absolute so a deep path doesn't break relative resolution. **Hub metadata persistence** (accounts, projects, orgs+invites, shares, devices — never blobs or journals) sits behind a pluggable `MetaStore` of typed repos (`db.go`): the service structs (`BuiltinAuth`, `OrgDB`, `ProjectDB`, `ShareDB`, `DeviceRegistry`) keep their in-memory maps + logic and persist each change as one record through a repo. Two backends — `db_file.go` (the historical JSON files, still the zero-dep default, reached via the `Open*(path)` constructors) and `db_sql.go` (one `database/sql` impl over pure-Go drivers: `modernc.org/sqlite` locally, `jackc/pgx` for Postgres/Supabase, portable schema + idempotent migrations + transactional multi-row writes). `web.go`'s `database` config (`{driver:file|sqlite|postgres, dsn}`) selects it; file is default and untouched. `db_conformance_test.go` runs the same service ops against every backend. +- **`webapp`** — the `bdrive web` server, in two modes. Single-volume: `Source` is a `DirSource` (plain folder from disk) or `RemoteSource` (folds journals into a file tree with per-file provenance). Hub: `Root` + `Projects` host many projects on one storage root, each under `//` via `remote.Prefixed`; `ProjectDB` (`projects.go`) is a file-backed registry (JSON, loaded at open, rewritten atomically per change) with create-or-join-by-name semantics, name-scoped per organization. Orgs (`orgs.go`, file-backed `orgs.json`) wall projects by membership (email → owner|member): every per-project route — viewer APIs, uploads, history, shares management, the `/store/*` sync proxy — 403s for non-members, `/api/projects` lists only your orgs' projects, owners mint expiring multi-use invite links (`/join/`), and a pre-org hub migrates all projects into a "default" org (all existing accounts join, oldest owns) at startup. `QuotaProvider` (`quota.go`) is the plan-enforcement seam mirroring `AuthProvider` — CheckWrite/RecordUsage on every write path, CheckSeat on invite redemption; OSS ships only `UnlimitedQuota`, managed deployments swap the provider. Renders markdown (goldmark + Obsidian `[[wikilinks]]`). With `--upload` it accepts writes: browser uploads (`upload.go` — direct-to-storage via presigned URLs when the backend implements `remote.PutSigner`, relayed otherwise; ops journaled under the server's own device) and the per-project `/api/p//store/*` proxy (`store.go`) that whole devices sync through — the `https://` remote backend (`remote/http.go`) is its client; journals are never presigned, only immutable blobs. Frontend is dependency-free vanilla JS embedded via `go:embed static`; it learns everything from `/api/config` (+ `/api/projects` in hub mode) and never sees storage info or credentials. It uses native History-API path routing (`//` in hub mode, `/` in volume mode, `/join/` for invites — no `#`, slashes stay literal); `Server.frontend` serves `index.html` as the SPA fallback for any non-asset, non-API/auth/share route so deep links and refreshes resolve, and all client API/asset URLs are root-absolute so a deep path doesn't break relative resolution. **Read heat** (`reads.go`): a `ReadLedger` (hub-only, nil = off, config `reads` block) aggregates read telemetry into daily per-actor buckets, debounced to 10-minute visits, folded into all-time rows past `retention_days` — viewer file/render/download = human (recorded via the project id the `proj()` resolver stashes in the request context), `/s/*` hits = share, device-reported reads (`POST /api/p//reads`) = agent; `/store/*` replication and history `/blob` views are NEVER reads. `GET /api/p//heat?prefix=&days=` returns counts/distinct-readers/last-read only — actor identities (the email/device/token in the buckets) must never appear in an API response. Recording and flushing degrade silently (log once); telemetry must never fail a request or a sync cycle. The frontend shows heat dots on folder listings for members and an admin/org-owner Insights quadrant (reads × staleness). **Hub metadata persistence** (accounts, projects, orgs+invites, shares, devices, read buckets — never blobs or journals) sits behind a pluggable `MetaStore` of typed repos (`db.go`): the service structs (`BuiltinAuth`, `OrgDB`, `ProjectDB`, `ShareDB`, `DeviceRegistry`, `ReadLedger`) keep their in-memory maps + logic and persist each change as one record through a repo (the `ReadRepo` alone is batch-oriented — one flush, one write). Two backends — `db_file.go` (the historical JSON files, still the zero-dep default, reached via the `Open*(path)` constructors) and `db_sql.go` (one `database/sql` impl over pure-Go drivers: `modernc.org/sqlite` locally, `jackc/pgx` for Postgres/Supabase, portable schema + idempotent migrations + transactional multi-row writes). `web.go`'s `database` config (`{driver:file|sqlite|postgres, dsn}`) selects it; file is default and untouched. `db_conformance_test.go` runs the same service ops against every backend. `cmd/bdrive/` is a thin cobra CLI over these packages (`login`, `logout`, `init`, `stop`, `sync`, `status`, `log`, `web`, `whoami`, `daemon`, `version` — `mnt`/`umnt`/`remote` are gone; `init` is the front door and `stop` pauses). `bdrive login` signs the device in (bare form uses the remembered server or `config.DefaultServer` = beardrive.ai; loopback-callback browser flow in `login.go`, `--device` for headless) and stores server+token+account in `settings.json`; `bdrive logout` clears the saved token+account (keeps the remembered server unless `--forget`). Switching hubs is `bdrive login ` then re-`init` — `init` is the only thing that writes a folder's remote (always a hub, `server + "/p/" + id`); there is no client command to point a folder at a raw bucket. `bdrive init` is interactive on a TTY (survey menus: create-new vs connect-existing with a project list; whole-folder vs `--shared `, which becomes the include list) with full flag bypass (`--name/--project/--shared/--yes`) and never prompts without a TTY; it runs the login flow first when there is no session, writes `.bdrive/config.json`, seeds `.bdriveignore`, and starts sync via `startSync`; re-running it resumes — including after a folder move. `bdrive web -c config.json` configures the server from a file, explicit flags winning. diff --git a/README.md b/README.md index d34e85b..52eba19 100644 --- a/README.md +++ b/README.md @@ -122,7 +122,8 @@ beardrive uses each provider's standard credential chain — nothing beardrive-s | `bdrive stop [folder]` | Stop syncing (files stay; `bdrive init` resumes) | | `bdrive share ` | Public URL for a synced file (`--list`, `--revoke`, `--expires`) | | `bdrive sync [folder]` | Run one sync cycle now. `--note ` stamps session context (e.g. an agent session id) onto changes — shown in `bdrive log` and hub history; keeps applying to daemon-committed changes until `--note-ttl` (default 30m) expires | -| `bdrive hooks [install]` | Register turn-boundary sync hooks with detected agent platforms (Claude Code, Codex, Gemini CLI, Hermes) — pull each turn, push after edits, session-note stamping; idempotent (`--agent` overrides detection) | +| `bdrive hooks [install]` | Register turn-boundary sync hooks with detected agent platforms (Claude Code, Codex, Gemini CLI, Hermes) — pull each turn, push after edits, session-note stamping, agent-read tracking; idempotent (`--agent` overrides detection) | +| `bdrive read-log [folder]` | Hook plumbing: queue agent file reads from a hook event (JSON on stdin) for the hub's read heatmap; drained on the next sync. Registered by `bdrive hooks install` | | `bdrive status [folder]` | Projects, daemon state, pending changes | | `bdrive log [folder] [-p path] [-n N]` | Change history: account, device, time, file | | `bdrive web [folder \| storage-root-url]` | Web server: viewer (rendered markdown, downloads, history), uploads, multi-project sync hub | @@ -200,6 +201,10 @@ the positional argument), `--upload` (allow client writes, off by default), "admins": ["admin@example.com"], "smtp": { "host": "smtp.example.com", "port": 587, "user": "drive@example.com", "pass": "…", "from": "drive@example.com" } + }, + "reads": { // read heatmap telemetry (hub mode) + "enabled": true, // default true; aggregate counts only + "retention_days": 400 // daily buckets older than this fold into all-time totals } } ``` @@ -310,6 +315,16 @@ phase and the API is already shaped for it). Folder rows have a history shortcut for a subtree feed; the topbar button shows the current file's versions or the whole project feed. +Hubs also track **read heat**: viewer opens and downloads count as human +reads, share-link hits as share reads, and agent tool reads (reported by +the sync hooks via `bdrive read-log`) as agent reads — sync replication +never counts. Folder listings show heat dots and 30-day read counts to +every member, and admins / org owners get an **Insights** view (⋯ menu) +plotting each file by reads × days since last change: the hot-but-stale +quadrant is the knowledge people rely on that nobody maintains. The API +(`GET /api/p//heat?prefix=&days=`) exposes only aggregate counts, +distinct-reader counts, and last-read times — never who read what. + ### Authentication Hubs always require sign-in — every change is attributed to a real account. diff --git a/docs/design/read-heatmap.md b/docs/design/read-heatmap.md index 9a0788a..15772d2 100644 --- a/docs/design/read-heatmap.md +++ b/docs/design/read-heatmap.md @@ -1,6 +1,6 @@ # Read heatmap — design -Status: proposed (2026-07-11) · Owner: snow · Prior art: session-linked notes (shipped), history API +Status: implemented (2026-07-11, all three phases) · Owner: snow · Prior art: session-linked notes (shipped), history API ## Problem diff --git a/internal/webapp/static/app.js b/internal/webapp/static/app.js index 8b9dbaf..a7baa5f 100644 --- a/internal/webapp/static/app.js +++ b/internal/webapp/static/app.js @@ -812,7 +812,7 @@ function renderNode(n) { openFolder(n.path); }; } else { - flatFiles.push({ path: n.path, name: n.name }); + flatFiles.push({ path: n.path, name: n.name, time: n.time }); row.onclick = () => openFile(n.path); } return li; @@ -1135,6 +1135,132 @@ function updateShareButton() { }; } +/* ---- insights: the read×write matrix ---- + Every file plotted by how much it is read (30 days, from the heat API) + against how long since it last changed (from the tree). The hot-but-stale + quadrant is the danger zone: knowledge people still rely on that nobody + maintains. Admin/org-owner only — members get the ambient heat dots. */ + +const HOT_READS = 3; // ≥ this many reads/30d = hot +const STALE_DAYS = 30; // ≥ this many days since last write = stale + +function canSeeInsights() { + if (!(serverConfig.mode === "hub" && currentProject)) return false; + if (!(serverConfig.reads && serverConfig.reads.enabled)) return false; + if (serverConfig.auth && serverConfig.auth.admin) return true; + const org = currentOrg(); + return !!(org && org.role === "owner"); +} + +async function showInsights() { + if (!canSeeInsights()) return; + await refreshHeat(true); + currentPath = null; + markActive(); + $("crumb").textContent = "Insights — " + currentProject.name; + $("meta").textContent = ""; + $("download").hidden = true; + $("more-btn").hidden = true; + const content = $("content"); + content.className = "view"; + renderInsights(content, "all"); +} + +function renderInsights(content, lens) { + content.innerHTML = ""; + const wrap = el(content, "div", "insights"); + el(wrap, "h1", "in-title", "Reads × freshness"); + el(wrap, "p", "dl-sub", + "Every file by 30-day reads and days since its last change. " + + "Hot but stale (top right) is the danger zone — read a lot, maintained by nobody."); + const bar = el(wrap, "div", "in-lens"); + for (const l of ["all", "human", "agent"]) { + const label = l === "all" ? "All reads" : l === "human" ? "Human reads" : "Agent reads"; + const b = el(bar, "button", "in-lens-btn" + (l === lens ? " active" : ""), label); + b.onclick = () => renderInsights(content, lens = l); + } + + const readsOf = (e) => (lens === "all" ? heatTotal(e) : e[lens] || 0); + const now = Date.now(); + const pts = flatFiles.map((f) => { + const e = (heatMap && heatMap[f.path]) || {}; + const days = f.time ? Math.max(0, (now - new Date(f.time).getTime()) / 86400000) : 0; + const reads = readsOf(e); + return { path: f.path, reads, days, danger: reads >= HOT_READS && days >= STALE_DAYS }; + }); + + wrap.appendChild(insightsChart(pts)); + + const danger = pts.filter((p) => p.danger) + .sort((a, b) => b.reads - a.reads || b.days - a.days).slice(0, 15); + el(wrap, "h3", "dl-h3", "Danger zone — fix these first"); + if (!danger.length) { + el(wrap, "div", "dl-empty", "No hot-but-stale files. The knowledge base is healthy."); + return; + } + const list = el(wrap, "div", "dl-items"); + for (const p of danger) { + const row = el(list, "div", "dl-row"); + row.tabIndex = 0; + row.setAttribute("role", "button"); + const icon = el(row, "span", "ticon"); + icon.innerHTML = svgIcon("alert"); + el(row, "span", "dl-name", p.path); + el(row, "span", "dl-meta", + p.reads + (p.reads === 1 ? " read" : " reads") + " · untouched " + Math.round(p.days) + "d"); + row.onclick = () => openFile(p.path); + row.onkeydown = (e) => { if (e.key === "Enter" || e.key === " ") { e.preventDefault(); row.click(); } }; + } +} + +/* Dependency-free SVG scatter: x = days since last write, y = reads, both + log-scaled; threshold lines split the quadrants. */ +function insightsChart(pts) { + const W = 720, H = 360, M = { l: 44, r: 16, t: 20, b: 34 }; + const maxDays = Math.max(STALE_DAYS * 2, ...pts.map((p) => p.days)); + const maxReads = Math.max(HOT_READS * 2, ...pts.map((p) => p.reads)); + const lx = (d) => Math.log10(d + 1) / Math.log10(maxDays + 1); + const ly = (r) => Math.log10(r + 1) / Math.log10(maxReads + 1); + const X = (d) => M.l + lx(d) * (W - M.l - M.r); + const Y = (r) => H - M.b - ly(r) * (H - M.t - M.b); + + const svg = document.createElementNS("http://www.w3.org/2000/svg", "svg"); + svg.setAttribute("viewBox", `0 0 ${W} ${H}`); + svg.setAttribute("class", "in-chart"); + const add = (tag, attrs, text) => { + const n = document.createElementNS("http://www.w3.org/2000/svg", tag); + for (const [k, v] of Object.entries(attrs)) n.setAttribute(k, v); + if (text != null) n.textContent = text; + svg.appendChild(n); + return n; + }; + + // danger quadrant shading + threshold lines + add("rect", { x: X(STALE_DAYS), y: M.t, width: W - M.r - X(STALE_DAYS), height: Y(HOT_READS) - M.t, class: "in-danger-zone" }); + add("line", { x1: X(STALE_DAYS), y1: M.t, x2: X(STALE_DAYS), y2: H - M.b, class: "in-threshold" }); + add("line", { x1: M.l, y1: Y(HOT_READS), x2: W - M.r, y2: Y(HOT_READS), class: "in-threshold" }); + // axes + add("line", { x1: M.l, y1: H - M.b, x2: W - M.r, y2: H - M.b, class: "in-axis" }); + add("line", { x1: M.l, y1: M.t, x2: M.l, y2: H - M.b, class: "in-axis" }); + add("text", { x: (M.l + W - M.r) / 2, y: H - 8, class: "in-label" }, "days since last change →"); + add("text", { x: 12, y: (M.t + H - M.b) / 2, class: "in-label", transform: `rotate(-90 12 ${(M.t + H - M.b) / 2})` }, "reads / 30d →"); + add("text", { x: W - M.r - 6, y: M.t + 14, class: "in-quad in-quad-danger", "text-anchor": "end" }, "hot + stale"); + add("text", { x: M.l + 6, y: M.t + 14, class: "in-quad" }, "hot + fresh"); + add("text", { x: W - M.r - 6, y: H - M.b - 8, class: "in-quad", "text-anchor": "end" }, "cold + stale"); + + for (const p of pts) { + const c = add("circle", { + cx: X(p.days).toFixed(1), cy: Y(p.reads).toFixed(1), r: 5, + class: "in-pt" + (p.danger ? " danger" : p.reads ? "" : " cold"), + }); + const tip = document.createElementNS("http://www.w3.org/2000/svg", "title"); + tip.textContent = `${p.path} — ${p.reads} read${p.reads === 1 ? "" : "s"} / 30d · changed ${Math.round(p.days)}d ago`; + c.appendChild(tip); + c.onclick = () => openFile(p.path); + } + return svg; +} + /* ---- history ---- Every change ever made, straight from the journals: who (account), when, from which device (name, OS, IP as the server saw it), with view/download @@ -1553,6 +1679,14 @@ function buildMoreMenu() { b.onclick = () => { $("more-menu").hidden = true; el.click(); }; menu.appendChild(b); } + if (canSeeInsights()) { + const b = document.createElement("button"); + b.className = "more-item"; + b.textContent = "Insights"; + b.onclick = () => { $("more-menu").hidden = true; showInsights(); }; + menu.appendChild(b); + return items.length + 1; + } return items.length; } $("more-btn").addEventListener("click", (e) => { diff --git a/internal/webapp/static/style.css b/internal/webapp/static/style.css index c0f5d94..25bf24b 100644 --- a/internal/webapp/static/style.css +++ b/internal/webapp/static/style.css @@ -281,6 +281,25 @@ button, input, a.btn { font-family: inherit; } .dl-hlist .hentry:last-child { border-bottom: none; } .dl-more { margin-top: 10px; } +/* ---- insights (read×write matrix) ---- */ +.insights { max-width: 760px; margin: 0 auto; } +.in-title { font-size: 21px; font-weight: 640; letter-spacing: -.02em; margin: 0 0 4px; color: #f4f6f9; } +.in-lens { display: flex; gap: 6px; margin: 0 0 14px; } +.in-lens-btn { font: inherit; font-size: 12px; padding: 5px 12px; border-radius: 999px; border: 1px solid var(--border); background: none; color: var(--text-faint); cursor: pointer; } +.in-lens-btn:hover { color: var(--text); } +.in-lens-btn.active { color: var(--accent); border-color: var(--accent); } +.in-chart { width: 100%; height: auto; border: 1px solid var(--border); border-radius: var(--r-card); background: var(--bg-side); margin-bottom: 6px; } +.in-axis { stroke: var(--border); stroke-width: 1; } +.in-threshold { stroke: var(--border); stroke-width: 1; stroke-dasharray: 4 4; } +.in-danger-zone { fill: rgba(242, 109, 109, .05); } +.in-label { fill: var(--text-ghost); font-size: 11px; } +.in-quad { fill: var(--text-ghost); font-size: 10.5px; text-transform: uppercase; letter-spacing: .06em; } +.in-quad-danger { fill: #e07070; } +.in-pt { fill: var(--accent); opacity: .75; cursor: pointer; } +.in-pt:hover { opacity: 1; } +.in-pt.cold { fill: var(--text-ghost); opacity: .35; } +.in-pt.danger { fill: #e05d5d; } + /* ---- history ---- */ .history { max-width: 860px; } .hentry { padding: 11px 12px; border-bottom: 1px solid var(--border); } diff --git a/plugin/commands/init.md b/plugin/commands/init.md index dd13a7a..e8c7901 100644 --- a/plugin/commands/init.md +++ b/plugin/commands/init.md @@ -56,10 +56,12 @@ Follow these steps: detects the agent platforms in use (Claude Code, Codex, Gemini CLI, Hermes — by their config dirs in the project or home) and idempotently merges beardrive's sync hooks into each platform's own hook config, so - files pull at every turn start, push after edits, and every change is - stamped with the agent session that made it. Tell the user which - platforms got hooks; if Codex is among them, mention they must run - `/hooks` inside Codex once to trust the project's `.codex` layer. + files pull at every turn start, push after edits, every change is + stamped with the agent session that made it, and agent file reads feed + the hub's read heatmap (queued locally by `bdrive read-log`, reported + on the next sync). Tell the user which platforms got hooks; if Codex is + among them, mention they must run `/hooks` inside Codex once to trust + the project's `.codex` layer. 6. **Verify**: run `bdrive status ` and confirm the daemon is running and pending is 0. Summarize: project name/id, what syncs, and diff --git a/plugin/commands/install.md b/plugin/commands/install.md index 1adaf7d..257c5ec 100644 --- a/plugin/commands/install.md +++ b/plugin/commands/install.md @@ -77,7 +77,11 @@ team's latest files), push right after edits (artifacts land on the server seconds after they're created — daemon or no daemon), and stamp every change with the agent session that made it (`bdrive sync --note " session "` — visible in `bdrive log` and the hub's history views). -They are fast no-ops in folders without `.bdrive/`. +A third hook on each platform's read tool (`bdrive read-log`) queues which +files the agent read, so the hub's read heatmap can show admins what the +team's agents actually consume — reads are reported on the next sync, +never from the hook itself. They are fast no-ops in folders without +`.bdrive/`. Tell the user which platforms got hooks (`bdrive hooks` shows the status table). If Codex is among them, mention they must run `/hooks` inside diff --git a/plugin/hooks/hooks.json b/plugin/hooks/hooks.json index 68b9cf0..f4d9f89 100644 --- a/plugin/hooks/hooks.json +++ b/plugin/hooks/hooks.json @@ -12,6 +12,18 @@ ] } ], + "PostToolUse": [ + { + "matcher": "Read", + "hooks": [ + { + "type": "command", + "command": "\"${CLAUDE_PLUGIN_ROOT}/scripts/beardrive-read.sh\"", + "async": true + } + ] + } + ], "Stop": [ { "hooks": [ diff --git a/plugin/scripts/beardrive-read.sh b/plugin/scripts/beardrive-read.sh new file mode 100755 index 0000000..114266e --- /dev/null +++ b/plugin/scripts/beardrive-read.sh @@ -0,0 +1,10 @@ +#!/bin/sh +# Queue the file reads from a Read tool call for the hub's read heatmap. +# `bdrive read-log` parses the hook's stdin JSON itself and only appends to +# a local spool (drained on the next sync) — no network, no locking, so this +# is safe to run on every Read in every project. Fast no-op outside +# beardrive projects. +cd "${CLAUDE_PROJECT_DIR:-.}" || exit 0 +[ -d .bdrive ] || exit 0 +command -v bdrive >/dev/null 2>&1 || exit 0 +bdrive read-log . >/dev/null 2>&1 || true diff --git a/plugin/skills/beardrive/SKILL.md b/plugin/skills/beardrive/SKILL.md index b7ad61d..25fbadb 100644 --- a/plugin/skills/beardrive/SKILL.md +++ b/plugin/skills/beardrive/SKILL.md @@ -17,7 +17,8 @@ Use this skill whenever the user is working with the `bdrive` CLI: initializing | Run the daemon in the foreground | `bdrive init -f` | | Stop syncing | `bdrive stop []` (`--forget` also unregisters) | | One sync cycle now | `bdrive sync []` | -| Register agent sync hooks (Claude Code, Codex, Gemini CLI, Hermes) | `bdrive hooks install []` — auto-detects the platforms in use and merges pull/push/session-note hooks into each one's own hook config, idempotently; bare `bdrive hooks` shows the status table | +| Register agent sync hooks (Claude Code, Codex, Gemini CLI, Hermes) | `bdrive hooks install []` — auto-detects the platforms in use and merges pull/push/session-note/read-tracking hooks into each one's own hook config, idempotently; bare `bdrive hooks` shows the status table | +| Record agent file reads (hook plumbing) | `bdrive read-log []` — parses a hook event JSON from stdin and queues in-project reads locally; drained to the hub on the next sync as agent traffic in the read heatmap. Registered automatically by `bdrive hooks install`; rarely run by hand | | Mounts + daemon + pending state | `bdrive status []` | | Change history | `bdrive log [] [-p path] [-n N]` | | This device's identity | `bdrive whoami` | @@ -130,20 +131,35 @@ conflict-copy ops keep their own `conflict copy of ` note. `bdrive hooks install []` registers turn-boundary sync for every agent platform it detects (by config dir, in the project or home): -| Platform | Config it writes | Pull / push events | +| Platform | Config it writes | Pull / push / read events | |---|---|---| -| Claude Code (& Cowork) | `/.claude/settings.json` | `UserPromptSubmit` / `PostToolUse` (Write\|Edit) | -| Codex (ChatGPT) | `/.codex/hooks.json` | `UserPromptSubmit` / `PostToolUse` (apply_patch) — user must `/hooks`-trust the layer once | -| Gemini CLI | `/.gemini/settings.json` | `BeforeAgent` / `AfterTool` (write_file\|replace) | -| Hermes | `~/.hermes/config.yaml` (per-user) | `pre_llm_call` / `post_tool_call` (write_file\|patch) | +| Claude Code (& Cowork) | `/.claude/settings.json` | `UserPromptSubmit` / `PostToolUse` (Write\|Edit) / `PostToolUse` (Read) | +| Codex (ChatGPT) | `/.codex/hooks.json` | `UserPromptSubmit` / `PostToolUse` (apply_patch) / `PostToolUse` (read_file, best-effort) — user must `/hooks`-trust the layer once | +| Gemini CLI | `/.gemini/settings.json` | `BeforeAgent` / `AfterTool` (write_file\|replace) / `AfterTool` (read_file\|read_many_files) | +| Hermes | `~/.hermes/config.yaml` (per-user) | `pre_llm_call` / `post_tool_call` (write_file\|patch) / `post_tool_call` (read_file) | Every platform pipes hook JSON with a `session_id`, so one hook command serves all four: it syncs the project (fast no-op outside bdrive folders) -and stamps changes with ` session `. Merging is idempotent and -preserves existing hooks; `--agent claude,codex,gemini,hermes` overrides +and stamps changes with ` session `. The read hook runs `bdrive +read-log`, which queues the read locally (no network) for the hub's read +heatmap. Merging is idempotent and preserves existing hooks — each hook +carries its own marker, so configs from before the read hook gain just the +missing group on re-install; `--agent claude,codex,gemini,hermes` overrides detection; bare `bdrive hooks` prints the detection/registration table. Project-level configs ride the repo, so hooks reach the whole team. +### Read heat (who actually reads what) + +Hubs aggregate reads per file — viewer opens and downloads count as human +reads, share-link hits as share reads, and hook-reported agent reads as +agent reads; `/store` sync replication never counts. The web UI shows heat +dots and read counts on folder listings (all members), and admins / org +owners get an **Insights** view (⋯ menu) plotting every file by 30-day +reads × days since last change — the hot-but-stale quadrant is the list of +files to fix first. Counts only, never reader identities. API: +`GET /api/p//heat?prefix=&days=30`. Server config: `"reads": +{"enabled": true, "retention_days": 400}` (on by default in hub mode). + ### Examples to walk a user through ```sh