mirror of
https://github.com/runbear-io/beardrive.git
synced 2026-08-25 08:08:08 +02:00
Merge pull request #13 from runbear-io/feat/read-heatmap
feat: read heatmap — heat ledger, agent read reporting, admin Insights
This commit is contained in:
Snow W. Lee (Sungwon)
@@ -39,7 +39,7 @@ Package roles (`internal/`):
|
||||
- **`syncer`** — the heart: `Session.Cycle()` runs one pass: scan → commit local ops → pull peer journals → preserve conflict copies → materialize merged state → push blobs + own journal. Read the package doc comment in `syncer.go` first. `ignore.go` holds the path filter (`.bdriveignore` rules + the `.bdrive` include list), applied symmetrically in scan and materialize; a newly filtered path is dropped from the cache *without* a delete op so opting out locally never deletes remotely.
|
||||
- **`daemon`** — per-mount background loop (detached process, `daemon.pid`/`daemon.log` in the mount's volume dir). Scans every `--scan-interval` (3s), talks to the remote every `--remote-interval` (10s) or immediately after local edits. Re-reads `.bdrive/config.json` each tick; if it vanishes (folder moved/renamed/deleted) the daemon **exits cleanly without propagating deletes** — the next bdrive command at the new location resumes it (self-heal on next touch).
|
||||
- **`config`** — global state under `$BDRIVE_HOME` (default `~/.bdrive`): device identity (`device.json`), settings (`settings.json`: default server + device token + signed-in account), and the mount registry (`mounts.json`, keyed by **stable mount id**, holding only each mount's last-known path). The per-folder `.bdrive/` directory (`project.go`) holds `config.json` with the mount id + volume/remote/include; **nothing is keyed by the folder path**, so renames/moves are free — `ResolveMount` self-heals the registry path, and the volume store lives at `~/.bdrive/volumes/<mount-id>/`. `.bdrive/` is never synced and holds no credentials.
|
||||
- **`webapp`** — the `bdrive web` server, in two modes. Single-volume: `Source` is a `DirSource` (plain folder from disk) or `RemoteSource` (folds journals into a file tree with per-file provenance). Hub: `Root` + `Projects` host many projects on one storage root, each under `<root>/<project-id>/` via `remote.Prefixed`; `ProjectDB` (`projects.go`) is a file-backed registry (JSON, loaded at open, rewritten atomically per change) with create-or-join-by-name semantics, name-scoped per organization. Orgs (`orgs.go`, file-backed `orgs.json`) wall projects by membership (email → owner|member): every per-project route — viewer APIs, uploads, history, shares management, the `/store/*` sync proxy — 403s for non-members, `/api/projects` lists only your orgs' projects, owners mint expiring multi-use invite links (`/join/<token>`), and a pre-org hub migrates all projects into a "default" org (all existing accounts join, oldest owns) at startup. `QuotaProvider` (`quota.go`) is the plan-enforcement seam mirroring `AuthProvider` — CheckWrite/RecordUsage on every write path, CheckSeat on invite redemption; OSS ships only `UnlimitedQuota`, managed deployments swap the provider. Renders markdown (goldmark + Obsidian `[[wikilinks]]`). With `--upload` it accepts writes: browser uploads (`upload.go` — direct-to-storage via presigned URLs when the backend implements `remote.PutSigner`, relayed otherwise; ops journaled under the server's own device) and the per-project `/api/p/<id>/store/*` proxy (`store.go`) that whole devices sync through — the `https://` remote backend (`remote/http.go`) is its client; journals are never presigned, only immutable blobs. Frontend is dependency-free vanilla JS embedded via `go:embed static`; it learns everything from `/api/config` (+ `/api/projects` in hub mode) and never sees storage info or credentials. It uses native History-API path routing (`/<project-id>/<path>` in hub mode, `/<path>` in volume mode, `/join/<token>` for invites — no `#`, slashes stay literal); `Server.frontend` serves `index.html` as the SPA fallback for any non-asset, non-API/auth/share route so deep links and refreshes resolve, and all client API/asset URLs are root-absolute so a deep path doesn't break relative resolution. **Hub metadata persistence** (accounts, projects, orgs+invites, shares, devices — never blobs or journals) sits behind a pluggable `MetaStore` of typed repos (`db.go`): the service structs (`BuiltinAuth`, `OrgDB`, `ProjectDB`, `ShareDB`, `DeviceRegistry`) keep their in-memory maps + logic and persist each change as one record through a repo. Two backends — `db_file.go` (the historical JSON files, still the zero-dep default, reached via the `Open*(path)` constructors) and `db_sql.go` (one `database/sql` impl over pure-Go drivers: `modernc.org/sqlite` locally, `jackc/pgx` for Postgres/Supabase, portable schema + idempotent migrations + transactional multi-row writes). `web.go`'s `database` config (`{driver:file|sqlite|postgres, dsn}`) selects it; file is default and untouched. `db_conformance_test.go` runs the same service ops against every backend.
|
||||
- **`webapp`** — the `bdrive web` server, in two modes. Single-volume: `Source` is a `DirSource` (plain folder from disk) or `RemoteSource` (folds journals into a file tree with per-file provenance). Hub: `Root` + `Projects` host many projects on one storage root, each under `<root>/<project-id>/` via `remote.Prefixed`; `ProjectDB` (`projects.go`) is a file-backed registry (JSON, loaded at open, rewritten atomically per change) with create-or-join-by-name semantics, name-scoped per organization. Orgs (`orgs.go`, file-backed `orgs.json`) wall projects by membership (email → owner|member): every per-project route — viewer APIs, uploads, history, shares management, the `/store/*` sync proxy — 403s for non-members, `/api/projects` lists only your orgs' projects, owners mint expiring multi-use invite links (`/join/<token>`), and a pre-org hub migrates all projects into a "default" org (all existing accounts join, oldest owns) at startup. `QuotaProvider` (`quota.go`) is the plan-enforcement seam mirroring `AuthProvider` — CheckWrite/RecordUsage on every write path, CheckSeat on invite redemption; OSS ships only `UnlimitedQuota`, managed deployments swap the provider. Renders markdown (goldmark + Obsidian `[[wikilinks]]`). With `--upload` it accepts writes: browser uploads (`upload.go` — direct-to-storage via presigned URLs when the backend implements `remote.PutSigner`, relayed otherwise; ops journaled under the server's own device) and the per-project `/api/p/<id>/store/*` proxy (`store.go`) that whole devices sync through — the `https://` remote backend (`remote/http.go`) is its client; journals are never presigned, only immutable blobs. Frontend is dependency-free vanilla JS embedded via `go:embed static`; it learns everything from `/api/config` (+ `/api/projects` in hub mode) and never sees storage info or credentials. It uses native History-API path routing (`/<project-id>/<path>` in hub mode, `/<path>` in volume mode, `/join/<token>` for invites — no `#`, slashes stay literal); `Server.frontend` serves `index.html` as the SPA fallback for any non-asset, non-API/auth/share route so deep links and refreshes resolve, and all client API/asset URLs are root-absolute so a deep path doesn't break relative resolution. **Read heat** (`reads.go`): a `ReadLedger` (hub-only, nil = off, config `reads` block) aggregates read telemetry into daily per-actor buckets, debounced to 10-minute visits, folded into all-time rows past `retention_days` — viewer file/render/download = human (recorded via the project id the `proj()` resolver stashes in the request context), `/s/*` hits = share, device-reported reads (`POST /api/p/<id>/reads`) = agent; `/store/*` replication and history `/blob` views are NEVER reads. `GET /api/p/<id>/heat?prefix=&days=` returns counts/distinct-readers/last-read only — actor identities (the email/device/token in the buckets) must never appear in an API response. Recording and flushing degrade silently (log once); telemetry must never fail a request or a sync cycle. The frontend shows heat dots on folder listings for members and an admin/org-owner Insights quadrant (reads × staleness). **Hub metadata persistence** (accounts, projects, orgs+invites, shares, devices, read buckets — never blobs or journals) sits behind a pluggable `MetaStore` of typed repos (`db.go`): the service structs (`BuiltinAuth`, `OrgDB`, `ProjectDB`, `ShareDB`, `DeviceRegistry`, `ReadLedger`) keep their in-memory maps + logic and persist each change as one record through a repo (the `ReadRepo` alone is batch-oriented — one flush, one write). Two backends — `db_file.go` (the historical JSON files, still the zero-dep default, reached via the `Open*(path)` constructors) and `db_sql.go` (one `database/sql` impl over pure-Go drivers: `modernc.org/sqlite` locally, `jackc/pgx` for Postgres/Supabase, portable schema + idempotent migrations + transactional multi-row writes). `web.go`'s `database` config (`{driver:file|sqlite|postgres, dsn}`) selects it; file is default and untouched. `db_conformance_test.go` runs the same service ops against every backend.
|
||||
|
||||
`cmd/bdrive/` is a thin cobra CLI over these packages (`login`, `logout`, `init`, `stop`, `sync`, `status`, `log`, `web`, `whoami`, `daemon`, `version` — `mnt`/`umnt`/`remote` are gone; `init` is the front door and `stop` pauses). `bdrive login` signs the device in (bare form uses the remembered server or `config.DefaultServer` = beardrive.ai; loopback-callback browser flow in `login.go`, `--device` for headless) and stores server+token+account in `settings.json`; `bdrive logout` clears the saved token+account (keeps the remembered server unless `--forget`). Switching hubs is `bdrive login <new-url>` then re-`init` — `init` is the only thing that writes a folder's remote (always a hub, `server + "/p/" + id`); there is no client command to point a folder at a raw bucket. `bdrive init` is interactive on a TTY (survey menus: create-new vs connect-existing with a project list; whole-folder vs `--shared <dir>`, which becomes the include list) with full flag bypass (`--name/--project/--shared/--yes`) and never prompts without a TTY; it runs the login flow first when there is no session, writes `.bdrive/config.json`, seeds `.bdriveignore`, and starts sync via `startSync`; re-running it resumes — including after a folder move. `bdrive web -c config.json` configures the server from a file, explicit flags winning.
|
||||
|
||||
|
||||
@@ -122,7 +122,8 @@ beardrive uses each provider's standard credential chain — nothing beardrive-s
|
||||
| `bdrive stop [folder]` | Stop syncing (files stay; `bdrive init` resumes) |
|
||||
| `bdrive share <file>` | Public URL for a synced file (`--list`, `--revoke`, `--expires`) |
|
||||
| `bdrive sync [folder]` | Run one sync cycle now. `--note <text>` stamps session context (e.g. an agent session id) onto changes — shown in `bdrive log` and hub history; keeps applying to daemon-committed changes until `--note-ttl` (default 30m) expires |
|
||||
| `bdrive hooks [install]` | Register turn-boundary sync hooks with detected agent platforms (Claude Code, Codex, Gemini CLI, Hermes) — pull each turn, push after edits, session-note stamping; idempotent (`--agent` overrides detection) |
|
||||
| `bdrive hooks [install]` | Register turn-boundary sync hooks with detected agent platforms (Claude Code, Codex, Gemini CLI, Hermes) — pull each turn, push after edits, session-note stamping, agent-read tracking; idempotent (`--agent` overrides detection) |
|
||||
| `bdrive read-log [folder]` | Hook plumbing: queue agent file reads from a hook event (JSON on stdin) for the hub's read heatmap; drained on the next sync. Registered by `bdrive hooks install` |
|
||||
| `bdrive status [folder]` | Projects, daemon state, pending changes |
|
||||
| `bdrive log [folder] [-p path] [-n N]` | Change history: account, device, time, file |
|
||||
| `bdrive web [folder \| storage-root-url]` | Web server: viewer (rendered markdown, downloads, history), uploads, multi-project sync hub |
|
||||
@@ -200,6 +201,10 @@ the positional argument), `--upload` (allow client writes, off by default),
|
||||
"admins": ["admin@example.com"],
|
||||
"smtp": { "host": "smtp.example.com", "port": 587,
|
||||
"user": "drive@example.com", "pass": "…", "from": "drive@example.com" }
|
||||
},
|
||||
"reads": { // read heatmap telemetry (hub mode)
|
||||
"enabled": true, // default true; aggregate counts only
|
||||
"retention_days": 400 // daily buckets older than this fold into all-time totals
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -310,6 +315,16 @@ phase and the API is already shaped for it). Folder rows have a history
|
||||
shortcut for a subtree feed; the topbar button shows the current file's
|
||||
versions or the whole project feed.
|
||||
|
||||
Hubs also track **read heat**: viewer opens and downloads count as human
|
||||
reads, share-link hits as share reads, and agent tool reads (reported by
|
||||
the sync hooks via `bdrive read-log`) as agent reads — sync replication
|
||||
never counts. Folder listings show heat dots and 30-day read counts to
|
||||
every member, and admins / org owners get an **Insights** view (⋯ menu)
|
||||
plotting each file by reads × days since last change: the hot-but-stale
|
||||
quadrant is the knowledge people rely on that nobody maintains. The API
|
||||
(`GET /api/p/<id>/heat?prefix=&days=`) exposes only aggregate counts,
|
||||
distinct-reader counts, and last-read times — never who read what.
|
||||
|
||||
### Authentication
|
||||
|
||||
Hubs always require sign-in — every change is attributed to a real account.
|
||||
|
||||
@@ -36,6 +36,7 @@ everything keeps working offline; changes sync when the remote is reachable.`,
|
||||
shareCmd(),
|
||||
stopCmd(),
|
||||
syncCmd(),
|
||||
readLogCmd(),
|
||||
hooksCmd(),
|
||||
statusCmd(),
|
||||
logCmd(),
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"github.com/runbear-io/beardrive/internal/config"
|
||||
"github.com/runbear-io/beardrive/internal/store"
|
||||
"github.com/runbear-io/beardrive/internal/syncer"
|
||||
)
|
||||
|
||||
// read-log is the agent read hook's command: it runs after every file-read
|
||||
// tool call, so it must be a fast, silent no-op in every case that isn't
|
||||
// "a synced file was just read" — no network, no locking, one appended line.
|
||||
func readLogCmd() *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "read-log [folder]",
|
||||
Short: "Record agent file reads from a hook event (JSON on stdin)",
|
||||
Long: `Record which project files an agent just read, from the hook event JSON
|
||||
piped on stdin (any platform's PostToolUse-style payload: file paths are
|
||||
found wherever they appear in the event).
|
||||
|
||||
Reads are queued locally in the volume store and drained to the hub on the
|
||||
next sync, where they show up as agent traffic in the project's read heat.
|
||||
Registered automatically by "bdrive hooks install"; there is rarely a reason
|
||||
to run it by hand.`,
|
||||
Args: cobra.MaximumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
folder, err := absFolder(args)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
proj, found, err := config.ResolveMount(folder)
|
||||
if err != nil || !found {
|
||||
return nil // not a beardrive project: fast no-op
|
||||
}
|
||||
data, _ := io.ReadAll(io.LimitReader(cmd.InOrStdin(), 1<<20))
|
||||
paths := extractEventPaths(data)
|
||||
if len(paths) == 0 {
|
||||
return nil
|
||||
}
|
||||
filter, err := syncer.LoadFilter(folder, proj.Include)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
vdir, err := config.VolumeDir(proj.ID)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
st, err := store.Open(vdir)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
for _, p := range paths {
|
||||
abs := p
|
||||
if !filepath.IsAbs(abs) {
|
||||
abs = filepath.Join(folder, p)
|
||||
}
|
||||
rel, err := filepath.Rel(folder, abs)
|
||||
if err != nil || rel == "." || strings.HasPrefix(rel, "..") {
|
||||
continue // outside the mount
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
if filter.Skip(rel) {
|
||||
continue // not part of the project (ignore/include rules)
|
||||
}
|
||||
st.LogRead(rel) // best-effort; the hook must never fail the turn
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// Keys that carry file paths in the hook payloads of the supported agent
|
||||
// platforms (Claude Code tool_input.file_path, Gemini/Hermes read_file
|
||||
// path/absolute_path, multi-read paths arrays).
|
||||
var (
|
||||
eventPathKeys = map[string]bool{"file_path": true, "absolute_path": true, "path": true, "notebook_path": true}
|
||||
eventPathListKeys = map[string]bool{"paths": true, "file_paths": true}
|
||||
)
|
||||
|
||||
// extractEventPaths pulls candidate file paths out of an arbitrary hook
|
||||
// event JSON. The hook only fires on read-tool matchers, so any path-shaped
|
||||
// field is a read; non-project paths are filtered by the caller.
|
||||
func extractEventPaths(data []byte) []string {
|
||||
var root any
|
||||
if json.Unmarshal(data, &root) != nil {
|
||||
return nil
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
var out []string
|
||||
add := func(s string) {
|
||||
if s != "" && !seen[s] {
|
||||
seen[s] = true
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
var walk func(v any)
|
||||
walk = func(v any) {
|
||||
switch t := v.(type) {
|
||||
case map[string]any:
|
||||
for k, val := range t {
|
||||
switch {
|
||||
case eventPathKeys[k]:
|
||||
if s, ok := val.(string); ok {
|
||||
add(s)
|
||||
}
|
||||
case eventPathListKeys[k]:
|
||||
if arr, ok := val.([]any); ok {
|
||||
for _, it := range arr {
|
||||
if s, ok := it.(string); ok {
|
||||
add(s)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(val)
|
||||
}
|
||||
case []any:
|
||||
for _, it := range t {
|
||||
walk(it)
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(root)
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/runbear-io/beardrive/internal/config"
|
||||
"github.com/runbear-io/beardrive/internal/store"
|
||||
)
|
||||
|
||||
// Real hook payload shapes from the supported platforms — read-log must find
|
||||
// the file paths wherever each platform puts them.
|
||||
func TestExtractEventPaths(t *testing.T) {
|
||||
cases := map[string]struct {
|
||||
payload string
|
||||
want []string
|
||||
}{
|
||||
"claude read": {
|
||||
`{"session_id":"abc","hook_event_name":"PostToolUse","tool_name":"Read",
|
||||
"tool_input":{"file_path":"/proj/wiki/a.md"},"tool_response":{"type":"text"}}`,
|
||||
[]string{"/proj/wiki/a.md"},
|
||||
},
|
||||
"gemini read_many_files": {
|
||||
`{"session_id":"g1","tool":{"name":"read_many_files","args":{"paths":["wiki/a.md","wiki/b.md"]}}}`,
|
||||
[]string{"wiki/a.md", "wiki/b.md"},
|
||||
},
|
||||
"gemini read_file absolute": {
|
||||
`{"session_id":"g2","tool":{"name":"read_file","args":{"absolute_path":"/proj/notes.md"}}}`,
|
||||
[]string{"/proj/notes.md"},
|
||||
},
|
||||
"hermes read_file": {
|
||||
`{"hook_event_name":"post_tool_call","tool_name":"read_file","tool_args":{"path":"docs/x.md"}}`,
|
||||
[]string{"docs/x.md"},
|
||||
},
|
||||
"duplicates collapse": {
|
||||
`{"tool_input":{"file_path":"a.md"},"extra":{"file_path":"a.md"}}`,
|
||||
[]string{"a.md"},
|
||||
},
|
||||
"no paths": {
|
||||
`{"session_id":"abc","prompt":"hello"}`,
|
||||
nil,
|
||||
},
|
||||
"not json": {
|
||||
`plain text`,
|
||||
nil,
|
||||
},
|
||||
}
|
||||
for name, c := range cases {
|
||||
got := extractEventPaths([]byte(c.payload))
|
||||
if strings.Join(got, ",") != strings.Join(c.want, ",") {
|
||||
t.Errorf("%s: paths = %v, want %v", name, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// End to end through the cobra command: a claude-style event lands in the
|
||||
// mount's read spool, filtered to project-relative synced paths.
|
||||
func TestReadLogCommand(t *testing.T) {
|
||||
t.Setenv("BDRIVE_HOME", t.TempDir())
|
||||
folder := t.TempDir()
|
||||
folder, _ = filepath.EvalSymlinks(folder)
|
||||
proj, err := config.SaveProject(folder, config.Project{Volume: "wiki"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
c := readLogCmd()
|
||||
c.SetIn(bytes.NewReader([]byte(`{"session_id":"abc","tool_name":"Read",
|
||||
"tool_input":{"file_path":"` + folder + `/wiki/a.md"},
|
||||
"other":{"file_path":"/somewhere/else/entirely.md"}}`)))
|
||||
c.SetArgs([]string{folder})
|
||||
if err := c.Execute(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
vdir, err := config.VolumeDir(proj.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := store.Open(vdir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
evs, err := st.PendingReads()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(evs) != 1 || evs[0].Path != "wiki/a.md" {
|
||||
t.Fatalf("spool = %+v, want just the in-project read, mount-relative", evs)
|
||||
}
|
||||
}
|
||||
@@ -57,6 +57,13 @@ type webConfig struct {
|
||||
Driver string `json:"driver,omitempty"` // file (default) | sqlite | postgres
|
||||
DSN string `json:"dsn,omitempty"` // sqlite file path, or a Postgres/Supabase URL
|
||||
} `json:"database,omitempty"`
|
||||
// Reads tunes read telemetry (the heat API): aggregate view counts per
|
||||
// file, split human/agent/share. On by default in hub mode; counts only,
|
||||
// no reader identities in any API.
|
||||
Reads *struct {
|
||||
Enabled *bool `json:"enabled,omitempty"` // default true
|
||||
RetentionDays int `json:"retention_days,omitempty"` // default 400; older days fold into all-time
|
||||
} `json:"reads,omitempty"`
|
||||
}
|
||||
|
||||
func loadWebConfig(path string) (webConfig, error) {
|
||||
@@ -340,6 +347,26 @@ credentials); otherwise it is relayed through this server.`,
|
||||
return fmt.Errorf("open share registry: %w", err)
|
||||
}
|
||||
srv.Shares = shares
|
||||
readsOn, retention := true, 0
|
||||
if cfg.Reads != nil {
|
||||
if cfg.Reads.Enabled != nil {
|
||||
readsOn = *cfg.Reads.Enabled
|
||||
}
|
||||
retention = cfg.Reads.RetentionDays
|
||||
}
|
||||
if readsOn {
|
||||
var reads *webapp.ReadLedger
|
||||
if meta != nil {
|
||||
reads, err = webapp.NewReadLedger(meta.Reads(), retention)
|
||||
} else {
|
||||
reads, err = webapp.OpenReadLedger(filepath.Join(filepath.Dir(projectsDB), "reads.json"), retention)
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("open read ledger: %w", err)
|
||||
}
|
||||
defer reads.Close()
|
||||
srv.Reads = reads
|
||||
}
|
||||
if meta != nil {
|
||||
display += " (db: " + cfg.Database.Driver + ")"
|
||||
} else {
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
# Read heatmap — design
|
||||
|
||||
Status: implemented (2026-07-11, all three phases) · Owner: snow · Prior art: session-linked notes (shipped), history API
|
||||
|
||||
## Problem
|
||||
|
||||
BearDrive knows everything about *writes* (journals: who, when, which session)
|
||||
and nothing about *reads*. Admins curating a shared knowledge folder can't see
|
||||
which files the team actually consumes, so they can't tell a load-bearing
|
||||
document from dead weight. The killer view is the **read×write matrix**:
|
||||
heavily-read + long-unwritten is the danger zone (stale knowledge people still
|
||||
rely on); unread + unwritten is archive material.
|
||||
|
||||
Two properties make this more than Confluence-style view counts:
|
||||
|
||||
1. **Write provenance is already perfect** (journals), so read data alone
|
||||
completes the matrix — competitors have reads but weak write history.
|
||||
2. **Agent reads are attributable.** The agent hooks + session notes shipped in
|
||||
PR #12 mean we can distinguish *human* reads from *agent* reads — the agent
|
||||
hot-path is effectively the team's context window, and nobody else can see it.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **`/store/*` sync traffic is never a read.** Devices replicating a volume is
|
||||
replication, not consumption. Only deliberate consumption counts.
|
||||
- No per-user browsing profiles in any API response. Aggregate counts only.
|
||||
- No general web analytics (referrers, dwell time, scroll depth).
|
||||
- Volume mode (`DirSource`, auth-free viewer) is out of scope — like history,
|
||||
this is a hub feature.
|
||||
|
||||
## What counts as a read
|
||||
|
||||
| Source | Kind | Actor (internal only) | Where recorded |
|
||||
|---|---|---|---|
|
||||
| Viewer `GET file` / `render` / `download` | `human` | account email | `serveBlob`, `handleRender` |
|
||||
| Share link hit `GET /s/<token>` | `share` | share token | `handleShared` |
|
||||
| Agent tool read (Read / read_file / …) | `agent` | device id | reported by client, phase 3 |
|
||||
| History version view (`/blob`) | — not counted | | spelunking ≠ consumption |
|
||||
| `/store/*` | — never | | replication |
|
||||
|
||||
Details:
|
||||
|
||||
- **Record before the ETag check.** A 304 render is still a person reading the
|
||||
file; skipping it would undercount exactly the hottest (most-cached) pages.
|
||||
- **Debounce to visits, not requests.** An in-memory `(project, path, actor)`
|
||||
seen-map with a 10-minute window collapses reload storms and the
|
||||
render-then-raw double fetch into one read. The map is pruned on the flush
|
||||
tick. Embedded assets fetched during a markdown render do count as reads of
|
||||
those assets ("this diagram is viewed a lot" is signal, not noise).
|
||||
|
||||
## Data model
|
||||
|
||||
One new MetaStore repo. Rows are daily aggregation buckets, not events — the
|
||||
ledger never stores an event log, so there is nothing sensitive to leak and
|
||||
nothing that grows per-request.
|
||||
|
||||
```go
|
||||
// ReadStat is one aggregation bucket: reads of one path by one actor on one
|
||||
// day. Day=="" is the all-time fold (see retention). Actor is an opaque
|
||||
// internal id (account email / device id / share token) used only to count
|
||||
// distinct readers; it never appears in an API response.
|
||||
type ReadStat struct {
|
||||
Project string `json:"project"`
|
||||
Path string `json:"path"`
|
||||
Day string `json:"day"` // "2026-07-11" UTC, or "" for all-time
|
||||
Kind string `json:"kind"` // human | agent | share
|
||||
Actor string `json:"actor"`
|
||||
Count int64 `json:"count"`
|
||||
Last time.Time `json:"last"`
|
||||
}
|
||||
|
||||
// ReadRepo persists read buckets. Unlike the other repos this one is batch-
|
||||
// oriented: reads are telemetry and flushes carry many dirty buckets at once —
|
||||
// one file rewrite / one SQL transaction per flush, not per bucket.
|
||||
type ReadRepo interface {
|
||||
Load() ([]ReadStat, error)
|
||||
PutBatch(stats []ReadStat) error // upsert by (project,path,day,kind,actor)
|
||||
DeleteBatch(keys []ReadStatKey) error // used by retention fold
|
||||
}
|
||||
```
|
||||
|
||||
- `MetaStore` gains `Reads() ReadRepo`; file backend adds `reads.json`
|
||||
(same load-all / rewrite-atomically discipline, 0o755 dir mode — no secrets),
|
||||
SQL backend adds one table with PK `(project, path, day, kind, actor)` and
|
||||
the usual idempotent `CREATE TABLE IF NOT EXISTS` migration.
|
||||
- `db_conformance_test.go` gets ReadRepo cases like every other repo.
|
||||
|
||||
### Cardinality & retention
|
||||
|
||||
Worst realistic case (1k files × 30 actors × 400 days, every actor reading
|
||||
every file daily) is implausible; actual rows ≈ files-actually-read × active
|
||||
actors × active days. Retention keeps it bounded regardless:
|
||||
|
||||
- Config `reads.retention_days` (default **400** — enough for a year-over-year
|
||||
view). On the daily compaction pass (at load + once per day on the flush
|
||||
loop), buckets older than the horizon are **folded into the all-time row**
|
||||
`(project, path, "", kind, actor)` and deleted. All-time totals survive
|
||||
forever; per-day resolution ages out.
|
||||
|
||||
## Server plumbing
|
||||
|
||||
New `ReadLedger` service (`webapp/reads.go`), same shape as `DeviceRegistry`:
|
||||
in-memory state over a repo, with write throttling.
|
||||
|
||||
```go
|
||||
type ReadLedger struct {
|
||||
repo ReadRepo
|
||||
mu sync.Mutex
|
||||
byKey map[readKey]ReadStat // loaded at open, bumped in memory
|
||||
dirty map[readKey]struct{} // flushed every 30s and on Close
|
||||
seen map[visitKey]time.Time // debounce window
|
||||
}
|
||||
func (l *ReadLedger) Record(project, path, kind, actor string)
|
||||
func (l *ReadLedger) Heat(project, prefix string, since time.Time) map[string]HeatEntry
|
||||
func (l *ReadLedger) Close() error // final flush
|
||||
```
|
||||
|
||||
- `Server.Reads *ReadLedger` — nil means the feature is off (mirrors
|
||||
`Devices`/`Shares`). `Record` on a nil ledger is a no-op, so call sites
|
||||
stay unconditional.
|
||||
- **Recording sites**: `serveBlob` (covers `file` + `download`),
|
||||
`handleRender`, and `handleShared`. The `proj()` resolver already knows the
|
||||
project id; it stashes it in the request context so `recordRead(r, path,
|
||||
kind)` can pick it up without changing every handler signature.
|
||||
- **Never on the failure path**: record only after the path resolved and
|
||||
membership passed (`proj()` runs `projectAllowed` first — a 403 records
|
||||
nothing).
|
||||
- Flush errors degrade silently (log once), same "never break the request"
|
||||
posture as sync: telemetry must never 500 a page view.
|
||||
|
||||
## API
|
||||
|
||||
One endpoint, membership-gated like every per-project route:
|
||||
|
||||
```
|
||||
GET /api/p/<id>/heat?prefix=<folder/>&days=30
|
||||
→ { "since": "2026-06-11", "entries": {
|
||||
"wiki/onboarding.md": { "human": 42, "agent": 17, "share": 3,
|
||||
"readers": 6, "last_read": "2026-07-10T…" },
|
||||
… } }
|
||||
```
|
||||
|
||||
- `days=0` → all-time (daily buckets + the all-time fold).
|
||||
- `readers` = distinct human actors in the window. Actor identities never
|
||||
leave the server; this is the only trace of them.
|
||||
- **No writes-join on the server**: the frontend already has per-path last
|
||||
write time from `tree`, so the read×write matrix is a client-side join.
|
||||
One source of truth for write recency, no new endpoint.
|
||||
- `/api/config` gains `"reads": {"enabled": true|false}` so the frontend knows
|
||||
whether to fetch heat at all.
|
||||
|
||||
## Frontend
|
||||
|
||||
**Phase 1 — ambient heat (all members).**
|
||||
- Folder listing rows (`renderFolderListing`): a heat dot before the meta text,
|
||||
intensity 0–4 on a log scale of 30-day reads (any-kind), plus `· 42 reads`
|
||||
in `dl-meta`. Folders show the sum of their subtree.
|
||||
- File view meta line gains `· 42 reads/30d`.
|
||||
- One `heat?days=30` fetch per project, cached alongside the tree and
|
||||
refreshed with it.
|
||||
|
||||
**Phase 2 — Insights (admins + org owners).**
|
||||
- An "Insights" panel per project: SVG quadrant scatter, **x = days since last
|
||||
write (log), y = reads in window (log)**, one point per file. Quadrants:
|
||||
hot+fresh (healthy), **hot+stale (danger zone — fix these first)**,
|
||||
cold+fresh (new, unproven), cold+stale (archive candidates).
|
||||
- Below it, the danger-zone list ranked by `reads × staleness`, each row
|
||||
linking into the file with its history.
|
||||
- A human/agent toggle (or stacked dot colors) — "what do the agents live on"
|
||||
is a different question from "what do people open".
|
||||
- Dependency-free vanilla JS/SVG like the rest of the frontend.
|
||||
|
||||
## Agent reads (phase 3)
|
||||
|
||||
Agents read files from the *local synced folder* — the hub never sees those
|
||||
reads. The hooks pipeline shipped for session notes closes the gap:
|
||||
|
||||
1. **Hook**: each platform gets one more matcher — `Read` (Claude Code),
|
||||
`read_file|read_many_files` (Gemini), `read_file` (Hermes); Codex reads are
|
||||
mostly shell commands, so coverage there is best-effort. The hook runs
|
||||
`bdrive read-log`, which reads the hook JSON on stdin, extracts
|
||||
`tool_input.file_path`, and — iff the path is inside the mount and not
|
||||
ignored — appends `{path, time}` to a spool (`reads.jsonl`, `O_APPEND`,
|
||||
capped at ~10k lines) in the volume store. No network in the hook path.
|
||||
2. **Flush**: the sync cycle (daemon remote tick and one-shot `bdrive sync`)
|
||||
drains the spool best-effort to `POST /api/p/<id>/reads` — a new
|
||||
device-token-authenticated endpoint that records each entry as
|
||||
`kind=agent, actor=<device id>`. Offline → the spool just waits; a failed
|
||||
flush never fails the cycle. `remote/http.go` gains the client call as an
|
||||
optional capability interface (the `PutSigner` pattern) so `file://`
|
||||
test remotes simply don't report.
|
||||
3. Session attribution rides along free: the persisted session note is active
|
||||
during the same turn, so the hub can tag agent read batches with the same
|
||||
session id the writes carry (kept server-side; surfaced later if wanted).
|
||||
|
||||
## Privacy & config
|
||||
|
||||
```json
|
||||
"reads": { "enabled": true, "retention_days": 400 }
|
||||
```
|
||||
|
||||
- Default **on** (aggregate-only data; a hub admin can disable it).
|
||||
- Exposed data is only ever: per-path counts by kind, distinct-reader count,
|
||||
last-read time. Who-read-what is not queryable through any API, including
|
||||
admin APIs — the actor column exists solely to make `readers` honest.
|
||||
- Heat is member-visible (it helps everyone find the good docs); the Insights
|
||||
panel is admin/org-owner only, matching the saved intent.
|
||||
|
||||
## Testing
|
||||
|
||||
- **Unit** (`reads_test.go`): debounce window, retention fold (daily → all-time),
|
||||
distinct-reader counting, nil-ledger no-ops, flush-on-close.
|
||||
- **Conformance**: ReadRepo ops across file/sqlite/postgres backends.
|
||||
- **Handler**: render/file/download/share record with the right kind; `/blob`
|
||||
and `/store/*` do not; non-member 403 records nothing; `heat` respects
|
||||
prefix + window; heat 404s in volume mode.
|
||||
- **Client** (phase 3): spool append from real hook JSON shapes (reuse the
|
||||
fixtures from `agenthooks_test.go`), flush drains + survives offline,
|
||||
cycle never fails on report errors.
|
||||
|
||||
## Phasing
|
||||
|
||||
1. **Ledger + heat** — ReadRepo (file/SQL), ReadLedger, recording sites,
|
||||
`/heat`, config block, folder heat dots + file read counts. Ships alone;
|
||||
human/share data starts accruing immediately.
|
||||
2. **Insights quadrant** — admin panel, danger-zone list. Pure frontend + the
|
||||
existing endpoints.
|
||||
3. **Agent reads** — `bdrive read-log`, hook matchers, spool, `POST /reads`,
|
||||
human/agent split in the UI. Depends on 1.
|
||||
|
||||
Phase 1 is the prerequisite for everything and is deliberately boring: one
|
||||
repo, one service, three record calls, one endpoint, two UI touches.
|
||||
|
||||
Phase 3 is the point of the feature, not tail work: human view counts are a
|
||||
commodity (every Confluence app has them); *agent* read visibility is the
|
||||
part nobody else can build. Phases are ordered by dependency, not value —
|
||||
ship 1 and 3 before polishing 2 if time is short.
|
||||
@@ -13,7 +13,10 @@
|
||||
//
|
||||
// The hook syncs the project and stamps changes with "<agent> session <id>"
|
||||
// (see `bdrive sync --note`), so hub history links every change to the agent
|
||||
// session that made it. Hooks are fast no-ops outside bdrive projects.
|
||||
// session that made it. A third hook on each platform's read-tool matcher
|
||||
// runs `bdrive read-log`, queueing agent file reads for the hub's read
|
||||
// heatmap (drained on the next sync — the hook itself never touches the
|
||||
// network). Hooks are fast no-ops outside bdrive projects.
|
||||
package agenthooks
|
||||
|
||||
import (
|
||||
@@ -28,8 +31,14 @@ import (
|
||||
"github.com/runbear-io/beardrive/internal/store"
|
||||
)
|
||||
|
||||
// marker identifies our hooks inside a config, for idempotency and status.
|
||||
const marker = "bdrive sync"
|
||||
// Markers identify our hooks inside a config, for idempotency and status.
|
||||
// The sync and read hooks are separate groups (different matchers), so each
|
||||
// carries its own marker — re-running install on a config that predates the
|
||||
// read hook adds just the missing group.
|
||||
const (
|
||||
marker = "bdrive sync"
|
||||
readMarker = "bdrive read-log"
|
||||
)
|
||||
|
||||
// Agent names, in the order they are reported.
|
||||
var Agents = []string{"claude", "codex", "gemini", "hermes"}
|
||||
@@ -53,6 +62,15 @@ func hookCommand(label string) string {
|
||||
`else bdrive sync . >/dev/null 2>&1 || true; fi'`
|
||||
}
|
||||
|
||||
// readHookCommand queues agent file reads for the hub's read heatmap:
|
||||
// `bdrive read-log` parses the hook's stdin JSON itself and only appends to
|
||||
// a local spool, so this stays cheap enough to run on every read-tool call.
|
||||
func readHookCommand() string {
|
||||
return `sh -c '` +
|
||||
`cd "${CLAUDE_PROJECT_DIR:-.}" && [ -d .bdrive ] && command -v bdrive >/dev/null || exit 0; ` +
|
||||
`bdrive read-log . >/dev/null 2>&1 || true'`
|
||||
}
|
||||
|
||||
type platform struct {
|
||||
label string // session-note label
|
||||
projectDir string // presence of this dir (project or home) = detected
|
||||
@@ -67,15 +85,17 @@ var platforms = map[string]platform{
|
||||
projectDir: ".claude",
|
||||
install: func(folder string) (string, bool, error) {
|
||||
return mergeJSONHooks(filepath.Join(folder, ".claude", "settings.json"),
|
||||
"UserPromptSubmit", "PostToolUse", "Write|Edit|MultiEdit", "claude-code", 30, true)
|
||||
"UserPromptSubmit", "PostToolUse", "Write|Edit|MultiEdit", "Read", "claude-code", 30, true)
|
||||
},
|
||||
},
|
||||
"codex": {
|
||||
label: "codex",
|
||||
projectDir: ".codex",
|
||||
install: func(folder string) (string, bool, error) {
|
||||
// Codex reads mostly happen through shell commands, so the
|
||||
// read_file matcher is best-effort coverage.
|
||||
return mergeJSONHooks(filepath.Join(folder, ".codex", "hooks.json"),
|
||||
"UserPromptSubmit", "PostToolUse", "apply_patch", "codex", 30, false)
|
||||
"UserPromptSubmit", "PostToolUse", "apply_patch", "read_file", "codex", 30, false)
|
||||
},
|
||||
note: "run /hooks inside Codex once to trust the project's .codex layer",
|
||||
},
|
||||
@@ -85,7 +105,7 @@ var platforms = map[string]platform{
|
||||
install: func(folder string) (string, bool, error) {
|
||||
// Gemini uses its own event names and millisecond timeouts.
|
||||
return mergeJSONHooks(filepath.Join(folder, ".gemini", "settings.json"),
|
||||
"BeforeAgent", "AfterTool", "write_file|replace|edit", "gemini", 30000, false)
|
||||
"BeforeAgent", "AfterTool", "write_file|replace|edit", "read_file|read_many_files", "gemini", 30000, false)
|
||||
},
|
||||
},
|
||||
"hermes": {
|
||||
@@ -159,10 +179,12 @@ func Install(folder string, agents []string) ([]Result, error) {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// mergeJSONHooks adds the pull + push hook pair to a Claude-style hooks JSON
|
||||
// file (Claude, Codex, and Gemini all use this shape: hooks.<Event> is an
|
||||
// array of {matcher?, hooks: [{type: "command", ...}]} groups).
|
||||
func mergeJSONHooks(path, pullEvent, pushEvent, pushMatcher, label string, timeout int, async bool) (string, bool, error) {
|
||||
// mergeJSONHooks adds the pull + push + read hook trio to a Claude-style
|
||||
// hooks JSON file (Claude, Codex, and Gemini all use this shape:
|
||||
// hooks.<Event> is an array of {matcher?, hooks: [{type: "command", ...}]}
|
||||
// groups). Push and read share the tool-use event under different matchers,
|
||||
// each idempotent on its own marker.
|
||||
func mergeJSONHooks(path, pullEvent, pushEvent, pushMatcher, readMatcher, label string, timeout int, async bool) (string, bool, error) {
|
||||
root := map[string]any{}
|
||||
if data, err := os.ReadFile(path); err == nil {
|
||||
if err := json.Unmarshal(data, &root); err != nil {
|
||||
@@ -182,18 +204,29 @@ func mergeJSONHooks(path, pullEvent, pushEvent, pushMatcher, label string, timeo
|
||||
"statusMessage": "beardrive: pulling latest files",
|
||||
}}}
|
||||
pushHook := map[string]any{"type": "command", "command": cmd, "timeout": timeout}
|
||||
readHook := map[string]any{"type": "command", "command": readHookCommand(), "timeout": timeout}
|
||||
if async {
|
||||
pushHook["async"] = true
|
||||
readHook["async"] = true
|
||||
}
|
||||
push := map[string]any{"matcher": pushMatcher, "hooks": []any{pushHook}}
|
||||
read := map[string]any{"matcher": readMatcher, "hooks": []any{readHook}}
|
||||
|
||||
changed := false
|
||||
for event, group := range map[string]any{pullEvent: pull, pushEvent: push} {
|
||||
arr, _ := hooks[event].([]any)
|
||||
if containsMarker(arr) {
|
||||
for _, g := range []struct {
|
||||
event string
|
||||
group map[string]any
|
||||
marker string
|
||||
}{
|
||||
{pullEvent, pull, marker},
|
||||
{pushEvent, push, marker},
|
||||
{pushEvent, read, readMarker},
|
||||
} {
|
||||
arr, _ := hooks[g.event].([]any)
|
||||
if containsMarker(arr, g.marker) {
|
||||
continue
|
||||
}
|
||||
hooks[event] = append(arr, group)
|
||||
hooks[g.event] = append(arr, g.group)
|
||||
changed = true
|
||||
}
|
||||
if !changed {
|
||||
@@ -222,17 +255,22 @@ func installHermes(string) (string, bool, error) {
|
||||
root["hooks"] = hooks
|
||||
}
|
||||
cmd := hookCommand("hermes")
|
||||
groups := map[string]any{
|
||||
"pre_llm_call": map[string]any{"command": cmd, "timeout": 30},
|
||||
"post_tool_call": map[string]any{"matcher": "write_file|patch", "command": cmd, "timeout": 30},
|
||||
groups := []struct {
|
||||
event string
|
||||
group map[string]any
|
||||
marker string
|
||||
}{
|
||||
{"pre_llm_call", map[string]any{"command": cmd, "timeout": 30}, marker},
|
||||
{"post_tool_call", map[string]any{"matcher": "write_file|patch", "command": cmd, "timeout": 30}, marker},
|
||||
{"post_tool_call", map[string]any{"matcher": "read_file", "command": readHookCommand(), "timeout": 30}, readMarker},
|
||||
}
|
||||
changed := false
|
||||
for event, group := range groups {
|
||||
arr, _ := hooks[event].([]any)
|
||||
if containsMarker(arr) {
|
||||
for _, g := range groups {
|
||||
arr, _ := hooks[g.event].([]any)
|
||||
if containsMarker(arr, g.marker) {
|
||||
continue
|
||||
}
|
||||
hooks[event] = append(arr, group)
|
||||
hooks[g.event] = append(arr, g.group)
|
||||
changed = true
|
||||
}
|
||||
if !changed {
|
||||
@@ -243,11 +281,12 @@ func installHermes(string) (string, bool, error) {
|
||||
})
|
||||
}
|
||||
|
||||
// containsMarker reports whether a hook array already holds one of ours.
|
||||
// Serializing sidesteps walking every platform's nesting by hand.
|
||||
func containsMarker(v any) bool {
|
||||
// containsMarker reports whether a hook array already holds the hook the
|
||||
// marker identifies. Serializing sidesteps walking every platform's nesting
|
||||
// by hand.
|
||||
func containsMarker(v any, m string) bool {
|
||||
data, err := json.Marshal(v)
|
||||
return err == nil && strings.Contains(string(data), marker)
|
||||
return err == nil && strings.Contains(string(data), m)
|
||||
}
|
||||
|
||||
func writeConfig(path string, marshal func() ([]byte, error)) error {
|
||||
|
||||
@@ -75,6 +75,10 @@ func TestInstallJSONPlatforms(t *testing.T) {
|
||||
if !strings.Contains(string(raw), `"async":true`) {
|
||||
t.Fatal("claude push hook should be async")
|
||||
}
|
||||
// …and the read hook, on its own matcher.
|
||||
if !strings.Contains(string(raw), "bdrive read-log") || !strings.Contains(string(raw), `"matcher":"Read"`) {
|
||||
t.Fatalf("claude read hook missing: %s", raw)
|
||||
}
|
||||
|
||||
// Codex: same schema, its own label and matcher, no async field.
|
||||
cx, _ := json.Marshal(readJSON(t, filepath.Join(folder, ".codex", "hooks.json")))
|
||||
@@ -84,10 +88,13 @@ func TestInstallJSONPlatforms(t *testing.T) {
|
||||
if strings.Contains(string(cx), "async") {
|
||||
t.Fatal("codex should not get the claude-only async field")
|
||||
}
|
||||
if !strings.Contains(string(cx), "bdrive read-log") || !strings.Contains(string(cx), `"matcher":"read_file"`) {
|
||||
t.Fatalf("codex read hook missing: %s", cx)
|
||||
}
|
||||
|
||||
// Gemini: its own event names and ms timeout.
|
||||
gm, _ := json.Marshal(readJSON(t, filepath.Join(folder, ".gemini", "settings.json")))
|
||||
for _, want := range []string{"BeforeAgent", "AfterTool", "gemini session $s", "30000"} {
|
||||
for _, want := range []string{"BeforeAgent", "AfterTool", "gemini session $s", "30000", "bdrive read-log", "read_file|read_many_files"} {
|
||||
if !strings.Contains(string(gm), want) {
|
||||
t.Fatalf("gemini hooks missing %q: %s", want, gm)
|
||||
}
|
||||
@@ -114,8 +121,8 @@ func TestInstallIdempotentAndPreserving(t *testing.T) {
|
||||
if _, ok := cfg["permissions"]; !ok {
|
||||
t.Fatal("merge dropped unrelated settings keys")
|
||||
}
|
||||
if got := len(cfg["hooks"].(map[string]any)["PostToolUse"].([]any)); got != 2 {
|
||||
t.Fatalf("PostToolUse groups = %d, want user's + ours", got)
|
||||
if got := len(cfg["hooks"].(map[string]any)["PostToolUse"].([]any)); got != 3 {
|
||||
t.Fatalf("PostToolUse groups = %d, want user's + our push + our read", got)
|
||||
}
|
||||
|
||||
// Second install: no change, byte-identical file.
|
||||
@@ -133,6 +140,38 @@ func TestInstallIdempotentAndPreserving(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A config from before the read hook existed (sync hooks only) gains just
|
||||
// the read group on re-install — the sync hooks are not duplicated.
|
||||
func TestInstallUpgradesSyncOnlyConfig(t *testing.T) {
|
||||
t.Setenv("HOME", t.TempDir())
|
||||
folder := t.TempDir()
|
||||
old := `{"hooks":{
|
||||
"UserPromptSubmit":[{"hooks":[{"type":"command","command":"sh -c 'bdrive sync .'"}]}],
|
||||
"PostToolUse":[{"matcher":"Write|Edit|MultiEdit","hooks":[{"type":"command","command":"sh -c 'bdrive sync .'"}]}]}}`
|
||||
os.MkdirAll(filepath.Join(folder, ".claude"), 0o755)
|
||||
os.WriteFile(filepath.Join(folder, ".claude", "settings.json"), []byte(old), 0o644)
|
||||
|
||||
results, err := Install(folder, []string{"claude"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !results[0].Changed {
|
||||
t.Fatal("upgrade install reported unchanged")
|
||||
}
|
||||
cfg := readJSON(t, filepath.Join(folder, ".claude", "settings.json"))
|
||||
hooks := cfg["hooks"].(map[string]any)
|
||||
if got := len(hooks["UserPromptSubmit"].([]any)); got != 1 {
|
||||
t.Fatalf("UserPromptSubmit groups = %d, want the old one only", got)
|
||||
}
|
||||
if got := len(hooks["PostToolUse"].([]any)); got != 2 {
|
||||
t.Fatalf("PostToolUse groups = %d, want old push + new read", got)
|
||||
}
|
||||
raw, _ := json.Marshal(cfg)
|
||||
if !strings.Contains(string(raw), "bdrive read-log") {
|
||||
t.Fatal("upgrade did not add the read hook")
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstallHermesYAML(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
t.Setenv("HOME", home)
|
||||
@@ -162,9 +201,15 @@ func TestInstallHermesYAML(t *testing.T) {
|
||||
t.Fatalf("hermes missing %s", ev)
|
||||
}
|
||||
}
|
||||
if got := len(hooks["post_tool_call"].([]any)); got != 2 {
|
||||
t.Fatalf("hermes post_tool_call groups = %d, want push + read", got)
|
||||
}
|
||||
if !strings.Contains(string(data), "hermes session $s") {
|
||||
t.Fatal("hermes hook lacks its session-note label")
|
||||
}
|
||||
if !strings.Contains(string(data), "bdrive read-log") {
|
||||
t.Fatal("hermes read hook missing")
|
||||
}
|
||||
|
||||
// Idempotent.
|
||||
results, _ = Install(t.TempDir(), []string{"hermes"})
|
||||
@@ -231,6 +276,47 @@ func TestHookCommandExtraction(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The read hook must hand the event JSON through to `bdrive read-log`
|
||||
// untouched — the binary does the parsing, the shell only guards.
|
||||
func TestReadHookCommand(t *testing.T) {
|
||||
if _, err := os.Stat("/bin/sh"); err != nil {
|
||||
t.Skip("no /bin/sh")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
os.MkdirAll(filepath.Join(dir, ".bdrive"), 0o755)
|
||||
bin := filepath.Join(dir, "bin")
|
||||
os.MkdirAll(bin, 0o755)
|
||||
fake := "#!/bin/sh\necho \"$@\" > \"" + dir + "/args.txt\"\ncat > \"" + dir + "/stdin.txt\"\n"
|
||||
os.WriteFile(filepath.Join(bin, "bdrive"), []byte(fake), 0o755)
|
||||
|
||||
payload := `{"session_id":"abc","tool_name":"Read","tool_input":{"file_path":"/x/wiki/a.md"}}`
|
||||
sh := "cd " + dir + " && PATH=" + bin + ":$PATH " + readHookCommand()
|
||||
if err := runShell(t, sh, payload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
args, err := os.ReadFile(filepath.Join(dir, "args.txt"))
|
||||
if err != nil {
|
||||
t.Fatalf("read hook never called bdrive: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(args), "read-log .") {
|
||||
t.Fatalf("bdrive argv = %q, want read-log .", args)
|
||||
}
|
||||
stdin, _ := os.ReadFile(filepath.Join(dir, "stdin.txt"))
|
||||
if string(stdin) != payload {
|
||||
t.Fatalf("event JSON not passed through: %q", stdin)
|
||||
}
|
||||
|
||||
// Outside a bdrive project the hook exits before invoking anything.
|
||||
os.Remove(filepath.Join(dir, "args.txt"))
|
||||
os.RemoveAll(filepath.Join(dir, ".bdrive"))
|
||||
if err := runShell(t, sh, payload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "args.txt")); !os.IsNotExist(err) {
|
||||
t.Fatal("hook invoked bdrive outside a project")
|
||||
}
|
||||
}
|
||||
|
||||
func runShell(t *testing.T, script, stdin string) error {
|
||||
t.Helper()
|
||||
cmd := exec.Command("/bin/sh", "-c", script)
|
||||
|
||||
@@ -248,4 +248,28 @@ func (b *httpBackend) putViaServer(ctx context.Context, key string, r io.Reader,
|
||||
return nil
|
||||
}
|
||||
|
||||
// ReportReads sends the device's queued agent reads to the hub's read
|
||||
// ledger, where they count as agent traffic (actor = this device).
|
||||
func (b *httpBackend) ReportReads(ctx context.Context, reads []ReadEvent) error {
|
||||
body, err := json.Marshal(map[string]any{"reads": reads})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
|
||||
b.base+"/api/p/"+b.project+"/reads", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := b.do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpError(resp)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *httpBackend) Close() error { return nil }
|
||||
|
||||
@@ -52,6 +52,20 @@ type Backend interface {
|
||||
Close() error
|
||||
}
|
||||
|
||||
// ReadEvent is one agent file read reported to the hub for its read heatmap.
|
||||
type ReadEvent struct {
|
||||
Path string `json:"path"`
|
||||
Time time.Time `json:"time,omitzero"`
|
||||
}
|
||||
|
||||
// ReadReporter is the optional read-telemetry capability, in the PutSigner
|
||||
// mold: backends that sync through a hub report the device's agent reads so
|
||||
// the heat view can split human from agent traffic. Object-store backends
|
||||
// simply don't implement it — there is no hub to tell.
|
||||
type ReadReporter interface {
|
||||
ReportReads(ctx context.Context, reads []ReadEvent) error
|
||||
}
|
||||
|
||||
// Open creates a backend from a remote URL.
|
||||
func Open(ctx context.Context, raw string) (Backend, error) {
|
||||
u, err := url.Parse(raw)
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The read spool queues agent tool reads observed by `bdrive read-log` (the
|
||||
// agent read hook) until a sync cycle drains it to the hub, where they count
|
||||
// as agent traffic in the read heatmap. Hooks only append locally — no
|
||||
// network on the hook path — and the flush is best-effort: offline, the
|
||||
// spool just waits.
|
||||
|
||||
// ReadEvent is one observed read of a synced file (mount-relative path).
|
||||
type ReadEvent struct {
|
||||
Path string `json:"path"`
|
||||
Time time.Time `json:"time"`
|
||||
}
|
||||
|
||||
// readSpoolMax caps the spool: past it new events are dropped rather than
|
||||
// letting an unreachable hub grow telemetry without bound.
|
||||
const readSpoolMax = 1 << 20
|
||||
|
||||
// readReportMax bounds one drained batch to what the hub accepts per report.
|
||||
const readReportMax = 4096
|
||||
|
||||
func (s *Store) readSpoolPath() string { return filepath.Join(s.dir, "reads.jsonl") }
|
||||
func (s *Store) readFlushPath() string { return filepath.Join(s.dir, "reads-flushing.jsonl") }
|
||||
|
||||
// LogRead appends one read event to the spool. Single-line O_APPEND writes
|
||||
// keep concurrent hook invocations from interleaving.
|
||||
func (s *Store) LogRead(rel string) error {
|
||||
if fi, err := os.Stat(s.readSpoolPath()); err == nil && fi.Size() > readSpoolMax {
|
||||
return nil // spool full: drop, never grow unbounded
|
||||
}
|
||||
line, err := json.Marshal(ReadEvent{Path: rel, Time: time.Now().UTC()})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.OpenFile(s.readSpoolPath(), os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
_, err = f.Write(append(line, '\n'))
|
||||
return err
|
||||
}
|
||||
|
||||
// PendingReads returns the queued batch awaiting report, deduplicated by path
|
||||
// (latest time wins). The spool is rotated aside first, so events logged
|
||||
// after this call land in a fresh spool; the batch survives until
|
||||
// ClearPendingReads — a failed report is simply retried next cycle.
|
||||
func (s *Store) PendingReads() ([]ReadEvent, error) {
|
||||
if _, err := os.Stat(s.readFlushPath()); os.IsNotExist(err) {
|
||||
if err := os.Rename(s.readSpoolPath(), s.readFlushPath()); err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil // nothing queued
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
data, err := os.ReadFile(s.readFlushPath())
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
latest := map[string]time.Time{}
|
||||
var order []string
|
||||
for _, line := range bytes.Split(data, []byte("\n")) {
|
||||
if len(bytes.TrimSpace(line)) == 0 {
|
||||
continue
|
||||
}
|
||||
var e ReadEvent
|
||||
if json.Unmarshal(line, &e) != nil || e.Path == "" {
|
||||
continue // torn or corrupt line; drop it
|
||||
}
|
||||
if _, ok := latest[e.Path]; !ok {
|
||||
order = append(order, e.Path)
|
||||
}
|
||||
if e.Time.After(latest[e.Path]) {
|
||||
latest[e.Path] = e.Time
|
||||
}
|
||||
}
|
||||
if len(order) > readReportMax {
|
||||
order = order[len(order)-readReportMax:]
|
||||
}
|
||||
out := make([]ReadEvent, 0, len(order))
|
||||
for _, p := range order {
|
||||
out = append(out, ReadEvent{Path: p, Time: latest[p]})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ClearPendingReads drops the batch PendingReads returned, after a
|
||||
// successful report.
|
||||
func (s *Store) ClearPendingReads() error {
|
||||
err := os.Remove(s.readFlushPath())
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func openTestStore(t *testing.T) *Store {
|
||||
t.Helper()
|
||||
s, err := Open(filepath.Join(t.TempDir(), "volume"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func TestReadSpool(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
|
||||
// Nothing queued: no batch, no error.
|
||||
if evs, err := s.PendingReads(); err != nil || len(evs) != 0 {
|
||||
t.Fatalf("empty spool = %v, %v", evs, err)
|
||||
}
|
||||
|
||||
// Repeat reads of one path dedupe to its latest event.
|
||||
for i := 0; i < 3; i++ {
|
||||
if err := s.LogRead("wiki/a.md"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := s.LogRead("b.md"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
evs, err := s.PendingReads()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(evs) != 2 || evs[0].Path != "wiki/a.md" || evs[1].Path != "b.md" {
|
||||
t.Fatalf("batch = %+v, want deduped a.md + b.md", evs)
|
||||
}
|
||||
if evs[0].Time.IsZero() {
|
||||
t.Fatal("events must carry their read time")
|
||||
}
|
||||
|
||||
// The batch survives until cleared — a failed report just retries — and
|
||||
// reads logged meanwhile land in a fresh spool behind it.
|
||||
if err := s.LogRead("c.md"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again, err := s.PendingReads()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(again) != 2 || again[0].Path != "wiki/a.md" {
|
||||
t.Fatalf("retry batch = %+v, want the same uncleared batch", again)
|
||||
}
|
||||
if err := s.ClearPendingReads(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
next, err := s.PendingReads()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(next) != 1 || next[0].Path != "c.md" {
|
||||
t.Fatalf("post-clear batch = %+v, want just c.md", next)
|
||||
}
|
||||
s.ClearPendingReads()
|
||||
if evs, _ := s.PendingReads(); len(evs) != 0 {
|
||||
t.Fatalf("drained spool still returned %+v", evs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadSpoolSurvivesCorruptLines(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
s.LogRead("good.md")
|
||||
f, err := os.OpenFile(s.readSpoolPath(), os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.WriteString(`{"path": "torn`) // a torn write
|
||||
f.Close()
|
||||
s.LogRead("also-good.md")
|
||||
evs, err := s.PendingReads()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The torn line joins the next event's line; both are dropped, but the
|
||||
// batch itself survives.
|
||||
if len(evs) == 0 || evs[0].Path != "good.md" {
|
||||
t.Fatalf("batch = %+v, want good.md to survive the torn line", evs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadSpoolCap(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
long := strings.Repeat("d", 1024)
|
||||
for i := 0; i < 1100; i++ { // ~1.1 MB of events
|
||||
if err := s.LogRead(long + "/" + string(rune('a'+i%26)) + ".md"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
fi, err := os.Stat(s.readSpoolPath())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if fi.Size() > readSpoolMax+4096 {
|
||||
t.Fatalf("spool grew past its cap: %d bytes", fi.Size())
|
||||
}
|
||||
}
|
||||
@@ -53,6 +53,14 @@ type pattern struct {
|
||||
negate bool
|
||||
}
|
||||
|
||||
// LoadFilter builds the filter for a folder from its .bdriveignore (if any)
|
||||
// plus the include list from the .bdrive settings file — the exact rules the
|
||||
// sync cycle applies, for callers outside the cycle (e.g. `bdrive read-log`
|
||||
// deciding whether an agent-read path is part of the project).
|
||||
func LoadFilter(folder string, include []string) (*Filter, error) {
|
||||
return loadFilter(folder, include)
|
||||
}
|
||||
|
||||
// loadFilter builds the filter for a folder from its .bdriveignore (if any)
|
||||
// plus the include list from the .bdrive settings file.
|
||||
func loadFilter(folder string, include []string) (*Filter, error) {
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
package syncer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/runbear-io/beardrive/internal/remote"
|
||||
)
|
||||
|
||||
// readReportingRemote wraps a backend with the hub's ReadReporter capability,
|
||||
// standing in for the https:// backend in the multi-device harness.
|
||||
type readReportingRemote struct {
|
||||
remote.Backend
|
||||
|
||||
mu sync.Mutex
|
||||
fail bool
|
||||
reports [][]remote.ReadEvent
|
||||
}
|
||||
|
||||
func (r *readReportingRemote) ReportReads(_ context.Context, reads []remote.ReadEvent) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.fail {
|
||||
return fmt.Errorf("hub unreachable")
|
||||
}
|
||||
cp := make([]remote.ReadEvent, len(reads))
|
||||
copy(cp, reads)
|
||||
r.reports = append(r.reports, cp)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *readReportingRemote) setFail(v bool) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.fail = v
|
||||
}
|
||||
|
||||
func (r *readReportingRemote) all() [][]remote.ReadEvent {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.reports
|
||||
}
|
||||
|
||||
// TestAgentReadReporting drives the read spool through real sync cycles: the
|
||||
// queued reads flush to a reporting hub (deduped), survive an unreachable hub
|
||||
// and retry, and never disturb the sync result itself.
|
||||
func TestAgentReadReporting(t *testing.T) {
|
||||
hub := &readReportingRemote{Backend: sharedRemote(t)}
|
||||
a := newDevice(t, "deva", hub)
|
||||
write(t, a.Folder, "wiki/a.md", "content")
|
||||
|
||||
// The agent read a.md twice and b.md once before this cycle.
|
||||
a.Store.LogRead("wiki/a.md")
|
||||
a.Store.LogRead("wiki/a.md")
|
||||
a.Store.LogRead("b.md")
|
||||
res := cycle(t, a)
|
||||
if !res.Pushed {
|
||||
t.Fatal("cycle should have pushed")
|
||||
}
|
||||
reports := hub.all()
|
||||
if len(reports) != 1 || len(reports[0]) != 2 {
|
||||
t.Fatalf("reports = %+v, want one deduped batch of 2", reports)
|
||||
}
|
||||
if reports[0][0].Path != "wiki/a.md" || reports[0][1].Path != "b.md" {
|
||||
t.Fatalf("batch = %+v", reports[0])
|
||||
}
|
||||
// Drained: an idle cycle reports nothing.
|
||||
cycle(t, a)
|
||||
if len(hub.all()) != 1 {
|
||||
t.Fatal("empty spool still produced a report")
|
||||
}
|
||||
|
||||
// Hub down: the cycle still succeeds and the batch stays queued.
|
||||
hub.setFail(true)
|
||||
a.Store.LogRead("wiki/a.md")
|
||||
if res := cycle(t, a); res.Offline {
|
||||
t.Fatal("a failed read report must not mark the cycle offline")
|
||||
}
|
||||
if len(hub.all()) != 1 {
|
||||
t.Fatal("failed report should not have landed")
|
||||
}
|
||||
// Hub back: the next cycle retries the same batch.
|
||||
hub.setFail(false)
|
||||
cycle(t, a)
|
||||
reports = hub.all()
|
||||
if len(reports) != 2 || len(reports[1]) != 1 || reports[1][0].Path != "wiki/a.md" {
|
||||
t.Fatalf("retry reports = %+v", reports)
|
||||
}
|
||||
|
||||
// A backend without the capability (plain object store) is untouched by
|
||||
// queued reads: the cycle runs, the spool just keeps waiting.
|
||||
b := newDevice(t, "devb", sharedRemote(t))
|
||||
write(t, b.Folder, "x.md", "x")
|
||||
b.Store.LogRead("x.md")
|
||||
cycle(t, b)
|
||||
if evs, err := b.Store.PendingReads(); err != nil || len(evs) != 1 {
|
||||
t.Fatalf("spool on a hubless device = %v, %v; want the read still queued", evs, err)
|
||||
}
|
||||
}
|
||||
@@ -199,6 +199,21 @@ func (s *Session) Cycle(ctx context.Context) (*Result, error) {
|
||||
}
|
||||
}
|
||||
|
||||
// 6. Drain the agent read spool to the hub (read heatmap telemetry).
|
||||
// Strictly best-effort: a failed report keeps the batch queued for the
|
||||
// next cycle and never fails — or even marks offline — this one.
|
||||
if rr, ok := s.Backend.(remote.ReadReporter); ok && !res.Offline {
|
||||
if evs, err := s.Store.PendingReads(); err == nil && len(evs) > 0 {
|
||||
reads := make([]remote.ReadEvent, len(evs))
|
||||
for i, e := range evs {
|
||||
reads[i] = remote.ReadEvent{Path: e.Path, Time: e.Time}
|
||||
}
|
||||
if rr.ReportReads(ctx, reads) == nil {
|
||||
s.Store.ClearPendingReads()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := s.Store.SaveCache(s.mountID(), cache); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@ type MetaStore interface {
|
||||
Orgs() OrgRepo
|
||||
Shares() ShareRepo
|
||||
Devices() DeviceRepo
|
||||
Reads() ReadRepo
|
||||
Close() error
|
||||
}
|
||||
|
||||
@@ -56,3 +57,13 @@ type DeviceRepo interface {
|
||||
Load() ([]DeviceInfo, error)
|
||||
Put(d DeviceInfo) error
|
||||
}
|
||||
|
||||
// ReadRepo persists read-telemetry buckets (see ReadStat). Unlike the other
|
||||
// repos it is batch-oriented: reads are telemetry, and the ledger flushes many
|
||||
// dirty buckets at once — one file rewrite / one SQL transaction per flush,
|
||||
// not one write per bucket.
|
||||
type ReadRepo interface {
|
||||
Load() ([]ReadStat, error)
|
||||
PutBatch(stats []ReadStat) error // upsert by (project, path, day, kind, actor)
|
||||
DeleteBatch(keys []ReadStatKey) error
|
||||
}
|
||||
|
||||
@@ -55,7 +55,7 @@ func metaBackends(t *testing.T) []metaBackend {
|
||||
t.Fatalf("postgres reset: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
db.Exec(`DROP TABLE IF EXISTS accounts, tokens, auth_policy, projects, orgs, org_members, invites, shares, devices`)
|
||||
db.Exec(`DROP TABLE IF EXISTS accounts, tokens, auth_policy, projects, orgs, org_members, invites, shares, devices, read_stats`)
|
||||
},
|
||||
open: func(t *testing.T) MetaStore {
|
||||
s, err := OpenSQLStore("pgx", dsn)
|
||||
@@ -176,6 +176,17 @@ func TestMetaStoreConformance(t *testing.T) {
|
||||
}
|
||||
devices.Observe(DeviceInfo{ID: "d1", Name: "laptop", OS: "mac", User: "dev@x.io", IP: "1.2.3.4"})
|
||||
|
||||
reads, err := NewReadLedger(st.Reads(), 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reads.Record(p1.ID, "handbook.md", ReadKindHuman, "dev@x.io")
|
||||
reads.Record(p1.ID, "handbook.md", ReadKindHuman, "boss@x.io")
|
||||
reads.Record(p1.ID, "wiki/deep.md", ReadKindAgent, "d1")
|
||||
if err := reads.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if err := st.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -236,6 +247,21 @@ func TestMetaStoreConformance(t *testing.T) {
|
||||
if !ok || d.Name != "laptop" || d.IP != "1.2.3.4" {
|
||||
t.Fatalf("device lost across reload: %+v", d)
|
||||
}
|
||||
|
||||
reads2, err := NewReadLedger(st2.Reads(), 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
heat := reads2.Heat(p1.ID, "", time.Time{})
|
||||
if e := heat["handbook.md"]; e.Human != 2 || e.Readers != 2 {
|
||||
t.Fatalf("read buckets lost across reload: %+v", e)
|
||||
}
|
||||
if e := heat["wiki/deep.md"]; e.Agent != 1 || e.Readers != 0 {
|
||||
t.Fatalf("agent read bucket lost across reload: %+v", e)
|
||||
}
|
||||
if sub := reads2.Heat(p1.ID, "wiki", time.Time{}); len(sub) != 1 {
|
||||
t.Fatalf("prefix heat = %+v, want only wiki/deep.md", sub)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -61,6 +61,7 @@ type fileMetaStore struct {
|
||||
orgs *fileOrgRepo
|
||||
shares *fileShareRepo
|
||||
devices *fileDeviceRepo
|
||||
reads *fileReadRepo
|
||||
}
|
||||
|
||||
// OpenFileStore builds the file backend over dir, using the historical
|
||||
@@ -72,6 +73,7 @@ func OpenFileStore(dir string) (MetaStore, error) {
|
||||
orgs: newFileOrgRepo(filepath.Join(dir, "orgs.json")),
|
||||
shares: newFileShareRepo(filepath.Join(dir, "shares.json")),
|
||||
devices: newFileDeviceRepo(filepath.Join(dir, "devices.json")),
|
||||
reads: newFileReadRepo(filepath.Join(dir, "reads.json")),
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -80,6 +82,7 @@ func (s *fileMetaStore) Projects() ProjectRepo { return s.projects }
|
||||
func (s *fileMetaStore) Orgs() OrgRepo { return s.orgs }
|
||||
func (s *fileMetaStore) Shares() ShareRepo { return s.shares }
|
||||
func (s *fileMetaStore) Devices() DeviceRepo { return s.devices }
|
||||
func (s *fileMetaStore) Reads() ReadRepo { return s.reads }
|
||||
func (s *fileMetaStore) Close() error { return nil }
|
||||
|
||||
// ---- accounts (auth.json: users + tokens + policy) ----
|
||||
@@ -419,3 +422,72 @@ func (r *fileDeviceRepo) Put(d DeviceInfo) error {
|
||||
r.byID[d.ID] = d
|
||||
return r.write()
|
||||
}
|
||||
|
||||
// ---- reads (reads.json) ----
|
||||
|
||||
type fileReadRepo struct {
|
||||
path string
|
||||
mu sync.Mutex
|
||||
byKey map[ReadStatKey]ReadStat
|
||||
}
|
||||
|
||||
func newFileReadRepo(path string) *fileReadRepo {
|
||||
return &fileReadRepo{path: path, byKey: map[ReadStatKey]ReadStat{}}
|
||||
}
|
||||
|
||||
func (r *fileReadRepo) Load() ([]ReadStat, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
var f struct {
|
||||
Reads []ReadStat `json:"reads"`
|
||||
}
|
||||
if _, err := readJSONFile(r.path, &f); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r.byKey = map[ReadStatKey]ReadStat{}
|
||||
for _, st := range f.Reads {
|
||||
r.byKey[st.key()] = st
|
||||
}
|
||||
return f.Reads, nil
|
||||
}
|
||||
|
||||
func (r *fileReadRepo) write() error {
|
||||
var f struct {
|
||||
Reads []ReadStat `json:"reads"`
|
||||
}
|
||||
f.Reads = make([]ReadStat, 0, len(r.byKey))
|
||||
for _, st := range r.byKey {
|
||||
f.Reads = append(f.Reads, st)
|
||||
}
|
||||
sort.Slice(f.Reads, func(i, j int) bool {
|
||||
a, b := f.Reads[i], f.Reads[j]
|
||||
if a.Path != b.Path {
|
||||
return a.Path < b.Path
|
||||
}
|
||||
return a.Day < b.Day
|
||||
})
|
||||
data, err := json.Marshal(f) // telemetry: compact beats pretty
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// 0700 dir: buckets carry actor emails, like auth.json carries accounts.
|
||||
return writeFileAtomic(r.path, append(data, '\n'), 0o700)
|
||||
}
|
||||
|
||||
func (r *fileReadRepo) PutBatch(stats []ReadStat) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
for _, st := range stats {
|
||||
r.byKey[st.key()] = st
|
||||
}
|
||||
return r.write()
|
||||
}
|
||||
|
||||
func (r *fileReadRepo) DeleteBatch(keys []ReadStatKey) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
for _, k := range keys {
|
||||
delete(r.byKey, k)
|
||||
}
|
||||
return r.write()
|
||||
}
|
||||
|
||||
@@ -33,6 +33,7 @@ type sqlMetaStore struct {
|
||||
orgs *sqlOrgRepo
|
||||
shares *sqlShareRepo
|
||||
devices *sqlDeviceRepo
|
||||
reads *sqlReadRepo
|
||||
}
|
||||
|
||||
// OpenSQLStore opens (and migrates) a SQL metadata store. driver is "sqlite"
|
||||
@@ -66,6 +67,7 @@ func OpenSQLStore(driver, dsn string) (MetaStore, error) {
|
||||
s.orgs = &sqlOrgRepo{s}
|
||||
s.shares = &sqlShareRepo{s}
|
||||
s.devices = &sqlDeviceRepo{s}
|
||||
s.reads = &sqlReadRepo{s}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
@@ -74,6 +76,7 @@ func (s *sqlMetaStore) Projects() ProjectRepo { return s.projects }
|
||||
func (s *sqlMetaStore) Orgs() OrgRepo { return s.orgs }
|
||||
func (s *sqlMetaStore) Shares() ShareRepo { return s.shares }
|
||||
func (s *sqlMetaStore) Devices() DeviceRepo { return s.devices }
|
||||
func (s *sqlMetaStore) Reads() ReadRepo { return s.reads }
|
||||
func (s *sqlMetaStore) Close() error { return s.db.Close() }
|
||||
|
||||
// q rebinds ?-placeholders to $1,$2,… for Postgres; SQLite keeps ?.
|
||||
@@ -153,6 +156,11 @@ func (s *sqlMetaStore) migrate() error {
|
||||
`CREATE TABLE IF NOT EXISTS devices (
|
||||
id TEXT PRIMARY KEY, name TEXT NOT NULL DEFAULT '', os TEXT NOT NULL DEFAULT '',
|
||||
user_email TEXT NOT NULL DEFAULT '', ip TEXT NOT NULL DEFAULT '', last_seen TEXT NOT NULL DEFAULT '')`,
|
||||
`CREATE TABLE IF NOT EXISTS read_stats (
|
||||
project TEXT NOT NULL, path TEXT NOT NULL, day TEXT NOT NULL DEFAULT '',
|
||||
kind TEXT NOT NULL, actor TEXT NOT NULL,
|
||||
count INTEGER NOT NULL DEFAULT 0, last TEXT NOT NULL DEFAULT '',
|
||||
PRIMARY KEY (project, path, day, kind, actor))`,
|
||||
}
|
||||
for _, st := range stmts {
|
||||
if _, err := s.db.Exec(st); err != nil {
|
||||
@@ -462,3 +470,59 @@ func (r *sqlDeviceRepo) Put(d DeviceInfo) error {
|
||||
ip=excluded.ip, last_seen=excluded.last_seen`,
|
||||
d.ID, d.Name, d.OS, d.User, d.IP, tenc(d.LastSeen))
|
||||
}
|
||||
|
||||
// ---- reads ----
|
||||
|
||||
type sqlReadRepo struct{ s *sqlMetaStore }
|
||||
|
||||
func (r *sqlReadRepo) Load() ([]ReadStat, error) {
|
||||
rows, err := r.s.db.Query(`SELECT project, path, day, kind, actor, count, last FROM read_stats`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []ReadStat
|
||||
for rows.Next() {
|
||||
var st ReadStat
|
||||
var last string
|
||||
if err := rows.Scan(&st.Project, &st.Path, &st.Day, &st.Kind, &st.Actor, &st.Count, &last); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
st.Last = tdec(last)
|
||||
out = append(out, st)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
func (r *sqlReadRepo) PutBatch(stats []ReadStat) error {
|
||||
tx, err := r.s.db.Begin()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
for _, st := range stats {
|
||||
if _, err := tx.Exec(r.s.q(`INSERT INTO read_stats (project,path,day,kind,actor,count,last)
|
||||
VALUES (?,?,?,?,?,?,?)
|
||||
ON CONFLICT(project,path,day,kind,actor) DO UPDATE SET count=excluded.count, last=excluded.last`),
|
||||
st.Project, st.Path, st.Day, st.Kind, st.Actor, st.Count, tenc(st.Last)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
func (r *sqlReadRepo) DeleteBatch(keys []ReadStatKey) error {
|
||||
tx, err := r.s.db.Begin()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
for _, k := range keys {
|
||||
if _, err := tx.Exec(r.s.q(`DELETE FROM read_stats
|
||||
WHERE project = ? AND path = ? AND day = ? AND kind = ? AND actor = ?`),
|
||||
k.Project, k.Path, k.Day, k.Kind, k.Actor); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
package webapp
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Read telemetry: who consumes what, aggregated. Together with the write
|
||||
// history the journals already carry, this completes the read×write matrix —
|
||||
// heavily-read + long-unwritten is the danger zone an admin should fix first
|
||||
// (see docs/design/read-heatmap.md).
|
||||
//
|
||||
// What counts as a read: viewer file/render/download hits (human), share-link
|
||||
// hits (share), and agent tool reads reported by syncing devices (agent).
|
||||
// /store/* sync traffic is replication, not reading, and is never counted;
|
||||
// history /blob views are spelunking, not consumption, and aren't either.
|
||||
//
|
||||
// Privacy: rows are daily aggregation buckets, never an event log. The actor
|
||||
// column (account email / device id / share token) exists only to count
|
||||
// distinct readers and never appears in an API response.
|
||||
|
||||
// Read kinds.
|
||||
const (
|
||||
ReadKindHuman = "human"
|
||||
ReadKindAgent = "agent"
|
||||
ReadKindShare = "share"
|
||||
)
|
||||
|
||||
// ReadStat is one aggregation bucket: reads of one path by one actor on one
|
||||
// day. Day == "" is the all-time fold that survives retention.
|
||||
type ReadStat struct {
|
||||
Project string `json:"project"`
|
||||
Path string `json:"path"`
|
||||
Day string `json:"day"` // "2006-01-02" UTC, or "" for all-time
|
||||
Kind string `json:"kind"`
|
||||
Actor string `json:"actor"`
|
||||
Count int64 `json:"count"`
|
||||
Last time.Time `json:"last"`
|
||||
}
|
||||
|
||||
// ReadStatKey identifies one bucket.
|
||||
type ReadStatKey struct {
|
||||
Project, Path, Day, Kind, Actor string
|
||||
}
|
||||
|
||||
func (s ReadStat) key() ReadStatKey {
|
||||
return ReadStatKey{s.Project, s.Path, s.Day, s.Kind, s.Actor}
|
||||
}
|
||||
|
||||
// HeatEntry is the per-path aggregate the heat API returns. Counts only —
|
||||
// never identities.
|
||||
type HeatEntry struct {
|
||||
Human int64 `json:"human,omitempty"`
|
||||
Agent int64 `json:"agent,omitempty"`
|
||||
Share int64 `json:"share,omitempty"`
|
||||
Readers int `json:"readers,omitempty"` // distinct human readers
|
||||
LastRead time.Time `json:"last_read,omitzero"`
|
||||
}
|
||||
|
||||
const (
|
||||
// readDebounce collapses request storms (reloads, render-then-raw double
|
||||
// fetches) into visits: repeat reads of a path by the same actor within
|
||||
// the window don't count again.
|
||||
readDebounce = 10 * time.Minute
|
||||
// readFlushEvery throttles persistence; dirty buckets ride in memory
|
||||
// between flushes, so a crash loses at most this much telemetry.
|
||||
readFlushEvery = 30 * time.Second
|
||||
// DefaultReadRetentionDays is how long daily buckets keep per-day
|
||||
// resolution before folding into the all-time row.
|
||||
DefaultReadRetentionDays = 400
|
||||
)
|
||||
|
||||
// ReadLedger is the in-memory read-telemetry service over a ReadRepo, in the
|
||||
// mold of DeviceRegistry: reads stay in memory, writes are throttled. There is
|
||||
// no background goroutine — flushes piggyback on Record calls, and telemetry
|
||||
// failures never surface to the request that triggered them.
|
||||
type ReadLedger struct {
|
||||
repo ReadRepo
|
||||
retention time.Duration
|
||||
|
||||
mu sync.Mutex
|
||||
byKey map[ReadStatKey]ReadStat
|
||||
dirty map[ReadStatKey]bool
|
||||
pendingDel []ReadStatKey // retention deletions awaiting a successful flush
|
||||
seen map[ReadStatKey]time.Time // debounce; Day field unused ("")
|
||||
lastFlush time.Time
|
||||
warned bool
|
||||
}
|
||||
|
||||
// NewReadLedger loads the ledger and immediately folds buckets older than the
|
||||
// retention horizon into their all-time rows. retentionDays <= 0 means the
|
||||
// default.
|
||||
func NewReadLedger(repo ReadRepo, retentionDays int) (*ReadLedger, error) {
|
||||
if retentionDays <= 0 {
|
||||
retentionDays = DefaultReadRetentionDays
|
||||
}
|
||||
l := &ReadLedger{
|
||||
repo: repo,
|
||||
retention: time.Duration(retentionDays) * 24 * time.Hour,
|
||||
byKey: map[ReadStatKey]ReadStat{},
|
||||
dirty: map[ReadStatKey]bool{},
|
||||
seen: map[ReadStatKey]time.Time{},
|
||||
lastFlush: time.Now(),
|
||||
}
|
||||
stats, err := repo.Load()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, st := range stats {
|
||||
l.byKey[st.key()] = st
|
||||
}
|
||||
// Fold anything past the retention horizon right away. A failed persist
|
||||
// is not a boot failure — the fold stays dirty and later flushes retry.
|
||||
l.mu.Lock()
|
||||
l.compactLocked()
|
||||
if err := l.persistLocked(); err != nil {
|
||||
log.Printf("beardrive: read telemetry compact failed (will retry): %v", err)
|
||||
}
|
||||
l.mu.Unlock()
|
||||
return l, nil
|
||||
}
|
||||
|
||||
// OpenReadLedger loads the file-backed ledger at path.
|
||||
func OpenReadLedger(path string, retentionDays int) (*ReadLedger, error) {
|
||||
return NewReadLedger(newFileReadRepo(path), retentionDays)
|
||||
}
|
||||
|
||||
// Record counts one read. Nil-safe and never fails: telemetry must not break
|
||||
// the page view (or sync cycle) that triggered it.
|
||||
func (l *ReadLedger) Record(project, path, kind, actor string) {
|
||||
if l == nil || project == "" || path == "" {
|
||||
return
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
visit := ReadStatKey{Project: project, Path: path, Kind: kind, Actor: actor}
|
||||
if t, ok := l.seen[visit]; ok && now.Sub(t) < readDebounce {
|
||||
return
|
||||
}
|
||||
l.seen[visit] = now
|
||||
key := visit
|
||||
key.Day = now.Format("2006-01-02")
|
||||
st := l.byKey[key]
|
||||
st.Project, st.Path, st.Day, st.Kind, st.Actor = project, path, key.Day, kind, actor
|
||||
st.Count++
|
||||
st.Last = now
|
||||
l.byKey[key] = st
|
||||
l.dirty[key] = true
|
||||
if now.Sub(l.lastFlush) >= readFlushEvery {
|
||||
l.flushLocked()
|
||||
}
|
||||
}
|
||||
|
||||
// Heat aggregates reads per path for one project. since bounds the window
|
||||
// (zero = all time, including retention folds); prefix "" means the whole
|
||||
// project, otherwise paths under "<prefix>/".
|
||||
func (l *ReadLedger) Heat(project, prefix string, since time.Time) map[string]HeatEntry {
|
||||
if l == nil {
|
||||
return nil
|
||||
}
|
||||
sinceDay := ""
|
||||
if !since.IsZero() {
|
||||
sinceDay = since.UTC().Format("2006-01-02")
|
||||
}
|
||||
prefix = strings.TrimSuffix(prefix, "/")
|
||||
out := map[string]HeatEntry{}
|
||||
humans := map[string]map[string]bool{} // path → distinct human actors
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
for key, st := range l.byKey {
|
||||
if key.Project != project {
|
||||
continue
|
||||
}
|
||||
if prefix != "" && !strings.HasPrefix(key.Path, prefix+"/") {
|
||||
continue
|
||||
}
|
||||
if key.Day == "" {
|
||||
if sinceDay != "" {
|
||||
continue // all-time fold is older than any windowed query
|
||||
}
|
||||
} else if key.Day < sinceDay {
|
||||
continue
|
||||
}
|
||||
e := out[key.Path]
|
||||
switch key.Kind {
|
||||
case ReadKindHuman:
|
||||
e.Human += st.Count
|
||||
set := humans[key.Path]
|
||||
if set == nil {
|
||||
set = map[string]bool{}
|
||||
humans[key.Path] = set
|
||||
}
|
||||
set[key.Actor] = true
|
||||
case ReadKindAgent:
|
||||
e.Agent += st.Count
|
||||
case ReadKindShare:
|
||||
e.Share += st.Count
|
||||
}
|
||||
if st.Last.After(e.LastRead) {
|
||||
e.LastRead = st.Last
|
||||
}
|
||||
out[key.Path] = e
|
||||
}
|
||||
for p, set := range humans {
|
||||
e := out[p]
|
||||
e.Readers = len(set)
|
||||
out[p] = e
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Close flushes any pending buckets.
|
||||
func (l *ReadLedger) Close() error {
|
||||
if l == nil {
|
||||
return nil
|
||||
}
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.flushLocked()
|
||||
if n := len(l.dirty); n > 0 {
|
||||
return fmt.Errorf("read ledger: flush failed, %d buckets pending", n)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// flushLocked persists dirty buckets (and, once a day, retention folds),
|
||||
// pruning the debounce map along the way. Failures keep the buckets dirty for
|
||||
// the next attempt and log once — telemetry never breaks a request.
|
||||
func (l *ReadLedger) flushLocked() {
|
||||
now := time.Now()
|
||||
l.lastFlush = now
|
||||
for k, t := range l.seen {
|
||||
if now.Sub(t) >= readDebounce {
|
||||
delete(l.seen, k)
|
||||
}
|
||||
}
|
||||
l.compactLocked()
|
||||
if err := l.persistLocked(); err != nil {
|
||||
if !l.warned {
|
||||
l.warned = true
|
||||
log.Printf("beardrive: read telemetry flush failed (will retry): %v", err)
|
||||
}
|
||||
} else {
|
||||
l.warned = false
|
||||
}
|
||||
}
|
||||
|
||||
// compactLocked folds daily buckets older than the retention horizon into
|
||||
// their all-time rows, queueing the daily rows for deletion. Callers hold mu.
|
||||
func (l *ReadLedger) compactLocked() {
|
||||
horizon := time.Now().UTC().Add(-l.retention).Format("2006-01-02")
|
||||
for key, st := range l.byKey {
|
||||
if key.Day == "" || key.Day >= horizon {
|
||||
continue
|
||||
}
|
||||
fold := key
|
||||
fold.Day = ""
|
||||
agg := l.byKey[fold]
|
||||
agg.Project, agg.Path, agg.Kind, agg.Actor = st.Project, st.Path, st.Kind, st.Actor
|
||||
agg.Day = ""
|
||||
agg.Count += st.Count
|
||||
if st.Last.After(agg.Last) {
|
||||
agg.Last = st.Last
|
||||
}
|
||||
l.byKey[fold] = agg
|
||||
l.dirty[fold] = true
|
||||
delete(l.byKey, key)
|
||||
delete(l.dirty, key)
|
||||
l.pendingDel = append(l.pendingDel, key)
|
||||
}
|
||||
}
|
||||
|
||||
// persistLocked writes queued deletions and dirty buckets through the repo.
|
||||
// Callers hold mu. Both queues survive a failure so the next flush retries —
|
||||
// dropping a deletion would resurrect folded rows on the next load and
|
||||
// double-count them.
|
||||
func (l *ReadLedger) persistLocked() error {
|
||||
if len(l.pendingDel) > 0 {
|
||||
if err := l.repo.DeleteBatch(l.pendingDel); err != nil {
|
||||
return err
|
||||
}
|
||||
l.pendingDel = nil
|
||||
}
|
||||
if len(l.dirty) == 0 {
|
||||
return nil
|
||||
}
|
||||
batch := make([]ReadStat, 0, len(l.dirty))
|
||||
for key := range l.dirty {
|
||||
batch = append(batch, l.byKey[key])
|
||||
}
|
||||
if err := l.repo.PutBatch(batch); err != nil {
|
||||
return err
|
||||
}
|
||||
l.dirty = map[ReadStatKey]bool{}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ---- server integration ----
|
||||
|
||||
// ctxProjectKey carries the resolved project id from the proj() route
|
||||
// resolver to handlers that record reads.
|
||||
type ctxProjectKey struct{}
|
||||
|
||||
func withProjectID(r *http.Request, id string) *http.Request {
|
||||
return r.WithContext(context.WithValue(r.Context(), ctxProjectKey{}, id))
|
||||
}
|
||||
|
||||
func projectID(r *http.Request) string {
|
||||
id, _ := r.Context().Value(ctxProjectKey{}).(string)
|
||||
return id
|
||||
}
|
||||
|
||||
// recordRead counts a human read of path for the request's project. No-op
|
||||
// outside hub mode (no project id) or when read tracking is off.
|
||||
func (s *Server) recordRead(r *http.Request, path string) {
|
||||
if s.Reads == nil {
|
||||
return
|
||||
}
|
||||
project := projectID(r)
|
||||
if project == "" {
|
||||
return
|
||||
}
|
||||
actor := s.requestUser(r).Email
|
||||
if actor == "" {
|
||||
actor = "anonymous"
|
||||
}
|
||||
s.Reads.Record(project, path, ReadKindHuman, actor)
|
||||
}
|
||||
|
||||
// handleHeat serves per-path read aggregates: ?prefix= bounds to a folder,
|
||||
// ?days= bounds the window (default 30, 0 = all time). Counts only — actor
|
||||
// identities never leave the server.
|
||||
func (s *Server) handleHeat(v *volume, w http.ResponseWriter, r *http.Request) {
|
||||
if s.Reads == nil {
|
||||
http.Error(w, "read tracking is not enabled on this server", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
_ = v
|
||||
q := r.URL.Query()
|
||||
days := 30
|
||||
if raw := q.Get("days"); raw != "" {
|
||||
var err error
|
||||
if days, err = strconv.Atoi(raw); err != nil || days < 0 {
|
||||
http.Error(w, "invalid days", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
}
|
||||
var since time.Time
|
||||
if days > 0 {
|
||||
since = time.Now().UTC().AddDate(0, 0, -days)
|
||||
}
|
||||
entries := s.Reads.Heat(projectID(r), q.Get("prefix"), since)
|
||||
out := map[string]any{"entries": entries}
|
||||
if !since.IsZero() {
|
||||
out["since"] = since.Format("2006-01-02")
|
||||
}
|
||||
writeJSON(w, out)
|
||||
}
|
||||
|
||||
// handleReadReport ingests agent reads from a syncing device: the client's
|
||||
// read spool, drained best-effort at sync time. Requires a device identity —
|
||||
// the device id is the actor, so reads count as agent traffic.
|
||||
func (s *Server) handleReadReport(v *volume, w http.ResponseWriter, r *http.Request) {
|
||||
if s.Reads == nil {
|
||||
http.Error(w, "read tracking is not enabled on this server", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
_ = v
|
||||
device := r.Header.Get("X-Bdrive-Device")
|
||||
if device == "" {
|
||||
http.Error(w, "agent read reports need a device identity", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Reads []struct {
|
||||
Path string `json:"path"`
|
||||
// Time is accepted for forward compatibility but buckets use
|
||||
// server time: client clocks are unreliable and late flushes are
|
||||
// telemetry noise, not data loss.
|
||||
Time time.Time `json:"time,omitzero"`
|
||||
} `json:"reads"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&req); err != nil {
|
||||
http.Error(w, "bad request: "+err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if len(req.Reads) > 4096 {
|
||||
http.Error(w, "too many reads in one report", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
s.observeDevice(r)
|
||||
project := projectID(r)
|
||||
n := 0
|
||||
for _, e := range req.Reads {
|
||||
if e.Path == "" || strings.Contains(e.Path, "..") {
|
||||
continue
|
||||
}
|
||||
s.Reads.Record(project, e.Path, ReadKindAgent, device)
|
||||
n++
|
||||
}
|
||||
writeJSON(w, map[string]any{"accepted": n})
|
||||
}
|
||||
@@ -0,0 +1,236 @@
|
||||
package webapp
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// doHdr is do() with request headers (device identity on read reports).
|
||||
func doHdr(t *testing.T, h http.Handler, method, url string, body any, hdr map[string]string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
data, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
req := httptest.NewRequest(method, url, bytes.NewReader(data))
|
||||
for k, v := range hdr {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
func openTestLedger(t *testing.T, retentionDays int) (*ReadLedger, *fileReadRepo) {
|
||||
t.Helper()
|
||||
repo := newFileReadRepo(filepath.Join(t.TempDir(), "reads.json"))
|
||||
l, err := NewReadLedger(repo, retentionDays)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return l, repo
|
||||
}
|
||||
|
||||
func TestReadLedgerDebounce(t *testing.T) {
|
||||
l, _ := openTestLedger(t, 0)
|
||||
// A reload storm and the render-then-raw double fetch are one visit…
|
||||
l.Record("p-1", "a.md", ReadKindHuman, "alice@x.io")
|
||||
l.Record("p-1", "a.md", ReadKindHuman, "alice@x.io")
|
||||
l.Record("p-1", "a.md", ReadKindHuman, "alice@x.io")
|
||||
// …but a different actor, kind, or path counts on its own.
|
||||
l.Record("p-1", "a.md", ReadKindHuman, "bob@x.io")
|
||||
l.Record("p-1", "a.md", ReadKindAgent, "alice@x.io")
|
||||
l.Record("p-1", "b.md", ReadKindHuman, "alice@x.io")
|
||||
heat := l.Heat("p-1", "", time.Time{})
|
||||
if e := heat["a.md"]; e.Human != 2 || e.Agent != 1 || e.Readers != 2 {
|
||||
t.Fatalf("a.md = %+v, want human 2, agent 1, readers 2", e)
|
||||
}
|
||||
if e := heat["b.md"]; e.Human != 1 || e.Readers != 1 {
|
||||
t.Fatalf("b.md = %+v", e)
|
||||
}
|
||||
if heat["a.md"].LastRead.IsZero() {
|
||||
t.Fatal("last_read not set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadLedgerWindow(t *testing.T) {
|
||||
l, repo := openTestLedger(t, 0)
|
||||
l.Record("p-1", "a.md", ReadKindHuman, "alice@x.io")
|
||||
if err := l.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// An old bucket inside retention: counted all-time, outside a 7-day window.
|
||||
old := ReadStat{Project: "p-1", Path: "a.md", Day: "2026-01-01", Kind: ReadKindHuman,
|
||||
Actor: "carol@x.io", Count: 5, Last: time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)}
|
||||
if err := repo.PutBatch([]ReadStat{old}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l2, err := NewReadLedger(repo, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if e := l2.Heat("p-1", "", time.Time{})["a.md"]; e.Human != 6 || e.Readers != 2 {
|
||||
t.Fatalf("all-time = %+v, want human 6, readers 2", e)
|
||||
}
|
||||
week := time.Now().UTC().AddDate(0, 0, -7)
|
||||
if e := l2.Heat("p-1", "", week)["a.md"]; e.Human != 1 || e.Readers != 1 {
|
||||
t.Fatalf("windowed = %+v, want only today's read", e)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadLedgerRetentionFold(t *testing.T) {
|
||||
repo := newFileReadRepo(filepath.Join(t.TempDir(), "reads.json"))
|
||||
seed := []ReadStat{
|
||||
{Project: "p-1", Path: "a.md", Day: "2020-01-01", Kind: ReadKindHuman, Actor: "alice@x.io", Count: 3,
|
||||
Last: time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)},
|
||||
{Project: "p-1", Path: "a.md", Day: "2020-01-02", Kind: ReadKindHuman, Actor: "alice@x.io", Count: 2,
|
||||
Last: time.Date(2020, 1, 2, 0, 0, 0, 0, time.UTC)},
|
||||
{Project: "p-1", Path: "a.md", Day: time.Now().UTC().Format("2006-01-02"), Kind: ReadKindHuman,
|
||||
Actor: "alice@x.io", Count: 1, Last: time.Now().UTC()},
|
||||
}
|
||||
if err := repo.PutBatch(seed); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l, err := NewReadLedger(repo, 30)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// All-time totals survive the fold; per-day resolution ages out.
|
||||
if e := l.Heat("p-1", "", time.Time{})["a.md"]; e.Human != 6 || e.Readers != 1 {
|
||||
t.Fatalf("after fold = %+v, want human 6, readers 1", e)
|
||||
}
|
||||
if err := l.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rows, err := repo.Load()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var folds, dailies int
|
||||
for _, st := range rows {
|
||||
if st.Day == "" {
|
||||
folds++
|
||||
if st.Count != 5 {
|
||||
t.Fatalf("fold count = %d, want 5", st.Count)
|
||||
}
|
||||
} else {
|
||||
dailies++
|
||||
}
|
||||
}
|
||||
if folds != 1 || dailies != 1 {
|
||||
t.Fatalf("rows after fold: %d folds, %d dailies; want 1 and 1 (%+v)", folds, dailies, rows)
|
||||
}
|
||||
// Reloading must not double-count: the fold replaced the old rows.
|
||||
l2, err := NewReadLedger(repo, 30)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if e := l2.Heat("p-1", "", time.Time{})["a.md"]; e.Human != 6 {
|
||||
t.Fatalf("after reload = %+v, want human still 6", e)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadLedgerNil(t *testing.T) {
|
||||
var l *ReadLedger
|
||||
l.Record("p-1", "a.md", ReadKindHuman, "x") // must not panic
|
||||
if l.Heat("p-1", "", time.Time{}) != nil {
|
||||
t.Fatal("nil ledger heat should be nil")
|
||||
}
|
||||
if err := l.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeatAPI drives the recording sites and the heat/report endpoints
|
||||
// through the real handler: renders and downloads count (debounced), the
|
||||
// store sync proxy and history blob views never count, and device-reported
|
||||
// agent reads land as agent traffic.
|
||||
func TestHeatAPI(t *testing.T) {
|
||||
srv, p, root := newHub(t, false, nil)
|
||||
f := newFakeRemoteAt(t, filepath.Join(root, p.ID))
|
||||
f.putAs("dev1", "alice@x.io", "Alice", "wiki/plan.md", "# plan")
|
||||
f.putAs("dev1", "alice@x.io", "Alice", "top.md", "top")
|
||||
var err error
|
||||
srv.Reads, err = OpenReadLedger(filepath.Join(t.TempDir(), "reads.json"), 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h := srv.Handler()
|
||||
base := "/api/p/" + p.ID + "/"
|
||||
|
||||
// render + raw fetch of the same file = one visit (debounced)
|
||||
for _, u := range []string{base + "render?path=wiki/plan.md", base + "file?path=wiki/plan.md"} {
|
||||
if rec := do(t, h, "GET", u, nil); rec.Code != 200 {
|
||||
t.Fatalf("GET %s: %d %s", u, rec.Code, rec.Body)
|
||||
}
|
||||
}
|
||||
// store proxy traffic is replication, not reading
|
||||
do(t, h, "GET", base+"store/list?prefix=journal/", nil)
|
||||
do(t, h, "GET", base+"store/object?key=journal/dev1.jsonl", nil)
|
||||
|
||||
rec := do(t, h, "GET", base+"heat", nil)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("heat: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
var out struct {
|
||||
Entries map[string]HeatEntry `json:"entries"`
|
||||
Since string `json:"since"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if e := out.Entries["wiki/plan.md"]; e.Human != 1 || e.Readers != 1 {
|
||||
t.Fatalf("plan.md heat = %+v, want one human visit", e)
|
||||
}
|
||||
if len(out.Entries) != 1 || out.Since == "" {
|
||||
t.Fatalf("heat = %+v; store traffic must not count", out)
|
||||
}
|
||||
|
||||
// an agent device reports its local tool reads
|
||||
report := map[string]any{
|
||||
"reads": []map[string]string{{"path": "wiki/plan.md"}, {"path": "top.md"}, {"path": "../evil"}},
|
||||
}
|
||||
rec = doHdr(t, h, "POST", base+"reads", report, map[string]string{"X-Bdrive-Device": "dev1"})
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("report: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
var acc struct {
|
||||
Accepted int `json:"accepted"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &acc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if acc.Accepted != 2 {
|
||||
t.Fatalf("accepted = %d, want 2 (traversal path dropped)", acc.Accepted)
|
||||
}
|
||||
// …and without a device identity the report is rejected
|
||||
if rec := doHdr(t, h, "POST", base+"reads", report, nil); rec.Code != 400 {
|
||||
t.Fatalf("device-less report: %d, want 400", rec.Code)
|
||||
}
|
||||
|
||||
rec = do(t, h, "GET", base+"heat?prefix=wiki", nil)
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if e := out.Entries["wiki/plan.md"]; e.Human != 1 || e.Agent != 1 {
|
||||
t.Fatalf("plan.md after report = %+v, want human 1 + agent 1", e)
|
||||
}
|
||||
if _, ok := out.Entries["top.md"]; ok {
|
||||
t.Fatal("prefix filter leaked top.md")
|
||||
}
|
||||
|
||||
if rec := do(t, h, "GET", base+"heat?days=x", nil); rec.Code != 400 {
|
||||
t.Fatalf("bad days: %d, want 400", rec.Code)
|
||||
}
|
||||
|
||||
// a hub without read tracking 404s cleanly
|
||||
srv.Reads = nil
|
||||
if rec := do(t, h, "GET", base+"heat", nil); rec.Code != 404 {
|
||||
t.Fatalf("disabled heat: %d, want 404", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -71,6 +71,9 @@ type Server struct {
|
||||
Devices *DeviceRegistry
|
||||
// Shares, when set, enables public share links (/s/<token>).
|
||||
Shares *ShareDB
|
||||
// Reads, when set, aggregates read telemetry (viewer, share, and agent
|
||||
// reads) for the heat API. Nil means read tracking is off.
|
||||
Reads *ReadLedger
|
||||
// Orgs, when set, walls projects off by organization membership.
|
||||
Orgs *OrgDB
|
||||
// Quota, when set, enforces plan limits (managed deployments). Nil
|
||||
@@ -305,7 +308,9 @@ func (s *Server) Handler() http.Handler {
|
||||
http.Error(w, "you are not a member of this project's organization", http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
h(v, w, r)
|
||||
// Read recording (and anything else downstream) finds the project
|
||||
// id in the context; membership has already passed at this point.
|
||||
h(v, w, withProjectID(r, id))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -348,6 +353,8 @@ func (s *Server) Handler() http.Handler {
|
||||
|
||||
mux.HandleFunc("GET /api/p/{project}/history", proj(s.handleHistory))
|
||||
mux.HandleFunc("GET /api/p/{project}/blob", proj(s.handleBlob))
|
||||
mux.HandleFunc("GET /api/p/{project}/heat", proj(s.handleHeat))
|
||||
mux.HandleFunc("POST /api/p/{project}/reads", proj(s.handleReadReport))
|
||||
mux.HandleFunc("POST /api/p/{project}/shares", proj(s.handleShareCreate))
|
||||
mux.HandleFunc("GET /api/p/{project}/shares", proj(s.handleShareList))
|
||||
mux.HandleFunc("DELETE /api/shares/{token}", s.handleShareRevoke)
|
||||
@@ -429,7 +436,8 @@ func (s *Server) handleConfig(w http.ResponseWriter, r *http.Request) {
|
||||
"upload": map[string]any{
|
||||
"enabled": s.Upload.Enabled,
|
||||
},
|
||||
"auth": auth,
|
||||
"auth": auth,
|
||||
"reads": map[string]any{"enabled": s.Reads != nil},
|
||||
}
|
||||
if me.Email != "" {
|
||||
out["me"] = map[string]string{"email": me.Email, "name": me.Name}
|
||||
@@ -607,12 +615,15 @@ func lookup(v *volume, r *http.Request) (string, FileInfo, int, error) {
|
||||
return p, fi, 0, nil
|
||||
}
|
||||
|
||||
func serveBlob(v *volume, w http.ResponseWriter, r *http.Request, attach bool) {
|
||||
func (s *Server) serveBlob(v *volume, w http.ResponseWriter, r *http.Request, attach bool) {
|
||||
p, fi, code, err := lookup(v, r)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), code)
|
||||
return
|
||||
}
|
||||
// Count the read before the ETag check: a 304 render is still a person
|
||||
// reading the file, and skipping it would undercount the hottest pages.
|
||||
s.recordRead(r, p)
|
||||
etag := `"` + fi.Blob + `"`
|
||||
if r.Header.Get("If-None-Match") == etag {
|
||||
w.WriteHeader(http.StatusNotModified)
|
||||
@@ -634,11 +645,11 @@ func serveBlob(v *volume, w http.ResponseWriter, r *http.Request, attach bool) {
|
||||
}
|
||||
|
||||
func (s *Server) handleFile(v *volume, w http.ResponseWriter, r *http.Request) {
|
||||
serveBlob(v, w, r, false)
|
||||
s.serveBlob(v, w, r, false)
|
||||
}
|
||||
|
||||
func (s *Server) handleDownload(v *volume, w http.ResponseWriter, r *http.Request) {
|
||||
serveBlob(v, w, r, true)
|
||||
s.serveBlob(v, w, r, true)
|
||||
}
|
||||
|
||||
func (s *Server) handleRender(v *volume, w http.ResponseWriter, r *http.Request) {
|
||||
@@ -647,6 +658,7 @@ func (s *Server) handleRender(v *volume, w http.ResponseWriter, r *http.Request)
|
||||
http.Error(w, err.Error(), code)
|
||||
return
|
||||
}
|
||||
s.recordRead(r, p)
|
||||
rc, err := v.source.Open(r.Context(), p, fi)
|
||||
if err != nil {
|
||||
http.Error(w, fmt.Sprintf("fetch content: %v", err), http.StatusBadGateway)
|
||||
|
||||
@@ -243,6 +243,9 @@ func (s *Server) handleShared(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "the shared file no longer exists", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
// A share hit is external consumption. Actor is token+IP: one audience
|
||||
// member reloading is debounced to a visit, distinct visitors still count.
|
||||
s.Reads.Record(sh.Project, sh.Path, ReadKindShare, sh.Token+"/"+clientIP(r))
|
||||
|
||||
// Sandbox everything under /s/: shared content executes in an opaque
|
||||
// origin (scripts allowed — charts in reports — but no cookies, no
|
||||
|
||||
@@ -21,6 +21,8 @@ let currentProject = null; // hub mode: the selected project
|
||||
let apiBase = "/api/"; // volume-scoped endpoint prefix
|
||||
let orgs = []; // hub mode: the orgs this account belongs to
|
||||
let joinedOrgId = null; // org just joined via an invite this page-load
|
||||
let heatMap = null; // hub: path → 30-day read counts, from the heat API
|
||||
let heatAt = 0; // when heatMap was last fetched (ms)
|
||||
|
||||
const fileURL = (p) => apiBase + "file?path=" + encodeURIComponent(p);
|
||||
|
||||
@@ -148,6 +150,8 @@ function selectProject(p, path) {
|
||||
currentProject = p;
|
||||
expanded = new Set(); // fresh collapse state for the new project's tree
|
||||
treeFirstLoad = true;
|
||||
heatMap = null; // the old project's heat means nothing here
|
||||
heatAt = 0;
|
||||
apiBase = "/api/p/" + p.id + "/";
|
||||
$("vault-name").textContent = p.name;
|
||||
document.title = p.name + " — BearDrive";
|
||||
@@ -694,6 +698,60 @@ async function refreshTree() {
|
||||
if (currentPath && dirIndex.has(currentPath) && $("content").querySelector(".dirlist")) {
|
||||
renderFolderListing(currentPath);
|
||||
}
|
||||
refreshHeat();
|
||||
}
|
||||
|
||||
/* ---- read heat ----
|
||||
30-day read counts per path from the heat API (hub only). Counts only —
|
||||
the server never says who read what. Fetched lazily alongside the tree. */
|
||||
async function refreshHeat(force) {
|
||||
if (!(serverConfig.mode === "hub" && currentProject)) return;
|
||||
if (!(serverConfig.reads && serverConfig.reads.enabled)) return;
|
||||
if (!force && Date.now() - heatAt < 60000) return;
|
||||
heatAt = Date.now(); // set before the fetch so failures don't hammer
|
||||
let out;
|
||||
try {
|
||||
out = await getJSON(apiBase + "heat?days=30");
|
||||
} catch { return; } // keep the last good heat
|
||||
heatMap = out.entries || {};
|
||||
if (currentPath && dirIndex.has(currentPath) && $("content").querySelector(".dirlist")) {
|
||||
renderFolderListing(currentPath);
|
||||
}
|
||||
}
|
||||
|
||||
/* Heat for one listing entry: a file's own bucket, or the subtree sum for a
|
||||
folder. Null when there is nothing to show. */
|
||||
function heatFor(path, isDir) {
|
||||
if (!heatMap) return null;
|
||||
if (!isDir) return heatMap[path] || null;
|
||||
const agg = { human: 0, agent: 0, share: 0 };
|
||||
for (const [p, e] of Object.entries(heatMap)) {
|
||||
if (!p.startsWith(path + "/")) continue;
|
||||
agg.human += e.human || 0;
|
||||
agg.agent += e.agent || 0;
|
||||
agg.share += e.share || 0;
|
||||
}
|
||||
return agg.human || agg.agent || agg.share ? agg : null;
|
||||
}
|
||||
|
||||
function heatTotal(e) { return (e.human || 0) + (e.agent || 0) + (e.share || 0); }
|
||||
|
||||
function heatText(e) {
|
||||
const total = heatTotal(e);
|
||||
if (!total) return "";
|
||||
let s = total + (total === 1 ? " read" : " reads");
|
||||
if (e.agent) s += " (" + e.agent + " agent)";
|
||||
return s;
|
||||
}
|
||||
|
||||
/* Dot intensity 1–4, log-ish steps: 1–2, 3–9, 10–29, 30+ reads. */
|
||||
function heatLevel(e) {
|
||||
const total = heatTotal(e);
|
||||
if (!total) return 0;
|
||||
if (total < 3) return 1;
|
||||
if (total < 10) return 2;
|
||||
if (total < 30) return 3;
|
||||
return 4;
|
||||
}
|
||||
|
||||
function renderChildren(children) {
|
||||
@@ -754,7 +812,7 @@ function renderNode(n) {
|
||||
openFolder(n.path);
|
||||
};
|
||||
} else {
|
||||
flatFiles.push({ path: n.path, name: n.name });
|
||||
flatFiles.push({ path: n.path, name: n.name, time: n.time });
|
||||
row.onclick = () => openFile(n.path);
|
||||
}
|
||||
return li;
|
||||
@@ -865,6 +923,8 @@ function renderFolderListing(p) {
|
||||
const counts = [];
|
||||
if (dirs) counts.push(dirs + (dirs === 1 ? " folder" : " folders"));
|
||||
if (files) counts.push(files + (files === 1 ? " file" : " files"));
|
||||
const folderHeat = heatFor(p, true);
|
||||
if (folderHeat) counts.push(heatText(folderHeat) + " in 30 days");
|
||||
el(wrap, "p", "dl-sub", counts.join(" · ") || "Empty folder");
|
||||
if (!kids.length) {
|
||||
el(wrap, "div", "dl-empty", "Nothing in this folder yet.");
|
||||
@@ -888,6 +948,12 @@ function renderFolderListing(p) {
|
||||
meta = [c.size ? humanSize(c.size) : "", c.time ? new Date(c.time).toLocaleDateString() : ""]
|
||||
.filter(Boolean).join(" · ");
|
||||
}
|
||||
const he = heatFor(c.path, !!c.dir);
|
||||
if (he) {
|
||||
const dot = el(row, "span", "heatdot lvl" + heatLevel(he));
|
||||
dot.title = heatText(he) + " in 30 days";
|
||||
meta = heatText(he) + (meta ? " · " + meta : "");
|
||||
}
|
||||
el(row, "span", "dl-meta", meta);
|
||||
row.onclick = () => openPath(c.path);
|
||||
row.onkeydown = (e) => { if (e.key === "Enter" || e.key === " ") { e.preventDefault(); row.click(); } };
|
||||
@@ -964,6 +1030,8 @@ function showMeta(doc) {
|
||||
const parts = [];
|
||||
if (doc.author) parts.push(doc.author + (doc.device ? " on " + doc.device : ""));
|
||||
if (doc.time) parts.push(new Date(doc.time).toLocaleString());
|
||||
const he = heatMap && heatMap[doc.path];
|
||||
if (he && heatTotal(he)) parts.push(heatText(he) + " / 30d");
|
||||
$("meta").textContent = parts.join(" · ");
|
||||
}
|
||||
|
||||
@@ -1067,6 +1135,132 @@ function updateShareButton() {
|
||||
};
|
||||
}
|
||||
|
||||
/* ---- insights: the read×write matrix ----
|
||||
Every file plotted by how much it is read (30 days, from the heat API)
|
||||
against how long since it last changed (from the tree). The hot-but-stale
|
||||
quadrant is the danger zone: knowledge people still rely on that nobody
|
||||
maintains. Admin/org-owner only — members get the ambient heat dots. */
|
||||
|
||||
const HOT_READS = 3; // ≥ this many reads/30d = hot
|
||||
const STALE_DAYS = 30; // ≥ this many days since last write = stale
|
||||
|
||||
function canSeeInsights() {
|
||||
if (!(serverConfig.mode === "hub" && currentProject)) return false;
|
||||
if (!(serverConfig.reads && serverConfig.reads.enabled)) return false;
|
||||
if (serverConfig.auth && serverConfig.auth.admin) return true;
|
||||
const org = currentOrg();
|
||||
return !!(org && org.role === "owner");
|
||||
}
|
||||
|
||||
async function showInsights() {
|
||||
if (!canSeeInsights()) return;
|
||||
await refreshHeat(true);
|
||||
currentPath = null;
|
||||
markActive();
|
||||
$("crumb").textContent = "Insights — " + currentProject.name;
|
||||
$("meta").textContent = "";
|
||||
$("download").hidden = true;
|
||||
$("more-btn").hidden = true;
|
||||
const content = $("content");
|
||||
content.className = "view";
|
||||
renderInsights(content, "all");
|
||||
}
|
||||
|
||||
function renderInsights(content, lens) {
|
||||
content.innerHTML = "";
|
||||
const wrap = el(content, "div", "insights");
|
||||
el(wrap, "h1", "in-title", "Reads × freshness");
|
||||
el(wrap, "p", "dl-sub",
|
||||
"Every file by 30-day reads and days since its last change. " +
|
||||
"Hot but stale (top right) is the danger zone — read a lot, maintained by nobody.");
|
||||
const bar = el(wrap, "div", "in-lens");
|
||||
for (const l of ["all", "human", "agent"]) {
|
||||
const label = l === "all" ? "All reads" : l === "human" ? "Human reads" : "Agent reads";
|
||||
const b = el(bar, "button", "in-lens-btn" + (l === lens ? " active" : ""), label);
|
||||
b.onclick = () => renderInsights(content, lens = l);
|
||||
}
|
||||
|
||||
const readsOf = (e) => (lens === "all" ? heatTotal(e) : e[lens] || 0);
|
||||
const now = Date.now();
|
||||
const pts = flatFiles.map((f) => {
|
||||
const e = (heatMap && heatMap[f.path]) || {};
|
||||
const days = f.time ? Math.max(0, (now - new Date(f.time).getTime()) / 86400000) : 0;
|
||||
const reads = readsOf(e);
|
||||
return { path: f.path, reads, days, danger: reads >= HOT_READS && days >= STALE_DAYS };
|
||||
});
|
||||
|
||||
wrap.appendChild(insightsChart(pts));
|
||||
|
||||
const danger = pts.filter((p) => p.danger)
|
||||
.sort((a, b) => b.reads - a.reads || b.days - a.days).slice(0, 15);
|
||||
el(wrap, "h3", "dl-h3", "Danger zone — fix these first");
|
||||
if (!danger.length) {
|
||||
el(wrap, "div", "dl-empty", "No hot-but-stale files. The knowledge base is healthy.");
|
||||
return;
|
||||
}
|
||||
const list = el(wrap, "div", "dl-items");
|
||||
for (const p of danger) {
|
||||
const row = el(list, "div", "dl-row");
|
||||
row.tabIndex = 0;
|
||||
row.setAttribute("role", "button");
|
||||
const icon = el(row, "span", "ticon");
|
||||
icon.innerHTML = svgIcon("alert");
|
||||
el(row, "span", "dl-name", p.path);
|
||||
el(row, "span", "dl-meta",
|
||||
p.reads + (p.reads === 1 ? " read" : " reads") + " · untouched " + Math.round(p.days) + "d");
|
||||
row.onclick = () => openFile(p.path);
|
||||
row.onkeydown = (e) => { if (e.key === "Enter" || e.key === " ") { e.preventDefault(); row.click(); } };
|
||||
}
|
||||
}
|
||||
|
||||
/* Dependency-free SVG scatter: x = days since last write, y = reads, both
|
||||
log-scaled; threshold lines split the quadrants. */
|
||||
function insightsChart(pts) {
|
||||
const W = 720, H = 360, M = { l: 44, r: 16, t: 20, b: 34 };
|
||||
const maxDays = Math.max(STALE_DAYS * 2, ...pts.map((p) => p.days));
|
||||
const maxReads = Math.max(HOT_READS * 2, ...pts.map((p) => p.reads));
|
||||
const lx = (d) => Math.log10(d + 1) / Math.log10(maxDays + 1);
|
||||
const ly = (r) => Math.log10(r + 1) / Math.log10(maxReads + 1);
|
||||
const X = (d) => M.l + lx(d) * (W - M.l - M.r);
|
||||
const Y = (r) => H - M.b - ly(r) * (H - M.t - M.b);
|
||||
|
||||
const svg = document.createElementNS("http://www.w3.org/2000/svg", "svg");
|
||||
svg.setAttribute("viewBox", `0 0 ${W} ${H}`);
|
||||
svg.setAttribute("class", "in-chart");
|
||||
const add = (tag, attrs, text) => {
|
||||
const n = document.createElementNS("http://www.w3.org/2000/svg", tag);
|
||||
for (const [k, v] of Object.entries(attrs)) n.setAttribute(k, v);
|
||||
if (text != null) n.textContent = text;
|
||||
svg.appendChild(n);
|
||||
return n;
|
||||
};
|
||||
|
||||
// danger quadrant shading + threshold lines
|
||||
add("rect", { x: X(STALE_DAYS), y: M.t, width: W - M.r - X(STALE_DAYS), height: Y(HOT_READS) - M.t, class: "in-danger-zone" });
|
||||
add("line", { x1: X(STALE_DAYS), y1: M.t, x2: X(STALE_DAYS), y2: H - M.b, class: "in-threshold" });
|
||||
add("line", { x1: M.l, y1: Y(HOT_READS), x2: W - M.r, y2: Y(HOT_READS), class: "in-threshold" });
|
||||
// axes
|
||||
add("line", { x1: M.l, y1: H - M.b, x2: W - M.r, y2: H - M.b, class: "in-axis" });
|
||||
add("line", { x1: M.l, y1: M.t, x2: M.l, y2: H - M.b, class: "in-axis" });
|
||||
add("text", { x: (M.l + W - M.r) / 2, y: H - 8, class: "in-label" }, "days since last change →");
|
||||
add("text", { x: 12, y: (M.t + H - M.b) / 2, class: "in-label", transform: `rotate(-90 12 ${(M.t + H - M.b) / 2})` }, "reads / 30d →");
|
||||
add("text", { x: W - M.r - 6, y: M.t + 14, class: "in-quad in-quad-danger", "text-anchor": "end" }, "hot + stale");
|
||||
add("text", { x: M.l + 6, y: M.t + 14, class: "in-quad" }, "hot + fresh");
|
||||
add("text", { x: W - M.r - 6, y: H - M.b - 8, class: "in-quad", "text-anchor": "end" }, "cold + stale");
|
||||
|
||||
for (const p of pts) {
|
||||
const c = add("circle", {
|
||||
cx: X(p.days).toFixed(1), cy: Y(p.reads).toFixed(1), r: 5,
|
||||
class: "in-pt" + (p.danger ? " danger" : p.reads ? "" : " cold"),
|
||||
});
|
||||
const tip = document.createElementNS("http://www.w3.org/2000/svg", "title");
|
||||
tip.textContent = `${p.path} — ${p.reads} read${p.reads === 1 ? "" : "s"} / 30d · changed ${Math.round(p.days)}d ago`;
|
||||
c.appendChild(tip);
|
||||
c.onclick = () => openFile(p.path);
|
||||
}
|
||||
return svg;
|
||||
}
|
||||
|
||||
/* ---- history ----
|
||||
Every change ever made, straight from the journals: who (account), when,
|
||||
from which device (name, OS, IP as the server saw it), with view/download
|
||||
@@ -1485,6 +1679,14 @@ function buildMoreMenu() {
|
||||
b.onclick = () => { $("more-menu").hidden = true; el.click(); };
|
||||
menu.appendChild(b);
|
||||
}
|
||||
if (canSeeInsights()) {
|
||||
const b = document.createElement("button");
|
||||
b.className = "more-item";
|
||||
b.textContent = "Insights";
|
||||
b.onclick = () => { $("more-menu").hidden = true; showInsights(); };
|
||||
menu.appendChild(b);
|
||||
return items.length + 1;
|
||||
}
|
||||
return items.length;
|
||||
}
|
||||
$("more-btn").addEventListener("click", (e) => {
|
||||
|
||||
@@ -269,12 +269,37 @@ button, input, a.btn { font-family: inherit; }
|
||||
.dl-row:hover .ticon { color: var(--text-faint); }
|
||||
.dl-name { flex: 1; min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-size: 13.5px; color: var(--text); }
|
||||
.dl-meta { flex: none; font-size: 12px; color: var(--text-faint); font-variant-numeric: tabular-nums; }
|
||||
/* read-heat dot: intensity grows with 30-day reads (heatLevel in app.js) */
|
||||
.heatdot { flex: none; width: 7px; height: 7px; border-radius: 50%; background: var(--accent); }
|
||||
.heatdot.lvl1 { opacity: .3; }
|
||||
.heatdot.lvl2 { opacity: .55; }
|
||||
.heatdot.lvl3 { opacity: .8; }
|
||||
.heatdot.lvl4 { opacity: 1; box-shadow: 0 0 6px rgba(245, 166, 35, .55); }
|
||||
.dl-empty { padding: 24px 14px; color: var(--text-faint); font-size: 13px; border: 1px dashed var(--border); border-radius: var(--r-card); text-align: center; }
|
||||
.dl-h3 { margin: 28px 0 8px; font-size: 10.5px; text-transform: uppercase; letter-spacing: .07em; color: var(--text-ghost); font-weight: 600; }
|
||||
.dl-hlist { border: 1px solid var(--border); border-radius: var(--r-card); background: var(--bg-side); overflow: hidden; max-width: none; }
|
||||
.dl-hlist .hentry:last-child { border-bottom: none; }
|
||||
.dl-more { margin-top: 10px; }
|
||||
|
||||
/* ---- insights (read×write matrix) ---- */
|
||||
.insights { max-width: 760px; margin: 0 auto; }
|
||||
.in-title { font-size: 21px; font-weight: 640; letter-spacing: -.02em; margin: 0 0 4px; color: #f4f6f9; }
|
||||
.in-lens { display: flex; gap: 6px; margin: 0 0 14px; }
|
||||
.in-lens-btn { font: inherit; font-size: 12px; padding: 5px 12px; border-radius: 999px; border: 1px solid var(--border); background: none; color: var(--text-faint); cursor: pointer; }
|
||||
.in-lens-btn:hover { color: var(--text); }
|
||||
.in-lens-btn.active { color: var(--accent); border-color: var(--accent); }
|
||||
.in-chart { width: 100%; height: auto; border: 1px solid var(--border); border-radius: var(--r-card); background: var(--bg-side); margin-bottom: 6px; }
|
||||
.in-axis { stroke: var(--border); stroke-width: 1; }
|
||||
.in-threshold { stroke: var(--border); stroke-width: 1; stroke-dasharray: 4 4; }
|
||||
.in-danger-zone { fill: rgba(242, 109, 109, .05); }
|
||||
.in-label { fill: var(--text-ghost); font-size: 11px; }
|
||||
.in-quad { fill: var(--text-ghost); font-size: 10.5px; text-transform: uppercase; letter-spacing: .06em; }
|
||||
.in-quad-danger { fill: #e07070; }
|
||||
.in-pt { fill: var(--accent); opacity: .75; cursor: pointer; }
|
||||
.in-pt:hover { opacity: 1; }
|
||||
.in-pt.cold { fill: var(--text-ghost); opacity: .35; }
|
||||
.in-pt.danger { fill: #e05d5d; }
|
||||
|
||||
/* ---- history ---- */
|
||||
.history { max-width: 860px; }
|
||||
.hentry { padding: 11px 12px; border-bottom: 1px solid var(--border); }
|
||||
|
||||
@@ -56,10 +56,12 @@ Follow these steps:
|
||||
detects the agent platforms in use (Claude Code, Codex, Gemini CLI,
|
||||
Hermes — by their config dirs in the project or home) and idempotently
|
||||
merges beardrive's sync hooks into each platform's own hook config, so
|
||||
files pull at every turn start, push after edits, and every change is
|
||||
stamped with the agent session that made it. Tell the user which
|
||||
platforms got hooks; if Codex is among them, mention they must run
|
||||
`/hooks` inside Codex once to trust the project's `.codex` layer.
|
||||
files pull at every turn start, push after edits, every change is
|
||||
stamped with the agent session that made it, and agent file reads feed
|
||||
the hub's read heatmap (queued locally by `bdrive read-log`, reported
|
||||
on the next sync). Tell the user which platforms got hooks; if Codex is
|
||||
among them, mention they must run `/hooks` inside Codex once to trust
|
||||
the project's `.codex` layer.
|
||||
|
||||
6. **Verify**: run `bdrive status <folder>` and confirm the daemon is
|
||||
running and pending is 0. Summarize: project name/id, what syncs, and
|
||||
|
||||
@@ -77,7 +77,11 @@ team's latest files), push right after edits (artifacts land on the server
|
||||
seconds after they're created — daemon or no daemon), and stamp every
|
||||
change with the agent session that made it (`bdrive sync --note "<agent>
|
||||
session <id>"` — visible in `bdrive log` and the hub's history views).
|
||||
They are fast no-ops in folders without `.bdrive/`.
|
||||
A third hook on each platform's read tool (`bdrive read-log`) queues which
|
||||
files the agent read, so the hub's read heatmap can show admins what the
|
||||
team's agents actually consume — reads are reported on the next sync,
|
||||
never from the hook itself. They are fast no-ops in folders without
|
||||
`.bdrive/`.
|
||||
|
||||
Tell the user which platforms got hooks (`bdrive hooks` shows the status
|
||||
table). If Codex is among them, mention they must run `/hooks` inside
|
||||
|
||||
@@ -12,6 +12,18 @@
|
||||
]
|
||||
}
|
||||
],
|
||||
"PostToolUse": [
|
||||
{
|
||||
"matcher": "Read",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "\"${CLAUDE_PLUGIN_ROOT}/scripts/beardrive-read.sh\"",
|
||||
"async": true
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"Stop": [
|
||||
{
|
||||
"hooks": [
|
||||
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
# Queue the file reads from a Read tool call for the hub's read heatmap.
|
||||
# `bdrive read-log` parses the hook's stdin JSON itself and only appends to
|
||||
# a local spool (drained on the next sync) — no network, no locking, so this
|
||||
# is safe to run on every Read in every project. Fast no-op outside
|
||||
# beardrive projects.
|
||||
cd "${CLAUDE_PROJECT_DIR:-.}" || exit 0
|
||||
[ -d .bdrive ] || exit 0
|
||||
command -v bdrive >/dev/null 2>&1 || exit 0
|
||||
bdrive read-log . >/dev/null 2>&1 || true
|
||||
@@ -17,7 +17,8 @@ Use this skill whenever the user is working with the `bdrive` CLI: initializing
|
||||
| Run the daemon in the foreground | `bdrive init -f` |
|
||||
| Stop syncing | `bdrive stop [<folder>]` (`--forget` also unregisters) |
|
||||
| One sync cycle now | `bdrive sync [<folder>]` |
|
||||
| Register agent sync hooks (Claude Code, Codex, Gemini CLI, Hermes) | `bdrive hooks install [<folder>]` — auto-detects the platforms in use and merges pull/push/session-note hooks into each one's own hook config, idempotently; bare `bdrive hooks` shows the status table |
|
||||
| Register agent sync hooks (Claude Code, Codex, Gemini CLI, Hermes) | `bdrive hooks install [<folder>]` — auto-detects the platforms in use and merges pull/push/session-note/read-tracking hooks into each one's own hook config, idempotently; bare `bdrive hooks` shows the status table |
|
||||
| Record agent file reads (hook plumbing) | `bdrive read-log [<folder>]` — parses a hook event JSON from stdin and queues in-project reads locally; drained to the hub on the next sync as agent traffic in the read heatmap. Registered automatically by `bdrive hooks install`; rarely run by hand |
|
||||
| Mounts + daemon + pending state | `bdrive status [<folder>]` |
|
||||
| Change history | `bdrive log [<folder>] [-p path] [-n N]` |
|
||||
| This device's identity | `bdrive whoami` |
|
||||
@@ -130,20 +131,35 @@ conflict-copy ops keep their own `conflict copy of <path>` note.
|
||||
`bdrive hooks install [<folder>]` registers turn-boundary sync for every
|
||||
agent platform it detects (by config dir, in the project or home):
|
||||
|
||||
| Platform | Config it writes | Pull / push events |
|
||||
| Platform | Config it writes | Pull / push / read events |
|
||||
|---|---|---|
|
||||
| Claude Code (& Cowork) | `<project>/.claude/settings.json` | `UserPromptSubmit` / `PostToolUse` (Write\|Edit) |
|
||||
| Codex (ChatGPT) | `<project>/.codex/hooks.json` | `UserPromptSubmit` / `PostToolUse` (apply_patch) — user must `/hooks`-trust the layer once |
|
||||
| Gemini CLI | `<project>/.gemini/settings.json` | `BeforeAgent` / `AfterTool` (write_file\|replace) |
|
||||
| Hermes | `~/.hermes/config.yaml` (per-user) | `pre_llm_call` / `post_tool_call` (write_file\|patch) |
|
||||
| Claude Code (& Cowork) | `<project>/.claude/settings.json` | `UserPromptSubmit` / `PostToolUse` (Write\|Edit) / `PostToolUse` (Read) |
|
||||
| Codex (ChatGPT) | `<project>/.codex/hooks.json` | `UserPromptSubmit` / `PostToolUse` (apply_patch) / `PostToolUse` (read_file, best-effort) — user must `/hooks`-trust the layer once |
|
||||
| Gemini CLI | `<project>/.gemini/settings.json` | `BeforeAgent` / `AfterTool` (write_file\|replace) / `AfterTool` (read_file\|read_many_files) |
|
||||
| Hermes | `~/.hermes/config.yaml` (per-user) | `pre_llm_call` / `post_tool_call` (write_file\|patch) / `post_tool_call` (read_file) |
|
||||
|
||||
Every platform pipes hook JSON with a `session_id`, so one hook command
|
||||
serves all four: it syncs the project (fast no-op outside bdrive folders)
|
||||
and stamps changes with `<agent> session <id>`. Merging is idempotent and
|
||||
preserves existing hooks; `--agent claude,codex,gemini,hermes` overrides
|
||||
and stamps changes with `<agent> session <id>`. The read hook runs `bdrive
|
||||
read-log`, which queues the read locally (no network) for the hub's read
|
||||
heatmap. Merging is idempotent and preserves existing hooks — each hook
|
||||
carries its own marker, so configs from before the read hook gain just the
|
||||
missing group on re-install; `--agent claude,codex,gemini,hermes` overrides
|
||||
detection; bare `bdrive hooks` prints the detection/registration table.
|
||||
Project-level configs ride the repo, so hooks reach the whole team.
|
||||
|
||||
### Read heat (who actually reads what)
|
||||
|
||||
Hubs aggregate reads per file — viewer opens and downloads count as human
|
||||
reads, share-link hits as share reads, and hook-reported agent reads as
|
||||
agent reads; `/store` sync replication never counts. The web UI shows heat
|
||||
dots and read counts on folder listings (all members), and admins / org
|
||||
owners get an **Insights** view (⋯ menu) plotting every file by 30-day
|
||||
reads × days since last change — the hot-but-stale quadrant is the list of
|
||||
files to fix first. Counts only, never reader identities. API:
|
||||
`GET /api/p/<id>/heat?prefix=&days=30`. Server config: `"reads":
|
||||
{"enabled": true, "retention_days": 400}` (on by default in hub mode).
|
||||
|
||||
### Examples to walk a user through
|
||||
|
||||
```sh
|
||||
|
||||
Reference in New Issue
Block a user