From 46cfeb7898cee975f055824859bb9b4fad7fd119 Mon Sep 17 00:00:00 2001 From: Snow Lee Date: Fri, 21 Aug 2026 09:24:23 -0700 Subject: [PATCH] feat(cli): bdrive verify proves this folder matches the hub (BEA-161) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Your folder is the same everywhere" was a belief with no receipt. `bdrive status` counts pending ops and unscanned changes but never reads a byte of content, so a file whose bytes changed while its size and mtime stayed put was invisible to every check we shipped. `bdrive verify` hashes every synced file and compares it against journal.Replay(AllOps), reporting drifted / never-pushed / missing-locally / not-yet-scanned, and with --remote also missing-on-hub. Exit 0 when every category is empty, 1 otherwise, so it composes as a pre-flight check. The logic lives in internal/syncer/verify.go next to Drift and Explain — the two read-only siblings it completes — because neverSync, chunkThreshold and loadFilter are unexported there, and because a multi-device test cannot drive a func main package. cmd/bdrive/verify.go is the thin cobra shell. Two things that would have made it silently wrong: - --remote must probe BOTH blobs/ and manifests/. Files over 4 MiB are pushed as chunks plus a manifest keyed by the file's own sha, so a check asking only blobs/ would call every large file missing from the hub. Size only orders the probe — it can never be a filter, because browser uploads always write blobs/ at any size, pushChunked falls back to a whole blob when the manifest key is refused, and pre-delta-sync history is whole blobs regardless. - missing-locally applies filter.Skip + neverSync, the same guard materialize uses. The rules are symmetric in scan and materialize, so a path the local filter excludes is legitimately absent — without this, every project narrowed by `bdrive scope --only` would report its whole out-of-scope set as missing. Pure read throughout: no Session, no volume flock, no ops, no journal writes, no materialize, and no network without --remote. LoadProject rather than ResolveMount and remote.Open rather than openSession, so a read never enrolls the device. An unreachable hub degrades to a printed warning and the local verdict still decides. The command says its own caveat out loud: the journals it replays are this device's local copies, so it proves "this folder matches what I last pulled". Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 1 + architecture/cli-sync.md | 25 +- cmd/bdrive/main.go | 1 + cmd/bdrive/verify.go | 203 ++++++++++++++ internal/syncer/verify.go | 243 +++++++++++++++++ internal/syncer/verify_test.go | 301 +++++++++++++++++++++ internal/webapp/cli_e2e_test.go | 83 ++++++ web/docs/src/content/docs/reference/cli.md | 62 +++++ 8 files changed, 918 insertions(+), 1 deletion(-) create mode 100644 cmd/bdrive/verify.go create mode 100644 internal/syncer/verify.go create mode 100644 internal/syncer/verify_test.go diff --git a/README.md b/README.md index 6b760de..bfb6dce 100644 --- a/README.md +++ b/README.md @@ -253,6 +253,7 @@ hub's own storage, never something a syncing client points at directly: | `bdrive scope [add\|rm ]` | Show or change which subfolders sync — edits the managed block of `.bdriveignore` rules that `init --only` writes, so no one hand-writes negation syntax. The daemon picks changes up in seconds; `rm` deletes nothing, locally or on the hub. `--explain` lists every path in the folder split into what syncs and what does not, so you can verify what leaves this machine (pure read — no daemon, no lock, no network) | | `bdrive grep [folder]` | Search the text **inside** the files a project syncs — Go RE2 regexp, or a literal with `-F`; `-i` ignores case, `-l` prints matching paths only, `-n` caps the lines printed (default 200, `0` = all). Output is `path:line: text`. Only files the project actually syncs are searched, so a `.bdriveignore` rule or a narrowed `bdrive scope` excludes a file from search exactly as it excludes it from sync; binary files are skipped. Pure local read — no daemon, no lock, no network, works offline and never blocks a sync in progress. Exit status 0 on match, 1 on none, so it composes in scripts | | `bdrive stale [folder]` | Find docs the code has outgrown: synced markdown (`.md`/`.markdown`) that links to a file written **after** the doc itself. Staleness here is not age — a doc goes stale when what it describes moves. Write times come from the **journal**, not `os.Stat`: materialize stamps a peer's file with this device's mtime, so on a freshly synced machine every mtime is identical and only the journal still knows. `-l` prints outgrown paths only, `-n` caps the docs printed (default 50, `0` = all). A reference that does not resolve to a file this project syncs — a URL, a `../` escape, a made-up path — is silently ignored. Pure local read — no daemon, no lock, no network. **Exit status is 0 whether or not anything is stale**: this is advisory, not a gate | +| `bdrive verify [folder]` | Prove this folder matches the content the journal records: **sha256 every file the project syncs** and compare. Reports `drifted` (on disk, but not what the journal records), `never-pushed` (committed here, never reached the hub), `missing-locally` (the journal has it, this folder does not), `not-yet-scanned` (on disk, no op yet), and with `--remote` also `missing-on-hub` (one existence check per distinct blob, asking **both** `blobs/` and `manifests/` — files over 4 MiB are pushed as chunks plus a manifest, so a check that asked only `blobs/` would call every large file missing). Unlike `bdrive status` this reads content, so a file whose bytes changed while its size and mtime stayed put is caught; there is deliberately no fast path, and it is slower than `status` on a big project. Pure read — no daemon, no lock, no ops, no journal writes, and without `--remote` no network. It never repairs anything. **Exit status is 0 when every category is empty, 1 otherwise**, so it composes as a pre-flight check. An unreachable hub degrades `--remote` to a warning, never a failure. The journals it replays are this device's local copies, so it proves "this folder matches what I last pulled" | | `bdrive forget ...` | Stop syncing a path *and* remove it from the hub — adds the rule to `.bdriveignore` (which syncs) and prunes in one step. Local files are never touched, here or on teammates' devices | | `bdrive url [path]` | Internal hub link for a file/folder (sign-in + membership required; `--sync` pushes first; no arg = project home). Computed locally | | `bdrive share ` | Public URL for a synced file (`--list`, `--revoke`, `--expires`) | diff --git a/architecture/cli-sync.md b/architecture/cli-sync.md index fbbbef7..e5b05a4 100644 --- a/architecture/cli-sync.md +++ b/architecture/cli-sync.md @@ -115,6 +115,22 @@ classDiagram } note for Drift "drift.go — the `local:` line in bdrive status: what is on disk that the state cache has not seen, using the scan's own size+mtime compare. Pure read like its siblings, and load-bearing that it stays one: status is what someone runs when sync is stuck, so it stores no blob, mints no op, rewrites no cache — it does not even mutate the cache map it is handed, which status prints `files:` from" + class Verify { + +Verify(ctx, folder, include, st, device, be) VerifyReport + -existsEither(ctx, be, blob, size) bool + } + class VerifyReport { + +Files int + +Bytes int64 + +Elapsed Duration + +Drifted / NeverPushed / MissingLocally paths + +NotYetScanned / MissingOnHub paths + +NotFetched int + +RemoteErr error + +Problems() int + } + note for Verify "verify.go — bdrive verify, the third read-only sibling and the only one that HASHES: sha256 every synced file against journal.Replay(st.AllOps()), which is why it catches the file whose bytes changed while size and mtime stayed put — precisely what Drift cannot see. Pure read like its siblings: no Session, no flock, no ops, no journal write, no materialize, and no network unless a Backend is passed. missing-locally applies filter.Skip + neverSync, the same guard materialize uses, so a scope-narrowed project does not report every out-of-scope path as missing. The --remote leg is one Exists per DISTINCT blob probing BOTH blobs/ and manifests/ — size only orders the probe, because a large file legitimately lives under blobs/ (browser upload, pushChunked fallback, pre-delta-sync history) — and any Exists error sets RemoteErr and stops that leg, never the local verdict" + class Explain { +Explain(folder, include, accepted) two lists +NotSyncedFiles(entries) int @@ -191,6 +207,11 @@ classDiagram Session --> Filter : SkipUp on scan, Skip on materialize Session --> walkFolder : scan Explain --> walkFolder : same predicate + Verify --> SyncedFiles : hashes what syncs + Verify --> Filter : own fresh instance, for missing-locally + Verify --> Store : AllOps / DeviceOps / LoadSync / HasBlob + Verify --> Backend : Exists per blob, --remote only + Verify ..> VerifyReport : findings and exit status Drift --> walkFolder : same predicate Drift --> Filter : own fresh instance SyncedFiles --> walkFolder : same predicate @@ -223,13 +244,15 @@ classDiagram class Commands { init login logout - sync stop scope grep stale forget status log + sync stop scope grep stale verify forget status log restore url share export import web daemon hooks read-log resume autostart } note for Commands "cmd/bdrive — thin cobra layer; init is the front door (one command: login + hooks + sync + link), stop pauses" note for Commands "grep searches file CONTENTS in the working folder via syncer.SyncedFiles — LoadProject not ResolveMount (a read must not enroll the device), no session, no flock, and the volume store is opened only if it already exists, so a search creates nothing. Exit 1 on no match is a status, not an error (errNoMatch + SilenceErrors). stale copies that whole posture and swaps the predicate: it extracts path-shaped references from synced markdown, keeps only the ones resolving into the SyncedFiles set, and flags a doc whose reference was written later. It dates a path from the JOURNAL, not os.Stat — materialize stamps a peer's file with this device's mtime, so mtime comparison reports nothing on a freshly cloned machine — folding st.AllOps() to the max syncer.DisplayTime per path, which drops a forged future stamp instead of dating that path to year 1. Unlike grep it exits 0 either way: advisory output, not a gate" + note for Commands "verify is the same pure-read shell again — LoadProject not ResolveMount, stat-guarded store.Open, safeField on every path (missing-locally and missing-on-hub come out of a PEER's journal) — and it opens its Backend with remote.Open directly rather than openSession, because openSession goes through mustProject and would enroll the device. Exit 1 when any category is non-empty is a status, not an error (errVerifyProblems + SilenceErrors), so it composes as a pre-flight check; an unreachable hub prints a warning and the local verdict still decides" + note for Commands "Every peer-authored string status / log / whoami print goes through safeField first — a teammate's file name is attacker-controlled text landing in your terminal, and an escape sequence there rewrites the line above it. grep runs BOTH the path and the matched line through it — a matched line is a teammate's file content, the widest version of that surface. login now does PKCE on the loopback callback (no compat arm) and both its client and init's refuse to follow a redirect off the hub's origin with the device token attached" class Templates { diff --git a/cmd/bdrive/main.go b/cmd/bdrive/main.go index 9ddcd67..d15322f 100644 --- a/cmd/bdrive/main.go +++ b/cmd/bdrive/main.go @@ -55,6 +55,7 @@ everything keeps working offline; changes sync when the remote is reachable.`, scopeCmd(), grepCmd(), staleCmd(), + verifyCmd(), forgetCmd(), syncCmd(), readLogCmd(), diff --git a/cmd/bdrive/verify.go b/cmd/bdrive/verify.go new file mode 100644 index 0000000..29edc06 --- /dev/null +++ b/cmd/bdrive/verify.go @@ -0,0 +1,203 @@ +package main + +import ( + "errors" + "fmt" + "io" + "time" + + "github.com/spf13/cobra" + + "github.com/runbear-io/beardrive/internal/config" + "github.com/runbear-io/beardrive/internal/remote" + "github.com/runbear-io/beardrive/internal/store" + "github.com/runbear-io/beardrive/internal/syncer" +) + +// errVerifyProblems is verify's "something is wrong" exit, not a failure of +// the command: the convention is status 1 with the findings already printed, +// so `bdrive verify || ...` composes in a script or a pre-flight check. Real +// errors (not a project, bad remote) still print — see verifyCmd. +var errVerifyProblems = errors.New("verify found problems") + +// verifyMaxList caps how many paths are listed per category. The counts are +// always exact; only the listing is bounded, so a project that drifted whole +// does not scroll the finding off the screen. +const verifyMaxList = 20 + +func verifyCmd() *cobra.Command { + var checkRemote bool + c := &cobra.Command{ + Use: "verify [folder]", + Short: "Prove this folder matches the content the journal records", + Long: `Hash every file this project syncs and compare it to the journal. + +"Your folder is the same everywhere" is the claim; this is the receipt. +` + "`bdrive status`" + ` counts pending ops and unscanned changes but never reads a +byte of content — so a file whose bytes changed while its size and mtime +stayed put is invisible to it. verify hashes, which is the whole point: there +is deliberately no size/mtime fast path, and on a large project it is +noticeably slower than status. + +It reports five things: + + drifted on disk, but not the content the journal records + never-pushed committed here, never reached the hub + missing-locally the journal has it, this folder does not + not-yet-scanned on disk, with no op anywhere yet + missing-on-hub synced here, absent from the hub (--remote only) + +It is a pure read: no daemon, no lock, no ops, no journal writes, and without +--remote no network at all — so it works offline and never blocks on a sync in +progress. It never repairs anything; run ` + "`bdrive sync`" + ` for that. + +Exit status is 0 when every category is empty and 1 when any is not. + +One caveat it prints for itself: the journals it replays are this device's +local copies, so without a pull it proves "this folder matches what I last +pulled" — a teammate's newer op is invisible until the next cycle.`, + Example: ` bdrive verify # hash the folder, compare to the journal + bdrive verify --remote # also ask the hub whether it still holds the content + bdrive verify ~/wiki # a project other than the current folder`, + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + err := runVerify(cmd, args, checkRemote) + // SilenceErrors below is for errVerifyProblems alone, so anything + // else has to print itself — cobra no longer will. + if err != nil && !errors.Is(err, errVerifyProblems) { + fmt.Fprintln(cmd.ErrOrStderr(), "Error:", err) + } + return err + }, + } + // Silenced so a findings exit is status 1 with the findings and nothing + // else — errVerifyProblems is a status, not a failure, and cobra would + // otherwise render it as an error and a usage block. Same mechanism + // `bdrive grep` uses for its no-match exit. + c.SilenceErrors = true + c.SilenceUsage = true + c.Flags().BoolVar(&checkRemote, "remote", false, "also ask the hub whether it still holds every blob (one check per blob)") + return c +} + +func runVerify(cmd *cobra.Command, folderArg []string, checkRemote bool) error { + folder, err := absFolder(folderArg) + if err != nil { + return err + } + // LoadProject, not ResolveMount: ResolveMount self-heals the registry + // path, i.e. it enrolls this device. A read-only query must not have that + // side effect — the same rule grep and stale follow. + proj, found, err := config.LoadProject(folder) + if err != nil { + return err + } + if !found { + return fmt.Errorf("%s is not a beardrive project (run `bdrive init` there first)", folder) + } + + out := cmd.OutOrStdout() + fmt.Fprintf(out, " project: %s (%s)\n", safeField(proj.Volume, 120), proj.ID) + + // Stat-guarded, because store.Open MkdirAlls and a read must not create a + // volume for a project that has never synced; unlocked, because store.Open + // takes no volume flock — a running daemon never blocks this. + vdir, err := config.VolumeDir(proj.ID) + if err != nil { + return err + } + if !dirExists(vdir) { + fmt.Fprintln(out, " nothing synced yet — no local history to verify against") + return nil + } + st, err := store.Open(vdir) + if err != nil { + return err + } + + var be remote.Backend + if checkRemote { + if proj.Remote == "" { + return errors.New("--remote needs a hub: this project is local only") + } + // remote.Open directly, not openSession: openSession goes through + // mustProject, which is the enrolling call this command must not make. + // It picks up the device token itself, so there is nothing to wire. + b, oerr := remote.Open(cmd.Context(), proj.Remote) + if oerr != nil { + // Unreachable hub is a warning, not a failure: the local verdict + // is still the answer, and this is the repo's "never break on the + // remote" posture. + fmt.Fprintf(out, " warning: hub unreachable, local check only (%s)\n", safeField(oerr.Error(), 200)) + } else { + be = b + defer be.Close() + } + } + + // Best-effort: without an identity the never-pushed count is simply + // empty, which is a worse answer than the others but not a reason to fail + // a read. + dev, _ := config.LoadDevice() + + rep, err := syncer.Verify(cmd.Context(), folder, proj.Include, st, dev.ID, be) + if err != nil { + return err + } + return printVerify(out, rep) +} + +func printVerify(out io.Writer, rep syncer.VerifyReport) error { + fmt.Fprintf(out, " checked: %s, %s hashed in %s\n", + plural(rep.Files, "file"), humanBytes(rep.Bytes), rep.Elapsed.Round(time.Millisecond)) + if rep.RemoteErr != nil { + fmt.Fprintf(out, " warning: hub check incomplete, local result stands (%s)\n", safeField(rep.RemoteErr.Error(), 200)) + } + + missingNote := "the journal has it, this folder does not" + if rep.NotFetched > 0 { + missingNote = fmt.Sprintf("%d not fetched yet — run `bdrive sync`", rep.NotFetched) + } + cats := []struct { + name, note string + paths []string + }{ + {"drifted", "on disk, but not the content the journal records", rep.Drifted}, + {"never-pushed", "committed here, never reached the hub", rep.NeverPushed}, + {"missing-locally", missingNote, rep.MissingLocally}, + {"not-yet-scanned", "on disk, with no op anywhere yet", rep.NotYetScanned}, + {"missing-on-hub", "synced here, absent from the hub", rep.MissingOnHub}, + } + for _, c := range cats { + if len(c.paths) == 0 { + continue + } + fmt.Fprintf(out, "\n %s (%d) - %s\n", c.name, len(c.paths), c.note) + shown := c.paths + if len(shown) > verifyMaxList { + shown = shown[:verifyMaxList] + } + for _, p := range shown { + // Every path in missing-locally and missing-on-hub comes out of a + // PEER's journal — a string someone else chose. safeField, the + // same treatment `bdrive log` and `bdrive grep` give journal + // strings, or a lone CR repaints the row. + fmt.Fprintf(out, " %s\n", safeField(p, 160)) + } + if len(c.paths) > len(shown) { + fmt.Fprintf(out, " ... and %d more\n", len(c.paths)-len(shown)) + } + } + + if n := rep.Problems(); n > 0 { + fmt.Fprintf(out, "\n %s. Run `bdrive sync` to reconcile, or `bdrive log ` for history.\n", plural(n, "problem")) + return errVerifyProblems + } + fmt.Fprintln(out, " OK - the folder matches the journal") + // Said out loud, not just documented: the journals replayed here are this + // device's local copies, so a clean verdict is about what was last pulled. + // Without this line the command overclaims exactly the thing it exists to + // prove. + fmt.Fprintln(out, " (this compares against what this device last pulled — run `bdrive sync` first for the hub's latest)") + return nil +} diff --git a/internal/syncer/verify.go b/internal/syncer/verify.go new file mode 100644 index 0000000..854a6d1 --- /dev/null +++ b/internal/syncer/verify.go @@ -0,0 +1,243 @@ +package syncer + +import ( + "context" + "os" + "path/filepath" + "sort" + "time" + + "github.com/runbear-io/beardrive/internal/journal" + "github.com/runbear-io/beardrive/internal/remote" + "github.com/runbear-io/beardrive/internal/store" +) + +// VerifyReport is what Verify found. Every slice holds mount-relative paths, +// sorted; an empty slice means that category is clean. +// +// A path can legitimately land in two categories at once — a file edited after +// its last scan and never pushed is both drifted and never-pushed — so the +// counts are independent and dedup happens only within a category. +type VerifyReport struct { + Files int // files hashed this run + Bytes int64 // bytes hashed this run + Elapsed time.Duration // how long the hashing took, so the cost is visible + + // Drifted: on disk and in the journal state, but sha256(disk) != state.Blob. + Drifted []string + // NeverPushed: this device's own ops past SyncState.PushedOps — content + // the hub has never seen. + NeverPushed []string + // MissingLocally: a path the journal state holds and the local filter + // includes, that this folder does not have. + MissingLocally []string + // NotFetched is how many of MissingLocally are simply not downloaded yet + // (their blob is absent from the local store). That is the benign half of + // this category — materializeFile returns early on it and the next cycle + // fixes it — and without the distinction a freshly cloned device reads as + // broken. + NotFetched int + // NotYetScanned: a file on disk the filter syncs, with no op anywhere yet. + NotYetScanned []string + // MissingOnHub: content this device believes is synced that the hub does + // not hold. Only populated with a backend. + MissingOnHub []string + // RemoteErr is set when the remote leg could not finish. The local half of + // the report still stands — this is the repo's "never break on the remote" + // posture, the same one Result.Offline takes. + RemoteErr error +} + +// Problems is how many findings the report holds, across every category. It is +// what decides `bdrive verify`'s exit status. +func (r VerifyReport) Problems() int { + return len(r.Drifted) + len(r.NeverPushed) + len(r.MissingLocally) + + len(r.NotYetScanned) + len(r.MissingOnHub) +} + +// Verify compares the bytes in the working folder against the content the +// journal records for them, hashing every synced file. +// +// It is a pure read, the same contract as Drift and Explain: no Session, no +// volume lock, no ops, no journal write, no materialize — and with be == nil, +// no network either. `bdrive verify` is what someone runs to check a folder +// they already suspect; a version of it that repaired anything would destroy +// the evidence it was asked to describe. +// +// Unlike Drift, which compares size+mtime against the state cache, this hashes +// content — that is the whole point, and there is deliberately no fast path. +// A file whose bytes changed but whose size and mtime were restored is exactly +// what Drift cannot see. +// +// device is this device's ID, for the never-pushed count. be may be nil, which +// skips the hub leg entirely. +// +// Known limitation, and the command prints it: the journals AllOps replays are +// this device's LOCAL copies. Without a pull, this proves "the folder matches +// what I last pulled" — and with a backend, "and the hub still holds all of +// it". A teammate's newer op committed since the last cycle is invisible here. +func Verify(ctx context.Context, folder string, include []string, st *store.Store, device string, be remote.Backend) (VerifyReport, error) { + var rep VerifyReport + start := time.Now() + + sync, err := st.LoadSync() + if err != nil { + return rep, err + } + + // A fresh filter: addNestedMount mutates it during a walk, so this must + // never be shared with a live cycle. AcceptRules for the reason Explain + // documents — the scan applies Filter.SkipUp, and omitting it would + // disagree with the very next cycle. + filter, err := loadFilter(folder, include) + if err != nil { + return rep, err + } + filter.AcceptRules(sync.IgnoreAccepted) + + ops, err := st.AllOps() + if err != nil { + return rep, err + } + // The identical object Cycle materializes from (syncer.go, the pull phase), + // so what this compares against is what the cycle would write. + state := journal.Replay(ops) + + paths, err := SyncedFiles(folder, include, sync.IgnoreAccepted) + if err != nil { + return rep, err + } + + seen := make(map[string]bool, len(paths)) + for _, rel := range paths { + seen[rel] = true + want, known := state[rel] + if !known { + rep.NotYetScanned = append(rep.NotYetScanned, rel) + continue + } + abs := filepath.Join(folder, filepath.FromSlash(rel)) + sum, err := hashFile(abs) + if err != nil { + continue // vanished or unreadable; skipped, never fatal, as everywhere in this walk + } + rep.Files++ + if fi, err := os.Stat(abs); err == nil { + rep.Bytes += fi.Size() + } + if sum != want.Blob { + rep.Drifted = append(rep.Drifted, rel) + } + } + + for rel, want := range state { + // The same guard materialize applies before it would write: a path the + // local filter excludes is LEGITIMATELY absent from disk, because the + // rules are applied symmetrically in scan and materialize. Without + // this, every project narrowed by `bdrive scope --only` would report + // every out-of-scope path as missing. + if seen[rel] || filter.Skip(rel) || neverSync(rel) { + continue + } + rep.MissingLocally = append(rep.MissingLocally, rel) + if !st.HasBlob(want.Blob) { + rep.NotFetched++ + } + } + + // The same arithmetic push and `bdrive status` use: our own ops past the + // push cursor are the ones the hub has never seen. + if myOps, err := st.DeviceOps(device); err == nil { + from := sync.PushedOps + if from > int64(len(myOps)) { + from = int64(len(myOps)) + } + if from < 0 { + from = 0 + } + dedup := map[string]bool{} + for _, op := range myOps[from:] { + if dedup[op.Path] { + continue + } + dedup[op.Path] = true + rep.NeverPushed = append(rep.NeverPushed, op.Path) + } + } + + if be != nil { + verifyRemote(ctx, be, state, &rep) + } + + for _, s := range [][]string{rep.Drifted, rep.NeverPushed, rep.MissingLocally, rep.NotYetScanned, rep.MissingOnHub} { + sort.Strings(s) + } + rep.Elapsed = time.Since(start) + return rep, nil +} + +// verifyRemote asks the hub whether it still holds every distinct blob the +// journal state references — one existence check per blob, not per path. +func verifyRemote(ctx context.Context, be remote.Backend, state map[string]journal.FileState, rep *VerifyReport) { + byBlob := map[string][]string{} + size := map[string]int64{} + for rel, want := range state { + if want.Blob == "" { + continue + } + byBlob[want.Blob] = append(byBlob[want.Blob], rel) + if want.Size > size[want.Blob] { + size[want.Blob] = want.Size + } + } + blobs := make([]string, 0, len(byBlob)) + for b := range byBlob { + blobs = append(blobs, b) + } + sort.Strings(blobs) // stable order, so a partial run after RemoteErr is reproducible + + for _, b := range blobs { + ok, err := existsEither(ctx, be, b, size[b]) + if err != nil { + // Stop the remote leg, keep the local report. Never a hard failure. + rep.RemoteErr = err + return + } + if !ok { + rep.MissingOnHub = append(rep.MissingOnHub, byBlob[b]...) + } + } +} + +// existsEither asks for a blob under both key shapes it can live under. +// +// A file over chunkThreshold is pushed as content-defined chunks plus a +// manifest keyed by the FILE's own sha256 — manifests/, never +// blobs/ (chunks.go). A check that only asked blobs/ would report +// every large file as missing from the hub. +// +// The size only picks which key to try FIRST; it can never be a filter, +// because a large file legitimately lives under blobs/ in three ways: the +// browser upload path always writes blobs/ at any size, pushChunked falls +// back to a whole-blob Put when the manifest key is refused, and anything +// pushed before delta sync existed is a whole blob regardless of size. +// +// Checking the manifest key alone is enough for a chunked file: +// "a manifest exists ⟹ its chunks exist" is enforced hub-side at ingest, not +// an honest-client convention, so there is no need to walk the chunk list. +func existsEither(ctx context.Context, be remote.Backend, blob string, size int64) (bool, error) { + keys := [2]string{"blobs/" + blob, "manifests/" + blob} + if size > chunkThreshold { + keys[0], keys[1] = keys[1], keys[0] + } + for _, k := range keys { + ok, err := be.Exists(ctx, k) + if err != nil { + return false, err + } + if ok { + return true, nil // short-circuit: the common case is one round trip + } + } + return false, nil +} diff --git a/internal/syncer/verify_test.go b/internal/syncer/verify_test.go new file mode 100644 index 0000000..e3a07f8 --- /dev/null +++ b/internal/syncer/verify_test.go @@ -0,0 +1,301 @@ +package syncer + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/runbear-io/beardrive/internal/remote" + "github.com/runbear-io/beardrive/internal/store" +) + +// verifyLocal runs Verify against a device the way `bdrive verify` does with +// no --remote: the device's own store and ID, no backend. +func verifyLocal(t *testing.T, s *Session) VerifyReport { + t.Helper() + rep, err := Verify(context.Background(), s.Folder, nil, s.Store, s.Device.ID, nil) + if err != nil { + t.Fatal(err) + } + return rep +} + +func verifyRemoteRun(t *testing.T, s *Session) VerifyReport { + t.Helper() + rep, err := Verify(context.Background(), s.Folder, nil, s.Store, s.Device.ID, s.Backend) + if err != nil { + t.Fatal(err) + } + if rep.RemoteErr != nil { + t.Fatalf("unexpected RemoteErr: %v", rep.RemoteErr) + } + return rep +} + +func wantEmpty(t *testing.T, name string, got []string) { + t.Helper() + if len(got) != 0 { + t.Fatalf("%s = %v, want empty", name, got) + } +} + +func wantOnly(t *testing.T, name string, got []string, want ...string) { + t.Helper() + if strings.Join(got, ",") != strings.Join(want, ",") { + t.Fatalf("%s = %v, want %v", name, got, want) + } +} + +// TestVerifyCleanAfterConverge: two devices that have converged both report a +// fully empty verdict — nothing drifted, nothing pending, nothing missing. +func TestVerifyCleanAfterConverge(t *testing.T) { + be := sharedRemote(t) + a := newDevice(t, "deva", be) + b := newDevice(t, "devb", be) + + write(t, a.Folder, "doc.txt", "v1") + write(t, a.Folder, "sub/nested.md", "deep") + cycle(t, a) + cycle(t, b) + + for _, d := range []*Session{a, b} { + rep := verifyLocal(t, d) + if rep.Problems() != 0 { + t.Fatalf("%s: Problems() = %d, want 0 (%+v)", d.Device.ID, rep.Problems(), rep) + } + if rep.Files != 2 { + t.Fatalf("%s: hashed %d file(s), want 2", d.Device.ID, rep.Files) + } + if rep.Bytes == 0 { + t.Fatalf("%s: Bytes should be > 0", d.Device.ID) + } + } +} + +// TestVerifyCatchesRestoredMtime is the criterion Drift cannot meet: bytes +// changed, size and mtime put back. Drift compares size+mtime and sees +// nothing; Verify hashes and catches it. +func TestVerifyCatchesRestoredMtime(t *testing.T) { + a := newDevice(t, "deva", sharedRemote(t)) + write(t, a.Folder, "doc.txt", "aaaa") + cycle(t, a) + + abs := filepath.Join(a.Folder, "doc.txt") + fi, err := os.Stat(abs) + if err != nil { + t.Fatal(err) + } + // Same length, different bytes, original mtime restored. + if err := os.WriteFile(abs, []byte("bbbb"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.Chtimes(abs, fi.ModTime(), fi.ModTime()); err != nil { + t.Fatal(err) + } + + rep := verifyLocal(t, a) + wantOnly(t, "Drifted", rep.Drifted, "doc.txt") + + // And the whole point: the cheap check reports nothing. + added, modified, removed, err := Drift(a.Folder, nil, "", cacheOf(t, a)) + if err != nil { + t.Fatal(err) + } + if added+modified+removed != 0 { + t.Fatalf("Drift saw %d/%d/%d changes; it is supposed to be blind to this", added, modified, removed) + } +} + +// cacheOf is the materialization cache Drift compares against, for the mount +// the test session materializes into. +func cacheOf(t *testing.T, s *Session) map[string]store.CachedFile { + t.Helper() + c, err := s.Store.LoadCache(s.mountID()) + if err != nil { + t.Fatal(err) + } + return c +} + +// TestVerifyMissingLocally: a path the journal holds and this folder does not. +func TestVerifyMissingLocally(t *testing.T) { + be := sharedRemote(t) + a := newDevice(t, "deva", be) + b := newDevice(t, "devb", be) + + write(t, a.Folder, "keep.txt", "k") + write(t, a.Folder, "gone.txt", "g") + cycle(t, a) + cycle(t, b) + + // Delete behind the daemon's back — no cycle, so no delete op. + if err := os.Remove(filepath.Join(b.Folder, "gone.txt")); err != nil { + t.Fatal(err) + } + rep := verifyLocal(t, b) + wantOnly(t, "MissingLocally", rep.MissingLocally, "gone.txt") + wantEmpty(t, "Drifted", rep.Drifted) + if rep.NotFetched != 0 { + t.Fatalf("NotFetched = %d, want 0 — the blob is right here", rep.NotFetched) + } +} + +// TestVerifyIgnoredPathNotMissing is the --only false-alarm case: a path the +// LOCAL filter excludes is legitimately absent from disk, because the rules +// are applied symmetrically in scan and materialize. +func TestVerifyIgnoredPathNotMissing(t *testing.T) { + be := sharedRemote(t) + a := newDevice(t, "deva", be) + b := newDevice(t, "devb", be) + + write(t, a.Folder, "keep.txt", "k") + write(t, a.Folder, "secret/thing.txt", "s") + cycle(t, a) + cycle(t, b) + + // B narrows its scope and drops the excluded path, the way materialize + // would on the next cycle. + if err := os.RemoveAll(filepath.Join(b.Folder, "secret")); err != nil { + t.Fatal(err) + } + rules := "secret/\n" + write(t, b.Folder, IgnoreFile, rules) + sync, err := b.Store.LoadSync() + if err != nil { + t.Fatal(err) + } + sync.IgnoreAccepted = rules + if err := b.Store.SaveSync(sync); err != nil { + t.Fatal(err) + } + + rep := verifyLocal(t, b) + wantEmpty(t, "MissingLocally", rep.MissingLocally) + + // Negative control: without the rule the very same folder DOES report it, + // so the assertion above is about the filter and not about the path + // having quietly fallen out of the journal state. + if err := os.Remove(filepath.Join(b.Folder, IgnoreFile)); err != nil { + t.Fatal(err) + } + sync.IgnoreAccepted = "" + if err := b.Store.SaveSync(sync); err != nil { + t.Fatal(err) + } + if rep := verifyLocal(t, b); len(rep.MissingLocally) != 1 || rep.MissingLocally[0] != "secret/thing.txt" { + t.Fatalf("without the rule MissingLocally = %v, want [secret/thing.txt]", rep.MissingLocally) + } +} + +// TestVerifyNeverPushed: an offline cycle commits ops the hub never sees. +func TestVerifyNeverPushed(t *testing.T) { + a := newDevice(t, "deva", nil) // nil backend: nothing can be pushed + write(t, a.Folder, "notes/draft.md", "wip") + cycle(t, a) + + rep := verifyLocal(t, a) + wantOnly(t, "NeverPushed", rep.NeverPushed, "notes/draft.md") + wantEmpty(t, "Drifted", rep.Drifted) + wantEmpty(t, "NotYetScanned", rep.NotYetScanned) +} + +// TestVerifyNotYetScanned: a file on disk the filter syncs, with no op at all. +func TestVerifyNotYetScanned(t *testing.T) { + a := newDevice(t, "deva", sharedRemote(t)) + write(t, a.Folder, "doc.txt", "v1") + cycle(t, a) + + write(t, a.Folder, "fresh.txt", "brand new") // no cycle + rep := verifyLocal(t, a) + wantOnly(t, "NotYetScanned", rep.NotYetScanned, "fresh.txt") + wantEmpty(t, "Drifted", rep.Drifted) + wantEmpty(t, "MissingLocally", rep.MissingLocally) +} + +// TestVerifyRemoteChunkedFile is the manifests/ criterion: a file over +// chunkThreshold is pushed as chunks plus a manifest keyed by the file's own +// sha, never blobs/. A --remote check that asked only blobs/ would report +// every large file as missing from the hub. +func TestVerifyRemoteChunkedFile(t *testing.T) { + dir := t.TempDir() + a := newDevice(t, "deva", fileRemote(t, dir)) + big := strings.Repeat("beardrive chunked payload — 64 bytes of filler here!!\n", (chunkThreshold/53)+200_000/53) + if len(big) <= chunkThreshold { + t.Fatalf("test payload is %d bytes, need > %d", len(big), chunkThreshold) + } + write(t, a.Folder, "video/demo.bin", big) + write(t, a.Folder, "small.txt", "tiny") + cycle(t, a) + + // The test is only meaningful if the big file really took the chunked + // path: its content must be reachable ONLY under manifests/, so a + // check that asked blobs/ alone would have to report it missing. + sum, err := hashFile(filepath.Join(a.Folder, "video/demo.bin")) + if err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(dir, "manifests", sum)); err != nil { + t.Fatalf("the >4MiB file was not chunked, so this test proves nothing: %v", err) + } + if _, err := os.Stat(filepath.Join(dir, "blobs", sum)); err == nil { + t.Fatal("the >4MiB file is also a whole blob, so this test proves nothing") + } + + rep := verifyRemoteRun(t, a) + wantEmpty(t, "MissingOnHub", rep.MissingOnHub) + if rep.Problems() != 0 { + t.Fatalf("Problems() = %d, want 0 (%+v)", rep.Problems(), rep) + } +} + +// TestVerifyRemoteMissingBlob: content this device believes is synced, gone +// from the hub's storage. +func TestVerifyRemoteMissingBlob(t *testing.T) { + dir := t.TempDir() + be := fileRemote(t, dir) + a := newDevice(t, "deva", be) + + write(t, a.Folder, "doc.txt", "v1") + cycle(t, a) + if rep := verifyRemoteRun(t, a); rep.Problems() != 0 { + t.Fatalf("before removal: Problems() = %d, want 0 (%+v)", rep.Problems(), rep) + } + + // Blow the blob out of the remote directory. + blobs := filepath.Join(dir, "blobs") + entries, err := os.ReadDir(blobs) + if err != nil { + t.Fatal(err) + } + removed := 0 + for _, e := range entries { + if err := os.RemoveAll(filepath.Join(blobs, e.Name())); err != nil { + t.Fatal(err) + } + removed++ + } + if removed == 0 { + t.Fatal("no blobs in the remote to remove") + } + + rep := verifyRemoteRun(t, a) + wantOnly(t, "MissingOnHub", rep.MissingOnHub, "doc.txt") + // The local half is still clean — the bytes are on this disk. + wantEmpty(t, "Drifted", rep.Drifted) + if rep.Elapsed <= 0 { + t.Fatal("Elapsed should be measured") + } +} + +// fileRemote is sharedRemote against a directory the test can reach into, so +// it can delete a blob out from under a converged device. +func fileRemote(t *testing.T, dir string) remote.Backend { + t.Helper() + be, err := remote.Open(context.Background(), "file://"+dir) + if err != nil { + t.Fatal(err) + } + return be +} diff --git a/internal/webapp/cli_e2e_test.go b/internal/webapp/cli_e2e_test.go index 9773a45..064fbcd 100644 --- a/internal/webapp/cli_e2e_test.go +++ b/internal/webapp/cli_e2e_test.go @@ -1113,3 +1113,86 @@ func TestCLIStatusReportsUnscannedWork(t *testing.T) { t.Fatalf("drift did not clear after a sync:\n%s", out) } } + +// TestCLIVerifyE2E drives `bdrive verify` against a real hub through the real +// binary: a clean project passes, an edit made behind the daemon's back is +// reported as drifted with a status-1 exit, --remote passes against the live +// hub, and --remote against a hub that has gone away still returns the local +// verdict rather than failing. +func TestCLIVerifyE2E(t *testing.T) { + e := newCLIEnv(t) + run := e.run + + work := t.TempDir() + if err := os.WriteFile(filepath.Join(work, "index.md"), []byte("# Index\n"), 0o644); err != nil { + t.Fatal(err) + } + if out, err := run(work, "init", "--name", "verify-e2e", "--yes"); err != nil { + t.Fatalf("init: %v\n%s", err, out) + } + defer run(work, "stop", work) + if out, err := run(work, "sync"); err != nil { + t.Fatalf("sync: %v\n%s", err, out) + } + // Stop the daemon, or it scans the edit below out from under the test. + if out, err := run(work, "stop", work); err != nil { + t.Fatalf("stop: %v\n%s", err, out) + } + + // --- Clean: exit 0, and it says so. + out, err := run(work, "verify") + if err != nil { + t.Fatalf("verify on a clean project should exit 0: %v\n%s", err, out) + } + if !strings.Contains(out, "OK - the folder matches the journal") { + t.Fatalf("clean verify did not report OK:\n%s", out) + } + // index.md plus the .bdriveignore init seeds — the point is that the cost + // is reported at all, not the exact count. + if !strings.Contains(out, "checked: 2 files, ") { + t.Fatalf("verify did not report what it hashed:\n%s", out) + } + + // --- The hub still holds it: --remote is clean too. + out, err = run(work, "verify", "--remote") + if err != nil { + t.Fatalf("verify --remote on a synced project should exit 0: %v\n%s", err, out) + } + if strings.Contains(out, "missing-on-hub") { + t.Fatalf("verify --remote reported content missing from the hub:\n%s", out) + } + + // --- Edited behind the daemon's back: drifted, exit 1. + if err := os.WriteFile(filepath.Join(work, "index.md"), []byte("# Index\n\nappended by hand\n"), 0o644); err != nil { + t.Fatal(err) + } + out, err = run(work, "verify") + if err == nil { + t.Fatalf("verify should exit 1 when something drifted:\n%s", out) + } + if !strings.Contains(out, "drifted (1)") || !strings.Contains(out, "index.md") { + t.Fatalf("verify did not name the drifted file:\n%s", out) + } + // A status is not a stack trace: cobra's usage block must stay silenced. + if strings.Contains(out, "Usage:") { + t.Fatalf("verify printed a usage block for a findings exit:\n%s", out) + } + + // --- Hub gone: the remote leg degrades to a warning, the local verdict + // still decides. Put the file back first so "local is clean" is the + // thing being asserted. + if err := os.WriteFile(filepath.Join(work, "index.md"), []byte("# Index\n"), 0o644); err != nil { + t.Fatal(err) + } + e.hub.Close() + out, err = run(work, "verify", "--remote") + if err != nil { + t.Fatalf("verify --remote against an unreachable hub must not hard-fail: %v\n%s", err, out) + } + if !strings.Contains(out, "warning:") { + t.Fatalf("verify --remote said nothing about the unreachable hub:\n%s", out) + } + if !strings.Contains(out, "OK - the folder matches the journal") { + t.Fatalf("verify --remote dropped the local verdict when the hub went away:\n%s", out) + } +} diff --git a/web/docs/src/content/docs/reference/cli.md b/web/docs/src/content/docs/reference/cli.md index d93ab63..5692349 100644 --- a/web/docs/src/content/docs/reference/cli.md +++ b/web/docs/src/content/docs/reference/cli.md @@ -19,6 +19,7 @@ One binary, `bdrive` — the CLI, the sync daemon, and the web server. | `bdrive scope --explain` | List every path in the folder, split into what syncs and what does not, with counts — the verifiable answer to "what leaves this machine". Pure read: no daemon, no lock, no network | | `bdrive grep [folder]` | Search the text **inside** the files a project syncs. `pattern` is a Go RE2 regexp, or a literal string with `-F`. `-i` ignores case, `-l` prints matching paths only, `-n` caps the lines printed (default 200, `0` = all). Pure read: no daemon, no lock, no network | | `bdrive stale [folder]` | Find synced markdown that links to a file written **after** the doc itself — staleness by what moved, not by the calendar. `-l` prints outgrown paths only, `-n` caps the docs printed (default 50, `0` = all). Pure read: no daemon, no lock, no network. Exit status is 0 whether or not anything is stale | +| `bdrive verify [folder]` | Prove this folder matches the content the journal records — sha256 every synced file and compare. Reports `drifted`, `never-pushed`, `missing-locally`, `not-yet-scanned`, and with `--remote` also `missing-on-hub`. Pure read: no daemon, no lock, no ops, and without `--remote` no network. Exit status is 0 when every category is empty, 1 otherwise | | `bdrive forget ...` | Stop syncing a path and remove it from the hub. Adds the rule to `.bdriveignore` (which syncs) and prunes in one step. Local files are never touched, here or on teammates' devices | | `bdrive url [path]` | Internal hub link for a file or folder — sign-in and membership required. `--sync` pushes first; no argument gives the project home. Computed locally | | `bdrive share ` | Public URL for a synced file — links are per-file, so a folder is refused with a file inside it named instead. `--list`, `--revoke`, `--expires` (the hub's Share dialog can also set an expiry on an existing link). Refuses a file whose first 1 MiB holds credential-shaped strings — `--force` shares it anyway | @@ -204,6 +205,67 @@ would invert here and fail on a clean project. Read heat, a badge on the hub's file view, and injecting the flag into an agent's session context are not built yet; this ships the signal. +### `bdrive verify` — prove this folder matches the hub + +"Your folder is the same everywhere" is the claim. `bdrive verify` is the +receipt. It hashes every file the project syncs and compares each one against +the content the journal records for it. + +This is what `bdrive status` cannot do. `status` counts pending ops and +unscanned changes, but it never reads a byte of content — so a file whose bytes +changed while its size and mtime stayed put is invisible to it. `verify` +hashes, which is the whole point: there is deliberately no size/mtime fast +path, and on a large project it is noticeably slower than `status`. It reports +what it hashed and how long it took for exactly that reason. + +```sh +bdrive verify +# project: team-wiki (m-a3f9c1d2) +# checked: 184 files, 41.2 MB hashed in 912ms +# OK - the folder matches the journal +# (this compares against what this device last pulled — run `bdrive sync` first for the hub's latest) + +bdrive verify --remote # also ask the hub whether it still holds the content +bdrive verify ~/wiki # a project other than the current folder +``` + +It reports five things: + +| Category | Meaning | +| --- | --- | +| `drifted` | on disk, but not the content the journal records | +| `never-pushed` | committed here, never reached the hub | +| `missing-locally` | the journal has it, this folder does not | +| `not-yet-scanned` | on disk, with no op anywhere yet | +| `missing-on-hub` | synced here, absent from the hub — `--remote` only | + +A path the local `.bdriveignore` excludes is **not** reported as missing. The +rules are applied symmetrically in scan and materialize, so its absence from +disk is correct — a project narrowed with `bdrive scope --only` stays quiet. +And when `missing-locally` is really just "not downloaded yet", the output says +so and points at `bdrive sync`. + +`--remote` adds one existence check per **distinct blob**, and asks for both +`blobs/` and `manifests/`. Files over 4 MiB move as content-defined +chunks plus a manifest keyed by the file's own sha256, so a check that asked +only `blobs/` would call every large file missing from the hub. A manifest +existing implies its chunks do — the hub enforces that at ingest — so there is +no chunk-by-chunk walk. If the hub is unreachable, the remote half degrades to +a printed warning and the local verdict still stands. + +**`verify` never repairs anything.** It is a pure read: no daemon, no lock, no +ops, no journal writes. Run `bdrive sync` to reconcile what it found, or +`bdrive log ` for a file's history. + +**Exit status is 0 when every category is empty and 1 when any is not,** so it +composes as a pre-flight check in a script or a CI job. + +One caveat the command prints for itself: the journals it replays are this +device's **local copies**. Without a pull it proves "this folder matches what I +last pulled" — and with `--remote`, "and the hub still holds all of it". A +teammate's newer op is invisible until the next cycle, so sync first if you +want the hub's latest. + ### `bdrive forget` and `bdrive sync --prune` — cleaning up the hub Adding a rule to `.bdriveignore` only stops *future* uploads. Anything that