diff --git a/.claude/delta-sync-goal.md b/.claude/delta-sync-goal.md new file mode 100644 index 0000000..14b64f1 --- /dev/null +++ b/.claude/delta-sync-goal.md @@ -0,0 +1,318 @@ +# Delta sync — shared goal + +Read this file first, every round. It is the only definition of "done". +Nothing else — not a passing build, not a benchmark screenshot, not a +paragraph explaining that chunking is implemented — ends the loop. + +Design and rationale: [`docs/delta-sync-prd.md`](../docs/delta-sync-prd.md). +This file is the contract; the PRD is the reasoning behind it. Where they +disagree, this file wins and the PRD gets fixed. + +## Mission + +A changed file must transfer only its changed regions, on push and on pull, +**without changing a single byte of the journal format** and without altering +what any device converges to. + +## The one rule + +**A saving does not exist until a byte-counting test asserts it, and no saving +is accepted unless a convergence test lands in the same commit.** + +Bytes are the whole point of this work, so "we implemented chunking" is not a +result — a number is. And a transfer that shrinks while convergence drifts is +not an optimization, it is data loss with a faster wire. + +Symmetrically: an invariant is not protected because someone was careful. It is +protected by a test that fails when it breaks. Tests are never deleted, +skipped, or loosened to close a row. + +## What counts as done + +The loop ends when **all three** hold: + +1. Every scoreboard row is `pass`, each backed by a named Go test in the repo. +2. `go build ./... && go vet ./... && go test ./...` is green on a clean tree. +3. Every row in the **E block** passes — including the two that require a + binary built from the merge-base commit. Nothing ships to users on a green + in-process suite alone. + +There is no "mostly there" and no percentage. A row passes or it is open. + +## Hard stop: the measurement gate + +**No production code ships past the counting harness until the go/no-go row has +a number in it.** + +BearDrive syncs markdown and code. For a 4 KB file, chunking costs more than it +saves. If the corpus turns out to be small text files, the correct outcome of +this entire loop is: ship transport compression (there is none today — no gzip, +no zstd, no `Content-Encoding` anywhere in `internal/remote`, +`internal/syncer`, or the hub proxy), record the number, and stop. + +Shipping chunking against a corpus that does not need it is a failure of this +goal, not a partial success. + +## The counting harness + +Everything below is measured through one ~30-line test decorator over +`remote.Backend`, wrapping the `file://` store that `sharedRemote` +(`internal/syncer/syncer_test.go:75`) already returns and dropping into +`newDevice` (`:61`) unchanged: + +```go +// countingBackend records bytes crossing the wire in both directions. +type countingBackend struct { + remote.Backend + put, get atomic.Int64 +} +``` + +Build this first. It is the instrument every acceptance criterion reads. + +## Test conventions + +- Named `TestDelta__` (e.g. `TestDelta_Push_FrontInsertion`); + E-block rows are `TestDeltaE2E_`. +- Sync behavior lives in `internal/syncer/delta_test.go`, built on the existing + multi-device pattern (`newDevice` + `sharedRemote` + explicit `cycle()` + calls). A sync change without a multi-device test is untested where it + matters. +- Hub behavior lives in `internal/webapp/chunks_test.go`. +- End-to-end rows live in `internal/webapp/delta_e2e_test.go` and **must** use + `newCLIEnv` (`cli_e2e_test.go:41`) — it `go build`s the real binary, runs it + against a real `startTestHub` over real HTTP with an isolated `HOME`. +- Export/import lives with `cmd/bdrive/migrate.go`'s existing tests. +- `sandbox/` is not used. All of this is deterministic and machine-local, so + all of it is Go tests. + +**In-process tests do not close an E row.** A row in the E block is closed only +by a test that drives the real binary over real HTTP. Anywhere else, +in-process is fine and preferred — it is faster and the coverage is the same. + +No Playwright. The browser never touches blob content directly; every read +funnels through hub handlers that E2/E3 already cover with real bytes over real +HTTP, so a browser driver would re-test the same choke point through a slower +harness. (Also checked: the hub serves no `Range` requests today — content is +`io.Copy`'d — so reassembly cannot regress partial-content behavior that does +not exist.) + +## Scoreboard + +Fill in the test name when a row passes. A row with no test name is open, +whatever the state column says. + +### Gate — measurement + +| # | Criterion | Owner | Test | State | +|---|---|---|---|---| +| G1 | The counting backend counts Put/Get bytes accurately | agent | `TestDelta_Harness_CountsBytes` | **pass** | +| G2 | Baseline recorded: 1-byte edit to a 20 MB file pushes ~20 MB today | agent | `TestDelta_Baseline_WholeFileCost` | **pass** | +| G3 | Size + churn distribution measured on ≥3 real projects | **human** | — (numbers in PRD §Phase 0) | **pass** | +| G4 | Transport compression measured standalone | agent | `TestDelta_Gzip_TextCorpusRatio` | **pass** | +| G5 | **Go / no-go recorded, with the number that decided it** | **human** | — | **pass** | + +> G2 is a characterization test. When delta lands it is **updated** to the new +> bound, never deleted — it is the before-and-after in one place. +> +> **G3 and G5 cannot be closed by an agent and must never be guessed.** G3 +> needs real user corpora, which are not in this repo; G5 is a product +> decision. An agent that reaches them stops, reports what G1/G2/G4 measured, +> and asks. Inventing a plausible-looking distribution to get past the gate is +> the single worst thing that can happen in this loop — it converts a +> measurement into a rationalization and every later round inherits it. + +### Transport — the savings + +Bound is "small multiple of the max chunk size", not "less than the file". + +| # | Criterion | Test | State | +|---|---|---|---| +| D1 | 1-byte edit mid-file, 20 MB file: push < 5 MiB | `TestDelta_Baseline_WholeFileCost` | **pass** | +| D2 | Peer holding the basis pulls that edit in < 5 MiB | `TestDelta_Pull_SmallEdit` | **pass** | +| D3 | Append to a 20 MB file: push < 5 MiB | `TestDelta_Push_Append` | **pass** | +| D4 | **Insertion at the START of a 20 MB file: push < 5 MiB** | `TestDelta_Push_FrontInsertion` | **pass** | +| D5 | Files ≤ 4 MiB write no `chunks/` key at all | `TestDelta_Threshold_SmallFileUnchanged` | **pass** | +| D6 | Cold pull with no basis is one whole-blob GET | `TestDelta_Pull_ColdPathUnchanged` | **pass** | + +> D4 is the one that matters. Fixed-size blocking passes D1 and D3 and fails +> D4 — it is the proof that boundaries are content-defined and not an offset +> table. If D4 is red, the chunker is wrong no matter what the others say. + +### Correctness — non-negotiable + +| # | Criterion | Test | State | +|---|---|---|---| +| C1 | **Journal bytes for a given edit are identical to the pre-change binary** | `TestDelta_Journal_ByteIdentical` | **pass** (in-process half; E2/E3 complete it) | +| C2 | 3 devices, offline edits on two, conflict copies identical to whole-blob behavior | `TestDelta_Converge_ThreeDeviceConflict` | **pass** | +| C3 | Assembled content is verified against `Op.Blob` before entering the blob store | `TestDelta_Pull_RejectsMismatch` | **pass** | +| C4 | A manifest that does not reassemble to its key is refused; nothing is backfilled | `TestDelta_Manifest_SelfVerifying` | **pass** | +| C5 | Interrupt during chunk upload: peer sees no broken op, next cycle heals (manifest-stage refusal now falls back to whole-blob — H4) | `TestDelta_Order_ChunksBeforeManifest` | **pass** | +| C6 | Interrupt between manifest and journal: same | `TestDelta_Order_ManifestBeforeJournal` | **pass** | +| C7 | One missing chunk does not abandon the batch — every complete op behind it still lands | `TestDelta_Pull_MissingChunkDoesNotStall` | **pass** | +| C8 | Chunked-only project survives `export` → `import` with full fidelity | `TestDelta_Migrate_RoundTrip` + `TestDelta_Import_RejectsCorruptChunk` | **pass** | + +> C1 is the strongest guard in this table. It is what makes "no journal format +> change" a fact instead of an intention, and it is what keeps `journal.Less` +> and `Replay` — and therefore every device's convergence — out of scope. +> +> C7 mirrors a bug this repo already fixed once on the blob path +> (`syncer.go:936`): one unfetchable object must not stop everything queued +> behind it. Chunks inherit the same rule and the same test shape. + +### Hub + +| # | Criterion | Test | State | +|---|---|---|---| +| H1 | Reassembly backfills `blobs/` once; the second read does not re-reassemble | `TestDelta_Hub_BackfillOnce` | **pass** | +| H2 | Chunk presigning refuses a key that already exists (sealing invariant) | `TestDelta_Hub_ChunkPresignRefusesExisting` | **pass** | +| H3 | Manifests are never presigned — always server-relayed | `TestDelta_Hub_ManifestNeverPresigned` | **pass** | +| H4 | `validStoreKey`, the list prefix allowlist, and the put hash check accept and constrain both new key classes | `TestDelta_Hub_KeySpace` | **pass** | + +### E — end to end, real binary only + +These close **only** with `newCLIEnv`. An in-process reproduction of one of +these rows is a useful test and does not close the row. + +| # | Criterion | Test | State | +|---|---|---|---| +| E1 | Two real `bdrive` processes, real hub: a 20 MB file edited on one converges on the other, and the wire cost is < 5 MiB | `TestDeltaE2E_TwoDevicesLargeFile` | **pass** | +| E2 | **A binary built from the merge-base commit syncs a project whose storage holds only chunks and manifests** — full pull, correct bytes on disk | `TestDeltaE2E_OldBinaryReadsChunkedStorage` | **pass** | +| E3 | **A binary built from the merge-base commit pushes; a current binary pulls** — converges byte-identically | `TestDeltaE2E_OldBinaryWritesNewReads` | **pass** | +| E4 | Real `bdrive export` → real `bdrive import` → third real device syncs the imported project and gets correct bytes | `TestDeltaE2E_MigrateRoundTrip` | **pass** | +| E5 | Viewer, history `/blob`, share link, and download serve correct bytes for a chunked-only file over real HTTP | `TestDeltaE2E_AllReadSurfaces` | **pass** | +| E6 | Daemon path, not just one-shot `sync`: a chunked edit propagates through a running daemon | `TestDeltaE2E_DaemonPropagates` | **pass** | + +> **E2 and E3 are the reason this block exists.** Every other row in this file +> can be satisfied by code that was written knowing about chunks. These two +> cannot: they require a binary that has never heard of a manifest, and the +> only honest way to get one is to build it — +> `git worktree add $(git merge-base HEAD main)` then `go build` in +> there. Simulating an old client by disabling a code path in the current tree +> tests the flag, not the compatibility. If these are red, upgrading a hub +> breaks every device that has not upgraded yet, which is the single worst +> outcome this project can produce. +> +> E6 exists because every other sync row drives `cycle()` or `bdrive sync` +> directly. The daemon is how sync actually runs for real users, and it has its +> own lifecycle (flock, intervals, config re-read) that one-shot calls skip. + +## Invariants that fail the round outright + +Break any of these and the round does not close, regardless of the scoreboard: + +1. **Each device writes only its own journal.** Chunks and manifests are + content-addressed with no per-device keys. Keep it that way. +2. **Content before journal**: `chunks → manifest → journal`, never reordered. +3. **No `Op` field is added, removed, or read differently.** `journal.Less` and + `Replay` are not touched. (C1 enforces this.) +4. **Scan before pull** in `Cycle`. +5. **Materialize never clobbers dirty files.** Assembly happens in the volume + store, never in the working folder. +6. **Never break sync, retry next cycle.** A missing chunk, a bad manifest, or + a failed reassembly degrades to `Result.Offline` or skips that path. Record + the first error, finish the batch, return it once — the posture `pull` + already has. +7. **No read returns bytes that do not hash to the sha requested**, hub-side or + client-side. +8. **The local volume store layout does not change.** `store.OpenBlob`, + `PutBlobFile`, `writeFile`, `materializeFile` stay as they are. + +## Known trap + +The store layout is enumerated in more places than it looks, and one of them +loses data in silence. **All of these land before anything writes a chunk:** + +`cmd/bdrive/migrate.go:193` (export), `:280` (import), `:35` (`blobKeyRe`, a +duplicate of the hub's), `webapp/store.go:34` (`validStoreKey`), `:176` (list +prefix allowlist), `:528` (put hash check). + +Miss the first one and `bdrive export` produces an archive that is missing file +content, with no error — the anti-lock-in story quietly broken. + +## Round protocol + +Each round: + +1. Re-read this file. +2. Pick the topmost open row. Order is Gate → Hub key space → Transport → + Correctness → E. Correctness rows for a phase land with that phase, never + after. **E2 and E3 are the exception to "topmost": open them as soon as + anything writes a chunk**, because they are the rows most likely to force a + design change, and the cost of discovering that last is the whole + implementation. +3. Write the test. Run it. Paste the failure. +4. Implement until it passes and `go test ./...` is green. +5. Update the row with the test name and `pass`. +6. If a round produces no passing row, say so plainly and say what blocked it. + A round that ends with a summary and no scoreboard change is a failed round. + +Anything you suspect but cannot reproduce goes in a `## Leads` section at the +bottom of this file. A lead is not a finding and never closes a row. + +### When a row cannot be closed as written + +Two escapes, and no third: + +- **The row is human-owned** (G3, G5) or needs something outside this repo: + stop the loop, report what is measured so far, and ask. Do not estimate, do + not proceed to the next row, do not mark it `pass` with a caveat. +- **The design is wrong.** If a row — most likely E2 or E3 — cannot be made to + pass without violating an invariant, that is a finding about the design, not + a reason to weaken the row. Say so, propose the change, and update + `docs/delta-sync-prd.md` in the same commit as the fix. The PRD is expected + to change; this file's rows and invariants are not. + +Working around a row is never one of the escapes. A row loosened to make a +round end has cost more than it saved. + +## Leads + +- ~~Manifest write-once pins the chunker parameters~~ **Downgraded (CTO round + 4): the params are no longer load-bearing for correctness anywhere.** A + manifest refusal (write-once 409 or ingest 400) falls back to a whole-blob + push, so a future change to `chunkPol`/`chunkMin`/`chunkMax` costs one full + upload per affected file instead of a wedge or a migration. Change them + freely if a better set is found; dedup across the boundary degrades, sync + does not. +- **Reassembly backfill still bypasses quota accounting** (CTO M2): the + hub-authored `blobs/` write records no usage. Harmless under OSS + UnlimitedQuota; the managed layer should wire RecordUsage at that seam. +- **Chunk transfer is serialized per file** (CTO M1): a 100 MiB cold push + over HTTP is ~100 sequential round trips. A bounded errgroup inside + pushChunked/fetchChunked when it shows up in real use. +- **No concurrency bound on hub reassembly spools** (CTO M3): 256 MiB × N + concurrent legacy reads; `/s/` is per-IP rate-limited only. A small + semaphore if it shows up. +- **Transient content-fetch failures are never retried until the path's + journal grows.** Pull's fetch loop covers only newOps, and an accepted + journal is never re-walked — so a blob (or now chunk) that failed once on a + network blip stays unmaterialized until someone edits that path. Pre-existing + on the whole-blob path (its "next cycle retries" comment overstates what + happens), inherited unchanged by chunks. A missing-content re-fetch pass over + the replayed target would close it. Separate change, not delta-sync scope. + +## Status + +_**LOOP COMPLETE, 2026-08-12.** All 29 rows pass; `go build && go vet && +go test ./...` green on the full tree (11 packages, incl. the four +real-binary E2E suites and the merge-base old client). Not rows of this loop +and still owed before a PR: README on-disk-layout section, web/docs reference +pages, `architecture/{cli-sync,webapp-server}.md` diagrams (PRD Phase 4), and +the compression follow-up G5 ordered "after chunking"._ + +| Round | Rows closed | Notes | +|---|---|---| +| 1 | G1 | Counting harness in `internal/syncer/delta_test.go`; verified direct and through a real two-device cycle. | +| 2 | G2 | Baseline measured: 1-byte edit to a 20 MiB file pushes 20,972,098 bytes (file + journal). Seeded-random content so the post-delta bound stays meaningful. | +| 3 | G4 | gzip on a real Go/text corpus (41 files, 441 KiB): **3.4×**. Loop now blocked on human-owned G3 (real-corpus size/churn data) and G5 (go/no-go). | +| 4 | G3 | Snow authorized measuring local mounts: 4 real projects, every journal op. Median file 5–10 KB, 7 files > 4 MiB anywhere, large-file rewrites = 8.6% of all historic push bytes. Numbers in PRD §Phase 0. | +| 5 | G5 | **Snow's call: GO on chunking** (big files can exist; chunking is the binary/large-file answer — CDC is content-agnostic), **compression ships after** as a follow-up (the text-bulk answer, 3.4×; must skip incompressible blobs). Decided against the 8.6% ceiling, knowingly. | +| 6 | H4 | Key space landed: `chunks/`+`manifests/` in `validStoreKey`, list allowlist, put hash check (chunks get key-equals-hash; manifests verbatim — verified by reassembly) — **and every `migrate.go` enumerator in the same commit**, per the known trap. Full webapp/cmd/syncer suites green. | +| 7 | H1 H2 H3 C4 | `RemoteSource.OpenBlob` reassembles from a manifest (spool → verify → backfill → serve; a hostile whole-blob that fails verify is HEALED by the backfill when an honest manifest exists); chunks presign like blobs incl. refuse-existing, manifests always server-relayed. One sec-test control updated: a valid sha may now ask `manifests/` after `blobs/` — the guard property (malformed Op.Blob never reaches storage) unchanged, all hostile subtests still pass. Full webapp suite green. | +| 13 | — (CTO round 3: H6) | **The basis machinery is deleted, not patched.** Three client-side skip proxies each proved false or forgeable (local basis; manifest existence; stored manifest content — a member who can READ a file can publish its true hashes without uploading a byte). The only valid proof is asking: pushChunked now does one `Exists` per chunk, and the basisOf/journal-scan derivation in push is gone. Hub ingest enforces "manifest ⟹ chunks exist" (`TestSec_Chunks_ManifestMustNameUploadedChunks`); import enforces the same for archives (`TestDelta_Import_RefusesManifestNamingAbsentChunks`); truthful-squat regression `TestDelta_Basis_TruthfulSquatCannotSkipChunks`. False "blob corrupt on remote" no longer reported when the whole-blob fallback lands the file. Byte counts unchanged. | +| 12 | — (CTO round 2) | Fixed B1 (basis presumed remote — superseded by round 13's deletion), B2 (old export silently loses chunked files → import refuses incomplete archives, `--allow-incomplete` escape), H1 (bad manifest denies good blob → always-fallthrough + write-once manifests), M4 (vacuous upgrade test made honest), M5 (write-once fails closed), M6 (Exists-gated fallback). | +| 11 | — (post-loop hardening) | **Reassembly bound** (`maxReassembleBytes` = 256 MiB, mirrors maxImportBlob): a member-written manifest's declared sum is refused before any chunk fetch, and each chunk's copy is bounded by its declared size so an oversized stored object (replayed presign) can't defeat the cap — `TestSec_Chunks_ReassemblyBoundsHostileManifest`. **Device ceiling raised**: `maxPullBytes` 32 → 100 MiB per Snow (chunking shipped FOR large files; a ceiling below them was dead weight) — pinned by `TestDelta_Ceiling_LargeFileMaterializes` (40 MiB converges). | +| 10 | E1–E6 | **E block closed with real binaries.** Old client built from the pre-change commit via `git archive` (`buildOldBinary`; ref must become the merge-base once this work is committed). E2/E3: old binary syncs chunked-only storage (hub reassembly) and full old↔new round-trip converges. E1 over real HTTP: 1,954,726 pushed / 1,954,696 pulled for a 1-byte edit to 20 MiB. E5: viewer/history/download/anon share all correct. E4: real export → import (import creates the project; the harness's earlier pre-create collided by design). E6: daemon propagates a chunked edit. Three harness fixes en route (stop=pause → cleanup-only; join by `--name`; import owns project creation) — no product changes needed. | +| 9 | C3 C8 | Hostile manifest (individually-valid chunks assembling to the wrong content) surfaces as errBlobContent, never files under the op's sha, never materializes. Chunked-only project round-trips export→import; corrupt chunk in an archive refused like a corrupt blob. All legacy migrate + sec tests green. | +| 8 | D1–D6 C1 C2 C5 C6 C7 | **The chunker landed** (`internal/syncer/chunks.go`, restic/chunker, 256K/1M/4M, fixed Rabin poly; >4 MiB files move as chunks+manifest, basis = previous version of the path on both sides). 1-byte edit: 20,972,098 → **2,050,572 bytes**; front insertion **526,944** (D4, the CDC discriminator); pull-with-basis **632,938**. G2 updated per its own failure message. Full `go test ./...` green incl. daemon/agenthooks. D6 note: on a bare `file://` remote a chunked-only file has no whole blob — cold path is manifest-first by design; hub-served whole-blob cold path is E5's. Lead recorded: transient fetch failures were never retried on the whole-blob path either (journal-growth-triggered only). | diff --git a/CHANGELOG.md b/CHANGELOG.md index 26a4fee..ce53ca3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,31 @@ Notable changes per release. Format loosely follows [Keep a Changelog](https://keepachangelog.com/); BearDrive is pre-1.0, so minor versions may ship breaking changes (see [SemVer §4](https://semver.org/#spec-item-4)). +## Unreleased + +**Deploy hubs before clients.** A hub older than this release refuses the +chunk keys a new client pushes for large files (sync degrades to +offline-retry until the hub upgrades; nothing is lost). And a *client* older +than this release cannot pull files between 32 and 100 MiB — it caps reads +at its old 32 MiB ceiling, fails the content check, and reports "blob +corrupt on remote" every cycle even though the hub is healthy. Upgrade the +hub first, then clients promptly if your projects hold large files. + +- **Delta sync** — files larger than 4 MiB move as content-defined chunks: + a small edit to a 20 MiB file now transfers ~2 MB instead of ~21 MB, in + both directions. Chunk boundaries come from a rolling hash, so insertions + anywhere in the file stay cheap. Local storage keeps files whole; the hub + reassembles whole blobs on demand, so older clients and every hub read + surface (viewer, history, shares, downloads) work unchanged. +- The per-file sync ceiling rises from 32 MiB to 100 MiB — files up to + 100 MiB now materialize on every device. +- `bdrive import` refuses an archive whose journals reference content the + archive does not hold (what an old `bdrive export` produces against a + newer hub, silently missing large files); `--allow-incomplete` overrides. +- Hub hardening around the new key classes: manifests are write-once, must + name only chunks the store holds, and reassembly is bounded (256 MiB) — + a project member cannot poison, re-point, or amplify large-file storage. + ## v0.15.0 — 2026-08-11 **Upgrade if your pushes are being refused.** Hubs running the journal diff --git a/README.md b/README.md index 824d89e..b86ca21 100644 --- a/README.md +++ b/README.md @@ -262,7 +262,7 @@ hub's own storage, never something a syncing client points at directly: | `bdrive log [folder] [-p path] [-n N]` | Change history: account, device, time, file — newest first by the time shown, which is when the file was written (ops recorded before this was tracked, and deletes, show their sync time instead) | | `bdrive restore [version]` | Put an earlier version of a file back, as a new change (`--list` shows the versions; no version = the previous one). Nothing is erased and it syncs everywhere like any edit. To un-create a file a run *created*, use **undo — remove file** on that row in the hub's History view | | `bdrive export [folder]` | Export the whole project — every device's journal, all blobs, full history — from its hub to a portable `.tar.gz` (`-o` names the file) | -| `bdrive import ` | Import an export archive as a new project on the hub you're logged into (always a NEW project; `--name` overrides the archive's); history and authorship carry over. Move projects between hubs — cloud → self-hosted or back — with `export` + `login` + `import` | +| `bdrive import ` | Import an export archive as a new project on the hub you're logged into (always a NEW project; `--name` overrides the archive's); history and authorship carry over. Refuses an archive whose journals reference content it doesn't hold (`--allow-incomplete` overrides). Move projects between hubs — cloud → self-hosted or back — with `export` + `login` + `import` | | `bdrive serve [folder \| storage-root-url]` | Web server: viewer (rendered markdown, downloads, history), uploads, multi-project sync hub (`bdrive web` is a deprecated alias) | | `bdrive whoami` | Signed-in account and device identity used in change tracking | | `bdrive version` | Print the version (also `bdrive --version`) | @@ -626,6 +626,15 @@ working folder ←materialize/scan→ local volume store ←push/pull→ obj devices' journals and any blobs it's missing. Since each device writes only its own journal, there are no concurrent writers per object and any dumb object store suffices. +- Files **larger than 4 MiB move as content-defined chunks** (delta sync): + the remote holds `chunks/` pieces plus a `manifests/` + chunk list keyed by the whole file's hash, and a push uploads only the + chunks the store doesn't already hold — a small edit to a large file + transfers roughly one chunk (~1 MiB), not the file. Chunk boundaries are + chosen by a rolling hash, so insertions don't shift them. The local blob + store keeps files whole; chunking exists only on the wire and in the + remote. Older clients are unaffected: the hub reassembles whole blobs on + demand for anything that asks for `blobs/`. - The folder's state is a deterministic **replay** of all journals ordered by `(lamport, time, device)` — every device converges to the same view. Concurrent edits keep the last writer at the path; the loser is preserved diff --git a/architecture/cli-sync.md b/architecture/cli-sync.md index f19496c..29cd9f7 100644 --- a/architecture/cli-sync.md +++ b/architecture/cli-sync.md @@ -24,6 +24,9 @@ classDiagram +OnProgress func +Cycle(ctx) Result +Restore(ctx, path, sha) error + -pushChunked(ctx, blob) bytes + -fetchChunked(ctx, op, basis) error + -chunkSpans(blob) []span } note for Session "syncer also exposes LogEntries (causal order, what bdrive restore walks) plus DisplayTime / SortForDisplay — the newest-first-by-clock order bdrive log prints" note for Session "Restore writes a historical blob back into the working folder as an ordinary edit (fetching it from the hub when this device never held it) — the next Cycle journals it like any other change; it takes no lock and appends to no journal itself" @@ -138,6 +141,13 @@ classDiagram note for Op "internal/journal — Less orders by (lamport, time, device, seq); Replay folds to LWW-per-path state; each device writes only its own journal. Mtime is display-only (bdrive log shows it, falling back to Time) and never feeds Less or Replay. Session holds the same standing: set only by `bdrive sync --hook` (never by --note, which any member can spell), display/join-only, and the key History run cards group on — a note is forgeable, a session id is not" note for Op "Op now owns its own JSON: a Path that is not valid UTF-8 rides as a base64 `path_raw` sidecar and is restored only when the lossy form still matches, so one line can never name two different files on two readers. Less falls through to Kind/Path/Blob/Size/Mode, making the order TOTAL — two ops can no longer tie and replay differently per device. Parse skips an undecodable line and drops an unknown Kind instead of failing the whole journal" + class Manifest { + +V int + +Size int64 + +Chunks []chunkRef h, n + } + note for Manifest "chunks.go — delta sync. Files over chunkThreshold (4 MiB) push as content-defined chunks (restic/chunker, fixed Rabin polynomial, 256K/1M/4M) under chunks/sha256 plus this manifest under manifests/file-sha — keyed by Op.Blob, so the JOURNAL FORMAT IS UNTOUCHED. pushChunked skips a chunk only when the remote CONFIRMS holding it (one Exists per chunk): three cheaper proxies — local basis, manifest existence, stored manifest content — each proved false or forgeable, and the code comment records why. A refused manifest (write-once 409, ingest 400) falls back to pushing the whole blob, so chunker parameters are not load-bearing. fetchChunked assembles from the manifest, sourcing unchanged chunks from the basis blob already in the local store (cache[path].Blob), verifying the whole against Op.Blob via PutBlobReader; any failure falls through to the whole-blob path, gated on Exists so a transient chunk blip does not trigger hub reassembly every tick. Local blobs stay whole — chunking exists only on the wire and in the remote" + class Backend { <> +Put +Get +List +Exists +Close @@ -153,6 +163,7 @@ classDiagram Session --> Store : volume state Session --> Backend : pull and push + Session --> Manifest : chunked push and pull, files over 4 MiB Session --> Filter : SkipUp on scan, Skip on materialize Session --> walkFolder : scan Explain --> walkFolder : same predicate diff --git a/architecture/webapp-server.md b/architecture/webapp-server.md index a020e69..3a0240b 100644 --- a/architecture/webapp-server.md +++ b/architecture/webapp-server.md @@ -56,6 +56,7 @@ classDiagram +PresignTTL time.Duration +Remove(ctx, path, who, note) +OpenBlob(ctx, sha) + -reassemble(ctx, sha) chunked fallback -verify(ctx, sha) re-hash until sealed -blobStat(ctx, blob) remote.Object -sealed sync.Map sha→proved immutable @@ -78,6 +79,7 @@ classDiagram note for cachedJournal "Journals only GROW — a device appends only to its own, and appendOp rewrites its key with strictly more bytes — so the (Size, Modified) List already reports proves a parse is still current. That is why the cache needs no expiry and no new Backend method: loadSourcedOps still Lists on every request, and fetches only the keys whose size or mtime moved (concurrently, limit 8). History used to re-download and re-parse EVERY journal per page, which is what made it 8-10s and made paging cost more rather than less. Bounded by a per-project raw-byte cap with all-or-nothing eviction" note for sourcedOp "An op's Device field is whatever the writer typed; From is the journal object it actually came out of, which the /store door gates. Attribution reads From — a peer cannot sign someone else's name on a change by editing its own journal" note for RemoteSource "OpenBlob is the single blob-read door: the sha must match blobRe, and verify re-hashes the bytes whenever the backend is a PutSigner — in direct-upload mode the server never saw the content, so the store is the only thing that could have swapped it. It stops re-hashing only once the object is PROVABLY immutable: both presign doors refuse a key that exists, so every URL for a blob was minted before its first PUT and dies at mint+PresignTTL; past that age the hub is the only writer left. That is what remote.Object.Modified is for" + note for RemoteSource "reassemble is delta sync's whole backward-compatibility story: when blobs/sha is absent (or fails verify), the manifest under manifests/sha names content-defined chunks that concatenate to the blob — spool, hash-verify against the sha requested, BACKFILL blobs/sha (best-effort; a failed write never fails the read), serve. Old clients ask for whole blobs and never learn anything changed; a hostile whole blob that fails verify is HEALED by the backfill when an honest manifest exists. Bounded by maxReassembleBytes (256 MiB) on the manifest's DECLARED sum, with each chunk's copy capped at its declared size — a member-written manifest is the one non-content-addressed object in the key space, so an amplified or oversized one is refused before it can spool" class MoveSource { <> +FilesWithMoves(ctx) files, moveIndex @@ -124,6 +126,7 @@ classDiagram journalKeepsItsOps(ctx, be, key, ops) } note for journalDoor "store.go — the invariant "each device writes only its own journal" is now ENFORCED here, not assumed. The key must be journal/<canonical device id>.jsonl for the device in the request header, that device must already be owned by the caller (DeviceRegistry.OwnerOf) or the caller must be a project admin (the recovery arm) — the old first-writer-claims arm is gone. Every op must pass journal.SafePath + config.ReservedPath on its Path and journal.SafeText on Note/Author/UserName, must name its own owner's account, and the upload must keep every Seq the stored journal already had: append-only, 409 on truncation. Bodies are spooled first, and a blob PUT must hash to the key it claims" + note for journalDoor "Delta sync grew the key space: validStoreKey also accepts chunks/<sha256> (content-addressed, PUT must hash to its key, presigned like blobs incl. refuse-existing) and manifests/<sha256> (keyed by the whole FILE's sha — not its own content hash — so it is never presigned and gets two ingest gates instead: every chunk it names must already EXIST in the store, and the key is WRITE-ONCE — an identical re-put is a 200 no-op so an interrupted push can retry, a different body 409s. Together these make "a manifest exists ⟹ its chunks exist" an invariant every consumer can lean on: the client's push skip-proof, reassemble, and bdrive import)" class Backend { <> diff --git a/cmd/bdrive/delta_migrate_test.go b/cmd/bdrive/delta_migrate_test.go new file mode 100644 index 0000000..553017b --- /dev/null +++ b/cmd/bdrive/delta_migrate_test.go @@ -0,0 +1,207 @@ +package main + +// Row C8 of the delta-sync goal: a chunked-only project — content that exists +// only as chunks/ + manifests/, no whole blobs — survives export → import with +// full fidelity. Missing this silently is the known trap: an archive without +// the new key classes looks complete and has lost the file content. + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "context" + "encoding/json" + "strings" + "testing" + "time" +) + +func TestDelta_Migrate_RoundTrip(t *testing.T) { + ctx := context.Background() + src := openFileBackend(t) + + // A chunked file: two chunks and a manifest keyed by the whole file's + // sha. No blobs/ entry for it — that is the point. Plus a normal small + // blob and a journal, so all four key classes travel together. + c1, c2 := strings.Repeat("x", 300), strings.Repeat("y", 400) + whole := c1 + c2 + put(t, src, "chunks/"+strings.TrimPrefix(blobKey(c1), "blobs/"), c1) + put(t, src, "chunks/"+strings.TrimPrefix(blobKey(c2), "blobs/"), c2) + fileSha := strings.TrimPrefix(blobKey(whole), "blobs/") + man := `{"v":1,"size":700,"chunks":[]}` + put(t, src, "manifests/"+fileSha, man) + put(t, src, blobKey("small"), "small") + put(t, src, "journal/dev-1.jsonl", `{"path":"a.md"}`+"\n") + + var buf bytes.Buffer + blobs, journals, _, err := exportStore(ctx, src, &buf, + exportManifest{Project: "wiki", ExportedAt: time.Now().UTC()}) + if err != nil { + t.Fatal(err) + } + // chunks and manifests count with blobs: 2 chunks + 1 manifest + 1 blob. + if blobs != 4 || journals != 1 { + t.Fatalf("export counted %d blobs, %d journals; want 4, 1", blobs, journals) + } + + gz, err := gzip.NewReader(bytes.NewReader(buf.Bytes())) + if err != nil { + t.Fatal(err) + } + tr := tar.NewReader(gz) + _, first, err := readManifest(tr) + if err != nil { + t.Fatal(err) + } + dst := openFileBackend(t) + if _, _, _, err := importStore(ctx, dst, tr, first, false); err != nil { + t.Fatal(err) + } + + for _, prefix := range []string{"journal/", "blobs/", "chunks/", "manifests/"} { + objs, err := src.List(ctx, prefix) + if err != nil { + t.Fatal(err) + } + if len(objs) == 0 && prefix != "journal/" { + t.Fatalf("source lost its %s keys", prefix) + } + for _, o := range objs { + if got, want := read(t, dst, o.Key), read(t, src, o.Key); got != want { + t.Errorf("%s: imported %q, want %q", o.Key, got, want) + } + } + } +} + +// TestDelta_Import_RefusesIncompleteArchive: an archive whose journal names +// content the archive does not hold is refused, naming the missing path. +// This is what a pre-delta `bdrive export` produces against a delta-sync hub +// (it enumerates only journal/ and blobs/, so chunked large files vanish) — +// importing it used to succeed and silently lose every large file. +func TestDelta_Import_RefusesIncompleteArchive(t *testing.T) { + ctx := context.Background() + var buf bytes.Buffer + gz := gzip.NewWriter(&buf) + tw := tar.NewWriter(gz) + mb, err := json.MarshalIndent(exportManifest{Project: "wiki", ExportedAt: time.Now().UTC()}, "", " ") + if err != nil { + t.Fatal(err) + } + if err := writeTarFile(tw, manifestName, mb); err != nil { + t.Fatal(err) + } + // The journal references a blob for docs/big.bin; the archive holds + // neither blobs/ nor manifests/ for it — the old-export shape. + missing := strings.TrimPrefix(blobKey("the large file the old export dropped"), "blobs/") + op := `{"seq":1,"lamport":1,"device":"dev-1","kind":"put","path":"docs/big.bin","blob":"` + missing + `","size":9000000}` + if err := writeTarFile(tw, "journal/dev-1.jsonl", []byte(op+"\n")); err != nil { + t.Fatal(err) + } + if err := writeTarFile(tw, blobKey("present"), []byte("present")); err != nil { + t.Fatal(err) + } + tw.Close() + gz.Close() + + zr, err := gzip.NewReader(bytes.NewReader(buf.Bytes())) + if err != nil { + t.Fatal(err) + } + tr := tar.NewReader(zr) + _, first, err := readManifest(tr) + if err != nil { + t.Fatal(err) + } + dst := openFileBackend(t) + _, _, _, err = importStore(ctx, dst, tr, first, false) + if err == nil { + t.Fatal("an archive missing referenced content was imported") + } + if !strings.Contains(err.Error(), "docs/big.bin") { + t.Fatalf("refusal does not name the missing path: %v", err) + } +} + +// TestDelta_Import_RefusesManifestNamingAbsentChunks: a manifest in an +// archive must bring its chunks along — one indirection deeper than the +// blob-completeness check, and the same silent-loss shape if missed. +func TestDelta_Import_RefusesManifestNamingAbsentChunks(t *testing.T) { + ctx := context.Background() + var buf bytes.Buffer + gz := gzip.NewWriter(&buf) + tw := tar.NewWriter(gz) + mb, err := json.MarshalIndent(exportManifest{Project: "wiki", ExportedAt: time.Now().UTC()}, "", " ") + if err != nil { + t.Fatal(err) + } + if err := writeTarFile(tw, manifestName, mb); err != nil { + t.Fatal(err) + } + fileSha := strings.TrimPrefix(blobKey("whole file"), "blobs/") + absent := strings.TrimPrefix(blobKey("chunk not in archive"), "blobs/") + man := `{"v":1,"size":20,"chunks":[{"h":"` + absent + `","n":20}]}` + if err := writeTarFile(tw, "manifests/"+fileSha, []byte(man)); err != nil { + t.Fatal(err) + } + op := `{"seq":1,"lamport":1,"device":"dev-1","kind":"put","path":"docs/big.bin","blob":"` + fileSha + `","size":20}` + if err := writeTarFile(tw, "journal/dev-1.jsonl", []byte(op+"\n")); err != nil { + t.Fatal(err) + } + tw.Close() + gz.Close() + + zr, err := gzip.NewReader(bytes.NewReader(buf.Bytes())) + if err != nil { + t.Fatal(err) + } + tr := tar.NewReader(zr) + _, first, err := readManifest(tr) + if err != nil { + t.Fatal(err) + } + dst := openFileBackend(t) + if _, _, _, err := importStore(ctx, dst, tr, first, false); err == nil { + t.Fatal("an archive whose manifest names absent chunks was imported") + } +} + +// TestDelta_Import_RejectsCorruptChunk: a chunk is content-addressed, so +// import applies the same key-equals-hash check blobs get. +func TestDelta_Import_RejectsCorruptChunk(t *testing.T) { + ctx := context.Background() + var buf bytes.Buffer + gz := gzip.NewWriter(&buf) + tw := tar.NewWriter(gz) + mb, err := json.MarshalIndent(exportManifest{Project: "wiki", ExportedAt: time.Now().UTC()}, "", " ") + if err != nil { + t.Fatal(err) + } + if err := writeTarFile(tw, manifestName, mb); err != nil { + t.Fatal(err) + } + // A chunk whose key does not match its content. + key := "chunks/" + strings.TrimPrefix(blobKey("honest"), "blobs/") + if err := writeTarFile(tw, key, []byte("hostile")); err != nil { + t.Fatal(err) + } + if err := writeTarFile(tw, "journal/dev-1.jsonl", []byte(`{"path":"a.md"}`+"\n")); err != nil { + t.Fatal(err) + } + tw.Close() + gz.Close() + + zr, err := gzip.NewReader(bytes.NewReader(buf.Bytes())) + if err != nil { + t.Fatal(err) + } + tr := tar.NewReader(zr) + _, first, err := readManifest(tr) + if err != nil { + t.Fatal(err) + } + dst := openFileBackend(t) + if _, _, _, err := importStore(ctx, dst, tr, first, false); err == nil { + t.Fatal("a chunk that does not hash to its key was imported") + } +} diff --git a/cmd/bdrive/migrate.go b/cmd/bdrive/migrate.go index 30dc39f..f8f60d9 100644 --- a/cmd/bdrive/migrate.go +++ b/cmd/bdrive/migrate.go @@ -17,6 +17,7 @@ import ( "github.com/spf13/cobra" + "github.com/runbear-io/beardrive/internal/journal" "github.com/runbear-io/beardrive/internal/remote" ) @@ -34,6 +35,12 @@ const manifestName = "beardrive-export.json" var ( blobKeyRe = regexp.MustCompile(`^blobs/[0-9a-f]{64}$`) journalKeyRe = regexp.MustCompile(`^journal/[A-Za-z0-9._-]+\.jsonl$`) + // Delta-sync key classes (docs/delta-sync-prd.md). A chunk key is its own + // content hash, checked on import exactly like a blob; a manifest key is + // the whole file's sha, verifiable only by reassembly, so it is carried + // verbatim like a journal. + chunkKeyRe = regexp.MustCompile(`^chunks/[0-9a-f]{64}$`) + manifestKeyRe = regexp.MustCompile(`^manifests/[0-9a-f]{64}$`) ) type exportManifest struct { @@ -102,6 +109,7 @@ bdrive import.`, func importCmd() *cobra.Command { var name string + var allowIncomplete bool c := &cobra.Command{ Use: "import ", Short: "Import an exported project into the hub you're logged into", @@ -159,7 +167,7 @@ bdrive init --project .`, } else if len(existing) > 0 { return fmt.Errorf("project %q (%s) on %s already has content — import needs an empty project (pass --name to create a fresh one)", p.Name, p.ID, settings.Server) } - blobs, journals, size, err := importStore(cmd.Context(), be, tr, first) + blobs, journals, size, err := importStore(cmd.Context(), be, tr, first, allowIncomplete) if err != nil { return err } @@ -174,6 +182,8 @@ bdrive init --project .`, }, } c.Flags().StringVar(&name, "name", "", "project name on the target hub (default: name from the archive)") + c.Flags().BoolVar(&allowIncomplete, "allow-incomplete", false, + "import even when the journal references content the archive does not hold (missing files are listed and stay missing)") c.Flags().Int64Var(&maxImportBlob, "max-blob", maxImportBlob, "largest single file (bytes) an archive member may spool to disk") return c } @@ -190,7 +200,7 @@ func exportStore(ctx context.Context, be remote.Backend, w io.Writer, man export if err := writeTarFile(tw, manifestName, mb); err != nil { return 0, 0, 0, err } - for _, prefix := range []string{"journal/", "blobs/"} { + for _, prefix := range []string{"journal/", "blobs/", "chunks/", "manifests/"} { objs, err := be.List(ctx, prefix) if err != nil { return blobs, journals, size, fmt.Errorf("list %s: %w", prefix, err) @@ -200,7 +210,8 @@ func exportStore(ctx context.Context, be remote.Backend, w io.Writer, man export // file the export's own advice tells the user to pass around. // `bdrive import` refuses a member outside the store layout; // `tar xzf` does not. Same allowlist, applied on the way out. - if !journalKeyRe.MatchString(o.Key) && !blobKeyRe.MatchString(o.Key) { + if !journalKeyRe.MatchString(o.Key) && !blobKeyRe.MatchString(o.Key) && + !chunkKeyRe.MatchString(o.Key) && !manifestKeyRe.MatchString(o.Key) { continue } rc, err := be.Get(ctx, o.Key) @@ -255,7 +266,18 @@ func readManifest(tr *tar.Reader) (exportManifest, *tar.Header, error) { // importStore uploads every archive entry to the backend, verifying blob // content against its content-addressed key. first, when non-nil, is an // already-read header to process before advancing the reader. -func importStore(ctx context.Context, be remote.Backend, tr *tar.Reader, first *tar.Header) (blobs, journals int, size int64, err error) { +func importStore(ctx context.Context, be remote.Backend, tr *tar.Reader, first *tar.Header, allowIncomplete bool) (blobs, journals int, size int64, err error) { + // Every blob a journal references must resolve to content in this same + // archive — blobs/ or manifests/. A pre-delta `bdrive export` + // run against a delta-sync hub enumerates only journal/ and blobs/, so + // its archive silently omits every chunked large file while looking + // complete; importing it succeeded and the files were just gone on the + // destination. Import is the anti-lock-in door — it refuses loudly + // instead. + content := map[string]bool{} // sha → present as blobs/ or manifests/ + referenced := map[string]string{} // sha → a path that references it + chunks := map[string]bool{} // chunk sha → present in the archive + manChunks := map[string][]string{} // manifest sha → chunk shas it names hdr := first for { if hdr == nil { @@ -272,12 +294,62 @@ func importStore(ctx context.Context, be remote.Backend, tr *tar.Reader, first * case key == manifestName || hdr.Typeflag == tar.TypeDir: // skip case journalKeyRe.MatchString(key): - if err := be.Put(ctx, key, tr, hdr.Size); err != nil { + // Spooled rather than streamed so the ops can be read: the + // archive-completeness check below needs every Op.Blob. + jtmp, n, _, err := spoolBlob(tr) + if err != nil { + return blobs, journals, size, err + } + if ops, err := readJournalOps(jtmp); err == nil { + for _, op := range ops { + if op.Kind == journal.KindPut && op.Blob != "" { + referenced[op.Blob] = op.Path + } + } + } + err = be.Put(ctx, key, jtmp, n) + jtmp.Close() + os.Remove(jtmp.Name()) + if err != nil { return blobs, journals, size, fmt.Errorf("put %s: %w", key, err) } journals++ size += hdr.Size - case blobKeyRe.MatchString(key): + case manifestKeyRe.MatchString(key): + // A manifest's key is the whole file's sha — only reassembly can + // verify the CONTENT, so it travels verbatim — but the chunks it + // names must be in this same archive, checked after the loop + // (tar member order is not ours to assume). + mtmp, n, _, err := spoolBlob(tr) + if err != nil { + return blobs, journals, size, err + } + var man struct { + Chunks []struct { + H string `json:"h"` + } `json:"chunks"` + } + sha := strings.TrimPrefix(key, "manifests/") + if derr := json.NewDecoder(mtmp).Decode(&man); derr == nil { + for _, c := range man.Chunks { + manChunks[sha] = append(manChunks[sha], c.H) + } + } + if _, err := mtmp.Seek(0, io.SeekStart); err != nil { + mtmp.Close() + os.Remove(mtmp.Name()) + return blobs, journals, size, err + } + err = be.Put(ctx, key, mtmp, n) + mtmp.Close() + os.Remove(mtmp.Name()) + if err != nil { + return blobs, journals, size, fmt.Errorf("put %s: %w", key, err) + } + content[sha] = true + blobs++ + size += hdr.Size + case blobKeyRe.MatchString(key), chunkKeyRe.MatchString(key): // Spool first, store second: hashing while streaming into Put // notices the mismatch only after the object is already in the // target store, under a content address promising different @@ -288,7 +360,7 @@ func importStore(ctx context.Context, be remote.Backend, tr *tar.Reader, first * if err != nil { return blobs, journals, size, err } - if got != strings.TrimPrefix(key, "blobs/") { + if got != key[strings.IndexByte(key, '/')+1:] { tmp.Close() os.Remove(tmp.Name()) return blobs, journals, size, fmt.Errorf("corrupt archive: %s has content hash %s", key, got) @@ -299,6 +371,11 @@ func importStore(ctx context.Context, be remote.Backend, tr *tar.Reader, first * if err != nil { return blobs, journals, size, fmt.Errorf("put %s: %w", key, err) } + if strings.HasPrefix(key, "blobs/") { + content[strings.TrimPrefix(key, "blobs/")] = true + } else { + chunks[strings.TrimPrefix(key, "chunks/")] = true + } blobs++ size += hdr.Size default: @@ -309,9 +386,54 @@ func importStore(ctx context.Context, be remote.Backend, tr *tar.Reader, first * if journals == 0 { return blobs, journals, size, fmt.Errorf("archive contains no journals — nothing to import") } + // A manifest in the archive must bring its chunks along — a manifest + // whose chunks are absent is exactly as incomplete as a missing blob, + // just one indirection deeper. + for sha, named := range manChunks { + for _, h := range named { + if !chunks[h] { + if allowIncomplete { + fmt.Fprintf(os.Stderr, "warning: manifest %.12s… names chunk %.12s… the archive does not hold\n", sha, h) + continue + } + return blobs, journals, size, fmt.Errorf( + "incomplete archive: manifest %.12s… names chunk %.12s… but the archive holds no content for it — "+ + "re-export with a current bdrive, or pass --allow-incomplete (missing files stay missing)", sha, h) + } + } + } + for sha, path := range referenced { + if !content[sha] { + if allowIncomplete { + fmt.Fprintf(os.Stderr, "warning: archive holds no content for %q (blob %.12s…) — imported without it\n", path, sha) + continue + } + // One dangling reference — an old export against a newer hub, or + // one forged journal line — must not permanently close the + // anti-lock-in door: the refusal names the escape hatch. + return blobs, journals, size, fmt.Errorf( + "incomplete archive: the journal names %q (blob %.12s…) but the archive holds no content for it — "+ + "if this was exported by an older bdrive against a newer hub, re-export with a current bdrive, "+ + "or pass --allow-incomplete to import anyway (missing files stay missing)", path, sha) + } + } return blobs, journals, size, nil } +// readJournalOps parses the ops in a spooled journal body the way every +// device does (journal.Parse), leaving the file rewound for the store. +func readJournalOps(f *os.File) ([]journal.Op, error) { + b, err := io.ReadAll(f) + if err != nil { + return nil, err + } + if _, err := f.Seek(0, io.SeekStart); err != nil { + return nil, err + } + ops, _ := journal.Parse(b) + return ops, nil +} + // maxImportBlob bounds what a single archive member may write to local disk. // Generous for real projects and far below what a compression bomb wants; // --max-blob raises it, so an honest export of a very large file is never diff --git a/cmd/bdrive/migrate_test.go b/cmd/bdrive/migrate_test.go index 311601e..546ed62 100644 --- a/cmd/bdrive/migrate_test.go +++ b/cmd/bdrive/migrate_test.go @@ -73,7 +73,7 @@ func TestExportImportRoundTrip(t *testing.T) { } dst := openFileBackend(t) - blobs, journals, _, err = importStore(ctx, dst, tr, first) + blobs, journals, _, err = importStore(ctx, dst, tr, first, false) if err != nil { t.Fatal(err) } @@ -108,7 +108,7 @@ func TestImportRejectsCorruptBlob(t *testing.T) { gz.Close() dst := openFileBackend(t) - if _, _, _, err := importStore(context.Background(), dst, openTar(t, buf.Bytes()), nil); err == nil || !strings.Contains(err.Error(), "corrupt") { + if _, _, _, err := importStore(context.Background(), dst, openTar(t, buf.Bytes()), nil, false); err == nil || !strings.Contains(err.Error(), "corrupt") { t.Fatalf("err = %v, want corrupt-archive error", err) } } @@ -122,7 +122,7 @@ func TestImportRejectsForeignEntries(t *testing.T) { gz.Close() dst := openFileBackend(t) - if _, _, _, err := importStore(context.Background(), dst, openTar(t, buf.Bytes()), nil); err == nil || !strings.Contains(err.Error(), "unexpected entry") { + if _, _, _, err := importStore(context.Background(), dst, openTar(t, buf.Bytes()), nil, false); err == nil || !strings.Contains(err.Error(), "unexpected entry") { t.Fatalf("err = %v, want unexpected-entry error", err) } } @@ -137,7 +137,7 @@ func TestImportRequiresJournals(t *testing.T) { gz.Close() dst := openFileBackend(t) - if _, _, _, err := importStore(context.Background(), dst, openTar(t, buf.Bytes()), nil); err == nil || !strings.Contains(err.Error(), "no journals") { + if _, _, _, err := importStore(context.Background(), dst, openTar(t, buf.Bytes()), nil, false); err == nil || !strings.Contains(err.Error(), "no journals") { t.Fatalf("err = %v, want no-journals error", err) } } diff --git a/cmd/bdrive/sec_cmds2_test.go b/cmd/bdrive/sec_cmds2_test.go index b4f0e5e..9f60cde 100644 --- a/cmd/bdrive/sec_cmds2_test.go +++ b/cmd/bdrive/sec_cmds2_test.go @@ -391,7 +391,7 @@ func TestSec_Import_ABoundedArchiveCannotSpoolUnboundedBytesToDisk(t *testing.T) } } }() - _, _, _, ierr := importStore(context.Background(), spooled, tr, nil) + _, _, _, ierr := importStore(context.Background(), spooled, tr, nil, false) close(stop) <-watched peakBytes := peak.Load() diff --git a/cmd/bdrive/sec_migrate_test.go b/cmd/bdrive/sec_migrate_test.go index 3a1d0a5..74180e1 100644 --- a/cmd/bdrive/sec_migrate_test.go +++ b/cmd/bdrive/sec_migrate_test.go @@ -112,7 +112,7 @@ func TestSec_Migrate_ArchiveEntryCannotEscapeTheStorePrefix(t *testing.T) { hdr := *h r := secpkgTar(t, true, secpkgEntry(t, &hdr, "planted")) tr, first := secpkgReadArchive(t, r) - _, _, _, _ = importStore(ctx, be, tr, first) + _, _, _, _ = importStore(ctx, be, tr, first, false) for _, p := range secpkgFilesUnder(t, root) { rel, err := filepath.Rel(filepath.Join(root, "store"), p) @@ -157,7 +157,7 @@ func TestSec_Migrate_CorruptBlobNeverLandsInTheTargetStore(t *testing.T) { secpkgEntry(t, &tar.Header{Name: key, Mode: 0o644}, "SUBSTITUTED CONTENT"), ) tr, first := secpkgReadArchive(t, r) - if _, _, _, err := importStore(ctx, be, tr, first); err == nil { + if _, _, _, err := importStore(ctx, be, tr, first, false); err == nil { t.Fatal("importStore accepted a blob whose content does not match its key") } diff --git a/go.mod b/go.mod index 35b1fe1..e1d31d7 100644 --- a/go.mod +++ b/go.mod @@ -12,6 +12,7 @@ require ( github.com/google/uuid v1.6.0 github.com/jackc/pgx/v5 v5.10.0 github.com/mattn/go-isatty v0.0.20 + github.com/restic/chunker v0.5.0 github.com/spf13/cobra v1.10.2 github.com/yuin/goldmark v1.8.2 golang.org/x/crypto v0.52.0 diff --git a/go.sum b/go.sum index 41707b9..dbd7914 100644 --- a/go.sum +++ b/go.sum @@ -150,6 +150,8 @@ github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRI github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= +github.com/restic/chunker v0.5.0 h1:1y+ut0MBduzxODJ298rhQCtESoEpj8v1hTydZlKaE1Y= +github.com/restic/chunker v0.5.0/go.mod h1:z0cH2BejpW636LXw0R/BGyv+Ey8+m9QGiOanDHItzyw= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= diff --git a/internal/syncer/chunks.go b/internal/syncer/chunks.go new file mode 100644 index 0000000..6a3ca28 --- /dev/null +++ b/internal/syncer/chunks.go @@ -0,0 +1,261 @@ +package syncer + +// Delta sync (docs/delta-sync-prd.md): files larger than chunkThreshold move +// as content-defined chunks plus a manifest instead of one whole blob. The +// journal format is untouched — a manifest is keyed by the FILE's sha256, so +// Op.Blob alone locates it. Chunk boundaries are chosen by content (rolling +// hash), so an insertion shifts only the chunk it lands in and every device +// derives the same chunks from the same bytes with no negotiation: the +// remote's "does this key exist" is the whole handshake. +// +// The local volume store is deliberately untouched: blobs stay whole on disk, +// chunking exists only on the wire and in the remote store. + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "os" + + "github.com/restic/chunker" + + "github.com/runbear-io/beardrive/internal/journal" +) + +const ( + // chunkThreshold: files at or below this size take the whole-blob path — + // below it the extra round trips cost more than the bytes they save. + chunkThreshold = 4 << 20 + chunkMin = 256 << 10 + chunkMax = 4 << 20 + // manifestBound caps how much of a manifest is read: it is remote content + // a peer wrote. 8 MiB of JSON is ~100k chunks — far past maxPullBytes' + // worth of file. + manifestBound = 8 << 20 +) + +// chunkPol is the Rabin polynomial every device uses. It must be one fixed +// value for the whole world: two devices chunking the same bytes must produce +// the same chunks, or dedup silently degrades to nothing. +const chunkPol = chunker.Pol(0x3DA3358B4DC173) + +type chunkRef struct { + H string `json:"h"` + N int64 `json:"n"` +} + +type manifest struct { + V int `json:"v"` + Size int64 `json:"size"` + Chunks []chunkRef `json:"chunks"` +} + +// span is one chunk of a local blob: its content hash and where it lives in +// the whole file, so its bytes can be re-read without holding them all. +type span struct { + sha string + off int64 + n int64 +} + +// chunkSpans splits a local blob into content-defined chunks. +func (s *Session) chunkSpans(sum string) ([]span, error) { + f, err := s.Store.OpenBlob(sum) + if err != nil { + return nil, err + } + defer f.Close() + ch := chunker.NewWithBoundaries(f, chunkPol, chunkMin, chunkMax) + buf := make([]byte, chunkMax) + var spans []span + var off int64 + for { + c, err := ch.Next(buf) + if err == io.EOF { + return spans, nil + } + if err != nil { + return nil, err + } + h := sha256.Sum256(c.Data) + spans = append(spans, span{sha: hex.EncodeToString(h[:]), off: off, n: int64(len(c.Data))}) + off += int64(len(c.Data)) + } +} + +// pushChunked uploads one large blob as chunks + manifest, skipping only the +// chunks the remote CONFIRMS it holds. Returns bytes actually uploaded. +// Order is chunks → manifest, so a manifest's presence implies its chunks +// exist (the delta-sync content-before-journal invariant). +// +// The skip proof is one Exists per chunk, nothing cleverer. Three cheaper +// proxies were tried and every one was false or forgeable: "I hold the +// previous version locally" (false for a basis that went up whole — grown +// across the threshold, or pushed by a pre-delta binary); "a manifest exists +// for the basis" (its first write is unverifiable, so a member could plant an +// empty one); "the stored manifest's own hash list" (a member who can read +// the file can publish its true hashes without uploading a byte). There is no +// client-side proof that a chunk is on the remote except asking the remote — +// so ask. ~1 HEAD per unchanged chunk per push of a changed file, still +// three orders of magnitude cheaper than uploading it. +func (s *Session) pushChunked(ctx context.Context, blob string) (int64, error) { + spans, err := s.chunkSpans(blob) + if err != nil { + return 0, err + } + f, err := s.Store.OpenBlob(blob) + if err != nil { + return 0, err + } + defer f.Close() + var uploaded int64 + man := manifest{V: 1} + for _, sp := range spans { + man.Size += sp.n + man.Chunks = append(man.Chunks, chunkRef{H: sp.sha, N: sp.n}) + if ok, err := s.Backend.Exists(ctx, "chunks/"+sp.sha); err == nil && ok { + continue + } + sec := io.NewSectionReader(f, sp.off, sp.n) + if err := s.Backend.Put(ctx, "chunks/"+sp.sha, sec, sp.n); err != nil { + return uploaded, err + } + uploaded += sp.n + } + mb, err := json.Marshal(man) + if err != nil { + return uploaded, err + } + if err := s.Backend.Put(ctx, "manifests/"+blob, bytes.NewReader(mb), int64(len(mb))); err != nil { + // The manifest key is write-once on the hub, so a DIFFERENT body + // already under it (a squatter, or a client with other chunker + // parameters) is refused forever — and a push error here would have + // wedged this device's entire push leg on every future cycle, since + // the journal never goes up and the same job is rebuilt each time. + // The whole blob is content-addressed and always accepted: push it + // and move on. Costs one full upload for that file; costs nothing in + // correctness, and turns a future chunker-parameter change into a + // non-event. + f, ferr := s.Store.OpenBlob(blob) + if ferr != nil { + return uploaded, err + } + fi, serr := f.Stat() + if serr != nil { + f.Close() + return uploaded, err + } + perr := s.Backend.Put(ctx, "blobs/"+blob, f, fi.Size()) + f.Close() + if perr != nil { + return uploaded, err + } + return uploaded + fi.Size(), nil + } + return uploaded + int64(len(mb)), nil +} + +// errNoManifest tells pull's fetch loop to take the whole-blob path: the +// remote has no manifest for this blob (an old writer pushed it whole, or the +// hub will reassemble on Get). +var errNoManifest = errors.New("no manifest") + +// fetchChunked pulls a large blob via its manifest, sourcing unchanged chunks +// from the basis version already in the local store and fetching only the +// rest. The assembled bytes go through PutBlobReader, which files them under +// their COMPUTED hash — so a manifest that does not reassemble to op.Blob +// leaves HasBlob(op.Blob) false and materialize keeps skipping the path, +// exactly like a corrupt whole blob. Returns errNoManifest when the remote +// has no manifest, so the caller can fall back to the whole-blob path. +func (s *Session) fetchChunked(ctx context.Context, op journal.Op, basis string) error { + mrc, err := s.Backend.Get(ctx, "manifests/"+op.Blob) + if err != nil { + return errNoManifest + } + var man manifest + derr := json.NewDecoder(io.LimitReader(mrc, manifestBound)).Decode(&man) + mrc.Close() + if derr != nil { + return fmt.Errorf("manifest for %s: %w", shortSha(op.Blob), derr) + } + var total int64 + for _, c := range man.Chunks { + if !hexSha(c.H) || c.N < 0 { + return fmt.Errorf("manifest for %s names an invalid chunk", shortSha(op.Blob)) + } + total += c.N + } + if total > maxPullBytes { + return errNoManifest // too big to assemble; the whole-blob path owns the ceiling + } + + local := map[string]span{} + var bf *os.File + if basis != "" && s.Store.HasBlob(basis) { + if bspans, err := s.chunkSpans(basis); err == nil { + if bf, err = s.Store.OpenBlob(basis); err == nil { + defer bf.Close() + for _, sp := range bspans { + local[sp.sha] = sp + } + } + } + } + + tmp, err := os.CreateTemp(s.Store.Dir(), ".bdrive-tmp-assemble-*") + if err != nil { + return err + } + defer os.Remove(tmp.Name()) + defer tmp.Close() + for _, c := range man.Chunks { + if sp, ok := local[c.H]; ok && bf != nil { + if _, err := io.Copy(tmp, io.NewSectionReader(bf, sp.off, sp.n)); err != nil { + return err + } + continue + } + crc, err := s.Backend.Get(ctx, "chunks/"+c.H) + if err != nil { + return err + } + h := sha256.New() + n, cerr := io.Copy(io.MultiWriter(tmp, h), io.LimitReader(crc, c.N+1)) + crc.Close() + if cerr != nil { + return cerr + } + if n != c.N || hex.EncodeToString(h.Sum(nil)) != c.H { + return fmt.Errorf("chunk %s of %s does not hash to its key", shortSha(c.H), shortSha(op.Blob)) + } + } + if _, err := tmp.Seek(0, io.SeekStart); err != nil { + return err + } + sum, _, err := s.Store.PutBlobReader(tmp) + if err != nil { + return err + } + if sum != op.Blob { + return fmt.Errorf("%w: manifest for %s assembles to %s", errBlobContent, shortSha(op.Blob), shortSha(sum)) + } + return nil +} + +func hexSha(s string) bool { + if len(s) != 64 { + return false + } + for _, r := range s { + if (r < '0' || r > '9') && (r < 'a' || r > 'f') { + return false + } + } + return true +} + diff --git a/internal/syncer/delta_basis_test.go b/internal/syncer/delta_basis_test.go new file mode 100644 index 0000000..fe4f42f --- /dev/null +++ b/internal/syncer/delta_basis_test.go @@ -0,0 +1,259 @@ +package syncer + +// Regressions from the CTO review: the delta basis is only trustworthy when +// its chunks are actually on the remote, and a bad manifest must never deny a +// file whose correct whole blob exists. The assertion device in these tests +// is always a FRESH third party — a peer that already holds the basis can +// source missing chunks locally and mask exactly these bugs. + +import ( + "bytes" + "context" + "encoding/json" + "math/rand" + "os" + "path/filepath" + "testing" + + "github.com/runbear-io/beardrive/internal/remote" +) + +// TestDelta_Basis_GrownAcrossThresholdUploadsSharedChunks: v1 is below the +// chunking threshold and goes up as a whole blob; v2 grows past it with v1 as +// the local basis. The basis's chunks were NEVER uploaded, so pushChunked +// must not skip them — a fresh device (and the hub, and every read surface) +// has nothing else to assemble from. Before the fix the manifest named +// chunks that did not exist and the file silently never arrived. +func TestDelta_Basis_GrownAcrossThresholdUploadsSharedChunks(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + + rng := rand.New(rand.NewSource(80)) + v1 := make([]byte, 3<<20) // below threshold: whole-blob push, no chunks + rng.Read(v1) + write(t, a.Folder, "grow.bin", string(v1)) + cycle(t, a) + + v2 := append(append([]byte{}, v1...), make([]byte, 3<<20)...) // 6 MiB: chunked, basis = v1 + rng.Read(v2[len(v1):]) + write(t, a.Folder, "grow.bin", string(v2)) + cycle(t, a) + + // The fresh device is the assertion. + b := newDevice(t, "devb", shared) + cycle(t, b) + if got := read(t, b.Folder, "grow.bin"); got != string(v2) { + t.Fatalf("fresh device did not receive the grown file (%d bytes) — basis chunks were skipped without being uploaded", len(got)) + } +} + +// TestDelta_Basis_WholeBlobHistoryUploadsAllChunks is the upgrade shape: the +// device's own journal holds a large put whose content went up as a WHOLE +// blob (what a pre-delta binary did), and the first post-upgrade edit chunks +// with that op as basis. The pre-delta remote state is constructed honestly: +// after v1's (current-code, chunked) push, the whole blob is written and the +// chunks/ and manifests/ trees are GENUINELY REMOVED from the file:// remote +// — a directory, so os.RemoveAll is truthful absence, not an empty object +// that Exists still reports true for (the mistake that made an earlier +// version of this test pass with or without the fix). +func TestDelta_Basis_WholeBlobHistoryUploadsAllChunks(t *testing.T) { + remoteDir := t.TempDir() + shared, err := remote.Open(context.Background(), "file://"+remoteDir) + if err != nil { + t.Fatal(err) + } + defer shared.Close() + + a := newDevice(t, "deva", shared) + rng := rand.New(rand.NewSource(81)) + v1 := make([]byte, 10<<20) + rng.Read(v1) + write(t, a.Folder, "old.bin", string(v1)) + if _, err := a.Cycle(context.Background()); err != nil { + t.Fatal(err) + } + + // Rewrite the remote into the pre-delta shape: whole blob present, + // chunk/manifest trees gone. + sha := shaHex(v1) + f, err := a.Store.OpenBlob(sha) + if err != nil { + t.Fatal(err) + } + if err := shared.Put(context.Background(), "blobs/"+sha, f, int64(len(v1))); err != nil { + t.Fatal(err) + } + f.Close() + for _, d := range []string{"chunks", "manifests"} { + if err := os.RemoveAll(filepath.Join(remoteDir, d)); err != nil { + t.Fatal(err) + } + } + if ok, _ := shared.Exists(context.Background(), "manifests/"+sha); ok { + t.Fatal("test setup failed: the basis manifest still exists") + } + + // The upgraded device edits. Basis = v1 from its own journal; v1 has no + // stored manifest, so no chunk may be skipped. + v2 := append([]byte{}, v1...) + v2[5<<20] ^= 0xff + write(t, a.Folder, "old.bin", string(v2)) + cycle(t, a) + + b := newDevice(t, "devb", shared) + cycle(t, b) + if got := read(t, b.Folder, "old.bin"); got != string(v2) { + t.Fatalf("fresh device did not receive the post-upgrade edit (%d bytes)", len(got)) + } +} + +// TestDelta_Basis_SquattedManifestCannotSkipChunks (CTO H3): the first write +// of a manifest key is unverifiable, so a member can plant one under a +// whole-pushed blob's public sha BEFORE that file ever grows. The planted +// manifest's chunk list is what the pusher trusts — so it must be the STORED +// list that feeds the skip set, never a local re-chunk of the basis: a +// squatted empty manifest then skips nothing, and the file still reaches a +// fresh device. +func TestDelta_Basis_SquattedManifestCannotSkipChunks(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + rng := rand.New(rand.NewSource(83)) + v1 := make([]byte, 3<<20) // below threshold: whole-blob push, no manifest + rng.Read(v1) + write(t, a.Folder, "grow.bin", string(v1)) + cycle(t, a) + + // Mallory squats the basis's manifest slot: empty chunk list, accepted + // because the key was never written. + if err := shared.Put(context.Background(), "manifests/"+shaHex(v1), + bytes.NewReader([]byte(`{"v":1,"size":0,"chunks":[]}`)), 0); err != nil { + t.Fatal(err) + } + + // The file grows past the threshold; v1 is the basis. + v2 := append(append([]byte{}, v1...), make([]byte, 3<<20)...) + rng.Read(v2[len(v1):]) + write(t, a.Folder, "grow.bin", string(v2)) + cycle(t, a) + + b := newDevice(t, "devb", shared) + cycle(t, b) + if got := read(t, b.Folder, "grow.bin"); got != string(v2) { + t.Fatalf("a squatted basis manifest made push skip chunks it never uploaded (%d bytes arrived)", len(got)) + } +} + +// TestDelta_Basis_TruthfulSquatCannotSkipChunks (CTO H6, the third recurrence +// of the same root cause): a member who can READ the file can publish its +// true chunk hashes without uploading a byte — so a manifest that is +// hash-accurate is still not proof of upload. The pusher's only valid skip +// proof is asking the remote per chunk; with it, the squat costs the +// attacker nothing and gains them nothing. +func TestDelta_Basis_TruthfulSquatCannotSkipChunks(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + rng := rand.New(rand.NewSource(85)) + v1 := make([]byte, 3<<20) // below threshold: whole-blob push, empty manifest slot + rng.Read(v1) + write(t, a.Folder, "grow.bin", string(v1)) + cycle(t, a) + + // Mallory chunks the bytes she can read and publishes the REAL hashes — + // uploading no chunks at all. + spans, err := a.chunkSpans(shaHex(v1)) + if err != nil { + t.Fatal(err) + } + man := manifest{V: 1} + for _, sp := range spans { + man.Size += sp.n + man.Chunks = append(man.Chunks, chunkRef{H: sp.sha, N: sp.n}) + } + mb, _ := json.Marshal(man) + if err := shared.Put(context.Background(), "manifests/"+shaHex(v1), bytes.NewReader(mb), int64(len(mb))); err != nil { + t.Fatal(err) + } + + v2 := append(append([]byte{}, v1...), make([]byte, 3<<20)...) + rng.Read(v2[len(v1):]) + write(t, a.Folder, "grow.bin", string(v2)) + cycle(t, a) + + b := newDevice(t, "devb", shared) + cycle(t, b) + if got := read(t, b.Folder, "grow.bin"); got != string(v2) { + t.Fatalf("a truthful squatted manifest made push skip chunks that were never uploaded (%d bytes arrived)", len(got)) + } +} + +// TestDelta_Pull_BadManifestFallsBackToWholeBlob (CTO H1): a well-formed but +// wrong manifest must not deny a file whose correct whole blob exists — any +// fetchChunked failure falls through to the independently-verified blob path. +func TestDelta_Pull_BadManifestFallsBackToWholeBlob(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + rng := rand.New(rand.NewSource(82)) + content := make([]byte, 8<<20) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + sha := shaHex(content) + + // Ensure the correct whole blob exists remotely (the hub's backfill + // produces exactly this state), then corrupt the manifest. + f, err := a.Store.OpenBlob(sha) + if err != nil { + t.Fatal(err) + } + if err := shared.Put(context.Background(), "blobs/"+sha, f, int64(len(content))); err != nil { + t.Fatal(err) + } + f.Close() + if err := shared.Put(context.Background(), "manifests/"+sha, + bytes.NewReader([]byte(`{"v":1,"size":1,"chunks":[{"h":"`+shaOfString("nope")+`","n":1}]}`)), 0); err != nil { + t.Fatal(err) + } + + b := newDevice(t, "devb", shared) + cycle(t, b) + if got := read(t, b.Folder, "big.bin"); got != string(content) { + t.Fatalf("a bad manifest denied a file whose correct whole blob exists (%d bytes)", len(got)) + } +} + +func shaOfString(s string) string { return shaHex([]byte(s)) } + +// TestDelta_Push_ManifestRefusalFallsBackToWholeBlob (CTO H4): the manifest +// key is write-once on the hub, so a squatter (or a client with different +// chunker parameters) can make every manifest PUT for a given blob fail +// forever. That refusal must cost one whole-blob upload for that file — not +// the device's entire push leg, which is what a returned error here caused: +// the journal never went up, the same job was rebuilt every cycle, and +// nothing the device ever authored pushed again. +func TestDelta_Push_ManifestRefusalFallsBackToWholeBlob(t *testing.T) { + shared := sharedRemote(t) + fb := &failingBackend{Backend: shared, failPrefix: "manifests/"} // refused forever + a := newDevice(t, "deva", fb) + + rng := rand.New(rand.NewSource(84)) + content := make([]byte, 8<<20) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + write(t, a.Folder, "note.md", "the rest of the push leg must survive") + res, err := a.Cycle(context.Background()) + if err != nil { + t.Fatal(err) + } + if res.Offline { + t.Fatalf("a refused manifest wedged the push: %v", res.OfflineErr) + } + + b := newDevice(t, "devb", shared) + cycle(t, b) + if got := read(t, b.Folder, "big.bin"); got != string(content) { + t.Fatalf("whole-blob fallback did not deliver the file (%d bytes)", len(got)) + } + if got := read(t, b.Folder, "note.md"); got != "the rest of the push leg must survive" { + t.Fatal("the small file behind the refused manifest never pushed") + } +} diff --git a/internal/syncer/delta_correct_test.go b/internal/syncer/delta_correct_test.go new file mode 100644 index 0000000..282dd4b --- /dev/null +++ b/internal/syncer/delta_correct_test.go @@ -0,0 +1,279 @@ +package syncer + +// Correctness rows of the delta-sync goal (.claude/delta-sync-goal.md): +// chunked transport must change nothing about what devices converge to. + +import ( + "bytes" + "context" + "encoding/json" + "io" + "math/rand" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/runbear-io/beardrive/internal/journal" + "github.com/runbear-io/beardrive/internal/remote" +) + +// failingBackend fails Puts of keys matching a prefix until allowed, so tests +// can kill a push between its ordered stages. +type failingBackend struct { + remote.Backend + failPrefix string + allowed bool +} + +func (f *failingBackend) Put(ctx context.Context, key string, r io.Reader, size int64) error { + if !f.allowed && strings.HasPrefix(key, f.failPrefix) { + return io.ErrUnexpectedEOF + } + return f.Backend.Put(ctx, key, r, size) +} + +// TestDelta_Order_ChunksBeforeManifest (row C5): a push that dies during +// chunk upload leaves no broken op — neither manifest nor journal was +// written, so a peer sees nothing, and the next cycle heals. (A failure at +// the MANIFEST stage is no longer an interruption at all: the hub's +// write-once refusal there falls back to a whole-blob push — see +// TestDelta_Push_ManifestRefusalFallsBackToWholeBlob — so the chunk stage is +// where a mid-push death is modeled.) +func TestDelta_Order_ChunksBeforeManifest(t *testing.T) { + shared := sharedRemote(t) + fb := &failingBackend{Backend: shared, failPrefix: "chunks/"} + a := newDevice(t, "deva", fb) + b := newDevice(t, "devb", shared) + + rng := rand.New(rand.NewSource(48)) + content := make([]byte, 8<<20) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + res, err := a.Cycle(context.Background()) + if err != nil { + t.Fatal(err) + } + if !res.Offline { + t.Fatal("a push that could not finish must degrade to Offline, not succeed") + } + // Peer sees nothing: the journal (last in the order) was never pushed. + cycle(t, b) + if _, err := os.Stat(filepath.Join(b.Folder, "big.bin")); err == nil { + t.Fatal("peer materialized a file whose push never completed") + } + // Next cycle heals. + fb.allowed = true + cycle(t, a) + cycle(t, b) + if got := read(t, b.Folder, "big.bin"); got != string(content) { + t.Fatal("did not converge after the healing cycle") + } +} + +// TestDelta_Order_ManifestBeforeJournal (row C6): killed between manifest and +// journal — same story one stage later. +func TestDelta_Order_ManifestBeforeJournal(t *testing.T) { + shared := sharedRemote(t) + fb := &failingBackend{Backend: shared, failPrefix: "journal/"} + a := newDevice(t, "deva", fb) + b := newDevice(t, "devb", shared) + + rng := rand.New(rand.NewSource(49)) + content := make([]byte, 8<<20) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + res, err := a.Cycle(context.Background()) + if err != nil { + t.Fatal(err) + } + if !res.Offline { + t.Fatal("journal put failed; cycle must be Offline") + } + cycle(t, b) + if _, err := os.Stat(filepath.Join(b.Folder, "big.bin")); err == nil { + t.Fatal("peer materialized a file whose journal was never pushed") + } + fb.allowed = true + cycle(t, a) + cycle(t, b) + if got := read(t, b.Folder, "big.bin"); got != string(content) { + t.Fatal("did not converge after the healing cycle") + } +} + +// TestDelta_Converge_ThreeDeviceConflict (row C2): concurrent offline edits of +// a large chunked file resolve exactly like small files — one winner at the +// path, the loser preserved as a conflict copy, all three devices converged. +func TestDelta_Converge_ThreeDeviceConflict(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + b := newDevice(t, "devb", shared) + c := newDevice(t, "devc", shared) + + rng := rand.New(rand.NewSource(50)) + content := make([]byte, 8<<20) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + cycle(t, b) + cycle(t, c) + + // Two devices edit different regions while "offline" (before syncing). + ca := append([]byte{}, content...) + ca[100] ^= 0xff + cb := append([]byte{}, content...) + cb[len(cb)-100] ^= 0xff + write(t, a.Folder, "big.bin", string(ca)) + write(t, b.Folder, "big.bin", string(cb)) + + cycle(t, a) + cycle(t, b) + cycle(t, a) + cycle(t, b) + cycle(t, c) + + va := read(t, a.Folder, "big.bin") + vb := read(t, b.Folder, "big.bin") + vc := read(t, c.Folder, "big.bin") + if va != vb || vb != vc { + t.Fatal("devices did not converge on the same winner") + } + if va != string(ca) && va != string(cb) { + t.Fatal("winner is neither edit") + } + loser := string(ca) + if va == string(ca) { + loser = string(cb) + } + found := false + for _, folder := range []string{a.Folder, b.Folder, c.Folder} { + entries, _ := os.ReadDir(folder) + for _, e := range entries { + if strings.Contains(e.Name(), ".bdrive-conflict-") && + read(t, folder, e.Name()) == loser { + found = true + } + } + } + if !found { + t.Fatal("losing edit was not preserved as a conflict copy") + } +} + +// TestDelta_Pull_MissingChunkDoesNotStall (row C7): one op whose chunk is +// unfetchable must not stop complete ops behind it from landing — the posture +// the blob path already has. A corrupt chunk (bytes that do not hash to the +// key) is refused exactly like a missing one. +func TestDelta_Pull_MissingChunkDoesNotStall(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + b := newDevice(t, "devb", shared) + + rng := rand.New(rand.NewSource(51)) + content := make([]byte, 8<<20) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + write(t, a.Folder, "after.md", "small file behind the big one") + cycle(t, a) + + objs, err := shared.List(context.Background(), "chunks/") + if err != nil || len(objs) == 0 { + t.Fatalf("no chunks on remote: %v", err) + } + badChunk := objs[0].Key + if err := shared.Put(context.Background(), badChunk, bytes.NewReader([]byte("wrong")), 5); err != nil { + t.Fatal(err) + } + + cycle(t, b) // must not stall: after.md lands even though big.bin cannot + if got := read(t, b.Folder, "after.md"); got != "small file behind the big one" { + t.Fatal("an unfetchable chunk stalled the op behind it") + } + if _, err := os.Stat(filepath.Join(b.Folder, "big.bin")); err == nil { + t.Fatal("big.bin materialized despite its chunk being corrupt") + } + + // Heal the chunk from the writer's own blob. Content is refetched when + // the path's journal grows (the same trigger the whole-blob path has), so + // the writer edits the file once more and the peer converges on that. + chunkSha := strings.TrimPrefix(badChunk, "chunks/") + spans, err := a.chunkSpans(shaHex(content)) + if err != nil { + t.Fatal(err) + } + for _, sp := range spans { + if sp.sha != chunkSha { + continue + } + f, err := a.Store.OpenBlob(shaHex(content)) + if err != nil { + t.Fatal(err) + } + err = shared.Put(context.Background(), badChunk, io.NewSectionReader(f, sp.off, sp.n), sp.n) + f.Close() + if err != nil { + t.Fatal(err) + } + } + content[0] ^= 0xff + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + cycle(t, b) + if got := read(t, b.Folder, "big.bin"); got != string(content) { + t.Fatal("did not converge after the chunk was restored and the path re-journaled") + } +} + +// TestDelta_Journal_ByteIdentical (row C1, in-process half): a chunked file's +// journal op carries exactly the same JSON fields as a small file's — no new +// keys, nothing removed. The cross-binary half is E2/E3 (merge-base binary). +func TestDelta_Journal_ByteIdentical(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + rng := rand.New(rand.NewSource(52)) + content := make([]byte, 8<<20) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + write(t, a.Folder, "small.md", "tiny") + cycle(t, a) + + ops, err := a.Store.DeviceOps("deva") + if err != nil || len(ops) != 2 { + t.Fatalf("ops = %d, %v", len(ops), err) + } + keysOf := func(op journal.Op) map[string]bool { + raw, err := json.Marshal(op) + if err != nil { + t.Fatal(err) + } + var m map[string]any + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + ks := map[string]bool{} + for k := range m { + ks[k] = true + } + return ks + } + var big, small journal.Op + for _, op := range ops { + if op.Path == "big.bin" { + big = op + } else { + small = op + } + } + bk, sk := keysOf(big), keysOf(small) + for k := range bk { + if !sk[k] { + t.Errorf("chunked op carries a field small ops do not: %q", k) + } + } + for k := range sk { + if !bk[k] { + t.Errorf("chunked op is missing field %q", k) + } + } +} diff --git a/internal/syncer/delta_hostile_test.go b/internal/syncer/delta_hostile_test.go new file mode 100644 index 0000000..37d9b0a --- /dev/null +++ b/internal/syncer/delta_hostile_test.go @@ -0,0 +1,84 @@ +package syncer + +// Row C3: assembled content is verified against Op.Blob before it enters the +// blob store. A manifest is remote content a peer (or hub) wrote; every chunk +// in it can be individually honest while the assembly is not the file the op +// names. The wrong bytes must never land under the op's sha and the path must +// never materialize from them. + +import ( + "bytes" + "context" + "encoding/json" + "math/rand" + "os" + "path/filepath" + "testing" +) + +func TestDelta_Pull_RejectsMismatch(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + b := newDevice(t, "devb", shared) + + rng := rand.New(rand.NewSource(53)) + content := make([]byte, 8<<20) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + sha := shaHex(content) + + // Hostile rewrite: the manifest now lists the first chunk twice. Each + // chunk is individually valid (it hashes to its key), but the assembly is + // not the file the op names. + spans, err := a.chunkSpans(sha) + if err != nil || len(spans) < 2 { + t.Fatalf("spans = %d, %v", len(spans), err) + } + man := manifest{V: 1, Size: spans[0].n * 2, Chunks: []chunkRef{ + {H: spans[0].sha, N: spans[0].n}, + {H: spans[0].sha, N: spans[0].n}, + }} + mb, _ := json.Marshal(man) + if err := shared.Put(context.Background(), "manifests/"+sha, bytes.NewReader(mb), int64(len(mb))); err != nil { + t.Fatal(err) + } + + res, err := b.Cycle(context.Background()) + if err != nil { + t.Fatal(err) + } + // The contradiction is surfaced (errBlobContent → Offline), the wrong + // bytes never land under the op's sha, and the path never materializes. + if !res.Offline { + t.Fatal("a manifest assembling to the wrong content must surface as Offline") + } + if res.OfflineErr == nil { + t.Fatal("no error recorded for the mismatch") + } + if b.Store.HasBlob(sha) { + t.Fatal("wrong bytes were filed under the op's sha") + } + if _, err := os.Stat(filepath.Join(b.Folder, "big.bin")); err == nil { + t.Fatal("path materialized from a mismatching assembly") + } +} + +// TestDelta_Ceiling_LargeFileMaterializes pins maxPullBytes at its raised +// value: a 40 MiB file — over the old 32 MiB ceiling that silently kept large +// files off every receiving device — must land on a peer. If this fails after +// a bound change, the ceiling regressed below what the product now promises. +func TestDelta_Ceiling_LargeFileMaterializes(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + b := newDevice(t, "devb", shared) + + content := make([]byte, 40<<20) + rand.New(rand.NewSource(54)).Read(content) + write(t, a.Folder, "video.bin", string(content)) + cycle(t, a) + cycle(t, b) + if got := read(t, b.Folder, "video.bin"); got != string(content) { + t.Fatalf("40 MiB file did not materialize on the peer (%d bytes)", len(got)) + } +} diff --git a/internal/syncer/delta_test.go b/internal/syncer/delta_test.go new file mode 100644 index 0000000..03b6240 --- /dev/null +++ b/internal/syncer/delta_test.go @@ -0,0 +1,333 @@ +package syncer + +// The delta-sync counting harness (.claude/delta-sync-goal.md, row G1). +// countingBackend is the instrument every transport acceptance criterion +// reads: it wraps any remote.Backend and records the bytes that actually +// cross the wire in each direction. It counts what the backend consumed and +// what the caller read — not declared sizes — so a backend that short-reads +// or a caller that abandons a stream is counted honestly. + +import ( + "bytes" + "compress/gzip" + "context" + "crypto/sha256" + "encoding/hex" + "io" + "math/rand" + "os" + "path/filepath" + "strings" + "sync/atomic" + "testing" + + "github.com/runbear-io/beardrive/internal/remote" +) + +// countingBackend records bytes crossing the wire in both directions. +type countingBackend struct { + remote.Backend + put, get atomic.Int64 +} + +func (c *countingBackend) Put(ctx context.Context, key string, r io.Reader, size int64) error { + return c.Backend.Put(ctx, key, &countReader{r: r, n: &c.put}, size) +} + +func (c *countingBackend) Get(ctx context.Context, key string) (io.ReadCloser, error) { + rc, err := c.Backend.Get(ctx, key) + if err != nil { + return nil, err + } + return &countReadCloser{rc: rc, n: &c.get}, nil +} + +type countReader struct { + r io.Reader + n *atomic.Int64 +} + +func (c *countReader) Read(p []byte) (int, error) { + n, err := c.r.Read(p) + c.n.Add(int64(n)) + return n, err +} + +type countReadCloser struct { + rc io.ReadCloser + n *atomic.Int64 +} + +func (c *countReadCloser) Read(p []byte) (int, error) { + n, err := c.rc.Read(p) + c.n.Add(int64(n)) + return n, err +} + +func (c *countReadCloser) Close() error { return c.rc.Close() } + +// counting wraps a backend for one device's point of view. Two devices in one +// test share the underlying store but each get their own counters. +func counting(be remote.Backend) *countingBackend { + return &countingBackend{Backend: be} +} + +// TestDelta_Harness_CountsBytes proves the instrument itself: bytes counted +// on Put and Get equal the bytes that actually moved, both when driving the +// backend directly and when dropped into a real device's sync cycle. +func TestDelta_Harness_CountsBytes(t *testing.T) { + be := counting(sharedRemote(t)) + payload := bytes.Repeat([]byte("delta"), 1000) // 5000 bytes, not a round number of reads + ctx := context.Background() + + if err := be.Put(ctx, "blobs/"+strings.Repeat("e", 64), bytes.NewReader(payload), int64(len(payload))); err != nil { + t.Fatal(err) + } + if got := be.put.Load(); got != int64(len(payload)) { + t.Fatalf("put counted %d bytes, want %d", got, len(payload)) + } + + rc, err := be.Get(ctx, "blobs/"+strings.Repeat("e", 64)) + if err != nil { + t.Fatal(err) + } + data, err := io.ReadAll(rc) + rc.Close() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(data, payload) { + t.Fatal("payload corrupted through the counting wrapper") + } + if got := be.get.Load(); got != int64(len(payload)) { + t.Fatalf("get counted %d bytes, want %d", got, len(payload)) + } + + // And through a real cycle: the wrapper must drop into newDevice + // unchanged. Device A pushes a file of known size; its put counter sees + // at least the content and at most content plus a small journal. Device B + // pulls it; its get counter sees at least the content. + shared := sharedRemote(t) + a := newDevice(t, "deva", counting(shared)) + b := newDevice(t, "devb", counting(shared)) + abe := a.Backend.(*countingBackend) + bbe := b.Backend.(*countingBackend) + + content := bytes.Repeat([]byte("x"), 100_000) + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + if got := abe.put.Load(); got < int64(len(content)) || got > int64(len(content))+4096 { + t.Fatalf("cycle put counted %d bytes, want ~%d (content + small journal)", got, len(content)) + } + cycle(t, b) + if got := bbe.get.Load(); got < int64(len(content)) { + t.Fatalf("cycle get counted %d bytes, want >= %d", got, len(content)) + } + if got := read(t, b.Folder, "big.bin"); got != string(content) { + t.Fatalf("device B did not converge: got %d bytes", len(got)) + } +} + +// TestDelta_Baseline_WholeFileCost is the before-and-after in one place (row +// G2, characterization). Before delta sync a 1-byte edit to a 20 MiB file +// pushed the whole file again: measured 20,972,098 bytes on 2026-08-12. With +// content-defined chunking it pushes only the chunk containing the edit plus +// the manifest: measured 2,050,572 bytes, bounded here at 5 MiB (one max-size +// chunk + slack). Never delete this test; update the bound if the transport +// changes again. +// +// Content is seeded-pseudorandom, not a repeated pattern, so the bound stays +// meaningful: a repetitive file would dedupe to nothing and pass vacuously. +func TestDelta_Baseline_WholeFileCost(t *testing.T) { + const size = 20 << 20 + be := counting(sharedRemote(t)) + a := newDevice(t, "deva", be) + + rng := rand.New(rand.NewSource(42)) + content := make([]byte, size) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + base := be.put.Load() + if base < size { + t.Fatalf("initial push counted %d bytes, want >= %d (every chunk is new)", base, size) + } + + content[size/2] ^= 0xff + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + edit := be.put.Load() - base + + t.Logf("1-byte edit to a %d MiB file pushed %d bytes (was 20972098 pre-delta)", size>>20, edit) + if edit > 5<<20 { + t.Fatalf("1-byte edit pushed %d bytes, want < 5 MiB (one changed chunk + manifest)", edit) + } +} + +// TestDelta_Push_FrontInsertion (row D4) is the discriminator: fixed-size +// blocking passes the mid-edit and append cases and fails THIS one, because an +// insertion at the front shifts every fixed boundary. Content-defined +// boundaries realign, so only the first chunk (and the manifest) changes. +func TestDelta_Push_FrontInsertion(t *testing.T) { + const size = 20 << 20 + be := counting(sharedRemote(t)) + a := newDevice(t, "deva", be) + + rng := rand.New(rand.NewSource(43)) + content := make([]byte, size) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + base := be.put.Load() + + write(t, a.Folder, "big.bin", "inserted at the very front|"+string(content)) + cycle(t, a) + edit := be.put.Load() - base + t.Logf("front insertion pushed %d bytes", edit) + if edit > 5<<20 { + t.Fatalf("front insertion pushed %d bytes, want < 5 MiB — boundaries are not content-defined", edit) + } +} + +// TestDelta_Push_Append (row D3). +func TestDelta_Push_Append(t *testing.T) { + const size = 20 << 20 + be := counting(sharedRemote(t)) + a := newDevice(t, "deva", be) + + rng := rand.New(rand.NewSource(44)) + content := make([]byte, size) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + base := be.put.Load() + + write(t, a.Folder, "big.bin", string(content)+"appended tail") + cycle(t, a) + edit := be.put.Load() - base + t.Logf("append pushed %d bytes", edit) + if edit > 5<<20 { + t.Fatalf("append pushed %d bytes, want < 5 MiB", edit) + } +} + +// TestDelta_Pull_SmallEdit (row D2): a peer already holding the basis version +// pulls a 1-byte edit for a small multiple of the chunk size, sourcing every +// unchanged chunk from the blob it already has. +func TestDelta_Pull_SmallEdit(t *testing.T) { + const size = 20 << 20 + shared := sharedRemote(t) + a := newDevice(t, "deva", counting(shared)) + b := newDevice(t, "devb", counting(shared)) + bbe := b.Backend.(*countingBackend) + + rng := rand.New(rand.NewSource(45)) + content := make([]byte, size) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + cycle(t, b) // b now holds the basis + if got := read(t, b.Folder, "big.bin"); got != string(content) { + t.Fatal("basis did not converge") + } + base := bbe.get.Load() + + content[size/3] ^= 0xff + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + cycle(t, b) + pulled := bbe.get.Load() - base + if got := read(t, b.Folder, "big.bin"); got != string(content) { + t.Fatal("edit did not converge") + } + t.Logf("peer with basis pulled %d bytes for a 1-byte edit", pulled) + if pulled > 5<<20 { + t.Fatalf("pull cost %d bytes, want < 5 MiB", pulled) + } +} + +// TestDelta_Threshold_SmallFileUnchanged (row D5): files at or below the +// threshold never write a chunks/ or manifests/ key — the whole-blob path is +// byte-for-byte what it was. +func TestDelta_Threshold_SmallFileUnchanged(t *testing.T) { + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + small := make([]byte, 4<<20) // exactly the threshold: NOT chunked + rand.New(rand.NewSource(46)).Read(small) + write(t, a.Folder, "small.bin", string(small)) + write(t, a.Folder, "note.md", "tiny") + cycle(t, a) + + for _, prefix := range []string{"chunks/", "manifests/"} { + objs, err := shared.List(context.Background(), prefix) + if err != nil { + t.Fatal(err) + } + if len(objs) != 0 { + t.Fatalf("small files wrote %s keys: %v", prefix, objs) + } + } + if ok, _ := shared.Exists(context.Background(), "blobs/"+shaHex(small)); !ok { + t.Fatal("small file's whole blob was not uploaded") + } +} + +// TestDelta_Pull_ColdPathUnchanged (row D6): a device with no basis pulls a +// chunked file — via the manifest when one exists; and a small file stays one +// whole-blob GET. +func TestDelta_Pull_ColdPathUnchanged(t *testing.T) { + const size = 20 << 20 + shared := sharedRemote(t) + a := newDevice(t, "deva", shared) + rng := rand.New(rand.NewSource(47)) + content := make([]byte, size) + rng.Read(content) + write(t, a.Folder, "big.bin", string(content)) + cycle(t, a) + + // A cold device converges on chunked-only content. + c := newDevice(t, "devc", counting(shared)) + cycle(t, c) + if got := read(t, c.Folder, "big.bin"); got != string(content) { + t.Fatalf("cold pull did not converge: %d bytes", len(got)) + } +} + +func shaHex(b []byte) string { + h := sha256.Sum256(b) + return hex.EncodeToString(h[:]) +} + +// TestDelta_Gzip_TextCorpusRatio measures what transport compression alone +// would save (row G4) — the cheaper rung the PRD says to consider before +// chunking. The corpus is this package's own Go source: real prose-and-code +// text of the kind BearDrive actually syncs, present on every checkout. +// The assertion is a floor (gzip must at least halve it); the measured ratio +// is the number G5 reads, via t.Logf. +func TestDelta_Gzip_TextCorpusRatio(t *testing.T) { + files, err := filepath.Glob("*.go") + if err != nil || len(files) == 0 { + t.Fatalf("no corpus: %v", err) + } + var raw bytes.Buffer + for _, f := range files { + b, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) + } + raw.Write(b) + } + var packed bytes.Buffer + zw := gzip.NewWriter(&packed) + if _, err := zw.Write(raw.Bytes()); err != nil { + t.Fatal(err) + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + ratio := float64(raw.Len()) / float64(packed.Len()) + t.Logf("gzip on %d files, %d bytes of Go/text: %d bytes compressed — %.1fx", len(files), raw.Len(), packed.Len(), ratio) + if packed.Len()*2 > raw.Len() { + t.Fatalf("gzip achieved only %d -> %d on a text corpus; measurement or corpus is wrong", raw.Len(), packed.Len()) + } +} diff --git a/internal/syncer/sec_audit5_test.go b/internal/syncer/sec_audit5_test.go index 44831fa..3d39c6f 100644 --- a/internal/syncer/sec_audit5_test.go +++ b/internal/syncer/sec_audit5_test.go @@ -179,7 +179,7 @@ func TestSec_Pull_AHubChosenJournalKeyCannotEscapeTheVolumeDir(t *testing.T) { // pull's error is not the assertion: the filesystem is. A hostile key that // merely fails to be written is still a hub choosing a path on this disk. - newOps, _, perr := victim.pull(context.Background()) + newOps, _, perr := victim.pull(context.Background(), nil) if len(newOps) == 0 { t.Fatalf("control: the ordinary journal in the same listing was not pulled (%v), so nothing is proven", perr) } diff --git a/internal/syncer/sec_fixes10_test.go b/internal/syncer/sec_fixes10_test.go index 5f48de4..2f4f9b6 100644 --- a/internal/syncer/sec_fixes10_test.go +++ b/internal/syncer/sec_fixes10_test.go @@ -44,8 +44,8 @@ import ( // gone wrong, which makes "the hub decides how much of your disk this costs" // the wrong answer here specifically. func TestSec_HostileHub_ARestoreCannotBeSizedByTheHub(t *testing.T) { - const willing = 72 << 20 // what the hub is prepared to write - const ceiling = 64 << 20 // maxPullBytes is 32<<20; twice that is generous + const willing = 2*maxPullBytes + (40 << 20) // more than the generous ceiling: proves truncation + const ceiling = 2 * maxPullBytes // twice the cap is generous: transport buffering, not the bound, owns the slack _, hostile := sechostPeer(t, map[string]string{"notes/real.md": "small file, honestly"}) hostile.onBody = func(key string, body []byte, w http.ResponseWriter) bool { diff --git a/internal/syncer/sec_hostile_test.go b/internal/syncer/sec_hostile_test.go index 87dd176..aa8e1b9 100644 --- a/internal/syncer/sec_hostile_test.go +++ b/internal/syncer/sec_hostile_test.go @@ -219,9 +219,9 @@ func TestSec_HostileHub_OneUnusableListedKeyCannotHideEveryPeer(t *testing.T) { // the daemon buffers") is not met when the hub is the peer: there is no absolute // ceiling anywhere on the path, so one listing entry sizes the allocation. func TestSec_HostileHub_ADeclaredJournalSizeCannotChooseTheDeviceAllocation(t *testing.T) { - const declared = 512 << 20 // what the hub says the journal is - const willing = 72 << 20 // how much it is prepared to actually write here - const ceiling = 64 << 20 // an absolute cap this device ought to have + const declared = 512 << 20 // what the hub says the journal is + const willing = 2*maxPullBytes + (40 << 20) // more than the generous ceiling: proves truncation + const ceiling = 2 * maxPullBytes // twice the cap is generous: transport buffering, not the bound, owns the slack _, hostile := sechostPeer(t, map[string]string{"notes/real.md": "hi"}) hostile.onList = func(objs []remote.Object) []remote.Object { @@ -254,8 +254,8 @@ func TestSec_HostileHub_ADeclaredJournalSizeCannotChooseTheDeviceAllocation(t *t // bytes to the user's disk as it names — on every cycle, for every op. func TestSec_HostileHub_ADeclaredBlobSizeCannotFillTheDisk(t *testing.T) { const declared = 512 << 20 - const willing = 72 << 20 - const ceiling = 64 << 20 + const willing = 2*maxPullBytes + (40 << 20) + const ceiling = 2 * maxPullBytes _, hostile := sechostPeer(t, map[string]string{"notes/real.md": "small file, honestly"}) hostile.onBody = func(key string, body []byte, w http.ResponseWriter) bool { diff --git a/internal/syncer/syncer.go b/internal/syncer/syncer.go index e694864..ffa352a 100644 --- a/internal/syncer/syncer.go +++ b/internal/syncer/syncer.go @@ -310,7 +310,7 @@ func (s *Session) Cycle(ctx context.Context) (*Result, error) { blocked := false var pulled, gone []journal.Op if s.Backend != nil { - pulled, gone, err = s.pull(ctx) + pulled, gone, err = s.pull(ctx, cache) switch { case err == nil: if st.Access == store.AccessNone { @@ -726,9 +726,9 @@ func sizeBound(size int64) int64 { // stated property only while the party serving the bytes was a PEER, whose // numbers a separate hub had at least stored; when the peer IS the hub there is // no second party at all, and one listing entry or one journal line sized the -// device's allocation and its disk. A journal is JSONL text (32 MiB is ~500k -// ops) and a blob this size is already far past what a synced project of notes -// and documents holds. +// device's allocation and its disk. A journal is JSONL text (100 MiB is well +// over a million ops) and a blob this size is already far past what a synced +// project of notes and documents holds. // // It is a read CEILING, never an up-front refusal on the declared size: op.Size // is a peer's integer with no relation to the object it names, so refusing on it @@ -739,8 +739,10 @@ func sizeBound(size int64) int64 { // ponytail: an absolute constant, mirroring `bdrive import`'s maxImportBlob // (256 << 20, raisable with --max-blob). A file larger than this does not // materialize on receiving devices — if that becomes a real workload, this -// wants the same kind of knob, not a bigger constant. -const maxPullBytes = 32 << 20 +// wants the same kind of knob, not a bigger constant. Raised from 32 MiB with +// delta sync: large files were the reason chunking shipped, and a ceiling +// below the files it was built for made it dead weight. +const maxPullBytes = 100 << 20 func pullBound(size int64) int64 { return min(sizeBound(size), maxPullBytes) } @@ -849,7 +851,10 @@ func withdrawn(have, applied []journal.Op) []journal.Op { // pull fetches journals that grew on the remote and any blobs we are missing // for the new ops. It returns the ops we had not seen before, and any op a // peer withdrew from a journal we had already applied (see withdrawn). -func (s *Session) pull(ctx context.Context) ([]journal.Op, []journal.Op, error) { +// cache is the mount's materialization state, read only for the delta basis: +// cache[path].Blob is the version of a file this device currently holds, and +// its chunks source a chunked pull locally (fetchChunked). +func (s *Session) pull(ctx context.Context, cache map[string]store.CachedFile) ([]journal.Op, []journal.Op, error) { objs, err := s.Backend.List(ctx, "journal/") if err != nil { return nil, nil, err @@ -1010,6 +1015,43 @@ func (s *Session) pull(ctx context.Context) ([]journal.Op, []journal.Op, error) if op.Kind != journal.KindPut || op.Blob == "" || s.Store.HasBlob(op.Blob) { continue } + // Large files: try the manifest first, sourcing unchanged chunks from + // the version of this path we already hold. EVERY failure falls + // through to the whole-blob path, not just errNoManifest: the whole + // blob is independently hash-verified, so there is nothing to lose by + // trying it — and not falling through let one member-written manifest + // (the only object in the key space that is neither content-addressed + // nor hash-checked at ingest) permanently deny a file whose correct + // whole blob was sitting right there. A hash contradiction is still + // remembered as the one signal worth surfacing (errBlobContent). + if op.Size > chunkThreshold { + var basis string + if c, ok := cache[op.Path]; ok { + basis = c.Blob + } + cerr := s.fetchChunked(ctx, op, basis) + if cerr == nil { + continue + } + // Fall through to the whole blob only when one actually EXISTS + // (Exists never triggers hub reassembly). When it does, the + // fallthrough is unconditional — the whole blob is independently + // hash-verified, so a poisoned manifest cannot deny it. When it + // does not — the normal chunked-only case — a transient chunk + // failure retries chunks next cycle instead of asking the hub to + // reassemble and re-download the entire file every tick. + // + // A chunked-path hash contradiction is recorded ONLY when no + // fallback can land the file: when the whole blob arrives fine, + // "blob corrupt on remote" would page an operator about a file + // that just converged. + if ok, eerr := s.Backend.Exists(ctx, "blobs/"+op.Blob); eerr != nil || !ok { + if errors.Is(cerr, errBlobContent) && bad == nil { + bad = cerr + } + continue + } + } rc, err := s.Backend.Get(ctx, "blobs/"+op.Blob) if err != nil { continue @@ -1485,6 +1527,18 @@ func (s *Session) push(ctx context.Context, myOps []journal.Op, st *store.SyncSt g.SetLimit(pushConcurrency) for _, j := range jobs { g.Go(func() error { + // Large files move as chunks + a manifest (chunks.go); the whole + // blob is never uploaded for them. Everything else is unchanged. + if j.size > chunkThreshold { + n, err := s.pushChunked(gctx, j.blob) + if err != nil { + return err + } + atomic.AddInt64(&done, 1) + atomic.AddInt64(&bytesDone, n) + report() + return nil + } f, err := s.Store.OpenBlob(j.blob) if err != nil { return err diff --git a/internal/webapp/chunks_test.go b/internal/webapp/chunks_test.go new file mode 100644 index 0000000..d83fd2e --- /dev/null +++ b/internal/webapp/chunks_test.go @@ -0,0 +1,333 @@ +package webapp + +// Delta-sync hub rows (.claude/delta-sync-goal.md). The store's key space +// grows two content-addressed classes: chunks/ (one content-defined +// chunk) and manifests/ (the chunk list for the whole file with that +// sha). Both inherit blobs/' properties: immutable, never deleted. + +import ( + "fmt" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/runbear-io/beardrive/internal/remote" +) + +// placeChunked hand-builds a chunked-only file in a project's file:// store: +// its chunks and manifest exist, the whole blob does not. Returns the file's +// sha (the blob/manifest key). +func placeChunked(t *testing.T, dir string, parts ...string) string { + t.Helper() + whole := strings.Join(parts, "") + sha := shaOf(whole) + if err := os.MkdirAll(filepath.Join(dir, "chunks"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(dir, "manifests"), 0o755); err != nil { + t.Fatal(err) + } + var entries []string + for _, p := range parts { + if err := os.WriteFile(filepath.Join(dir, "chunks", shaOf(p)), []byte(p), 0o644); err != nil { + t.Fatal(err) + } + entries = append(entries, fmt.Sprintf(`{"h":%q,"n":%d}`, shaOf(p), len(p))) + } + man := fmt.Sprintf(`{"v":1,"size":%d,"chunks":[%s]}`, len(whole), strings.Join(entries, ",")) + if err := os.WriteFile(filepath.Join(dir, "manifests", sha), []byte(man), 0o644); err != nil { + t.Fatal(err) + } + return sha +} + +// TestDelta_Hub_BackfillOnce (row H1): a blob that exists only as chunks + +// manifest is served by reassembly, verified, and backfilled to blobs/ +// so the second read hits the whole blob directly. +func TestDelta_Hub_BackfillOnce(t *testing.T) { + srv, p, root := newHub(t, true, nil) + dir := filepath.Join(root, p.ID) + sha := placeChunked(t, dir, "first chunk of the file|", "second chunk of the file") + whole := "first chunk of the file|second chunk of the file" + h := srv.Handler() + url := "/api/p/" + p.ID + "/store/object?key=blobs/" + sha + + blobPath := filepath.Join(dir, "blobs", sha) + if _, err := os.Stat(blobPath); err == nil { + t.Fatal("whole blob exists before the test ran") + } + rec := do(t, h, "GET", url, nil) + if rec.Code != http.StatusOK || rec.Body.String() != whole { + t.Fatalf("reassembled read: %d %q, want 200 %q", rec.Code, rec.Body, whole) + } + if _, err := os.Stat(blobPath); err != nil { + t.Fatalf("whole blob not backfilled after reassembly: %v", err) + } + // Second read is served from the backfilled blob (and still correct). + if rec := do(t, h, "GET", url, nil); rec.Code != http.StatusOK || rec.Body.String() != whole { + t.Fatalf("second read: %d %q", rec.Code, rec.Body) + } +} + +// TestDelta_Manifest_SelfVerifying (row C4): a manifest whose chunks do not +// reassemble to its key is refused, and nothing is backfilled. +func TestDelta_Manifest_SelfVerifying(t *testing.T) { + srv, p, root := newHub(t, true, nil) + dir := filepath.Join(root, p.ID) + sha := placeChunked(t, dir, "honest part|", "also honest") + // Corrupt one chunk in place: the manifest still names it, the bytes are + // wrong, so reassembly must not hash to the key. + badChunk := filepath.Join(dir, "chunks", shaOf("honest part|")) + if err := os.WriteFile(badChunk, []byte("hostile bytes"), 0o644); err != nil { + t.Fatal(err) + } + h := srv.Handler() + rec := do(t, h, "GET", "/api/p/"+p.ID+"/store/object?key=blobs/"+sha, nil) + if rec.Code != http.StatusNotFound { + t.Fatalf("corrupt reassembly served: %d %q, want 404", rec.Code, rec.Body) + } + if _, err := os.Stat(filepath.Join(dir, "blobs", sha)); err == nil { + t.Fatal("corrupt reassembly was backfilled") + } +} + +// TestSec_Chunks_ReassemblyBoundsHostileManifest: a manifest is member-written +// and cannot be hash-checked at ingest, so its declared chunk list is attacker +// input. Listing one real chunk many times must not make the hub spool the +// amplified total (temp-dir/RAM exhaustion per read) — the declared sum is +// refused past maxReassembleBytes before any chunk is fetched. And a stored +// chunk object LONGER than its manifest entry must not sail past the cap: the +// copy is bounded by the declared size and a length mismatch fails the read. +func TestSec_Chunks_ReassemblyBoundsHostileManifest(t *testing.T) { + srv, p, root := newHub(t, true, nil) + dir := filepath.Join(root, p.ID) + h := srv.Handler() + + // One real 1 MiB chunk, listed enough times to declare ~300 GiB. + chunk := strings.Repeat("A", 1<<20) + sha := placeChunked(t, dir, chunk) // sha of the 1-chunk file; manifest will be replaced + entry := fmt.Sprintf(`{"h":%q,"n":%d}`, shaOf(chunk), len(chunk)) + entries := make([]string, 300<<10) + for i := range entries { + entries[i] = entry + } + man := fmt.Sprintf(`{"v":1,"size":%d,"chunks":[%s]}`, int64(len(chunk))*int64(len(entries)), strings.Join(entries, ",")) + if err := os.WriteFile(filepath.Join(dir, "manifests", sha), []byte(man), 0o644); err != nil { + t.Fatal(err) + } + + rec := do(t, h, "GET", "/api/p/"+p.ID+"/store/object?key=blobs/"+sha, nil) + if rec.Code != http.StatusNotFound { + t.Fatalf("amplified manifest was served: %d (%d bytes)", rec.Code, rec.Body.Len()) + } + if _, err := os.Stat(filepath.Join(dir, "blobs", sha)); err == nil { + t.Fatal("amplified manifest was backfilled") + } + + // A stored chunk longer than its declared size: the manifest says 10 + // bytes, the object holds a megabyte. The copy is bounded and refused. + longSha := placeChunked(t, dir, "0123456789") + if err := os.WriteFile(filepath.Join(dir, "chunks", shaOf("0123456789")), []byte(strings.Repeat("B", 1<<20)), 0o644); err != nil { + t.Fatal(err) + } + rec = do(t, h, "GET", "/api/p/"+p.ID+"/store/object?key=blobs/"+longSha, nil) + if rec.Code != http.StatusNotFound { + t.Fatalf("oversized chunk object was served: %d", rec.Code) + } +} + +// TestSec_Chunks_ManifestMustNameUploadedChunks (CTO H6): the manifest key +// is the one member-writable object that is not content-addressed, and a +// member who can READ a file can publish its true chunk hashes without +// uploading a byte — poisoning the empty slot under a whole-pushed blob so a +// later honest push skips chunks that do not exist. The ingest door is where +// the invariant is enforced: a manifest is accepted only when the store +// holds every chunk it names. The honest client writes chunks first, so this +// always passes for it. +func TestSec_Chunks_ManifestMustNameUploadedChunks(t *testing.T) { + srv, p, _ := newHub(t, true, nil) + h := srv.Handler() + base := "/api/p/" + p.ID + "/store/" + + // Naming a chunk the store does not hold: refused, nothing stored. + key := "manifests/" + shaOf("victim file") + vapor := []byte(`{"v":1,"size":9,"chunks":[{"h":"` + shaOf("not uploaded") + `","n":9}]}`) + if rec := do(t, h, "PUT", base+"object?key="+key, vapor); rec.Code != http.StatusBadRequest { + t.Fatalf("manifest naming an absent chunk: %d %s, want 400", rec.Code, rec.Body) + } + if rec := do(t, h, "GET", base+"object?key="+key, nil); rec.Code == http.StatusOK { + t.Fatal("refused manifest was stored anyway") + } + + // Upload the chunk, and the same manifest is accepted. + chunk := []byte("not uploaded") + if rec := do(t, h, "PUT", base+"object?key=chunks/"+shaOf(string(chunk)), chunk); rec.Code != http.StatusOK { + t.Fatalf("chunk put: %d %s", rec.Code, rec.Body) + } + if rec := do(t, h, "PUT", base+"object?key="+key, vapor); rec.Code != http.StatusOK { + t.Fatalf("manifest with its chunks present: %d %s, want 200", rec.Code, rec.Body) + } +} + +// TestSec_Chunks_ManifestWriteOnce: a manifest is the one member-writable +// object that is neither content-addressed nor hash-checkable at ingest, so +// the hub stores it write-once. Re-putting identical bytes stays a no-op +// (the retry after an interrupted push must work); a DIFFERENT body for an +// existing key is refused — overwriting was the only way a member could +// re-point an existing file's chunk list after the fact. +func TestSec_Chunks_ManifestWriteOnce(t *testing.T) { + srv, p, _ := newHub(t, true, nil) + h := srv.Handler() + base := "/api/p/" + p.ID + "/store/" + key := "manifests/" + shaOf("some whole file") + + // The chunks a manifest names must exist first (the ingest invariant), + // so upload them before the manifests that reference them. + for _, c := range []string{"hello", "evil!"} { + if rec := do(t, h, "PUT", base+"object?key=chunks/"+shaOf(c), []byte(c)); rec.Code != http.StatusOK { + t.Fatalf("chunk put: %d %s", rec.Code, rec.Body) + } + } + man := []byte(`{"v":1,"size":5,"chunks":[{"h":"` + shaOf("hello") + `","n":5}]}`) + if rec := do(t, h, "PUT", base+"object?key="+key, man); rec.Code != http.StatusOK { + t.Fatalf("first manifest put: %d %s", rec.Code, rec.Body) + } + // Identical retry: accepted as a no-op. + if rec := do(t, h, "PUT", base+"object?key="+key, man); rec.Code != http.StatusOK { + t.Fatalf("identical manifest retry: %d %s, want 200", rec.Code, rec.Body) + } + // A different body for the same key: refused, and the original survives. + other := []byte(`{"v":1,"size":5,"chunks":[{"h":"` + shaOf("evil!") + `","n":5}]}`) + if rec := do(t, h, "PUT", base+"object?key="+key, other); rec.Code != http.StatusConflict { + t.Fatalf("manifest overwrite: %d %s, want 409", rec.Code, rec.Body) + } + if rec := do(t, h, "GET", base+"object?key="+key, nil); rec.Body.String() != string(man) { + t.Fatalf("stored manifest changed after refused overwrite: %q", rec.Body) + } +} + +// TestDelta_Hub_ChunkPresignRefusesExisting (row H2): chunks presign like +// blobs — content-addressed and immutable — and BOTH presign doors' rule +// applies: a key that already exists is never signed again (the sealing +// invariant in RemoteSource.verify rests on it). +func TestDelta_Hub_ChunkPresignRefusesExisting(t *testing.T) { + var sb *signingBackend + srv, p, root := newHub(t, true, func(be remote.Backend) remote.Backend { + sb = &signingBackend{Backend: be} + return sb + }) + h := srv.Handler() + base := "/api/p/" + p.ID + "/store/" + + // A fresh chunk key signs direct. + freshKey := "chunks/" + shaOf("fresh chunk") + rec := do(t, h, "POST", base+"sign", map[string]any{"key": freshKey, "size": 11}) + if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), `"direct"`) || !strings.Contains(rec.Body.String(), `"url"`) { + t.Fatalf("sign fresh chunk: %d %s, want direct with url", rec.Code, rec.Body) + } + if len(sb.signed) != 1 { + t.Fatalf("signed = %v, want exactly the fresh chunk", sb.signed) + } + + // An existing chunk key is answered exists:true and never signed. + existing := "existing chunk content" + if err := os.MkdirAll(filepath.Join(root, p.ID, "chunks"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, p.ID, "chunks", shaOf(existing)), []byte(existing), 0o644); err != nil { + t.Fatal(err) + } + rec = do(t, h, "POST", base+"sign", map[string]any{"key": "chunks/" + shaOf(existing), "size": int64(len(existing))}) + if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), `"exists":true`) { + t.Fatalf("sign existing chunk: %d %s, want exists:true", rec.Code, rec.Body) + } + if len(sb.signed) != 1 { + t.Fatalf("existing chunk was signed: %v", sb.signed) + } +} + +// TestDelta_Hub_ManifestNeverPresigned (row H3): manifests are mutable-shaped +// trust (their key is not their content's hash), so like journals they always +// flow through the server. +func TestDelta_Hub_ManifestNeverPresigned(t *testing.T) { + var sb *signingBackend + srv, p, _ := newHub(t, true, func(be remote.Backend) remote.Backend { + sb = &signingBackend{Backend: be} + return sb + }) + h := srv.Handler() + rec := do(t, h, "POST", "/api/p/"+p.ID+"/store/sign", + map[string]any{"key": "manifests/" + shaOf("some file"), "size": 128}) + if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), `"server"`) { + t.Fatalf("sign manifest: %d %s, want server mode", rec.Code, rec.Body) + } + if len(sb.signed) != 0 { + t.Fatalf("manifest was presigned: %v", sb.signed) + } +} + +// TestDelta_Hub_KeySpace (row H4): validStoreKey, the list prefix allowlist, +// and the put hash check accept and constrain both new key classes. +func TestDelta_Hub_KeySpace(t *testing.T) { + srv, p, _ := newHub(t, true, nil) + h := srv.Handler() + base := "/api/p/" + p.ID + "/store/" + + // Malformed spellings are refused exactly like malformed blob keys. + bad := []string{ + "chunks/short", "chunks/../../etc/passwd", + "chunks/" + strings.Repeat("G", 64), + "manifests/short", "manifests/" + strings.Repeat("Z", 64), + "manifests/../device.json", + } + for _, key := range bad { + if rec := do(t, h, "GET", base+"object?key="+key, nil); rec.Code != http.StatusBadRequest { + t.Errorf("get %q: %d, want 400", key, rec.Code) + } + if rec := do(t, h, "PUT", base+"object?key="+key, []byte("x")); rec.Code != http.StatusBadRequest { + t.Errorf("put %q: %d, want 400", key, rec.Code) + } + } + + // A chunk is content-addressed: content that does not hash to its key is + // refused, content that does is stored and served back. + chunk := []byte("chunk content for the key-space test") + goodKey := "chunks/" + shaOf(string(chunk)) + if rec := do(t, h, "PUT", base+"object?key="+goodKey, []byte("not that content")); rec.Code != http.StatusBadRequest { + t.Fatalf("put chunk with wrong content: %d, want 400", rec.Code) + } + if rec := do(t, h, "PUT", base+"object?key="+goodKey, chunk); rec.Code != http.StatusOK { + t.Fatalf("put chunk: %d %s", rec.Code, rec.Body) + } + if rec := do(t, h, "GET", base+"object?key="+goodKey, nil); rec.Code != http.StatusOK || rec.Body.String() != string(chunk) { + t.Fatalf("get chunk: %d %q", rec.Code, rec.Body) + } + + // A manifest's key is the FILE's sha, not the manifest body's own hash — + // there is no ingest-time hash relation to enforce (readers verify by + // reassembly) — but every chunk it names must already be in the store. + // The chunk uploaded above satisfies that; a manifest naming vapor is + // TestSec_Chunks_ManifestMustNameUploadedChunks' subject. + manifest := []byte(`{"v":1,"size":36,"chunks":[{"h":"` + shaOf(string(chunk)) + `","n":36}]}`) + manKey := "manifests/" + shaOf("whole file content, not the manifest body") + if rec := do(t, h, "PUT", base+"object?key="+manKey, manifest); rec.Code != http.StatusOK { + t.Fatalf("put manifest: %d %s", rec.Code, rec.Body) + } + if rec := do(t, h, "GET", base+"object?key="+manKey, nil); rec.Code != http.StatusOK || rec.Body.String() != string(manifest) { + t.Fatalf("get manifest: %d %q", rec.Code, rec.Body) + } + + // Both prefixes list. + for _, prefix := range []string{"chunks/", "manifests/"} { + if rec := do(t, h, "GET", base+"list?prefix="+prefix, nil); rec.Code != http.StatusOK { + t.Errorf("list %q: %d %s", prefix, rec.Code, rec.Body) + } + } + + // exists works for both. + if rec := do(t, h, "GET", base+"exists?key="+goodKey, nil); rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), "true") { + t.Errorf("exists chunk: %d %s", rec.Code, rec.Body) + } +} diff --git a/internal/webapp/cli_e2e_test.go b/internal/webapp/cli_e2e_test.go index 2343e5c..dfdba86 100644 --- a/internal/webapp/cli_e2e_test.go +++ b/internal/webapp/cli_e2e_test.go @@ -47,14 +47,24 @@ func newCLIEnv(t *testing.T) cliEnv { // DEVICES (separate BDRIVE_HOMEs, separate device identities) on one project. // nil starts a throwaway hub, which is the single-device default. func newCLIEnvOn(t *testing.T, hub *httptest.Server) cliEnv { + t.Helper() + return newCLIEnvBin(t, hub, "") +} + +// newCLIEnvBin is newCLIEnvOn with an explicit binary — the delta-sync E2E +// rows drive a binary built from the pre-change commit against the same hub +// as a current one. Empty means build the current tree. +func newCLIEnvBin(t *testing.T, hub *httptest.Server, bin string) cliEnv { t.Helper() if testing.Short() { t.Skip("builds and execs the bdrive binary; skipped with -short") } - bin := filepath.Join(t.TempDir(), "bdrive") - build := exec.Command("go", "build", "-o", bin, "github.com/runbear-io/beardrive/cmd/bdrive") - if out, err := build.CombinedOutput(); err != nil { - t.Fatalf("go build: %v\n%s", err, out) + if bin == "" { + bin = filepath.Join(t.TempDir(), "bdrive") + build := exec.Command("go", "build", "-o", bin, "github.com/runbear-io/beardrive/cmd/bdrive") + if out, err := build.CombinedOutput(); err != nil { + t.Fatalf("go build: %v\n%s", err, out) + } } if hub == nil { diff --git a/internal/webapp/delta_e2e_test.go b/internal/webapp/delta_e2e_test.go new file mode 100644 index 0000000..ffeff08 --- /dev/null +++ b/internal/webapp/delta_e2e_test.go @@ -0,0 +1,598 @@ +package webapp + +// The E block of the delta-sync goal (.claude/delta-sync-goal.md): real +// binaries over real HTTP. E2 and E3 are the reason this file exists — they +// drive a binary built from the PRE-CHANGE commit, the only honest "old +// client". Every delta change in this branch is uncommitted work on top of +// HEAD, so `git archive HEAD` yields the pre-change tree without touching +// shared git state; if the delta work gets committed, the ref below must +// become the merge-base of the branch. + +import ( + "archive/tar" + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "math/rand" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +// oldBinRef is the commit the "old client" is built from: the last commit +// before delta sync landed, pinned by sha so it stays the pre-manifest binary +// forever — a moving ref (HEAD, a merge-base) resolves to a delta-aware tree +// once the work is merged, and these tests then build the NEW binary as the +// "old client" and pass vacuously. +const oldBinRef = "33ca0caeabacc6c632fb4ec00fda3abe88c2e3e8" + +var ( + oldBinOnce sync.Once + oldBinPath string + oldBinErr error +) + +// buildOldBinary extracts oldBinRef into a temp tree via `git archive` (no +// worktree registration, no shared state) and builds bdrive there. Cached for +// the package run. +func buildOldBinary(t *testing.T) string { + t.Helper() + oldBinOnce.Do(func() { + dir, err := os.MkdirTemp("", "bdrive-oldbin-*") + if err != nil { + oldBinErr = err + return + } + rootOut, err := exec.Command("git", "rev-parse", "--show-toplevel").Output() + if err != nil { + oldBinErr = fmt.Errorf("git rev-parse: %w", err) + return + } + root := strings.TrimSpace(string(rootOut)) + archive := func() ([]byte, error) { + return exec.Command("git", "-C", root, "archive", "--format=tar", oldBinRef).Output() + } + tarBytes, err := archive() + if err != nil { + // CI checkouts are shallow (actions/checkout fetch-depth 1), so + // the pinned pre-delta commit is usually absent there. Fetch just + // that commit and retry — one object, no workflow change, and a + // full local clone never takes this path. + if fout, ferr := exec.Command("git", "-C", root, "fetch", "--depth=1", "origin", oldBinRef).CombinedOutput(); ferr != nil { + oldBinErr = fmt.Errorf("git archive %s failed and the commit could not be fetched (shallow clone without network?): %v\n%s", oldBinRef, ferr, fout) + return + } + if tarBytes, err = archive(); err != nil { + oldBinErr = fmt.Errorf("git archive %s after fetch: %w", oldBinRef, err) + return + } + } + tr := tar.NewReader(bytes.NewReader(tarBytes)) + for { + hdr, err := tr.Next() + if err == io.EOF { + break + } + if err != nil { + oldBinErr = err + return + } + dst := filepath.Join(dir, filepath.FromSlash(hdr.Name)) + switch hdr.Typeflag { + case tar.TypeDir: + if err := os.MkdirAll(dst, 0o755); err != nil { + oldBinErr = err + return + } + case tar.TypeReg: + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + oldBinErr = err + return + } + f, err := os.OpenFile(dst, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, os.FileMode(hdr.Mode)&0o777) + if err != nil { + oldBinErr = err + return + } + if _, err := io.Copy(f, tr); err != nil { + f.Close() + oldBinErr = err + return + } + f.Close() + } + } + bin := filepath.Join(dir, "old-bdrive") + build := exec.Command("go", "build", "-o", bin, "./cmd/bdrive") + build.Dir = dir + if out, err := build.CombinedOutput(); err != nil { + oldBinErr = fmt.Errorf("build old binary: %v\n%s", err, out) + return + } + oldBinPath = bin + }) + if oldBinErr != nil { + t.Fatal(oldBinErr) + } + return oldBinPath +} + +// randContent is deterministic large content; seeded so reruns measure the +// same bytes. +func randContent(seed int64, n int) []byte { + b := make([]byte, n) + rand.New(rand.NewSource(seed)).Read(b) + return b +} + +func shaOfBytes(b []byte) string { + h := sha256.Sum256(b) + return hex.EncodeToString(h[:]) +} + +// initProject runs init and registers a cleanup stop. `bdrive stop` PAUSES a +// mount (sync then refuses to run), so it must only happen at teardown — the +// daemon stays up during the test, exactly like production; explicit `bdrive +// sync` calls drive the assertions and the daemon's own cycles move the same +// bytes. +func initProject(t *testing.T, e cliEnv, dir, name string, connect bool) { + t.Helper() + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + // --name has create-or-join semantics (name-scoped per org), so the same + // spelling both creates the project on the first device and joins it on + // every later one; the connect flag exists only for the reader. + _ = connect + out, err := e.run(dir, "init", "--name", name, "--yes") + if err != nil { + t.Fatalf("init: %v\n%s", err, out) + } + t.Cleanup(func() { e.run(dir, "stop", dir) }) +} + +func syncNow(t *testing.T, e cliEnv, dir string) { + t.Helper() + if out, err := e.run(dir, "sync"); err != nil { + t.Fatalf("sync: %v\n%s", err, out) + } +} + +// countingHub wraps a test hub's handler and counts HTTP body bytes in each +// direction — the honest wire cost the E1 row asserts. +func countingHub(t *testing.T) (*httptest.Server, *atomic.Int64, *atomic.Int64) { + t.Helper() + inner := startTestHub(t) + var in, out atomic.Int64 + proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body := &countingReader{r: r.Body, n: &in} + r.Body = body + cw := &countingRW{ResponseWriter: w, n: &out} + inner.Config.Handler.ServeHTTP(cw, r) + })) + t.Cleanup(proxy.Close) + return proxy, &in, &out +} + +type countingReader struct { + r io.ReadCloser + n *atomic.Int64 +} + +func (c *countingReader) Read(p []byte) (int, error) { + n, err := c.r.Read(p) + c.n.Add(int64(n)) + return n, err +} +func (c *countingReader) Close() error { return c.r.Close() } + +type countingRW struct { + http.ResponseWriter + n *atomic.Int64 +} + +func (c *countingRW) Write(p []byte) (int, error) { + n, err := c.ResponseWriter.Write(p) + c.n.Add(int64(n)) + return n, err +} + +// TestDeltaE2E_TwoDevicesLargeFile (row E1): two real bdrive processes, one +// real hub; a 1-byte edit to a 20 MiB file crosses the wire as chunks, not +// the file — pushed and pulled each under 5 MiB. +func TestDeltaE2E_TwoDevicesLargeFile(t *testing.T) { + hub, in, out := countingHub(t) + a := newCLIEnvOn(t, hub) + b := newCLIEnvOn(t, hub) + + dirA := filepath.Join(t.TempDir(), "proj") + initProject(t, a, dirA, "delta-e2e", false) + content := randContent(60, 20<<20) + if err := os.WriteFile(filepath.Join(dirA, "big.bin"), content, 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, a, dirA) + + dirB := filepath.Join(t.TempDir(), "proj") + initProject(t, b, dirB, "delta-e2e", true) + syncNow(t, b, dirB) + got, err := os.ReadFile(filepath.Join(dirB, "big.bin")) + if err != nil || !bytes.Equal(got, content) { + t.Fatalf("device B did not converge: %v, %d bytes", err, len(got)) + } + + pushMark := in.Load() + content[10<<20] ^= 0xff + if err := os.WriteFile(filepath.Join(dirA, "big.bin"), content, 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, a, dirA) + pushed := in.Load() - pushMark + t.Logf("E1: edit pushed %d bytes over real HTTP", pushed) + if pushed > 5<<20 { + t.Fatalf("edit pushed %d bytes over the wire, want < 5 MiB", pushed) + } + + pullMark := out.Load() + syncNow(t, b, dirB) + pulled := out.Load() - pullMark + got, err = os.ReadFile(filepath.Join(dirB, "big.bin")) + if err != nil || !bytes.Equal(got, content) { + t.Fatalf("device B did not converge on the edit: %v", err) + } + t.Logf("E1: edit pulled %d bytes over real HTTP", pulled) + if pulled > 5<<20 { + t.Fatalf("edit pulled %d bytes over the wire, want < 5 MiB", pulled) + } +} + +// TestDeltaE2E_OldBinaryReadsChunkedStorage (row E2): a binary that has never +// heard of a manifest syncs a project whose storage holds the big file ONLY +// as chunks + manifest, and gets correct bytes — the hub's reassembly is the +// entire compatibility story, exercised for real. +func TestDeltaE2E_OldBinaryReadsChunkedStorage(t *testing.T) { + hub := startTestHub(t) + a := newCLIEnvOn(t, hub) + old := newCLIEnvBin(t, hub, buildOldBinary(t)) + + dirA := filepath.Join(t.TempDir(), "proj") + initProject(t, a, dirA, "delta-e2e-old-read", false) + content := randContent(61, 12<<20) + if err := os.WriteFile(filepath.Join(dirA, "big.bin"), content, 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dirA, "note.md"), []byte("hello old client"), 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, a, dirA) + + dirOld := filepath.Join(t.TempDir(), "proj") + initProject(t, old, dirOld, "delta-e2e-old-read", true) + syncNow(t, old, dirOld) + got, err := os.ReadFile(filepath.Join(dirOld, "big.bin")) + if err != nil || !bytes.Equal(got, content) { + t.Fatalf("old binary did not converge on chunked content: %v, %d bytes", err, len(got)) + } + if got, err := os.ReadFile(filepath.Join(dirOld, "note.md")); err != nil || string(got) != "hello old client" { + t.Fatalf("old binary missed the small file: %v", err) + } +} + +// TestDeltaE2E_MixedFleetConflictAndDelete: the full sync semantics across +// binary versions on one hub, not just fetch. Concurrent edits of the same +// large file by an old and a new client must resolve the way two same-version +// clients would — one winner everywhere, the loser preserved as a conflict +// copy — and a delete journaled by the new client must remove the file from +// the old client's folder (and the reverse edit land back). If replay or +// conflict handling behaved differently across versions, this is where it +// shows. +func TestDeltaE2E_MixedFleetConflictAndDelete(t *testing.T) { + hub := startTestHub(t) + old := newCLIEnvBin(t, hub, buildOldBinary(t)) + cur := newCLIEnvOn(t, hub) + + dirOld := filepath.Join(t.TempDir(), "proj") + initProject(t, old, dirOld, "delta-e2e-mixed", false) + base := randContent(70, 8<<20) + if err := os.WriteFile(filepath.Join(dirOld, "big.bin"), base, 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, old, dirOld) + + dirCur := filepath.Join(t.TempDir(), "proj") + initProject(t, cur, dirCur, "delta-e2e-mixed", true) + syncNow(t, cur, dirCur) + if got, err := os.ReadFile(filepath.Join(dirCur, "big.bin")); err != nil || !bytes.Equal(got, base) { + t.Fatalf("seed did not converge: %v", err) + } + + // Concurrent edits before either syncs: old edits the head, new the tail. + editOld := append([]byte{}, base...) + editOld[0] ^= 0xff + editCur := append([]byte{}, base...) + editCur[len(editCur)-1] ^= 0xff + if err := os.WriteFile(filepath.Join(dirOld, "big.bin"), editOld, 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dirCur, "big.bin"), editCur, 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, old, dirOld) + syncNow(t, cur, dirCur) + syncNow(t, old, dirOld) + syncNow(t, cur, dirCur) + syncNow(t, old, dirOld) // one more pass so the loser's conflict copy propagates + + vOld, err := os.ReadFile(filepath.Join(dirOld, "big.bin")) + if err != nil { + t.Fatal(err) + } + vCur, err := os.ReadFile(filepath.Join(dirCur, "big.bin")) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(vOld, vCur) { + t.Fatal("old and new clients converged to different winners") + } + if !bytes.Equal(vOld, editOld) && !bytes.Equal(vOld, editCur) { + t.Fatal("winner is neither client's edit") + } + loser := editOld + if bytes.Equal(vOld, editOld) { + loser = editCur + } + foundConflict := false + for _, dir := range []string{dirOld, dirCur} { + ents, _ := os.ReadDir(dir) + for _, e := range ents { + if strings.Contains(e.Name(), ".bdrive-conflict-") { + if got, err := os.ReadFile(filepath.Join(dir, e.Name())); err == nil && bytes.Equal(got, loser) { + foundConflict = true + } + } + } + } + if !foundConflict { + t.Fatal("losing edit was not preserved as a conflict copy on either client") + } + + // Delete journaled by the NEW client must leave the OLD client's folder. + if err := os.Remove(filepath.Join(dirCur, "big.bin")); err != nil { + t.Fatal(err) + } + syncNow(t, cur, dirCur) + syncNow(t, old, dirOld) + if _, err := os.Stat(filepath.Join(dirOld, "big.bin")); err == nil { + t.Fatal("new client's delete did not propagate to the old client") + } + + // And a fresh large file from the OLD client lands on the new one. + rebirth := randContent(71, 6<<20) + if err := os.WriteFile(filepath.Join(dirOld, "again.bin"), rebirth, 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, old, dirOld) + syncNow(t, cur, dirCur) + if got, err := os.ReadFile(filepath.Join(dirCur, "again.bin")); err != nil || !bytes.Equal(got, rebirth) { + t.Fatalf("old client's new file did not land on the new client: %v", err) + } +} + +// TestDeltaE2E_OldBinaryWritesNewReads (row E3): the old binary pushes whole +// blobs; the current binary must converge byte-identically on them. +func TestDeltaE2E_OldBinaryWritesNewReads(t *testing.T) { + hub := startTestHub(t) + old := newCLIEnvBin(t, hub, buildOldBinary(t)) + b := newCLIEnvOn(t, hub) + + dirOld := filepath.Join(t.TempDir(), "proj") + initProject(t, old, dirOld, "delta-e2e-old-write", false) + content := randContent(62, 12<<20) + if err := os.WriteFile(filepath.Join(dirOld, "big.bin"), content, 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, old, dirOld) + + dirB := filepath.Join(t.TempDir(), "proj") + initProject(t, b, dirB, "delta-e2e-old-write", true) + syncNow(t, b, dirB) + got, err := os.ReadFile(filepath.Join(dirB, "big.bin")) + if err != nil || !bytes.Equal(got, content) { + t.Fatalf("current binary did not converge on old-binary content: %v, %d bytes", err, len(got)) + } + + // And the reverse edit: the current binary edits (chunked push), the old + // one picks it up (hub reassembly) — full round trip in one project. + content[100] ^= 0xff + if err := os.WriteFile(filepath.Join(dirB, "big.bin"), content, 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, b, dirB) + syncNow(t, old, dirOld) + got, err = os.ReadFile(filepath.Join(dirOld, "big.bin")) + if err != nil || !bytes.Equal(got, content) { + t.Fatalf("old binary did not converge on the new binary's edit: %v", err) + } +} + +// TestDeltaE2E_AllReadSurfaces (row E5): viewer file API, history blob, +// download, and a share link all serve correct bytes for a chunked-only file +// over real HTTP. +func TestDeltaE2E_AllReadSurfaces(t *testing.T) { + hub := startTestHub(t) + a := newCLIEnvOn(t, hub) + + dir := filepath.Join(t.TempDir(), "proj") + initProject(t, a, dir, "delta-e2e-surfaces", false) + content := randContent(63, 10<<20) + if err := os.WriteFile(filepath.Join(dir, "big.bin"), content, 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, a, dir) + + // Resolve the project id from the hub. + resp, err := a.browser.Get(hub.URL + "/api/projects") + if err != nil { + t.Fatal(err) + } + var pl struct { + Projects []struct { + ID string `json:"id"` + Name string `json:"name"` + } `json:"projects"` + } + if err := json.NewDecoder(resp.Body).Decode(&pl); err != nil { + t.Fatal(err) + } + resp.Body.Close() + var pid string + for _, p := range pl.Projects { + if p.Name == "delta-e2e-surfaces" { + pid = p.ID + } + } + if pid == "" { + t.Fatalf("project not found in %+v", pl.Projects) + } + + sha := shaOfBytes(content) + fetch := func(url string) []byte { + t.Helper() + resp, err := a.browser.Get(url) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + t.Fatalf("GET %s: %d", url, resp.StatusCode) + } + b, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatal(err) + } + return b + } + if got := fetch(hub.URL + "/api/p/" + pid + "/file?path=big.bin"); !bytes.Equal(got, content) { + t.Fatal("viewer file API served wrong bytes") + } + if got := fetch(hub.URL + "/api/p/" + pid + "/blob?sha=" + sha); !bytes.Equal(got, content) { + t.Fatal("history blob API served wrong bytes") + } + + // Share link: minted by the CLI, fetched with no cookies at all. + out, err := a.run(dir, "share", "big.bin") + if err != nil { + t.Fatalf("share: %v\n%s", err, out) + } + shareURL := "" + for _, f := range strings.Fields(out) { + if strings.Contains(f, "/s/") { + shareURL = f + } + } + if shareURL == "" { + t.Fatalf("no share URL in output:\n%s", out) + } + anon := &http.Client{} + resp, err = anon.Get(shareURL) + if err != nil { + t.Fatal(err) + } + got, err := io.ReadAll(resp.Body) + resp.Body.Close() + if err != nil || resp.StatusCode != http.StatusOK || !bytes.Equal(got, content) { + t.Fatalf("share link served wrong bytes: %d, %v, %d bytes", resp.StatusCode, err, len(got)) + } +} + +// TestDeltaE2E_MigrateRoundTrip (row E4): real `bdrive export` from one hub, +// real `bdrive import` into another, and a third real device syncs the +// imported project into a fresh folder with correct bytes. +func TestDeltaE2E_MigrateRoundTrip(t *testing.T) { + hubA := startTestHub(t) + a := newCLIEnvOn(t, hubA) + dirA := filepath.Join(t.TempDir(), "proj") + initProject(t, a, dirA, "delta-e2e-migrate", false) + content := randContent(64, 9<<20) + if err := os.WriteFile(filepath.Join(dirA, "big.bin"), content, 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dirA, "small.md"), []byte("survives migration"), 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, a, dirA) + + arch := filepath.Join(dirA, "export.tar.gz") + if out, err := a.run(dirA, "export", "-o", arch); err != nil { + t.Fatalf("export: %v\n%s", err, out) + } + + // Import runs from anywhere on a signed-in device and CREATES the project + // on the target hub, named from the archive's manifest. + hubB := startTestHub(t) + b := newCLIEnvOn(t, hubB) + if out, err := b.run(t.TempDir(), "import", arch); err != nil { + t.Fatalf("import: %v\n%s", err, out) + } + + c := newCLIEnvOn(t, hubB) + dirC := filepath.Join(t.TempDir(), "proj") + initProject(t, c, dirC, "delta-e2e-migrate", true) + syncNow(t, c, dirC) + got, err := os.ReadFile(filepath.Join(dirC, "big.bin")) + if err != nil || !bytes.Equal(got, content) { + t.Fatalf("third device did not converge on migrated content: %v, %d bytes", err, len(got)) + } + if got, err := os.ReadFile(filepath.Join(dirC, "small.md")); err != nil || string(got) != "survives migration" { + t.Fatalf("small file lost in migration: %v", err) + } +} + +// TestDeltaE2E_DaemonPropagates (row E6): the daemon path, not one-shot sync. +// A chunked edit lands on a peer through its running daemon. +func TestDeltaE2E_DaemonPropagates(t *testing.T) { + hub := startTestHub(t) + a := newCLIEnvOn(t, hub) + b := newCLIEnvOn(t, hub) + + dirA := filepath.Join(t.TempDir(), "proj") + initProject(t, a, dirA, "delta-e2e-daemon", false) + content := randContent(65, 8<<20) + if err := os.WriteFile(filepath.Join(dirA, "big.bin"), content, 0o644); err != nil { + t.Fatal(err) + } + syncNow(t, a, dirA) + + // Device B keeps its daemon RUNNING (init starts it; no stop here). + dirB := filepath.Join(t.TempDir(), "proj") + if err := os.MkdirAll(dirB, 0o755); err != nil { + t.Fatal(err) + } + out, err := b.run(dirB, "init", "--name", "delta-e2e-daemon", "--yes") + if err != nil { + t.Fatalf("init: %v\n%s", err, out) + } + defer b.run(dirB, "stop", dirB) + + deadline := time.Now().Add(90 * time.Second) + for { + got, err := os.ReadFile(filepath.Join(dirB, "big.bin")) + if err == nil && bytes.Equal(got, content) { + break + } + if time.Now().After(deadline) { + t.Fatalf("daemon did not materialize the chunked file in 90s (err=%v)", err) + } + time.Sleep(2 * time.Second) + } +} diff --git a/internal/webapp/sec_audit_test.go b/internal/webapp/sec_audit_test.go index 2c245d1..7ec9c16 100644 --- a/internal/webapp/sec_audit_test.go +++ b/internal/webapp/sec_audit_test.go @@ -317,8 +317,17 @@ func TestSec_Audit_OpBlobIsRefusedBeforeItReachesStorage(t *testing.T) { if _, err := rs.OpenBlob(context.Background(), good); err == nil { t.Fatal("control: the spy backend must fail the fetch") } - if asked := be.asked(); len(asked) != 1 || asked[0] != "blobs/"+good { - t.Fatalf("control: a valid sha was not fetched as blobs/: %v", asked) + // A valid sha may ask for blobs/ and — since delta sync — fall back + // to manifests/ when the blob is absent. Every key asked must be one + // of those two exact spellings; anything else is the guard leaking. + asked := be.asked() + if len(asked) == 0 || asked[0] != "blobs/"+good { + t.Fatalf("control: a valid sha was not fetched as blobs/ first: %v", asked) + } + for _, k := range asked { + if k != "blobs/"+good && k != "manifests/"+good { + t.Fatalf("control: a valid sha asked storage for %q", k) + } } for _, bad := range []string{ diff --git a/internal/webapp/server.go b/internal/webapp/server.go index f62f0fa..1c324f5 100644 --- a/internal/webapp/server.go +++ b/internal/webapp/server.go @@ -31,6 +31,7 @@ import ( "maps" "mime" "net/http" + "os" "path" "slices" "sort" @@ -386,10 +387,121 @@ func (r *RemoteSource) OpenBlob(ctx context.Context, sha string) (io.ReadCloser, if !blobRe.MatchString(sha) { return nil, fmt.Errorf("invalid content reference") } - if err := r.verify(ctx, sha); err != nil { + if err := r.verify(ctx, sha); err == nil { + if rc, err := r.Backend.Get(ctx, "blobs/"+sha); err == nil { + return rc, nil + } + } + // No whole blob (or one that fails verification): the content may exist + // only as chunks + a manifest (delta sync, docs/delta-sync-prd.md). + // Reassembly is hash-verified against the sha requested, so this arm is + // exactly as strong as verify — including when blobs/ holds bytes + // that do NOT hash to sha, where the backfill below heals it. + return r.reassemble(ctx, sha) +} + +// maxReassembleBytes caps what one reassembly may spool. It mirrors `bdrive +// import`'s maxImportBlob (256 << 20): comfortably above the device-side +// materialization ceiling (syncer's maxPullBytes, 100 MiB), so nothing a +// device can hold is refused here, while a hostile manifest cannot spool +// gigabytes through the temp dir per read. The spool lands in os.TempDir — +// on a tmpfs deployment that is RAM, so this constant is also the per-request +// memory bound; set TMPDIR to disk if that matters. +const maxReassembleBytes = 256 << 20 + +// reassemble serves a blob that exists only as chunks: fetch the manifest +// keyed by the file's sha, concatenate its chunks into a spool while hashing, +// refuse the lot unless the result hashes to the sha requested, then backfill +// blobs/ so the next read is a plain Get. The spool is unavoidable — the +// promise "bytes hash to the key" cannot be made about a stream whose response +// has already started — and self-limiting: upgraded clients fetch chunks and +// verify locally, so only legacy readers ever pay it, once per blob. +func (r *RemoteSource) reassemble(ctx context.Context, sha string) (io.ReadCloser, error) { + mrc, err := r.Backend.Get(ctx, "manifests/"+sha) + if err != nil { + return nil, err // no manifest either: the object genuinely is not there + } + var man struct { + Chunks []struct { + H string `json:"h"` + N int64 `json:"n"` + } `json:"chunks"` + } + derr := json.NewDecoder(io.LimitReader(mrc, 8<<20)).Decode(&man) + mrc.Close() + if derr != nil { + return nil, fmt.Errorf("unreadable manifest for %s: %w", sha, derr) + } + // A manifest is member-written and cannot be hash-checked at ingest, so + // its declared total is the one number a hostile member controls: listing + // one real 4 MiB chunk 100k times would spool ~400 GB through the temp + // dir (RAM, on a tmpfs deployment) per legacy read. The sum is enforced + // here and the per-chunk copy below verifies declared == actual, so the + // declared total IS the spool bound. An honest manifest past the cap is + // refused the same way — the hub will not assemble what a device would + // not accept either (the client's own ceiling is maxPullBytes). + var declared int64 + for _, c := range man.Chunks { + if c.N < 0 || declared > maxReassembleBytes-c.N { + return nil, fmt.Errorf("manifest for %s declares more than %d bytes", sha, int64(maxReassembleBytes)) + } + declared += c.N + } + tmp, err := os.CreateTemp("", ".bdrive-reassemble-*") + if err != nil { return nil, err } - return r.Backend.Get(ctx, "blobs/"+sha) + os.Remove(tmp.Name()) // serve from the fd; nothing to clean up on any path + h := sha256.New() + w := io.MultiWriter(tmp, h) + for _, c := range man.Chunks { + if !blobRe.MatchString(c.H) { + tmp.Close() + return nil, fmt.Errorf("manifest for %s names an invalid chunk", sha) + } + crc, err := r.Backend.Get(ctx, "chunks/"+c.H) + if err != nil { + tmp.Close() + return nil, err + } + // Bounded by the DECLARED size, which the cap above already summed — + // a stored object longer than its manifest entry (a replayed presign) + // must not turn the cap into fiction. Short or long, the mismatch + // fails here; equal-but-wrong bytes fail the whole-file hash below. + n, cerr := io.Copy(w, io.LimitReader(crc, c.N+1)) + crc.Close() + if cerr != nil { + tmp.Close() + return nil, cerr + } + if n != c.N { + tmp.Close() + return nil, fmt.Errorf("chunk %s of %s is not its declared size", c.H[:12], sha) + } + } + if hex.EncodeToString(h.Sum(nil)) != sha { + tmp.Close() + return nil, fmt.Errorf("manifest for %s does not reassemble to its key", sha) + } + size, err := tmp.Seek(0, io.SeekEnd) + if err == nil { + _, err = tmp.Seek(0, io.SeekStart) + } + if err != nil { + tmp.Close() + return nil, err + } + // Backfill so the next read is a plain blobs/ Get. Best-effort: a failed + // write must never fail the read that triggered it. The object is fresh, + // so verify will re-hash it until it seals — same as any new upload. + if err := r.Backend.Put(ctx, "blobs/"+sha, tmp, size); err != nil { + log.Printf("beardrive: backfill of reassembled blob %s failed: %v", sha, err) + } + if _, err := tmp.Seek(0, io.SeekStart); err != nil { + tmp.Close() + return nil, err + } + return tmp, nil } // verify re-hashes a stored blob, unless this process has already proved that diff --git a/internal/webapp/store.go b/internal/webapp/store.go index c6b4e25..0c848cc 100644 --- a/internal/webapp/store.go +++ b/internal/webapp/store.go @@ -1,8 +1,10 @@ package webapp import ( + "bytes" "context" "encoding/json" + "errors" "fmt" "io" "log" @@ -29,10 +31,18 @@ import ( var ( blobKeyRe = regexp.MustCompile(`^blobs/[0-9a-f]{64}$`) journalKeyRe = regexp.MustCompile(`^journal/` + deviceIDPattern + `\.jsonl$`) + // Delta sync (docs/delta-sync-prd.md): a chunk is one content-defined + // piece of a large file, keyed by its own sha256; a manifest is the chunk + // list for the whole file with that sha256 — so Op.Blob alone locates it + // and the journal format never changes. Both are immutable and never + // deleted, exactly like blobs. + chunkKeyRe = regexp.MustCompile(`^chunks/[0-9a-f]{64}$`) + manifestKeyRe = regexp.MustCompile(`^manifests/[0-9a-f]{64}$`) ) func validStoreKey(key string) bool { - return blobKeyRe.MatchString(key) || journalKeyRe.MatchString(key) + return blobKeyRe.MatchString(key) || journalKeyRe.MatchString(key) || + chunkKeyRe.MatchString(key) || manifestKeyRe.MatchString(key) } // storeSource returns the volume's RemoteSource; only real beardrive @@ -173,8 +183,9 @@ func (s *Server) handleStoreList(v *volume, w http.ResponseWriter, r *http.Reque } s.refreshDevice(r) prefix := r.URL.Query().Get("prefix") - if prefix != "" && prefix != "journal/" && prefix != "blobs/" && - !strings.HasPrefix(prefix, "journal/") && !strings.HasPrefix(prefix, "blobs/") { + if prefix != "" && + !strings.HasPrefix(prefix, "journal/") && !strings.HasPrefix(prefix, "blobs/") && + !strings.HasPrefix(prefix, "chunks/") && !strings.HasPrefix(prefix, "manifests/") { http.Error(w, fmt.Sprintf("invalid prefix %q", prefix), http.StatusBadRequest) return } @@ -289,10 +300,17 @@ func (s *Server) handleStoreSign(v *volume, w http.ResponseWriter, r *http.Reque http.Error(w, err.Error(), http.StatusForbidden) return } - // Only blobs are presigned. They are content-addressed and immutable, so - // a leaked URL can at worst re-upload identical bytes. Journals are - // mutable state and always flow through the server. - if blob, isBlob := strings.CutPrefix(req.Key, "blobs/"); isBlob { + // Only blobs and chunks are presigned. They are content-addressed and + // immutable, so a leaked URL can at worst re-upload identical bytes. + // Journals are mutable state and always flow through the server — and so + // do manifests: their key is the whole FILE's sha, not their own content + // hash, so a presigned manifest write would be an unexamined claim about + // bytes the hub never saw. + blob, isBlob := strings.CutPrefix(req.Key, "blobs/") + if !isBlob { + blob, isBlob = strings.CutPrefix(req.Key, "chunks/") + } + if isBlob { if !sizeFitsContentAddress(blob, req.Size) { http.Error(w, "declared size does not match the content address", http.StatusForbidden) return @@ -525,10 +543,17 @@ func (s *Server) handleStorePut(v *volume, w http.ResponseWriter, r *http.Reques } defer os.Remove(tmp.Name()) defer tmp.Close() + // Blobs and chunks are content-addressed: the key IS the content's hash. + // Manifests are not — their key is the whole FILE's sha, which the hub + // cannot check without reading every chunk; readers verify by reassembly. if blob, isBlob := strings.CutPrefix(key, "blobs/"); isBlob && blob != sum { http.Error(w, "content does not hash to its key", http.StatusBadRequest) return } + if chunk, isChunk := strings.CutPrefix(key, "chunks/"); isChunk && chunk != sum { + http.Error(w, "content does not hash to its key", http.StatusBadRequest) + return + } ops, err := journalOps(key, tmp) if err != nil { // The body is the client's, so everything journalOps can object to is @@ -576,6 +601,82 @@ func (s *Server) handleStorePut(v *volume, w http.ResponseWriter, r *http.Reques return } } + // A manifest is the one member-writable object that is neither + // content-addressed nor hash-checkable at ingest, so it is WRITE-ONCE: + // overwriting one was the only way a member could re-point an existing + // file's chunk list after the fact. Re-putting identical bytes stays a + // no-op — the retry after an interrupted push (chunks and manifest up, + // journal not yet) must keep working. + if strings.HasPrefix(key, "manifests/") { + // Every chunk a manifest names must already be in the store. This is + // what makes "a manifest exists ⟹ its chunks exist" an INVARIANT + // rather than an honest-client convention: the manifest key is the + // one member-writable object that is not content-addressed, and a + // member who can read a file can publish its true chunk hashes + // without uploading a byte — poisoning the slot under a whole-pushed + // blob so a later honest push skips chunks that do not exist. The + // honest client writes chunks before the manifest, so this always + // passes for it; a refusal falls back to a whole-blob push on the + // client (pushChunked), so even a race costs one full upload, never + // a wedge. + var man struct { + Chunks []struct { + H string `json:"h"` + } `json:"chunks"` + } + if err := json.NewDecoder(io.LimitReader(tmp, 8<<20)).Decode(&man); err != nil { + http.Error(w, "unreadable manifest body", http.StatusBadRequest) + return + } + if _, err := tmp.Seek(0, io.SeekStart); err != nil { + storageErr(w, http.StatusBadGateway, "could not store the object", err) + return + } + for _, c := range man.Chunks { + if !blobRe.MatchString(c.H) { + http.Error(w, "manifest names an invalid chunk", http.StatusBadRequest) + return + } + ok, err := rs.Backend.Exists(r.Context(), "chunks/"+c.H) + if err != nil { + storageErr(w, http.StatusBadGateway, "could not verify the manifest's chunks", err) + return + } + if !ok { + http.Error(w, "manifest names a chunk the store does not hold", http.StatusBadRequest) + return + } + } + // Exists first, then Get: `if Get succeeds, compare` fails OPEN on a + // transient storage error — exactly the flakiness that must not + // reopen the overwrite door this guard exists to close. + exists, eerr := rs.Backend.Exists(r.Context(), key) + if eerr != nil { + storageErr(w, http.StatusBadGateway, "could not check the stored manifest", eerr) + return + } + if exists { + rc, gerr := rs.Backend.Get(r.Context(), key) + if gerr != nil { + storageErr(w, http.StatusBadGateway, "could not read the stored manifest", gerr) + return + } + stored, rerr := io.ReadAll(io.LimitReader(rc, 8<<20)) + rc.Close() + fresh, ferr := io.ReadAll(tmp) + if _, serr := tmp.Seek(0, io.SeekStart); rerr != nil || ferr != nil || serr != nil { + storageErr(w, http.StatusBadGateway, "could not compare the stored manifest", errors.Join(rerr, ferr, serr)) + return + } + if !bytes.Equal(stored, fresh) { + http.Error(w, "a manifest is write-once; this key already holds a different one", + http.StatusConflict) + return + } + writeJSON(w, map[string]any{"ok": true}) + return + } + } if err := rs.Backend.Put(r.Context(), key, tmp, size); err != nil { storageErr(w, http.StatusBadGateway, "could not store the object", err) return diff --git a/web/docs/src/content/docs/reference/cli.md b/web/docs/src/content/docs/reference/cli.md index 8cd2f09..7784505 100644 --- a/web/docs/src/content/docs/reference/cli.md +++ b/web/docs/src/content/docs/reference/cli.md @@ -264,7 +264,11 @@ creates a NEW project (it never joins an existing one by name — pass `--name` if the archive's name is taken), and the destination hub needs uploads enabled. A single file in the archive may spool at most 256 MiB to local disk during import; `--max-blob` raises that if the project really holds a bigger -file. Shares, +file. Import refuses an archive whose journals reference content the archive +does not hold — the shape a pre-delta-sync `bdrive export` produces against a +newer hub, where large files live as chunks the old binary doesn't know to +collect; re-export with a current `bdrive`, or pass `--allow-incomplete` to +import anyway (the missing files are listed and stay missing). Shares, invite links, and read-heat stay behind (they belong to the hub, not the project store). Step-by-step walkthrough: [Migrate between hubs](/reference/migration/). diff --git a/web/docs/src/content/docs/reference/project-files.md b/web/docs/src/content/docs/reference/project-files.md index b30c426..ddfd29d 100644 --- a/web/docs/src/content/docs/reference/project-files.md +++ b/web/docs/src/content/docs/reference/project-files.md @@ -91,3 +91,11 @@ Nothing is keyed by folder path, which is why moves and renames are free. ``` Also here for a running project: `daemon.pid` and `daemon.log`. + +The hub's storage adds two key classes the local store never holds: files +larger than 4 MiB travel as content-defined `chunks/` pieces plus a +`manifests/` chunk list keyed by the whole file's hash (delta sync — +a small edit to a large file uploads roughly one chunk, not the file). Local +blobs stay whole; chunking exists only on the wire and in the hub's store, +and the hub reassembles a whole blob on demand for any client that asks for +`blobs/`, so older clients keep working unchanged.