All data is keyed by a 32-byte content hash — identical content is stored only once. The caller (Bichon) strips attachments from emails and stores the "holed" raw email + attachments into this store, tracking reference counts in its own schema.

On-disk layout

<root>/
├── meta.bin              # global metadata (bincode + CRC32)
├── index.redb            # key → (segment_id, offset, size) index (redb B-tree)
├── segments/
│   ├── 00000001.seg      # append-only segment files (≤ 1 GB each)
│   ├── 00000002.seg
│   └── ...

Segment entry format (50-byte fixed header + variable data)

magic(4)  crc32(4)  flags(1)  codec(1)  key(32)  raw_size(4)  data_size(4)  data(*)

magic: 0xB3DB_0001 — entry boundary validation
crc32: covers everything after this field
flags: 0 = live, 1 = tombstone
codec: 0 = none, 1 = Zstd, 2 = Lz4
key: 32-byte content hash (BLAKE3 / SHA-256)
raw_size: original uncompressed size
data_size: on-disk data size (after compression)

Index store

The key → (segment_id, offset, data_size, flags) mapping is stored in a single redb database (index.redb). redb provides:

B-tree + mmap: O(log N) point lookups with zero heap allocation — pages are faulted in on demand.
ACID transactions: every index write is durable and atomic.
Crash recovery: handled transparently by redb's WAL — no manual reload or rebuild logic.
O(1) startup: only the B-tree root page is read at open time.

Records are stored as fixed-size 56-byte blobs, each carrying an internal CRC32 checksum.

Read path

get(key)
  → index_store.get(key)           # redb B-tree lookup, zero-copy
  → IndexRecord CRC32 verify       # (segment_id, offset, data_size, flags)
  → pread entry from segment file
  → entry CRC32 verify → decompress → return value

The index record and the segment entry carry independent CRC32 checksums. Corruption in one record or entry is contained — it never affects other keys.

Write path

put(key, value, codec)
  → compress value (Zstd/Lz4 if ≥ 4 KB, else store raw)
  → append entry to active segment file
  → insert IndexRecord into redb (single write txn)
  → update metadata (indexed_up_to_offset)
  → if segment ≥ 1 GB → seal it, create new segment

Delete

Deletes are tombstones — an entry with flags=1 and empty data is appended to the active segment. Before writing the tombstone, the existing index record is consulted to increment deleted_bytes on the original segment (the one that holds the live data). This drives the GC threshold.

delete(key)
  → index_store.get(key) → find original (segment_id, data_size)
  → original_segment.deleted_bytes += data_size
  → recompute deleted_ratio on original segment
  → append tombstone entry to active segment
  → insert tombstone IndexRecord into redb
  → index_store.get(key) now returns None

GC

Segment GC: two-phase, driven by the deleted_ratio tracked per segment.

Trigger

Background: the blob-gc thread wakes up every gc_interval_secs (default 300s), checks whether any sealed segment's deleted_ratio ≥ gc_deleted_ratio (default 0.30), and runs GC on the worst segment if so.
Manual: engine.gc_if_needed() or engine.gc().

Phase 1 — scan & compact (read-only, no write lock)

Pick the sealed segment with the highest deleted_ratio.
Scan only that segment's entries.
For each entry, ask the index: "is this entry still the latest version for its key?"
- If the index points to this exact (segment_id, offset) → keep, write to a temp segment file.
- If the index points elsewhere (overwritten by a later segment, or a tombstone) → skip (stale).
Fsync the temp file.

Phase 1 holds only the read lock — put / delete continue uninterrupted.

Phase 2 — commit & update index (write lock)

Atomically rename the temp file over the original segment.
Batch-insert new IndexRecords (now at new offsets) into redb. Old records with the same key are naturally overwritten.
Reset the segment's deleted_bytes and deleted_ratio to zero.
Persist metadata.

Phase 2 holds the write lock, but is fast — no full segment scan, no full index rebuild.

Data integrity

Every record on disk is independently checksummed:

Layer	Format	Protection
Segment entry	50-byte header + data	CRC32 covers all fields + data
Index record	56 bytes	CRC32 covers key + segment_id + offset + data_size + flags
Global metadata	bincode blob	CRC32 + version header

Corruption is contained — a bad segment entry or index record produces an error for that key only. Recovery and GC skip corrupt records (with a warning) rather than aborting. The index is backed by redb's B-tree which maintains its own internal integrity.

Crash recovery

Temp files from interrupted GC are cleaned up on open.
Any segment data beyond indexed_up_to_offset is scanned and inserted into the index.
Partial writes at the tail of a segment (detected via CRC32 mismatch near EOF) are truncated.
redb's WAL ensures the index is always consistent — no manual reload or rebuild needed.

Background threads

Set Config.flush_interval_secs and Config.gc_interval_secs to positive values to enable periodic background work:

Thread	Config	Default	What it does
`blob-flush`	`flush_interval_secs`	`0` (off)	Fsync the active segment and save metadata
`blob-gc`	`gc_interval_secs`	`0` (off)	Check deleted-ratio, compact one segment if needed

The two threads are independent — a long GC run never blocks fsync.

Config

Field	Default	Notes
`compress_threshold`	4096	Bytes; smaller values stored uncompressed
`default_codec`	Zstd	Also supports Lz4
`compression_level`	0	Zstd compression level
`gc_deleted_ratio`	0.30	Trigger GC when a sealed segment exceeds this
`flush_interval_secs`	0	0 = disabled; ≥ 5 for periodic background fsync
`gc_interval_secs`	0	0 = disabled; ≥ 10 for periodic background GC

Basic usage

use bichon_blob::{Codec, Config, Engine};

let engine = Engine::open(path, Config::default())?;

// Store
let hash = blake3::hash(b"email body").into();
engine.put(hash, b"email body", Codec::Zstd)?;

// Retrieve
let value = engine.get(&hash)?; // Some(Vec<u8>) or None

// Delete (caller must track refcounts)
engine.delete(&hash)?;

// Batch operations
engine.put_batch(&[(hash1, data1, Codec::Zstd), (hash2, data2, Codec::Lz4)])?;
engine.delete_batch(&[hash1, hash2])?;

// GC
engine.gc_if_needed()?;

// Clean shutdown
engine.shutdown()?;