All data is keyed by a 32-byte content hash — identical content is stored only once. The caller (Bichon) strips attachments from emails and stores the "holed" raw email + attachments into this store, tracking reference counts in its own schema.

On-disk layout

<root>/
├── meta.bin              # global metadata (bincode + CRC32)
├── segments/
│   ├── 00000001.seg      # append-only segment files (≤ 256 MB each)
│   ├── 00000002.seg
│   └── ...
└── buckets/
    ├── 00.idx            # 256 bucket index files (mmap'd, binary-searchable)
    ├── 01.idx
    └── ...

Segment entry format (50-byte fixed header + variable data)

magic(4)  crc32(4)  flags(1)  codec(1)  key(32)  raw_size(4)  data_size(4)  data(*)

magic: 0xB3DB_0001 — entry boundary validation
crc32: covers everything after this field
flags: 0 = live, 1 = tombstone
codec: 0 = none, 1 = Zstd, 2 = Lz4
key: 32-byte content hash (BLAKE3 / SHA-256)
raw_size: original uncompressed size
data_size: on-disk data size (after compression)

Bucket index record format (56 bytes)

key(32)  segment_id(4)  offset(8)  data_size(4)  flags(1)  _pad(3)  crc32(4)

Every index record is CRC32-protected — a corrupted record is detected on read and never silently returned. Records are sorted by key, deduplicated (newest segment_id + offset wins), and mmap'd for zero-heap binary search. Pending writes (since the last compaction) live in a small in-memory HashMap.

Read path

get(key)
  → bucket_id = (key[0..2] as u16) % 256
  → check pending HashMap (most recent wins)
  → binary search mmap'd bucket file
  → IndexRecord CRC32 verify → (segment_id, offset, data_size)
  → pread entry from segment file
  → entry CRC32 verify → decompress → return value

Both the bucket index record and the segment entry carry independent CRC32 checksums. Corruption in one record or entry is contained — it never affects other keys.

Write path

put(key, value, codec)
  → compress value (Zstd/Lz4 if ≥ 4 KB, else store raw)
  → append entry to active segment file
  → insert IndexRecord into bucket store (append to .idx file + HashMap)
  → update metadata (indexed_up_to_offset)
  → if segment ≥ 256 MB → seal it, create new segment

Delete

Deletes are tombstones — an entry with flags=1 and empty data is appended. The bucket index maps the key to this tombstone. Read returns None. GC later reclaims the space.

Compaction & GC

Bucket compaction: when a bucket's pending HashMap grows past compact_threshold (default 10,000), the mmap + pending records are merged, sorted, deduplicated, and atomically rewritten. This keeps binary search fast.

Garbage collection: when a sealed segment's deleted-ratio exceeds gc_deleted_ratio (default 0.30), GC scans all segments to find the latest entry per key, then rewrites the target segment keeping only live entries. Tombstones and overwritten entries are dropped. Bucket indices are rebuilt afterward.

Data integrity

Every record on disk is independently checksummed:

Layer	Format	Protection
Segment entry	50-byte header + data	CRC32 covers all fields + data
Bucket index record	56 bytes	CRC32 covers key + segment_id + offset + data_size + flags
Global metadata	bincode blob	CRC32 + version header

Corruption is contained — a bad segment entry or bucket record produces an error for that key only. Compaction and recovery skip corrupt records (with a warning) rather than aborting. Bucket indices can always be fully rebuilt from segments via rebuild_from_segments.

Crash recovery

Temp files from interrupted GC are cleaned up on open.
Any segment data beyond indexed_up_to_offset is scanned and indexed into bucket files.
After appending recovered records, bucket mmaps are reloaded so they are immediately visible.
Partial writes at the tail of a segment (detected via CRC32 mismatch near EOF) are truncated.
Buckets are always repairable by re-scanning segments (rebuild_from_segments).

Background flush

Set Config.flush_interval_secs to a positive value to have a background thread fsync the active segment and save metadata periodically. This bounds recovery time after a crash at the cost of a small I/O overhead.

Config

Field	Default	Notes
`compress_threshold`	4096	Bytes; smaller values stored uncompressed
`default_codec`	Zstd	Also supports Lz4
`compression_level`	0	Zstd compression level
`compact_threshold`	10000	Pending records per bucket before auto-compact
`gc_deleted_ratio`	0.30	Trigger GC when a sealed segment exceeds this
`flush_interval_secs`	0	0 = disabled; ≥ 5 for periodic background fsync

Basic usage

use bichon_blob::{Codec, Config, Engine};

let engine = Engine::open(path, Config::default())?;

// Store
let hash = blake3::hash(b"email body").into();
engine.put(hash, b"email body", Codec::Zstd)?;

// Retrieve
let value = engine.get(&hash)?; // Some(Vec<u8>) or None

// Delete (caller must track refcounts)
engine.delete(&hash)?;

// Batch operations
engine.put_batch(&[(hash1, data1, Codec::Zstd), (hash2, data2, Codec::Lz4)])?;
engine.delete_batch(&[hash1, hash2])?;

// GC
engine.gc_if_needed()?;

// Clean shutdown
engine.shutdown()?;