External/bichon
1
0
Fork 0
mirror of https://github.com/rustmailer/bichon.git synced 2026-08-03 07:48:34 +02:00
Code Issues Packages Projects Releases Wiki Activity

Updated De‐duplication (markdown)

rustmailer
2026-05-21 09:09:14 +08:00
parent 19cc807f12
commit 4cccf6702d
1 changed files with 146 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
De‐duplication.md
+146 -35
@@ -1,46 +1,157 @@
## Overview
Bichon is designed with a **Single-Instance Storage** philosophy at the **Account Level**. The primary goal is to ensure that any unique email is stored only once within an account, regardless of how many folders it appears in.
Bichon handles deduplication through several independent mechanisms, each operating at a different point in the pipeline. Together they ensure that no matter how many times the same email is downloaded, moved across folders, or imported, only one canonical copy persists at the storage level.
## 1. Storage Architecture
Bichon decouples email information into two separate storage layers to optimize both search performance and storage integrity:
## BLAKE3 content hashing
* **Metadata Index**: Extracted headers and properties (e.g., Subject, Sender, Date) are stored in a dedicated index for rapid querying.
* **Blob Storage**: The full, raw email content (MIME) is stored in a separate directory.
Every deduplication decision starts with `compute_content_hash()` at `crates/core/src/utils/mod.rs:350`:
### The Primary Key
Both the Metadata and the Full Content use a unique key derived from the email's **`Message-ID`**. This serves as the "Primary Key" for the lifecycle of that email within the system.
```rust
pub fn compute_content_hash(content: &[u8]) -> String {
let hash = blake3::hash(content);
hash.to_hex().to_string()
}
```
## 2. De-duplication Logic: "Delete-then-Write"
To maximize **write efficiency**, Bichon does not perform a traditional "update" or "check-if-exists" read operation. Instead, it follows a strict sequence:
- Uses **BLAKE3-256**, producing a 64-character hex string.
- Computed over the **full raw bytes** of the email body for the message-level fingerprint.
- Computed over each attachment's raw bytes separately for per-attachment fingerprints.
- The hash is a stable identity for the content — it does not depend on IMAP UIDs or folder location.
1. **Extract**: Identify the `Message-ID` of the incoming email.
2. **Purge**: Immediately delete any existing Metadata and Full Content associated with that `Message-ID`.
3. **Insert**: Write the new Metadata and Full Content to the storage layers.
## Fjall blob store: immediate key-level dedup
This "Last-In-Wins" strategy ensures that the database remains clean and that write operations are not slowed down by complex conflict resolution.
**Location:** `crates/core/src/store/blob.rs:58-87`
## 3. Practical Implications
This is the first line of defense — it fires immediately on every write to persistent blob storage.
### Folder Synchronization (e.g., The Trash Scenario)
Because Bichon maintains only one copy per `Message-ID` at the account level, moving emails between folders results in an "overwrite" rather than a "duplicate":
```mermaid
flowchart LR
A[New email / attachment] --> B[Compute BLAKE3 hash]
B --> C{Fjall Keyspace<br/>contains_key?}
C -->|No| D[Insert blob]
C -->|Yes| E[Silently skip]
```
* **Example**:
1. Email **X** is synced from the `Inbox`. Bichon stores it.
2. You move Email **X** to the `Trash` folder in your mail client.
3. When Bichon syncs the `Trash` folder, it sees Email **X** again.
4. Bichon deletes the "Inbox version" of Email **X** and writes the "Trash version."
* **Outcome**: The email appears to have moved. While Bichon's primary intent is simply to ensure only one copy exists, the side effect is a clean representation of the email's latest state.
- `BLOB_MANAGER` manages two Fjall keyspaces: `"email"` and `"attachments"`.
- Before every insert, it calls `contains_key()` on the content hash.
- If the hash already exists the write is a no-op — no overwrite, no error.
- This is fully automatic and requires no background task.
### Bulk Imports via `nosync`
When using the `nosync` tool to import large datasets:
* If the source data contains duplicate emails (same `Message-ID`), the version that is **processed last** will be the one that persists in Bichon.
* This ensures that no matter how many times a duplicate is imported, the storage footprint does not grow unnecessarily.
Net effect: if the same email is downloaded twice (e.g. it appears in two folders), Fjall stores only one copy of the raw bytes. Both index entries point to the same content hash key.
## 4. Summary Table
| Feature | Implementation |
| :--- | :--- |
| **Deduplication Scope** | Account Level |
| **Primary Key** | Derived from `Message-ID` |
| **Storage Strategy** | Separate Metadata (Index) vs. Full Content (Filesystem) |
| **Write Pattern** | Atomic Delete-then-Write |
| **Design Goal** | High write throughput & Single-instance storage |
## Tantivy index: periodic dedup task
**Location:** `crates/core/src/store/tantivy/dedup.rs` (733 lines)
A background task that runs every **12 hours**, scanning the full-text index and removing duplicate documents.
### Grouping key
Duplicates are identified by `(mailbox_id, content_hash)`:
```
Mailbox A, hash 0xabc... → keep at most 1 copy
Mailbox B, hash 0xabc... → allowed (different mailbox)
```
Key design decisions:
- Only documents with the **same content hash inside the same mailbox** are considered duplicates.
- **Cross-mailbox duplicates are preserved** — a user may intentionally archive the same email into multiple folders.
- Documents belonging to different accounts are **never deduplicated** against each other (the task processes accounts one at a time).
### Retention policy: keep the newest `ingest_at`
When duplicates are found, the copy with the **highest `ingest_at`** timestamp is kept; the rest are deleted.
The rationale is tied to UIDVALIDITY resets: after a server reassigns UIDs, the stale copy carries an outdated UID. If the stale copy were kept, UID-based incremental sync would see the new UIDs as missing and re-download emails already present. Keeping the most recently ingested copy (which bears the new UID) prevents this.
### Cascading attachment cleanup
When a duplicate email is removed, its attachments are cleaned up from the attachment index:
```
1. Delete the duplicate email doc from the email index by f_id term
2. Delete attachment docs from the attachment index by f_envelope_id term (matching the removed email's f_id)
3. Commit both indexes per account
```
## UIDVALIDITY-triggered mailbox rebuild
**Location:** `crates/core/src/cache/imap/download/flow.rs:424-531`
This is not deduplication in the traditional sense — it prevents data inconsistency by detecting stale data and purging it.
### How it works
On every sync cycle, `reconcile_mailboxes()` compares the local and remote `uid_validity` for each mailbox:
1. **UIDVALIDITY unchanged** → perform incremental sync (fetch only UIDs > local max UID).
2. **UIDVALIDITY changed** → the mailbox data is considered invalid. The system:
- Calls `rebuild_mailbox_cache()` or `rebuild_mailbox_cache_by_date()`.
- Deletes all existing envelope documents for the affected mailbox.
- Re-downloads the entire mailbox from the server.
## Reference-counted blob cleanup
**Location:** `crates/core/src/store/tantivy/envelope.rs:775-851`
When envelopes are deleted (whether by user action or UIDVALIDITY rebuild), the corresponding blobs are not deleted outright. Instead, the system checks whether any remaining document still references each content hash:
```
Delete envelopes
└→ collect_content_hashes() // gather every content_hash from the docs being deleted
└→ cleanup_unused_content()
├→ For each email content hash: Count query → refs == 0? → delete from Fjall
└→ For each attachment content hash: Count query → refs == 0? → delete from Fjall
```
- Uses Tantivy's `Count` query to check whether each hash is still referenced by any surviving document.
- A blob is removed from Fjall only when **no document** references it.
- This guarantees that a blob shared by multiple index entries is not deleted when one of those entries is removed.
## Edge cases
### Email moved between folders
When an email is moved from Inbox to Archive:
1. The IMAP server presents the same message (same Message-ID / content) in both folders.
2. Bichon downloads it twice → two index documents in Tantivy (different `mailbox_id`).
3. Fjall stores only one copy of the blob (immediate key-level dedup).
4. The periodic dedup task does **not** merge them (different `mailbox_id`).
5. Both index documents point to the same blob → reference-counted cleanup protects the blob when either one is deleted.
### UIDVALIDITY reset
1. The IMAP server changes UIDVALIDITY.
2. `reconcile_mailboxes()` detects the change → deletes stale data → performs a full re-download.
3. If detection fails (e.g. server bug), the periodic dedup task catches the resulting duplicates within the same mailbox and keeps the most recent `ingest_at` copy.
### Same email under different accounts
Two accounts subscribed to the same mailing list receive identical content → **never deduplicated against each other**. Each user has an independent archive.
### EML import / SMTP ingestion
- Import and SMTP paths use `uid=0`.
- They go through the same pipeline: BLAKE3 hash → Fjall immediate dedup → Tantivy periodic dedup.
- If the same email is later synced via IMAP (with a real UID), the periodic dedup task keeps the more recently ingested copy.
## Summary
| Mechanism | Where | When | Scope |
|-----------|-------|------|-------|
| BLAKE3-256 hashing | `utils/mod.rs:350` | Every ingestion | Per-byte |
| Fjall `contains_key` check | `blob.rs:58-87` | **Immediate** (every write) | Blob storage, global |
| Periodic index dedup | `dedup.rs` | **Every 12 hours** | Tantivy index, per `(mailbox_id, content_hash)` |
| UIDVALIDITY rebuild | `flow.rs:451` | **Every sync cycle** | Entire mailbox |
| Reference-counted blob cleanup | `envelope.rs:813-851` | **On envelope deletion** | Fjall blobs, global |
| Migration buffer dedup | `migrate/store.rs:494` | **During migration** | In-memory buffers |
## Dedup does not mean "store once"
A core design principle: **the same email content may exist as multiple index entries across different mailboxes.** The goals are:
1. **Storage efficiency** — the blob layer ensures identical content occupies disk space only once.
2. **Index cleanliness** — no duplicate entries within the same mailbox (no repeated search results).
3. **Cross-mailbox fidelity** — when a user files the same email into multiple folders, it remains searchable from each one.
This balances **storage efficiency** against **user visibility**: blob dedup saves space, while the index allows cross-mailbox coexistence so every folder perspective is complete.