From 1adc5048459a63e2226700bdc56b2392bd711742 Mon Sep 17 00:00:00 2001 From: rustmailer Date: Sat, 16 May 2026 22:52:05 +0800 Subject: [PATCH] =?UTF-8?q?Created=20Bichon=20Data=20Migration:=20v0.3.7?= =?UTF-8?q?=20=E2=86=92=20v1.0=20(markdown)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Bichon-Data-Migration:-v0.3.7-→-v1.0.md | 176 ++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 Bichon-Data-Migration:-v0.3.7-→-v1.0.md diff --git a/Bichon-Data-Migration:-v0.3.7-→-v1.0.md b/Bichon-Data-Migration:-v0.3.7-→-v1.0.md new file mode 100644 index 0000000..f235539 --- /dev/null +++ b/Bichon-Data-Migration:-v0.3.7-→-v1.0.md @@ -0,0 +1,176 @@ + +## Overview + +Bichon v1.0.0 introduces a new storage architecture that separates email indexes (Tantivy) +from raw message and attachment data (Fjall). Migration is required for all v0.3.7 installations. +The migration is **non-destructive** — legacy files are left intact. + +--- + +## Directory Structure + +### Before (v0.3.7) + +``` +/ +├── envelope/ # Tantivy index — envelope metadata +└── eml/ # Tantivy index — raw message bodies + attachments +``` + +### After (v1.0.0) + +``` +/ +├── envelope/ # (legacy, untouched) +├── eml/ # (legacy, untouched) +├── bichon-indices/ +│ ├── mail_metadata/ # Tantivy index — email metadata + full-text +│ └── attachment_metadata/ # Tantivy index — attachment metadata +└── bichon-storage/ + ├── keyspaces/ # Fjall keyspaces — email bodies & attachment blobs + ├── 0.jnl # Fjall journal file + ├── lock # Fjall lock file + └── version # Fjall version file +``` + +If `--bichon-index-dir` or `--bichon-data-dir` were customized, `bichon-indices/` and +`bichon-storage/` are created under those directories respectively. + +--- + +## Running the Migration + +Run `bichon-admin` and select: + ```Migrate Legacy v0.3.7 Storage to v1.0.0``` + +> **Note:** bichon-admin v1.0.0 has a known memory issue during migration of large mailboxes. +> Please use bichon-admin v1.0.1 which includes an emergency fix. The menu option still shows +> v1.0.0 in the label — this is a cosmetic issue, simply select it as normal. + + +The tool will prompt for: + +1. `--bichon-root-dir` — same value used by the old server +2. `--bichon-index-dir` — leave blank to use default (`/envelope`) +3. `--bichon-data-dir` — leave blank to use default (`/eml`) +4. **Batch size** — controls memory usage per batch (see below) + +Migration proceeds in two steps: + +**Step 1 — Metadata migration** (usually completes in seconds): + + > Mail Settings done (0 items) + > Accounts done (1 items) + > OAuth2 Entities done (0 items) + > OAuth2 Pending done (0 items) + > OAuth2 Access Tokens done (0 items) + > Proxy Settings done (0 items) + > User Roles done (5 items) + > Users done (1 items) + > Access Tokens done (2 items) + > Mailboxes done (1 items) + Metadata migration completed successfully. + +**Step 2 — Email index and blob migration**: proceeds segment by segment. +Progress is displayed for both the envelope scanning phase (Phase 1) +and the ingestion phase (Phase 2). + +--- + +## Memory Usage + +Batch size directly controls peak memory consumption. Choose based on your available RAM +and average email size (larger attachments = higher memory per batch). + +| Batch Size | Approx. RAM | Notes | +|------------|-------------|------------------------------| +| 1000 | ~500 MB | Safe for low-memory servers | +| 3000 | ~1 GB | Recommended default | +| 5000 | ~2 GB | Faster on high-memory servers| + +After each batch is ingested, Tantivy performs a commit to flush the in-memory buffer to disk. +This commit can take **2–3 minutes** depending on batch size and disk speed — this is normal, +please wait patiently. Segment merging is deferred to the end of migration to avoid +mid-process stalls, and may also take several minutes once all batches are complete. + +For large mailboxes (hundreds of GB), the full migration can take **several hours**. +Reducing the batch size will lower memory usage but increase the number of commits, +which may extend the total migration time. + +--- + +## Re-running the Migration + +If migration needs to be re-run from scratch, delete the new output directories before +running the tool again. + +**Default paths** (when no custom dirs were configured): + +```bash +rm -rf /bichon-indices +rm -rf /bichon-storage +rm -rf /memdb +``` + +**Custom paths** (when `--bichon-index-dir` or `--bichon-data-dir` were configured): + +```bash +rm -rf /bichon-indices +rm -rf /bichon-storage +rm -rf /memdb # memdb is always under bichon-root-dir +``` + +> **Warning:** Only delete `bichon-indices`, `bichon-storage` and `memdb`. Do **not** delete +> or modify the legacy `envelope/` and `eml/` directories — they are the migration source and +> must remain intact. Deleting them will make re-migration impossible without restoring from backup. + +--- + + +## After a Successful Migration + +1. **Start the v1.0.x server** — it will automatically detect the new storage layout and skip migration prompts. +2. **Verify** that emails and attachments are accessible and searchable. +3. **Clean up legacy files** once you have confirmed the migration is complete and the new server is stable: + +**Default paths:** +```bash +rm -rf /envelope +rm -rf /eml +rm -f /mailbox.db +rm -f /meta.db +``` + +If `--bichon-index-dir` was configured, the directory contains a mix of legacy Tantivy +segment files (`.fast`, `.fieldnorm`, `.idx`, `.pos`, `.store`, `.term`, `meta.json`, +`.managed.json`, `.tantivy-*.lock`) alongside the new `bichon-indices/` subdirectory. +Remove all the legacy Tantivy files, but keep `bichon-indices/` intact. + +Same applies to `--bichon-data-dir` — remove the legacy Tantivy files, keep `bichon-storage/`. + +``` +/ +├── bichon-indices/ ← KEEP — new index +├── .managed.json ← delete +├── .tantivy-meta.lock ← delete +├── .tantivy-writer.lock ← delete +├── meta.json ← delete +├── 0d4e7e91a66d44e58...fast ← delete +├── 0d4e7e91a66d44e58...store← delete +├── 0d4e7e91a66d44e58...term ← delete +└── ... ← delete all other segment files + +/ +├── bichon-storage/ ← KEEP — new storage +├── .managed.json ← delete +├── meta.json ← delete +└── ... ← delete all other segment files +``` + + + + + +> **Warning:** Double-check paths before running any `rm` command. If `--bichon-index-dir` +> or `--bichon-data-dir` point to directories that also contain `bichon-indices` or +> `bichon-storage` as subdirectories, remove only the legacy contents, not the parent directory.