mirror of
https://github.com/rustmailer/bichon.git
synced 2026-08-03 07:48:34 +02:00
Created Bichon Data Migration: v0.3.7 → v1.0 (markdown)
rustmailer
@@ -0,0 +1,176 @@
|
|||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Bichon v1.0.0 introduces a new storage architecture that separates email indexes (Tantivy)
|
||||||
|
from raw message and attachment data (Fjall). Migration is required for all v0.3.7 installations.
|
||||||
|
The migration is **non-destructive** — legacy files are left intact.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Directory Structure
|
||||||
|
|
||||||
|
### Before (v0.3.7)
|
||||||
|
|
||||||
|
```
|
||||||
|
<bichon-root-dir>/
|
||||||
|
├── envelope/ # Tantivy index — envelope metadata
|
||||||
|
└── eml/ # Tantivy index — raw message bodies + attachments
|
||||||
|
```
|
||||||
|
|
||||||
|
### After (v1.0.0)
|
||||||
|
|
||||||
|
```
|
||||||
|
<bichon-root-dir>/
|
||||||
|
├── envelope/ # (legacy, untouched)
|
||||||
|
├── eml/ # (legacy, untouched)
|
||||||
|
├── bichon-indices/
|
||||||
|
│ ├── mail_metadata/ # Tantivy index — email metadata + full-text
|
||||||
|
│ └── attachment_metadata/ # Tantivy index — attachment metadata
|
||||||
|
└── bichon-storage/
|
||||||
|
├── keyspaces/ # Fjall keyspaces — email bodies & attachment blobs
|
||||||
|
├── 0.jnl # Fjall journal file
|
||||||
|
├── lock # Fjall lock file
|
||||||
|
└── version # Fjall version file
|
||||||
|
```
|
||||||
|
|
||||||
|
If `--bichon-index-dir` or `--bichon-data-dir` were customized, `bichon-indices/` and
|
||||||
|
`bichon-storage/` are created under those directories respectively.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Running the Migration
|
||||||
|
|
||||||
|
Run `bichon-admin` and select:
|
||||||
|
```Migrate Legacy v0.3.7 Storage to v1.0.0```
|
||||||
|
|
||||||
|
> **Note:** bichon-admin v1.0.0 has a known memory issue during migration of large mailboxes.
|
||||||
|
> Please use bichon-admin v1.0.1 which includes an emergency fix. The menu option still shows
|
||||||
|
> v1.0.0 in the label — this is a cosmetic issue, simply select it as normal.
|
||||||
|
|
||||||
|
|
||||||
|
The tool will prompt for:
|
||||||
|
|
||||||
|
1. `--bichon-root-dir` — same value used by the old server
|
||||||
|
2. `--bichon-index-dir` — leave blank to use default (`<root>/envelope`)
|
||||||
|
3. `--bichon-data-dir` — leave blank to use default (`<root>/eml`)
|
||||||
|
4. **Batch size** — controls memory usage per batch (see below)
|
||||||
|
|
||||||
|
Migration proceeds in two steps:
|
||||||
|
|
||||||
|
**Step 1 — Metadata migration** (usually completes in seconds):
|
||||||
|
|
||||||
|
> Mail Settings done (0 items)
|
||||||
|
> Accounts done (1 items)
|
||||||
|
> OAuth2 Entities done (0 items)
|
||||||
|
> OAuth2 Pending done (0 items)
|
||||||
|
> OAuth2 Access Tokens done (0 items)
|
||||||
|
> Proxy Settings done (0 items)
|
||||||
|
> User Roles done (5 items)
|
||||||
|
> Users done (1 items)
|
||||||
|
> Access Tokens done (2 items)
|
||||||
|
> Mailboxes done (1 items)
|
||||||
|
Metadata migration completed successfully.
|
||||||
|
|
||||||
|
**Step 2 — Email index and blob migration**: proceeds segment by segment.
|
||||||
|
Progress is displayed for both the envelope scanning phase (Phase 1)
|
||||||
|
and the ingestion phase (Phase 2).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Memory Usage
|
||||||
|
|
||||||
|
Batch size directly controls peak memory consumption. Choose based on your available RAM
|
||||||
|
and average email size (larger attachments = higher memory per batch).
|
||||||
|
|
||||||
|
| Batch Size | Approx. RAM | Notes |
|
||||||
|
|------------|-------------|------------------------------|
|
||||||
|
| 1000 | ~500 MB | Safe for low-memory servers |
|
||||||
|
| 3000 | ~1 GB | Recommended default |
|
||||||
|
| 5000 | ~2 GB | Faster on high-memory servers|
|
||||||
|
|
||||||
|
After each batch is ingested, Tantivy performs a commit to flush the in-memory buffer to disk.
|
||||||
|
This commit can take **2–3 minutes** depending on batch size and disk speed — this is normal,
|
||||||
|
please wait patiently. Segment merging is deferred to the end of migration to avoid
|
||||||
|
mid-process stalls, and may also take several minutes once all batches are complete.
|
||||||
|
|
||||||
|
For large mailboxes (hundreds of GB), the full migration can take **several hours**.
|
||||||
|
Reducing the batch size will lower memory usage but increase the number of commits,
|
||||||
|
which may extend the total migration time.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Re-running the Migration
|
||||||
|
|
||||||
|
If migration needs to be re-run from scratch, delete the new output directories before
|
||||||
|
running the tool again.
|
||||||
|
|
||||||
|
**Default paths** (when no custom dirs were configured):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rm -rf <bichon-root-dir>/bichon-indices
|
||||||
|
rm -rf <bichon-root-dir>/bichon-storage
|
||||||
|
rm -rf <bichon-root-dir>/memdb
|
||||||
|
```
|
||||||
|
|
||||||
|
**Custom paths** (when `--bichon-index-dir` or `--bichon-data-dir` were configured):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rm -rf <bichon-index-dir>/bichon-indices
|
||||||
|
rm -rf <bichon-data-dir>/bichon-storage
|
||||||
|
rm -rf <bichon-root-dir>/memdb # memdb is always under bichon-root-dir
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Warning:** Only delete `bichon-indices`, `bichon-storage` and `memdb`. Do **not** delete
|
||||||
|
> or modify the legacy `envelope/` and `eml/` directories — they are the migration source and
|
||||||
|
> must remain intact. Deleting them will make re-migration impossible without restoring from backup.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
## After a Successful Migration
|
||||||
|
|
||||||
|
1. **Start the v1.0.x server** — it will automatically detect the new storage layout and skip migration prompts.
|
||||||
|
2. **Verify** that emails and attachments are accessible and searchable.
|
||||||
|
3. **Clean up legacy files** once you have confirmed the migration is complete and the new server is stable:
|
||||||
|
|
||||||
|
**Default paths:**
|
||||||
|
```bash
|
||||||
|
rm -rf <bichon-root-dir>/envelope
|
||||||
|
rm -rf <bichon-root-dir>/eml
|
||||||
|
rm -f <bichon-root-dir>/mailbox.db
|
||||||
|
rm -f <bichon-root-dir>/meta.db
|
||||||
|
```
|
||||||
|
|
||||||
|
If `--bichon-index-dir` was configured, the directory contains a mix of legacy Tantivy
|
||||||
|
segment files (`.fast`, `.fieldnorm`, `.idx`, `.pos`, `.store`, `.term`, `meta.json`,
|
||||||
|
`.managed.json`, `.tantivy-*.lock`) alongside the new `bichon-indices/` subdirectory.
|
||||||
|
Remove all the legacy Tantivy files, but keep `bichon-indices/` intact.
|
||||||
|
|
||||||
|
Same applies to `--bichon-data-dir` — remove the legacy Tantivy files, keep `bichon-storage/`.
|
||||||
|
|
||||||
|
```
|
||||||
|
<bichon-index-dir>/
|
||||||
|
├── bichon-indices/ ← KEEP — new index
|
||||||
|
├── .managed.json ← delete
|
||||||
|
├── .tantivy-meta.lock ← delete
|
||||||
|
├── .tantivy-writer.lock ← delete
|
||||||
|
├── meta.json ← delete
|
||||||
|
├── 0d4e7e91a66d44e58...fast ← delete
|
||||||
|
├── 0d4e7e91a66d44e58...store← delete
|
||||||
|
├── 0d4e7e91a66d44e58...term ← delete
|
||||||
|
└── ... ← delete all other segment files
|
||||||
|
|
||||||
|
<bichon-data-dir>/
|
||||||
|
├── bichon-storage/ ← KEEP — new storage
|
||||||
|
├── .managed.json ← delete
|
||||||
|
├── meta.json ← delete
|
||||||
|
└── ... ← delete all other segment files
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
> **Warning:** Double-check paths before running any `rm` command. If `--bichon-index-dir`
|
||||||
|
> or `--bichon-data-dir` point to directories that also contain `bichon-indices` or
|
||||||
|
> `bichon-storage` as subdirectories, remove only the legacy contents, not the parent directory.
|
||||||
Reference in New Issue
Block a user