mirror of
https://github.com/rustmailer/bichon.git
synced 2026-08-03 07:48:34 +02:00
Created Bichon Data Migration: v0.3.7 → v1.0 (markdown)
rustmailer
@@ -0,0 +1,176 @@
|
||||
|
||||
## Overview
|
||||
|
||||
Bichon v1.0.0 introduces a new storage architecture that separates email indexes (Tantivy)
|
||||
from raw message and attachment data (Fjall). Migration is required for all v0.3.7 installations.
|
||||
The migration is **non-destructive** — legacy files are left intact.
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
### Before (v0.3.7)
|
||||
|
||||
```
|
||||
<bichon-root-dir>/
|
||||
├── envelope/ # Tantivy index — envelope metadata
|
||||
└── eml/ # Tantivy index — raw message bodies + attachments
|
||||
```
|
||||
|
||||
### After (v1.0.0)
|
||||
|
||||
```
|
||||
<bichon-root-dir>/
|
||||
├── envelope/ # (legacy, untouched)
|
||||
├── eml/ # (legacy, untouched)
|
||||
├── bichon-indices/
|
||||
│ ├── mail_metadata/ # Tantivy index — email metadata + full-text
|
||||
│ └── attachment_metadata/ # Tantivy index — attachment metadata
|
||||
└── bichon-storage/
|
||||
├── keyspaces/ # Fjall keyspaces — email bodies & attachment blobs
|
||||
├── 0.jnl # Fjall journal file
|
||||
├── lock # Fjall lock file
|
||||
└── version # Fjall version file
|
||||
```
|
||||
|
||||
If `--bichon-index-dir` or `--bichon-data-dir` were customized, `bichon-indices/` and
|
||||
`bichon-storage/` are created under those directories respectively.
|
||||
|
||||
---
|
||||
|
||||
## Running the Migration
|
||||
|
||||
Run `bichon-admin` and select:
|
||||
```Migrate Legacy v0.3.7 Storage to v1.0.0```
|
||||
|
||||
> **Note:** bichon-admin v1.0.0 has a known memory issue during migration of large mailboxes.
|
||||
> Please use bichon-admin v1.0.1 which includes an emergency fix. The menu option still shows
|
||||
> v1.0.0 in the label — this is a cosmetic issue, simply select it as normal.
|
||||
|
||||
|
||||
The tool will prompt for:
|
||||
|
||||
1. `--bichon-root-dir` — same value used by the old server
|
||||
2. `--bichon-index-dir` — leave blank to use default (`<root>/envelope`)
|
||||
3. `--bichon-data-dir` — leave blank to use default (`<root>/eml`)
|
||||
4. **Batch size** — controls memory usage per batch (see below)
|
||||
|
||||
Migration proceeds in two steps:
|
||||
|
||||
**Step 1 — Metadata migration** (usually completes in seconds):
|
||||
|
||||
> Mail Settings done (0 items)
|
||||
> Accounts done (1 items)
|
||||
> OAuth2 Entities done (0 items)
|
||||
> OAuth2 Pending done (0 items)
|
||||
> OAuth2 Access Tokens done (0 items)
|
||||
> Proxy Settings done (0 items)
|
||||
> User Roles done (5 items)
|
||||
> Users done (1 items)
|
||||
> Access Tokens done (2 items)
|
||||
> Mailboxes done (1 items)
|
||||
Metadata migration completed successfully.
|
||||
|
||||
**Step 2 — Email index and blob migration**: proceeds segment by segment.
|
||||
Progress is displayed for both the envelope scanning phase (Phase 1)
|
||||
and the ingestion phase (Phase 2).
|
||||
|
||||
---
|
||||
|
||||
## Memory Usage
|
||||
|
||||
Batch size directly controls peak memory consumption. Choose based on your available RAM
|
||||
and average email size (larger attachments = higher memory per batch).
|
||||
|
||||
| Batch Size | Approx. RAM | Notes |
|
||||
|------------|-------------|------------------------------|
|
||||
| 1000 | ~500 MB | Safe for low-memory servers |
|
||||
| 3000 | ~1 GB | Recommended default |
|
||||
| 5000 | ~2 GB | Faster on high-memory servers|
|
||||
|
||||
After each batch is ingested, Tantivy performs a commit to flush the in-memory buffer to disk.
|
||||
This commit can take **2–3 minutes** depending on batch size and disk speed — this is normal,
|
||||
please wait patiently. Segment merging is deferred to the end of migration to avoid
|
||||
mid-process stalls, and may also take several minutes once all batches are complete.
|
||||
|
||||
For large mailboxes (hundreds of GB), the full migration can take **several hours**.
|
||||
Reducing the batch size will lower memory usage but increase the number of commits,
|
||||
which may extend the total migration time.
|
||||
|
||||
---
|
||||
|
||||
## Re-running the Migration
|
||||
|
||||
If migration needs to be re-run from scratch, delete the new output directories before
|
||||
running the tool again.
|
||||
|
||||
**Default paths** (when no custom dirs were configured):
|
||||
|
||||
```bash
|
||||
rm -rf <bichon-root-dir>/bichon-indices
|
||||
rm -rf <bichon-root-dir>/bichon-storage
|
||||
rm -rf <bichon-root-dir>/memdb
|
||||
```
|
||||
|
||||
**Custom paths** (when `--bichon-index-dir` or `--bichon-data-dir` were configured):
|
||||
|
||||
```bash
|
||||
rm -rf <bichon-index-dir>/bichon-indices
|
||||
rm -rf <bichon-data-dir>/bichon-storage
|
||||
rm -rf <bichon-root-dir>/memdb # memdb is always under bichon-root-dir
|
||||
```
|
||||
|
||||
> **Warning:** Only delete `bichon-indices`, `bichon-storage` and `memdb`. Do **not** delete
|
||||
> or modify the legacy `envelope/` and `eml/` directories — they are the migration source and
|
||||
> must remain intact. Deleting them will make re-migration impossible without restoring from backup.
|
||||
|
||||
---
|
||||
|
||||
|
||||
## After a Successful Migration
|
||||
|
||||
1. **Start the v1.0.x server** — it will automatically detect the new storage layout and skip migration prompts.
|
||||
2. **Verify** that emails and attachments are accessible and searchable.
|
||||
3. **Clean up legacy files** once you have confirmed the migration is complete and the new server is stable:
|
||||
|
||||
**Default paths:**
|
||||
```bash
|
||||
rm -rf <bichon-root-dir>/envelope
|
||||
rm -rf <bichon-root-dir>/eml
|
||||
rm -f <bichon-root-dir>/mailbox.db
|
||||
rm -f <bichon-root-dir>/meta.db
|
||||
```
|
||||
|
||||
If `--bichon-index-dir` was configured, the directory contains a mix of legacy Tantivy
|
||||
segment files (`.fast`, `.fieldnorm`, `.idx`, `.pos`, `.store`, `.term`, `meta.json`,
|
||||
`.managed.json`, `.tantivy-*.lock`) alongside the new `bichon-indices/` subdirectory.
|
||||
Remove all the legacy Tantivy files, but keep `bichon-indices/` intact.
|
||||
|
||||
Same applies to `--bichon-data-dir` — remove the legacy Tantivy files, keep `bichon-storage/`.
|
||||
|
||||
```
|
||||
<bichon-index-dir>/
|
||||
├── bichon-indices/ ← KEEP — new index
|
||||
├── .managed.json ← delete
|
||||
├── .tantivy-meta.lock ← delete
|
||||
├── .tantivy-writer.lock ← delete
|
||||
├── meta.json ← delete
|
||||
├── 0d4e7e91a66d44e58...fast ← delete
|
||||
├── 0d4e7e91a66d44e58...store← delete
|
||||
├── 0d4e7e91a66d44e58...term ← delete
|
||||
└── ... ← delete all other segment files
|
||||
|
||||
<bichon-data-dir>/
|
||||
├── bichon-storage/ ← KEEP — new storage
|
||||
├── .managed.json ← delete
|
||||
├── meta.json ← delete
|
||||
└── ... ← delete all other segment files
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
> **Warning:** Double-check paths before running any `rm` command. If `--bichon-index-dir`
|
||||
> or `--bichon-data-dir` point to directories that also contain `bichon-indices` or
|
||||
> `bichon-storage` as subdirectories, remove only the legacy contents, not the parent directory.
|
||||
Reference in New Issue
Block a user