- use a single Engine instance for email + attachment blobs
- add delete_batch, gc_if_needed, background flush to blob crate
- fix Entry.raw_size storing compressed length instead of original
- move fjall-dependent migration code from core to admin crate
- add STORAGE_VERSION file for layout version detection
- Make MBOX/PST upload size limits configurable via SETTINGS
(bichon_web_mbox_upload_limit_mb defaults to 1 GB,
bichon_web_pst_upload_limit_mb defaults to 2 GB)
update_envelope_tags lost f_body, f_from_text, f_to_text, f_cc_text,
f_bcc_text, f_attachment_name_text and f_attachment_name_exact because
they are not STORED and field_values() skipped them during delete+add.
Rebuild these from stored counterparts and the blob-store EML.
Previously, do_migrate_segment created a fresh NewIndexWriter (and
therefore a new Fjall Database) on every call, meaning the Fjall
database at bichon-storage/ was opened and closed once per segment.
This caused the migration to fail mid-way through (observed at segment
9/16) with:
Storage(InvalidTag(("ChecksumType", 171)))
Root cause: after segment N writes email blobs via Fjall's ingestion
API (start_ingestion / write / finish), those SSTables and KV-separated
blob files are flushed to disk and the Database is dropped. When segment
N+1 calls Database::builder(storage_dir).open(), Fjall must discover and
catalog all on-disk files produced by the previous segments. During that
discovery it reads SSTable or blob-file block headers and encounters a
ChecksumType discriminant byte (171 / 0xAB) that lsm-tree 3.1.4 does
not recognise, causing the fatal error.
The first N segments succeed because the cumulative set of ingested
SSTables stays small enough that Fjall does not need to read the
offending headers during reopen. Once enough data has accumulated the
reopen triggers a manifest or compaction read that exposes the mismatch.
Fix: open NewIndexWriter once, before the segment loop, and pass a
&mut reference into each do_migrate_segment call. finish_writers() is
called a single time after all segments complete. The Fjall Database
stays open for the entire migration and is never closed and reopened,
eliminating the incompatible-reopen path entirely.
Use (account_id, mailbox_id, content_hash) as dedup key with time-based
eviction to bound memory at ~50MB. Check happens before mail parsing,
so duplicate emails skip all expensive work entirely.
- Populate cache from Tantivy FAST columns on startup (7-day window)
- Evict oldest 1/4 of entries when exceeding 300K capacity
- Graceful degradation: populate failure → empty cache, still works
This commit adds support for IMAP servers that don't provide UIDVALIDITY,
such as Tencent Enterprise Mail (腾讯企业邮箱).
Changes:
- Added `generate_synthetic_uidvalidity()` function that creates a stable
hash-based UIDVALIDITY from the mailbox name
- Modified `reconcile_mailboxes()` to use synthetic UIDVALIDITY when the
server doesn't provide one
- Servers without UIDVALIDITY can now sync all mailboxes including system
folders (Sent Messages, Drafts, Deleted Messages)
- Incremental sync is supported via the synthetic UIDVALIDITY
- Added warning logs to indicate when synthetic UIDVALIDITY is in use
- Updated mailbox metadata to store the resolved UIDVALIDITY
Fixes issues with:
- Tencent Enterprise Mail (腾讯企业邮箱)
- Other non-compliant IMAP servers
- Mailboxes that don't properly support UIDVALIDITY"
When the request returns an empty body, choose to skip it and print the account ID and UID information, leaving it for the user to investigate themselves. Otherwise, the IMAP download process will be blocked by this.
- Treat MIME parts with Content-ID but no Content-Disposition as inline
- Add account_ids filter to CLI export search to avoid pulling all accounts
- Skip failed emails during export instead of aborting the entire batch
- Switch memdb durability from Full to Batch(100) with 10s flush worker
- Offload BlobManager fjall writes to spawn_blocking
- Wrap Tantivy commit operations in block_in_place
- Flush memdb WAL on graceful shutdown
InternalDate and IngestAt were renamed for the wire with #[serde(rename)] only. SortBy derives poem_openapi::Enum, which does not honour serde attributes, so the REST deserializer exposed them under their Rust identifiers instead of the intended INTERNAL_DATE / INGEST_AT — inconsistent with the existing DATE/SIZE values and rejecting the documented names with HTTP 400. Add #[oai(rename = ...)] alongside the serde rename.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /api/v1/search-messages previously filtered and sorted only on the
sender-controlled Date: header. Add support for two server-controlled
timestamps that are already indexed as FAST i64 fields:
- internal_date (IMAP INTERNALDATE)
- ingest_at (Bichon's archival time)
EmailSearchFilter gains internal_date_since/before and ingest_since/before
range bounds, mirroring the existing `since`/`before` Date: handling.
SortBy gains InternalDate and IngestAt variants (wire values INTERNAL_DATE
and INGEST_AT), mirroring the existing DATE/SIZE sort handling.
The envelope and attachment Tantivy schemas already declare these fields
as INDEXED | STORED | FAST, so no re-index or migration is required.
Attachments carry no IMAP INTERNALDATE, so the attachment search maps the
InternalDate sort to the attachment's own date field as a defined fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The download-message REST endpoint returned 404 [30000] "Original email
content not found" whenever an indexed message's raw content blob was
missing from the blob store, leaving the message permanently
unrecoverable even though it still existed on the IMAP server.
Make the endpoint self-healing: when the content blob is absent, fetch
that one message on demand from the IMAP server via
UID FETCH <uid> (BODY.PEEK[]), repopulate the detached blob, and return
the content. The 404 is now only produced when the on-demand fetch
itself fails (mailbox gone, UID gone, connection failure, or the fetched
bytes no longer match the archived content_hash).
- Add ImapExecutor::fetch_single_message_body: examines the mailbox
read-only and fetches one message by UID, reusing the existing
BODY.PEEK[] fetch command.
- Add reattach_eml_content_self_healing / recover_message_blob in the
envelope extractor: fast-path delegates to reattach_eml_content when
the blob exists; otherwise recovers it via IMAP and re-stores the
stripped EML + attachments through the existing blob queue.
- Make store::blob::get_reader async and route it through the
self-healing path; update the single caller (download_message).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rustmailer