Use (account_id, mailbox_id, content_hash) as dedup key with time-based
eviction to bound memory at ~50MB. Check happens before mail parsing,
so duplicate emails skip all expensive work entirely.
- Populate cache from Tantivy FAST columns on startup (7-day window)
- Evict oldest 1/4 of entries when exceeding 300K capacity
- Graceful degradation: populate failure → empty cache, still works
- Switch memdb durability from Full to Batch(100) with 10s flush worker
- Offload BlobManager fjall writes to spawn_blocking
- Wrap Tantivy commit operations in block_in_place
- Flush memdb WAL on graceful shutdown
POST /api/v1/search-messages previously filtered and sorted only on the
sender-controlled Date: header. Add support for two server-controlled
timestamps that are already indexed as FAST i64 fields:
- internal_date (IMAP INTERNALDATE)
- ingest_at (Bichon's archival time)
EmailSearchFilter gains internal_date_since/before and ingest_since/before
range bounds, mirroring the existing `since`/`before` Date: handling.
SortBy gains InternalDate and IngestAt variants (wire values INTERNAL_DATE
and INGEST_AT), mirroring the existing DATE/SIZE sort handling.
The envelope and attachment Tantivy schemas already declare these fields
as INDEXED | STORED | FAST, so no re-index or migration is required.
Attachments carry no IMAP INTERNALDATE, so the attachment search maps the
InternalDate sort to the attachment's own date field as a defined fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cleanup_unused_content decides whether to delete a deduplicated blob by
running a Tantivy Count of envelopes referencing each content_hash. The
searcher it used reflected only the committed index state at the time it
was built, so an envelope that shared a content hash but was still
sitting uncommitted in the IndexWriter buffer (for example added by the
background ingest task before the delete operation acquired the writer
lock) was invisible to the count. The count read 0 and a
still-referenced blob was deleted, permanently 404ing that envelope's
download-message.
Pass the locked IndexWriter into cleanup_unused_content and fatal_commit
it immediately before creating the searcher. create_searcher already
reloads the reader, so the Count is now evaluated against a fully
committed, freshly-reloaded index state. The barrier is local to the GC
path and self-contained, independent of what the caller committed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rustmailer