A server-provided retry-after / suspension-time was clamped to 60s, so a
longer server-requested cooldown was ignored (impolite, and the server may
penalize a client that does not honor it). Honor the server's value up to a
300s safety ceiling; the tighter 60s cap now applies only to our own
header-less escalating backoff.
Add a MockBackend error mode and a test asserting a failed on-demand body
fetch returns NO [UNAVAILABLE] (not a placeholder body) and arms the
cooldown, and that a cooled-down mail then short-circuits to NO without
touching the backend. Also prune elapsed entries when arming a cooldown so
the map stays bounded by the mails currently cooling down rather than every
mail that has ever failed.
needs_body() now triggers an on-demand fetch only for items that actually
need the body (BODY[] / BODY.PEEK[] / standalone RFC822). It used to fire for
ENVELOPE and every BODY[...] section, including BODY[HEADER...], so the
client's list-building did one full body+attachment download per message: on
a 19535-mail inbox that meant downloading the whole mailbox just to render
the list (the request storm, and an empty list while it ground on). Envelope,
header, size and structure items are answered from local metadata.
On a genuine fetch failure the bridge now returns a tagged NO [UNAVAILABLE]
instead of a successful response carrying a placeholder body (which made the
client cache a fake message and re-request forever). The failed mail is put
on a 30s cooldown, shared across IMAP connections via MailStore, so the
bridge does not re-hit a throttled server on every client redraw.
Tests: needs_body (skip metadata/header, trigger only on real body),
body-fetch cooldown set/expire.
Route every SDK request through one GovernedRestClient (installed via
Sdk::new_without_suspension, replacing the SDK's header-only suspension
layer). It enforces:
- Bounded concurrency: at most MAX_IN_FLIGHT (4) requests in flight, so
attachment sub-loops, on-demand IMAP fetches across connections, and the
syncer can no longer stampede the API in parallel.
- Throttle backoff: on HTTP 429/503 it suspends all outbound traffic,
honoring the server's retry-after / suspension-time header, or an
escalating default backoff (2s to 60s) when the server gives no hint
(the gap the SDK left, which let the bridge keep hammering).
Because a 429 still returns to the caller while the gate is armed, the
existing retry layers become self-correcting: their next attempt blocks on
the gate instead of amplifying the flood.
7 unit tests: concurrency cap, throttle classification, backoff
honor/clamp/escalate/reset, suspension gate timing.
A mail subject or sender name containing a raw newline (or 8-bit bytes)
was emitted inside a quoted string, which IMAP forbids. The resulting
malformed FETCH response broke the client's parse of the message list,
leaving the mailbox empty in Thunderbird.
Add imap_string(): a quoted string for safe 7-bit text, a server-side
literal ({N}CRLF<octets>) when the value holds CR, LF or 8-bit bytes.
Applied to subject, message-id and envelope address fields.
Adds 6 unit tests (imap_string + a newline-subject regression); the old
imap_quote tests are migrated. Full lib suite green (259 tests).
Mail clients save a copy of each sent message to the Sent folder with an
IMAP APPEND. The bridge did not implement APPEND, so Thunderbird reported
"a copy was not placed in your Sent folder" after every send.
Tuta saves sent mail server-side and the syncer brings that copy back, so
an APPEND to Sent is a no-op: read and discard the literal, reply OK,
which avoids creating a duplicate. APPEND to any other folder is rejected
before the literal is sent (the client then aborts the synchronizing
literal and the stream stays in sync); real APPEND-to-Drafts is left for
a follow-up.
The literal is read at the socket level since the session layer is line
based. Tested: parse_append, the Sent-folder decision, and the full
handle_append flow over an in-memory pipe (Sent reads the literal and
returns OK, a non-Sent folder is rejected with no continuation).
* mail: do not split a quoted display name on its comma
parse_address_list tracked angle-bracket depth but not quotes, so a
recipient like `"Doe, John" <john@x.com>` was split on the comma inside
the quoted name, yielding a bogus recipient (`"Doe`) next to the real
one. With a real contact named "Last, First" that either gets the whole
send rejected by Tuta or delivers to a garbage address.
Track the quote state too: inside `"..."`, commas and angle brackets are
literal. Tested with a quoted-comma recipient and a plain comma list.
* mail: decode non-UTF-8 bodies instead of echoing base64 or QP source
When a base64 or quoted-printable body decoded to bytes that were not
valid UTF-8 (e.g. a Latin-1 message), the parser fell back to returning
the still-encoded source: the recipient saw a wall of base64, or raw
=XX sequences. Decode the bytes lossily instead, so the text is readable
(non-UTF-8 bytes become the replacement char rather than garbage).
Full charset-aware decoding (Content-Type charset via encoding_rs) is a
follow-up; this fixes the worst symptom with no new dependency. Tested
with a non-UTF-8 base64 body and a non-UTF-8 quoted-printable byte.
* smtp: enforce message size and line length limits
The server advertised SIZE 26214400 in EHLO but never enforced it, and
the DATA loop appended every line into an in-memory buffer with no cap,
so a single local client could grow the process memory without bound (a
line with no terminator was read unboundedly too).
Enforce both: reject a MAIL FROM that declares an over-limit SIZE, stop
buffering and reply 552 once a message exceeds the cap, and bound each
protocol line. handle_connection is now generic over the stream so the
whole conversation can be exercised over an in-memory pipe; tests cover
the size param, the DATA cap, the per-line cap, and a normal send.
* backup: run mail decryption and writes off the async runtime
export_eml decrypted each cached .eml.enc and wrote the output file
inline on the async task. A GUI backup reuses the running bridge's
runtime, so over a large already-synced mailbox that tight, non-yielding
loop pinned a worker and froze the live IMAP/SMTP servers for the whole
export (the same failure class as the cached-folder load).
Wrap the per-mail decrypt and file write in block_in_place so the worker
hands its other tasks off and the servers stay responsive. The backup
integration tests run on a multi-thread runtime now (block_in_place
requires it) and still assert the same cache/server/resume behaviour.
* net: tolerant accept loop with a connection cap and handshake timeout
Both servers ran `loop { listener.accept().await? }`. A single transient
accept error (EMFILE, ECONNABORTED, ...) propagated out and stopped the
server for good, nothing bounded concurrent connections, and a stalled
TLS handshake was never timed out (a client that connects but never
negotiates parked a task and a file descriptor forever).
Extract a shared net::accept_loop that logs and retries a failed accept,
caps concurrency with a semaphore (64 connections), and wrap each
handshake in a 15s timeout. The loop is transport agnostic so it is unit
tested without TLS: one test proves it keeps accepting across
connections, another that it bounds concurrency at the cap.
* event-bus: recover poisoned last_batch_ids lock instead of panicking
last_batch_ids is a std Mutex shared between the bridge, the event
handler, and the SDK's reconnect path. Every accessor used
.lock().unwrap(), so one panic while holding it would poison the mutex
and make every later lock (the SDK reconnect included) panic, killing
realtime sync for the rest of the process's life.
Add util::lock_recover (locks, recovering the guard from poisoning) and
use it at the bridge-side accessors. Tested against a poisoned mutex.
Loading a cached folder, and the one-time full-text backfill, both
decrypt every cached .eml.enc body (AES-CBC plus an HMAC-SHA256
verification) in a tight loop with no await points. Run inline on a
tokio worker, that loop keeps the worker and the IO driver it holds busy
for the whole duration, so the IMAP and SMTP accept loops stop being
polled. On a large mailbox, connecting a client or sending a message
times out for the first 10 to 90 seconds after launch while the cache
loads, even though most cores sit idle.
Profiling during the stall showed 15 of 16 workers parked, 1 grinding
through SHA-256 and AES, and nothing polling kqueue.
Move the per mail decode (metadata deserialize, body read and decrypt)
onto the blocking pool via spawn_blocking, for both the startup cache
load and the FTS backfill. The worker threads stay free to drive IO, so
IMAP and SMTP answer immediately while the mailbox loads in the
background.
Expose the mailbox to an LLM client (Claude Desktop / Code) over an
in-process MCP server, so the bridge itself hosts it and the GUI controls
it live. Strictly read-only: there is no tool that sends, moves, deletes or
mutates mail — by design and asserted in tests.
Transport: Streamable HTTP (MCP 2025-06-18) on a single POST /mcp endpoint
bound to 127.0.0.1, answering each JSON-RPC request with application/json
(no SSE — the server never pushes). Auth is a bearer token (the bridge
password); the Origin header is validated to block DNS-rebinding.
Permission tiers (config.McpPermission, default Disabled = server off):
- Metadata — folders, metadata search (subject/sender/date), headers only.
- Full — the above plus full-text body search and message body text.
Tools: list_folders, search_messages, list_unread, get_message. Search
combines subject/sender always and the encrypted FTS body index under Full;
get_message returns headers always and body only under Full.
Wiring: spawned in-process by both the CLI (main.rs) and the GUI bridge
task (bridge.rs); a Disabled tier makes serve() a no-op, and it is kept out
of the select! so it never triggers teardown. GUI gains an MCP section
(tier selector, port, full-read warning, "copy client config" button) and a
get_mcp_client_config command that emits the ready-to-paste client snippet.
Validated live on a ~19k-message mailbox: initialize / tools/list /
tools/call all conform; 401 without the bearer token, 403 on a foreign
Origin, 202 on notifications; list_folders, body search and get_message
(HTML stripped to text) all return correctly. 240 unit tests.
BODY/TEXT searches previously matched only bodies that happened to be
decoded in memory, so results were inconsistent. Add a persistent FTS5
index (a virtual table inside the SQLCipher store, encrypted at rest) over
the plain-text body of every message we download.
- store.rs: mail_fts(element_id UNINDEXED, body) with unicode61 +
remove_diacritics; index_body / unindex_body / search_body / fts_count.
Terms become prefix tokens ANDed together (factur -> factur*), built by
fts_match_expr which strips everything but alphanumerics so it is
injection-safe.
- rfc2822.rs: strip_html (drops tags + script/style + entities) and
extract_body_text (decodes the text part of our own .eml) feed the index.
- sync.rs: index inline at prefetch; one-time backfill at boot
(body_fts_indexed_v1) for bodies cached before the index existed;
unindex on delete.
- search.rs: BODY/TEXT resolve through the index — the session collects the
distinct body terms, queries the index once each, and passes the hit sets
to matches() via a SearchContext. A body term only matches messages whose
body has actually been downloaded (full coverage needs sync_limit = 0).
- LocalStore threaded into ImapSession (Option; None in unit tests).
Validated live: backfilled 7,687 cached bodies, then BODY/TEXT/AND/OR/NOT
queries returned coherent subsets — NOT BODY x == total - (BODY x), an
exact complement. 233 unit tests, incl. real FTS5 MATCH against the bundled
SQLCipher (confirms FTS5 is compiled in for the cross-OS release).
cmd_search only ever special-cased UNSEEN — every other query (SUBJECT,
FROM, SINCE, …) fell through to "return all message ids", so a search in
Thunderbird silently matched the entire mailbox. Now that the full mailbox
is listed, that made search actively misleading.
New imap/search.rs parses the RFC 3501 SEARCH grammar into a SearchKey
tree (AND/OR/NOT, parens, CHARSET prefix, quoted strings, sequence/UID
sets) and matches each message via a lightweight MsgView the session
projects from its cached mail. Coverage is metadata-first: SUBJECT, FROM,
TO, CC, BCC, HEADER, flags, dates (BEFORE/ON/SINCE + SENT*), LARGER/
SMALLER, UID and sequence sets. Flag predicates resolve consistently with
what we report over FETCH (only \Seen and \Deleted exist). BODY/TEXT match
the body only when it's already decoded — whole-mailbox full-text body
search is the next increment, backed by an on-disk index.
Unknown criteria degrade to a non-restrictive match so search never hides
a message.
Validated live on a 19,322-message INBOX: SEEN+UNSEEN partition the
mailbox exactly, NOT SEEN == UNSEEN, AND/OR compose, and a nonexistent
subject now returns 0 hits instead of everything. 223 unit tests.
The local store was capped at sync_limit, so IMAP only ever listed the
newest N messages — a search in Thunderbird (the only search UI we have)
silently missed everything older. Now the syncer lists the *full* mailbox
metadata for every folder, and sync_limit governs only how many recent
message bodies are pre-warmed offline. Bodies outside that window are
fetched on demand the first time a client opens the message.
A one-time full-metadata sync (marker full_metadata_synced_v1) completes
the mailbox view on first launch after upgrade.
Crucially, an empty body is now stored as rfc2822 = None rather than a
rendered "(No body available)" placeholder: the placeholder looked like a
real body to the IMAP layer and suppressed the on-demand fetch. CachedMail
gains body_loaded to track whether the body (not just the headers) is final.
Validated live on a 19,322-message INBOX (26,965 mails total across
folders): full listing, on-demand body fetch (~0.1-0.4s), in-memory cache
on re-fetch.
The bundled-sqlcipher feature relied on a system OpenSSL for SQLCipher's
crypto — fine on macOS/Linux, but absent on Windows, where the release
build failed at libsqlite3-sys. Switch to
bundled-sqlcipher-vendored-openssl: OpenSSL is built from source, so the
build is self-contained and identical across all three OSes (and the AUR
package needs no system OpenSSL). Caught by the multi-OS release dispatch.
The keyring dep was hard-pinned to the macOS `apple-native` feature, so
the headless CLI/core didn't compile anywhere else — blocking an AUR
package or any Linux/Windows use. Split it into per-target features:
apple-native (macOS), windows-native (Windows), and
sync-secret-service + crypto-rust (Linux, via gnome-keyring/KWallet,
pure-Rust crypto so no OpenSSL build dep).
TutaBridge links Tuta's Rust SDK, which is part of the GPLv3-licensed
tutanota project, so the bridge must carry the same license. Add the
full GPLv3 text, set `license = "GPL-3.0-or-later"` on all three
crates, and note it in the README.
* Add complete mailbox backup to .eml files (CLI)
`tutabridge backup <dir>` exports every mail of every folder to a
plaintext `.eml` tree, mirroring the IMAP folder hierarchy.
A backup must be *complete*: it enumerates all mails per folder from
the server (`limit == 0`), not just the `sync_limit`-capped subset the
bridge keeps cached. Dumping only the synced subset would silently drop
mail — live-tested here against an INBOX with 6288 server-side mails vs
1050 cached, all 6288 exported. The encrypted local cache
(`.eml.enc`) is used as a fast path; only never-synced mails trigger a
rate-limited (150ms) server fetch.
Format: one `.eml` per mail in `<output>/<folder path>/<YYYYMMDD-HHMMSS>_<id>.eml`.
EML is the most portable target — native to Thunderbird/Apple Mail/
Outlook, no Maildir `:2,S` colons that break on Windows, and a single
corrupt file never takes down the whole archive. Folder path segments
are sanitised for cross-platform filesystems (Windows-illegal chars +
trailing dot/space stripped); the date prefix makes a directory listing
sort chronologically.
`backup::export_eml` is surface-agnostic (takes a progress callback) so
a GUI button can wrap the same engine later. Per-mail failures are
collected in `BackupStats::errors` rather than aborting the run. The CLI
shares the keychain/password login flow with the bridge via the new
`login_session` helper, and opens the cache without the bridge's
reset-on-key-mismatch (a backup must never destroy the cache).
8 backup unit/integration tests: filename + folder sanitisation,
date stamp, and an end-to-end export over a mock backend asserting the
cache-vs-server split, file tree layout, and verbatim cached bodies.
GUI button is a follow-up (needs the Tauri dialog plugin).
* Make backup resumable / incremental
Skip a mail when its `.eml` is already on disk, before any cache read
or server fetch. The filename is deterministic (stable receivedDate +
element id) and mail content is immutable, so an existing file is never
stale. This turns an interrupted backup into a resume (re-run continues
where it stopped) and a periodic re-backup into an incremental one
(only new mail is fetched — the expensive part). New
`BackupStats::skipped` counter, surfaced in the CLI summary.
Two tests: a re-run skips every already-exported mail (zero server
loads), and an incremental run fetches only the newly-arrived mail.
* Add Backup tab to the GUI
A "Backup" tab wraps the same `backup::export_eml` engine as the CLI:
a native folder picker (tauri-plugin-dialog), a live per-folder
progress bar driven by `bridge://backup-progress` events, and a result
summary (mails written, folders, MB, cache vs server vs skipped).
`BridgeHandle` now keeps the logged-in backend + local cache after
`start` and exposes them via `backend_and_store()`, so the
`export_mails` command reuses the live session instead of opening a
second one — and drops the handle lock before the (minutes-long)
export so status/stats stay responsive. The button is disabled unless
the bridge is running.
`BackupStats` is now `Serialize` so it can cross the Tauri boundary.
* Keep backup state across tab switches
The Backup tab is conditionally rendered, so switching away unmounted
`BackupPanel` mid-export — dropping its progress + result state and the
`bridge://backup-progress` listener while the Rust task kept running.
Coming back showed an idle panel even though the backup was still going.
Lift all backup state (busy / progress / result / error), the
`startBackup` action, and the progress listener into the always-mounted
`useBridge` hook. The listener is now active regardless of which tab is
shown, and `BackupPanel` is purely presentational — switch tabs freely
mid-backup and the progress is intact on return. `startBackup` guards
against a double launch while one is in flight.
The original implementation consumed the cache entry on first lookup,
which meant the prefetch sweep that ran first (typically the Sent
copy) got its multipart envelope rebuilt correctly, but the second
sweep (the Inbox copy, which is the one suffering from the
`File._ownerEncSessionKey` race) found an empty cache and fell back
to the failing `crypto_client.load` path — leaving its .eml body-only.
A self-send produces `load_attachments` calls for both folder copies
of the same envelope, so the cache must serve as many lookups as
arrive within the TTL. Clone the cached attachments out of the entry
instead of removing it; the existing TTL (1h) + soft cap
(50 entries) keep memory bounded. Also move the cache insert ahead of
`DraftService.post` so the WS event for the inbox copy cannot beat
the insert.
Live-verified: both Sent and Inbox copies of a self-sent mail with a
PDF attachment now expose a proper `multipart/mixed` BODYSTRUCTURE
with the file part — no more session-key-transient retry storms in the
prefetch logs.
Tuta's server never publishes `File._ownerEncSessionKey` on the
recipient-side copy of a mail the user sent to themselves over SMTP.
The TS client survives this because it caches each file's plaintext
session key locally at send time and re-uses it for the inbox copy
without going through `crypto_client.load::<TutanotaFile>()` — the
Rust SDK has no such cache, so the bridge's previous retry-on-WS-event
mechanism logged 'still not decryptable after retries' forever.
Mirror the TS behaviour in `TutaSession`:
* A small `HashMap<key, SelfSendCacheEntry>` keyed by
`subject + from + first_recipient` (lower-cased + trimmed) — those
three fields are preserved verbatim across the Sent and Inbox copies
of a self-send, so the inbox-side lookup always finds the entry the
send side just dropped in.
* `cache_self_send_attachments` runs from `send_mail_impl` only when
`is_self_recipient` is true (the recipient list contains the bridge's
own address). Third-party recipients hit Tuta's normal pipeline,
which populates File metadata before delivery, so caching there
would just waste memory.
* `try_self_send_cache` short-circuits `load_attachments_impl` for
the inbox copy, returning synthetic `TutanotaFile` records (only the
fields `mail_to_rfc2822` actually reads) alongside the plaintext
bytes. The entry is consumed on hit — the .eml the prefetch writes
next becomes the durable cache.
* TTL = 1h, soft cap 50 entries (LRU eviction on insert).
Seven unit tests pin the cache key normalisation and the self-send
detection (positive when To/Cc match From, negative when either
diverges, case-insensitive throughout).
The single in-line retry budget (2/4/8/16s, ~30s total) bumps the
session-key-transient race down but does not solve it: self-sends can
take longer than 30s to surface a usable File entity. Worse, the
in-line wait blocks the whole prefetch sweep on the offending mail.
Replace the strategy with an async pending state:
* New `attachments_pending: bool` on `StoredMail`. Set when a
prefetch_details pass sees `is_transient_attachment_error(e)`; the
body-only RFC 2822 is still written to disk + the store so IMAP keeps
serving the message immediately.
* Second pass in `prefetch_details` retries the attachment-only step
for any mail flagged pending, throttled by a per-mail
`HashMap<element_id, Instant>` (`ATTACHMENT_RETRY_THROTTLE = 60s`).
On success the cached .eml is rewritten as multipart, the flag is
cleared, and the throttle entry is forgotten. On permanent failure
the flag is also cleared so we stop hammering the server.
* `prefetch_loop` now wakes either on a store mutation **or** after
`ATTACHMENT_RETRY_THROTTLE` when any throttle entry exists, so the
retry happens even if no other mail traffic touches the store.
Bridge-side adds `MailStore::update_mail_rfc2822` (rewrite the body
without touching cached details), and threads the new field through
every `StoredMail` construction (tests included). SDK-side bumps the
in-line `load_file_with_retry` budget to 2/4/8/16s so most propagation
delays still resolve before we surface the transient error to the
pending path.
When a brand-new mail lands through the realtime event bus, its
attached `File` entities are queryable on the server but the
encryption metadata (`_ownerEncSessionKey` / `_ownerGroup`) may not
have propagated yet — `CryptoEntityClient::load` then returns
'instance missing owner key/group data' a few seconds before the same
load would succeed.
The previous code surfaced that as a permanent attachment-load failure
inside `prefetch_details` (best-effort fallback to a body-only
multipart), so a PDF sent via SMTP appeared body-only in Thunderbird
and only recovered after a manual sync.
Add `load_file_with_retry` in `TutaSession`: when
`is_session_key_transient(e)` (matching on the SDK error message),
back off 1s / 3s / 6s before retrying, then bubble up so the existing
ship-body-only fallback still applies. \~10s of tolerance is enough
in practice while staying well under the SMTP/IMAP latency budget.
Four unit tests cover the helper: matches the exact 'missing owner
key/group data' wording, the generic 'Session key resolution failure'
prefix, rejects unrelated SDK errors, accepts a bare 'missing owner
key/group data' message (paranoid catch).
The CLI (`src/main.rs`) only spawned the syncer + IMAP + SMTP servers
and never started an `EventBusClient` — so the realtime push the
GUI's `BridgeHandle` ships had no effect when running `cargo run` or
the headless binary. Mails that arrived after a bootstrap sync were
silently missed until the next restart with a forced full re-sync;
that's the irritant that triggered today's WS heartbeat/timeout audit.
Replicate the bridge.rs initialisation directly: build an
`EventBusClient`, hydrate `last_batch_ids` from
`event_bus_state` (with the same 44-day expiration guard), spawn
`bus_client.run` alongside the syncer and an
`event_handler::run_event_handler` to consume the mpsc, and log
WsState transitions at INFO so reconnect storms are visible without
`RUST_LOG=debug`. Shutdown aborts the bus + handler handles in the
same Ctrl-C arm as the syncer.
To avoid duplicating the model-version + client-name plumbing, the
helpers `bridge::sys_model_version`, `bridge::tutanota_model_version`
and `bridge::CLIENT_NAME` are now public, and the root crate gains a
direct `tuta-sdk` dependency (already present transitively through
`tutabridge-core`).
The bridge already turns WsState changes into stats pulses for the UI
but never logged the transition itself, so an INFO-only log stream had
no record of reconnects. Track the previous state in the watcher and
emit `info!` on each genuine transition (skipping duplicate sets from
the same state — `publish()` is called a few times redundantly during
the connect handshake). Picks up [[sdk]] commit
1036d6f2e5 (SDK heartbeat + idle timeout) via the submodule bump.
The IMAP server was hardcoding BODYSTRUCTURE to a single
\"text/html\" entry regardless of the cached envelope. Thunderbird
parses the body itself and survived that, but stricter IMAP clients
use BODYSTRUCTURE as the source of truth for whether a message has
files to save — so multipart messages were rendering with no
attachment hints anywhere.
New module `mail::bodystructure` walks the cached RFC 2822 (reusing
the parser helpers — now `pub(super)` for sibling access), produces a
parenthesised RFC 3501 §7.4.2 structure with one entry per MIME part,
and propagates Content-Disposition so attachment parts carry their
filename. Single-part bodies still emit the previous shape verbatim
(no behaviour change for the common case).
Eight unit tests cover the happy path (text/html), multipart/mixed
with a PDF attachment (asserts `MIXED`, `BOUNDARY`, type/subtype,
disposition + filename), nested `multipart/alternative` inside
`multipart/mixed` (real-world MUAs), and the helpers (count_lines,
quoted, build_params, missing boundary).
Wire the SMTP send path so that when Thunderbird hands the bridge a
`multipart/mixed` message its non-text parts are forwarded as real
Tuta attachments rather than dropped.
Parser side (mail/parser.rs)
* `ParsedMessage` grows an `attachments: Vec<Attachment>` field.
* `extract_multipart_body_and_attachments` walks every part, treats
anything carrying `Content-Disposition: attachment` or a
`name=` parameter (and not a text/* type) as a file, decodes
base64 / quoted-printable, picks up the filename from
Content-Disposition first then Content-Type's `name=`.
Send side (tuta.rs::send_mail_impl)
* `build_added_attachments` generates a per-file session key, uses
the existing `BlobFacade::encrypt_and_upload_multiple` to ship the
encrypted blob bytes, then assembles a `DraftAttachment` aggregate
with the random aggregate `_id`s the instance mapper requires.
* After `DraftService` persists the draft, `build_attachment_key_data`
reloads the resulting Mail, zips its `attachments[]` IdTuples with
the session keys we kept locally, and produces
`AttachmentKeyData[]` for both the top-level `SendDraftData` and
its nested `parameters` aggregate (server reads from the latter).
* Empty-attachments path is a no-op, matching the previous behaviour.
Three new parser unit tests pin down the multipart-with-PDF happy
path, the filename-fallback to Content-Type `name=`, and that
`multipart/alternative` text parts are not misclassified as files.
When the realtime path inline-decrypts a brand-new mail \"and\" its
MailDetailsBlob in the same batch, it used to render the RFC 2822
envelope right there and persist it to disk so the prefetch loop has
no work left for that mail. That shortcut is correct for mails with
zero attachments — but mails carrying attachments would end up cached
with a text/html-only body and no MIME parts for the attached files,
because the event-bus handler cannot synchronously fetch and decrypt
each File blob without making the WebSocket dispatch loop sleep.
Skip the inline-render when `mail.attachments` is non-empty so the
mail keeps `has_details=0` and the prefetch sweep picks it up,
emitting a proper `multipart/mixed` envelope through the same path
as historical mails.
When the prefetch loop loads a mail's body, also load every entity in
`mail.attachments`, decrypt the blob data with the new SDK helper and
embed each attachment as a base64 part of a `multipart/mixed` RFC 2822
message. Thunderbird now renders PDFs / images / etc. inline rather
than showing a body-only mail with no hint anything was attached.
* `mail_to_rfc2822` now takes a slice of (TutanotaFile, &[u8]) and emits
a multi-part envelope when it's non-empty; the simple text/html case
is unchanged. The boundary is derived from the mail's IdTuple so the
cached .eml.enc bytes stay stable across rewrites.
* `MailBackend::load_attachments` is the new trait method; the
TutaSession implementation loads each File via `crypto_client.load`
(auto-decrypted via the file's own `_ownerEncSessionKey`) then asks
the new `MailFacade::load_file_attachment_data` for the concatenated
decrypted bytes.
* `prefetch_details` does a best-effort fetch — partial failure logs a
warning and ships the body alone, on the assumption the user can
re-open later and the next sweep will retry.
A new unit test asserts the multipart structure (boundary, body part,
attachment part with name/MIME/filename, closing boundary).
After a SQLite schema migration the `mails` table is dropped and
recreated empty; the encrypted `.eml.enc` files on disk survive that
migration (they are keyed by element id, not by row id). Phase 0 of the
syncer was gating the body recovery on `meta.has_details`, so the post-
migration boot would deliver a `mail_to_rfc2822(mail, None)`
headers-only body to IMAP even though the full body sat right there on
disk — a real mail's content quietly showed up as a placeholder in
Thunderbird until something forced a refetch.
Try `read_eml` unconditionally. When it returns the body, also flip
`has_details = 1` on the row so subsequent prefetch sweeps skip the
mail and the heal is permanent. Mismatch → still falls back to
headers-only as before. The extra read at boot is a `Path::exists()`
plus an AES decrypt per mail, negligible vs the network costs we
already pay.
While here, demote the misleading IMAP-fetch log: when `details` is
None but `rfc2822` is populated we *do* serve the real body, so log
"placeholder" only when `rfc2822` is also missing.
166 bridge lib tests pass.
`load_mail_details_impl` now mirrors the TS `loadMailDetails` router:
when `mail.mailDetailsDraft` is set, call the new
`MailFacade::load_mail_details_draft` (sdk-mail-draft-details); when
`mail.mailDetails` is set, keep the existing blob path; otherwise return
`Ok(None)` — the legacy/malformed leaf the prefetch loop has to handle
anyway. Practical effect: the ~88 drafts on the test account that
previously logged "No details for mail" on every store change now decrypt
their body normally and surface it through IMAP FETCH.
The `Ok(None)` branch in `prefetch_details` was the secondary cause of
the log spam — with the prefetch loop now event-driven (Phase 2.5), every
`MailStore` bump re-queued the same 88 drafts because `has_eml` stayed
false. Persist a headers-only `.eml` and `mark_has_details = 1` on that
branch so the row is considered done; the next sweep skips it, and the
client at least gets the headers for a mail whose body we genuinely
cannot locate.
Phase 2.5 — `prefetch_loop` no longer wakes every 30s. It subscribes to
`MailStore::subscribe()` and only catches up missing bodies when the
generation counter ticks (event-bus delta, sync_folder finishing,
bootstrap). When the store is quiet — every cached mail has its .eml on
disk — the loop sleeps indefinitely on the watch channel. The
`PREFETCH_INTERVAL` constant is gone.
Phase 2.6 — the dashboard 1s `setInterval` is gone too. `BridgeHandle`
gets a `stats_dirty_tx: broadcast::Sender<()>` that pulses on every
`MailStore` bump, every `WsState` transition, and the start / stop
status transitions. A small watcher task inside `start()` plumbs the
two `watch::Receiver`s into the broadcast. The Tauri layer
(`stream_stats`) subscribes via `BridgeHandle::subscribe_stats()`,
takes a stats + status snapshot on every pulse (and once at startup),
and emits two events `bridge://stats` / `bridge://status` to the
webview. The React hook replaces `setInterval(refresh, 1000)` with two
`listen()` subscriptions; the initial `refresh()` still seeds the
state for the first frame.
Backend now has zero `time::sleep`-driven polling loops: the syncer is
driven by Phase 0 + bootstrap + the event bus, prefetch is driven by
store changes, and the UI is driven by pushes. The only remaining
delays are throttling (`INTER_FOLDER_DELAY`, `INTER_REQUEST_DELAY`)
and reconnect backoff, which are not polls.
166 bridge lib tests, full workspace builds, UI tsc clean.
A Mail CREATE event carries the encrypted Mail in `event.instance` and
its MailDetailsBlob in `event.blob_instance`. The bridge now pre-decrypts
both at the start of each batch into a `pending: HashMap<eid, PendingMail>`
pool, then the matching `MailSetEntry CREATE` consumes the entry by
element id — no `load_mail` REST call, and when the blob was present the
RFC 2822 `.eml` is rendered + written + `has_details = 1` on the spot,
so the prefetch loop never has to fetch the body either.
Total: a brand-new mail arriving over the event bus now needs 0 REST
calls when the server bundles the inline payloads (the common case).
Old path required 2 (load_mail + load_mail_details_blob).
New MailBackend method `decrypt_inline_mail_details_blob` delegates to
the SDK's `CryptoEntityClient::decrypt_inline_and_parse::<MailDetailsBlob>`
and extracts the `details` aggregate so the caller stays in terms of
`MailDetails`. The bucketer now routes Mail CREATEs to their own
`mail_creates` bucket so the pre-decrypt step runs ahead of the
MailSetEntry CREATE consumers. Existing CREATE-on-Mail behaviour
(nothing happens directly, MailSetEntry CREATE drives placement) is
preserved.
Fallback chain unchanged: a missing payload, an unresolvable session
key, or a decrypt error leaves the pool entry absent and the
MailSetEntry handler falls back to its existing inline-MailSetEntry →
`load_mail` ladder. 166 bridge lib tests pass (1 new bucket test
covers the routing).
Two new MailBackend methods, `decrypt_inline_mail` and
`decrypt_inline_mail_set_entry`, delegate to the SDK's
`CryptoEntityClient::decrypt_inline_and_parse<T>`. The event handler now
takes the inline path first, and only falls back to `load_mail` if the
payload was missing or its session key was unresolvable:
- Mail UPDATE: a new `resolve_mail` helper centralises the
decrypt-then-fallback policy. Most UPDATE events (read/unread, label
moves, …) now do zero REST calls, since the encrypted Mail rides
inside `event.instance`.
- MailSetEntry CREATE miss path: decode `event.instance` of the
MailSetEntry inline, read its `mail: IdTupleGenerated` field, then
`load_mail` it with the correct list_id. Drops the previous
`mail_list_id_cache` sniffing hack on MailStore (and its test) —
the inline payload always carries the correct list_id, no need to
guess from any cached Mail.
`resolve_mail_set_entry` factors the same try-inline-first pattern for
MailSetEntry events. Both helpers fall back to a `sync_folder` if every
path fails, preserving the no-silent-miss guarantee from Phase 2.
`MockBackend` returns `Ok(None)` from both new methods by default so
existing handler tests keep their REST behaviour; the live SDK fixture
test (`tests/decrypt_inline_test.rs` on the SDK side) covers the real
decryption.
165 bridge lib tests pass.
The new MailSetEntry CREATE/DELETE path applies changes silently to
MailStore + LocalStore; debug logs make it visible in the dev log so
we (and users running with RUST_LOG=debug) can confirm which path was
taken on a given event:
- `Event bus: cloning mail X from <source> → <target> (no REST)` —
hit path, the mail was already cached in another folder.
- `Event bus: targeted load_mail(...) → <target> (1 REST call)` —
miss path, the mail had never been seen.
- `Event bus: removed mail X from <source> (no REST)` — DELETE
applied without re-listing.
Live-tested: a TB-initiated MOVE between two cached folders now
logs the cloning + removed lines and zero `Pre-fetching` / `Removed N
deleted` lines for the affected folders, confirming we no longer touch
the REST API for that case.
The bridge no longer asks the server for a full folder listing on every
MailSetEntry CREATE/DELETE event. Instead it uses the encoding of the
entry id (4-byte timestamp + 9-byte Mail element id, see
`tuta-sdk::mail_set_entry_id`) to recover the affected mail directly,
and applies the delta to MailStore + LocalStore:
- MailSetEntry CREATE first (so a MOVE clones from the source folder
before the matching DELETE runs). Hit path = `find_mail_anywhere` →
clone the already-decrypted StoredMail into the target folder with a
fresh UID. Miss path = a single `load_mail` against the cached
`Mail.list_id`. Any decode failure / unknown folder / `load_mail`
error queues a fallback full `sync_folder` for that folder — no
silent miss.
- MailSetEntry DELETE second. Removes from the source folder only; the
`.eml` and DB row are dropped only if no folder still holds the mail
(multi-folder placement preserved, in-batch MOVE is correct because
the target was upserted by the CREATE loop).
- Mail UPDATE / DELETE unchanged.
- MailSet folder-list dirty handling unchanged.
Saves the `load_range(1000)` round-trip on the common MOVE case. The
fallback path keeps the previous behaviour available so the change is
strictly an optimisation, not a behavioural change.
New MailStore helpers (5 new tests):
- `find_mail_anywhere(eid)` / `is_mail_anywhere(eid)` — multi-folder
lookup.
- `remove_mail_from_folder(folder_id, eid)` — scoped remove (vs the
existing `remove_mail_everywhere`).
- `upsert_mail_in_folder` — idempotent insert/replace by element_id.
- `mail_list_id()` — sniff once, cache; needed by the load_mail miss
path.
`mail_to_metadata` made `pub(crate)` for the handler.
Bucketing in the event handler also split into `mail_set_entry_creates`
vs `mail_set_entry_deletes` so the order is explicit; 8 handler tests
cover empty / mixed / order-preserved / immutable-UPDATE-ignored shapes.
166/166 bridge lib tests pass.
`check_new_mail` only emitted `* N EXISTS` when the folder count changed.
That misses a real-world case: a mail moved between two folders that
share the same `sync_limit` window keeps the count constant (one mail
in, one mail out) while swapping the set — so the IDLE'd Thunderbird
session is never told and silently sticks to a stale view.
Diff by element id instead. For every mail in `self.mails` whose id is
no longer in the refreshed store, push an `* N EXPUNGE` (in descending
seqno per RFC 3501 so subsequent values don't shift). Push one `EXISTS`
afterwards with the new count; the client refetches and discovers any
freshly added mail. Same-set updates stay quiet.
5 new tests cover the four shapes (no-op, growth, removal, swap with
unchanged count) and the descending-EXPUNGE ordering.
SDK (sdk-event-bus amended on the fork): a new `WsState` enum
(`Stopped`/`Connecting`/`Connected`/`Reconnecting`) is broadcast through a
`watch::Sender` inside `EventBusClient`, exposed via `state()` for
observers. Transitions are emitted at every step of the reconnect loop,
plus a `Drop` guard guarantees a final `Stopped` even on a panic in the
caller's task tree. Two new SDK tests cover the initial value and
multi-subscriber broadcast.
Bridge: capture `bus_client.state()` at start, mirror it into a serde
`WsStatus` field on `BridgeStats`, and clear on stop.
UI: add a third stat card "Realtime" with a colored dot (green Connected,
orange pulsing Connecting/Reconnecting, gray Off) reading
`stats.ws_status`. Stats grid switches from 2 to 3 columns.
Submodule pointer bumped to the rebuilt `tutabridge-integration` which
cherry-picks all six SDK branches on `upstream/master`; the move-mails
fixup (use `make_test_facade` so its test compiles alongside the blob
branch) is re-applied. 341/341 SDK lib tests pass, 154/154 bridge tests
pass.
Three Phase-3 polish items, bridge-only:
1. Folder CRUD events. A MailSet (typeId 429) event in a batch now flips
the `Bucketed.folder_list_dirty` flag; the handler refreshes the
folder list and prunes any folder that disappeared from the server
(both in-memory and from LocalStore + .eml files). New helpers
`MailStore::prune_unknown_folders` and `LocalStore::delete_folder_mails`.
2. Out-of-sync detection. The server only replays missed batches for
~44 days. At startup we now check the oldest `event_bus_state` row;
if it predates that window we wipe the table so the syncer falls
through to a bootstrap full sync instead of looping on a server
refusal. New helpers `event_bus_state_min_updated_at_ms` and
`clear_event_bus_state`.
3. Model versions. Drop the hard-coded `SYS_MODEL_VERSION = 150` /
`TUTANOTA_MODEL_VERSION = 108` and read them at compile time from the
vendored SDK's `type_models/{sys,tutanota}.json` via `include_str!` +
`LazyLock`. They now track every SDK submodule bump automatically.
154/154 lib tests pass (6 new across bucket_marks_folder_list_dirty,
prune_unknown_folders, delete_folder_mails, event_bus_state min+clear,
parse_model_version + sanity check on the included JSON).
Pull the event-routing decision out of event_handler::apply_batch into a
pure bucket_updates(&[EntityUpdateEvent]) -> Bucketed function so it can
be tested without standing up a MailStore or hitting the network. Six
new tests cover empty batches, foreign apps, unknown type ids, the
MailSetEntry-list de-duplication, mail-event ordering and a mixed batch.
Add four MailStore tests for the new helpers: refresh_mail_in_place
must update the metadata in every folder that holds the mail (Tuta's
model allows multi-folder placement) while preserving the per-folder
UID, and must no-op on an unknown id; remove_mail_everywhere drops from
all folders and no-ops on an unknown id.
Drop the dead `let _ = bus_event_groups;` in bridge.rs and document why
event_groups() is not passed to the bus: the WebSocket subscribes
implicitly via the auth, and the URL's `groupsToLastEventBatchIds=` is
purely a per-group catch-up cursor.
148/148 lib tests pass.
Replace the silent `min(limit, 1000)` cap with proper pagination: the
server rejects a single `load_range` count > 1000, so for any user-facing
`sync_limit` above that we now loop 1000-entry pages, advancing the
cursor with the last (oldest in DESC) entry's element id, until we have
the requested count or the list is exhausted. `limit == 0` still
delegates to `load_all`, which already paginates the whole list.
Honors what the user typed (1050 means 1050, not 1000).
The Tuta entity REST endpoint rejects `count` > 1000 with a `Bad request`
400. A `sync_limit` above that (e.g. 1050 in config) was silently turning
every bootstrap and folder-re-sync into a 3-attempt retry loop that
always failed. Cap the per-request count and let any excess be picked up
by the realtime event bus going forward; multi-page stitching can come
back if a higher initial snapshot is ever needed.
The 60s list-sync loop is replaced by the WebSocket event bus from the
SDK (sdk-event-bus). On startup the syncer still does Phase 0 (load the
local store into memory), then a one-shot bootstrap sync only if no
event-bus catch-up state is cached. From there on:
- `EventBusClient` runs in its own task, streams `EventBusMessage`s into
an mpsc channel and reconnects with backoff.
- `event_handler` consumes the channel: MailSetEntry CREATE/DELETE
triggers a targeted `sync_folder` for the affected folder; Mail UPDATE
refreshes metadata in place; Mail DELETE drops the cache + .eml.
- After each batch the `(group_id, batch_id)` is persisted in the new
`event_bus_state` SQLite table (schema bumped to v4) and mirrored in
the bus's in-memory map, so the next reconnect resumes catch-up via
`groupsToLastEventBatchIds`.
`stop()` aborts and awaits the new bus + handler tasks alongside the
existing syncer/IMAP/SMTP teardown, so ports release before the next
start rebinds them.
138/138 bridge unit tests pass (incl. 2 new ones for the event-bus state
table). End-to-end behaviour to be verified against the live server.
- Expose the sync limit in the config panel: "Fetch all mail" checkbox (sync_limit=0) + a max-per-folder number input.
- Config fields are editable while the bridge runs; Save persists anytime, a Restart button (stop+start) applies changes.
- Fix the in-process restart: stop() was fire-and-forget and left IMAP/SMTP tasks holding their ports, so the next start failed to bind and aborted the new syncer before it listed folders (empty store). stop() now aborts all three tasks and awaits full teardown.
Previously sync_limit=0 still loaded a single 1000-entry page, so the
mailbox was effectively capped. Use the SDK's paginated load_all to walk
the entire MailSetEntry list when the limit is 0, keeping a single capped
load_range for a finite limit.
Live-tested: with sync_limit=0, INBOX loads all 19288 mails (was capped
at 500) with stable UIDs 1..19288.
UIDs were assigned from an in-memory counter that reset to 1 on every
bridge restart, so the UID<->mail mapping changed each run and IMAP
clients (Thunderbird) re-downloaded the whole mailbox on reconnect.
Persist a per-folder monotonic UID in the local store (schema v3):
- mails gain a `uid` column; sync_state gains a `next_uid` counter that
only ever advances (UIDs are never reused).
- The syncer keeps each mail's existing UID and allocates new ones for
new mail (oldest-first, so newer mail gets higher UIDs).
- refresh_mails uses the persisted UID instead of allocating; messages
are ordered by UID. UIDVALIDITY stays constant.
Migration v2->v3 drops the cache tables and re-syncs once (encrypted
.eml files survive). Live-tested: UID<->mail mapping is identical before
and after a restart (range 1..500 unchanged), so clients fetch only the
delta instead of re-downloading.
Implement MOVE / UID MOVE (RFC 6851): resolve the target by IMAP path
(UTF-7 decoded), call the SDK move_mails for the selected messages, then
expunge them from the source view. Advertise the MOVE capability.
COPY is rejected with NO — Tuta folders are exclusive, so duplication
isn't supported; clients use MOVE instead.
FolderInfo gains the folder's list id so the target MailSet IdTuple can
be reconstructed. Pulls in the SDK move_mails (tuta-repo submodule bump).
Live-tested: a mail moved from one custom folder to a nested UTF-7 folder
lands in the target and leaves the source server-side.
The syncer ran one loop: phase 1 (folder list + mail-id lists) then
phase 2 (body prefetch). On a large mailbox the cold prefetch pass takes
many minutes, so the next folder refresh was stuck behind it and new
folders/mail only showed up after a restart.
Split into two independent loops sharing the store: a fast list_sync_loop
(folder list + mail ids, ~every SYNC_INTERVAL) and a slow prefetch_loop
(bodies, background). Folder and new-mail refresh no longer wait on
prefetch.
Live-tested: a folder created while the bridge runs appears over IMAP in
~18s, no restart.
Add an imap::utf7 module (RFC 3501 §5.1.3) and apply it at the protocol
boundary: encode mailbox names in LIST, decode them in SELECT/STATUS.
Folder names internally stay UTF-8; only the IMAP wire form is UTF-7.
Also fix STATUS argument parsing to handle quoted mailbox names with
spaces (it split on the first space, truncating names like
"Not Important" or nested paths).
Live-tested: "Café" lists as "Caf&AOk-", a nested child lists as
"Caf&AOk-/Test dossier avec espace", and SELECT/STATUS resolve both.
Key the syncer, local store and IMAP server by Tuta MailSet folder id
instead of the system folder kind, so custom (and nested) folders are
first-class.
- tuta.rs: add FolderInfo; MailBackend.list_folders enumerates system +
custom folders via the SDK FolderSystem tree, building IMAP paths and
RFC 6154 special-use flags; load mails by the folder's entries list.
- store.rs: folder_kind INTEGER -> folder_id TEXT, schema v2 with a
migration that drops the cache tables and re-syncs (encrypted .eml
files survive).
- sync.rs: MailStore keyed by folder id; the syncer enumerates the live
folder list each cycle.
- imap/session.rs: dynamic LIST/SELECT/STATUS driven by the folder list;
drop the hardcoded six-folder mapping.
Pulls in the SDK FolderSystem tree (tuta-repo submodule bump).
Live-tested: 9 custom folders listed and selectable over IMAP, with
headers and decrypted bodies. Not yet covered: nested-folder paths,
modified UTF-7 for non-ASCII names, labels, and IMAP MOVE/COPY.
Anthony