[spec] Port MailSweep to Synchronous Rust #44

New issue

Closed

opened 2026-08-30 17:56:08 +03:00 by jalmari · 1 comment

jalmari commented

2026-08-30 17:56:08 +03:00

Owner

Specification: MailSweep Synchronous Rust Port

Problem Statement

The existing Python implementation of MailSweep is a robust zero-dependency toolkit, but executing full scans, multi-language regex matching, and streaming backups across mailboxes with tens of thousands of messages encounters memory overhead and execution limits inherent to interpreted runtimes. Deploying across different operating systems also requires Python runtime management and environment configuration. Users require a single, fast, self-contained native executable with zero startup latency, high-throughput regex classification across 13 European languages, and inviolable safety guarantees.

Solution

Rewrite the MailSweep engine and CLI in synchronous Rust as a single unified native binary. The Rust implementation will maintain 100% domain and heuristic parity with the existing architecture, respecting all domain boundaries in CONTEXT.md and architectural decisions in ADR 0008. It will provide sub-second regex evaluation, reliable streaming backups with SHA-256 validation, incremental JSON scan caching, atomic safety aborts on UIDVALIDITY changes, and non-destructive cleanup semantics.

User Stories

As a mailbox administrator, I want to scan my IMAP mailbox using a single self-contained native executable, so that I do not need to install Python runtimes or configure virtual environments.
As a privacy-conscious user, I want the scanner to execute purely locally on my machine without transmitting credentials or message contents to third parties, so that my mailbox privacy remains secure.
As an email user with thousands of messages, I want multi-pass regex classification across 13 European languages to execute in sub-second time, so that I get immediate analysis of my clutter.
As a user with custom workplace domains, I want Tier 0 Domain Immunity to unconditionally protect messages from configured domains, so that critical work communications are never marked as clutter.
As a conversational email user, I want Tier 1 Human Conversation protection to detect In-Reply-To and References headers, so that ongoing direct dialogues are preserved.
As a user who receives invoices and financial records, I want Tier 2 and Tier 3 protections to detect PDF attachments, receipts, and property notices across all 13 supported languages, so that financial records are safe.
As an employee and taxpayer, I want Tier 4 Critical Records protection to detect contracts, tax documents, and healthcare records, so that legal records are never moved to trash.
As an online shopper, I want Tier 5 protections to detect parcel tracking notices and calendar invitations, so that active deliveries and schedules are not disrupted.
As a mailbox owner, I want promotional emails, newsletters, social pings, and surveys to be accurately classified into designated clutter tiers, so that they can be reviewed and cleaned.
As a Gmail user, I want X-GM-EXT-1 smart labels and message IDs to be parsed and enriched, so that Google-specific categorizations and virtual folder deduplication function correctly.
As a user of non-ASCII mailboxes, I want RFC 3501 Modified UTF-7 folder names and RFC 2047 MIME encoded-words to be decoded accurately across UTF-8, ISO-8859-*, and Windows-125x charsets, so that international folder and subject names are rendered without corruption.
As a user performing repeat scans, I want an incremental JSON scan cache that skips unchanged folders based on UIDVALIDITY snapshots, so that delta scans complete in under one second.
As a cautious user, I want an atomic safety abort if a folder's UIDVALIDITY changes, so that out-of-sync message UIDs cannot cause unintended message operations.
As an operator running a cleanup, I want a timestamped ZIP backup containing raw RFC 822 .eml messages with an SHA-256 manifest to be generated and verified before any IMAP modification occurs, so that data loss is mathematically prevented.
As an operator executing a cleanup, I want clutter messages to be moved to the Trash folder rather than permanently deleted, so that a server-side 30-day recovery window is always maintained.
As a user reviewing cleanup plans, I want a clear dry-run summary with an interactive confirmation prompt, so that I can inspect the planned actions before committing them.
As an automation developer, I want a --yes flag to bypass interactive prompts in automated CI or scheduled cron jobs, so that workflows can run unattended.
As a user recovering from an accidental sweep, I want a restore command that moves backed-up messages from Trash back to their original folders, so that cleanup actions can be reversed.
As a subscriber overwhelmed by marketing emails, I want an unsubscribe command that parses RFC 2369 / RFC 8058 List-Unsubscribe headers and extracts direct links or mailto endpoints, so that I can easily unsubscribe from unwanted senders.
As a terminal user, I want real-time progress indicators during header fetching and backup generation, so that I know the tool is actively progressing on large mailboxes.

Implementation Decisions

Domain Architecture Alignment: The codebase will directly reflect the single-context domain architecture documented in CONTEXT.md, implementing the following domain modules:
- Mailbox Gateway: Synchronous IMAP client lifecycle over standard TCP/TLS streams (imap + rustls), managing authentication, folder discovery, batch UID fetching, and atomic message relocation.
- RFC Codecs & Envelopes: Multi-pass RFC 2047 MIME decoding (email-encoding, encoding_rs), RFC 3501 Modified UTF-7 folder codec, IMAP BODYSTRUCTURE attachment detection, X-GM-EXT-1 label decoding, and List-Unsubscribe parsing.
- Rules & Classifier Engine: Pure domain evaluation of the 13 European language dictionaries and heuristic patterns, compiled statically with std::sync::LazyLock and regex. Safety Tiers (0 to 5) take absolute precedence over Clutter Tiers.
- Archive Subsystem: Streaming RFC 822 message extractor writing into timestamped .zip archives with on-the-fly SHA-256 checksum verification (zip, sha2).
- Mailbox Pipeline: Deep domain coordinator encapsulating end-to-end scanning, cache persistence, dry-run safety auditing, backup creation, cleanup execution, and restore operations behind a single testable interface.
- CLI & Formatting: Command parsing with clap (derive mode), progress indicators with indicatif, and standard std::io::stdin confirmation prompts.
Error Handling: Domain errors implemented as structured enums with thiserror to enforce strict domain invariants and safety aborts (UIDVALIDITY changes, backup verification errors, connection drops).
Serialization & Scan Cache: Full JSON schema parity with existing Python .mailsweep_cache/scan_cache.json using serde and serde_json.
Configuration Loading: Environment and .env parsing handled via dotenvy with strong validation into a Config domain struct.
Single Crate Architecture: Structured as a clean single root crate (Cargo.toml) with domain modules under src/.

Testing Decisions

Seam Strategy: The primary test seam is the MailboxPipeline and its pure domain sub-seams (Classifier, RFC Codecs, Archive, ScanCache). Tests will exercise high-level domain behaviors rather than internal implementation details.
Golden Fixture & Parity Testing: Unit and integration tests will utilize existing golden email fixtures (.eml), RFC test vectors, and classification test matrices to guarantee 100% behavioral parity with the Python test oracle.
Deterministic Offline Suite: All codec, classification, cache delta, and archive tests will execute fully offline and deterministically in milliseconds without requiring live IMAP connections.
Rust Quality Verification: All code must strictly conform to:
1. cargo fmt --check
2. cargo clippy --all-targets --all-features -- -D warnings
3. 100% test pass rate with zero compiler warnings.

Out of Scope

Asynchronous tokio network runtime (the engine is deliberately synchronous to prioritize simplicity, predictable memory, and linear error traces).
Graphical User Interface (GUI) or web server interfaces (MailSweep remains a pure CLI toolkit).
Permanent hard deletion of messages (all cleanup operations move messages to Trash to preserve recovery windows).
Third-party cloud telemetry or external analytics.

Further Notes

The Rust implementation will be developed alongside the Python implementation, allowing the existing Python test suite and CLI to serve as a reference oracle during development until full feature parity and verification are achieved.
All error messages, documentation, and user-facing messages will be written in Controlled Language (ASD-STE100).

# Specification: MailSweep Synchronous Rust Port ## Problem Statement The existing Python implementation of MailSweep is a robust zero-dependency toolkit, but executing full scans, multi-language regex matching, and streaming backups across mailboxes with tens of thousands of messages encounters memory overhead and execution limits inherent to interpreted runtimes. Deploying across different operating systems also requires Python runtime management and environment configuration. Users require a single, fast, self-contained native executable with zero startup latency, high-throughput regex classification across 13 European languages, and inviolable safety guarantees. ## Solution Rewrite the MailSweep engine and CLI in synchronous Rust as a single unified native binary. The Rust implementation will maintain 100% domain and heuristic parity with the existing architecture, respecting all domain boundaries in CONTEXT.md and architectural decisions in ADR 0008. It will provide sub-second regex evaluation, reliable streaming backups with SHA-256 validation, incremental JSON scan caching, atomic safety aborts on UIDVALIDITY changes, and non-destructive cleanup semantics. ## User Stories 1. As a mailbox administrator, I want to scan my IMAP mailbox using a single self-contained native executable, so that I do not need to install Python runtimes or configure virtual environments. 2. As a privacy-conscious user, I want the scanner to execute purely locally on my machine without transmitting credentials or message contents to third parties, so that my mailbox privacy remains secure. 3. As an email user with thousands of messages, I want multi-pass regex classification across 13 European languages to execute in sub-second time, so that I get immediate analysis of my clutter. 4. As a user with custom workplace domains, I want Tier 0 Domain Immunity to unconditionally protect messages from configured domains, so that critical work communications are never marked as clutter. 5. As a conversational email user, I want Tier 1 Human Conversation protection to detect In-Reply-To and References headers, so that ongoing direct dialogues are preserved. 6. As a user who receives invoices and financial records, I want Tier 2 and Tier 3 protections to detect PDF attachments, receipts, and property notices across all 13 supported languages, so that financial records are safe. 7. As an employee and taxpayer, I want Tier 4 Critical Records protection to detect contracts, tax documents, and healthcare records, so that legal records are never moved to trash. 8. As an online shopper, I want Tier 5 protections to detect parcel tracking notices and calendar invitations, so that active deliveries and schedules are not disrupted. 9. As a mailbox owner, I want promotional emails, newsletters, social pings, and surveys to be accurately classified into designated clutter tiers, so that they can be reviewed and cleaned. 10. As a Gmail user, I want X-GM-EXT-1 smart labels and message IDs to be parsed and enriched, so that Google-specific categorizations and virtual folder deduplication function correctly. 11. As a user of non-ASCII mailboxes, I want RFC 3501 Modified UTF-7 folder names and RFC 2047 MIME encoded-words to be decoded accurately across UTF-8, ISO-8859-*, and Windows-125x charsets, so that international folder and subject names are rendered without corruption. 12. As a user performing repeat scans, I want an incremental JSON scan cache that skips unchanged folders based on UIDVALIDITY snapshots, so that delta scans complete in under one second. 13. As a cautious user, I want an atomic safety abort if a folder's UIDVALIDITY changes, so that out-of-sync message UIDs cannot cause unintended message operations. 14. As an operator running a cleanup, I want a timestamped ZIP backup containing raw RFC 822 .eml messages with an SHA-256 manifest to be generated and verified before any IMAP modification occurs, so that data loss is mathematically prevented. 15. As an operator executing a cleanup, I want clutter messages to be moved to the Trash folder rather than permanently deleted, so that a server-side 30-day recovery window is always maintained. 16. As a user reviewing cleanup plans, I want a clear dry-run summary with an interactive confirmation prompt, so that I can inspect the planned actions before committing them. 17. As an automation developer, I want a `--yes` flag to bypass interactive prompts in automated CI or scheduled cron jobs, so that workflows can run unattended. 18. As a user recovering from an accidental sweep, I want a restore command that moves backed-up messages from Trash back to their original folders, so that cleanup actions can be reversed. 19. As a subscriber overwhelmed by marketing emails, I want an unsubscribe command that parses RFC 2369 / RFC 8058 List-Unsubscribe headers and extracts direct links or mailto endpoints, so that I can easily unsubscribe from unwanted senders. 20. As a terminal user, I want real-time progress indicators during header fetching and backup generation, so that I know the tool is actively progressing on large mailboxes. ## Implementation Decisions - **Domain Architecture Alignment**: The codebase will directly reflect the single-context domain architecture documented in CONTEXT.md, implementing the following domain modules: - **Mailbox Gateway**: Synchronous IMAP client lifecycle over standard TCP/TLS streams (`imap` + `rustls`), managing authentication, folder discovery, batch UID fetching, and atomic message relocation. - **RFC Codecs & Envelopes**: Multi-pass RFC 2047 MIME decoding (`email-encoding`, `encoding_rs`), RFC 3501 Modified UTF-7 folder codec, IMAP `BODYSTRUCTURE` attachment detection, `X-GM-EXT-1` label decoding, and `List-Unsubscribe` parsing. - **Rules & Classifier Engine**: Pure domain evaluation of the 13 European language dictionaries and heuristic patterns, compiled statically with `std::sync::LazyLock` and `regex`. Safety Tiers (0 to 5) take absolute precedence over Clutter Tiers. - **Archive Subsystem**: Streaming RFC 822 message extractor writing into timestamped `.zip` archives with on-the-fly `SHA-256` checksum verification (`zip`, `sha2`). - **Mailbox Pipeline**: Deep domain coordinator encapsulating end-to-end scanning, cache persistence, dry-run safety auditing, backup creation, cleanup execution, and restore operations behind a single testable interface. - **CLI & Formatting**: Command parsing with `clap` (derive mode), progress indicators with `indicatif`, and standard `std::io::stdin` confirmation prompts. - **Error Handling**: Domain errors implemented as structured enums with `thiserror` to enforce strict domain invariants and safety aborts (`UIDVALIDITY` changes, backup verification errors, connection drops). - **Serialization & Scan Cache**: Full JSON schema parity with existing Python `.mailsweep_cache/scan_cache.json` using `serde` and `serde_json`. - **Configuration Loading**: Environment and `.env` parsing handled via `dotenvy` with strong validation into a `Config` domain struct. - **Single Crate Architecture**: Structured as a clean single root crate (`Cargo.toml`) with domain modules under `src/`. ## Testing Decisions - **Seam Strategy**: The primary test seam is the `MailboxPipeline` and its pure domain sub-seams (`Classifier`, `RFC Codecs`, `Archive`, `ScanCache`). Tests will exercise high-level domain behaviors rather than internal implementation details. - **Golden Fixture & Parity Testing**: Unit and integration tests will utilize existing golden email fixtures (`.eml`), RFC test vectors, and classification test matrices to guarantee 100% behavioral parity with the Python test oracle. - **Deterministic Offline Suite**: All codec, classification, cache delta, and archive tests will execute fully offline and deterministically in milliseconds without requiring live IMAP connections. - **Rust Quality Verification**: All code must strictly conform to: 1. `cargo fmt --check` 2. `cargo clippy --all-targets --all-features -- -D warnings` 3. 100% test pass rate with zero compiler warnings. ## Out of Scope - Asynchronous `tokio` network runtime (the engine is deliberately synchronous to prioritize simplicity, predictable memory, and linear error traces). - Graphical User Interface (GUI) or web server interfaces (MailSweep remains a pure CLI toolkit). - Permanent hard deletion of messages (all cleanup operations move messages to Trash to preserve recovery windows). - Third-party cloud telemetry or external analytics. ## Further Notes - The Rust implementation will be developed alongside the Python implementation, allowing the existing Python test suite and CLI to serve as a reference oracle during development until full feature parity and verification are achieved. - All error messages, documentation, and user-facing messages will be written in Controlled Language (ASD-STE100).

jalmari added the

ready-for-agent

spec

labels

2026-08-30 17:56:08 +03:00

jalmari referenced this issue

2026-08-30 17:58:52 +03:00

[feat] Core Domain Types, Configuration & RFC Codecs #45

jalmari referenced this issue

2026-08-30 17:58:52 +03:00

[feat] 13-Language Heuristic Classifier & Safety Tier Engine #46

jalmari referenced this issue