This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

7. Incremental Scan Cache, Virtual Folder Exclusion, and Real-Time Progress Feedback

Date: 2026-08-24

Status

Accepted

Context

MailSweep's primary entrypoint (mailsweep sweep / mailsweep scan) scans IMAP mailbox headers to classify clutter and formulate safety-audited cleanup plans.

During full-mailbox scans on large accounts (e.g. 30,000+ messages on Gmail):

1. Redundant Virtual Folder Scanning: Default folder discovery scanned Gmail virtual aggregate folders (\All / [Google Mail]/Kaikki viestit, \Starred, \Important), causing the entire historical email archive to be scanned multiple times.
2. Stateless Re-Scans: Every execution re-fetched all email headers over IMAP SSL, taking up to 25+ minutes even when zero new emails arrived.
3. Lack of Intra-Folder Progress Feedback: MailboxGateway.fetch_headers fetched UIDs in batches of 100 with zero progress callbacks until an entire folder completed, leading users to believe the CLI had frozen on large folders.

Decision

We resolved these performance and usability bottlenecks across three layers:

1. RFC 6154 Virtual Folder Exclusion (src/mailsweep/gateway.py):

   • Automatically exclude RFC 6154 SPECIAL-USE \All, \Starred, and \Important attributes from default list_folders() discovery.
   • Include a 13-language fallback dictionary (Kaikki viestit, Alla meddelanden, Alle Nachrichten, Tous les messages, etc.) for servers lacking RFC 6154 attributes.
   • Retain ability for users to explicitly target virtual folders via IMAP_FOLDER configuration.

2. Incremental Scan Cache (src/mailsweep/gateway.py, src/mailsweep/pipeline.py):

   • Introduce .mailsweep_cache/scan_cache.json storing ScanResult envelope metadata keyed by folder and UIDVALIDITY.
   • On scan:
     • If folder UIDVALIDITY matches cached snapshot, query IMAP only for delta UID > max_cached_uid.
     • If UIDVALIDITY changes (mailbox re-indexed), invalidate cache for that folder and perform a fresh scan.
   • Provide --fresh / --no-cache CLI arguments to bypass local cache on demand.
   • Add .mailsweep_cache/ to .gitignore.

3. Intra-Batch Real-Time Progress Reporting (src/mailsweep/gateway.py, src/mailsweep/cli.py):

   • Pass batch progress (stage, current_in_folder, total_in_folder, folder_name) directly from fetch_headers through MailboxGateway.scan and MailboxPipeline.scan.
   • Implement a zero-dependency in-place terminal updater in cli.py using standard library sys.stdout.write('\r...') with interactive TTY detection.
   • Increase default IMAP header batch size from 100 to 500 for a 5x reduction in network roundtrips.

Consequences

• Full mailbox scans on Gmail and large IMAP accounts drop from 25+ minutes to ~1–2 minutes on first run, and < 1 second on subsequent runs.
• CLI provides immediate, real-time feedback with message counts, percentages, and stage indicators.
• MailSweep remains 100% zero-dependency with no external terminal formatting or database libraries.
• All safety guarantees (UIDVALIDITY assertions and backup invariants) remain strictly enforced.