mailsift

Unified IMAP monitoring for important email. mailsift scans every configured mailbox—including spam folders—uses an LLM plus local rules to classify messages, and sends important results to Feishu or a generic webhook.

mailsift is an assistant, not a mail client. It is read-only by default. A separate write token can enable a small audited set of mailbox actions (mark, star, and move); it never sends, replies to, deletes, or silently changes mail. Classification can be wrong, so keep using your mail client as the source of truth.

中文文档：README.zh-CN.md。

Features

• Any IMAP provider, with multiple accounts.
• Spam-folder monitoring with provider-specific folder-name detection.
• Personal rules plus LLM classification; keyword fallback when the LLM is not configured or fails.
• Real-time push notifications and a daily digest.
• Crash-safe cursor handling and recovery for interrupted processing.
• Optional MCP server for querying local triage results.
• Conversation grouping, bounded search across triage text and attachment metadata, and an optional audited mailbox action surface.

Quick start

git clone https://github.com/itswl/mailsift.git
cd mailsift
cp .env.example .env
vim .env
docker compose up -d

At minimum, configure one mailbox, one output (FEISHU_WEBHOOK_URL or WEBHOOK_URL), and optionally an LLM API key. For local development with Node >= 22.5:

npm install
npm run check
npm run dev

Configuration

All settings are environment variables. See .env.example for the complete template.

MAIL_ACCOUNT_1=qq|[email protected]|app-password
MAIL_ACCOUNT_1_NAME=personal-qq
MAIL_ACCOUNT_1_FOLDERS=all

MAIL_ACCOUNT_2=imap|[email protected]|password|imap.example.com

FEISHU_WEBHOOK_URL=
WEBHOOK_URL=
WEBHOOK_TOKEN=

LLM_PROVIDER=deepseek
LLM_MODEL=deepseek-flash
LLM_API_KEY=

PUSH_MIN_IMPORTANCE=warning
SPAM_RANK_BONUS=0
DIGEST_ENABLED=true
DIGEST_HOUR=9

MAIL_ACCOUNT_N_FOLDERS accepts all, INBOX,spam, or explicit folder names. The all token excludes sent, drafts, trash, and provider-wide virtual views. The spam token detects the provider’s spam folder.

SUPPRESS_SELF_FORWARDS=true (the default) stops one message notifying twice when your mailboxes forward to each other. A forwarded copy is a new message with its own Message-ID, so deduplication cannot recognise it, but its sender is one of the configured accounts. Such a copy is recorded and stays searchable through list_mail, but it is neither pushed nor repeated in the daily digest, since the original was already triaged at its source. It costs no LLM call either. Mail you send to yourself inside a single mailbox is unaffected, and an explicit MAIL_ALWAYS_IMPORTANT sender still wins.

SPAM_RANK_BONUS lifts spam-folder mail by whole importance levels, so with the default PUSH_MIN_IMPORTANCE=warning a bonus of 1 pushes every spam-folder message: warning and critical were already above the line and info now sits on it. Leave it at 0 unless PUSH_MIN_IMPORTANCE is critical and you want spam-folder warnings to break through anyway. Catching a message wrongly filed as spam does not need it, because triage assesses spam normally and rates a real bill or code warning or critical there. --check warns about the combinations that deliver the whole spam folder.

MAIL_CONTEXT, MAIL_ALWAYS_IMPORTANT, MAIL_NEVER_IMPORTANT, and MAIL_KEYWORDS customize classification. Keep MAIL_CONTEXT on one line in .env. Sender rules take three forms: @bank.com matches the address domain or a subdomain of it, [email protected] matches exactly that address, and anything else is a substring test on the address and display name. The first two ignore the display name so a look-alike domain or a crafted name cannot trigger an always-important rule.

Gmail and Outlook OAuth

Two guides in Discussions cover this:

• Authorising Gmail and Outlook, step by step against the scopes and redirect this code actually uses, including the two traps that cost the most time: a Google consent screen left in Testing expires refresh tokens every 7 days, and Outlook.com ships with IMAP switched off.
• Forwarding, which is how to skip Gmail’s OAuth altogether by forwarding the whole mailbox to an account that uses a password. Spam needs a filter to come along, since plain forwarding leaves it behind. Outlook has no equivalent, because neither its forwarding setting nor an inbox rule touches Junk, so an Outlook mailbox is worth the OAuth. The same guide covers what to do when two monitored mailboxes forward to each other.

Gmail and Outlook require OAuth for most accounts:

npm run oauth

For a headless server or Docker:

docker compose run --rm mailsift node dist/scripts/oauth-setup.js --manual

The refresh-token store is data/tokens.json. Treat it as a long-lived mailbox credential, keep it out of version control, and preserve its file ownership in Docker (uid 10001). Personal Gmail can use gmail_pw with an app password instead of OAuth; Google Workspace cannot.

Polling and backfill limits

The first scan uses INITIAL_LOOKBACK_DAYS (default 3). If a fresh folder or a folder whose UIDVALIDITY changed returns more than MAX_MESSAGES_PER_LOOKBACK messages (default 500), mailsift processes the oldest bounded chunk and continues the backfill on later polls. It never advances past an unprocessed backlog just to avoid repeated work.

Each folder is limited to MAX_MESSAGES_PER_POLL (default 200). All accounts and folders share MAX_MESSAGES_PER_POLL_TOTAL (default 500). When the ordinary total cap is reached, unprocessed folders keep their cursors and continue on the next poll. Selected UIDs are processed oldest-first so a cap does not skip older mail.

Messages up to MAX_MESSAGE_SOURCE_BYTES (default 5 MiB) use the normal MIME parser. For larger messages, mailsift reads only the selected text/plain body part, falling back to HTML, so a large attachment is not pulled into triage. If that text body alone exceeds the limit, the message is recorded as a dead letter and can be inspected through list_dead_letters; list_attachments can still look up its files by account and Message-ID.

The MCP recovery tools expose recovery_status always, and retry_dead_letter only when the write credential below is configured. Retrying rewinds one folder cursor so the next normal poll can fetch the message again; it does not modify the mailbox.

MCP also provides observability for processing, delivery, dead-letter, feedback, and mailbox-action totals, plus the access audit grouped by tool. list_audit shows recent tool/resource events and audited mailbox actions with bounded targets, without credentials or message bodies. record_feedback records false_positive, missed, handled, and correct labels for later rule and triage evaluation, and also requires the write credential. list_threads groups messages by their References/In-Reply-To root, while search_mail also checks the stored body preview and attachment metadata.

After two feedback records for the same sender, missed feedback infers an always-important rule and false_positive feedback infers a never-important rule. Explicit environment rules take precedence; inspect inferred rules with MCP feedback_rules.

State is stored in SQLite (data/mailsift.db). Docker persists it in the mailsift-data named volume.

Real-time wake-ups with IMAP IDLE

IMAP_IDLE_ENABLED=true keeps one read-only connection per account and IDLE folder (IMAP_IDLE_FOLDERS, default INBOX) open. When the server reports new mail, mailsift fetches that folder at once on the same connection and runs the usual deduplicate, triage, and deliver pipeline, so a wake-up costs no extra login. Polling continues as the reconciliation pass and catches anything a notification missed; with IDLE on, POLL_INTERVAL_SECONDS can usually be raised to 900-1800. npm run probe shows whether a server advertises IDLE; accounts without it stay on polling. Listeners reconnect with backoff after a drop, log out cleanly on shutdown, and the MCP health tool reports the last wake-up per account.

OpenTelemetry metrics

Metrics are instrumented with OpenTelemetry and disabled by default. Set OTEL_EXPORTER_OTLP_ENDPOINT to send OTLP metrics to an OpenTelemetry Collector. The instruments cover poll duration, account outcomes, triage decisions, notifications, dead letters, and the outbox. Attributes intentionally use only low-cardinality values such as provider, outcome, channel, importance, and decision source.

LLM_SKIP_SENSITIVE=true keeps verification codes, one-time passwords, and auth-code messages in the local keyword/rule path. They remain eligible for normal notifications, but their content is never sent to the configured LLM.

LLM_REDACT_PII=true (the default) redacts email addresses, phone numbers, payment card numbers, and resident ID numbers from fields sent to the LLM. Card and ID numbers are only redacted when their checksum holds, so waybill, order, and invoice numbers stay readable in summaries. Local state and notifications keep the original values. Set it to false only when the configured endpoint is trusted and the additional context is necessary.

LLM_THREAD_CONTEXT_MESSAGES (default 3, range 0-5) adds recent earlier messages from the same mailbox thread to a new message’s LLM input. It uses stored subjects, senders, summaries, and bounded snippets, plus bounded snippets from earlier messages fetched in the same poll. It does not fetch older messages or persist additional body text. Context is sent only when the message already needs LLM classification, and the normal PII redaction applies. Set it to 0 to disable.

The LLM integration is OpenAI-compatible: mailsift posts to ${LLM_BASE_URL}/chat/completions with model, messages, and (when enabled) response_format: {"type":"json_object"}. The response must provide JSON in choices[0].message.content with one result per input index. Set LLM_JSON_MODE=false for providers that reject response_format; mailsift also retries once without it when the rejection is explicit.

LLM_OUTPUT_LANGUAGE sets the language of summaries and reasons: en (default), zh-CN, zh-TW, auto to follow each message, or any language name.

Categories are not translated. They come from a fixed vocabulary, Security, Finance, Delivery, Travel, Health, Legal, Work, Personal, Social, Marketing, System, Other, and anything else the model returns is folded into Other. The daily digest groups by category and MCP callers search on it, so a stable vocabulary matters more than a vivid label. Rule-decided mail may also carry Always important, Never important, Feedback rule or Forwarded copy.

If an LLM request fails, returns invalid JSON/schema, or returns duplicate or out-of-range indices, the affected batch uses the local fallback, because a misaligned index list cannot be trusted for any message in it. A message that is merely omitted from an otherwise valid response falls back on its own. Keyword matches remain warning-level and can be pushed; messages without a high-risk keyword become info and are queued for digest/review. Fallback does not stop polling. After LLM_ALERT_AFTER_FAILURES consecutive failures, mailsift sends an outage alert. When calls recover, it re-triages fallback messages that are still waiting in the digest, using their locally stored snippets; already delivered messages are left alone. If re-triage also fails, those digest entries stay available for a later retry.

Signal events

Webhook deliveries retain the existing mail / triage payload and add a small source-neutral signal.v1 event. It carries the triage summary and an MCP lookup reference instead of copying the raw message body. The reference can fetch a bounded normalized body from IMAP on demand without persisting it; set MCP_LIVE_BODY_CHARS to tune the returned body cap. Live lookup is read-only and bounded by MCP_LIVE_SOURCE_BYTES, MCP_LIVE_LOOKBACK_DAYS, and MCP_LIVE_SEARCH_MAX_MESSAGES.

The reference is Bearer-authenticated only when MCP_TOKEN is configured. A loopback MCP listener with an empty token is intentionally unauthenticated, so do not expose it beyond the host. See docs/signal-event-v1.md for the contract.

Running

docker compose up -d
docker compose logs -f
docker compose pull && docker compose up -d

npm run check
npm run dev -- --once
npm run dev -- --once --dry-run
npm run probe
npm run probe -- --account [email protected] --no-counts
npm run oauth -- --account [email protected] --manual --force
npm run dev -- --digest-now
npm run dev -- --healthcheck
npm run dev -- --recover

--once polls once and exits with status 2 if an account fetch fails. --check validates configuration and OAuth authorization; --dry-run logs notifications without sending them; --digest-now sends the current digest; --healthcheck checks only whether a recent poll heartbeat exists; and --recover drops undispatched records and rewinds cursors after an interrupted run.

probe checks folders and recent counts by default. Use --no-counts for a lighter connectivity/folder check, or --account to select one account. OAuth supports --account, --manual, and --force.

Compose uses the released image version 1.0.0 by default. Set MAILSIFT_VERSION in .env when upgrading; this keeps deployments away from the mutable latest tag. The container stores SQLite state and OAuth refresh tokens in the mailsift-data named volume. A host-side npm run oauth writes to ./data/tokens.json, which is not the Docker volume; run the OAuth command through docker compose run --rm mailsift ... when authorizing the container’s accounts.

To build from the checked-out source instead of pulling the image:

docker compose -f docker-compose.yml -f docker-compose.build.yml up -d --build

For systemd, first run npm install and npm run build, then use deploy/mailsift.service. Change WorkingDirectory to the directory containing .env and data/, and change ExecStart if which node is not /usr/bin/node. The service user must own or be able to write the data directory; a dedicated low-privilege user is recommended. Recovery after an interrupted dispatch:

docker compose run --rm mailsift node dist/src/main.js --recover

MCP

The embedded MCP server is enabled by default in the main process. In Docker, MCP_BIND is the container listen address and MCP_PUBLIC_HOST is the host-side publish address. The Compose file uses MCP_PORT for both host and container ports; by default it is reachable only at 127.0.0.1:8410 on the host:

MCP_ENABLED=true
MCP_PUBLIC_HOST=127.0.0.1
MCP_BIND=0.0.0.0
MCP_PORT=8410
MCP_TOKEN=

For public HTTP access, use a strong token and set MCP_PUBLIC_HOST deliberately. A non-loopback MCP_BIND refuses to start without MCP_TOKEN; loopback with an empty token is unauthenticated by design. The endpoint is /mcp, uses stateless Streamable HTTP, and applies MCP_RATE_LIMIT_PER_MINUTE (default 120). Use HTTPS through a reverse proxy or tunnel because MCP carries mailbox data and bearer tokens must not cross the public internet over plain HTTP. For local stdio clients, point the command at dist/src/mcp.js.

The surface is read-only unless you opt in

manage_mail, record_feedback, and retry_dead_letter are write tools, and they are not registered at all unless MCP_WRITE_TOKEN is set. manage_mail supports mark_read, mark_unread, star, unstar, and move; it refuses delete/trash destinations and never sends mail. Every operation resolves the Message-ID again inside a writable IMAP lock and records an audit row. The separate token keeps a leaked read credential from changing either mailbox contents or future triage policy.

Set MCP_WRITE_TOKEN to a second secret that differs from MCP_TOKEN; reusing one value for both is refused at startup. Present it as the bearer token to get the full surface. MCP_TOKEN then stays a read-only credential, which is what the browser view and any shared client should hold. Over stdio the caller can already read the env file, so setting MCP_WRITE_TOKEN alone enables the write tools there without a second presentation.

Reading attachments

list_attachments takes a configured account and Message-ID, and returns MIME metadata plus a mailsift://imap/.../attachment/{partId} resource URI for each file. Read the resource to fetch that one file from IMAP. The web view also lists and downloads attachments from a message’s detail panel. Reads use a read-only mailbox lock; attachment bytes are never stored. New triage records keep only a bounded manifest of names, MIME types, part IDs, and reported sizes.

MCP_ATTACHMENT_MAX_BYTES limits the decoded size of each on-demand file and defaults to 10 MiB (configuration is capped at 100 MiB). This limit is separate from MCP_LIVE_SOURCE_BYTES, so attachment reads fetch the selected MIME part without downloading the entire message source. Filenames are treated as untrusted data and files are delivered as downloads. mailsift returns image and PDF files as-is; it does not recognize images or extract PDF text.

scan_attachment and the Scan attachment button read only a bounded decoded prefix, controlled by MCP_ATTACHMENT_SCAN_BYTES (default 1 MiB, hard cap 4 MiB). The local check looks for suspicious filename/MIME combinations and common executable, script, active-content, and container signatures. It does not unpack, execute, upload, persist, or claim to replace antivirus software.

Link parsing reads at most MCP_SAFETY_BODY_BYTES (default 256 KiB, hard cap 1 MiB) from a selected text body part. It reports scheme, IP-host, punycode, userinfo, and common shortener clues while omitting URL paths and query values.

Spam and safety clues

The inspect_mail_safety tool checks a message’s configured folder, mailing-list or bulk-mail headers, a From/Reply-To domain mismatch, attachment filename/MIME types, and links parsed from a bounded text body prefix. The web detail view offers the same check, and attachment lists show file-type cautions and a bounded scan button. Mailbox reads use a read-only lock; only the selected text body prefix is read for link parsing, and no link is opened.

These are clues, not a spam or malware verdict. Mailing-list headers do not prove spam; sender fields can be forged. mailsift does not verify SPF, DKIM or DMARC, resolve or reputation-check links, unpack attachments, or provide an antivirus verdict. It does not move, delete, flag, or otherwise change messages as part of these checks.

Discovering folders

list_folders takes the exact account address returned by list_accounts and lists its decoded IMAP folder paths, hierarchy delimiter, flags, special-use marker, and selectability. It uses IMAP LIST without opening folders or changing mail.

Every tool and live resource call is audited by name, outcome, caller fingerprint, and (where useful) a bounded account/message/part target; rejected authentication attempts are included. list_audit returns recent rows and mailbox-action audit. Credentials, message bodies, and URL query strings are never stored in the audit.

health reports the last poll heartbeat, account failures, LLM failure count, and pending queues. The Docker healthcheck checks only the poll heartbeat; a healthy container does not prove that every account, LLM call, or notification sink is healthy. Use MCP health and recovery_status for those details.

Read-only web view

WEB_UI_ENABLED=true serves a browser view of the triage results at http://:/, on the same port and behind the same MCP_TOKEN as MCP, because it is the same trust boundary. It lists messages or conversation-like threads, filters by window, level, category and mailbox, searches subject, sender, summary and attachment metadata, previews the queue behind the next digest, and links out to the provider’s own client. A message body and individual attachments can be fetched live on request; attachment files are downloaded and never cached or persisted by mailsift.

The browser view remains read-only in the strict sense: mailbox calls fetch a message body or a selected MIME part under a read-only lock. Use MCP with the separate write token for the small audited action set; sending, replying, deleting, and other mailbox management still belong in a real webmail client such as Roundcube.

Off by default. A page a browser can reach is a wider surface than a JSON-RPC endpoint, so enable it deliberately, keep MCP_BIND on loopback, and put TLS in front of it if you reach it from a phone.

Signing in posts the token and exchanges it for an HttpOnly cookie that lasts six months and is renewed on each visit, so a browser in regular use is never asked again. The form carries an account field so a password manager will offer to save the token rather than leaving you to type it. An installed app keeps its own cookies on iOS, so expect to sign in once inside it even if the browser is already signed in.

The header carries a two-state light/dark switch. The choice is stored in the browser and applied before the body renders, so it never flashes the other theme, and it moves the status bar colour with it rather than leaving the top of an installed app behind.

Next to it is a language switch between English and Simplified Chinese. The page follows the browser’s language until the switch is used; that choice is kept in a cookie, so the server renders the page, and the sign-in form before it, in that language from the first paint. Only the page’s own words change. Subjects, summaries and reasons are shown as stored, in the language LLM_OUTPUT_LANGUAGE told the model to write, and the digest preview is the text that will actually be sent.

It installs to a home screen: the manifest, a maskable icon and a service worker that caches nothing are served without a session, since a browser asks for them before it has one. Installed, it opens without browser chrome, paints the status bar to match the page in either colour scheme, and reaches under a notch with the safe-area insets padded back. The worker deliberately caches nothing, because a page showing yesterday’s triage would be worse than no page. Regenerate the icon with npx tsx scripts/icons.ts after changing the mark.

Behind a reverse proxy, set TRUSTED_PROXY=true. Otherwise every request arrives from the proxy’s address, the per-client rate limit collapses into one shared budget, and a single noisy caller can lock out your own browser and any MCP client. With it set, the limit and the audit log key on CF-Connecting-IP, X-Real-IP or the first X-Forwarded-For entry. Leave it off when the port is reached directly, since anyone can send those headers. Sign-in gets a separate, much smaller budget of its own, because it is the only route that answers without credentials.

Troubleshooting

• Repeated notifications: messages are deduplicated by account plus Message-ID. --recover is for interrupted, undecided records.
• Nothing was notified: call MCP health first. Its delivery block reports the push and digest thresholds, the spam bonus, and whether self-forwards are suppressed, which usually explains the decision without further digging. recovery_status then shows folder UID progress and any notification still queued for retry.
• Delivery failures: a push that fails with a network error, a timeout, or HTTP 408/429/5xx is retried up to SINK_RETRY_ATTEMPTS times (default 3, seconds apart) and then waits in the durable outbox. Outbox retries use persisted exponential backoff (OUTBOX_RETRY_BASE_SECONDS, default 60; OUTBOX_RETRY_MAX_SECONDS, default 3600), so a failed endpoint is not hammered on every poll. recovery_status shows the next attempt time.
• Outbox backlog: only real failures wait there. An output that declines a message under its own threshold, such as FEISHU_MIN_IMPORTANCE, settles it immediately and the digest carries it, so a pending count in MCP health always means something is genuinely undelivered.
• QQ scans many messages: some QQ IMAP endpoints ignore SINCE; the local UID cursor still preserves correctness.
• Messages marked read: mailsift uses read-only mailbox locks and BODY.PEEK[].
• LLM privacy: LLM_SKIP_SENSITIVE=true keeps verification-code messages local, and LLM_REDACT_PII=true redacts common direct identifiers before sending fields to the LLM. Omit LLM_API_KEY for local keyword fallback or use a self-hosted endpoint.
• OAuth expiry: access tokens refresh automatically. Re-authorize with npm run oauth -- --account [email protected] --force after revocation or credential changes.

Development

npm test
npm run typecheck
npm run build

License

MIT