The problem
The platform's sending path ran on a commercial MTA fleet. It worked, but three things about it were structurally wrong for the way campaigns were actually being run.
Recovery was a guess. When a send was interrupted — a restart, a host problem, an operator pausing mid-campaign — there was no authoritative record of which individual messages had reached a terminal state. Resuming meant choosing between re-sending messages that had already been accepted, or dropping messages that had not. In email, the first option damages sender reputation and annoys recipients; the second silently loses revenue. Neither is acceptable, and the choice was being made by operators under time pressure.
Throughput was configured, not learned. Per-provider limits were static numbers in a config file. Real receiving infrastructure does not behave like a static number: it throttles differently by time of day, by reputation, and by how hard you have just pushed it. Static limits are either too conservative (campaigns take longer than they need to) or too aggressive (deferrals and blocks that cost far more than the time saved).
Configuration had sharp edges that had already cost us. One example that
became an invariant: in that MTA's version 5, a rate directive of 0 means zero
throughput, not unlimited. A reasonable person reading the config sees "no
limit". The system sees "send nothing". Traps like this are not documented
failures — they are silent ones.
Why existing tools were insufficient. The commercial MTA is a good product; it was not built for our shape of work. We inject millions of pre-personalized messages at import time and then deliver them as fast as receiving infrastructure allows. That shape wants an injection-time inventory, a delivery-time event log, and a scheduler that can rebuild its entire state from those two things. Bolting that onto a closed system was not possible.
Previous workflow
Before: campaign delivery on the commercial MTA fleet
Every box that follows a failure has a human in it.
- Human action
Operator prepares campaign
Recipient list, template and sending identity assembled by hand.
- Automated software
Platform injects messages over SMTP
Messages are wrapped and handed to the MTA fleet.
- Problem: No injection-time inventory the platform could later reconcile against
- Problem: Injection and delivery state lived in two systems that could disagree
- External service
MTA fleet delivers to receiving providers
- Problem: Static per-provider rate limits, tuned by guesswork
- Problem: Config traps where a plausible-looking value silently means zero throughput
- Automated software
Accounting logs parsed for outcomes
Delivery results are recovered by reading log files after the fact.
- Problem: Outcome visibility lagged behind the send
- Problem: Three fleet nodes had to be kept in configuration lockstep by hand
- Human action
Interruption — operator decides what to do
Decision point: Re-send the whole segment and risk duplicates, or skip it and risk dropping messages?
- Problem: No per-message terminal record to resume from
- Problem: The decision was made live, under time pressure, with incomplete information
- Problem: Ownership gap: nobody owned 'what actually happened to message X'
- Human action
- Automated software
- External service
Goals and non-goals
Goals
- A send interrupted at any point resumes without re-delivering a message that already reached a terminal state.
- Per-destination concurrency adapts to observed behavior rather than static configuration.
- Delivery identity is explicit and enforced — one campaign delivers from one sending identity, never fanned across several for speed.
- Every delivery outcome is recorded in an append-only log that can be replayed independently of the running process.
- An operator can see what the engine is doing without reading log files.
Non-goals
- Not a general-purpose MTA. No inbound mail, no local delivery, no mailbox handling. It is an outbound delivery engine for one workload.
- Not a replacement for the platform. Campaign preparation, suppression, tracking and reporting stay in the platform. giMTA delivers.
- Not a throughput record attempt. Faster than receiving infrastructure will accept is not a feature; it is a reputation problem.
- No silent fan-out. Throughput is never bought by spreading one campaign across multiple sending identities.
My role
I owned this as the product and program lead, working with AI assistance under the discipline described in the AI-assisted engineering workflow.
- Discovery — traced the real failure modes on the existing fleet, including the interrupted-send decision that had no good answer.
- Requirements — wrote the recovery contract as a testable invariant before any code existed: index plus event log must be sufficient to rebuild scheduler state.
- Architecture — set the crate boundaries and the three-way relationship between spool, index and event log that the whole design rests on.
- Deliverability policy — made one-campaign-one-lane a hard invariant rather than a tunable, and wrote down why.
- Implementation coordination — decomposed the work into crate-sized units with explicit contracts.
- Test planning — specified the crash-recovery, DKIM drift and DNS pre-warm gate tests.
- Production cutover — ran the canary sequence and the rollback plan.
- Operation — diagnosed live throughput and deferral behavior and fed it back into the tuning loop.
Product solution
giMTA is a single binary. An operator imports a campaign, starts the engine, and watches a dashboard.
Import takes a recipient set and a template, writes the rendered messages into segment files, optionally signs them, and writes an atomic index describing exactly what was injected and where each message lives.
Start loads that index, replays the campaign's event log to discover what has already happened, and reconstructs the scheduler's live state from the merge. Messages with a terminal outcome are excluded from the delivery queues automatically — not by an operator's judgment.
Delivery leases batches to engine workers, which hold pooled SMTP connections partitioned by destination group. Every outcome — accepted, deferred, failed — is appended to the event log as it happens.
Tuning runs continuously. A learning agent maintains smoothed per-destination metrics and recommends a concurrency level, which the engine applies within configured ceilings.
The unit an operator thinks in is the lane: one outbound address, one sending domain, one signing key. Capacity is a separate concept — engines are parallel workers operating for a lane. Keeping those two ideas separate is what lets capacity scale without ever compromising sending identity.
Workflow
After: campaign import to delivery
The pre-warm gate and the event log are the two additions that changed the operating experience.
- Human action
Operator imports the campaign
Recipient set plus template, named as a campaign.
- Automated software
Render and spool
Messages written to segment files as raw bytes.
- Automated software
Sign
Messages signed at spool time or immediately before transmission, depending on configuration.
- Data store
Atomic index written
Written to a temporary file, flushed, then renamed — so the index is either complete or absent, never half-written.
- Automated software
Signing-key drift check
Decision point: Does the signing key match the one recorded at import? If not, stop rather than send unverifiable mail.
- Approval gate
DNS pre-warm gate
All recipient domains are resolved before sending starts.
Decision point: If the resolution success ratio is below the configured floor, abort the start instead of sending into an unknown routing picture.
- Data store
Event log replay
The campaign's append-only log is replayed to find every terminal outcome.
- Automated software
Scheduler state rebuilt
Index plus replayed outcomes produce the live message set. Anything already terminal is excluded.
- Automated software
Engines lease and deliver
Workers lease batches and deliver over pooled connections partitioned by destination group.
- External service
Receiving infrastructure responds
Decision point: Accepted, deferred or permanently failed — each drives a different scheduler transition.
- Data store
Outcome appended to event log
Append-only, per campaign. This is the record of truth for anything after injection.
- Monitoring / feedback
Learning agent updates
Smoothed per-destination metrics feed a recommended concurrency the engine applies within ceilings.
- Human action
Operator watches the dashboard
Live queue depth, throughput and per-destination behavior, without reading log files.
- Human action
- Automated software
- Data store
- External service
- Approval gate
- Monitoring / feedback
Architecture
giMTA — sanitized component view
A workspace of focused crates. Boundaries are enforced by the crate graph, not by convention.
Operator surface
Command line
Human
Import, start, status, tune, benchmark, canary planning and signing-key checks.
Control plane and dashboard
Automated
HTTP API and live view of queue state and throughput.
Durable state
The three-way relationship these form is the central architectural invariant.
Segment files
Data store
Raw message bytes, written once at import.
Atomic index
Data store
Injection-time inventory. Written once via write-flush-rename; never mutated afterwards.
Event log
Data store
Append-only per-campaign record of every delivery outcome. Authoritative for anything after injection.
Delivery core
Scheduler
Automated
Sharded ready queues, lease accounting and a timer wheel for deferred retries.
Engine workers
Automated
Bounded concurrency per engine; lease, transmit, record outcome.
Connection pool
Automated
Pooled sessions partitioned by destination group, with reuse and cap limits.
Supporting subsystems
Signing
Automated
Standards-compliant message signing with a key fingerprint recorded at import.
DNS cache
Automated
Pre-warmed routing lookups with a start-time success-ratio gate.
Content checks
Automated
Pre-send content and link inspection.
Metrics registry
Monitoring
Counters and gauges exported for scraping.
Learning agent
Monitoring
Smoothed per-destination outcome tracking driving concurrency recommendations.
Test infrastructure
Built as first-class crates rather than test fixtures, because reproducing provider behavior on demand is what made iteration possible.
SMTP laboratory
Automated
A test receiver with round-trip-time and slow-server modes, so provider throttling can be reproduced deterministically instead of waited for.
Provider simulator
Automated
Simulates receiving-provider behavior offline, so the scheduler and learning agent can be exercised without touching a real destination.
Concurrency models
Monitoring
Models runnable under a permutation checker, kept deliberately outside the main build so the check is opt-in rather than a build cost.
External
Receiving infrastructure
External service
Destination mail exchangers. Their responses are the only real signal about how hard we may push.
Diagram is sanitized for publication: it shows responsibilities and data flow, not hostnames, addresses or credentials.
The invariant the design rests on
| Component | Written | Updated after | Authoritative for |
|---|---|---|---|
| Segment files | At import | Never | Raw message bytes |
| Index | At import, atomically | Never | What was injected, and where it lives |
| Event log | During delivery | Append-only | What actually happened |
| Scheduler state | At startup | Continuously | Live leasing |
Recovery is then mechanical rather than judgmental: load the index, replay the log, merge, exclude terminal outcomes. The index is never updated after its initial write — the moment it becomes mutable, the event log stops being the single answer to "what happened", and the guarantee is gone.
Where AI is used — and where it is not
AI / LLM responsibilities
- Architecture review — challenging the crate boundaries and the recovery contract before implementation.
- Implementation assistance under an explicit context pack, task card and token budget.
- Incident reasoning — proposing hypotheses when throughput or deferral behavior changed unexpectedly.
- Documentation synthesis from the code, so architecture notes track reality rather than intent.
- Reviewing diffs before push for contract violations a human reviewer skims past.
Deterministic responsibilities
- Every delivery decision. No model is in the send path.
- Scheduler state reconstruction — index load, log replay, merge.
- Message signing and signing-key drift detection.
- DNS resolution and the pre-warm success-ratio gate.
- Concurrency recommendations — smoothed statistics, not inference.
- Queue depth, throughput and all reported counters.
- Retry and deferral timing.
- Test execution and the release gate.
Decisions and tradeoffs
Build a purpose-built MTA instead of tuning the commercial fleet
- Options considered
- Keep tuning the existing fleet · Wrap it in better reconciliation tooling · Build a delivery engine around an injection index and an event log
- Chosen
- Build it, with the commercial fleet kept operational as a fallback path throughout
- Reason
- The recovery guarantee we needed depends on owning the injection-time inventory and the outcome log. That cannot be added from outside a closed system.
- Cost and risk accepted
- A substantial engineering investment, and a long period running two sending paths in parallel.
- Result
- The active sending path is now giMTA; the legacy fleet remains supported for existing campaigns rather than being ripped out.
Rust rather than Go, which the rest of the platform uses
- Options considered
- Go, matching the platform · Rust · Extend an existing open-source MTA
- Chosen
- Rust on the tokio runtime
- Reason
- This component is a long-lived process holding tens of thousands of sockets with strict memory behavior, and it is the piece where a data race becomes a duplicate send.
- Cost and risk accepted
- A second language in the stack, a smaller pool of people who can work on it, and slower early iteration.
- Result
- Accepted deliberately. The boundary is narrow — one process with an HTTP and file interface — so the split does not leak across the platform.
Append-only event log as the source of truth after injection
- Options considered
- Mutable per-message status in a database · Status held in the index file · Append-only log replayed at startup
- Chosen
- Append-only log, with the index immutable after write
- Reason
- A crash mid-update leaves mutable state ambiguous. An append-only log is either missing an entry or has it — and a missing entry means the outcome never happened.
- Cost and risk accepted
- Replay cost at startup that grows with campaign size, and log files to retain and eventually prune.
- Result
- Interrupted sends resume mechanically. The 'do we re-send?' decision no longer exists.
One campaign delivers from exactly one lane — a hard rule, not a setting
- Options considered
- Allow fan-out across lanes for throughput · Allow it behind a flag · Forbid it entirely
- Chosen
- Forbid it, and document why next to the rule
- Reason
- Fanning a campaign across sending identities is the fastest available throughput win and one of the most reliable ways to damage domain reputation. Left as a tunable, it will eventually be tuned during an incident.
- Cost and risk accepted
- A ceiling on single-campaign throughput. Capacity has to come from engines within the lane instead.
- Result
- Capacity and identity became independent concepts, which is what made the ceiling acceptable.
Abort the start when DNS pre-warm falls below a floor
- Options considered
- Warn and continue · Retry silently · Refuse to start
- Chosen
- Refuse to start
- Reason
- Sending into an unresolved routing picture produces deferrals that look like throttling. That misdiagnosis is expensive, and it happens hours later.
- Cost and risk accepted
- A campaign can fail to start for a reason that is not the campaign's fault, and an operator has to intervene.
- Result
- A class of confusing deferral incidents disappeared, replaced by an immediate, legible failure.
Challenges
Signing-key drift was found the hard way. Messages signed at import carry a signature tied to a specific key. If the key changes before delivery, every signature becomes unverifiable — and nothing fails loudly. Delivery continues, reputation degrades, and the cause is several steps removed from the symptom. The fix was to record the key fingerprint at import and compare it at start, refusing to run on a mismatch. This is now a check that runs before every campaign start.
Recovery was easy to get subtly wrong. The first version of the replay logic correctly excluded accepted and failed messages, but deferred messages with a future retry time were being reinstated without their timer state. Nothing crashed. The engine simply retried them immediately, which is exactly the behavior a receiving provider reads as abuse. Caught by a crash-recovery test that asserted timer state rather than just message counts.
The rate-zero trap. Discovering that a rate directive of 0 means "no
throughput" rather than "no limit" cost a campaign's sending window. It is now
written down as an invariant with the evidence next to it, because the next
person to read that config will make the same reasonable assumption.
Two languages is a real cost, not a footnote. Debugging across the Go platform and the Rust engine means holding two mental models and two toolchains. It is worth it for this component, and I would not repeat the split for anything less critical.
Running two sending paths in parallel took longer than planned. Keeping the legacy fleet operational while the new engine matured was correct, but it doubled the surface area for operational mistakes for months. The alternative — a hard cutover — was not something I was willing to do to live campaigns.
Testing and validation
| Layer | What it covers | Evidence |
|---|---|---|
| Acceptance criteria | Recovery, pre-warm gate and signing-drift behavior written as testable statements before implementation. | Requirements written first; each maps to a named test. |
| Unit | Scheduler transitions, timer-wheel behavior, index encode and decode, signing. | Per-crate test suites run on every change. |
| Integration | Import to delivery against a local SMTP laboratory, including deferral and permanent-failure paths. | Dedicated SMTP lab and simulator crates in the workspace. |
| Deterministic throttle reproduction | Provider throttling and slow-server behavior reproduced on demand rather than observed in production. | Round-trip-time and slow-server modes in the SMTP laboratory crate. |
| Memory safety | Unsafe code forbidden across the entire workspace, with no exceptions granted. | Workspace-level lint, zero unsafe blocks in the tree. |
| Concurrency verification | Interleaving checks over the concurrent data structures under a permutation model checker. | A dedicated models crate, deliberately excluded from the default build so the check stays opt-in. |
| Crash recovery | Kill mid-send, restart, assert no terminal message is re-delivered and deferred timers survive. | The test that caught the deferred-timer defect described above. |
| Canary | A planned partial-volume send compared against the legacy path before full cutover. | Canary planning and execution built into the command line. |
| Production smoke | First campaign on a new lane verified end to end before scheduled volume follows. | First-lane checklist run per lane. |
| Monitoring | Queue depth, throughput, deferral and acceptance rates per destination. | Metrics exported for scraping; live dashboard. |
| Rollback | The legacy fleet stayed operational for the whole migration. | Documented cutover runbook with an explicit reverse path. |
Results
| Measure | Before | After | How it is measured |
|---|---|---|---|
| Interrupted send recovery | An operator decides between duplicate sends and dropped messages, live. | State rebuilt mechanically from index plus event log; terminal messages excluded automatically. | Verified by crash-recovery tests that kill the process mid-send and assert message-level outcomes. |
| Throughput control | Static per-provider limits, tuned by guesswork. | Smoothed per-destination metrics drive a recommended concurrency inside configured ceilings. | Learning agent updates on every delivery outcome. |
| Pre-send routing confidence | Sending began before routing was known; failures surfaced hours later as ambiguous deferrals. | Start is refused when the pre-warm success ratio is below its floor. | Gate evaluated at every start, before the first message. |
| Signature integrity | A key change between import and delivery silently produced unverifiable mail. | Key fingerprint recorded at import and compared at start; mismatch stops the run. | Drift check runs on every campaign start. |
| Sending identity discipline | Fan-out across identities was available as a throughput lever. | One campaign, one lane — enforced, documented, with capacity scaling separately via engines. | Written as an invariant with its rationale, not as a configuration default. |
Deliberately qualitative. I do not have permission to publish campaign volumes or per-provider delivery rates, and I would rather state a capability I can stand behind than a number I cannot show the working for.
Evidence
Fleet across six lanes, 48 engines live — Real fleet scale: six sending identities, all healthy, 48 engines live, each lane running eight. The lane, domain and address columns are blurred as one block — that table is the mapping from lane identity to sending domain and outbound address, which is why lane identities are covered everywhere else on this site. Click to enlarge. A real campaign mid-flight — 40,000 messages — Status sending, 40,000 messages handed to the engine, eight of eight engines running, DNS cache warm at 98.7 percent. Captured before delivery counters populated, so this is a loaded and released campaign rather than a throughput reading — the distinction matters and the numbers say so. Host, domain and address fields render empty here, so no redaction was needed. Click to enlarge. Provider funnel with real outcomes — The only capture with populated delivery outcomes rather than an idle instance — per-provider accepted, deferred and in-flight percentages, with share of volume. Provider names are public mailbox providers, so only the host address needed covering. Click to enlarge. Operations console — The command center, the handoff readiness checklist and the lane reputation checks in one view. Captured idle, so the checklist shows the first two steps still pending — which is the barrier working. Only the DKIM selector needed covering. Click to enlarge. Operational reports — Per-provider delivery funnel, deferral and bounce reason ranking, and throttle hotlist, captured on the idle local instance for comparison with the populated report above. Click to enlarge. Queue and scheduler state — Per-engine lease state, scheduler latency and event-log buffer health. Host address and the entire lane column are covered. Click to enlarge. Configuration control plane — Fleet-wide rollout tooling with a validate-then-preview-then-apply sequence. Host address, config path and the lane row are covered, and the crop stops above the per-lane identity section. Click to enlarge.
Lessons
What worked. Writing the recovery contract as a testable invariant before any code existed. Everything downstream — the immutable index, the append-only log, the replay-on-start sequence — follows from that one sentence. When an implementation question came up, the contract answered it.
What failed. I under-specified deferred-message state in the first recovery design. The tests I had asserted counts, and counts were correct while behavior was wrong. Assert on the state that matters, not on the number that is easy to compare.
What changed. Reliability checks moved from post-incident to pre-send. The pre-warm gate and the signing-drift check both exist because the failure they prevent was previously discovered hours later, from the wrong symptom.
What I would do differently. I would build the SMTP laboratory and simulator first rather than alongside. Once a realistic local receiver existed, iteration speed changed completely, and I had spent weeks before that testing against conditions I could not reproduce on demand.
What this demonstrates about AI-product leadership. The most important decision on this project was where not to put a model. giMTA is full of places where a model would look impressive in a demo — predicting throttling, choosing retry timing, classifying failures. Every one of those is a place where being confidently wrong costs sender reputation, and where a smoothed statistic is both cheaper and more accurate. Knowing that boundary, and being able to defend it, is the job.
Next milestone
- Multi-lane host consolidation. Move from one process per lane to a single host process serving several lanes, with per-lane control-plane routing. The configuration merge and a migration helper exist; the operational cutover is the remaining work.
- Event log compaction and retention. Replay cost grows with campaign size. A compaction step that preserves terminal outcomes while shrinking the log is the next scaling constraint to address.
- Richer per-destination policy. Today the learning agent recommends concurrency. The next step is per-destination retry shaping driven by the same observed signal.