The problem
Email marketing platforms fail in a specific and expensive way: they work fine until they quietly stop being correct.
Suppression is the highest-stakes rule in the system and the easiest to bypass. If someone has unsubscribed or hard-bounced, sending to them again is a compliance problem, a reputation problem, and a trust problem. But suppression is a check, and checks live in code paths. The moment a second send path exists — a retry, a resend, an automation, a workflow trigger — there is a new place where that check can be forgotten. Not maliciously. Just by someone shipping a feature who did not know the rule existed.
Delivery problems escalate faster than humans notice. A campaign that starts generating deferrals or complaints does not wait for someone to check a dashboard. By the time a person reacts, the damage is already priced into sender reputation for weeks.
Deploying and changing behavior were the same event. A release that changed how sending worked could only be evaluated in production, at full volume, with rollback as the only lever.
Most of the ways this system breaks are silent. Not a stack trace — a counter that stops incrementing, a queue that stops draining, a bounce processor that exits and takes the suppression feed with it. The absence of an error is not evidence that anything is working.
Previous workflow
Before: campaign operations without enforced invariants
Each problem below was survivable alone. Together they meant nobody could state confidently what the system had just done.
- Human action
Operator builds the list and campaign
- Problem: List hygiene depended on who prepared it
- Problem: No single place that said what had already been sent to whom
- Automated software
Send path picks up the campaign
- Problem: Multiple send paths, each responsible for remembering to check suppression
- Problem: A new feature could introduce a fourth path without anyone noticing the omission
- External service
Messages delivered by the MTA fleet
- Problem: Static rate configuration
- Problem: Fleet nodes kept in configuration lockstep by hand
- Automated software
Outcomes recovered from logs
- Problem: Delivery signal lagged the send
- Problem: A stopped processor looked identical to a healthy quiet period
- Human action
Someone notices a problem
Decision point: Pause the campaign, throttle it, or wait and see?
- Problem: Reaction time measured in hours
- Problem: Reputation damage already committed by the time a person acted
- Problem: The decision depended on which person happened to be looking
- Human action
- Automated software
- External service
Goals and non-goals
Goals
- No send path can reach the delivery engine without consulting suppression.
- Suppression is append-only in production — no bulk deletes, no truncation.
- Delivery protection reacts on a fixed cycle without waiting for a human.
- Message state transitions are a closed, documented graph with safe claim semantics under concurrency.
- Behavior changes ship dark and are enabled deliberately, separately from deployment.
- The ways this system fails silently are written down, with a check for each.
Non-goals
- Not a general marketing suite. No CRM, no landing-page builder beyond what tracking requires, no social scheduling.
- Not a self-service signup product. It serves operators who know what a sending domain is.
- Not throughput at the expense of the invariants. Suppression, unsubscribe handling and bounce processing are never traded for speed.
- No AI in the send decision. Stated as a boundary, not an omission.
My role
I own the platform: product, architecture and delivery — with AI assistance applied under a defined workflow rather than ad hoc.
- Product discovery — worked directly with operators to find where the real time and risk sat, which turned out to be recovery and diagnosis rather than authoring.
- Requirements and PRDs — wrote the service-level specifications, including the enrichment service's hard constraints and the delivery-protection rules.
- Invariants — this is the piece I consider most important. I wrote down the rules that must never break, each with a pointer to the code or document that proves it, so a new contributor inherits them instead of rediscovering them.
- Architecture decisions — service boundaries, the choice to run platform services natively rather than in containers, and the split between the platform and the delivery engine.
- Workflow and state design — the message state machine and its claim semantics under concurrent workers.
- Development coordination — decomposing work across a large surface: many binaries and packages.
- Test planning, UAT and defect management — including the end-to-end browser suite and the release checklist.
- Deployment and production verification — cutover sequencing, canary sends and post-deploy checks.
- Operational improvement — turning each incident into either an invariant, a check, or a runbook.
Product solution
An operator composes a campaign, selects an audience, and schedules it. The platform then does the parts that are easy to get wrong.
Preparation resolves the audience, filters it against suppression, renders personalized messages, and produces a bundle for the delivery engine. Suppression is consulted here, in the one path that all sends funnel through.
Handoff imports that bundle into the delivery engine, gated on a readiness check that fails closed — if the queue, cache and readiness state are not complete, the send is refused rather than started early.
Protection runs continuously on a fixed cycle, evaluating a set of delivery health rules and acting on them — throttling or pausing without waiting for a person.
Ingestion pulls per-recipient lifecycle events back from the delivery engine and classifies bounces, feeding the suppression list and the reporting layer.
Automation — workflows, RSS-triggered campaigns and enrichment — sits around that core loop and is explicitly forbidden from bypassing it.
Workflow
After: campaign creation to reported outcome
The two additions that mattered most are the single suppression gate and the fail-closed readiness barrier.
- Human action
Operator composes and schedules
Audience, template, sending identity and schedule.
- AI / LLM
Optional AI drafting assistance
Subject and body drafting support in the editor. Output is always reviewed by the operator before it can be scheduled.
- Automated software
Audience resolution
Segment and list membership resolved to a concrete recipient set.
- Approval gate
Suppression filter
The single mandatory gate. Every send path passes through here.
Decision point: Suppressed recipients are removed before anything downstream sees them. Suppression is append-only in production.
- Automated software
Message generation
Personalized rendering, including the mandatory unsubscribe mechanism on every outbound marketing message.
- Approval gate
Readiness barrier
Decision point: Are queue, cache and readiness state complete? If not, refuse the send. The gate fails closed.
- Automated software
Handoff to the delivery engine
Bundle imported over an authenticated internal interface. One campaign delivers from one sending identity.
- External service
Delivery to receiving providers
- Monitoring / feedback
Delivery protection cycle
Health rules evaluated on a fixed tick.
Decision point: Throttle, pause or continue — decided by rule, not by whoever is watching.
- Automated software
Event ingestion and bounce classification
Per-recipient lifecycle events pulled back and classified.
- Data store
Suppression updated
Hard bounces and complaints append to the suppression list, closing the loop.
- Automated software
Engagement and click tracking
Served by a separate process with a deliberately minimal surface — no dashboard, no authenticated API.
- Human action
Operator reviews reporting
Funnel, placement and engagement for the campaign.
- Human action
- Automated software
- AI / LLM
- Data store
- External service
- Approval gate
- Monitoring / feedback
Architecture
MailerClub — sanitized service view
Platform services run as independent long-lived processes, each with one job. The tracking surface is isolated by process boundary rather than by configuration.
Entry
Reverse proxy
External service
TLS termination and hostname routing.
Application API and web UI
Automated
Campaign management, templates, reporting, administration.
Tracking service
Automated
Click, open, unsubscribe and abuse endpoints only. No authentication surface, no dashboard, no administrative API — enforced by running as a separate process rather than by proxy rules.
Campaign pipeline
Campaign worker
Automated
Scheduling, audience resolution, message generation and handoff to the delivery engine.
Sending agent
Automated
Campaign orchestration, slot allocation and send-time optimization.
Delivery protection
Monitoring
Fourteen rules in four priority bands evaluated on a fixed cycle, able to throttle or pause autonomously.
Workflow runner
Automated
Automation sequences, which are explicitly forbidden from bypassing the suppression gate.
Feedback loop
Event consumer
Automated
Pulls per-recipient lifecycle events from the delivery engine's event log.
Bounce processor
Automated
Classifies bounces and complaints and appends to suppression. Must always be running — its absence is a silent failure mode.
Enrichment service
Automated
Background-only contact enrichment for segmentation. Explicitly must never block sending.
State
Relational store
Data store
Campaigns, recipients, messages, suppression, workflows. Suppression is append-only in production.
Cache and queue layer
Data store
Split between ephemeral working state and durable queue state.
Metrics
Monitoring
Counters and gauges exported for scraping and alerting.
Delivery
giMTA — active path
Automated
Purpose-built Rust delivery engine. See the separate case study.
Commercial MTA — legacy path
External service
Retained for existing campaigns rather than removed. New work targets the active path.
AI and agent surface
Everything here sits beside the core loop. None of it can reach the send decision.
In-app assistant
AI / LLM
Twenty-six registered platform tools. An action it wants to take is written as a pending row and stops there until a human confirms — the state machine is pending, executing, then completed, failed or canceled.
Confirm-before-execute audit
Approval gate
A fleet-wide view of every tool run across all conversations, filterable by state and annotated with the role each tool requires.
Agent registry and guardrails
Automated
A separate service with its own database holding the agent registry, per-agent signing keys that can be minted and revoked, clamped deliverability guardrails and a global kill switch.
Decision explainer
Data store
A decision can be stored with its reasoning tree, the signals used, a confidence breakdown, an input snapshot and the alternatives considered — and two decisions compared signal by signal.
Model credential store
Automated
Three allowlisted providers, keys encrypted at rest with only a last-four fragment retained, plus live credential validation and an A/B view over the routing pool.
Scraper egress guard
Approval gate
Classifies a destination before a proxied request is permitted and blocks egress to financial, payment, checkout and login endpoints. Per-feature kill switches, fully audited.
Diagram is sanitized for publication: it shows responsibilities and data flow, not hostnames, addresses or credentials.
The AI surface, and why it is shaped like this
The platform has a visible AI and agents area, and the design rule across all of it is the same one that governs the send path: a model may propose, and a deterministic gate decides.
The assistant is the clearest example. It has twenty-six real platform tools —
pause a campaign, check suppression, read delivery stats, count a segment. It
cannot call any of them directly. When it decides to act it writes a row in
pending_confirmation and stops. A human confirms, and only then does the action
execute. Every run is auditable across the whole fleet, not just within one
conversation.
The egress guard is the other one worth naming, because it protects against a failure mode that is easy to miss: the platform runs scrapers through a residential proxy pool, and without a guard those scrapers could proxy requests to banks, payment processors, checkout flows and login portals. So a destination is classified before the connection is allowed, and those categories are refused outright.
The invariants
The platform carries a written list of rules that must not break, each with a pointer to the code or document that proves it holds. The ones that shape the architecture most:
- One campaign delivers from one sending identity. Never fanned across several for throughput. This is a deliverability rule, not a performance tradeoff.
- Suppression is append-only in production, and every send path consults it before handoff.
- Message state transitions stay inside a closed graph — queued, leased, delivered, deferred, failed, suppressed — with claim semantics that are safe when several workers compete for the same rows.
- Every outbound marketing message carries an unsubscribe mechanism.
- Bounce and complaint processing must always be running, end to end.
- Feature flags default to off. Always.
Writing these down with their evidence is the single highest-leverage thing I have done on this platform. It converts tribal knowledge into something a new contributor — or an AI assistant — inherits on day one.
Where AI is used — and where it is not
AI / LLM responsibilities
- Campaign copy drafting assistance in the editor, always operator-reviewed before scheduling.
- Summarizing incident context and proposing hypotheses during diagnosis.
- Explaining anomalies in delivery metrics — narrative, not detection.
- Code assistance and pre-push diff review under the engineering workflow.
- Retrieval over internal documentation and runbooks via the knowledge engine.
- Documentation synthesis and drift detection against the code.
Deterministic responsibilities
- Suppression filtering. Every time, no exceptions, no model involved.
- The message state machine and its concurrent claim semantics.
- The readiness barrier, which fails closed.
- All fourteen delivery-protection rules and the eleven action types they can take.
- Bounce and complaint classification.
- Every reported number: queue depth, funnel counts, engagement rates.
- Authentication, authorization and rate limiting.
- Schema validation, database constraints and migrations.
- Test execution and release gating.
Decisions and tradeoffs
One mandatory suppression gate rather than a check in each send path
- Options considered
- A suppression check in every send path · A shared helper each path calls · A single funnel every path must pass through
- Chosen
- A single funnel, with the rule written as an invariant carrying its evidence
- Reason
- A helper still has to be called. A funnel cannot be forgotten by someone who did not know it existed — which is the actual failure mode.
- Cost and risk accepted
- The funnel is a bottleneck in the architecture and a constraint on how new send paths can be built.
- Result
- New send paths inherit compliance instead of re-implementing it.
Run platform services natively rather than in containers
- Options considered
- Containerize everything · Keep containers for local development only · Fully containerized orchestration
- Chosen
- Native long-lived services, containers reserved for the legacy fleet and local development
- Reason
- These are long-lived processes with straightforward dependencies. The orchestration layer was adding operational surface without solving a problem we had.
- Cost and risk accepted
- Less portable, and the environment has to be managed rather than declared. Requires discipline about which host a command belongs to.
- Result
- Fewer moving parts in production. Documented explicitly, because the mixed model is genuinely confusing otherwise.
Feature flags default to off, always
- Options considered
- Ship changes on and revert if wrong · Default on for low-risk changes · Default off without exception
- Chosen
- Default off without exception
- Reason
- 'Low risk' is an assessment made before the incident. A rule with exceptions requires judgment at exactly the moment judgment is worst — during a release.
- Cost and risk accepted
- Over one hundred flags to track, and dead flags that need periodic cleanup.
- Result
- Deploying and changing behavior became separate events with separate rollbacks.
Isolate the tracking surface by process, not by proxy rules
- Options considered
- Proxy-level path allow-list · Application-level host checks · A separate process with only tracking routes
- Chosen
- A separate process exposing click, open, unsubscribe and abuse endpoints and nothing else
- Reason
- Tracking endpoints are the most publicly exposed part of the platform. An allow-list regex is one edit away from exposing the administrative API; a process that does not contain those routes cannot.
- Cost and risk accepted
- Another service to deploy, monitor and keep in step.
- Result
- The exposure boundary became structural rather than configured.
Enrichment is background-only and may never block sending
- Options considered
- Enrich inline during message generation · Enrich on demand with a timeout · Background-only, tolerating absent data
- Chosen
- Background-only, with every consumer required to tolerate missing enrichment
- Reason
- Enrichment depends on external sources. Inline dependency means an external slowdown becomes a sending outage.
- Cost and risk accepted
- Segmentation may run against incomplete data, and consumers carry null-handling everywhere.
- Result
- An external dependency cannot take the send path down.
Challenges
Most failures here are silent. Early on, our operating assumption was that problems announce themselves. They do not. A stopped bounce processor produces no error — it produces a suppression list that stops growing, which looks exactly like a quiet week. The response was a written catalog of how this system actually breaks, with a positive check for each: not "is there an error?" but "is this counter still moving?"
Naming collisions cost real debugging time. Across a large codebase, three different things ended up sharing one name and two more shared another. Someone reading a stack trace could reasonably work on the wrong component for an hour. The fix was a code map that states plainly which tree is the real one — including a substantial subtree that does not run at all and is retained for reference.
Documentation drifts faster than code. Runbooks referenced service names that no longer existed and commands that silently print nothing when run without elevated privileges — which reads as "no logs" rather than "wrong command". We now run a documentation check that fails on exactly the drift that has bitten us before.
Two sending paths for an extended period. Keeping the legacy fleet operational while the new engine matured doubled operational surface for months. It was the right call for live campaigns and it was genuinely painful.
Feature-flag sprawl is a real cost. Over a hundred flags, living in configuration files, environment and the database. Defaulting to off is correct; the cleanup discipline to retire flags after rollout is something I still find easy to postpone.
Testing and validation
| Layer | What it covers | Evidence |
|---|---|---|
| Acceptance criteria | Written per feature before implementation, tied to the invariant it must not break. | Definition-of-done document applied at review. |
| Unit | Store patterns, state transitions, classification logic, rendering. | Package test suites run on every change. |
| Integration | Campaign preparation through handoff against a simulated delivery backend. | Dedicated simulator used in place of live sending. |
| End-to-end | Operator journeys through the browser: authentication, campaign creation, scheduling, reporting. | Browser automation suite in the repository. |
| Invariant checks | Documented rules paired with an evidence file or code reference that must still hold. | Invariants directory, each entry citing checkable evidence. |
| UAT | Operators run their real campaign flow on a release candidate before rollout. | Release checklist with operator sign-off. |
| Production smoke | First campaign after a release verified end to end at low volume before scheduled volume follows. | Post-deploy verification runbook. |
| Monitoring | Queue depth, delivery outcomes, bounce processing liveness, protection-rule firings. | Metrics exported for scraping and alerting. |
| Rollback | Behavior reverts by disabling a flag; code reverts by redeploying the previous build. | Flags default off, so the reverse path is the default state. |
Results
| Measure | Before | After | How it is measured |
|---|---|---|---|
| Suppression enforcement | Each send path responsible for remembering the check. | A single mandatory gate every send path passes through, with suppression append-only in production. | Written as an invariant with a named evidence file; new send paths inherit it. |
| Delivery incident response | A person had to notice, then decide. | Fourteen rules across four priority bands evaluated on a fixed cycle, able to throttle or pause autonomously. | Protection service running continuously against live delivery signal. |
| Release risk | Deploying and changing behavior were one event. | Changes ship dark behind flags that default to off and are enabled deliberately. | Applied without exception across more than one hundred flags. |
| Silent failure detection | Absence of errors treated as evidence of health. | A written catalog of real failure modes, each with a positive liveness check. | Derived from incidents that actually occurred, not from a generic checklist. |
| Onboarding to a large codebase | Component ownership and naming collisions learned by asking, or by debugging the wrong thing. | A code map stating which tree is authoritative, plus invariants with evidence. | Same documents used to orient AI assistants working in the repository. |
Qualitative by choice. Campaign volumes, delivery rates and client identities are not mine to publish, so this table describes capability changes I can substantiate from the system itself.
Evidence
Campaign detail and delivery funnel — The definition line under the counters is the point — it stops injected, sent and delivered being conflated in reporting. Account identifier, campaign name and sending-lane identity covered; the figures are from a four-recipient test campaign. Click to enlarge. MTA operations console — The platform's own view into the delivery engine: command center, handoff readiness checklist and per-engine state. Only the DKIM selector needed covering — every counter is zero and both data panels are empty, so no recipient data appears. Click to enlarge. Send pipeline progress — The pipeline is staged and observable rather than a single opaque job — preflight, generation, attribution and send-time computation each report separately. Account identifier, campaign name and sending-lane identity covered. Click to enlarge. Campaign index — Lifecycle status model and per-campaign drill-down into the delivery engine. Account identifier covered; campaign and template name columns blurred because they are client offer names. Click to enlarge. Status filtering — The same index filtered to a single lifecycle state — how an operator finds the campaign they need among many. Account identifier covered; campaign name column blurred. Click to enlarge. Blocklist reputation policy — Note the stated boundary printed on the page itself: it reports, it never blocks a send. Heaviest redaction of the set — account identifier, the address pair in the banner, all six sending-lane cards and the recipient-domain table. Click to enlarge. Campaign creation — A five-step wizard sequencing recipients, content, sender, delivery and review — audience resolution happens before content, so suppression is applied to a known set. List names and contact counts blurred. Click to enlarge. Delivery protection firing
A protection rule throttling a campaign automatically, with the reason recorded. Not yet captured — reproducing it means waiting for a live degradation.
Placeholder — evidence pending sanitization review before publication.
Lessons
What worked. Writing invariants down with their evidence. It is the cheapest document I maintain and the one that has prevented the most expensive mistakes. It also turned out to be the thing that makes AI assistance safe in this repository — the assistant inherits the rules instead of inferring them.
What failed. I assumed for too long that failures would be loud. They were not, and the gap between a component stopping and anyone noticing was measured in days. Monitoring absence of progress, not presence of errors, was the correction.
What changed. Incidents stopped producing fixes and started producing artifacts: an invariant, a liveness check, or a runbook. A fix helps once; an invariant helps every contributor afterwards.
What I would do differently. I would have built the code map on day one. The cost of naming collisions and an unused legacy tree was paid repeatedly in debugging time before I wrote two pages that would have prevented all of it.
What this demonstrates about AI-product leadership. This platform is full of AI-shaped opportunities — predicting deliverability, classifying bounces, choosing send times. I put a model behind none of them. The value I added was deciding where the deterministic boundary sits and enforcing it structurally, so the AI that is used — drafting, diagnosis, retrieval, code review — operates where being wrong is cheap and recoverable.
Next milestone
- Retire the legacy sending path once remaining campaigns have migrated, removing an entire class of configuration traps and the dual-path operational burden.
- Feature-flag lifecycle discipline — an expiry and review process so flags are retired after rollout instead of accumulating.
- Invariant checks in continuous integration. Today invariants are documented with evidence and enforced at review. The goal is a mechanical check for each one that fails a build rather than relying on a reviewer noticing.