The business problem
A marketing operation was running campaigns on a stack that worked, right up until it quietly stopped being correct.
The compliance rule that mattered most was the easiest one to bypass. If someone has unsubscribed or hard-bounced, sending to them again is a compliance problem, a reputation problem and a trust problem. Suppression was a check, and checks live in code paths. Every additional send path — a retry, a resend, an automation — was a new place for that check to be forgotten by someone who did not know the rule existed.
Delivery problems escalated faster than people noticed. A campaign that starts generating deferrals or complaints does not wait for someone to open a dashboard. Reaction time was measured in hours, and reputation damage was already committed by the time a person acted.
An interrupted send had no good answer. With no authoritative per-message record, resuming meant choosing between re-sending messages that had already been accepted or dropping messages that had not. One damages sender reputation, the other silently loses revenue, and the choice was being made live, under time pressure.
Manual work sat in the middle of everything. List preparation depended on who prepared it. Prospect research was a person with a browser and a spreadsheet whose provenance lived in their memory. Configuration across the sending fleet was kept in step by hand.
Deploying and changing behavior were the same event. A release that changed how sending worked could only be evaluated in production, at full volume, with rollback as the only lever.
And most of the failures were silent. Not a stack trace — a counter that stops incrementing, a queue that stops draining, a bounce processor that exits and takes the suppression feed with it. The absence of an error is not evidence that anything is working.
Product vision and scope
The vision was narrow on purpose: make the rules that must never break impossible to bypass, and make everything else observable.
In scope
- One enforced path from campaign to delivered mail, with compliance applied inside it rather than alongside it.
- Delivery health handled by rules on a fixed cycle instead of by whoever happened to be watching.
- Recovery that is mechanical rather than judgmental.
- Releases where deploying and changing behavior are separate decisions.
- Data work — discovery, enrichment, segmentation — kept structurally unable to block or bypass sending.
Explicitly not in scope
- Not a general marketing suite. No CRM, no social scheduling, no page builder beyond what tracking requires.
- Not a self-service product. It serves operators who know what a sending domain is.
- Not throughput at the expense of the rules. Suppression, unsubscribe handling and bounce processing are never traded for speed.
- No model in the send decision. Stated as a boundary, not left as an omission.
My role
I set the product vision, wrote the requirements, made the architecture decisions, coordinated delivery, managed testing and planned the releases, and I run the system in production. There is no permanent engineering team — delivery is contractors for defined scopes plus AI assistance under a documented workflow.
The three decisions I would point to as mine:
- Writing the invariants down with their evidence. The rules that must never break, each with a pointer to the code or document that proves it holds. It is the cheapest document I maintain and it has prevented the most expensive mistakes — and it is what makes AI assistance safe in these repositories, because the assistant inherits the rules instead of inferring them.
- Refusing a hard cutover. The legacy sending path stayed operational throughout the migration, as the documented reverse path. It doubled operational surface for months. It was still correct: live campaigns were depending on it.
- Keeping models out of the send decision. This platform is full of AI-shaped opportunities — predicting throttling, classifying failures, choosing send times. There is no model behind any of them, and that was a decision rather than an oversight.
Discovery and requirements
Discovery was time with the operators, walking the process step by step. The finding that reshaped the plan: the expensive part was not authoring campaigns, it was recovery and diagnosis — the time spent working out what the system had just done.
The requirements that followed were written as statements that could be tested, not as qualities to aspire to:
| Requirement | How it is verified |
|---|---|
| No send path reaches the delivery engine without consulting suppression | A single funnel; new paths cannot construct a handoff without passing through it |
| Suppression is append-only in production | Bulk deletion rejected in production |
| Delivery protection reacts on a fixed cycle without human input | Rules evaluated continuously against live delivery signal |
| Message state transitions form a closed, documented graph, safe under concurrent workers | Claim semantics tested with competing workers |
| A send interrupted at any point resumes without re-delivering a terminal message | Crash-recovery tests kill the process mid-send and assert message-level outcomes |
| Behavior changes ship dark and are enabled deliberately | Flags default off, without exception |
| Every way this system fails silently is written down, with a check for each | A failure-mode catalog, each entry paired with a positive liveness check |
The last one is the requirement I would write first if I did this again. It came out of an incident rather than out of planning.
What was built
Three things, delivered in sequence.
A campaign platform that resolves an audience, filters it against suppression, renders personalized messages and hands a prepared bundle to the delivery engine — gated on a readiness check that fails closed. If the queue, cache and readiness state are not complete, the send is refused rather than started early.
A purpose-built delivery engine that replaced a commercial MTA fleet as the active sending path. It records what was injected in an index written once and never modified, and records every outcome in an append-only log. Recovery is then mechanical: load the index, replay the log, exclude anything already terminal.
A feedback loop that pulls per-recipient outcomes back, classifies bounces and complaints, appends them to suppression, and feeds reporting where injected, sent and delivered are defined separately so they cannot be conflated.
Around that core sit the automation surfaces — workflows, enrichment, prospect discovery — each of which is explicitly forbidden from bypassing the core loop.
Operational controls
The controls are the product. Each one exists because of something that happened.
| Control | What it prevents | How it fails |
|---|---|---|
| Single suppression gate | Sending to someone who has unsubscribed or hard-bounced, via a path that forgot to check | Cannot be bypassed: there is one funnel, not a helper to call |
| Append-only suppression | A bulk delete quietly re-permissioning contacts | Deletion rejected in production |
| Readiness barrier | A send starting before its preconditions are complete | Fails closed — refuses, rather than warning and proceeding |
| Routing pre-warm floor | Sending into an unresolved routing picture, producing deferrals that look like throttling hours later | Refuses to start |
| Signing-key drift check | Unverifiable mail after a key change between preparation and delivery | Stops the run on a fingerprint mismatch |
| One campaign, one sending identity | Buying throughput by spreading a campaign across sending identities and damaging domain reputation | Not configurable — there is no flag for it |
| Delivery protection rules | A degrading campaign continuing until a human notices | Throttles or pauses autonomously |
| Flags default off | A deploy silently changing behavior | The reverse path is the default state |
Two things are worth saying about this table. First, every one of these fails closed — refusing is the safe direction, and a control that warns and continues is not a control. Second, none of them depends on a person remembering a rule during an incident, which is exactly when rules with exceptions get their exception.
Where AI is used — and where it is not
AI — interpretation, acceleration, recommendation
- Campaign copy drafting in the editor, always operator-reviewed before scheduling.
- Client intake acknowledgment and qualification, with a human owning every commitment.
- Incident diagnosis — summarizing context and proposing hypotheses, read-only.
- Implementation assistance and pre-push diff review under the engineering workflow.
- Retrieval over internal documentation, so answers come from what the system is rather than what a model infers.
Deterministic — permission, enforcement, irreversible action
- Suppression filtering. Every time, no exceptions.
- Whether a message is sent, to whom, and how fast.
- The readiness barrier and the routing floor, both fail-closed.
- Bounce and complaint classification.
- Every reported number.
- Authentication, authorization and rate limiting.
- Release gating and test execution.
Delivery approach, testing and release
The program ran as parallel workstreams with one person holding the dependencies — which only works if the seams are written down. The workstream map, dependency map and RAID log are published as artifacts.
Requirement to verified production change
The gates are the point. Two of them are human, two are mechanical, and none of them can be skipped by someone in a hurry.
- Human action
Requirement written
Problem, goal, explicit non-goals, acceptance criteria, and the invariant it must not break.
- AI / LLM
AI-assisted implementation
Against a context pack and a task card, under an explicit budget.
- Approval gate
Human review
Decision point: A person reviews every change. No model applies a production change.
- Automated software
Deterministic commit gates
Formatting, linting, type checking, secret scanning and schema validation. No model in the loop — a gate has to give the same answer every time.
- Automated software
Automated test suites
Unit, integration against a simulated delivery backend, and end-to-end operator journeys in a browser.
- Human action
Operator UAT
Operators run their real campaign flow on the release candidate.
- Approval gate
Release readiness
Decision point: Defect position, rollback route and post-release verification agreed before the go decision.
- Automated software
Deploy with the flag off
The change is present and inert. Deploying and changing behavior are separate events.
- Human action
Enable deliberately
Flag switched on as its own decision, with the reverse path being the default state.
- Monitoring / feedback
Post-release verification
First campaign verified end to end at low volume before scheduled volume follows.
- Human action
- Automated software
- AI / LLM
- Approval gate
- Monitoring / feedback
Observability, and what healthy means
The hardest lesson on this program: absence of an error is not evidence of health. A stopped bounce processor produces no error. It produces a suppression list that stops growing, which looks exactly like a quiet week.
So monitoring asks a different question — not "is there an error?" but "is this counter still moving?" What is watched:
- Queue depth and whether it is draining.
- Bounce and complaint processing liveness, end to end. Its absence is treated as a failure, not as silence.
- Event ingestion continuing to advance.
- Per-destination acceptance and deferral behavior.
- Protection-rule firings, with the reason recorded.
Each of those came from a documented failure mode rather than from a generic checklist, which is why the list is short.
Outcomes
| Measure | Before | After | How it is measured |
|---|---|---|---|
| Compliance enforcement | Each send path responsible for remembering the suppression check. | One mandatory gate every send path passes through, with suppression append-only in production. | Recorded as an invariant with a named evidence file; new send paths inherit it. |
| Delivery incident response | A person had to notice, then decide. Reaction measured in hours. | Fourteen rules across four priority bands evaluated on a fixed cycle, able to throttle or pause autonomously. | Protection service running continuously against live delivery signal. |
| Interrupted send recovery | An operator chose between duplicate sends and dropped messages, live. | State rebuilt mechanically from an immutable index plus an append-only outcome log. | Verified by crash-recovery tests that kill the process mid-send and assert message-level outcomes. |
| Release risk | Deploying and changing behavior were one event, with rollback as the only lever. | Changes ship dark behind flags that default to off and are enabled as a separate decision. | Applied without exception across more than one hundred flags. |
| Silent failure detection | Absence of errors treated as evidence of health. | A written catalog of real failure modes, each with a positive liveness check. | Derived from incidents that actually occurred. |
| Requirement to production | Bounded by how much one person could implement and verify. | 50% faster, self-reported, under the AI-assisted engineering workflow with deterministic gates. | Self-reported by Liorry Herisnor. The gates are mechanical; the speed-up is in the implementation and review loop. |
Qualitative by choice. Campaign volumes, delivery rates and client identities are not mine to publish, so this table describes capability changes I can substantiate from the system itself. The one percentage figure is self-reported.
What I learned, and what I would improve
What worked. Writing the invariants down with their evidence, and making the controls fail closed. Both convert a rule that depends on memory into a property of the system.
What failed. I assumed for too long that failures would announce themselves. The gap between a component stopping and anyone noticing was measured in days.
What I would do differently. Three things. Build the code map on day one — naming collisions cost debugging time repeatedly before I wrote the two pages that would have prevented all of it. Build the local test receiver first rather than alongside, because iteration speed changed completely once provider behavior could be reproduced on demand. And put flag retirement on a schedule from the start: defaulting to off is correct, and over a hundred flags accumulate faster than they retire.
What I would keep exactly as it is. Refusing the hard cutover. It cost months of doubled operational surface and I would pay it again rather than do that to live campaigns.
Technical deep dives
This page deliberately stops short of the engineering detail. It is all published:
- MailerClub — the platform: service boundaries, the message state machine, the AI and agent surface, the tradeoff table and the test evidence.
- giMTA — the delivery engine: the three-way relationship between spool, index and event log that the recovery guarantee rests on, and why it is written in a different language from everything else.
- Lead discovery and enrichment — the data side, and the review gate that keeps it out of the sending path.
- Agent governance — what an AI agent is permitted to do to a production system.
- Workflow library and platform capabilities — every process design and every capability, each labeled with whether it actually runs.
Next milestone
- Retire the legacy sending path once remaining campaigns have migrated, removing a class of configuration traps and halving operational surface.
- Mechanical invariant checks in continuous integration, so a violated invariant fails a build instead of relying on a reviewer noticing.
- Feature-flag lifecycle discipline — expiry and review, so flags retire after rollout.
- Event-log compaction and retention, which is the next scaling constraint rather than a current problem.