Using AI to Improve Product Development

Using AI to write code is easy. Using it to deliver production systems responsibly, repeatedly, across several large codebases, is a different problem with a different failure mode.

The assistant does not know what the system is. Given a large repository it infers structure from names and directory layout. In our codebases that inference is actively wrong: several components share names, and a substantial subtree does not run at all. The result is not gibberish — it is a confident, well-argued change to the wrong component.

Context is expensive and mostly wasted. Letting an assistant explore freely burns a large token budget rediscovering things that are already written down, then runs out of room for the actual task.

Every task gets the same model. Extracting a field from a struct and designing a recovery contract are not the same problem, but they were routed to the same model at the same cost. Overpaying on trivial work and underpowering hard work, simultaneously.

Review quality is inconsistent when I am the only reviewer. Reviewing my own work — including work an assistant produced under my direction — misses the same class of thing every time, because it is my blind spot doing the reviewing.

AI in the commit path is slow and non-deterministic. Calling a model on every commit makes commits slow, expensive and variable. The same commit can pass and fail on different days, which destroys trust in the gate.

The dangerous failure is a confident wrong change. Not a syntax error — those are caught immediately. A plausible change that violates an invariant nobody wrote down.

Goals

1. An assistant starts from written architectural truth rather than inference.
2. Context is assembled deliberately, not discovered expensively.
3. Model selection matches the task class, in capability and in cost.
4. Deterministic checks run on every commit with no model involved.
5. AI review adds a second perspective before push, where it is cheap.
6. No production mutation happens without human approval.
7. Diagnosis is read-only until an action is explicitly approved.

Non-goals

• No AI in the commit gate. Deliberate. Gates must be fast and deterministic.
• No autonomous deployment. Ever.
• Not a replacement for understanding the system. The workflow makes me faster at work I could do; it does not let me ship what I do not understand.
• Not model-vendor-specific. The routing tiers are roles, not product names.

I wrote this workflow and I am its primary user. It is the reason the other case studies in this portfolio exist at the scale they do.

• Authored the repository operating manuals — the architectural truth, the invariants with their evidence, the commands, and the definition of done.
• Designed the context strategy — context packs, generated repository maps and task cards, replacing free exploration.
• Set the model routing policy — which class of task gets which tier, and which tasks get no model at all.
• Built the deterministic gate set — formatting, linting, type checking, secret scanning and content-schema validation, all without an LLM call.
• Defined the approval boundary — read-only diagnosis, human approval for mutation, audit trail throughout.
• Ran it across every project here — platform, delivery engine, data pipelines, agent governance and this portfolio site.
• Revised it from failures — most rules here exist because something went wrong without them.

The operating manual is the foundation. Each repository carries a document that states what the system is, which architectural claims are true today, the invariants that must not break with a pointer to the evidence for each, the build and validation commands, and the definition of done. An assistant reads this first, every time. It is also what a new human contributor reads first — the two audiences want the same document.

Context packs replace exploration. Rather than letting an assistant search a large repository, the relevant material is assembled: the operating manual, a generated map of the relevant subtree, the specific files in scope, and the task card. Exploration beyond that is limited and deliberate.

Task cards make the unit small. A task card carries the goal, the acceptance criteria, the files in scope, the invariants that apply, and the sensitivity level of the data involved. Small enough to fit in context with room to think.

Model routing by task class. Deterministic facts get no model — a query, a grep or a script answers better and for free. Extraction and summarization go to an inexpensive tier. Architecture and security review go to a strong tier. Production incidents get the best available reasoning, because that is where being wrong is most expensive. Fallbacks exist for provider trouble, and spend is tracked.

Deterministic gates at commit. Formatting, linting, type checking, secret scanning and content-schema validation run on every commit. Fast, repeatable, no model, no network.

AI review before push. Once the deterministic gates pass, the diff goes to a model for review against the repository's invariants. This is where a second perspective is cheap and I am at my weakest — reviewing my own direction.

Human approval for anything that mutates production. Diagnosis is read-only. An action requires a person. Enforced structurally by the agent governance layer for automated agents, and by process for interactive work.

Confident wrong changes were the real danger. An assistant produced a well-reasoned change to a component that does not run in production, because the directory looked authoritative. It passed review because the reasoning was sound. The code map exists because of that, and it is the artifact I would write first in any new repository.

Documentation drift makes the whole method degrade. The workflow rests on written truth. When the writing goes stale, assistants confidently reproduce stale architecture, and the confidence is unchanged. A documentation check that fails on the specific drift that has bitten us before was the answer — including commands that silently print nothing when run without elevated privileges, which reads as "no logs" rather than "wrong command".

Task cards were too large at first. Multi-part cards produced changes that were partly right, and reviewing a partly-right large change is harder than reviewing two small ones. Smaller cards with tighter acceptance criteria fixed it, at the cost of more decomposition work up front.

Model routing needs actual cost visibility. The initial routing rules were my guesses about which tier suited which task. Without tracked spend they stayed guesses. Tracking cost per task class turned the policy into something evidence can adjust.

I am the bottleneck, by design. Human approval on every production change means everything queues behind me. I have not solved this and I am not trying very hard to, because the alternative is relaxing the rule I trust most.

The 50% figure is self-reported. It comes from my own observation of requirement-to-verified-change time across these projects, not from an instrumented measurement system. It is labeled that way wherever it appears.

• Repository operating manual

  Architectural truth, invariants with evidence, commands and definition of done.

• Invariants document

  Rules that must not break, each with a pointer to its proof.

• Deterministic commit gate output

  Format, lint, type check, secret scan and schema validation on a real commit.

• Model routing policy

  Task classes mapped to tiers, including the no-model tier.

What worked. Writing invariants with evidence. It is the cheapest artifact in the workflow and it does the most work — it onboards humans, grounds assistants, and gives the pre-push review something concrete to check against.

What failed. My first instinct was to put AI review in the commit gate, on the theory that earlier is better. It made commits slow and non-deterministic, and a gate that gives different answers on different days stops being a gate — people start working around it. Fast and deterministic beats thorough, at that position in the pipeline.

What changed. The no-model tier. Recognizing that a large share of what I was asking a model to do was a factual question a query answers exactly — faster, free and without the possibility of being wrong — changed both cost and accuracy.

What I would do differently. I would write the code map before the first task, not after the first confident wrong change. Every repository has a version of "which tree is real", and it is two pages that prevent hours of debugging the wrong component.

What this demonstrates about AI-product leadership. This workflow is an argument that the value of AI in delivery comes from the structure around it, not from the model. Written truth, assembled context, routing by task class, deterministic gates and human approval at the boundary. The model is replaceable; the structure is what makes the output trustworthy — and being able to say precisely where a model is not allowed to operate is what makes it defensible to anyone responsible for the system.

• Mechanical invariant checks. Invariants are documented with evidence and enforced at review. Turning each into a check that fails a build removes the dependence on a reviewer noticing.
• Instrumented cycle-time measurement, so the improvement figure comes from data rather than from my own observation.
• Routing policy driven by tracked spend, replacing the initial guesses with evidence about which tier actually suits which task class.