Skip to main content
Liorry Herisnor
Runtime state: Running in productionInfrastructure

Closed-Loop Throughput Tuning

Adjust delivery concurrency from smoothed per-destination outcomes rather than from static configuration — with ceilings a loop can never exceed.

Runtime state: The learning agent updates on every delivery outcome inside the running engine, clamped by configured ceilings.

Business objective
Deliver as fast as receiving infrastructure will actually accept, which is a moving number that no static configuration can express.
Trigger
Every delivery outcome during an active send.
Inputs
  • Per-message outcomes — accepted, deferred, permanently failed
  • Existing smoothed metrics for the destination group
  • Configured concurrency ceilings for the lane
  • Destination classification from filter-sensitive routing
Outputs
  • A recommended concurrency per destination group
  • Applied concurrency, always within ceilings
  • Updated smoothed metrics
  • Operator-visible throughput and deferral behavior

Why static limits fail

A per-provider rate limit in a configuration file is a guess about a number that moves. Receiving infrastructure throttles differently by time of day, by sender reputation, and by how hard you have just pushed it.

Set the limit low and every campaign takes longer than it needs to. Set it high and you buy deferrals, then blocks, which cost far more than the time saved. The limit is only correct at the moment it was written, and it is never revisited.

The workflow

Outcome to applied concurrency

A control loop, not a prediction. Every input is a measured outcome.

  1. External service

    Delivery outcome received

    Accepted, deferred with a retry time, or permanently failed.

  2. Automated software

    Attribute to a destination group

    Grouped by routing target, so behavior is learned per receiver rather than globally.

  3. Monitoring / feedback

    Update smoothed metrics

    Exponentially weighted averages, so recent behavior dominates without a single outcome causing a swing.

  4. Automated software

    Compute recommended concurrency

    Decision point: Acceptance improving and deferrals low — increase, slowly. Deferrals rising — decrease, immediately.

  5. Approval gate

    Clamp to configured ceilings

    Decision point: The recommendation is advisory. Lane ceilings and per-destination caps are hard limits the loop can never exceed.

  6. Automated software

    Apply to the engine

    Worker concurrency and connection pool behavior adjusted for that destination group.

  7. Automated software

    Deliver at the new level

    Producing the next set of outcomes, closing the loop.

  8. Human action

    Operator observes

    Live throughput and per-destination behavior on the dashboard. Ceilings are adjusted by a person, never by the loop.

  • Human action
  • Automated software
  • External service
  • Approval gate
  • Monitoring / feedback

Decision points

  • Increase, hold or decrease? Driven by smoothed acceptance and deferral rates.
  • How fast to recover? Deliberately asymmetric — decrease quickly, increase slowly.
  • Is the recommendation within ceilings? Clamped without exception.
  • Should the ceiling itself change? Human decision, informed by observation.

Responsibilities

AI / LLM

  • Explaining an unexpected throughput pattern to an operator during diagnosis — after the fact, never in the loop.

Deterministic

  • Outcome attribution to destination groups.
  • Smoothed metric calculation.
  • The concurrency recommendation.
  • Ceiling clamping.
  • Applying concurrency to engine workers.
  • Every number shown to the operator.

Human approval points

  1. Setting the ceilings. The loop operates inside them and can never raise them.
  2. Changing the recovery curve or smoothing parameters.
  3. Intervening when the loop stabilizes somewhere unexpected, which is a signal about reputation or content rather than about tuning.

Failure handling

FailureResponse
Recommendation exceeds the ceilingClamped. The ceiling is the contract; the recommendation is advisory.
Deferrals spikeConcurrency reduced immediately. Decrease is fast by design.
Metrics unavailable for a new destinationStart at the conservative default for its class rather than at the lane default.
Loop oscillatesSmoothing parameters are the fix; oscillation means the average is too responsive.
Sustained low throughput at ceilingEscalate to a person — the constraint is reputation or content, not concurrency.
Blocking responseDelivery to that group stops and the incident is escalated. The loop does not retry through a block.

Monitoring

Applied versus recommended concurrency, acceptance and deferral rates per destination group, time at ceiling, backoff frequency, and oscillation amplitude. Time spent at the ceiling is the signal that a human should reconsider the ceiling.

More workflow designs

Back to the library