circuit breaker pattern for agent email services: keeping your pipeline alive

Your agent sends a password reset email. The email service times out. Your agent retries. Times out again. Retries harder. Now your agent is stuck in a loop, burning through rate limits, backing up its task queue, and ignoring everything else it should be doing.

This is exactly the kind of failure the circuit breaker pattern was designed to prevent. Borrowed from electrical engineering (where a tripped breaker stops current from frying your wiring), the software version does the same thing for service calls. When a downstream dependency starts failing, the circuit breaker cuts it off before the failures cascade through your entire system.

For agents that depend on email (sending verification codes, dispatching notifications, handling outbound workflows), circuit breaker implementation isn't optional. It's the difference between a momentary hiccup and a full pipeline meltdown.

How the circuit breaker pattern works for agent email services#

The pattern operates through three states:

1. Closed (normal operation): Your agent sends emails as usual. The circuit breaker counts failures silently. Everything works.
2. Open (service cut off): Failures cross a threshold. The breaker trips. All email requests are immediately rejected without contacting the email service, and your agent falls back to an alternative behavior.
3. Half-open (testing recovery): After a cooldown period, the breaker allows one test request through. If it succeeds, the breaker resets to Closed. If it fails, the breaker returns to Open and the timer restarts.

This three-state cycle means your agent never hammers a dead service. It fails fast, preserves resources, and automatically recovers when the email provider comes back online.

Why agents need this more than traditional services#

A human-operated application can show an error message. "Email failed, try again later." The user grumbles, waits, clicks retry. Problem contained.

An autonomous agent doesn't grumble. It retries. And retries. And retries. Without a circuit breaker, an agent encountering email delivery failures will typically:

• Exhaust its retry budget in seconds
• Queue hundreds of duplicate send attempts
• Consume API rate limits that affect other agents sharing the same account
• Block its own execution loop waiting for timeouts to resolve
• Miss time-sensitive tasks because it's stuck on a failed email call

I've seen agents burn through an entire day's send quota in under a minute because a provider had a 30-second outage. The provider recovered. The agent's reputation didn't.

The circuit breaker pattern solves this by making failure a discrete, handled state rather than something the agent fights against.

Setting failure thresholds for email services#

Generic circuit breaker tutorials tell you to trip after "5 failures in 60 seconds." That's fine for HTTP APIs. Email services are different.

Email failures come in two flavors. Transient failures (timeouts, 429 rate limits, 503 service unavailable) are candidates for circuit breaker protection. Permanent failures (550 rejections, invalid addresses, authentication errors) are not. A 550 will never succeed on retry, so counting it toward your circuit breaker threshold creates false trips.

Here's what I'd recommend for an agent email pipeline:

• Count only transient failures (timeouts, 5xx responses, connection errors)
• Threshold: 3 consecutive transient failures, or 5 within a 30-second window
• Open duration: 30 seconds for initial trip, doubling on each consecutive trip (30s, 60s, 120s), capped at 5 minutes
• Half-open probe: Send a single low-priority test email, not a queued production message

The doubling backoff matters. If a provider is having a real outage (not just a blip), you don't want your circuit breaker testing recovery every 30 seconds for an hour. Let the intervals grow.

For agents using LobsterMail, the SDK already handles basic retry logic with exponential backoff on transient errors. A circuit breaker sits one layer above that. If the SDK's own retries are exhausted and the call still fails, that's when your circuit breaker should count a failure.

Fallback strategies when the circuit is open#

The breaker tripped. Your agent can't send email right now. What should it do?

This depends on what the email is for:

Time-sensitive transactional emails (OTP codes, verification links, password resets): Queue to a dead-letter queue with a TTL. If the circuit closes within the TTL, send immediately. If not, log the failure and notify the orchestrating agent or a monitoring channel. Don't silently drop these.

Non-urgent notifications (welcome emails, status updates, summaries): Queue locally and retry when the circuit closes. These can wait minutes or even hours without consequence.

Outbound sequences (drip campaigns, follow-ups): Pause the sequence entirely. Sending half a sequence is worse than delaying the whole thing.

The key insight: your agent should know which category each email falls into before it tries to send. Tag your emails with a priority level at composition time, and let the fallback logic route accordingly.

If you're running webhooks to receive inbound email while your outbound circuit is open, inbound delivery is unaffected. The circuit breaker only gates outbound calls. We covered the tradeoffs of different inbound approaches in webhooks vs polling: how your agent should receive emails.

Circuit breaker vs. retry: they're not the same thing#

A common confusion. Retries and circuit breakers solve different problems.

Retries handle individual request failures. "This one call failed, try it again." Good for transient blips. Retries are optimistic: they assume the next attempt will probably work.

Circuit breakers handle systemic failures. "This service is down, stop trying." Good for outages. Circuit breakers are pessimistic: they assume the next attempt will probably fail too, so don't waste the resources.

You need both. Retries handle the 0.1% of requests that fail randomly. The circuit breaker catches the scenario where retries themselves are failing repeatedly, and stops the bleeding.

In practice, the stack looks like this: your agent calls the email SDK, the SDK retries transient failures 2-3 times with backoff, and if it still fails, your circuit breaker implementation records the failure. Three of those recorded failures trip the breaker.

Multi-agent coordination#

What happens when five agents share the same email service, and they each have their own circuit breaker? Agent A's breaker trips. Agents B through E keep hammering the failing service until their breakers trip too. That's four agents' worth of unnecessary failed requests.

The better approach: share circuit breaker state across agents. This doesn't require anything fancy. A shared Redis key, a small SQLite table, or even a file on disk works fine. When one agent detects the service is down, all agents stop calling it immediately.

The tradeoff is coordination overhead. If your agents run on separate machines, sharing state adds a network dependency. For self-hosted setups, a local shared store (SQLite or an in-memory cache) keeps it simple. For distributed deployments, Redis is the standard choice.

Watch out for the thundering herd problem on recovery. When the circuit closes, don't let all five agents send their queued emails simultaneously. Stagger the drain: each agent should add a small random delay before flushing its queue.

Monitoring circuit breaker state#

A circuit breaker that trips silently is worse than no circuit breaker at all. You need to know when it trips, how often, and how long it stays open.

Track these metrics:

• Trip count per hour/day. A breaker that trips once a week is protecting you from rare outages. A breaker tripping five times a day means your email provider has a reliability problem.
• Time spent in open state. This is your email downtime. If you're in open state for 15 minutes a day, that's 15 minutes of queued or dropped emails.
• Half-open success rate. If your probe requests are failing frequently during half-open, your cooldown period is too short. Increase it.
• Queue depth during open state. If your dead-letter queue or local queue grows faster than you can drain it on recovery, you have a capacity problem.

Log every state transition (Closed → Open, Open → Half-Open, Half-Open → Closed or Open) with a timestamp, the failure count that triggered it, and the last error message. When something goes wrong at 3 AM, these logs are what you'll actually read.

A practical implementation sketch#

If you're building this in Node.js with LobsterMail's SDK, the structure looks roughly like this:

import { LobsterMail } from '@lobsterkit/lobstermail';

const breaker = {
  state: 'closed' as 'closed' | 'open' | 'half-open',
  failures: 0,
  threshold: 3,
  cooldown: 30_000,
  lastFailure: 0,
};

async function sendWithBreaker(inbox: any, to: string, subject: string, body: string) {
  if (breaker.state === 'open') {
    if (Date.now() - breaker.lastFailure > breaker.cooldown) {
      breaker.state = 'half-open';
    } else {
      throw new Error('Circuit open: email service unavailable');
    }
  }

  try {
    const result = await inbox.send({ to, subject, body });
    breaker.state = 'closed';
    breaker.failures = 0;
    return result;
  } catch (err: any) {
    if (isTransient(err)) {
      breaker.failures++;
      breaker.lastFailure = Date.now();
      if (breaker.failures >= breaker.threshold) {
        breaker.state = 'open';
        console.log(`Circuit breaker tripped at ${new Date().toISOString()}`);
      }
    }
    throw err;
  }
}

function isTransient(err: any): boolean {
  const code = err?.statusCode || err?.code;
  return code === 429 || code === 503 || code === 'ETIMEDOUT' || code === 'ECONNREFUSED';
}

This is intentionally minimal. Production implementations should add the doubling backoff, shared state, queue management, and monitoring hooks. Libraries like opossum (Node.js) or pybreaker (Python) give you all of that out of the box.

Where to start#

If your agent sends fewer than 50 emails a day, you probably don't need a circuit breaker yet. A simple retry with exponential backoff covers most failure scenarios at that scale.

Once you're past that, or if your agent handles time-sensitive transactional email, add a breaker. Start with the basic three-state implementation, set conservative thresholds (3 failures, 30-second cooldown), and tune from there based on your monitoring data.

The goal isn't zero email failures. That's impossible. The goal is making sure a failing email service doesn't take down everything else your agent is doing.