testing agent email in your GitHub Actions CI/CD pipeline

Your GitHub Actions pipeline runs 47 tests on every push. All green. You merge with confidence and deploy. Then a customer mentions they never got the welcome email.

CI/CD pipelines are thorough about testing business logic and API contracts. But email? Email is treated as a fire-and-forget side effect. You call sendEmail(), assume it works, and move on. That assumption breaks in production more often than anyone likes to admit.

The problem gets worse when agents are involved. An AI agent that provisions its own inbox, sends transactional messages, or processes inbound mail needs its email tested like any other integration. Not manually. Not "I'll check my inbox after deploy." Tested in the pipeline, with assertions, on every commit.

This guide covers how to send email notifications from GitHub Actions when tests fail, how to test your email-sending code inside the pipeline without hitting real inboxes, and how to verify actual delivery using an agent with its own email address.

How to send email notifications from GitHub Actions on test failure#

1. Store your email API credentials (API key or SMTP password) as encrypted secrets in your repository's Settings → Secrets and variables → Actions.
2. Define a workflow triggered on push or pull_request events in your .github/workflows/ directory.
3. Add a test job step that runs your test suite and captures exit codes (e.g., npm test or pytest).
4. Add a conditional notification step using if: failure() so it only executes when a previous step fails.
5. Call your email API with a payload containing the job name, commit SHA, branch, and a test summary.
6. Verify delivery by checking that the notification arrives and isn't classified as spam.

Here's what steps 4 and 5 look like in practice:

- name: Notify on failure
  if: failure()
  run: |
    curl -s -X POST https://api.lobstermail.ai/v1/emails \
      -H "Authorization: Bearer ${{ secrets.LOBSTERMAIL_TOKEN }}" \
      -H "Content-Type: application/json" \
      -d '{
        "from": "ci-bot@yourproject.lobstermail.ai",
        "to": "team@yourcompany.com",
        "subject": "❌ Tests failed on ${{ github.ref_name }}",
        "text": "Commit ${{ github.sha }} broke the build. See: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
      }'

That's the minimal version. Most production pipelines parse test output and include a structured summary in the email body, which is what the next section covers.

Parsing test output into email reports#

Most test runners output JUnit XML or JSON results. You can parse these in a workflow step and format them into something a human actually wants to read.

- name: Parse test results
  if: failure()
  id: parse
  run: |
    FAILED=$(cat test-results.json | jq '[.testResults[].testCases[] | select(.status == "failed")] | length')
    SUMMARY=$(cat test-results.json | jq -r '[.testResults[].testCases[] | select(.status == "failed") | .name] | join(", ")')
    echo "failed_count=$FAILED" >> $GITHUB_OUTPUT
    echo "summary=$SUMMARY" >> $GITHUB_OUTPUT

Reference ${{ steps.parse.outputs.failed_count }} and ${{ steps.parse.outputs.summary }} in your email payload. This beats a generic "tests failed" message that tells the on-call engineer nothing about what actually broke. Include the failed test names, the triggering commit SHA, and a direct link to the Actions run. The goal is for the recipient to understand the failure without opening GitHub at all.

Testing email-sending code without delivering real messages#

Here's where most teams get stuck. Your application sends transactional email (password resets, order confirmations, verification codes), and you want to test that code in CI. But you don't want GitHub Actions firing off real emails to real people on every push.

Three approaches work, each trading off fidelity for speed.

The most thorough option is sandbox inboxes. Provision a test inbox that your CI pipeline sends to, then assert against what arrives. The inbox catches the email without forwarding it to a real person. This is where an agent with its own email address becomes useful: your test suite sends to the agent's inbox, then checks that the right message arrived with the right subject and body. We covered this pattern in our guide on testing agent email without hitting production.

If you need faster feedback, mock the transport layer. Replace your email provider's SDK with a mock during tests. This verifies that your code calls the send function with the correct arguments, but it doesn't test email rendering or the delivery path. Fast, but incomplete.

A middle ground is a local SMTP trap. Tools like MailHog or smtp4dev spin up a local SMTP server that accepts all mail and stores it for inspection. You can run MailHog as a Docker service container inside GitHub Actions:

services:
  mailhog:
    image: mailhog/mailhog
    ports:
      - 1025:1025
      - 8025:8025

Point your app's SMTP config at localhost:1025 and query the MailHog API on port 8025 to verify messages were received. This tests the full sending path without touching real inboxes, though it tells you nothing about spam filtering or deliverability on the receiving end.

Mocking is fastest but tests the least. Sandbox inboxes test the full delivery path, including spam classification, but add a few seconds per test. Pick based on what's actually failing in production: if emails disappear silently, you need the full-path test.

Email vs. Slack vs. webhooks for CI notifications#

Slack is the default notification channel for most teams. It's instant, it's where everyone already is, and setup takes five minutes. But email has properties that matter once you operate more than a couple of services.

Email creates a durable, searchable audit trail that exists outside your team's chat tool. When a compliance review asks "were stakeholders notified of the deployment failure on March 12th?", an email thread is evidence. A Slack message buried in a channel with 200 other messages is harder to surface and easier to dispute.

Email also handles routing better. A failure notification for the payments service should reach the payments team, not a noisy #ci-alerts channel that everyone muted three months ago. With email, you send to a distribution list or a specific person. With Slack, you're either @mentioning (and annoying) or posting to a channel (and being ignored).

For a comparison of real-time delivery approaches for agent inboxes, see the breakdown of webhooks vs polling: how your agent should receive emails. Many of the same tradeoffs apply when your CI pipeline needs to confirm whether a notification was delivered.

That said, email isn't instant the way Slack is. If you need a human to react within seconds, Slack or PagerDuty wins. The practical setup uses both: Slack for real-time alerts, email for structured reports and audit trails.

Avoiding the spam folder with CI email notifications#

Automated emails from CI runners hit spam filters more often than you'd expect. The sending IP has no reputation, the "from" address was just created, the content looks templated, and the volume spikes unpredictably. Four things help:

• Set up SPF, DKIM, and DMARC records for your sending domain. If you're using LobsterMail, authentication is configured automatically for @lobstermail.ai addresses. For custom domains, the setup guide walks you through the DNS records.
• Send from a consistent address. Don't generate a new "from" per workflow run. Something stable like ci@yourproject.lobstermail.ai lets receiving servers build reputation over time.
• Keep volume predictable. If you're running matrix builds that each fire off notification emails, you can hit rate limits fast. Aggregate results into a single summary email per workflow run instead of one per matrix cell.
• Avoid all-caps subjects, excessive links, and missing plain-text alternatives. Keep notification emails simple, short, and text-heavy. Spam classifiers are trained on exactly the kind of templated, link-stuffed content that automated systems love to produce.

Running workflows locally before pushing#

You can test GitHub Actions workflows on your machine using nektos/act. It runs workflows in Docker containers locally, so you can iterate on your email notification step without pushing 15 commits to trigger the real pipeline.

act -j test --secret-file .env

This won't replicate GitHub's runner environment perfectly (especially around secrets injection and service containers), but it catches YAML syntax errors and logic bugs before they waste CI minutes. Worth the five-minute install.

Where an agent fits in all of this#

The common thread is that email in CI/CD works best when something is actively managing inboxes, parsing inbound messages, and verifying delivery. That's exactly what agents do well. An agent that provisions its own inbox, receives test emails, and asserts on their content turns email from a fire-and-forget side effect into a testable, observable part of your pipeline.

If you want your agent to handle email in your CI/CD workflow, set up a LobsterMail inbox in about 30 seconds. The agent provisions its own address and starts receiving immediately, no DNS configuration required.