Cabalmail

Host your own email and enhance your privacy

View the Project on GitHub cabalmail/cabal-infra

Container Runtime Hardening Plan

Context

The three mail-tier containers (imap, smtp-in, smtp-out) and the optional monitoring tier (prometheus, alertmanager, grafana, exporters, kuma, ntfy, healthchecks) were lifted off the original Chef/EC2 baseline during the 0.4.x containerisation work. The migration faithfully preserved the behaviour of the EC2 install — same sendmail .mc macros, same Dovecot conf, same supervisord process tree — but did not replace EC2-era assumptions with container-era ones. The result is a stack of amazonlinux:2023-based images running as root with full Linux capabilities, mounting EFS volumes without noexec, signing DKIM for any source IP that can reach the daemon, accepting unbounded message sizes, and shipping a fail2ban that is wired into supervisord but commented out.

None of these have produced an incident. Together they form a meaningful blast-radius surface that we can shrink in deliberate, mostly-additive PRs.

This plan is the container-and-mail-tier counterpart to application-surface-hardening-plan.md (which covers the Lambda Python code) and iac-quality-gates-plan.md (which catches the IaC-flaggable issues). The Trivy IaC scanner from that plan will pick up a subset of these findings once it lands; this plan addresses the architectural and runtime-config issues that scanners would not catch.

Six themes:

  1. Runtime privilege posture. Drop NET_ADMIN, set readOnlyRootFilesystem, noNewPrivileges, and a user override per container.
  2. Image supply chain. Pin amazonlinux:2023 and the third-party Prometheus/Grafana/etc. images to digest; standardise the rebuild cadence; enable ECR scan-on-push.
  3. Sendmail hardening. Add confMAX_MESSAGE_SIZE, confCONNECTION_RATE_THROTTLE, confMAX_DAEMON_CHILDREN, restrictmailq. Drop weak AUTH mechanisms on smtp-in. Atomic config writes.
  4. Dovecot hardening. Flip disable_plaintext_auth = yes. Add login-attempt rate limiting via Dovecot’s own knobs (the only viable replacement for fail2ban in the container-network model).
  5. OpenDKIM scope. Replace TrustedHosts = 0.0.0.0/0 with 127.0.0.1 + the loopback shape sendmail actually uses to hand mail to the milter.
  6. EFS posture. Add transit_encryption = ENABLED to the IMAP mailstore mount; enforce nosuid and (where feasible) noexec on the mailstore filesystem; clean up the fail2ban dead code.

The themes ship as five phases. Phase 4 (Dovecot rate limiting) is the highest-touch piece — it changes auth behaviour. The rest are largely transparent.

Goals

Non-goals

Current state (audit)

Runtime privilege

All three mail tiers add NET_ADMIN (terraform/infra/modules/ecs/task-definitions.tf:63-67, 134-138, 242-246). The capability was added so fail2ban could manipulate iptables. fail2ban is commented out in every tier’s supervisord config: docker/imap/supervisord.conf:37-40, docker/smtp-in/supervisord.conf:38-40, docker/smtp-out/supervisord.conf:51-54. The capability is unused.

No readOnlyRootFilesystem, no linuxParameters.initProcessEnabled, no securityContext.noNewPrivileges. All containers run as root for the duration of supervisord; sendmail and dovecot drop privileges via their own User= directives once spawned, but the container’s process 1 is root, and any process the entrypoint shells out to runs as root by default.

The non-mail tiers fare somewhat better. Prometheus/Alertmanager/Grafana inherit non-root users from their upstream images, but the Dockerfile USER directive is not set explicitly, so a change to upstream image conventions would silently switch them.

Image supply chain

All FROM lines reference floating tags: amazonlinux:2023, prom/prometheus:v3.5.0, grafana/grafana:11.4.0, etc. None are digest-pinned. The CI pipeline (.github/workflows/app.yml) rebuilds on push but does not record provenance; tag re-pulls are silent on the next rebuild.

ECR repositories are created in terraform/infra/modules/ecr/main.tf without image_scanning_configuration { scan_on_push = true } and without image_tag_mutability = "IMMUTABLE". Re-tagging a published image to a different SHA is permitted.

Sendmail config

All three templates set confPRIVACY_FLAGS but omit restrictmailq in two of them (docker/templates/in-sendmail.mc:8, docker/templates/out-sendmail.mc:8). The IMAP template has restrictqrun but not restrictmailq.

No template defines confMAX_MESSAGE_SIZE, confMAX_DAEMON_CHILDREN, or confCONNECTION_RATE_THROTTLE. A burst of inbound connections from a single IP is bounded only by the per-tier ECS container’s memory ceiling.

The smtp-in template enables a generous AUTH mechanisms list — EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN (docker/templates/in-sendmail.mc:27-29) — but smtp-in is the inbound relay; it should not be doing AUTH at all (submission auth is on smtp-out via Dovecot). DIGEST-MD5 and CRAM-MD5 are weak by modern standards (RFC 6331 obsoleted DIGEST-MD5).

docker/shared/generate-config.sh writes sendmail maps directly to their final paths (/etc/mail/virtusertable, etc.) rather than write-temp-then-rename. A SIGHUP or restart that lands mid-write reads a partial file. Race window is small (millisecond-scale) but non-zero.

Dovecot config

Both IMAP and SMTP-OUT submission set disable_plaintext_auth = no (docker/imap/configs/dovecot/10-auth.conf:1, docker/smtp-out/configs/dovecot/10-auth.conf:1). The architectural justification is that TLS terminates at the NLB for IMAP (port 993 → 143) and at Dovecot itself for submission (the NLB passes 587/465 through TCP unencrypted). Anything that talks to Dovecot inside the container’s network namespace can therefore do plaintext auth. If the NLB is ever bypassed (operator running kubectl port-forward-equivalent, sidecar shipped, ECS Exec session for debugging), Dovecot will accept credentials in clear.

ssl_min_protocol = TLSv1.2 is set in the entrypoint docker/shared/entrypoint.sh:110. No explicit cipher list — Dovecot’s default is fine but unspecified is unspecified.

IMAP mail_max_userip_connections = 100 is set. No login_max_processes, no auth_failure_delay, no auth_cache_size settings that would slow down brute-force.

OpenDKIM scope

docker/shared/generate-config.sh:230 writes /etc/opendkim/TrustedHosts with a single line: 0.0.0.0/0\n. docker/smtp-out/configs/opendkim.conf:15-16 references the same file for both ExternalIgnoreList and InternalHosts. The effect is “any source IP is treated as internal,” meaning OpenDKIM signs any matching From: regardless of source.

The blast radius is moderated by the SigningTable: gen_dkim_signingtable writes *@<subdomain>.<tld> cabal._domainkey.<subdomain>.<tld> for each configured subdomain. A message with From: arbitrary@external.com would not match any SigningTable entry and would not be signed. So the practical risk is bounded: an attacker who reaches port 25 on the container and presents a From: on a domain Cabalmail signs (which they cannot do unless they also pass the dovecot auth layer for that user) gets a signature.

Still, defence in depth: the failure mode is “any single piece of the chain breaks and we are an open DKIM signer.” That is the kind of finding worth fixing while it is cheap.

EFS posture

The IMAP task’s EFS mount is plain: file_system_id = var.efs_id; root_directory = "/" (terraform/infra/modules/ecs/task-definitions.tf:80-86). No transit_encryption, no authorization_config. The smtp-out queue mount does set transit_encryption = "ENABLED" and uses an access point (task-definitions.tf:266-276), so the model is there — it just was not retrofitted onto IMAP when the access-point pattern landed.

The EFS filesystem itself is encrypted at rest (terraform/infra/modules/efs/main.tf); transit encryption is the missing piece for IMAP.

Mount options for noexec/nosuid/nodev are not currently exposable through efs_volume_configuration. ECS sets reasonable defaults; the access-point posture is the practical control surface.

fail2ban dead config

The supervisord entries are commented out; the entrypoint nevertheless writes /etc/fail2ban/jail.local with the VPC CIDR allowlist (docker/shared/entrypoint.sh:159-171) and the NET_ADMIN capability is still requested. The image install also still pulls in fail2ban via dnf install. Code, capability, and runtime artefact are all present for a feature that is off.

Target state

Phase 1 — Drop fail2ban dead weight; drop NET_ADMIN

Each tier’s Dockerfile removes fail2ban from the dnf install line and the supervisord.conf [program:fail2ban] blocks are deleted (not just commented). docker/shared/entrypoint.sh:159-171’s jail.local stanza is removed.

terraform/infra/modules/ecs/task-definitions.tf drops the linuxParameters.capabilities.add = ["NET_ADMIN"] blocks on all three mail tiers.

This is a no-op for behaviour (nothing using the cap) and a real reduction in attack surface (escape-to-host scenarios that exploit NET_ADMIN are off the table).

Phase 2 — Container runtime posture

The capability-and-privilege posture applies cleanly to every tier:

linuxParameters = {
  initProcessEnabled = true
  capabilities       = { drop = ["ALL"], add = [] }  # opt back in below
}

dockerSecurityOptions = ["no-new-privileges:true"] on each container.

Per-tier capability adds:

readOnlyRootFilesystem is not a flag flip for the mail tiers

The mail tiers are heavy runtime config-mutators, not just at startup. reconfigure.sh runs as a sidecar loop and re-runs the full config generation on every address change (SQS-triggered) and on the periodic fallback (reconfigure.sh:30-84). Each pass writes across the root filesystem; with a read-only root and set -euo pipefail, the first EROFS aborts the regeneration, and new addresses/subdomains silently stop propagating — the periodic fallback fails too, so there is no self-heal.

The full writable-path inventory — everything that needs a writable mount before ROFS can be set on a mail tier:

Path Written by When
/etc/mail/{access,virtusertable,mailertable,relay-domains,local-host-names,masq-domains} generate-config.sh startup + every reconfigure
/etc/mail/*.db makemap (reconfigure), make -C /etc/mail (entrypoint) startup + every reconfigure
/etc/mail/sendmail.mc, /etc/mail/sendmail.cf entrypoint render + make startup
/etc/opendkim/{KeyTable,SigningTable,TrustedHosts} generate-config.sh startup + every reconfigure (smtp-out)
/etc/opendkim/keys/cabal entrypoint startup (smtp-out)
/etc/aliases, /etc/aliases.dynamic, /etc/aliases.db entrypoint + reconfigure (newaliases) startup + every reconfigure (imap)
/etc/dovecot/conf.d/{10-ssl,05-login}.conf entrypoint startup (imap, smtp-out)
/etc/dovecot/master-users entrypoint (htpasswd) startup (imap)
/etc/pki/tls/{certs,private}/* entrypoint startup
/usr/bin/cognito.bash entrypoint startup
/etc/fail2ban/jail.local entrypoint startup (removed in Phase 1)
/var/lib/rsyslog entrypoint (mkdir) startup
/etc/hosts hosts-pin.sh startup + on IMAP IP change (smtp-in)
/tmp (scratch via mktemp) generate-config.sh startup + every reconfigure

Two tmpfs mounts (/tmp, /var/run) — the original scope of this phase — cover almost none of this. The real mutation surface is the mail config itself.

The tmpfs-shadowing trap. /etc/mail and /etc/opendkim cannot simply be tmpfs-mounted: both ship image-baked content. /etc/mail holds the sendmail-cf m4 sources and Makefile that make -C /etc/mail needs, the COPYed sendmail.mc.template, and (smtp-out) the static access map; /etc/opendkim is laid out by the package. An empty tmpfs over either shadows that content, so the entrypoint would have to seed the tmpfs (copy baked files in) before generating — a change to entrypoint ordering, not just a task-def edit.

Three ways to reconcile ROFS with this, in increasing order of effort and payoff:

  1. Posture-without-ROFS for the mail tiers (default for 0.10.x). Apply drop=ALL, no-new-privileges, and initProcessEnabled to imap/smtp-in/smtp-out but leave readOnlyRootFilesystem unset. Keeps the bulk of the hardening value and avoids the seeding problem entirely. This is the pragmatic landing spot, given that “rewrite /etc/mail from DynamoDB” is these images’ normal operation.

  2. Full tmpfs + entrypoint seeding. tmpfs-mount every writable path above and seed /etc/mail//etc/opendkim from baked copies at entrypoint. Achieves ROFS, but the entire mail-config surface is writable tmpfs anyway, so the marginal benefit over option 1 is modest and the entrypoint complexity is real. The tmpfs shape (ECS on EC2 supports it via linuxParameters.tmpfs):

    linuxParameters = {
  initProcessEnabled = true
  tmpfs = [
    { containerPath = "/tmp",     size = 64,  mountOptions = ["rw","nosuid","nodev","noexec"] },
    { containerPath = "/var/run", size = 32,  mountOptions = ["rw","nosuid","nodev"] },
    # ...plus /etc/mail, /etc/opendkim, /etc/dovecot/conf.d, /etc/pki/tls, /var/lib/rsyslog, /etc/aliases*
  ]
  capabilities = { drop = ["ALL"], add = [] }
}
  3. Relocate generated config to a single writable prefix. Refactor generate-config.sh, reconfigure.sh, and the daemon configs so all generated maps/tables/aliases live under one writable path (e.g. /run/cabal/...) and point sendmail/dovecot/opendkim at it. Cleanest end state — the actual root stays read-only — but a meaningful refactor that touches the daemon config wiring.

The monitoring tier does no runtime config regeneration and is a clean ROFS candidate; it can go read-only independently of the mail tiers and should not be gated on this decision.

Recommendation

Ship option 1 for the three mail tiers in 0.10.x (posture hardening without ROFS), set readOnlyRootFilesystem = true on the monitoring tier, and capture option 3 as a follow-up if ROFS on the mail tiers later becomes a requirement. The entrypoint and reconfigure write paths are the gating constraint here, not the capability drop.

The posture changes are still the highest-touch part of this plan. Migration order is dev → stage → prod with at least one mail-roundtrip end-to-end test per environment between flips. Other things to verify:

Pre-flight: a development-environment soak under load is essential before stage rollout.

Phase 2a as implemented (0.10.8) — capability tightening

The 0.10.4 add-back sets were flagged to tighten during the soak. Reviewed per tier against what each actually runs:

Shipped as two ordered deploys: (1) the entrypoint sync-users gate via app.yml, with smtp-in verified healthy and relaying; then (2) the task-def cap drop via Terraform (smtp-in marker v3 → v4). The cap drop must not roll before the gate is live, or smtp-in startup would fail without caps it is still using.

Phase 3 — Image digest pinning + ECR scan on push

Three pieces:

  1. Each Dockerfile’s FROM line gains a digest. FROM amazonlinux:2023@sha256:<...>. Renovate config gets a docker ecosystem entry that auto-bumps the digest when the upstream tag advances.
  2. The ECS task definitions reference ECR images by digest, not by tag. The image-tag-via-SSM pattern stays; the value in SSM is the digest (<repo>@sha256:<...>) instead of <repo>:sha-<8>. .github/scripts/deploy-ecs-service.sh extracts the digest from docker buildx --metadata-file and writes it to SSM.
  3. ECR repositories gain image_scanning_configuration { scan_on_push = true } and image_tag_mutability = "IMMUTABLE" in terraform/infra/modules/ecr/main.tf. Findings surfaced via the existing app.yml deploy job (poll aws ecr describe-image-scan-findings after push; warn on HIGH/CRITICAL until a baseline is established, then fail).

Nightly Trivy scan against the current digest-pinned images, results to GitHub Code Scanning (same surface as the Trivy IaC scan from iac-quality-gates-plan.md).

Phase 3 as implemented (0.10.6)

Reconciled against the codebase as it actually stood, Phase 3 shipped three of the four pieces above; piece 2 was dropped.

Phase 4 — Dovecot plaintext-off + login throttling

Flip docker/imap/configs/dovecot/10-auth.conf:1 and docker/smtp-out/configs/dovecot/10-auth.conf:1 to disable_plaintext_auth = yes. Add to both configs:

auth_failure_delay = 2 secs
auth_mechanisms = plain login

Add to IMAP:

service imap-login {
  process_limit = 1024
  client_limit = 1
}
service auth {
  client_limit = 4096
}

And to submission:

service submission-login {
  process_limit = 512
  client_limit = 1
}

The TLS terminator question: IMAP traffic arrives at the container in clear (NLB terminates 993→143). disable_plaintext_auth = yes would reject the IMAP login path unless we tell Dovecot the connection is effectively TLS. Two options:

  1. login_trusted_networks = <NLB subnet CIDR>. Dovecot treats sessions from these source IPs as already-TLS for auth purposes. The NLB source IP range is stable (the NLB IPs themselves are stable; backing them with proxy protocol is not configured today). Risk: anyone inside the VPC who can connect to the IMAP service port can bypass TLS. Acceptable given the VPC posture.
  2. End-to-end TLS (NLB passthrough mode), Dovecot terminates TLS itself. Higher operational complexity; cert rotation has to land at Dovecot via SSM/EFS. Defer.

Recommendation: option (1), with the NLB-subnet CIDR plumbed as an env var (LOGIN_TRUSTED_NETWORKS) injected by the ECS task definition. The entrypoint writes it to the dovecot config.

Phase 4 reconnaissance (verified live 2026-06-06)

The option-(1) assumption was checked against the running infrastructure before committing to it; it holds, with one refinement.

For submission, NLB does TCP passthrough; Dovecot already terminates TLS; disable_plaintext_auth = yes works out of the box without trusted-networks games.

Phase 4 as implemented (0.10.11)

Shipped as designed, with these specifics:

Unlike the 2a capability work, there is no startup-crash or lockout failure mode: disable_plaintext_auth = yes only takes effect with the new image, and that image always writes some login_trusted_networks (the NLB CIDRs, or the VPC fallback), so it never rejects legitimate NLB-forwarded auth. The stage gate is therefore about confirming the intended effect — real clients (React, Apple, raw IMAP) still authenticate, and a non-TLS bypass path is now refused — not about avoiding a self-inflicted outage.

Phase 5 — Sendmail and OpenDKIM hardening

Sendmail .mc templates

Add to all three:

define(`confMAX_MESSAGE_SIZE',         `52428800')dnl
define(`confMAX_DAEMON_CHILDREN',      `40')dnl
define(`confCONNECTION_RATE_THROTTLE', `5')dnl
define(`confREJECT_LOG_INTERVAL',      `3h')dnl

Append restrictmailq to confPRIVACY_FLAGS in docker/templates/in-sendmail.mc:8 and docker/templates/out-sendmail.mc:8.

Remove DIGEST-MD5 and CRAM-MD5 from confAUTH_MECHANISMS in docker/templates/in-sendmail.mc:27-29; for inbound relay, the right answer is to remove AUTH entirely (smtp-in does not need it — submission auth is on smtp-out via Dovecot). Cut to: define('confAUTH_MECHANISMS', 'EXTERNAL')dnl or remove the macro.

Atomic config writes

In docker/shared/generate-config.sh:

def write(path, content):
    os.makedirs(os.path.dirname(path), exist_ok=True)
    tmp = path + ".tmp"
    with open(tmp, "w") as f:
        f.write(content)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)
    print(f"  Generated {path}")

os.replace is atomic on POSIX filesystems.

OpenDKIM scope

docker/shared/generate-config.sh:230 writes a tighter TrustedHosts:

write("/etc/opendkim/TrustedHosts",
      "127.0.0.1\n::1\nlocalhost\n")

Sendmail hands mail to OpenDKIM via the milter socket on loopback; the only legitimate signing source is local. The SigningTable still constrains which domain to sign for, but now both axes (network position and From-domain match) have to be true.

Phase 6 — EFS transit encryption on IMAP

terraform/infra/modules/ecs/task-definitions.tf:80-86 is updated to:

volume {
  name = "mailstore"
  efs_volume_configuration {
    file_system_id     = var.efs_id
    root_directory     = "/"
    transit_encryption = "ENABLED"
    authorization_config {
      access_point_id = var.mailstore_access_point_id
      iam             = "DISABLED"
    }
  }
}

A new EFS access point on the mailstore (mirroring the existing smtp_queue_access_point_id pattern) constrains the IMAP task to /maildir (or wherever the existing mount root effectively points). Created in terraform/infra/modules/efs/main.tf.

The access-point creation has to land before the task-definition change references it; that is one Terraform apply, since both files are in the same stack.

Phase 6 as implemented (0.10.7)

Shipped as transit encryption only — no access point. The sketch above mirrors the smtp-queue access point, but the mailstore is not analogous to the queue: it is a multi-user tree rooted at the EFS root / (mounted to /home, one maildir per user, each owned by that user’s UID via sync-users.sh), whereas the queue is a single /smtp-queue subtree. An access point would therefore have to be root_directory = "/" with no posix_user — any posix override squashes the per-user ownership and every mailbox reads empty — and iam = "DISABLED" (the queue AP already disables IAM “for parity with the IMAP mount”). That is a transparent pass-through: zero gain over plain transit encryption, plus a data-path footgun if the root were ever mis-set. And transit_encryption = "ENABLED" does not require an access point — the queue pairs them only because it needed the AP to pin /smtp-queue and set the creation owner.

So the change is just transit_encryption = "ENABLED" on the existing root_directory = "/" mailstore volume, with the imap revision marker bumped v3 -> v4 so the volume edit actually deploys (a volume edit, like a container_definitions edit, is otherwise held back by ignore_changes). The smtp-out tier already runs ENABLED on these same ECS EC2 hosts, so the transit path is proven; the roll is one task replacement (brief IMAP blip). If per-tier IAM auth on EFS is ever wanted (the identity-IAM-hardening plan), add the access point then with iam = "ENABLED" — that is when an AP earns its keep.

Migration sequence

Each phase is one PR (or a small PR set) and is independently revertable.

Phase Scope Risk
1 — fail2ban + NET_ADMIN Docker + Terraform ECS task defs Low. No-op on behaviour; reduces capabilities.
2 — Runtime posture (caps, no-new-privs, init) Terraform ECS task defs Medium for the caps/no-new-privs/init posture (all tiers). ROFS is deferred for the mail tiers — they regenerate /etc/mail + /etc/opendkim at runtime; only the monitoring tier gets ROFS in 0.10.x. See Phase 2 for the option 1/2/3 tradeoff.
3 — Image digest pinning + ECR scan-on-push Dockerfiles, ECR module, deploy script, Renovate config Medium. Build cadence changes; expect some early scan-on-push noise from the AL2023 base.
4 — Dovecot plaintext-off + throttling Docker dovecot configs + entrypoint Medium. Auth path change; test with React webmail and Apple clients before promotion.
5 — Sendmail/OpenDKIM hardening Docker .mc templates + generate-config.sh Low to medium. confMAX_MESSAGE_SIZE change is the user-visible one (50 MB cap); document in release notes. OpenDKIM scope change is invisible to clients.
6 — EFS transit encryption on IMAP Terraform EFS + ECS Low. Mailstore data path is unchanged; only the wire is now TLS. Brief outage on the access-point creation roll-forward (one task-set replacement).

Each phase runs the full dev → stage → prod sequence per the standard branching rules. Phases 2 and 4 are the ones to slow-roll; the rest can move as quickly as CI feedback supports.

Rollback

CI changes

Acceptance

Open questions

Out of scope for 0.10.x