Why Docker containers lose connection on the bridge network — how to fix intermittent production outages

By · Published on · 8 min read

Blue network cables plugged into a patch panel, focus on the connectors with a blurred background
Physical connections in a network patch panel, representing infrastructure and connectivity diagnostics. Photo: Brett Sayles / Pexels

If Docker containers experience intermittent connection drops on the bridge network, a focused diagnosis saves hours of trial and error. Common root causes are MTU/PMTU, conntrack/NAT, rp_filter/ARP, or the embedded DNS. Below I show how to identify each root cause and apply safe fixes in production.

Symptoms and when the intermittency appears

Bridge intermittency appears as timeouts, TCP resets (RST), and connections that work and then suddenly break. Sometimes only large transfers fail while short pings pass.

The common pattern: services in Docker are reachable via published ports, but internal or external clients see occasional failures—especially under load, when many containers start/stop, or after network changes on the host.

  • TCP connects, transfers a few KB and then stalls until timeout.
  • Large uploads/downloads fail while small requests succeed.
  • Kernel messages about conntrack/ARP or ‘martian’ packets.
  • DNS resolution inside containers is slow or intermittent.

Most common technical causes on the Docker bridge

A few root causes show up repeatedly in real production incidents. Understand the mechanism so you can apply the right fix.

Good news: most issues have objective diagnostics that you can run on the host and inside containers.

  • MTU/PMTU: veth/docker0 MTU mismatches the physical network or tunnels, causing blackholes for large packets.
  • Conntrack/NAT: the nf_conntrack table fills up, ephemeral ports exhaust, and sessions drop randomly.
  • rp_filter/routes: asymmetric routing from VRRP/VPN or multiple NICs makes the kernel drop legitimate replies.
  • ARP/neighbor cache: stale entries or aggressive GC after heavy container churn.
  • Docker DNS (127.0.0.11): the embedded resolver struggles under high QPS or with poor timeouts.
  • DOCKER-USER/iptables rules: broad DROP policies affecting docker0 traffic without proper exceptions.

Quick diagnosis on the host and inside containers

Run an objective triage before changing configuration. Look for signs of MTU, conntrack, and firewall/routing drops.

Execute checks from both the host and an affected container. Log timestamps and compare them against traffic spikes.

# 1) Sinais no kernel (conntrack, ARP/neighbor, martian)
dmesg -T | egrep -i 'conntrack|nf_conntrack|neighbor|arp|martian|FRAG'

# 2) Estado de conntrack e portas efêmeras
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
ss -s
cat /proc/sys/net/ipv4/ip_local_port_range

# 3) PMTU/MTU: teste DF e caminho
GW=$(ip route | awk '/default/ {print $3; exit}')
ping -c 3 -M do -s 1472 "$GW" || echo "Possível MTU < 1500 no caminho"
tracepath -n "$GW" | tail -n +1

# 4) ARP/neigh suspeito (incomplete/failed)
ip -s neigh | egrep -i 'FAILED|INCOMPLETE' -n || true

# 5) Regras do Docker e DOCKER-USER
iptables -S DOCKER-USER 2>/dev/null || true
iptables -t nat -S | sed -n '1,120p' 2>/dev/null || true

# 6) De dentro de um container afetado: DNS e MTU
getent hosts example.com || nslookup example.com || dig +short example.com || true
ip route show
echo $(cat /sys/class/net/eth0/mtu) MTU-no-container

MTU and PMTU: detect and fix without taking everything down

Failures that only affect large payloads almost always point to MTU/PMTU. In cloud environments with tunnels (VXLAN/GRE) the effective MTU is often 1450–1460. If docker0/eth0 inside the container stays at 1500, you get a blackhole.

Confirm with a DF ping. If a smaller MTU is required, change Docker’s MTU to match the network. This requires restarting Docker and recreating the bridge—plan a maintenance window.

  • Validate with ping -M do to the gateway or service.
  • Adjust Docker’s daemon MTU so veth interfaces inherit the correct size.
  • Retest and monitor for fragmentation errors.
# Teste PMTU (ajuste o destino para um serviço fora do host)
ping -c 3 -M do -s 1472 $DEST || echo "Tente 1432/1400 para descobrir a MTU segura"

# Ajuste de MTU no Docker (crie/edite /etc/docker/daemon.json)
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json >/dev/null <<'EOF'
{
  "mtu": 1450
}
EOF

# Reinicie com cuidado (interrompe containers)
sudo systemctl restart docker

# Verifique
aioctl=$(ip -o link show docker0 | awk '{print $2,$9,$10,$11}')
echo "docker0 ->" $ioctl
ip -o link show | grep veth | head -n 3

Conntrack, NAT and ephemeral ports: limits and tuning

A full nf_conntrack table prevents new connections and can drop existing ones. High NAT usage also suffers if the ephemeral port range is too narrow.

Tune conntrack limits carefully, widen the ephemeral port range, and if you manage NAT rules manually prefer MASQUERADE with --random-fully to reduce 5‑tuple collisions.

  • Monitor count vs max and keep a healthy headroom.
  • Widen ip_local_port_range on hosts with heavy outbound traffic.
  • Avoid rules that force expensive per-packet inspection across all traffic.
# Medir e ajustar conntrack
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# Exemplo: elevar (persistir em /etc/sysctl.d)
echo 'net.netfilter.nf_conntrack_max = 524288' | sudo tee /etc/sysctl.d/90-conntrack.conf
sudo sysctl --system

# Portas efêmeras mais amplas
sudo sysctl -w net.ipv4.ip_local_port_range="10240 65535"
echo 'net.ipv4.ip_local_port_range = 10240 65535' | sudo tee /etc/sysctl.d/90-ports.conf
sudo sysctl --system

# Se você administra NAT manualmente (fora das regras do Docker):
# iptables -t nat -C POSTROUTING -o eth0 -j MASQUERADE --random-fully || \
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE --random-fully

ARP, neighbor cache and rp_filter: routes that vanish

Asymmetric routing (VPN, VRRP, or multiple NICs) with strict rp_filter makes the kernel drop legitimate responses. A small ARP/neighbor cache under churn also causes intermittent failures.

Set rp_filter to ‘loose’ where there is asymmetry and increase neighbor GC thresholds. Make changes with caution and document them.

  • Look for ‘martian source’ messages in the logs.
  • Confirm return path with ip route get.
  • Increase neighbor GC thresholds on hosts with many peers/containers.
# rp_filter atual
sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.default.rp_filter
# Tornar ‘loose’ (2) se há assimetria/VRRP/VPN
sudo sysctl -w net.ipv4.conf.all.rp_filter=2
sudo sysctl -w net.ipv4.conf.default.rp_filter=2

echo -e 'net.ipv4.conf.all.rp_filter=2\nnet.ipv4.conf.default.rp_filter=2' | \
  sudo tee /etc/sysctl.d/90-rpfilter.conf

# Neighbour cache mais folgado
sysctl net.ipv4.neigh.default.gc_thresh1 net.ipv4.neigh.default.gc_thresh2 net.ipv4.neigh.default.gc_thresh3
sudo bash -c 'cat > /etc/sysctl.d/90-neigh.conf <<EOF
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 16384
EOF'
sudo sysctl --system

# Limpar entradas problemáticas (impacta momentaneamente a resolução ARP)
sudo ip neigh flush dev docker0 || true

Docker DNS (127.0.0.11) and firewall: when it’s not the network

The embedded Docker resolver (127.0.0.11) can become a bottleneck under high QPS or with poor timeouts. In locked-down environments, a DOCKER-USER chain with a default DROP can block bridge traffic.

Test with an external/caching resolver and set dns_opt in Compose. Review DOCKER-USER to ensure explicit exceptions before applying broad DROP rules.

  • Use a local caching resolver (e.g., 10.0.0.53) to reduce upstream QPS.
  • Lower DNS timeouts/attempts so failures fail fast and fall back to another server.
  • Allow RELATED, ESTABLISHED and docker0 traffic in the firewall rules.
# docker-compose.yml (trecho)
services:
  app:
    image: example/app
    dns:
      - 10.0.0.53
      - 1.1.1.1
    dns_opt:
      - timeout:1
      - attempts:3
    # ... restante do serviço

Observability, tests and next steps

After applying fixes, validate under load and keep monitoring. Conntrack metrics, TCP drops and DNS latency help prevent regressions.

Implement health checks to remove failing instances early, and use a well-configured reverse proxy for failover and certificate renewal.

  • Collect metrics with Prometheus/Grafana and create actionable alerts.
  • Centralize logs (kernel, iptables, apps) for correlation.
  • Automate readiness checks and circuit breakers in the application.
  • Test PMTU in deploy pipelines and after network changes.
  • Avoid the default bridge; prefer user-defined networks per logical domain.

Frequently asked questions

How can I tell if the problem is MTU/PMTU or something else?

If small pings work but large packets with DF fail, that's a strong sign of MTU/PMTU. Tracepath helps reveal the hop that reduces MTU. Random drops under load with nf_conntrack messages point to conntrack/NAT. ‘Martian’ logs suggest rp_filter/route issues.

Does increasing nf_conntrack_max permanently solve the problem?

It helps absorb spikes but isn’t a silver bullet. If the traffic pattern creates many short connections, also tune timeouts, widen ephemeral ports, and review architecture (connection pooling/reuse). Monitor the count to avoid unbounded growth.

Is disabling rp_filter safe?

Avoid disabling it completely. Use ‘loose’ mode (2) when routing is asymmetric, such as with VRRP or VPNs. Strict mode on multi-homed hosts causes false positives and drops. Document changes and monitor ‘martian’ logs afterwards.

Should I use Docker’s embedded DNS or an external resolver?

For low traffic, Docker’s DNS works fine. Under high QPS or low-latency requirements, use a local caching resolver (unbound/dnsmasq) and point services to it in Compose. Also set dns_opt for short timeouts and multiple servers.

Does a user-defined bridge improve stability?

Yes. User-defined networks isolate broadcast domains, have more predictable rules and offer service-level DNS. They reduce collisions and make firewall policies simpler. Avoid placing all services on the default bridge—segment per logical domain.

Need to stabilize your containers’ network?

If you’re seeing Docker bridge intermittency and want a low-risk diagnostic and remediation plan, let’s discuss your environment and practical next steps.

Discuss Docker diagnosis

See also