How to diagnose and capture packets for intermittent network issues between Docker containers on Debian

By · Published on · 10 min read

Cranes and stacks of shipping containers at a port terminal
Visual metaphor for Docker containers and network flow Photo: Frank Rietsch / Pexels

Intermittent networking between Docker containers on Debian is often hard to reproduce. The shortest path to root cause is to confirm a hypothesis with packet captures at the right spot and controlled tests. Learn how to diagnose and capture packets for intermittent network issues between Docker containers on Debian using tcpdump/tshark, nsenter, tc netem, and MTU checks.

Signs of container-to-container intermittency that justify immediate capture

Real intermittency usually appears as variable response times, occasional connection resets, or DNS that sometimes succeeds and sometimes fails. If you observe any of these signs, prioritize capturing packets near the affected containers before the problem disappears.

Use this checklist to decide where to capture and what to note while reproducing the error.

  • Latency spikes on internal routes (p99 increases while average stays stable)
  • Occasional TCP RST errors when opening connections
  • DNS resolves sometimes and times out or returns NXDOMAIN at other times
  • gRPC/HTTP2 requests with intermittently broken streams
  • Logs show retries and timeouts without a clear pattern
  • Difference between intra-host traffic vs inter-host (Swarm/K8s)

Map of likely causes and how to quickly validate them

Use the mapping below to go straight to the most plausible hypothesis and the command that produces objective evidence. Capture both on the host and in the container's network namespace when possible to compare.

If you've already seen bridge network drops, dive deeper into why containers lose connectivity on the bridge and possible fixes in causes and fixes for Docker bridge connections (/en/docker-bridge-drops-connections).

Symptom Hypothesis Where to capture Check command
Intermittent timeout without server logs MTU/MSS and PMTUD blocked (ICMP 'Fragmentation Needed') host eth0 and container veth ping -M do -s 1472 <dest> | tcpdump icmp
Failures only for specific names Intermittent DNS (ndots/search, cache) container loopback and DNS egress getent hosts name; tcpdump port 53
RST right after SYN, sometimes Conntrack full or competing healthchecks Host (docker0/bridge) ss -s; conntrack -S; tcpdump 'tcp[tcpflags] & tcp-rst != 0'
Slow inside host, normal outside veth/bridge contention (qdisc), NIC offloads container veth and docker0 tc -s qdisc; ethtool -k <iface>
Only breaks inter-host Overlay MTU/encapsulation underlay and overlay interfaces tracepath <IP> and tcpdump gre/vxlan
IPv6 fails, IPv4 OK RA/NDP or IPv6 firewall policy internal and external IPv6 interfaces ping6 -s 1232 -M do; tcpdump icmp6

How to tell if the intermittency is DNS, routing, or MTU?

Separate by layer. Start with DNS, then connectivity/routing, and finally MTU/MSS. Measure each hypothesis with a deterministic test that can be repeated while capturing.

For IPv6 in containers and bridge networks on Debian, see the dedicated walkthrough on enabling and debugging IPv6 in Docker on Debian (/en/enable-debug-ipv6-docker-debian).

  • DNS: measure resolution time and distinguish NXDOMAIN from timeouts.
  • Routing: use tracepath/mtr and compare intra-host vs inter-host traces.
  • MTU: test ICMP with DF set and look for 'Fragmentation Needed' in captures.
# DNS (mede apenas resolução)
time getent hosts api.internal.svc || echo "falha DNS"

# Rota e MTU (Linux)
tracepath -n 10.0.0.25 | sed -n '1,6p'
# PMTUD: DF (do not fragment) e payload próximo a 1500
ping -M do -s 1472 10.0.0.25 -c 3

# TCP handshake cronometrado
TIMEFORMAT=%R; time bash -c 'exec 3<>/dev/tcp/10.0.0.25/443; echo ok >&3; exec 3<&- 3>&-'

Which command to use to capture packets inside the container?

The most reliable approach is to enter the container's network namespace with nsenter and run tcpdump/tshark there. That avoids having to map the veth pair on the host manually.

If you prefer to capture on the host, identify the correct veth and use filters for the container's IP/port.

# 1) Descubra o PID do container
eval PID=$(docker inspect -f '{{.State.Pid}}' nome_ou_id)

# 2) Entre no netns do container e capture (arquivo rotativo)
sudo nsenter -t "$PID" -n \
  tcpdump -ni any -w /tmp/cap.pcap -C 50 -W 5 \
  'host 10.0.0.25 and (tcp port 80 or 443)'

# Alternativa: tshark em modo texto
sudo nsenter -t "$PID" -n tshark -i any -Y 'tcp.analysis.retransmission or tcp.flags.reset==1' -T fields \
  -e frame.time -e ip.src -e ip.dst -e tcp.analysis.retransmission -e tcp.flags.reset

Safe workflow to reproduce and capture without disrupting production

Reproduce the issue with controlled traffic, capture to disk with a ring buffer, and limit the test window. This way you can correlate application and network timestamps accurately.

Set capture filters before starting the test to reduce volume and noise.

  • Define the target: IP/port and scenario (intra-host or inter-host).
  • Create a short 2–5 minute test window.
  • Use a ring buffer (-C/-W) to avoid filling disks.
  • Note start/end timestamps and request IDs.
# Terminal A: captura rotativa no namespace do cliente
PID_CLIENT=$(docker inspect -f '{{.State.Pid}}' svc_a)
sudo nsenter -t "$PID_CLIENT" -n tcpdump -ni any -w /tmp/cap_a.pcap -C 25 -W 4 'host 10.0.0.25 and tcp'

# Terminal B: tráfego de reprodução (baixa taxa para isolar)
for i in $(seq 1 200); do
  TS=$(date -Is)
  curl -sS --max-time 2 https://10.0.0.25/health || echo "$TS FAIL" >&2
  sleep 0.2
done

# Terminal C: marcação de eventos
echo "START $(date -Is)" >> /tmp/marks.log; sleep 150; echo "END $(date -Is)" >> /tmp/marks.log

# Depois: analisar
sudo nsenter -t "$PID_CLIENT" -n tshark -r /tmp/cap_a.pcap -qz io,stat,1,'COUNT(tcp.flags.reset) tcp.flags.reset==1'

When to use tc netem to simulate loss and latency?

Use tc netem to reproduce loss/jitter and verify whether the application fails under conditions similar to the ones observed. Apply it in the container's namespace or on the bridge to scope the impact.

Important: netem affects real traffic. Apply it in a test environment or during a controlled window and remove the rule afterward.

# Aplicar latência e perda no netns do container servidor
PID_SRV=$(docker inspect -f '{{.State.Pid}}' svc_b)
nsenter -t "$PID_SRV" -n bash -lc '
  IF=$(ip -o -4 route show default | awk "/default/ {print \$5; exit}")
  tc qdisc add dev "$IF" root netem delay 80ms 20ms loss 1%
  tc -s qdisc show dev "$IF"
'

# Remover	nsenter -t "$PID_SRV" -n bash -lc 'tc qdisc del dev $(ip -o -4 route show default | awk "/default/ {print \$5; exit}") root || true'

MTU and MSS: why they cause intermittency and how to measure

Encapsulation and asymmetric paths can reduce the effective MTU. If ICMP 'Fragmentation Needed' packets don't reach the sender, a handshake can fail intermittently. Validate MTU with ICMP DF tests and adjust MSS when necessary.

If you use overlay networking, consider a lower MTU for Docker's bridge or adjust TCPMSS on the host. Restarting Docker changes networks: plan a maintenance window, since interfaces and containers may reconnect.

  • Test PMTUD: look for 'Fragmentation Needed' in captures.
  • Temporary fix: TCPMSS clamp on egress.
  • Definitive fix: consistent MTU across Docker and host interfaces.
# Medir MTU efetiva (ICMP com DF)
ping -M do -s 1472 10.0.0.25 -c 2 || echo "tente tamanhos menores: 1464, 1452, 1432"

# Clamp MSS (temporário) no host
sudo iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN \
  -o docker0 -j TCPMSS --clamp-mss-to-pmtu

# Definir MTU da bridge do Docker (requer restart do serviço)
# /etc/docker/daemon.json
# {
#   "mtu": 1450
# }
# sudo systemctl restart docker  # ATENÇÃO: interrompe redes do Docker momentaneamente

How to confirm the fix and avoid regressions

Validate with side-by-side measurements: no errors for a sufficient period, stable latencies, and absence of anomalous resets/retransmissions. Then institutionalize tests and observability so you don't return to the blind spot.

Properly configured health and readiness checks prevent cascading failures during degradation.

  • Run mtr/tracepath for 15–30 minutes without loss.
  • Capture 5 minutes and confirm 0 unexpected RSTs.
  • Add synthetic internal checks (curl with timeout).
  • Configure health/readiness with backoff and a circuit breaker.
  • Document the target PMTUD/MTU in your runbook.
# Métrica rápida de retransmissão (deveria ser 0 ou baixo)
sudo tshark -r /tmp/cap_a.pcap -Y tcp.analysis.retransmission -T fields -e frame.number | wc -l

# Health/readiness em apps PHP (exemplos e práticas)
# Veja: health checks e readiness em Docker Compose
# Link: /health-checks-readiness-docker-compose-php

Frequently asked questions

Do I need to install tcpdump inside the container?

No. It's better to use nsenter to enter the container's network namespace and run tcpdump from the host. That avoids changing the image and ensures access to any interface in the netns. Capturing on the host also works but requires mapping the correct veth.

Which interface should I capture: docker0, veth, or any?

To isolate a container, capture inside the namespace with -i any. On the host, docker0 sees bridged traffic; the container's veth shows packets specific to that service. For overlay networks, also capture on the underlay interface to see encapsulation and check MTU.

Does TLS traffic make analysis harder?

TLS encrypts payloads, but metadata is still useful: examine the handshake, SNI, resets, key exchange latency, and retransmissions to confirm transport-layer intermittency. Combine tcpdump/tshark with application logs and synthetic checks to complete the diagnosis.

Will running tcpdump in production affect performance?

The impact is low when you use filters and write to pcap with a ring buffer. Avoid full-interface captures on very busy links. Monitor disk I/O and CPU during the test and limit capture windows to a few minutes.

How do I confirm an MTU problem between containers?

Use ping with DF (-M do) varying the size (-s) and capture ICMP; 'Fragmentation Needed' confirms that the effective MTU is smaller. In TCP, look for SYN/SYN-ACK followed by RST or incomplete handshakes. Fix by aligning MTU values or clamping TCPMSS on the path.

When should I use tc netem instead of relying on real traffic?

Use tc netem to reproduce conditions that are hard to capture live, like low packet loss and jitter. Apply it in the affected container's namespace and remove it afterward. It doesn't replace real traffic captures, but it validates application resilience under controlled degradation.

Need help structuring diagnostics without risking production?

Tell us about your scenario and we'll create a capture, controlled replay, and validation plan for Debian with Docker.

Discuss my case

See also