Computer Networks Transport Layer

SYN Floods and TCP-Layer Attacks

Half-open connections, SYN cookies, retransmission timeouts — the security and resilience knobs on TCP.

Concept Intermediate
8 min read
syn-flood tcp security ddos syn-cookies

Summary#

A SYN flood exploits the cheapest asymmetry in TCP: the server commits real state on receiving a SYN (allocates a Transmission Control Block, starts a retransmit timer for the SYN-ACK), but the client commits nothing — and the source IP in the SYN is trivially spoofable. An attacker sends a stream of SYNs from random source IPs, fills the listen queue with half-open connections, and starves legitimate clients. The defence — SYN cookies, plus connection-table limits, plus retransmission tuning — is one of the oldest deployed counters to a DoS attack and is still the right answer.

Beyond SYN floods, the same kind of resource asymmetry shows up in other TCP-layer attacks: RST injection, sequence-number guessing, slow-read (Slowloris), and amplification via misconfigured services. The TCP retransmission timer is at the centre of several of them — it’s also the place defences hook in. This writeup covers SYN floods in depth and the broader family of TCP-layer attacks around them.

Why it matters#

Every service that listens on a public port faces this attack. Network appliances and cloud load balancers handle it for you when you’re behind them; bare-metal servers need to know the kernel knobs. SRE interviews ask “what is a SYN flood and how do you defend against it” routinely — the wrong answer (“more bandwidth”) signals shallow protocol knowledge.

It also matters because the SYN cookie technique is a beautiful piece of protocol engineering — stateless validation using only the SYN-ACK sequence number as a cryptographic carrier. Understanding it sharpens your sense of what’s possible when you accept that “the client will echo back our chosen value.”

How it works#

The vulnerability#

Recall the three-way handshake:

client                              server
  | -- SYN, seq=X ---------------->  |  (server allocates TCB, queues half-open)
  | <- SYN-ACK, seq=Y, ack=X+1 ----  |  (server starts retransmit timer)
  | -- ACK, seq=X+1, ack=Y+1 ----->  |  (handshake complete; moves to ESTABLISHED)

On step 1 the server has committed memory (a Transmission Control Block) and a timer (to retransmit the SYN-ACK if the third ACK never comes). The client has sent one packet from a forgeable source IP. If the third ACK never arrives, the server eventually times out the half-open slot — but the default RTO retries take 30+ seconds.

An attacker spraying SYNs at 100k/s from random source IPs fills the half-open queue immediately. Legitimate clients’ SYNs are dropped (tcp_max_syn_backlog exceeded), or the queue accepts but never makes progress.

SYN cookies#

Bernstein’s 1996 trick: instead of allocating a TCB on the SYN, encode the connection’s identifying state into the SYN-ACK’s initial sequence number Y itself. If the client is real, it echoes Y+1 in the third ACK; the server validates that the echoed value is a properly-signed cookie and only then allocates the TCB.

The cookie structure (simplified):

Y = MAC(secret, client_IP, client_port, server_IP, server_port, t) || t || mss_index

where t is a coarse timestamp (so old cookies expire) and mss_index encodes the MSS into 3 bits (since the server doesn’t have the original SYN’s options anymore). On the third ACK, the server recomputes the MAC, checks it, decodes the MSS, and synthesises a TCB.

Result: no state until the handshake is complete. Spoofed SYNs cost the server CPU but no memory. Legitimate clients see a normal handshake. Linux enables tcp_syncookies = 1 by default and triggers them when the SYN queue is near full.

Trade-offs of SYN cookies#

  • Lose negotiated options. Window scale, SACK permitted, and timestamps are negotiated at SYN/SYN-ACK; SYN cookies discard them. Recent Linux smuggles a few back via the timestamp field, but it’s imperfect.
  • No early data. TCP Fast Open’s SYN-borne payload requires per-client cookies; SYN cookies (which are per-connection) clash with it.
  • CPU cost on cookie validation. Each ACK requires a MAC computation. Cheap, but non-zero.

Net: keep them as a fallback, enabled only when the SYN queue is saturated. Don’t make them the steady state.

Retransmission timer in the loop#

The server retransmits the SYN-ACK if the third ACK doesn’t arrive. Linux defaults to 5 retries, exponentially backing off — about 63 seconds total. A spoofed SYN holds a queue slot for that entire window. net.ipv4.tcp_synack_retries = 2 cuts the per-spoof cost dramatically. Combined with SYN cookies, this neutralises most floods.

Other TCP-layer attacks#

  • RST injection. An off-path attacker who can guess the 5-tuple and approximate sequence number sends a forged RST. Defences: randomise initial sequence numbers (RFC 6528), require the RST’s sequence to fall in the current receive window, RFC 5961’s relaxed acceptance with explicit ACKs.
  • Sequence-number guessing. Predictable ISNs let attackers inject data into a connection without seeing it. Mitigation: cryptographically strong ISN generation per RFC 6528.
  • Slowloris / slow-read. Application-layer cousin of SYN flood. Client opens a real TCP connection and dribbles bytes one at a time, holding a worker thread. Defences: time-out idle reads, cap concurrent connections per source IP, use event-driven servers (nginx, Envoy) where slow clients cost a socket but not a thread.
  • Reflection / amplification. Attacker sends a TCP SYN with the victim’s source IP spoofed; the server replies with SYN-ACK to the victim. Modest amplification (1:1 by packet, but with retransmits). UDP-based reflection (DNS, NTP, memcached) is far more dangerous; TCP’s three-way handshake provides natural rate-limiting.
  • TCP-MD5 / TCP-AO bypass. BGP sessions use TCP-MD5 to authenticate peers; attackers who break the MD5 key can inject route updates. TCP-AO (RFC 5925) replaced it with stronger algorithms.

Variants and trade-offs#

SYN cookies on (always-on or threshold) — stateless under attack, no half-open queue exhaustion. Lose option negotiation (wscale, SACK, timestamps). Worth the trade at the edge — most clients don’t notice the lost options.
SYN cookies off / threshold not reached — full option negotiation, standard TCP behaviour, slightly cheaper per-handshake CPU. Vulnerable to flood unless the SYN queue is large and synack_retries is tuned down.

Other defence trade-offs:

  • tcp_max_syn_backlog. Larger queue absorbs bursts but uses more memory. Default 128–1024; tune to ~8192 for public-facing servers.
  • tcp_synack_retries. Lower (1–2) abandons spoofed SYNs faster — at the cost of legitimate clients on lossy links seeing more handshake failures.
  • tcp_abort_on_overflow. When the accept queue (not SYN queue) overflows, send RST instead of silently dropping. Faster failure visibility; client retries land sooner.
  • Connection-rate limits per source IP. Effective when sources are real; useless against IP-spoofed floods.
  • Cloud / CDN scrubbing. Anti-DDoS services absorb floods at edge points-of-presence with anycast and SYN-proxying. The economical answer for most internet-facing services.
Why is the SYN-ACK retransmission timer the key knob?

The attacker’s per-SYN cost is fixed (one packet). The server’s cost is the half-open slot held for the SYN-ACK retransmission window. If the server holds a slot for 63 seconds per SYN, a 100k SYN/s attacker pins 6.3 million slots before any timer fires. Cutting retransmit retries from 5 to 2 reduces per-SYN slot holding from 63 s to about 7 s — an order of magnitude less queue pressure for the same defensive cost.

When this is asked in interviews#

Standard SRE / security / networking-deep-dive question. Expect “explain a SYN flood and how you’d defend a service against it.” Strong answers cover: the asymmetry of state (server commits, client doesn’t, source IP spoofable), SYN cookies as the canonical defence (and what they cost), the kernel knobs (syncookies, max_syn_backlog, synack_retries), and the edge-layer mitigation (CDN / scrubbing service).

Common follow-ups:

  • “What does a SYN cookie actually encode?” — A MAC over the 5-tuple plus a timestamp, packed into the SYN-ACK’s initial sequence number. The third ACK echoes it; the server re-verifies and reconstructs the TCB.
  • “What’s the cost of always-on SYN cookies?” — Loss of negotiated options (window scale, SACK, timestamps) for cookied connections. Modern Linux smuggles some back through timestamps but imperfectly.
  • “How is Slowloris different from a SYN flood?” — Slowloris uses real connections and dribbles application-layer data; it attacks application worker pools, not the TCP handshake. Defences are at L7 (idle timeouts, event-driven servers), not L4.
  • “Why is UDP amplification worse than TCP reflection?” — UDP has no handshake to validate the source IP, so spoofed requests reach the reflector and the response goes to the victim. TCP’s three-way handshake forces the client to echo the server’s sequence number, preventing one-way reflection at scale.
  • “What’s the difference between the SYN queue and the accept queue?” — SYN queue holds half-open connections (after SYN, before ACK). Accept queue holds fully-handshaked connections waiting for the application’s accept(). Both can overflow; both have separate knobs (tcp_max_syn_backlog, somaxconn).

Asked across security, SRE, and infrastructure loops. Tier-one anti-DoS interview question.

Search ESC

Keyboard shortcuts

Shortcuts are disabled while typing in inputs.