TCP Three-Way Handshake: The Formal Greeting

This is the moment a TCP connection is born. It's a formal three-step greeting that ensures both the client and the server are ready, listening, and agree on the details before any actual data is sent.

Metaphors:

  • The Phone Call:
    • SYN: You dial a friend. This is the moment you hear the phone ring on their end.
    • SYN-ACK: Your friend picks up and says, "Hello? I hear you. Can you hear me?"
    • ACK: You respond, "Got it, ready to talk now!"
  • The Secure Zoom Call:
    • SYN (The Invitation): You send the meeting link.
    • SYN-ACK (The Confirmation): The host confirms they are ready and sends you an acknowledgment.
    • ACK (The Join): You acknowledge the confirmation and hit "Join."

Why this matters for security:

  • The Handshake's flags (SYN, ACK) are the language of port scanning.
  • Attackers abuse the initial step to launch denial-of-service (DDoS) attacks.
  • Firewall rules must understand the Handshake to properly allow or deny connections.

18a. The Handshake Process (Step-by-Step)

One-line definition: The three-way handshake is the process TCP uses to establish a reliable, sequenced connection between a client and a server.

Why it exists: It ensures that both sides agree on the initial sequence numbers (ISN) and buffer sizes, guaranteeing that data can be tracked and reassembled correctly.

One-line definition: The Synchronization flag is the very first packet sent by the client to request a connection.

Why it exists: It tells the server, "Hey, I want to talk to you," and sends the client's ISN.

  • Flag Set: SYN = 1 in the TCP header.
  • Sequence Number: The client chooses a random 32-bit ISN.
  • Server State: Transitions from LISTEN to SYN_RECEIVED (or SYN_SENT depending on who initiates).

One-line definition: The server's response simultaneously acknowledges the client and sends its own request back.

Why it exists: It confirms the server heard the client and provides the server's own ISN.

  • Flags Set: Both SYN and ACK are set to 1.
  • Acknowledgement Number: Set to Client_ISN + 1 ("I received X, send X+1").
  • Sequence Number: The server chooses its own random ISN for the return flow.

One-line definition: The final packet from the client confirms the connection and transitions to established.

Why it exists: It's the final agreement, confirming the client received the server's SYN-ACK and is ready for data.

  • Flag Set: Only the ACK flag is set to 1.
  • Acknowledgement Number: Set to Server_ISN + 1.
  • Final State: Both sides move to ESTABLISHED.

🌍 Example (Cybersecurity - Attacker/Defender Perspective):

Scenario: A penetration tester wants to perform a stealthy port scan that is less likely to be logged by a firewall.

Attack Technique: SYN Scan ("Half-Open" Scan)

  1. The attacker sends a SYN packet to a port.
  2. If the port is OPEN, the server sends back a SYN-ACK.
  3. CRITICAL: Instead of the final ACK, the attacker immediately sends a RST (Reset) packet, or simply drops the connection.

Why it works: The connection never hits the ESTABLISHED state, bypassing simple logging systems while still confirming the port is open.

  • Prevention (Stateful Firewall): A modern firewall tracks the Handshake. If it sees high volumes of incomplete connections, it flags this as a probable stealth scan and blocks the source IP.
  • Detection (IDS/IPS): An IDS monitors for patterns of sequential SYN packets followed by RSTs from the same source.

18b. Security Implications & Attacks

One-line definition: A DDoS attack that prevents legitimate users from connecting by overwhelming the server with half-open TCP connections.

The Restaurant Analogy: Imagine a restaurant with 100 tables. Pranksters call and make 1,000 reservations. The host writes them all down, reserving spots and waiting. Soon, the host's notebook is full, and real, paying customers can't even get on the waiting list.

How it works:

  1. Attacker sends a massive flood of SYN packets.
  2. The server replies with SYN-ACK and places the request in the Backlog Queue.
  3. The attacker never sends the final ACK (often using spoofed source IPs), so the server sends the SYN-ACK to a fake address and waits forever.
  4. The backlog queue fills up, and the server drops legitimate SYN requests.

One-line definition: A cryptographic technique allowing a server to respond to a SYN without consuming resources until the final ACK arrives.

The Restaurant Fix: Instead of writing down a reservation, the host gives the caller a coded ticket number. Only if the caller physically shows up and presents that valid ticket is a table actually reserved.

The 4-Step Validation:

  1. SYN: Server receives the request.
  2. SYN-ACK with Cookie: The server calculates a cryptographic hash (cookie) based on the client IP, port, server IP, server port, and the client's ISN. This hash becomes the server's sequence number.
  3. ACK: A legitimate client sends the final ACK using that cookie as its acknowledgement number.
  4. Validation: The server re-calculates the hash. If it matches, it then builds the connection state. Malicious packets without the valid cookie are dropped.

🌍 Example 2 (Cybersecurity - Attacker/Defender Perspective):

Scenario: A popular e-commerce site is undergoing a major SYN flood attack.

  • Attacker's Steps (Exploiting Handshake):
    1. Uses a botnet to send millions of SYN packets with spoofed source IPs.
    2. The server's CPU spikes to 100%, and the Backlog Queue fills instantly.
    3. Legitimate users receive "Connection Timed Out" because there's no room for their requests.
  • Defender's Steps (Applying Mitigation):
    1. Network security enables SYN Cookies on the firewall/OS.
    2. The server stops allocating memory for half-open connections.
    3. Malicious spoofed packets fail validation and are dropped; legitimate clients establish connections, restoring service.

📝 Recap (3. TCP Three-Way Handshake)

  1. The Handshake uses three packets: SYN, SYN-ACK, and ACK.
  2. It's crucial because it ensures both sides agree on Initial Sequence Numbers (ISN) for reliable, ordered data delivery.
  3. SYN Flooding is a DDoS attack that exploits the server's need to reserve memory for half-open connections.
  4. SYN Cookies is the primary mitigation, which delays resource allocation until the connection is fully confirmed via a validated cryptographic hash.

📚 Sources & References

  • RFC 793: Transmission Control Protocol specifications.
  • MITRE ATT&CK T1499.001: Endpoint Denial of Service: Application or System Impairment.
  • SYN Cookies: Primary technique in modern Linux/Unix kernels.
  • BCP 38 / RFC 2827: Network Ingress Filtering.