What is IPsec? Tunnel vs Transport Mode, ESP, AH, and IKE Phases

IPsec remains the dominant enterprise VPN protocol because it operates at Layer 3, making it transparent to applications and compatible with any IP-based traffic—TCP, UDP, ICMP, or proprietary protocols. Defined across RFC 4301 (architecture), RFC 4302 (AH), RFC 4303 (ESP), and RFC 7296 (IKEv2), IPsec is vendor-neutral and interoperates across Cisco ASA, Palo Alto, Fortinet, Juniper SRX, and cloud gateways (AWS VPN, Azure VPN Gateway, GCP Cloud VPN). Unlike SSL/TLS VPNs that terminate at Layer 4–7 and require client software, IPsec is kernel-integrated on Windows, Linux, macOS, iOS, and Android. This makes it the default choice for site-to-site VPNs connecting branch offices, data centers, and cloud VPCs. In India, BFSI institutions (HDFC Bank, ICICI, SBI) mandate IPsec for inter-branch connectivity to comply with RBI's IT Framework for NBFC guidelines, which require end-to-end encryption for financial transaction networks. IPsec's longevity stems from its cryptographic agility: it supports AES-128/256-GCM, ChaCha20-Poly1305, SHA-256/384/512 for integrity, and Diffie-Hellman groups 14–21 for key exchange. Modern IKEv2 (RFC 7296) adds MOBIKE (RFC 4555) for seamless roaming across network changes—critical for 5G and SD-WAN deployments. Cisco's FlexVPN and Viptela SD-WAN both use IKEv2 as the control plane. Despite WireGuard's emergence, IPsec's mature tooling (StrongSwan, Libreswan, Cisco IOS crypto maps) and compliance certifications (FIPS 140-2, Common Criteria EAL4+) ensure it will dominate enterprise networks through 2030.

Tunnel mode encapsulates the entire original IP packet (header + payload) inside a new IP packet. The original packet becomes the payload of the outer packet, and a new IP header is prepended with the VPN gateway's public IP addresses as source and destination. This is defined in RFC 4301 §3.2: "In tunnel mode, the entire original IP datagram is protected and becomes the payload of a new IP datagram." The process works as follows: (1) Host A (10.1.1.10) sends a packet to Host B (10.2.2.20). (2) Gateway R1 intercepts the packet, encrypts it using ESP (Encapsulating Security Payload), and wraps it in a new IP header with R1's public IP (203.0.113.1) as source and R2's public IP (198.51.100.1) as destination. (3) The encrypted packet traverses the Internet. (4) Gateway R2 decrypts it and forwards the original packet to Host B. Tunnel mode is mandatory for site-to-site VPNs because: - Private addressing: Internal RFC 1918 addresses (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) are not routable on the Internet; tunnel mode hides them. - NAT traversal: RFC 3948 defines NAT-T (NAT Traversal), which encapsulates ESP inside UDP port 4500 to pass through NAT devices. Tunnel mode's outer header carries the NAT-translated public IPs. - Policy-based routing: The outer header allows ISPs and SD-WAN controllers to route VPN traffic independently of the inner payload. In our HSR Layout lab, we benchmark tunnel mode overhead: AES-256-GCM adds ~50–70 bytes (new IP header 20 bytes, ESP header 8 bytes, IV 8 bytes, ESP trailer 2–255 bytes, ICV 16 bytes). For a 1500-byte MTU, effective payload drops to ~1430 bytes, requiring MSS clamping (tcp adjust-mss 1360 on Cisco IOS). Production deployments at Cisco India's Bengaluru campus use tunnel mode for 200+ branch offices, with IKEv2 rekeying every 3600 seconds (1 hour) to limit per-SA data volume and comply with cryptographic best practices.

Transport mode encrypts only the Layer 4 payload (TCP/UDP data) and leaves the original IP header intact. RFC 4301 §3.2 states: "In transport mode, only the payload of the original IP datagram is protected; the IP header is left unmodified (except for the IP protocol field, TTL, and checksum)." The original IP header's protocol field is changed from 6 (TCP) or 17 (UDP) to 50 (ESP) or 51 (AH). The encrypted payload is inserted between the IP header and the ESP/AH trailer. This reduces overhead: transport mode adds only ~30–50 bytes (ESP header 8 bytes, IV 8 bytes, ESP trailer 2–255 bytes, ICV 16 bytes) compared to tunnel mode's ~50–70 bytes. Transport mode is used in three scenarios: 1. Host-to-host VPNs: When two endpoints have routable public IPs and no NAT is involved. Example: A Linux server at 203.0.113.10 communicating with another at 198.51.100.20. Both run StrongSwan in transport mode to encrypt database replication traffic. 2. L2TP/IPsec: Microsoft Windows and macOS remote-access VPNs use L2TP (Layer 2 Tunneling Protocol) over IPsec transport mode. L2TP creates the tunnel; IPsec transport mode encrypts the L2TP packets. This is common in Indian SMBs using Windows Server RRAS. 3. GRE over IPsec: Cisco routers use GRE (Generic Routing Encapsulation) to carry multicast/broadcast traffic (OSPF, EIGRP hello packets), then apply IPsec transport mode to encrypt the GRE tunnel. The configuration is crypto ipsec profile applied to a tunnel interface. This is taught in CCNP ENCOR (350-401) and CCNP Security (350-701). Transport mode cannot traverse NAT without additional encapsulation because NAT rewrites IP headers, breaking ESP's integrity checks. RFC 3948 NAT-T solves this by encapsulating ESP in UDP, but this effectively creates a pseudo-tunnel mode. For this reason, 95% of production IPsec deployments use tunnel mode.

IPsec defines two security protocols: ESP (Encapsulating Security Payload, RFC 4303) and AH (Authentication Header, RFC 4302). ESP provides: - Confidentiality: Encrypts the payload using AES-GCM, AES-CBC, ChaCha20-Poly1305. - Integrity: Authenticates the payload and ESP header using HMAC-SHA-256/384/512 or AEAD (Authenticated Encryption with Associated Data) like GCM. - Anti-replay: Uses a 32-bit or 64-bit (RFC 4304 Extended Sequence Numbers) sequence number to detect replayed packets. ESP does not authenticate the outer IP header in tunnel mode. This is acceptable because the outer header is transient (created by the VPN gateway) and not part of the original packet. AH provides: - Integrity: Authenticates the entire IP packet, including the IP header (except mutable fields like TTL, which are zeroed before hashing). - Anti-replay: Same sequence number mechanism as ESP. - No confidentiality: AH does not encrypt; it only authenticates. AH is obsolete in 2026 for three reasons: 1. NAT incompatibility: AH authenticates the IP header, so any NAT device that rewrites source/destination IPs breaks the authentication. RFC 3947 explicitly states AH cannot traverse NAT. 2. Firewall interference: Many stateful firewalls and DPI (Deep Packet Inspection) engines cannot parse AH (IP protocol 51) and drop it. ESP (IP protocol 50) is widely recognized. 3. ESP with null encryption: If you need authentication without confidentiality (e.g., for compliance reasons), you can use ESP with NULL encryption (RFC 2410). This provides the same integrity as AH while remaining NAT-traversable via NAT-T. In our 4-month paid internship at the Network Security Operations Division, students configure Cisco ASA and Palo Alto firewalls exclusively with ESP. We tested AH in lab scenarios to demonstrate its NAT failure, but no production partner (HCL, Akamai India, Barracuda) uses AH. Cisco IOS defaults to transform-set ESP-AES-256-SHA for this reason.

IKE (Internet Key Exchange) Phase 1 establishes a secure, authenticated control channel called the ISAKMP SA (Internet Security Association and Key Management Protocol Security Association). This bidirectional SA is used to negotiate Phase 2 parameters. RFC 7296 (IKEv2) and RFC 2409 (IKEv1) define two modes: IKEv1 Phase 1 modes: - Main Mode: 6-message exchange. Provides identity protection (peer identities are encrypted). Used when both peers have static public IPs. - Aggressive Mode: 3-message exchange. Faster but exposes peer identities in cleartext. Used when one peer has a dynamic IP (remote-access VPN). IKEv2 (the current standard) uses a single 4-message exchange (IKE_SA_INIT + IKE_AUTH) that combines the functions of IKEv1 Main Mode and Quick Mode. IKEv2 is mandatory in Cisco FlexVPN, Azure VPN Gateway, and AWS VPN. Phase 1 negotiates: 1. Encryption algorithm: AES-128/256-CBC or AES-GCM. 2. Integrity algorithm: SHA-256, SHA-384, SHA-512. 3. Diffie-Hellman group: Group 14 (2048-bit MODP), Group 19 (256-bit ECP), Group 20 (384-bit ECP). NIST SP 800-77 Rev. 1 deprecates groups 1, 2, 5 (≤1024-bit) as of 2023. 4. Authentication method: Pre-shared key (PSK), RSA signatures, ECDSA, or EAP (for remote-access). 5. Lifetime: Typically 86400 seconds (24 hours). After expiry, Phase 1 renegotiates. The Diffie-Hellman exchange generates a shared secret without transmitting it. Both peers derive the SKEYID (IKEv1) or SKEYSEED (IKEv2) from the DH shared secret and nonces. From SKEYSEED, IKEv2 derives: - SK_d: Used to derive child SA keys (Phase 2). - SK_ai, SK_ar: Integrity keys for IKE messages. - SK_ei, SK_er: Encryption keys for IKE messages. - SK_pi, SK_pr: Authentication keys. In production, we see Phase 1 failures due to: - Mismatched proposals: One peer offers AES-256-SHA-256-DH14, the other expects AES-128-SHA-1-DH5. Cisco IOS logs %CRYPTO-4-IKMP_NO_SA. - Clock skew: IKEv2 certificates have notBefore/notAfter timestamps. If clock skew exceeds 5 minutes, authentication fails. NTP is mandatory. - Firewall blocking UDP 500/4500: IKE Phase 1 uses UDP 500. If NAT is detected, it switches to UDP 4500 (NAT-T). Many Indian ISPs block UDP 4500; we configure policy-based routing to tunnel VPN traffic over a dedicated MPLS link.

IKE Phase 2 negotiates the IPsec SA (Security Association) that encrypts actual user data. In IKEv1, this is called Quick Mode; in IKEv2, it's the CREATE_CHILD_SA exchange. Phase 2 runs inside the encrypted Phase 1 tunnel, so all parameters are confidential. Phase 2 negotiates: 1. IPsec protocol: ESP or AH (always ESP in practice). 2. Encryption algorithm: AES-128/256-GCM, AES-CBC, ChaCha20-Poly1305. 3. Integrity algorithm: HMAC-SHA-256/384/512 (not needed if using AEAD like GCM). 4. Encapsulation mode: Tunnel or transport. 5. Lifetime: Typically 3600 seconds (1 hour) or 100 MB of data. Shorter lifetimes limit cryptanalysis exposure. 6. Perfect Forward Secrecy (PFS): Optional. If enabled, Phase 2 performs a new Diffie-Hellman exchange to derive keys independently of Phase 1. This ensures that compromise of Phase 1 keys does not compromise past Phase 2 sessions. PFS adds ~200ms latency but is mandatory for high-security environments (BFSI, defense). Each Phase 2 negotiation creates two unidirectional SAs: one inbound, one outbound. Each SA is identified by a 32-bit SPI (Security Parameter Index) and the destination IP. The tuple (SPI, destination IP, protocol) uniquely identifies an SA. You can view SAs on Cisco IOS with show crypto ipsec sa. Rekeying: Before the SA lifetime expires, IKE initiates a new Phase 2 exchange to create a new SA. For a brief overlap period, both old and new SAs are active. Once the new SA is confirmed, the old SA is deleted. This is called "make-before-break" and ensures zero packet loss during rekey. Failure scenarios: - Proxy ID mismatch: In policy-based VPNs, each Phase 2 SA is tied to a traffic selector (e.g., 10.1.0.0/16 ↔ 10.2.0.0/16). If one peer proposes 10.1.0.0/16 ↔ 10.2.0.0/24 and the other expects 10.1.0.0/16 ↔ 10.2.0.0/16, Phase 2 fails. Cisco logs %CRYPTO-4-IKMP_NO_SA: remote proxy does not match. - PFS group mismatch: If one peer requires PFS group 14 and the other offers group 5, negotiation fails. In our HSR Layout lab, we configure Cisco ASA with crypto ipsec ikev2 ipsec-proposal AES256-SHA256 and crypto map CMAP 10 set pfs group14. Students troubleshoot Phase 2 failures using debug crypto ikev2 protocol 127 and debug crypto ipsec 127. This is a core CCNP Security (350-701) lab scenario.

A Security Association (SA) is a one-way relationship between two IPsec peers that defines encryption/authentication parameters for a specific traffic flow. RFC 4301 §4.1 defines the SA as a tuple: (SPI, destination IP, protocol). Each SA is unidirectional, so a bidirectional VPN requires two SAs (one inbound, one outbound). IPsec maintains two databases: 1. SPD (Security Policy Database): Defines which traffic requires IPsec protection. Each SPD entry is a rule: - Selector: Source IP, destination IP, protocol, ports. - Action: PROTECT (apply IPsec), BYPASS (send in cleartext), or DISCARD (drop). - SA parameters: Which transform set (encryption/integrity algorithms) to use. Example SPD entry on Cisco IOS:

access-list 100 permit ip 10.1.0.0 0.0.255.255 10.2.0.0 0.0.255.255
crypto map CMAP 10 ipsec-isakmp
 set peer 198.51.100.1
 set transform-set ESP-AES-SHA
 match address 100

This says: "Traffic from 10.1.0.0/16 to 10.2.0.0/16 must be encrypted using ESP-AES-SHA and sent to peer 198.51.100.1." 2. SAD (Security Association Database): Stores active SAs. Each SAD entry contains: - SPI: 32-bit identifier (randomly generated during IKE Phase 2). - Sequence number: Current anti-replay counter. - Encryption key: Derived from IKE Phase 2. - Integrity key: Derived from IKE Phase 2. - Lifetime: Remaining seconds or bytes before rekey. - Mode: Tunnel or transport. You can view the SAD on Cisco IOS with show crypto ipsec sa. Example output:

interface: GigabitEthernet0/0
    Crypto map tag: CMAP, local addr 203.0.113.1
   protected vrf: (none)
   local  ident (addr/mask/prot/port): (10.1.0.0/255.255.0.0/0/0)
   remote ident (addr/mask/prot/port): (10.2.0.0/255.255.0.0/0/0)
   current_peer 198.51.100.1 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 5000, #pkts encrypt: 5000, #pkts digest: 5000
    #pkts decaps: 4800, #pkts decrypt: 4800, #pkts verify: 4800
    inbound esp sas:
      spi: 0x12345678(305419896)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 1, flow_id: SW:1, sibling_flags 80004040, crypto map: CMAP
        sa timing: remaining key lifetime (k/sec): (4500000/3200)

The SPD is consulted on every outbound packet. If a match is found and no SA exists, IKE Phase 2 is triggered. Once the SA is established, the SAD is used for fast-path encryption/decryption. This two-database architecture is why IPsec scales to thousands of tunnels: the SPD is small (policy rules), and the SAD is indexed by SPI for O(1) lookup.

IPsec troubleshooting follows a layered approach: connectivity → Phase 1 → Phase 2 → data plane. Here are the most common failures we see in production: 1. UDP 500/4500 blocked: IKE Phase 1 uses UDP 500. If NAT is detected, it switches to UDP 4500. Many Indian ISPs (Airtel, Jio Fiber) block UDP 4500 by default. Solution: Configure port forwarding on the CPE router or use a dedicated MPLS link. Verify with telnet <peer-ip> 500 (won't connect, but checks if port is open). 2. Mismatched Phase 1 proposals: One peer offers aes-256-sha-256-dh14, the other expects aes-128-sha-1-dh5. Cisco logs %CRYPTO-4-IKMP_NO_SA. Solution: Use show crypto isakmp sa to see the peer's proposal, then align both sides. Best practice: Configure multiple proposals in priority order. 3. Pre-shared key mismatch: Case-sensitive. MySecret123 ≠ mysecret123. Cisco logs %CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from <peer> failed its sanity check or is malformed. Solution: Re-enter the PSK on both sides. For production, use RSA signatures (certificates) instead of PSK. 4. Proxy ID mismatch (Phase 2): Cisco policy-based VPNs use access-lists as proxy IDs. If R1 has 10.1.0.0/16 ↔ 10.2.0.0/16 and R2 has 10.1.0.0/24 ↔ 10.2.0.0/16, Phase 2 fails. Cisco logs %CRYPTO-4-IKMP_NO_SA: remote proxy does not match. Solution: Ensure both ACLs are mirror images. Use show crypto ipsec sa to see the negotiated selectors. 5. MTU/fragmentation issues: IPsec adds 50–70 bytes overhead. If the original packet is 1500 bytes, the encrypted packet is 1550–1570 bytes, exceeding the interface MTU. The packet is fragmented, but many firewalls drop IPsec fragments. Solution: Configure MSS clamping on the tunnel interface: ip tcp adjust-mss 1360 (Cisco IOS). This forces TCP to use smaller segments. 6. Asymmetric routing: In hub-and-spoke topologies, traffic from Spoke1 to Spoke2 might route via the hub, but return traffic might take a direct path (if spokes have a separate connection). IPsec SAs are unidirectional and tied to the peer IP, so asymmetric routing breaks the tunnel. Solution: Use route-based VPNs (VTI - Virtual Tunnel Interface) instead of policy-based, and configure OSPF/BGP over the VTI to ensure symmetric routing. 7. NAT before IPsec: If a router performs NAT on the source IP before applying IPsec, the SPD selector won't match. Example: Host 10.1.1.10 sends to 10.2.2.20. Router NATs it to 203.0.113.10 → 10.2.2.20. The SPD expects 10.1.1.10 → 10.2.2.20, so IPsec is not applied. Solution: Configure crypto map CMAP local-address GigabitEthernet0/0 to ensure IPsec uses the pre-NAT address, or use route-based VPN with VTI. In our HSR Layout lab, we simulate all seven scenarios across Cisco ASA, Palo Alto PA-220, and Fortinet FortiGate 60F. Students use Wireshark to capture IKE exchanges (UDP 500/4500) and identify failure points. This is a mandatory module in our CCNP Security and Full Stack Network Security courses.

Modern SD-WAN platforms (Cisco Viptela, VMware VeloCloud, Fortinet SD-WAN, Aryaka) use IPsec as the data-plane encryption protocol. The SD-WAN controller (vManage, VeloCloud Orchestrator) automates IKE configuration, eliminating manual crypto map configuration. Cisco SD-WAN (Viptela): Uses IKEv2 with AES-256-GCM and DH group 14. Each vEdge router establishes IPsec tunnels to all other vEdges (full mesh) or to hub vEdges (hub-and-spoke). The controller pushes IPsec policies via OMP (Overlay Management Protocol). Rekeying happens every 86400 seconds (24 hours). Cisco SD-WAN supports IPsec over multiple transports: MPLS, broadband, LTE. The controller performs path selection based on latency, jitter, and loss, then steers traffic over the best IPsec tunnel. Cloud VPN gateways: - AWS VPN: Supports IKEv1 and IKEv2. Uses AES-128/256, SHA-1/256, DH groups 2/14/15/16/17/18/19/20/21/22/23/24. AWS requires route-based VPN (BGP over IPsec). You configure a Virtual Private Gateway (VGW) or Transit Gateway, download the configuration file, and apply it to your on-premises router. AWS charges $0.05/hour per VPN connection + data transfer. - Azure VPN Gateway: Supports IKEv2 only (IKEv1 deprecated in 2024). Uses AES-256-GCM, SHA-256, DH group 14. Azure supports both policy-based (legacy) and route-based VPN. Route-based is mandatory for active-active HA and ExpressRoute coexistence. Azure charges based on gateway SKU (VpnGw1: $0.19/hour, VpnGw2: $0.49/hour). - GCP Cloud VPN: Supports IKEv2 with AES-128/256-GCM/CBC, SHA-1/256/384/512, DH groups 14/15/16. GCP uses route-based VPN with BGP. You create a Cloud VPN Gateway, configure peer IPs and PSKs, then establish BGP sessions. GCP charges $0.05/hour per tunnel + $0.10/GB egress. Indian enterprises (HCL, Wipro, TCS) use hybrid cloud architectures: on-premises data centers in Mumbai/Bengaluru connected to AWS Mumbai (ap-south-1) and Azure Central India via IPsec. For compliance with DPDP Act 2023 (Data Protection and Digital Privacy), sensitive data (PII, financial records) must remain in India. IPsec tunnels ensure data-in-transit encryption between Indian regions. In our 4-month paid internship, students configure AWS Site-to-Site VPN with Cisco CSR 1000v, establish BGP over IPsec, and verify route propagation using show ip bgp and show crypto session. This is a core module in our Cloud Security & Cybersecurity course.

IPsec performance depends on CPU, encryption offload, and tunnel count. Here's what we measure in our HSR Layout lab: 1. Encryption overhead: AES-256-GCM on a Cisco ASA 5516-X (no hardware crypto accelerator) achieves ~500 Mbps throughput. The same ASA with AES-NI (Intel CPU instruction set) reaches ~1.2 Gbps. Palo Alto PA-3220 (dedicated crypto ASIC) reaches ~4 Gbps. For 10 Gbps+ throughput, you need appliances like Cisco ASR 1000 with ESP-200 (hardware crypto module) or Palo Alto PA-5220. 2. Tunnel count: Each IPsec SA consumes ~4 KB of memory (keys, sequence numbers, timers). A router with 8 GB RAM can theoretically support 2 million SAs, but practical limits are lower due to SPD lookup overhead. Cisco ASA 5516-X supports 10,000 tunnels; ASR 1000 supports 64,000 tunnels. SD-WAN controllers (Cisco vManage) support 10,000+ vEdges, each with 10–50 tunnels, totaling 100,000–500,000 SAs. 3. Rekeying storms: If 10,000 tunnels are configured with the same Phase 2 lifetime (3600 seconds) and all start simultaneously, they will all rekey at the same time, causing a CPU spike. Solution: Configure lifetime jitter: Cisco IOS crypto ipsec security-association lifetime seconds 3600 jitter 10 adds ±10% randomness (3240–3960 seconds). 4. Fragmentation: IPsec adds 50–70 bytes. For 1500-byte MTU links, this causes fragmentation. Fragmented IPsec packets are often dropped by stateful firewalls. Solution: (a) Reduce MTU on the tunnel interface: ip mtu 1400. (b) Enable MSS clamping: ip tcp adjust-mss 1360. (c) Use Path MTU Discovery (PMTUD), but many firewalls block ICMP Type 3 Code 4 (Fragmentation Needed), breaking PMTUD. 5. Anti-replay window size: RFC 4303 defines a 32-bit sequence number, allowing 2^32 packets per SA (~4 billion). At 10 Gbps with 1500-byte packets, this is exhausted in ~50 minutes. RFC 4304 Extended Sequence Numbers (ESN) uses 64-bit sequence numbers, extending the limit to 2^64 packets (~5.8 million years at 10 Gbps). Cisco IOS enables ESN with crypto ipsec security-association replay window-size 1024. 6. NAT-T overhead: Encapsulating ESP in UDP adds 8 bytes per packet. At 10 Gbps with 64-byte packets (worst case), this is 12.5 million packets/sec × 8 bytes = 100 MB/sec = 800 Mbps overhead. For high-PPS environments (DDoS mitigation, CDN), avoid NAT-T; use public IPs. Our 800+ hiring partners (Cisco, HCL, Akamai India, Barracuda) deploy IPsec at scale: Cisco's Bengaluru campus has 5000+ IPsec tunnels on ASR 1000 routers; Akamai India uses IPsec to secure CDN node-to-origin traffic across 50+ Indian PoPs. Students in our Full Stack Network Security course benchmark IPsec performance using iperf3 and analyze packet captures with Wireshark to identify bottlenecks.