System Administration Guide: IP Services

Encapsulating Security Payload

The encapsulating security payload (ESP) module provides confidentiality over what the ESP encapsulates. ESP also provides the services that AH provides. However, ESP only provides its protections over the part of the datagram that ESP encapsulates. ESP provides optional authentication services to ensure the integrity of the protected packet. Because ESP uses encryption-enabling technology, a system that provides ESP can be subject to import and export control laws.

ESP encapsulates its data, so ESP only protects the data that follows its beginning in the datagram, as shown in the following illustration.

Diagram shows the ESP header between the IP header and
the TCP header. The TCP header is encrypted by the ESP header.

In a TCP packet, ESP encapsulates only the TCP header and its data. If the packet is an IP-in-IP datagram, ESP protects the inner IP datagram. Per-socket policy allows self-encapsulation, so ESP can encapsulate IP options when ESP needs to.

If self-encapsulation is set, a copy of the IP header is made to construct an IP-in-IP datagram. For example, when self-encapsulation is not set on a TCP socket, the datagram is sent in the following format:

[ IP(a -> b) options + TCP + data ]

When self-encapsulation is set on that TCP socket, the datagram is sent in the following format:

[ IP(a -> b) + ESP [ IP(a -> b) options + TCP + data ] ]

For further discussion, see Transport and Tunnel Modes in IPsec.

Security Considerations When Using AH and ESP

The following table compares the protections that are provided by AH and ESP.

Table 19–2 Protections Provided by AH and ESP in IPsec


Packet Coverage 


Against Attacks 


Protects packet from the IP header to the transport header 

Provides strong integrity, data authentication: 

  • Ensures that the receiver receives exactly what the sender sent

  • Is susceptible to replay attacks when an AH does not enable replay protection

Replay, cut-and-paste 


Protects packet following the beginning of ESP in the datagram. 

With encryption option, encrypts the IP datagram. Ensures confidentiality 


With authentication option, provides the same protection as AH 

Replay, cut-and-paste 

With both options, provides strong integrity, data authentication, and confidentiality 

Replay, cut-and-paste, eavesdropping