Securing the Network in Oracle® Solaris 11.2

Exit Print View

Updated: August 2014

Encapsulating Security Payload

The encapsulating security payload (ESP) protocol provides confidentiality over what the ESP encapsulates. ESP also provides the services that AH provides. However, ESP does not protect the outer IP header. ESP provides authentication services to ensure the integrity of the protected packet. Because ESP uses encryption-enabling technology, a system that provides ESP can be subject to import and export control laws.

The ESP header and trailer encapsulate the IP payload. When encryption is used with ESP, it is applied only over the IP payload data, as shown in the following illustration.

image:Graphic shows the ESP header between the IP header and the TCP header. The TCP header is encrypted by the ESP header.

In a TCP packet, the ESP header is authenticated and it encapsulates the TCP header and its data. If the packet is an IP-in-IP packet, ESP protects the inner IP packet. Per-socket policy allows self-encapsulation, so ESP can encapsulate IP options when necessary.

Self-encapsulation can be used by writing a program that uses the setsockopt(). If self-encapsulation is set, a copy of the IP header is made to construct an IP-in-IP packet. For example, when self-encapsulation is not set on a TCP socket, the packet is sent in the following format:

[ IP(a -> b) options + TCP + data ]

When self-encapsulation is set on that TCP socket, the packet is sent in the following format:

[ IP(a -> b) + ESP [ IP(a -> b) options + TCP + data ] ]

For further discussion, see Transport and Tunnel Modes in IPsec.

Security Considerations When Using AH and ESP

The following table compares the protections that are provided by AH and ESP.

Table 6-1  Protections Provided by AH and ESP in IPsec
Packet Coverage
Against Attacks
Protects packet from the IP header to the end of the transport data
Provides strong integrity, data authentication:
  • Ensures that the receiver receives exactly what the sender sent

  • Is susceptible to replay attacks when an AH does not enable replay protection

Replay, cut-and-paste
Protects packet from the ESP header to the end of the transport data
With encryption option, encrypts the IP payload. Ensures confidentiality
With authentication option, provides the same payload protection as AH
Replay, cut-and-paste
With both options, provides strong integrity, data authentication, and confidentiality
Replay, cut-and-paste, eavesdropping