When you invoke ESP or AH after the IP header to protect a datagram, you are using transport mode. An example follows. A packet starts off as:
ESP, in transport mode, protects the data as follows:
AH, in transport mode, protects the data as follows:
AH actually covers the data before it appears in the datagram. Consequently, the protection that is provided by AH, even in transport mode, covers some of the IP header.
When an entire datagram is inside the protection of an IPsec header, IPsec is protecting the datagram in tunnel mode. Because AH covers most of its preceding IP header, tunnel mode is usually performed only on ESP. The previous example datagram would be protected in tunnel mode as follows:
Often, in tunnel mode, the outer (unprotected) IP header has different source and destination addresses from the inner (protected) IP header. The inner and outer IP headers can match if, for example, an IPsec-aware network program uses self-encapsulation with ESP. Self-encapsulation with ESP protects an IP header option.
The Solaris implementation of IPsec is primarily a transport mode IPsec implementation, which implements the tunnel mode as a special instance of the transport mode. The implementation treats IP-in-IP tunnels as a special transport provider. The ifconfig(1M) configuration options to set tunnels are nearly identical to the options available to socket programmers when enabling per-socket IPsec. Also, tunnel mode can be enabled in per-socket IPsec. In per-socket tunnel mode, the inner packet IP header has the same addresses as the outer IP header. See the ipsec(7P) man page for details on per-socket policy.
A configured tunnel is a point-to-point interface. It enables an IP packet to be encapsulated within an IP packet. Configuring a tunnel requires both a tunnel source and tunnel destination. See the tun(7M) man page and Solaris Tunneling Interfaces for IPv6 for more information.
A tunnel creates an apparent physical interface to IP. The physical link's integrity depends on the underlying security protocols. If you set up the security associations securely, then you can trust the tunnel. That is, packets that exit the tunnel originated from the peer that was specified in the tunnel destination. If this trust exists, you can use per-interface IP forwarding to create a virtual private network.