System Administration Guide: IP Services

Chapter 19 IP Security Architecture (Overview)

The IP Security Architecture (IPsec) provides cryptographic protection for IP datagrams in IPv4 and IPv6 network packets.

This chapter contains the following information:

To implement IPsec on your network, see Chapter 20, Configuring IPsec (Tasks). For reference information, see Chapter 21, IP Security Architecture (Reference).

What's New in IPsec?

Solaris 10 4/09: Starting in this release, the Service Management Facility (SMF) manages IPsec as a set of services.

By default, two IPsec services are enabled at system boot:

By default, the key management services are disabled at system boot:

To activate IPsec policies under SMF, you perform the following steps:

  1. Add IPsec policy entries to the ipsecinit.conf file.

  2. Configure the Internet Key Exchange (IKE) or manually configure keys.

  3. Refresh the IPsec policy service.

  4. Enable the key management service.

For more information about SMF, see Chapter 18, Managing Services (Overview), in System Administration Guide: Basic Administration. Also see the smf(5) and svcadm(1M) man pages.

Starting in this release, the ipsecconf and ipseckey commands have a -c option for checking the syntax of their respective configuration files. Also, the Network IPsec Management rights profile is provided for administering IPsec and IKE.

Solaris 10 7/07: Starting in this release, IPsec fully implements tunnels in tunnel mode, and the utilities that support tunnels are modified.

Solaris 10 1/06: Starting in this release, IKE is fully compliant with NAT-Traversal support as described in RFC 3947 and RFC 3948. IKE operations use the PKCS #11 library from the cryptographic framework, which improves performance.

The cryptographic framework provides a softtoken keystore for applications that use the metaslot. When IKE uses the metaslot, you have the option of storing the keys on disk, on an attached board, or in the softtoken keystore.

Introduction to IPsec

IPsec protects IP packets by authenticating the packets, by encrypting the packets, or by doing both. IPsec is performed inside the IP module, well below the application layer. Therefore, an Internet application can take advantage of IPsec while not having to configure itself to use IPsec. When used properly, IPsec is an effective tool in securing network traffic.

IPsec protection involves five main components:

IPsec applies the security mechanisms to IP datagrams that travel to the IP destination address. The receiver uses information in its SADB to verify that the arriving packets are legitimate and to decrypt them. Applications can invoke IPsec to apply security mechanisms to IP datagrams on a per-socket level as well.

Note that sockets behave differently from ports:

IPsec RFCs

The Internet Engineering Task Force (IETF) has published a number of Requests for Comment (RFCs) that describe the security architecture for the IP layer. All RFCs are copyrighted by the Internet Society. For a link to the RFCs, see The following list of RFCs covers the more general IP security references:

IPsec Terminology

The IPsec RFCs define a number of terms that are useful to recognize when implementing IPsec on your systems. The following table lists IPsec terms, provides their commonly used acronyms, and defines each term. For a list of terminology used in key negotiation, see Table 22–1.

Table 19–1 IPsec Terms, Acronyms, and Uses

IPsec Term 



Security association 


A unique connection between two nodes on a network. The connection is defined by a triplet: a security protocol, a security parameter index, and an IP destination. The IP destination can be an IP address or a socket. 

Security associations database 


Database that contains all active security associations. 

Security parameter index 


The indexing value for a security association. An SPI is a 32-bit value that distinguishes among SAs that have the same IP destination and security protocol. 

Security policy database


Database that determines if outbound packets and inbound packets have the specified level of protection. 

Key exchange 


The process of generating keys for asymmetric cryptographic algorithms. The two main methods are RSA protocols and the Diffie-Hellman protocol. 

Diffie-Hellman protocol 


A key exchange protocol that involves key generation and key authentication. Often called authenticated key exchange.

RSA protocol 


A key exchange protocol that involves key generation and key distribution. The protocol is named for its three creators, Rivest, Shamir, and Adleman. 

Internet Security Association and Key Management Protocol 


The common framework for establishing the format of SA attributes, and for negotiating, modifying, and deleting SAs. ISAKMP is the IETF standard for handling IPsec SAs. 

IPsec Packet Flow

Figure 19–1 shows how an IP addressed packet, as part of an IP datagram, proceeds when IPsec has been invoked on an outbound packet. The flow diagram illustrates where authentication header (AH) and encapsulating security payload (ESP) entities can be applied to the packet. How to apply these entities, as well as how to choose the algorithms, are described in subsequent sections.

Figure 19–2 shows the IPsec inbound process.

Figure 19–1 IPsec Applied to Outbound Packet Process

Flow diagram shows that the outbound packet is first
protected by ESP, and then by AH. The packet then goes to a tunnel or a physical

Figure 19–2 IPsec Applied to Inbound Packet Process

Flow diagram shows that IPsec first processes the AH
header, then the ESP header on inbound packets. A packet that is not protected
enough is dropped.

IPsec Security Associations

An IPsec security association (SA) specifies security properties that are recognized by communicating hosts. A single SA protects data in one direction. The protection is either to a single host or to a group (multicast) address. Because most communication is either peer-to-peer or client-server, two SAs must be present to secure traffic in both directions.

The following three elements uniquely identify an IPsec SA:

The SPI, an arbitrary 32-bit value, is transmitted with an AH or ESP packet. The ipsecah(7P) and ipsecesp(7P) man pages explain the extent of protection that is provided by AH and ESP. An integrity checksum value is used to authenticate a packet. If the authentication fails, the packet is dropped.

Security associations are stored in a security associations database (SADB). A socket-based administration engine, the PF_KEY interface enables privileged applications to manage the database. For example, the IKE application and the ipseckeys command use the PF_KEY socket interface.

Key Management in IPsec

Security associations (SAs) require keying material for authentication and for encryption. The managing of this keying material is called key management. The Internet Key Exchange (IKE) protocol handles key management automatically. You can also manage keys manually with the ipseckey command.

SAs on IPv4 and IPv6 packets can use either method of key management. Unless you have an overriding reason to use manual key management, automatic key management is preferred. For example, to interoperate with systems other than Solaris systems might require manual key management.

In the current release, SMF provides the following key management services for IPsec:

In the releases prior to the Solaris 10 4/09 release, the in.iked and ipseckey commands manage keying material.

IPsec Protection Mechanisms

IPsec provides two security protocols for protecting data:

An AH protects data with an authentication algorithm. An ESP protects data with an encryption algorithm. Optionally, an ESP protects data with an authentication algorithm. Each implementation of an algorithm is called a mechanism.

Authentication Header

The authentication header provides data authentication, strong integrity, and replay protection to IP datagrams. AH protects the greater part of the IP datagram. As the following illustration shows, AH is inserted between the IP header and the transport header.

Diagram shows the AH header between the IP header and
the TCP header.

The transport header can be TCP, UDP, SCTP, or ICMP. If a tunnel is being used, the transport header can be another IP header.

Encapsulating Security Payload

The encapsulating security payload (ESP) module provides confidentiality over what the ESP encapsulates. ESP also provides the services that AH provides. However, ESP only provides its protections over the part of the datagram that ESP encapsulates. ESP provides optional authentication services to ensure the integrity of the protected packet. Because ESP uses encryption-enabling technology, a system that provides ESP can be subject to import and export control laws.

ESP encapsulates its data, so ESP only protects the data that follows its beginning in the datagram, as shown in the following illustration.

Diagram shows the ESP header between the IP header and
the TCP header. The TCP header is encrypted by the ESP header.

In a TCP packet, ESP encapsulates only the TCP header and its data. If the packet is an IP-in-IP datagram, ESP protects the inner IP datagram. Per-socket policy allows self-encapsulation, so ESP can encapsulate IP options when ESP needs to.

If self-encapsulation is set, a copy of the IP header is made to construct an IP-in-IP datagram. For example, when self-encapsulation is not set on a TCP socket, the datagram is sent in the following format:

[ IP(a -> b) options + TCP + data ]

When self-encapsulation is set on that TCP socket, the datagram is sent in the following format:

[ IP(a -> b) + ESP [ IP(a -> b) options + TCP + data ] ]

For further discussion, see Transport and Tunnel Modes in IPsec.

Security Considerations When Using AH and ESP

The following table compares the protections that are provided by AH and ESP.

Table 19–2 Protections Provided by AH and ESP in IPsec


Packet Coverage 


Against Attacks 


Protects packet from the IP header to the transport header 

Provides strong integrity, data authentication: 

  • Ensures that the receiver receives exactly what the sender sent

  • Is susceptible to replay attacks when an AH does not enable replay protection

Replay, cut-and-paste 


Protects packet following the beginning of ESP in the datagram. 

With encryption option, encrypts the IP datagram. Ensures confidentiality 


With authentication option, provides the same protection as AH 

Replay, cut-and-paste 

With both options, provides strong integrity, data authentication, and confidentiality 

Replay, cut-and-paste, eavesdropping 

Authentication and Encryption Algorithms in IPsec

IPsec security protocols use two types of algorithms, authentication and encryption. The AH module uses authentication algorithms. The ESP module can use encryption as well as authentication algorithms. You can obtain a list of the algorithms on your system and their properties by using the ipsecalgs command. For more information, see the ipsecalgs(1M) man page. You can also use the functions that are described in the getipsecalgbyname(3NSL) man page to retrieve the properties of algorithms.

IPsec on a Solaris system uses the Solaris cryptographic framework to access the algorithms. The framework provides a central repository for algorithms, in addition to other services. The framework enables IPsec to take advantage of high performance cryptographic hardware accelerators. The framework also provides resource control features. For example, the framework enables you to limit the amount of CPU time spent in cryptographic operations in the kernel.

For more information, see the following:

Authentication Algorithms in IPsec

Authentication algorithms produce an integrity checksum value or digest that is based on the data and a key. The AH module uses authentication algorithms. The ESP module can use authentication algorithms as well.

Encryption Algorithms in IPsec

Encryption algorithms encrypt data with a key. The ESP module in IPsec uses encryption algorithms. The algorithms operate on data in units of a block size.

Different releases of the Solaris 10 OS provide different default encryption algorithms.

Caution – Caution –

Starting in the Solaris 10 7/07 release, do not add the Solaris Encryption Kit to your system. The kit downgrades the patch level for encryption on your system. The kit is incompatible with the encryption on your system.

IPsec Protection Policies

IPsec protection policies can use any of the security mechanisms. IPsec policies can be applied at the following levels:

IPsec applies the system-wide policy to outbound datagrams and inbound datagrams. Outbound datagrams are either sent with protection or without protection. If protection is applied, the algorithms are either specific or non-specific. You can apply some additional rules to outbound datagrams, because of the additional data that is known by the system. Inbound datagrams can be either accepted or dropped. The decision to drop or accept an inbound datagram is based on several criteria, which sometimes overlap or conflict. Conflicts are resolved by determining which rule is parsed first. The traffic is automatically accepted, except when a policy entry states that traffic should bypass all other policies.

The policy that normally protects a datagram can be bypassed. You can either specify an exception in the system-wide policy, or you can request a bypass in the per-socket policy. For traffic within a system, policies are enforced, but actual security mechanisms are not applied. Instead, the outbound policy on an intra-system packet translates into an inbound packet that has had those mechanisms applied.

You use the ipsecinit.conf file and the ipsecconf command to configure IPsec policies. For details and examples, see the ipsecconf(1M) man page.

Transport and Tunnel Modes in IPsec

The IPsec standards define two distinct modes of IPsec operation, transport mode and tunnel mode. The modes do not affect the encoding of packets. The packets are protected by AH, ESP, or both in each mode. The modes differ in policy application when the inner packet is an IP packet, as follows:

In transport mode, the outer header, the next header, and any ports that the next header supports, can be used to determine IPsec policy. In effect, IPsec can enforce different transport mode policies between two IP addresses to the granularity of a single port. For example, if the next header is TCP, which supports ports, then IPsec policy can be set for a TCP port of the outer IP address. Similarly, if the next header is an IP header, the outer header and the inner IP header can be used to determine IPsec policy.

Tunnel mode works only for IP-in-IP datagrams. Tunneling in tunnel mode can be useful when computer workers at home are connecting to a central computer location. In tunnel mode, IPsec policy is enforced on the contents of the inner IP datagram. Different IPsec policies can be enforced for different inner IP addresses. That is, the inner IP header, its next header, and the ports that the next header supports, can enforce a policy. Unlike transport mode, in tunnel mode the outer IP header does not dictate the policy of its inner IP datagram.

Therefore, in tunnel mode, IPsec policy can be specified for subnets of a LAN behind a router and for ports on those subnets. IPsec policy can also be specified for particular IP addresses, that is, hosts, on those subnets. The ports of those hosts can also have a specific IPsec policy. However, if a dynamic routing protocol is run over a tunnel, do not use subnet selection or address selection because the view of the network topology on the peer network could change. Changes would invalidate the static IPsec policy. For examples of tunneling procedures that include configuring static routes, see Protecting a VPN With IPsec.

In the Solaris OS, tunnel mode can be enforced only on an IP tunneling network interface.The ipsecconf command provides a tunnel keyword to select an IP tunneling network interface. When the tunnel keyword is present in a rule, all selectors that are specified in that rule apply to the inner packet.

In transport mode, ESP, AH, or both, can protect the datagram.

The following figure shows an IP header with an unprotected TCP packet.

Figure 19–3 Unprotected IP Packet Carrying TCP Information

Diagram shows the IP header followed by the TCP header.
The TCP header is not protected.

In transport mode, ESP protects the data as shown in the following figure. The shaded area shows the encrypted part of the packet.

Figure 19–4 Protected IP Packet Carrying TCP Information

Diagram shows the ESP header between the IP header and
the TCP header. The TCP header is encrypted by the ESP header.

In transport mode, AH protects the data as shown in the following figure.

Figure 19–5 Packet Protected by an Authentication Header

Diagram shows the AH header between the IP header and
the TCP header.

AH actually covers the data before the data appears in the datagram. Consequently, the protection that is provided by AH, even in transport mode, covers some of the IP header.

In tunnel mode, the entire datagram is inside the protection of an IPsec header. The datagram in Figure 19–3 is protected in tunnel mode by an outer IPsec header, and in this case ESP, as is shown in the following figure.

Figure 19–6 IPsec Packet Protected in Tunnel Mode

Diagram shows the ESP header after the IP header and
before an IP header and a TCP header. The last 2 headers are protected by

The ipsecconf command includes keywords to set tunnels in tunnel mode or transport mode.

Virtual Private Networks and IPsec

A configured tunnel is a point-to-point interface. The tunnel enables one IP packet to be encapsulated within another IP packet. A correctly configured tunnel requires both a tunnel source and a tunnel destination. For more information, see the tun(7M) man page and Configuring Tunnels for IPv6 Support.

A tunnel creates an apparent physical interface to IP. The physical link's integrity depends on the underlying security protocols. If you set up the security associations (SAs) securely, then you can trust the tunnel. Packets that exit the tunnel must have originated from the peer that was specified in the tunnel destination. If this trust exists, you can use per-interface IP forwarding to create a virtual private network (VPN).

You can use IPsec to construct a VPN. IPsec secures the connection. For example, an organization that uses VPN technology to connect offices with separate networks can deploy IPsec to secure traffic between the two offices.

The following figure illustrates how two offices use the Internet to form their VPN with IPsec deployed on their network systems.

Figure 19–7 Virtual Private Network

Diagram shows that Offices 1 and 2 use the hme0 interface
to communicate with each other. Each office uses hme1 for internal communication.

For a detailed example of the setup procedure, see How to Protect a VPN With an IPsec Tunnel in Tunnel Mode Over IPv4.

For a similar example with IPv6 addresses, see How to Protect a VPN With an IPsec Tunnel in Tunnel Mode Over IPv6.

IPsec and NAT Traversal

IKE can negotiate IPsec SAs across a NAT box. This ability enables systems to securely connect from a remote network, even when the systems are behind a NAT device. For example, employees who work from home, or who log on from a conference site can protect their traffic with IPsec.

NAT stands for network address translation. A NAT box is used to translate a private internal address into a unique Internet address. NATs are very common at public access points to the Internet, such as hotels. For a fuller discussion, see Using Oracle Solaris IP Filter's NAT Feature.

The ability to use IKE when a NAT box is between communicating systems is called NAT traversal, or NAT-T. In the Solaris 10 release, NAT-T has the following limitations:

The following RFCs describe NAT functionality and the limits of NAT-T. Copies of the RFCs can be retrieved from

To use IPsec across a NAT, see Configuring IKE for Mobile Systems (Task Map).

IPsec and SCTP

The Solaris OS supports the Streams Control Transmission Protocol (SCTP). The use of the SCTP protocol and SCTP port number to specify IPsec policy is supported, but is not robust. The IPsec extensions for SCTP as specified in RFC 3554 are not yet implemented. These limitations can create complications in creating IPsec policy for SCTP.

SCTP can make use of multiple source and destination addresses in the context of a single SCTP association. When IPsec policy is applied to a single source or a single destination address, communication can fail when SCTP switches the source or the destination address of that association. IPsec policy only recognizes the original address. For information about SCTP, read the RFCs and SCTP Protocol.

IPsec and Solaris Zones

For shared-IP zones, IPsec is configured from the global zone. The IPsec policy configuration file, ipsecinit.conf, exists in the global zone only. The file can have entries that apply to non-global zones, as well as entries that apply to the global zone.

For exclusive-IP zones, IPsec is configured in the non-global zone.

For information about how to use IPsec with zones, see Protecting Traffic With IPsec. For information about zones, see Chapter 16, Introduction to Solaris Zones, in System Administration Guide: Oracle Solaris Containers-Resource Management and Oracle Solaris Zones.

IPsec and Logical Domains

IPsec works with logical domains. The logical domain must be running a version of the Solaris OS that includes IPsec, such as the Solaris 10 release.

To create logical domains, you must use the Oracle VM Server for SPARC, which was previously called Logical Domains. For information about how to configure logical domains, see Logical Domains 1.2 Administration Guide or Oracle VM Server for SPARC 2.0 Administration Guide.

IPsec Utilities and Files

Table 19–3 describes the files, commands, and service identifiers that are used to configure and manage IPsec. For completeness, the table includes key management files, socket interfaces, and commands.

Starting in the Solaris 10 4/09 release, IPsec is managed by SMF. For more information about service identifiers, see Chapter 18, Managing Services (Overview), in System Administration Guide: Basic Administration.

Table 19–3 List of Selected IPsec Utilities and Files

IPsec Utility, File, or Service 


Man Page 


In the current release, the SMF service that manages IPsec algorithms. 

smf(5), ipsecalgs(1M)


In the current release, the SMF service that manages manual security associations (SAs). 

smf(5), ipseckey(1M)


In the current release, the SMF service that manages IPsec policy.

smf(5), ipsecconf(1M)


In the current release, the SMF service for the automatic management of IPsec SAs. 

smf(5), in.iked(1M)

/etc/inet/ipsecinit.conf file

IPsec policy file. In releases prior to the Solaris 10 4/09 release, if this file exists, IPsec is activated at boot time.

In the current release, the SMF policy service uses this file to configure IPsec policy at system boot.


ipsecconf command

IPsec policy command. Useful for viewing and modifying the current IPsec policy, and for testing. In releases prior to the Solaris 10 4/09 release, the boot scripts use ipsecconf to read the /etc/inet/ipsecinit.conf file and activate IPsec.

In the current release, ipsecconf is used by the SMF policy service to configure IPsec policy at system boot.


PF_KEY socket interface

Interface for the security associations database (SADB). Handles manual key management and automatic key management.


ipseckey command

IPsec SAs keying command. ipseckey is a command-line front end to the PF_KEY interface. ipseckey can create, destroy, or modify SAs.


/etc/inet/secret/ipseckeys file

Keys for IPsec SAs. In releases prior to the Solaris 10 4/09 release, if the ipsecinit.conf file exists, the ipseckeys file is automatically read at boot time.

In the current release, ipseckeys is used by the SMF manual-key service to configure SAs manually at system boot.


ipsecalgs command

IPsec algorithms command. Useful for viewing and modifying the list of IPsec algorithms and their properties.

In the current release, is used by the SMF ipsecalgs service to synchronize known IPsec algorithms with the kernel at system boot.


/etc/inet/ipsecalgs file

Contains the configured IPsec protocols and algorithm definitions. This file is managed by the ipsecalgs command and must never be edited manually.


/etc/inet/ike/config file

IKE configuration and policy file. By default, this file does not exist. In releases prior to the Solaris 10 4/09 release, if this file exists, the IKE daemon, in.iked, provides automatic key management. The management is based on rules and global parameters in the /etc/inet/ike/config file. See IKE Utilities and Files.

In the current release, if this file exists, the svc:/network/ipsec/ike service starts the IKE daemon, in.iked, to provide automatic key management.


Changes to IPsec for the Solaris 10 Release

For a complete listing of new Solaris features and a description of Solaris releases, see Oracle Solaris 10 9/10 What’s New. Since the Solaris 9 release, IPsec includes the following functionality: