Chapter 1 IP Security Architecture (Overview)

The IP security architecture (IPsec) provides cryptographic protection for IP datagrams in IPv4 and IPv6 network packets. This protection can include confidentiality, strong integrity of the data, data authentication, and partial sequence integrity. Partial sequence integrity is also known as replay protection.

IPsec is performed inside the IP module. IPsec can be applied with or without the knowledge of an Internet application. When used properly, IPsec is an effective tool in securing network traffic.

This chapter contains the following information:

• Introduction to IPsec

• IPsec Security Associations

• Protection Mechanisms

• Protection Policy and Enforcement Mechanisms

• Transport and Tunnel Modes

• Virtual Private Networks

• IPsec Utilities and Files

For instructions on implementing IPsec on your network, see Chapter 2, Administering IPsec (Tasks).

Introduction to IPsec

IPsec provides security mechanisms that include secure datagram authentication and encryption mechanisms within IP. When you invoke IPsec, IPsec applies the security mechanisms to IP datagrams that you have enabled in the IPsec global policy file. Applications can invoke IPsec to apply security mechanisms to IP datagrams on a per-socket level.

Figure 1–1 shows how an IP addressed packet, as part of an IP datagram, proceeds when IPsec has been invoked on an outbound packet. As you can see from the flow diagram, authentication header (AH) and encapsulating security payload (ESP) entities can be applied to the packet. Subsequent sections describe how you apply these entities, as well as authentication and encryption algorithms.

Figure 1–1 IPsec Applied to Outbound Packet Process

Figure 1–2 shows the IPsec inbound process.

Figure 1–2 IPsec Applied to Inbound Packet Process

IPsec Security Associations

An IPsec security association (SA) specifies security properties that are recognized by communicating hosts. These hosts typically require two SAs to communicate securely. A single SA protects data in one direction. The protection is either to a single host or a group (multicast) address. Because most communication is peer-to-peer or client-to-server, two SAs must be present to secure traffic in both directions.

The security protocol (AH or ESP), destination IP address, and security parameter index (SPI) identify an IPsec SA. The SPI, an arbitrary 32-bit value, is transmitted with an AH or ESP packet. The ipsecah(7P) and ipsecesp(7P) man pages explain the extent of protection that is provided by AH and ESP. An integrity checksum value is used to authenticate a packet. If the authentication fails, the packet is dropped.

Security associations are stored in a security associations database. A socket-based administration engine, the pf_key interface, enables privileged applications to manage the database. The in.iked daemon provides automatic key management. See the pf_key(7P) and in.iked(1M) man pages.

Key Management

A security association contains the following information:

• Material for keys for encryption and authentication

• The algorithms that can be used

• The identities of the endpoints

• Other parameters that are used by the system

SAs require keying material for authentication and encryption. The managing of keying material that SAs require is called key management. The Internet Key Exchange (IKE) protocol handles key management automatically. You can also manage keys manually with the ipseckey command. SAs on IPv4 and IPv6 packets can use automatic key management.

See IKE Overview, for how IKE manages cryptographic keys automatically. See Keying Utilities, for how you can manually manage the cryptographic keys by using the ipseckey command. The ipseckey(1M) man page provides a detailed description of the command options.

Protection Mechanisms

IPsec provides two mechanisms for protecting data:

• Authentication Header (AH)

• Encapsulating Security Payload (ESP)

Both mechanisms have their own Security Association Database (SADB).

Authentication Header

The authentication header provides data authentication, strong integrity, and replay protection to IP datagrams. AH protects the greater part of the IP datagram. AH cannot protect fields that change nondeterministically between sender and receiver. For example, the IP TTL field is not a predictable field and, consequently, not protected by AH. AH is inserted between the IP header and the transport header. The transport header can be TCP, UDP, ICMP, or another IP header when tunnels are being used. See the tun(7M) man page for details on tunneling.

Authentication Algorithms and the AH Module

IPsec implements AH as a module that is automatically pushed on top of IP. The /dev/ipsecah entry tunes AH with the ndd command. Future authentication algorithms can be loaded on top of AH. Current authentication algorithms include HMAC-MD5 and HMAC-SHA-1. Each authentication algorithm has its own key size and key format properties. See the authmd5h(7M) and authsha1(7M) man pages for details. For tuning IP configuration parameters, see the ndd(1M) man page.

Security Considerations for AH

Replay attacks threaten an AH when an AH does not enable replay protection. An AH does not protect against eavesdropping. Adversaries can still see data that is protected with AH.

Encapsulating Security Payload

The encapsulating security payload (ESP) header provides confidentiality over what the ESP encapsulates, as well as the services that AH provides. However, ESP only provides its protections over the part of the datagram that ESP encapsulates. ESP's authentication services are optional. These services enable you to use ESP and AH together on the same datagram without redundancy. Because ESP uses encryption-enabling technology, ESP must conform to U.S. export control laws.

ESP encapsulates its data, so ESP only protects the data that follows its beginning in the datagram. In a TCP packet, ESP encapsulates only the TCP header and its data. If the packet is an IP-in-IP datagram, ESP protects the inner IP datagram. Per-socket policy allows self-encapsulation, so ESP can encapsulate IP options when ESP needs to. Unlike the authentication header (AH), ESP allows multiple kinds of datagram protection. Using only a single form of datagram protection can make the datagram vulnerable. For example, if you use ESP to provide confidentiality only, the datagram is still vulnerable to replay attacks and cut-and-paste attacks. Similarly, if ESP protects only integrity, ESP could provide weaker protection than AH. The datagram would be vulnerable to eavesdropping.

Algorithms and the ESP Module

IPsec ESP implements ESP as a module that is automatically pushed on top of IP. The /dev/ipsecesp entry tunes ESP with the ndd command. ESP allows encryption algorithms to be pushed on top of ESP, in addition to the authentication algorithms that are used in AH. Encryption algorithms include Data Encryption Standard (DES), Triple-DES (3DES), Blowfish, and AES. Each encryption algorithm has its own key size and key format properties. Because of export laws in the United States and import laws in other countries, not all encryption algorithms are available outside of the United States. For tuning IP configuration parameters, see the ndd(1M) man page.

Security Considerations for ESP

An ESP without authentication is vulnerable to cut-and-paste cryptographic attacks and to replay attacks. When you use ESP without confidentiality, ESP is as vulnerable to eavesdropping as AH is.

Authentication and Encryption Algorithms

IPsec uses two types of algorithms, authentication and encryption. The authentication algorithms and the DES encryption algorithms are part of core Solaris installation. If you plan to use other algorithms that are supported for IPsec, you must install the Solaris Encryption Kit. The Solaris Encryption Kit is provided on a separate CD.

Authentication Algorithms

Authentication algorithms produce an integrity checksum value or digest that is based on the data and a key. The man pages for authentication algorithms describe the size of both the digest and key. The following table lists the authentication algorithms that are supported in the Solaris operating environment. The table also lists the format of the algorithms when the algorithms are used as security options to the IPsec utilities and their man page names.

Encryption Algorithms

Encryption algorithms encrypt data with a key. The algorithms operate on data in units of a block size. The man pages for encryption algorithms describe the block size and the key size for each algorithm. By default, the DES–CBC and 3DES-CBC algorithms are installed.

The AES and Blowfish algorithms are available to IPsec when you install the Solaris Encryption Kit. The kit is available on a separate CD that is not part of the Solaris 9 installation box. The Solaris 9 Encryption Kit Installation Guide describes how to install the Solaris Encryption Kit.

The following table lists the encryption algorithms that are supported in the Solaris operating environment. The table lists the format of the algorithms when the algorithms are used as security options to the IPsec utilities. The table also lists their man page names, and lists the package that contains the algorithm.

Protection Policy and Enforcement Mechanisms

IPsec separates its protection policy from its enforcement mechanisms. You can enforce IPsec policies in the following places:

• On a system-wide level

• On a per-socket level

You use the ipsecconf command to configure the system-wide policy. See the ipsecconf(1M) man page.

IPsec applies the system-wide policy to incoming datagrams and outgoing datagrams. You can apply some additional rules to outgoing datagrams, because of the additional data that is known by the system. Inbound datagrams can be either accepted or dropped. The decision to drop or accept an inbound datagram is based on several criteria, which sometimes overlap or conflict. Conflicts are resolved by determining which rule is parsed first. Except when a policy entry states that traffic should bypass all other policy, the traffic is automatically accepted. Outbound datagrams are either sent with protection or without protection. If protection is applied, the algorithms are either specific or non-specific.

The policy that normally protects a datagram can be bypassed. You can either specify an exception in the system-wide policy, or you can request a bypass in the per-socket policy. For intra-system traffic, policies are enforced, but actual security mechanisms are not applied. Instead, the outbound policy on an intra-system packet translates into an inbound packet that has had those mechanisms applied.

Transport and Tunnel Modes

When you invoke ESP or AH after the IP header to protect a datagram, you are using transport mode. An example follows. A packet starts off with the following header:

ESP, in transport mode, protects the data as follows:

AH, in transport mode, protects the data as follows:

AH actually covers the data before the data appears in the datagram. Consequently, the protection that is provided by AH, even in transport mode, covers some of the IP header.

When an entire datagram is inside the protection of an IPsec header, IPsec is protecting the datagram in tunnel mode. Because AH covers most of its preceding IP header, tunnel mode is usually performed only on ESP. The previous example datagram would be protected in tunnel mode as follows:

In tunnel mode, the inner header is protected, while the outer IP header is unprotected. Often, the outer IP header has different source and different destination addresses from the inner IP header. The inner and outer IP headers can match if, for example, an IPsec-aware network program uses self-encapsulation with ESP. Self-encapsulation with ESP protects an IP header option.

The Solaris implementation of IPsec is primarily an implementation of IPsec in transport mode. Tunnel mode is implemented as a special instance of the transport mode. The implementation treats IP-in-IP tunnels as a special transport provider. The ifconfig configuration options to set tunnels are nearly identical to the options that are available to socket programmers when enabling per-socket IPsec. Also, tunnel mode can be enabled in per-socket IPsec. In per-socket tunnel mode, the inner packet IP header has the same addresses as the outer IP header. For details on per-socket policy, see the ipsec(7P) man page. For configuring tunnels, see the ifconfig(1M) man page.

Trusted Tunnels

A configured tunnel is a point-to-point interface. The tunnel enables an IP packet to be encapsulated within an IP packet. A correctly configured tunnel requires both a tunnel source and a tunnel destination. For more information, see the tun(7M) man page and “Solaris Tunneling Interfaces for IPv6” in System Administration Guide: IP Services.

A tunnel creates an apparent physical interface to IP. The physical link's integrity depends on the underlying security protocols. If you set up the security associations securely, then you can trust the tunnel. Packets that exit the tunnel must have originated from the peer that was specified in the tunnel destination. If this trust exists, you can use per-interface IP forwarding to create a virtual private network.

Virtual Private Networks

You can use IPsec to construct a virtual private network (VPN). You use IPsec by constructing an Intranet that uses the Internet infrastructure. For example, an organization that uses VPN technology to connect offices with separate networks, can deploy IPsec to secure traffic between the two offices.

The following figure illustrates how two offices use the Internet to form their VPN with IPsec deployed on their network systems.

Figure 1–3 Virtual Private Network

See How to Set Up a Virtual Private Network (VPN) for a description of the setup procedure.

IPsec Utilities and Files

This section describes the configuration file that initializes IPsec. This section also describes various commands that enable you to manage IPsec within your network. For instructions about how to implement IPsec within your network, see Implementing IPsec (Task Map).

IPsec Policy Command

You use the ipsecconf command to configure the IPsec policy for a host. When you run the command to configure the policy, the system creates a temporary file that is named ipsecpolicy.conf. This file holds the IPsec policy entries that were set in the kernel by the ipsecconf command. The system uses the in-kernel IPsec policy entries to check all outbound and inbound IP datagrams for policy. Forwarded datagrams are not subjected to policy checks that are added by using this command. For information on how to protect forwarded packets, see the ifconfig(1M) and tun(7M) man pages. For IPsec policy options, see the ipsecconf(1M) man page.

You must become superuser or assume an equivalent role to invoke the ipsecconf command. The command accepts entries that protect traffic in both directions, and entries that protect traffic in only one direction.

Policy entries with a format of local address and remote address can protect traffic in both directions with a single policy entry. For example, entries that contain the patterns laddr host1 and raddr host2, protect traffic in both directions if no direction is specified for the named host. Thus, you need only one policy entry for each host. Policy entries with a format of source address to destination address protect traffic in only one direction. For example, a policy entry of the pattern saddr host1 daddr host2 protects inbound traffic or outbound traffic, not both directions. Thus, to protect traffic in both directions, you need to pass the ipsecconf command another entry, as in saddr host2 daddr host1.

You can see the policies that are configured in the system when you issue the ipsecconf command without any arguments. The command displays each entry with an index followed by a number. You can use the -d option with the index to delete a particular policy in the system. The command displays the entries in the order that the entries were added, which is not necessarily the order in which the traffic match occurs. To view the order in which the traffic match occurs, use the -l option.

The ipsecpolicy.conf file is deleted when the system shuts down. To ensure that the IPsec policy is active when the machine boots, you can create an IPsec policy file, /etc/inet/ipsecinit.conf, that the inetinit script reads during startup.

IPsec Policy File

To invoke IPsec security policies when you start the Solaris operating environment, you create a configuration file to initialize IPsec with your specific IPsec policy entries. You should name the file /etc/inet/ipsecinit.conf. See the ipsecconf(1M) man page for details about policy entries and their format. After policies are configured, you can use the ipsecconf command to delete a policy temporarily, or to view the existing configuration.

Example—ipsecinit.conf File

The Solaris software includes an IPsec policy file as a sample. This sample file is named ipsecinit.sample. You can use the file as a template to create your own ipsecinit.conf file. The ipsecinit.sample file contains the following examples:

#
# For example,
#
#	 {rport 23} ipsec {encr_algs des encr_auth_algs md5}
#
# will protect the telnet traffic originating from the host with ESP using
# DES and MD5. Also:
#
#	 {raddr 10.5.5.0/24} ipsec {auth_algs any}
#
# will protect traffic to or from the 10.5.5.0 subnet with AH 
# using any available algorithm.
#
#
# To do basic filtering, a drop rule may be used. For example:
#
#    {lport 23 dir in} drop {}
#    {lport 23 dir out} drop {}
#
# will disallow any remote system from telnetting in.

Security Considerations for ipsecinit.conf and ipsecconf

If, for example, the /etc/inet/ipsecinit.conf file is sent from an NFS-mounted file system, an adversary can modify the data contained in the file. The outcome would be a change to the configured policy. Consequently, you should use extreme caution if transmitting a copy of the ipsecinit.conf file over a network.

The policy cannot be changed for TCP sockets or UDP sockets on which a connect() or accept() function call has been issued. A socket whose policy cannot be changed is called a latched socket. New policy entries do not protect sockets that are already latched. See the connect(3SOCKET) and accept(3SOCKET) man pages.

Ensure that you set up the policies before starting any communications, because existing connections might be affected by the addition of new policy entries. Similarly, do not change policies in the middle of a communication.

Protect your naming system. If the following two conditions are met, then your host names are no longer trustworthy:

• Your source address is a host that can be looked up over the network

• Your naming system is compromised

Security weaknesses often lie in misapplication of tools, not the actual tools. You should be cautious when using the ipsecconf command. Use a console or other hard-connected TTY for the safest mode of operation.

Security Associations Database for IPsec

Information on keying material for IPsec security services is maintained in a security association database (SADB). Security associations protect both inbound packets and outbound packets. A user process, or possibly multiple cooperating processes, maintains SADBs by sending messages over a special kind of socket. This method of maintaining SADBs is analogous to the method that is described in the route(7P) man page. Only a superuser or someone who has assumed an equivalent role can access an SADB.

The operating system might spontaneously emit messages in response to external events. For example, the system might request for a new SA for an outbound datagram, or the system might report the expiration of an existing SA. You open the channel for passing SADB control messages by using the socket call that is mentioned in the previous section. More than one key socket can be open per system.

Messages include a small base header, followed by a number of extension messages. The number of messages might be zero or more. Some messages require additional data. The base message and all extensions must be 8-byte aligned. The GET message serves as an example. This message requires the base header, the SA extension, and the ADDRESS_DST extension. See the pf_key(7P) man page for details.

Keying Utilities

The IKE protocol is the automatic keying utility for IPv4 and IPv6 addresses. See Chapter 4, Administering IKE (Tasks) for how to set up IKE. The manual keying utility is the ipseckey command. See the ipseckey(1M) man page.

You use the ipseckey command to manually manipulate the security association databases with the ipsecah and ipsecesp protection mechanisms. You can also use the ipseckey command to set up security associations between communicating parties when automated key management is not used.

While the ipseckey command has only a limited number of general options, the command supports a rich command language. You can specify that requests should be delivered by means of a programmatic interface specific for manual keying. See the pf_key(7P) man page for additional information. When you invoke the ipseckey command with no arguments, the command enters an interactive mode that displays a prompt that enables you to make entries. Some commands require an explicit security association (SA) type, while others permit you to specify the SA type and act on all SA types.

Security Considerations for ipseckey

The ipseckey command enables a privileged user to enter sensitive cryptographic keying information. If an adversary gains access to this information, the adversary can compromise the security of IPsec traffic. You should consider the following issues when you handle keying material and use the ipseckey command:

1. Have you refreshed the keying material? Periodic key refreshment is a fundamental security practice. Key refreshment guards against potential weaknesses of the algorithm and keys, and limits the damage of an exposed key.

2. Is the TTY going over a network? Is the ipseckey command in interactive mode?

   • In interactive mode, the security of the keying material is the security of the network path for this TTY's traffic. You should avoid using the ipseckey command over a clear-text telnet or rlogin session.

   • Even local windows might be vulnerable to attacks by a concealed program that reads window events.

3. Is the file being accessed over the network? Can the file be read by the world? Have you used the -f option?

   • An adversary can read a network-mounted file as the file is being read. You should avoid using a world-readable file that contains keying material.

   • Protect your naming system. If the following two conditions are met, then your host names are no longer trustworthy:

     • Your source address is a host that can be looked up over the network

     • Your naming system is compromised

Security weaknesses often lie in misapplication of tools, not the actual tools. You should be cautious when using the ipseckey command. Use a console or other hard-connected TTY for the safest mode of operation.

IPsec Extensions to Other Utilities

The ifconfig command has options to manage the IPsec policy on a tunnel interface. The snoop command can parse AH and ESP headers.

ifconfig Command

To support IPsec, the following security options have been added to the ifconfig command:

• auth_algs

• encr_auth_algs

• encr_algs

You must specify all IPsec security options for a tunnel in one invocation. For example, if you are using only ESP to protect traffic, you would configure the tunnel, ip.tun0, once with both security options, as in:

# ifconfig ip.tun0 … encr_algs 3DES encr_auth_algs MD5

Similarly, an ipsecinit.conf entry would configure the tunnel once with both security options, as in:

# WAN traffic uses ESP with 3DES and MD5.
   {} ipsec {encr_algs 3des encr_auth_algs md5}

auth_algs Security Option

This option enables IPsec AH for a tunnel with a specified authentication algorithm. The auth_algs option has the following format:

auth_algs authentication-algorithm

For the algorithm, you can specify either a number or an algorithm name, including the parameter any, to express no specific algorithm preference. To disable tunnel security, specify the following option:

auth_algs none

See Table 1–1 for a list of available authentication algorithms and for pointers to the algorithm man pages.

encr_auth_algs Security Option

This option enables IPsec ESP for a tunnel with a specified authentication algorithm. The encr_auth_algs option has the following format:

encr_auth_algs authentication-algorithm

For the algorithm, you can specify either a number or an algorithm name, including the parameter any, to express no specific algorithm preference. If you specify an ESP encryption algorithm, but you do not specify the authentication algorithm, the ESP authentication algorithm value defaults to the parameter any.

See Table 1–1 for a list of available authentication algorithms and for pointers to the algorithm man pages.

encr_algs Security Option

This option enables IPsec ESP for a tunnel with a specified encryption algorithm. The encr_algs option has the following format:

encr_algs encryption-algorithm

For the algorithm, you can specify either a number or an algorithm name. To disable tunnel security, specify the following option:

encr_algs none

If you specify an ESP authentication algorithm, but not an encryption algorithm, ESP's encryption value defaults to the parameter null.

For a list of available encryption algorithms and for pointers to the algorithm man pages, see the ipsecesp(7P) man page or Table 1–2.

snoop Command

The snoop command can now parse AH and ESP headers. Because ESP encrypts its data, the snoop command cannot see encrypted headers that are protected by ESP. AH does not encrypt data, so traffic can still be inspected with this command. The snoop -V option shows when AH is in use on a packet. See the snoop(1M) man page for more details.

For a sample of verbose snoop output on a protected packet, see How to Verify That Packets Are Protected.