Chapter 19 IPsec (Overview)

The IP Security Architecture (IPsec) provides cryptographic protection for IP datagrams in IPv4 and IPv6 network packets. The protection can include confidentiality, strong integrity of the data, partial sequence integrity (replay protection), and data authentication. IPsec is performed inside the IP module, and it can be applied with or without the knowledge of an Internet application. When used properly, IPsec is an effective tool in securing network traffic.

This chapter contains the following information:

• Introduction to IPsec

• IPsec Security Associations

• Protection Mechanisms

• Protection Policy and Enforcement Mechanisms

• Transport and Tunnel Modes

• Virtual Private Networks

• IPsec Utilities and Files

Introduction to IPsec

IPsec provides security mechanisms that include secure datagram authentication and encryption mechanisms within IP. When you invoke IPsec, it applies the security mechanisms to IP datagrams that you have enabled in the IPsec global policy file. Applications can invoke IPsec to apply security mechanisms to IP datagrams on a per-socket level.

Figure 19–1 shows how an IP addressed packet, as part of an IP datagram, proceeds when IPsec has been invoked on an outbound packet. As you can see from the flow diagram, authentication header (AH) and encapsulating security payload (ESP) entities can be applied to the packet. Subsequent sections describe how you apply these entities, as well as authentication and encryption algorithms.

Figure 19–1 IPsec Applied to Outbound Packet Process

Figure 19–2 shows the IPsec inbound process.

Figure 19–2 IPsec Applied to Inbound Packet Process

IPsec Security Associations

An IPsec security associations (SA) specifies security properties that are recognized by communicating hosts. Communicating hosts typically require two SAs to communicate securely. A single SA protects data in one direction — either to a single host or a group (multicast) address. Because most communication is peer-to-peer or client-to-server, two SAs must be present to secure traffic in both directions.

The AH or ESP, destination IP address, and Security Parameter Index (SPI) identifies an IPsec SA. The security parameters index, an arbitrary 32-bit value, is transmitted with an AH or ESP packet. The ipsecah(7P) and ipsecesp(7P) man pages explain the extent of protection that is provided by AH and ESP. An integrity checksum value is used to authenticate a packet. If the authentication fails, the packet is dropped.

Security associations are stored in a security associations database. A socket-based administration engine, the pf_key(7P) interface, enables privileged applications to manage the database. The in.iked(1M)in.iked(1M) daemon provides automatic key management.

Key Management

A security association contains keying information, algorithm choices, endpoint identities, and other parameters. Managing the keying material that SAs require for authentication and for encryption is called key management. The Internet Key Exchange (IKE) protocol handles key management automatically. You can also manage keys manually with the ipseckey(1M) command. Currently, SAs on IPv4 packets can use automatic key management, while SAs on IPv6 packets require manual management.

See IKE Overview for how IKE manages cryptographic keys automatically for IPv4 hosts. See Keying Utilities for how the administrator can manually manage the cryptographic keys by using the ipseckey command.

Protection Mechanisms

IPsec provides two mechanisms for protecting data:

• Authentication Header (AH)

• Encapsulating Security Payload (ESP)

Both mechanisms use security associations.

Authentication Header

The authentication header, a new IP header, provides strong integrity, partial sequence integrity (replay protection), and data authentication to IP datagrams. AH protects as much of the IP datagram as it can. AH cannot protect fields that change nondeterministically between sender and receiver. For example, the IP TTL field is not a predictable field and, consequently, not protected by AH. AH is inserted between the IP header and the transport header. The transport header can be TCP, UDP, ICMP, or another IP header when tunnels are being used. See the tun(7M) man page for details on tunneling.

Authentication Algorithms and the AH Module

IPsec implements AH as a module that is automatically pushed on top of IP. The /dev/ipsecah entry tunes AH with ndd(1M). Future authentication algorithms can be loaded on top of AH. Current authentication algorithms include HMAC-MD5 and HMAC-SHA-1. Each authentication algorithm has its own key size and key format properties. See the authmd5h(7M) and authsha1(7M) man pages for details.

Security Considerations

Replay attacks threaten any AH that does not enable replay protection. An AH does not protect against eavesdropping. Adversaries can still see data protected with AH.

Encapsulating Security Payload

The ESP provides confidentiality over what it encapsulates, as well as the services that AH provides, but only over that which it encapsulates. ESP's authentication services are optional. These services enable you to use ESP and AH together on the same datagram without redundancy. Because ESP uses encryption-enabling technology, it must conform to U.S. export control laws.

ESP encapsulates its data, so it only protects the data that follows its beginning in the datagram. In a TCP packet, ESP encapsulates only the TCP header and its data. If the packet is an IP-in-IP datagram, ESP protects the inner IP datagram. Per-socket policy allows self-encapsulation, so ESP can encapsulate IP options when it needs to. Unlike the authentication header (AH), ESP allows multiple kinds of datagram protection. Using only a single form of datagram protection can make the datagram vulnerable. For example, if you use ESP to provide confidentiality only, the datagram is still vulnerable to replay attacks and cut-and-paste attacks. Similarly, if ESP protects only integrity and does not fully protect against eavesdropping, it could provide weaker protection than AH.

Algorithms and the ESP Module

IPsec ESP implements ESP as a module that is automatically pushed on top of IP. The /dev/ipsecesp entry tunes ESP with ndd(1M). ESP allows encryption algorithms to be pushed on top of it, in addition to the authentication algorithms that are used in AH. Encryption algorithms include United States Data Encryption Standard (DES), Triple-DES (3DES), Blowfish, and AES. Each encryption algorithm has its own key size and key format properties. Because of export laws in the United States and import laws in other countries, not all encryption algorithms are available outside of the United States.

Security Considerations

An ESP without authentication is vulnerable to cut-and-paste cryptographic attacks and to eavesdropping attacks. When you use ESP without confidentiality, ESP is as vulnerable to replay as AH is.

Authentication and Encryption Algorithms

IPsec uses two types of algorithms, authentication and encryption. The authentication algorithms and the DES encryption algorithms are part of core Solaris installation. If you plan to use other algorithms that are supported for IPsec, you must install the Solaris Encryption Kit, which is provided on a separate CD.

Authentication Algorithms

Authentication algorithms produce an integrity checksum value or digest that is based on the data and a key. The authentication algorithm man pages describe the size of both the digest and key. The following table lists the authentication algorithms that are supported in the Solaris operating environment. The table also lists the format of the algorithms when they are used as security options to the IPsec utilities and their man page names.

Encryption Algorithms

Encryption algorithms encrypt data with a key. The algorithms operate on data in units of a block size. The encryption algorithm man pages describe the size of both the block size and the key size. By default, the DES-CBC and 3DES-CBC algorithms are installed. You must install the Solaris Encryption Kit to make the AES and Blowfish algorithms available to IPsec. The kit is available on a separate CD that is not part of the Solaris 9 installation box. The Encryption Kit Installation Guide describes how to install the Solaris Encryption Kit.

The following table lists the encryption algorithms that are supported in the Solaris operating environment. The table also lists the format of the algorithms when they are used as security options to the IPsec utilities, their man page names, and the package that contains them.

Protection Policy and Enforcement Mechanisms

IPsec separates protection policy and enforcement mechanisms. You can enforce IPsec policies in the following places:

• On a system-wide level

• On a per-socket level

You use the ipsecconf(1M) command to configure system-wide policy.

IPsec applies system-wide policy to incoming and outgoing datagrams. You can apply some additional rules to outgoing datagrams, because of the additional data that is known by the system. Inbound datagrams can be either accepted or dropped. The decision to drop or accept an inbound datagram is based on several criteria, which sometimes overlap or conflict. Resolving a conflict depends on which rule is parsed first. Except when a policy entry states that traffic should bypass all other policy, the traffic is automatically accepted. Outbound datagrams are either sent with protection or without protection. If protection is applied, the algorithms are either specific or non-specific. If policy normally protects a datagram, it can be bypassed by either an exception in system-wide policy, or by requesting a bypass in per-socket policy.

For intra-system traffic, policies are enforced, but actual security mechanisms are not applied. Instead, the outbound policy on an intra-system packet translates into an inbound packet that has had those mechanisms applied.

Transport and Tunnel Modes

When you invoke ESP or AH after the IP header to protect a datagram, you are using transport mode. An example follows. A packet starts off as:

ESP, in transport mode, protects the data as follows:

AH, in transport mode, protects the data as follows:

AH actually covers the data before it appears in the datagram. Consequently, the protection that is provided by AH, even in transport mode, covers some of the IP header.

When an entire datagram is inside the protection of an IPsec header, IPsec is protecting the datagram in tunnel mode. Because AH covers most of its preceding IP header, tunnel mode is usually performed only on ESP. The previous example datagram would be protected in tunnel mode as follows:

Often, in tunnel mode, the outer (unprotected) IP header has different source and destination addresses from the inner (protected) IP header. The inner and outer IP headers can match if, for example, an IPsec-aware network program uses self-encapsulation with ESP. Self-encapsulation with ESP protects an IP header option.

The Solaris implementation of IPsec is primarily a transport mode IPsec implementation, which implements the tunnel mode as a special instance of the transport mode. The implementation treats IP-in-IP tunnels as a special transport provider. The ifconfig(1M) configuration options to set tunnels are nearly identical to the options available to socket programmers when enabling per-socket IPsec. Also, tunnel mode can be enabled in per-socket IPsec. In per-socket tunnel mode, the inner packet IP header has the same addresses as the outer IP header. See the ipsec(7P) man page for details on per-socket policy.

Trusted Tunnels

A configured tunnel is a point-to-point interface. It enables an IP packet to be encapsulated within an IP packet. Configuring a tunnel requires both a tunnel source and tunnel destination. See the tun(7M) man page and Solaris Tunneling Interfaces for IPv6 for more information.

A tunnel creates an apparent physical interface to IP. The physical link's integrity depends on the underlying security protocols. If you set up the security associations securely, then you can trust the tunnel. That is, packets that exit the tunnel originated from the peer that was specified in the tunnel destination. If this trust exists, you can use per-interface IP forwarding to create a virtual private network.

Virtual Private Networks

You can use IPsec to construct a Virtual Private Network (VPN). You do this by constructing an Intranet that uses the Internet infrastructure. For example, an organization that uses VPN technology to connect offices with separate networks, can deploy IPsec to secure traffic between the two offices.

The following figure illustrates how two offices use the Internet to form their VPN with IPsec deployed on their network systems.

Figure 19–3 Virtual Private Network

See How to Set Up a Virtual Private Network for a description of the setup procedure.

IPsec Utilities and Files

This section describes the IPsec initialization configuration file and various commands that enable you to manage IPsec within your network. For instructions about how to implement IPsec within your network, see Implementing IPsec Task Map.

IPsec Policy Command

You use the ipsecconf(1M) command to configure the IPsec policy for a host. When you run the command to configure policy, the system creates a temporary file named ipsecpolicy.conf to hold the IPsec policy entries. The system immediately uses the file to check all outbound and inbound IP datagrams for policy. Forwarded datagrams are not subjected to policy checks that are added by using this command. See ifconfig(1M) and tun(7M) for information on how to protect forwarded packets.

You must become superuser to invoke the ipsecconf command. The command accepts entries that protect traffic in both directions, and entries that protect traffic in only one direction.

Policy entries that do not specify a direction and contain the patterns laddr host1 (local address) and raddr host2 (remote address) protect traffic in both directions for the named host. Thus, you need only one entry for each host. A policy entry of the pattern saddr host1 daddr host2 (source address to destination address) protects traffic in only one direction, that is, either outbound or inbound. Thus, to protect traffic in both directions, you need to pass the ipsecconf command another entry, as in saddr host2 daddr host1.

You can see the policies that are configured in the system when you issue the ipsecconf command without any arguments. The command displays each entry with an index followed by a number. You can use the -d option with the index to delete a particular policy in the system. The command displays the entries in the order that they were added, which is not necessarily the order in which the traffic match occurs. To view the order in which the traffic match occurs, use the -l option.

The ipsecpolicy.conf file is deleted when the system shuts down. To ensure that IPsec policy is active when the machine boots, you can create an IPsec policy file, /etc/inet/ipsecinit.conf, that the inetinit script reads during startup.

IPsec Policy File

To invoke IPsec security policies when you start the Solaris operating environment, you create an IPsec initialization configuration file with your specific IPsec policy entries. You should name the file /etc/inet/ipsecinit.conf. See the ipsecconf(1M) man page for details about policy entries and their format. After policies are configured, you can use the ipsecconf command to delete a policy temporarily, or to view the existing configuration.

Example—ipsecinit.conf File

The Solaris software includes a sample IPsec policy file that you can use as a template to create your own ipsecinit.conf file. This sample file is named ipsecinit.sample and it contains the following entries:

#
#ident	"@(#)ipsecinit.sample	1.6  01/10/18 SMI"
#
# Copyright (c) 1999,2001 by Sun Microsystems, Inc.
# All rights reserved.
#
# This file should be copied to /etc/inet/ipsecinit.conf to enable IPsec
# systemwide policy (and as a side-effect, load IPsec kernel modules).
# Even if this file has no entries, IPsec will be loaded if
# /etc/inet/ipsecinit.conf exists.
#
# Add entries to protect the traffic using IPsec. The entries in this
# file are currently configured using ipsecconf from inetinit script
# after /usr is mounted.
#
# For example,
#
#	 {rport 23} ipsec {encr_algs des encr_auth_algs md5}
#
# Or, in the older (but still usable) syntax
#
#    {dport 23} apply {encr_algs des encr_auth_algs md5 sa shared}
#    {sport 23} permit {encr_algs des encr_auth_algs md5}
#
# will protect the telnet traffic originating from the host with ESP using
# DES and MD5. Also:
#
#	 {raddr 10.5.5.0/24} ipsec {auth_algs any}
#
# Or, in the older (but still usable) syntax
#
#    {daddr 10.5.5.0/24} apply {auth_algs any sa shared}
#    {saddr 10.5.5.0/24} permit {auth_algs any}
#
# will protect traffic to or from the 10.5.5.0 subnet with AH 
# using any available algorithm.
#
#
# To do basic filtering, a drop rule may be used. For example:
#
#    {lport 23 dir in} drop {}
#    {lport 23 dir out} drop {}
#
# will disallow any remote system from telnetting in.
#
#
# WARNING:	This file is read before default routes are established, and
#		before any naming services have been started. The
#		ipsecconf(1M) command attempts to resolve names, but it will
#		fail unless the machine uses files, or DNS and the DNS server
#		is reachable via routing information before ipsecconf(1M)
#		invocation.  (that is, the DNS server is on-subnet, or DHCP
#		has loaded up the default router already.)
#
#		It is suggested that for this file, use hostnames only if
#		they are in /etc/hosts, or use numeric IP addresses.
#
#		If DNS gets used, the DNS server is implicitly trusted, which
#		could lead to compromise of this machine if the DNS server
#		has been compromised.
#

Security Considerations

If, for example, the /etc/inet/ipsecinit.conf file is sent from an NFS-mounted file system, an adversary can modify the data contained in the file. The outcome would be a change to the configured policy. Consequently, you should use extreme caution if transmitting a copy of the ipsecinit.conf file over a network.

Policy cannot be changed (is “latched”) for TCP/UDP sockets on which a connect(3SOCKET) or accept(3SOCKET) has been issued. Adding new policy entries does not affect the latched sockets. This latching feature might change in the future, so you should not depend on this feature.

Ensure that you set up the policies before starting any communications, because existing connections might be affected by the addition of new policy entries. Similarly, do not change policies in the middle of a communication.

If your source address is a host that can be looked up over the network, and your naming system itself is compromised, then any names that are used are no longer trustworthy.

Security weaknesses often lie in misapplication of tools, not the tools themselves. You should be cautious when using the ipsecconf command. Use a console or other hard-connected TTY for the safest mode of operation.

IPsec Security Associations Database

Keying information for IPsec security services is maintained in a security association database (SADB). Security associations protect both inbound and outbound packets. A user process (or possibly multiple cooperating processes) maintains SADBs by sending messages over a special kind of socket. This is analogous to the method that is described in the route(7P) man page. Only a superuser can access an SADB.

The operating system might spontaneously emit messages in response to external events, such as a request for a new SA for an outbound datagram, or to report the expiration of an existing SA. You open the channel for passing SADB control messages by using the socket call that is described in the previous section. More than one key socket can be open per system.

Messages include a small base header, followed by a number of extension messages (zero or more). Some messages require additional data. The base message and all extensions must be 8-byte aligned. The GET message serves as an example. This message requires the base header, the SA extension, and the ADDRESS_DST extension. See the pf_key(7P) man page for details.

Keying Utilities

The IKE protocol is the automatic keying utility for IPv4 addresses. See Chapter 21, Internet Key Exchange for how to set up IKE. The manual keying utility is the ipseckey(1M) command.

You use the ipseckey command to manually manipulate the security association databases with the ipsecah(7P) and ipsecesp(7P) protection mechanisms. You can also use the ipseckey command to set up security associations between communicating parties when automated key management is not available. An example is communicating parties that have IPv6 addresses.

While the ipseckey command has only a limited number of general options, it supports a rich command language. You can specify that requests should be delivered by means of a programmatic interface specific for manual keying. See the pf_key(7P) man page for additional information. When you invoke ipseckey with no arguments, it enters an interactive mode that displays a prompt that enables you to make entries. Some commands require an explicit security association (SA) type, while others permit you to specify the SA type and act on all SA types.

Security Considerations

The ipseckey command enables a privileged user to enter sensitive cryptographic keying information. If an adversary gains access to this information, the adversary can compromise the security of IPsec traffic. You should consider the following issues when you handle keying material and use the ipseckey command:

1. Have you refreshed the keying material? Periodic key refreshment is a fundamental security practice. Changing keys guards against potential weaknesses of the algorithm and keys, and limits the damage of an exposed key.

2. Is the TTY going over a network (interactive mode)?

   • If the TTY is in interactive mode, then the security of the keying material is the security of the network path for this TTY's traffic. You should avoid using the ipseckey command over a clear-text telnet or rlogin session.

   • Even local windows might be vulnerable to attacks by a concealed program that reads window events.

3. Is the file accessed over the network or readable to the world (-f option)?

   • An adversary can read a network-mounted file as it is being read. You should avoid using a world-readable file with keying material in it.

   • If your source address is a host that can be looked up over the network, and your naming system is compromised, then any names used are no longer trustworthy.

Security weaknesses often lie in misapplication of tools, not the tools themselves. You should be cautious when using the ipseckey command. Use a console or other hard-connected TTY for the safest mode of operation.

IPsec Extensions to Other Utilities

The ifconfig command has options to manage IPsec policy on a tunnel interface, and the snoop command can parse AH and ESP headers.

ifconfig Command

To support IPsec, the following security options have been added to the ifconfig(1M) command:

• auth_algs

• encr_auth_algs

• encr_algs

auth_algs

This option enables IPsec AH for a tunnel, with the authentication algorithm specified. The auth_algs option has the following format:

auth_algs authentication_algorithm

The algorithm can be either a number or an algorithm name, including the parameter any, to express no specific algorithm preference. You must specify all IPsec tunnel properties on the same command line. To disable tunnel security, specify the following option:

auth_alg none

See Table 19–1 for a list of available authentication algorithms and for pointers to the algorithm man pages.

encr_auth_algs

This option enables IPsec ESP for a tunnel, with the authentication algorithm specified. The encr_auth_algs option has the following format:

encr_auth_algs authentication_algorithm

For the algorithm, you can specify either a number or an algorithm name, including the parameter any, to express no specific algorithm preference. If you specify an ESP encryption algorithm, but you do not specify the authentication algorithm, the ESP authentication algorithm value defaults to the parameter, any.

See Table 19–1 for a list of available authentication algorithms and for pointers to the algorithm man pages.

encr_algs

This option enables IPsec ESP for a tunnel with the encryption algorithm specified. The option has the following format:

encr_algs encryption_algorithm

For the algorithm, you can specify either a number or an algorithm name. You must specify all IPsec tunnel properties on the same command line. To disable tunnel security, specify the following option:

encr_alg none

If you specify an ESP authentication algorithm, but not an encryption algorithm, the ESP encryption value defaults to the parameter null.

See the ipsecesp(7P) man page or Table 19–2 for a list of available encryption algorithms and for pointers to the algorithm man pages.

snoop Command

The snoop command can now parse AH and ESP headers. Because ESP encrypts its data, snoop cannot see encrypted headers that are protected by ESP. AH does not encrypt data, so traffic can still be inspected with snoop. The snoop -V option shows when AH is in use on a packet. See the snoop(1M) man page for more details.