System Administration Guide, Volume 3

Chapter 18 Overview of IPsec

The IP Security Architecture (IPsec) provides protection for IP datagrams. The protection can include confidentiality, strong integrity of the data, partial sequence integrity (replay protection), and data authentication. IPsec is performed inside the IP process, and it can be applied with or without the knowledge of an Internet application. While IPsec is an effective tool in securing network traffic, it does not eliminate security problems.

Introduction to IPsec

IPsec provides security associations that include secure datagram authentication and encryption mechanisms within IP. When you invoke IPsec, it applies the security mechanisms to IP datagrams that you have enabled in the IPsec global policy file.

The following figure shows how an IP addressed packet, as part of an IP datagram, proceeds when IPsec has been invoked on an outbound packet. As you can see from the flow diagram, authentication header (AH) and encapsulated security payload (ESP) entities can be applied to the packet. Subsequent sections describe how you apply these entities, as well as athentication and encryption algorithms.

Figure 18-1 IPsec Applied to Outbound Packet Process

Graphic

The following figure shows the IPsec inbound process.

Figure 18-2 IPsec Applied to Inbound Packet Process

Graphic

Security Associations

Security associations (SA) specify security properties from one host to another. Two communicating systems need at least two SAs to communicate securely, unless they use multicast, in which case they can use the same multicast SA. The pf_key(7P) interface manages Security Associations. IPsec does not currently support automatic SA management, but you can use ipseckey(1M) as a command-line front-end. The AH or ESP, destination IP address, and security parameters index (SPI) identifies an IPsec SA. The security parameters index, an arbitrary 32-bit value, is transmitted with an AH or ESP packet. See ipsecah(7P) or ipsecesp(7P) man pages for an explanation about where the SPI resides in a protected packet.

Key Management

A security association contains keying information, algorithm choices, endpoint identities, and other parameters. Managing SAs is called key management. Currently, you must manually do key management.

Protection Mechanisms

IPsec provides two mechanisms for protecting data:

Both mechanisms use security associations.

Authentication Header

The authentication header, a new IP header, provides strong integrity, partial sequence integrity (replay protection), and data authentication to IP datagrams. AH protects as much of the IP datagram as it can. AH cannot protect fields that change nondeterministically between sender and receiver. For example, the IP TTL field is not a predictable field and, consequently, not protected by AH. AH is inserted between the IP header and the transport header. The transport header can be TCP, UDP, ICMP, or another IP header when tunnels are being used. See the tun(7M) man page for details on tunneling.

Authentication Algorithms and the AH Device

IPsec implements AH as a module that is automatically pushed on top of IP. The /dev/ipsecah entry tunes AH with ndd(1M), in addition to allowing future authentication algorithms to be loaded on top of AH. Current authentication algorithms include HMAC-MD5 and HMAC-SHA-1. Each authentication algorithm has its own key size and key format properties. See the authmd5h(7M) and authsha1(7M) man pages for details.

Security Considerations

Without replay protection enabled, all replay attacks jeopardize AH. AH does not protect against eavesdropping. Adversaries can still see data protected with AH.

Encapsulating Security Payload

The ESP provides confidentiality over what it encapsulates, as well as the services that AH provides, but only over that which it encapsulates. ESP's authentication services are optional. These services enable you to use ESP and AH together on the same datagram without redundancy. Because ESP uses encryption-enabling technology, it falls under U.S. export control laws.

ESP encapsulates its data, so it only protects the data that follows its beginning in the datagram. In a TCP packet, ESP encapsulates only the TCP header and its data. If the packet is an IP in IP datagram, ESP protects the inner IP datagram. Per-socket policy allows self-encapsulation, so ESP can encapsulate IP options, when it needs to. Unlike the authentication header (AH) , ESP allows multiple kinds of datagram protection. Using only a single form of datagram protection can expose the datagram to vulnerabilities. For example, you can use only ESP to provide confidentiality, but protecting confidentiality alone exposes vulnerabilities in both replay attacks and cut-and-paste attacks. Similarly, if ESP protects only integrity, and does not fully protect against eavesdropping, it could provide weaker protection than AH.

Algorithms and the ESP Device

IPsec ESP implements ESP as a module that is automatically pushed on top of IP. Use the /dev/ipsecesp entry to tune ESP with ndd(1M), as well as to allow future algorithms to be loaded on top of ESP. ESP allows encryption algorithms to be pushed on top of it, in addition to the authentication algorithms used in AH. Encryption algorithms include United States Data Encryption Standard (DES) and Triple-DES (3DES). Each encryption algorithm has its own key size and key format properties. Because of export laws in the United States, not all encryption algorithms are available outside of the United States.

Security Considerations

ESP without authentication exposes vulnerabilities to cut-and-paste cryptographic attacks, as well as eavesdropping attacks. When you use ESP without confidentiality, its vulnerability to replay is similar to AH. Because of United States export control laws, the encryption strength available on ESP is weaker for versions of the SunOS sold outside the United States.

Authentication and Encryption Algorithms

IPsec uses two types of algorithms:

Authentication Algorithms

Authentication algorithms produce an integrity checksum value or digest based on the data and a key. The authentication algorithm man pages describe the size of both the digest and key (see, for example, authmd5h(7M) and authsha1(7M) man pages).

Encryption Algorithms

Encryption algorithms encrypt data with a key. Encryption algorithms operate on data in units of a block size. The encryption algorithm man pages describe the size of both the block size and the key size (see, for example, encrdes(7M) and encr3des(7M) man pages).

Protection Policy and Enforcement Mechanisms

IPsec separates protection policy and enforcement mechanisms. You can enforce IPsec policies in the following places:

You use the ipsecconf(1M) command to configure system-wide policy.

IPsec applies system-wide policy to incoming and outgoing datagrams. You can apply some additional rules to outgoing datagrams, because of the additional data known by the system. Inbound datagrams can be either accepted or dropped. The decision to drop or accept an inbound datagram is based on several criteria, which sometimes overlap or conflict. Resolving conflict resolution depends on which rule is parsed first; except when a policy entry states that traffic should bypass all other policy, it is automaticaly accepted. Outbound datagrams are either sent with protection or without protection. If protection is applied, the algorithms are either specific or non-specific. If policy normally protects a datagram, it can be bypassed by either an exception in system-wide policy, or by requesting a bypass in per-socket policy.

For intra-system traffic, policies are enforced, but actual security mechanisms are not applied. Instead, the outbound policy on an intra-system packet translates into an inbound packet that has had those mechanisms applied.

Transport and Tunnel Modes

When you invoke ESP or AH after the IP header to protect a datagram, this is referred to as transport mode. For example, if a packet starts off as:

Graphic

Then in transport mode, ESP protects the data as follows:

Graphic

AH, in transport mode, protects the data as follows:

Graphic

AH actually covers the data before it appears in the datagram. Consequently, the protection provided by AH, even in transport mode, does cover some of the IP header.

When an entire datagram is inside the protection of an IPsec header, this is referred to as tunnel mode. Since AH covers most of its preceding IP header, tunnel mode is usually performed only on ESP. The previous example datagram would be protected in tunnel mode as follows:

Graphic

Often, in tunnel mode, the outer (unprotected) IP header has different source and destination addresses from the inner (protected) IP header. The inner and outer IP headers can match if, for example, an IPsec-aware network program uses self-encapuslation with ESP. This is done in case of an IP header option that needs to be protected with ESP.

The Solaris implementation of IPsec is primarily a transport mode IPsec implementation, which implements the tunnel mode as a special case of the transport mode. This is accomplished by treating IP-in-IP tunnels as a special transport provider. When you use ifconfig(1M) configuration options to set tunnels, they are nearly identical to the options available to socket programmers when enabling per-socket IPsec. Also, tunnel mode can be enabled in per-socket IPsec.

Tunneling Module for IPsec Tunnels

A configured tunnel is a point-to-point interface. It enables an IP packet to be encapsulated within an IP packet. Configuring a tunnel requires both a tunnel source and tunnel destination. See the tun(7M) man page and "Solaris Tunneling Interfaces for IPv6" for more information.

A tunnel creates an apparent physical interface to IP. The physical link's integrity depends on the underlying security protocols. If you set up the security associations securely, then you can trust the tunnel; that is, packets that come off the tunnel came from the peer specified in the tunnel destination. If this trust exists, you can use per-interface IP forwarding to create a virtual private network.

Enabling Virtual Private Networks

You can use IPsec to construct virtual private networks (VPN). You do this by constructing an Intranet using the Internet infrastructure. For example, an organization that has separate offices (with separate networks), and uses VPN technology to connect their offices, can deploy IPsec to secure traffic between the two offices.

The following figure illustrates how two offices use the Internet to form their VPN with IPsec deployed on their network systems.

Figure 18-3 Virtual Private Network

Graphic

See "How to Set Up a Virtual Private Network" for a description of the setup procedure.

Managing IPsec

This section describes the IPsec initialization configuration file and various commands that enable you to manage IPsec within your network. See Chapter 19, Implementing IPsec for a description of procedures you can use to manage IPsec.

IPsec Initialization Configuration File

To invoke IPsec security policies when you start the Solaris operating environment, you need to create an IPsec initialization configuration file with your specific IPsec entries. You should name the file, /etc/inet/ipsecinit.conf. See the ipsecconf(1M) man page for details about policy entries and their format.

Example--ipsecinit.conf File

The Solaris software includes a sample ipsecinit.conf file that you can use as a template to create your own ipsecinit.conf file. This sample file is named ipsecinit.sample and it contains the following entries:


#
#ident	"@(#)ipsecinit.sample	1.4	99/04/28 SMI"
#
# Copyright (c) 1999 by Sun Microsystems, Inc.
# All rights reserved.
#
# This file should be copied to /etc/inet/ipsecinit.conf to enable IPsec.
# Even if this file has no entries, IPsec will be loaded if
# /etc/inet/ipsecinit.conf exists.
#
# Add entries to protect the traffic using IPSEC. The entries in this
# file are currently configured using ipsecconf from inetinit script
# after /usr is mounted.
#
# For example,
#
#	 {dport 23} apply {encr_algs des encr_auth_algs md5 sa shared}
#	 {sport 23} permit {encr_algs des encr_auth_algs md5}
#
# will protect the telnet traffic to/from the host with ESP using DES and
# MD5.  Also:
#
#	 {daddr 10.5.5.0/24} apply {auth_algs any sa shared}
#	 {saddr 10.5.5.0/24} permit {auth_algs any}
#
# will protect traffic to or from the 10.5.5.0 subnet with AH 
# using any available algorithm.
#
#
# WARNING:	This file is read before default routes are established, and
#		before any naming services have been started.  The
#		ipsecconf(1M) command attempts to resolve names, but it will
#		fail unless the machine uses files, or DNS and the DNS server
#		is on-subnet (i.e. reachable without a default route).
#
#		It is suggested that for this file, use hostnames only if
#		they are in /etc/hosts, or use numeric IP addresses.
#
#		If DNS gets used, the DNS server is implicitly trusted, which
#		could lead to compromise of this machine if the DNS server
#		has been compromised.
#

Global Policy Setter

You use the ipsecconf(1M) command to configure the IPsec policy for a host. After you configure the policy, IPsec subjects all outbound and inbound datagrams to policy checks as they exit and enter the host. If no entries are found, no policy checks are completed, and all the traffic passes through. Forwarded datagrams are not subjected to policy checks added using this command. See ifconfig(1M) and tun(7M) for information on how to protect forwarded packets. You can use the ifconfig command to delete a policy entry from the /etc/inet/ipsecpolicy.conf file, or to view the existing configuration.

You must become superuser to invoke this command. Each entry protects traffic in only one direction, that is, either outbound or inbound. Thus, to protect traffic in both directions, you need to have separate entries for each direction.

You can see the policies configured in the system when you issue the command without any arguments. The command displays each entry with an index followed by a number. You can use the -d option with the index to delete a given policy in the system. The command displays the entries in the order that they were added, which is not necessarily the order in which the traffic match takes place. To view the order in which the traffic match takes place, use the -l option.

IPsec does not preserve policy entries across reboots. Thus, you need to add the policy every time the system reboots. To configure policies early in the boot process, you can set up policies in the /etc/inet/ipsecinit.conf file, so that the inetinit startup script reads them.

Security Considerations

If, for example, the /etc/inet/ipsecpolicy.conf file is sent from an NFS mounted file system, an adversary can modify the data contained in the file and actually make changes to the configured policy. Consequently, you should not transmit a copy of the /etc/inet/ipsecpolicy.conf file over a network.

Policy is latched for TCP/UDP sockets on which a connect(3N) or accept(3N) has been issued. Adding new policy entries does not have any effect on them. This latching feature might change in the future, so you should not depend on this feature.

Make sure you set up the policies before starting any communications, because existing connections might be affected by the addition of new policy entries. Similarly, do not change policies in the middle of a communication.

If your source address is a host that can be looked up over the network, and your naming system itself is compromised, then any names used are no longer trustworthy.

Security weaknesses often lie in misapplication of tools, not the tools themselves. You should be cautious when using ipseckey. Use a console for the safest mode of operation, or other hard-connected TTY.

Security Associations Database

Keying information for IPsec security services is maintained in security association databases (SADBs). Security associations protect both inbound and outbound packets. A user process (or possibly multiple co-operating processes) maintains SADBs by sending messages over a special kind of socket. This is analogous to the method described in the route(7P) man page. Only a superuser can access an SADB.

The operating system might spontaneously emit messages in response to external events, such as a request for a new SA for an outbound datagram, or to report the expiration of an existing SA. You open the channel for passing SADB control messages by using the socket call described in the previous section. More than one key socket can be open per system.

Messages include a small base header, followed by a number of extension messages (zero or more). Some messages require additional data. The base message and all extensions must be eight-byte aligned. The GET message serves as an example. It requires the base header, the SA extension, and the ADDRESS_DST extension. See the pf_key(7P) man pages for details.

Manual Keying Program

You use the ipseckey(1M) command to manually manipulate the security association databases with the ipsecah(7P) and ipsecesp(7P) network security services. You can also use the ipseckey command to set up security associations between communicating parties when automated key management is not available.

While the ipseckey command has only a limited number of general options, it supports a rich command language. You can specify that requests should be delivered by means of a programmatic interface specific for manual keying. See the pf_key(7P) man page for additional information. When you invoke ipseckey with no arguments, it enters an interactive mode that displays a prompt enabling you to make entries. Some commands require an explicit security association (SA) type, while others permit you to specify the SA type and act on all SA types.

Security Considerations

The ipseckey command handles sensitive cryptographic keying information. It enables a privileged user to enter cryptographic keying information. If an adversary gains access to this information, the adversary can compromise the security of IPsec traffic. You should take the following issues into account when using the ipseckey command:

  1. Is the TTY going over a network (interactive mode)?

    • If it is, then the security of the keying material is the security of the network path for this TTY's traffic. You should avoid using ipseckey(1M) over a clear-text telnet or rlogin session.

    • Even local windows might be vulnerable to attacks by a concealed program that reads window events.

  2. Is the file accessed over the network or readable to the world (-f option)?

    • An adversary can read a network-mounted file as it is being read. You should avoid using a world-readable file with keying material in it.

    • If your source address is a host that can be looked up over the network, and your naming system is compromised, then any names used are no longer trustworthy.

Security weaknesses often lie in misapplication of tools, not the tools themselves. You should be cautious when using ipseckey. Use a console for the safest mode of operation, or other hard-connected TTY.

IPsec Extensions to Existing Utilities

ifconfig

To support IPsec, the following security options have been added to ifconfig(1M):

auth_algs

This option enables IPsec AH for a tunnel, with the authentication algorithm specified. It has the following format:


auth_algs authentication_algorithm

The algorithm can be either a number or an algorithm name, including the parameter any, to express no specific algorithm preference. You must specify all IPsec tunnel properties on the same command line. To disable tunnel security, specify the following option:


auth_alg none

encr_auth_algs

This option enables IPsec ESP for a tunnel, with the authentication algorithm specified. It has the following format:


encr_auth_algs authentication_algorithm

For the algorithm, you can specify either a number or an algorithm name, including the parameter any, to express no specific algorithm preference. If you specify an ESP encryption algorithm, but you do not specify the authentication algorithm, the ESP authentication algorithm value defaults to the parameter, any.

encr_algs

This option enables IPsec ESP for a tunnel with the encryption algorithm specified. It has the following format:


encr_auth_algs encryption_algorithm

For the algorithm, you can specify either a number or an algorithm name. You must specify all IPsec tunnel properties on the same command line. To disable tunnel security, specify the following option:


encr_alg none

If you specify an ESP authentication algorithm, but not encryption algorithm, the ESP encryption value defaults to the parameter null.

snoop(1M)

The snoop command can now parse AH and ESP headers. Since ESP encrypts its data, snoop cannot see encrypted headers protected by ESP. AH does not encrypt data, so traffic can still be inspected with snoop. The snoop -V option can show when AH is in use on a packet. See the snoop(1M) man page for more details.