11.4 Example Protocols

Several widely used Internet protocols illustrate different facets of cryptographic techniques. This section examines three such protocols, each at a different layer. PEM is a privacy-enhanced electronic mail protocol at the applications layer and demonstrates the considerations needed when designing such a protocol. Its techniques are similar to those of PGP, another widely used security-enhanced electronic mail protocol. SSL provides transport layer security. Application layer protocols such as HTTP can use SSL to ensure secure connections. IPsec provides security mechanisms at the network, or IP, layer.

11.4.1 Secure Electronic Mail: PEM

Electronic mail is a widely used mechanism for communication over the Internet. It is also a good example of how practical considerations affect the design of security-related protocols. We begin by describing the state of electronic mail and then show how security services can be added.

Figure 11-3 shows a typical network mail service. The UA (user agent) interacts directly with the sender. When the message is composed, the UA hands it to the MTA (message transport, or transfer, agent). The MTA transfers the message to its destination host, or to another MTA, which in turn transfers the message further. At the destination host, the MTA invokes a user agent to deliver the message.

An attacker can read electronic mail at any of the computers on which MTAs handling the message reside, as well as on the network itself. An attacker could also modify the message without the recipient detecting the change. Because authentication mechanisms are minimal and easily evaded, a sender could forge a letter from another and inject it into the message handling system at any MTA, from which it would be forwarded to the destination. Finally, a sender could deny having sent a letter, and the recipient could not prove otherwise to a disinterested party. These four types of attacks (violation of confidentiality, authentication, message integrity, and nonrepudiation) make electronic mail nonsecure.

In 1985, the Internet Research Task Force on Privacy (also called the Privacy Research Group) began studying the problem of enhancing the privacy of electronic mail. The goal of this study was to develop electronic mail protocols that would provide the following services.

The protocols were christened Privacy-enhanced Electronic Mail (or PEM).

11.4.1.1 Design Principles

Creating a viable protocol requires the developers to consider several design aspects. Otherwise, acceptance and use of the protocol will be very limited.

Related protocols should not be changed. A protocol is designed to provide specific services (in this case, the privacy enhancements discussed in the preceding section). It should not require alteration of other protocols (such as those that transmit electronic mail). The Privacy Research Group developed new protocols rather than modifying the mail transfer protocols. This also requires development of new software rather than modification of existing software to implement the protocol (although existing software can be modified to support it).

A corollary is compatibility. A general protocol (such as PEM) must be compatible with as many other protocols and programs as possible. The protocols must work with a wide range of software, including software in all environments that connect to the Internet.

Another important principle is independence. The privacy enhancements should be available if desired but should not be mandatory. If a new protocol provides specific services, the user should be able to use the services desired, which may (or may not) be all the ones that the protocol provides. For example, a sender might care about sender authentication but not confidentiality. This also enables some users to send privacy-enhanced electronic mail, and others to send unprotected electronic mail, on the same system. Recipients can also read either type of mail.

Finally, two parties should be able to use the protocol to communicate without prearrangement. Arranging a communications key out of band (such as in person or over the telephone) can be time-consuming and prone to error. Furthermore, callers must authenticate themselves to the recipients. This is difficult and is another error-prone operation.

To summarize, the design goals of PEM were as follows.

11.4.1.2 Basic Design

PEM defines two types of keys. The message to be sent is enciphered with a data encipherment key (DEK), corresponding to a session key. This key is generated randomly and is used only once. It must be sent to the recipient, so it is enciphered with an interchange key. The interchange keys of the sender and recipient must be obtained in some way other than through the message.

This requires several assumptions. First, the interchange key must be available to the respective parties. If symmetric ciphers are used, the keys must be exchanged out of bands—for example, by telephone or courier. If public keys are used, the sender needs to obtain the certificate of the recipient.

If Alice wants to send a confidential message to Bob, she obtains Bob's interchange key k_Bob. She generates a random DEK k_session and enciphers the message m. She then enciphers the DEK using the interchange key. She sends both to Bob.

Alice Bob: { m }k_session{ k_session }k_Bob

Bob can then decipher the session key and from it obtain the message.

If Alice wants to send an authenticated, integrity-checked message to Bob, she first computes a cryptographic hash h(m) of the message, possibly using a random session key (if the hash function requires one). The value that the hash function computes is called a message integrity check (MIC). She then enciphers the MIC (and the session key, if one was used) with her interchange key k_Alice and sends it to Bob:

Alice Bob: m { h(m) }k_Alice

Bob uses Alice's interchange key to decipher the MIC, recomputes it from m, and compares the two. If they do not match, either the message or the value of the hash has been changed. In either case, the message cannot be trusted.

To send an enciphered, authenticated, integrity-checked message, combine the operations discussed above, as follows.

Alice Bob: { m }k_session{ h(m) }k_Alice{ k_session }k_Bob

The nonrepudiation service comes from the use of public key cryptography. If Alice's interchange key is her private key, a third party can verify that she signed the message by deciphering it with her public key. Alice cannot refute that her private key was used to sign the message. (She can dispute that she signed it by claiming her private key was compromised. Preventing this is beyond the scope of technical protocols. In this context, "nonrepudiation" refers only to the inability to deny that the private key was used to sign the message.)

11.4.1.3 Other Considerations

When the interchange keys are for public key cryptosystems, PEM suggests the use of a certificate-based key management scheme (see Section 14.5, "Naming and Certificates"). However, it is not a requirement.

A major problem is the specification of Internet electronic mail. Among the restrictions placed on it, the requirements that the letter contain only ASCII characters and that the lines be of limited length are the most onerous. Related to this is the difference among character sets. A letter typed on an ASCII-based system will be unreadable on an EBCDIC-based system.

A three-step encoding procedure overcomes these problems.

The resulting ASCII message has PEM headers (indicating algorithms and key) prepended. PEM headers and body are surrounded by lines indicating the start and end of the PEM message.

If the recipient has PEM-compliant software, she can read the message. Otherwise, she cannot. If the message is authenticated and integrity-checked (but not encrypted), she should be able to read the message even if she does not have PEM-compliant software (remember that one of the design components is compatibility with existing mail programs). The special mode MIC-CLEAR handles this case. In this mode, the message check is computed and added, but the message is not transformed into the representation of step 3. On receipt, the authentication and message integrity check may fail because some MTAs add blank lines, change the end-of-line character, or delete terminating white space from lines. Although this does not alter the meaning of the message, it does change the content. Hence, PEM-compliant software will report that the message has been altered in transit. But people can use normal mail reading programs to read the letter. (Whether they should trust it is another matter. Given that the PEM software has reported changes, the recipients should at least verify the contents in some way before trusting the letter.)

11.4.1.4 Conclusion

PEM demonstrates how system factors influence the use of cryptographic protocols. While central to the design and implementation of PEM systems, the cryptographic protocols require a supporting infrastructure. The need for compatibility guides many design choices for this infrastructure. The environment of development also affects the infrastructure.

Comparing PGP and PEM illustrates this. Both use the same cryptographic protocols, but by default, PGP uses the IDEA cipher instead of the DES. PGP also uses a different, nonhierarchical certificate management scheme described in Sections 10.4.2.2 and 14.5. Finally, PGP handles line termination characters differently. Messages are labeled binary or text. If binary, line terminators are untransformed. If text, they are canonicalized (if enciphering) or mapped into the end-of-line character sequence for the current host (if deciphering).

11.4.2 Security at the Transport Layer: SSL

The Secure Socket Layer (SSL) [375] is a standard developed by Netscape Corporation to provide security in WWW browsers and servers. The current version, SSLv3, is the basis for an Internet standard protocol under development. This protocol, the Transport Layer Security (TLS) protocol [293], is compatible with SSLv3 and has only minor changes. It has not yet been adopted formally, so we will focus on the widely used SSLv3.

SSL works in terms of connections and sessions between clients and servers.

Definition 11–6. An SSL session is an association between two peers. An SSL connection is the set of mechanisms used to transport data in an SSL session.

A single session may have many connections. Two peers may have many sessions active at the same time, but this is not common.

Each party keeps information related to a session with each peer. The data associated with a session includes the following information.

A connection describes how data is sent to, and received from, the peer. Each party keeps information related to a connection. Each peer has its own parameters. The key with which the client enciphers data is (probably) not the same as the key with which the server enciphers data. The information associated with the connection includes the following.

SSLv3 consists of two layers supported by numerous cryptographic mechanisms. We begin by describing the mechanisms, then the lower layer, and finally the upper layer.

11.4.2.1 Supporting Cryptographic Mechanisms

During the setup negotiations, the peers determine a cryptographic mechanism for providing confidentiality and message and origin integrity. Interchange keys are used to establish a session key. Because all parts of SSL use cryptographic mechanisms, we discuss them first.

The initial phase of session setup uses a public key cryptosystem to exchange keys. The messages are enciphered using a classical cipher and are checksummed using a cryptographic checksum.

Initially, the handshake protocol assumes no interchange cipher, no classical cipher, and no checksum. The handshake protocol then negotiates the selection of these mechanisms.

If the interchange cipher is RSA, the server must provide an RSA certificate for key exchange. The server may request that the client provide either an RSA or DSS certificate for signatures. Table 11-1 shows the legal combinations of classical ciphers and MACs.

Table 11-1. RSA, Classical Ciphers, and Checksum Combinations

Interchange cipher	Classical cipher	MAC algorithm
RSA (key at most 512 bits)	None RC4 (40-bit key) RC2 (40-bit key), CBC mode DES (40-bit key), CBC mode	MD5, SHA MD5 MD5 SHA
RSA	None RC4 (128-bit key) IDEA, CBC mode DES, CBC mode DES, EDE mode, CBC mode	MD5, SHA MD5, SHA SHA SHA SHA

Three types of Diffie-Hellman ciphers serve as interchange ciphers. "Diffie-Hellman" denotes a cipher system in which the certificate contains the cipher parameters and is signed by an appropriate certification authority. "Ephemeral Diffie-Hellman" refers to a cipher system in which a DSS or RSA certificate is used to sign the parameters to the Diffie-Hellman cipher. This implies that the parameters will not be used again (or else they would have been in a certificate) and so are "ephemeral." "Anonymous Diffie-Hellman" refers to use of Diffie-Hellman without either party being authenticated. This cipher is vulnerable to attacks, and its use is "strongly discouraged" according to the SSLv3 specification. Table 11-2 summarizes the legal combinations of classical ciphers and MACs.

Table 11-2. Diffie-Hellman, Classical Ciphers, and Checksum Combination

Interchange cipher	Classical cipher	MAC algorithm
Diffie-Hellman, DSS certificate	DES (40-bit key), CBC mode DES, CBC mode DES, EDE mode, CBC mode	SHA SHA SHA
Diffie-Hellman (key at most 512 bits), RSA certificate	DES (40-bit key), CBC mode DES, CBC mode DES, EDE mode, CBC mode	SHA SHA SHA
Ephemeral Diffie-Hellman, DSS certificate	DES (40-bit key), CBC mode DES, CBC mode DES, EDE mode, CBC mode	SHA SHA SHA
Ephemeral Diffie-Hellman (key at most 512 bits), RSA certificate	DES (40-bit key), CBC mode DES, CBC mode DES, EDE mode, CBC mode	SHA SHA SHA
Anonymous Diffie-Hellman	RC4 (40-bit key) RC4 (128-bit key) DES (40-bit key), CBC mode DES, CBC mode DES, EDE mode, CBC mode	MD5 MD5 SHA SHA SHA

The third set of ciphers is for the Fortezza cryptographic token [758, 759] used by the U.S. Department of Defense. Table 11-3 summarizes the legal combinations of classical ciphers and MACs.

Table 11-3. Fortezza, Classical Ciphers, and Checksum Combinations

Interchange cipher	Classical cipher	MAC algorithm
Fortezza key exchange	None Fortezza, CBC mode	SHA SHA
Algorithm	RC4 (128-bit key)	SHA

An RSA digital signature is computed by concatenating two hashes (the first computed using MD5, the second by computing SHA) and signing them with the appropriate public key. For other ciphers, the SHA hash is signed.

The record layer computes a MAC for each block. In SSLv3, the MAC is

hash(MAC_ws||opad||hash(MAC_ws||ipad||seq||SSL_comp||SSL_len||block))

where hash is the particular hash function used (MD5 or SHA), MAC_ws is the write MAC key of the entity, ipad and opad are the ipad and opad from HMAC (see Section 9.4.1), seq is the sequence number, SSL_comp is the message type (taken from the higher layer protocol), SSL_len is the length of the block, and block is the block itself.

When the record layer (see Section 11.4.2.2) enciphers a block, it uses the cryptosystem negotiated during the initialization of the session. The cryptosystem may be a stream cipher or a block cipher. If it is the former, the blocks are enciphered sequentially, with the state of the stream at the end of one block carrying over to the next block. If it is the latter, the block is padded to the block size by adding bytes after the MAC. The padding is arranged so that the length of the padding is added just before the cipher block boundary.

During the handshake protocol, the client and server each compute a secret from some shared data. SSLv3 uses the following function:

master	=	MD5(pre || SHA('A' || pre || random_1 || random_2)) || MD5(pre || SHA('BB' || pre || random_1 || random_2)) || MD5(pre || SHA('CCC' || pre || random_1 || random_2))

where pre is the shared data and random_1 and random_2 are shared random numbers.

11.4.2.2 Lower Layer: SSL Record Protocol

The SSL record protocol provides a basis for secure communication. The protocol to be secured runs on top of it. For example, SSL adds security to HTTP by taking HTTP messages and applying the steps listed below to each message. However, the SSL record protocol need not handle retransmissions or unreliable delivery, because it runs on top of TCP. TCP handles the transport layer functions such as retransmissions and reliable delivery.

The SSL record protocol provides both confidentiality and message integrity for the records being sent. It accepts messages from the higher layer. Each message is split, if needed, into multiple parts. The SSL record protocol can transport a block of at most 2¹⁴ = 16,384 bytes. The following steps are applied to each block, and the blocks are reassembled at the end.

The block is compressed. The MAC of the (compressed) block is computed, and the compressed block and MAC are enciphered. Finally, the SSL record header is prepended. The header contains the message type, the major version number (3 for both SSLv3 and TLS), the minor version number (0 for SSLv3, 1 for TLS), and the length of the block. Figure 11-4 shows these steps.

If the MAC and enciphering key have not been selected (which happens when the record layer carries messages for the handshake protocol; see Section 11.4.2.3), the encipherment algorithm is the identity transformation and no MAC is used. (In other words, the record layer does neither encipherment nor computation of the MAC.)

11.4.2.3 Upper Layer: SSL Handshake Protocol

The SSL handshake protocol sets up the parameters for the SSL record protocol. It consists of four rounds that enable the peers to agree on keys, ciphers, and MAC algorithms. The exact sequence depends on the level of security desired. For clarity, in the following discussion we assume that the client and server use RSA as the interchange cryptosystem.

The first round creates the SSL connection between the client C and the server S. If the server initiates the handshake, it sends an empty server_hello message, and the client begins the protocol. The client can initiate the handshake by sending the message without the server's request:

where version is the version of SSL that the client uses and rand₁ is a nonce composed of a timestamp and a set of 28 random bytes. The session_id is either 0 (meaning a new session) or the session identification number of an existing session (meaning that parameters are to be updated). The cipher_list is a list of ciphers that the client understands, in order of preference, and the compression_list identifies the compression algorithms that the client understands, again in order of preference.

The server responds with its own message:

which indicates the selection of the parameters for the connection. Here, version is the highest version of SSL that both the client and server understand, the cipher and compression fields identify the cipher and compression algorithms, session_id identifies the session (if the client sent a 0, this is the identification number of the new session), and rand₂ is a nonce (timestamp and 28 random bytes) that the server generated. For this example, assume that the cipher selected is RSA. This ends the first round.

The server authenticates itself in the second round. (If the server is not going to authenticate itself, it sends the final message in this round—message 6—immediately.) First, it sends the client its X.509v3 certificate.^[4]

^[4] If the client is not using RSA, the server sends a certificate appropriate for the cryptosystem in use.

The next message sets the parameters. If the RSA certificate is for signatures only, the server sends the public key modulus and exponent used for confidentiality, followed by a hash of the data.

Here, rand₁ and rand₂ are the random data from the first two messages; this prevents replay. The hash is the concatenation of the MD5 and SHA-1 hashes for the given parameter, digitally signed by enciphering them with the private key k_S of the server.^[5]

^[5] If DSS is used, the server sends the modulus, generator, and public value. If Fortezza is used, the random number for the key exchange algorithm is sent (and no hash is appended).

If the server is not anonymous, it may request a certificate from the client:

where cert_type identifies the type of certificate requested (by cryptosystem used) and good_cert_authorities identifies certification authorities that the server will accept. The server then sends a message ending the second round.

Round 3 begins with the client validating the server's certificate (if any) and the parameters (again, if any were sent). If the server requested a certificate, the client obliges or replies with an alert indicating that it has no certificate:

The client next sends a message to begin the key exchange. Since our peers are using RSA, the appropriate public key is used to encipher 48 random bytes to produce a "pre master secret" pre. The server and client will use this to generate a shared master secret.^[6]

^[6] If Diffie-Hellman is used, and a Diffie-Hellman certificate was sent, this message is empty; otherwise, it contains the client's public value. If Fortezza is used, this message contains its parameters, including the client's public key.

Both parties now compute the master secret from the pre_master_secret, as discussed above. The client closes with a verification message so the server can check that all previous messages were received correctly.

Here, mastr is the master secret computed from the pre sent in the previous step, ipad and opad are as defined for the HMAC, and messages is the concatenation of messages 1 through 8.

The fourth round consists of acknowledgments between the server and the client. First, the client tells the server to begin using the cipher specified (using a "change cipher spec" message; see Section 11.4.2.4). The client updates its session and connection information to reflect the cipher it uses. It then sends a "finished" message. As for the preceding message, the client creates this message by hashing the previously sent messages; unlike the preceding one, it includes message 9. The SSL record layer applies the ciphers and MACs to this message.

The server responds with a "change cipher spec" message telling the client to use the ciphers that the server specified, and it updates its session and connection information accordingly. It then sends a "finish" message.

In both the client's and the server's "finish" message, the quantities are as for message 9, except that messages includes message 9.

At this point, enciphered communications can begin.

11.4.2.4 Upper Layer: SSL Change Cipher Spec Protocol

The change cipher spec protocol is a single byte. It is sent after new cipher parameters have been negotiated (or renegotiated). The new parameters are considered "pending" because the old parameters are in use. When this protocol message is received, the "pending" parameters become the parameters in use.

11.4.2.5 Upper Layer: SSL Alert Protocol

The alert protocol signals that an unusual condition exists. A closure alert signals that the sender will not send any more messages over the connection. All pending data is delivered, but any new messages are ignored.

All other alerts are error alerts. Errors are either warnings or fatal errors. In the latter case, as soon as the error is sent or received, the connection is torn down. The cause of the error and its type (warning or fatal) are independent; however, some errors are always fatal. Table 11-4 summarizes the error messages and their levels.

Table 11-4. SSL Alert Protocol Error Messages

Message	Level	Message	Level
unexpected_message	fatal	no_certificate	warning, fatal
bad_record_mac	fatal	bad_certificate	warning, fatal
decompression_failure	fatal	unsupported_certificate	warning, fatal
handshake_failure	fatal	certificate_revoked	warning, fatal
illegal_parameter	fatal	certificate_expired certificate_unknown	warning, fatal warning, fatal

11.4.2.6 Upper Layer: Application Data Protocol

This protocol simply passes data from the application layer to the SSL Record Protocol layer. The record protocol transports the data to the peer using the current compression and cipher algorithms.

11.4.2.7 Summary

The SSL layer provides a transport mechanism that supports confidentiality and integrity. It supports a variety of ciphers and MACs. Any higher-level application protocol can use it, and it is the basis for much Internet security. World Wide Web browsers and servers generally support SSLv3.

11.4.3 Security at the Network Layer: IPsec

IPsec is a collection of protocols and mechanisms that provide confidentiality, authentication, message integrity, and replay detection at the IP layer. Because cryptography forms the basis for these services, the protocols also include a key management scheme, which we will not discuss here.

Conceptually, think of messages being sent between two hosts as following a path between the hosts. The path also passes through other intermediate hosts. IPsec mechanisms protect all messages sent along a path. If the IPsec mechanisms reside on an intermediate host (for example, a firewall or gateway), that host is called a security gateway.

IPsec has two modes. Transport mode encapsulates the IP packet data area (which is the upper layer packet) in an IPsec envelope, and then uses IP to send the IPsec-wrapped packet. The IP header is not protected. Tunnel mode encapsulates an entire IP packet in an IPsec envelope and then forwards it using IP. Here, the IP header of the encapsulated packet is protected. (Figure 11-5 illustrates these modes.) Transport mode is used when both endpoints support IPsec. Tunnel mode is used when either or both endpoints do not support IPsec but two intermediate hosts do.

EXAMPLE: Secure Corp. and Guards Inc. wish to exchange confidential information about a pending fraud case. The hosts main.secure.com and fraud.guards.com both support IPsec. The messages between the systems are encapsulated using transport mode at the sender and processed into cleartext at the receiver. Red Dog LLC is a third corporation that needs access to the data. The data is to be sent to gotcha.reddog.com. Red Dog's systems do not support IPsec, with one exception. That exception is the host, firewall.reddog.com, that is connected to both Red Dog's internal network and the Internet. Because none of Red Dog's other hosts is connected to the Internet, all traffic to gotcha from Secure Corp. must pass through firewall.reddog.com. So main.secure.com uses tunnel mode to send its IPsec packets to Red Dog. When the packets arrive at firewall, the IPsec information is removed and validated, and the enclosed IP packet is forwarded to gotcha. In this context, firewall.reddog.com is a security gateway.

Two protocols provide message security. The authentication header (AH) protocol provides message integrity and origin authentication and can provide antireplay services. The encapsulating security payload (ESP) protocol provides confidentiality and can provide the same services as those provided by the AH protocol. Both protocols are based on cryptography, with key management supplied by the Internet Key Exchange (IKE) protocol (although other key exchange protocols, including manual keying, may be used).

11.4.3.1 IPsec Architecture

IPsec mechanisms use a security policy database (SPD) to determine how to handle messages. Legal actions are discarding the message, applying security services to the message, and forwarding the message with no change. The action taken depends on information in the IP and transport layer headers.

IPsec mechanisms determine the security services needed on the basis of the SPD and the path that the packet takes.

When a packet arrives, the IPsec mechanism consults the SPD for the relevant network interface. The SPD determines which entry applies on the basis of the attributes of the packet. These attributes include the source and destination port and address, the transport layer protocol involved, and other data.

EXAMPLE: An SPD has two entries for destination addresses 10.1.2.3 to 10.1.2.103. The first applies to packets with destination port 25. The second applies to packets transporting the protocol HTTP. If a packet arrives with destination address 10.1.2.50, and its destination port is 25, the first entry applies; if its destination port is 80, the second entry applies.

Entries are checked in order. If one has a different policy for securing electronic mail depending on its destination, the more specific entries are placed where they will be searched first. If no entry matches the incoming packet, it is discarded.

EXAMPLE: In the example above, the administrator wants to discard SMTP packets coming from host 192.168.2.9 and forward packets from host 192.168.19.7 without applying IPsec services. Assuming that the SPD entries are searched from first to last, the SPD would have these three entries: source 192.168.2.9, destination 10.1.2.3 to 10.1.2.103, port 25, discard source 192.168.19.7, destination 10.1.2.3 to 10.1.2.103, port 25, bypass destination 10.1.2.3 to 10.1.2.103, port 25, apply IPsec

The heart of applying IPsec is the security association.

Definition 11–7. A security association (SA) is an association between peers for security services. The security association is unidirectional.

A security association is a set of security enhancements to a channel along which packets are sent. It is defined uniquely by the destination address, the security protocol (AH or ESP), and a unique 32-bit security parameter index (SPI). It defines the security protocol that is to be applied to packets sent over that association.

Each SA uses either ESP or AH, but not both. If both are required, two SAs are created. Similarly, if IPsec is to provide security between two peers in both directions, two SAs are needed.

When IPsec services are to be applied, the SPD entry identifies one or more security associations and parameters. The parameters describe how to determine which security association(s) to use to process the packet. This leads to the security association database (SAD), which consists of a set of selectors and corresponding security associations.

EXAMPLE: Continuing the example above, focus on the case in which IPsec is to be applied. The SPD entry for 10.1.2.101 could take the selector for the SAD from the packet (so the selector might be the SA with the destination address 10.1.2.101) or from the SPD entry (so the selector might be the SA with the destination addresses in the range 10.1.2.3 to 10.1.2.103).

Each SAD entry contains information about the SA. Key fields are as follows.

• The AH algorithm identifier and keys are used when the SA uses the AH protocol.

• The ESP encipherment algorithm identifier and keys are used when the SA uses the confidentiality service of the ESP protocol.

• The ESP authentication algorithm identifier and keys are used when the SA uses the authentication and data integrity services of the ESP protocol.

• The lifetime of the SA is either the time at which the SA must be deleted and a new one formed or a count of the maximum number of bytes allowed over this SA.

• The IPsec protocol mode is tunnel mode, transport mode, or a wildcard. If it is a wildcard, either protocol mode is acceptable. Security gateways need to support only tunnel mode, but host implementations must support both modes.

An additional field checks for replay in inbound packets.

• The antireplay window field is used to detect replay (see Section 11.4.3.2). If the SA does not use the antireplay feature, this field is not used.

Outbound packets have sequence numbers.

• The sequence number counter generates the AH or ESP sequence number.

• The sequence counter overflow field stops further traffic over the SA if the sequence counter overflows.

• Path Maximum Transmission Unit and aging variables detect time-outs.

When inbound traffic arrives, the destination address, security protocol, and SPI are used to find the associated SA in the SAD. This verifies the properties that the packet should have and enables the replay check (if desired). If the packet is to be forwarded, the SPD determines the relevant services, the appropriate services are supplied, and the packet is forwarded.

In some situations, multiple SAs may protect packets.

Definition 11–8. A security association bundle (SA bundle) is a sequence of security associations that the IPsec mechanisms apply to packets.

Tunnel mode SAs can be nested. This is called iterated tunneling and occurs when multiple hosts build tunnels through which they send traffic. The endpoints may be the same, although support for iterated tunneling is required only when at least one endpoint of the two tunnels is different. The tunnels may be entirely nested.

EXAMPLE: Return to Secure Corp. and Red Dog LLC. The fraud group within Secure has a host, frauds, that has IPsec mechanisms. The Red Dog fraud group has a new system, equity, that also has IPsec mechanisms. Both Secure's gateway to the internet, gateway, and Red Dog's gateway to the Internet, firewall, have IPsec mechanisms. Because the data is so sensitive, the fraud groups decide that they need to protect their data within each company. The SA between the gateways is not enough. The data transfer now has two SAs. The first goes from gateway.secure.com to firewall.reddog.com and is in tunnel mode. The second, also in tunnel mode, begins at frauds.secure.com, tunnels through the SA from gateway.secure.com to firewall.reddog.com, and terminates at equity.reddog.com.

Iteration of transport mode SAs occurs when both the AH and ESP protocols are used. This is called transport adjacency, and when it is used, application of the ESP protocol should precede application of the AH protocol. The idea is that the ESP protocol protects the higher-layer (transport) protocol and the AH protocol protects the IP packet. Were the AH protocol to be applied first, the ESP protocol would not protect the IP packet headers.

It is instructive to examine the appearance of the packets in the example above. Figure 11-6 shows the packet layout as it travels between the two companies. Notice that the packet generated by frauds is encapsulated in another IP packet with the IPsec services applied to the inner packet. Both headers identify equity as the destination. When the packet arrives at gateway, the original IP header is (probably) not visible to gateway. In this case, the SAD and SPD use a special identifier to indicate that the source is obscured. (See Exercise 8.) The appropriate SA directs the packet to be encapsulated and forwarded to firewall, so the added IP header identifies firewall as the destination IP address. When the packet arrives at firewall, it uses the incoming packet's destination IP address (firewall), security protocol, and SPI to locate the SA. This bundle tells firewall to authenticate and decrypt the contents of the packet. The inner IP packet is then used to look up the appropriate action in the SPD, which (in this case) is to bypass IPsec. The packet is then forwarded to equity, which repeats the processing. The innermost IP packet is then forwarded to equity and processed.

We now examine the AH and ESP protocols.

11.4.3.2 Authentication Header Protocol

The goal of the authentication header (AH) protocol is to provide origin authentication, message integrity, and protection against replay, if desired. It protects static fields of the IP packet header as well as the contents of the packet.

The important parameters included in the AH header are an indication of the length of the header, the SPI of the SA under which this protocol is applied, a sequence number used to prevent replay, and an Integrity Value Check (IVC)^[7] padded to a multiple of 32 bits (for IPv4) or 64 bits (for IPv6).

^[7] This is another term for a message integrity check (MIC); we use the AH protocol specification term here for consistency.

The AH protocol has two steps. The first checks that replay is not occurring. The second checks the authentication data.

When a packet is sent, the sender assumes that antireplay is used unless it is told otherwise. The sender first checks that the sequence number will not cycle. (If it will, a new SA must be created; see the discussion above.) It adds 1 to the current sequence number. The sender then calculates the IVC of the packet. The IVC includes all fields in the IP header that will not change in transit or that can be predicted (such as the destination field), the AH header (with the authentication data field set to 0 for this computation), and any encapsulated or higher-layer data. Mutable fields in the IP header (such as the type of service, flags, fragment offset, time to live, and header checksum fields) are set to 0 for this computation.

When a packet arrives, the IPsec mechanism determines if the packet contains an authentication header. If so, it uses the SPI and destination address to find the associated SA in the SAD. If no such SA exists, the packet is discarded. Otherwise, the key, IVC algorithm, and antireplay settings are obtained from the SAD entry.

If the antireplay service is desired, a "sliding window" mechanism checks that the packet is new. Think of the SA as operating on a stream of packets. Conceptually, the window contains slots for at least 32 packets. Each slot has the sequence number of the packet for that slot. When a packet arrives, the mechanism checks that the packet's sequence number is at least that of the leftmost slot in the window. If the packet's sequence number is to the left of the window, the packet is discarded. The IVC of the packet is then verified, and if it is incorrect, the packet is discarded. Otherwise, if the packet's sequence number lies within the window, but the slot with that sequence number is occupied, the packet is discarded. If the slot is empty, the packet is inserted into the slot. Finally, if the packet lies to the right of the window, the window is advanced to create a slot for the packet. The packet is then placed in that slot, which is the rightmost slot in the window.

If the antireplay service is not used, the IVC is verified. The IVC is computed in the same way as the sender (that is, appropriate fields are replaced by zeros) and is compared with the IVC in the AH. If the two differ, the packet is discarded.

All implementations of the AH protocol must support HMAC_MD5 and HMAC_SHA-1. They may support others as well.

11.4.3.3 Encapsulating Security Payload Protocol

The goal of the encapsulating security payload (ESP) protocol is to provide confidentiality, origin authentication, message integrity, protection against replay if desired, and a limited form of traffic flow confidentiality. It protects only the transport data or encapsulated IP data; it does not protect the IP header.

The important parameters included in the ESP header are the SPI of the SA under which this protocol is applied, a sequence number used to prevent replay, a generic "payload data" field, padding, the length of the padding, and an optional authentication data field.

The data in the payload data field depends on the ESP services enabled. For example, if an SA needs to resynchronize a cryptographic algorithm used in chaining mode, the sender could include an initialization vector here. As more algorithms for the ESP are defined, they may specify data to be included in this field.

Because the ESP protocol begins enciphering with the payload data field and protects both header fields and data, the IPsec mechanism may need to pad the packet in order to have the number of bits or bytes required by the cryptographic algorithm. The padding field allows for this adjustment. The padding length field contains the number of padding bytes; no more than 255 bytes of padding are allowed.

At least one of the confidentiality and authentication services must be selected. Furthermore, because packets may not arrive in order, any synchronization material must be carried in the payload field. Otherwise, the packets that follow a missing packet may be unintelligible.

When a packet is sent, the sender adds an ESP header, including any required padding, to the payload (either the transport data or an encapsulated IP packet). The sender enciphers the result (except for the SPI and sequence numbers). If authentication is desired, the authentication is computed as for the AH protocol, except that it is over the ESP header and payload. It does not include the IP header that encapsulates the ESP header and payload. The relevant SA dictates the cryptographic keys and algorithms that are used.

When a packet arrives, the IPsec mechanism determines if the packet contains an ESP header. If so, it uses the SPI and destination address to find the associated SA in the SAD. If no such SA exists, the packet is discarded. Otherwise, the SA parameters are obtained from the SAD entry.

If the authentication service is used, the antireplay feature and the MAC verification proceed as for the AH, again except that only the ESP and the payload are used. Because the authentication data is inserted after encipherment, it is not enciphered and so can be used directly.

If the confidentiality service is used, the IPsec mechanisms decipher the enciphered portion of the ESP header. Any padding is processed, and the payload is deciphered. If the SA specifies transport mode, the IP header and payload are treated as the original IP packet. If the SA specifies tunnel mode, the encapsulated IP packet is treated as the original IP packet.

Typical implementations of public key cryptosystems are far slower than implementations of classical cryptosystems. Hence, implementations of ESP assume a classical cryptosystem, although this is not required.

All implementations of ESP must support DES in CBC mode and the NULL encipherment algorithms, as well as the HMAC_MD5, HMAC_SHA-1, and NULL MACs. (The NULL encipherment algorithm and MAC mean that those algorithms are not used. Both should never be NULL at the same time.) Implementations may support other algorithms.

11.4.4 Conclusion

Each of the three protocols adds security to network communications. The "best" protocol to use depends on a variety of factors.

To what do the requisite security services apply? If they are specific to one particular application, such as remote logins, then using a program with application layer security is appropriate. When a program that requires security services is used in an environment that does not supply those services, or that the user does not trust to supply the requisite services, the application should supply its own security.

If more generic services are needed, lower-layer security protocols can supply security services to multiple applications and can do so whether or not the application has its own mechanisms for security services. Transport layer protocols such as SSL are end-to-end security mechanisms. They are appropriate when the intermediate hosts are not trusted, when the end hosts support the transport protocol, and when the application uses a connection-oriented (transport) protocol. Network layer mechanisms such as IPsec may provide security services on either an end-to-end or a link basis. They are appropriate when securing connectionless channels or when the infrastructure supports the network layer security mechanisms.

The application layer security protocol PEM provides security services for electronic mail messages. Consider using SSL for this goal. SSL does not authenticate the message to the recipient; it merely authenticates the transport connection. Specifically, if Alice sends Bob a message, PEM will authenticate that Alice composed the message and that Bob received it unaltered (and possibly that the message was kept confidential). SSL can authenticate that Alice sent the message to Bob, that it arrived as sent, and possibly that it was confidential in transit. SSL does not verify that Alice composed the message or that the message was confidential and unchanged on Alice's system or Bob's system. In other words, SSL secures the connection; PEM secures the electronic mail (the contents of the connection). Similarly, IPsec protects the packets and their contents in transit, but authentication is of the hosts and not of Alice or Bob.