25.6 Intrusion Response

Once an intrusion is detected, how can the system be protected? The field of intrusion response deals with this problem. Its goal is to handle the (attempted) attack in such a way that damage is minimized (as determined by the security policy). Some intrusion detection mechanisms may be augmented to thwart intrusions. Otherwise, the security officers must respond to the attack and attempt to repair any damage.

25.6.1 Incident Prevention

Ideally, intrusion attempts will be detected and stopped before they succeed. This typically involves closely monitoring the system (usually with an intrusion detection mechanism) and taking action to defeat the attack.

In the context of response, prevention requires that the attack be identified before it completes. The defenders then take measures to prevent the attack from completing. This may be done manually or automatically.

EXAMPLE: Jailing of attackers is an approach that allows the attackers to think that their attacks have succeeded, but places them in a confined area in which their behavior can be controlled and, if necessary, manipulated. Cheswick [192] used this approach to examine an attack. His system recorded a break-in attempt using the SMTP server. After several attempts to break in had failed, Cheswick created a highly restrictive account and monitored the intruder's actions, including which machines were attacked. (None of the attempts succeeded; Cheswick notified the administrators of those systems.) The jail had a file system that closely resembled a real UNIX file system (but without some programs that would reveal system information, and the deception), and access times to certain critical files were also masked. The attacker returned numerous times. Cheswick finally shut down the jail at the request of his management.

Amoroso [22] points out that multilevel secure systems are excellent places to implement jails, because they provide much greater degrees of confinement than do ordinary systems. The attacker is placed into a security compartment isolated from other compartments. The built-in security mechanisms are designed to limit the access of the subjects in the compartment, thereby confining the attacker.

More sophisticated host-based approaches may be integrated with intrusion detection mechanisms. Signature-based methods enable one to monitor transitions for potential attacks. Anomaly-based methods enable one to monitor relevant system characteristics for anomalies and to react when anomalies are detected in real time.

EXAMPLE: Somayaji and Forrest [948] extended intrusion detection using system calls (see the example that begins on page 729) to respond to suspected intrusions. They first modified the intrusion detection system to record anomalous system calls in the locality frame buffer. When the number of anomalous system calls (the locality frame count or LFC) exceeded a user-defined threshold, the system delayed the evaluation of system calls by d x 2LFC, where d was a tunable parameter. If the maximum LFC exceeded an abort_execve parameter, any attempt to spawn a child process failed. This scheme was implemented in the kernel of a Linux system. The first test examined an ssh daemon, and found that attempts to use a global password installed as a back door in the daemon were detected. In one set of experiments, the attacker's connection was slowed down significantly. In a second set, the abort_execve parameter was set to 1. This prevented the attacker from obtaining a login shell. The second experiment used sendmail, the standard Linux SMTP daemon. In those experiments, the delays that were produced quickly grew to more than 2 hours, defeating all but the most patient attacker. The performance impact of the mechanism was minimal if delays were turned off. When delays were turned on and programs were being used legitimately, the performance of system calls was substantially degraded. However, this did not appear to affect the user's view of system performance significantly. Because system calls are a small part of the runtimes most programs, this result is not surprising.

25.6.2 Intrusion Handling

When an intrusion occurs, the security policy of the site has been violated. Handling the intrusion means restoring the system to comply with the site security policy and taking any actions against the attacker that the policy specifies. Intrusion handling consists of six phases [779].

In the following discussions, we focus on the containment, eradication, and follow-up phases.

25.6.2.1 Containment Phase

Containing or confining an attack means limiting the access of the attacker to system resources. The protection domain of the attacker is reduced as much as possible. There are two approaches: passively monitoring the attack, and constraining access to prevent further damage to the system. In this context, "damage" refers to any action that causes the system to deviate from a "secure" state as defined by the site security policy.

Passive monitoring simply records the attacker's actions for later use. The monitors do not interfere with the attack in any way. This technique is marginally useful. It will reveal information about the attack and, possibly, the goals of the attacker. However, not only is the intruded system vulnerable throughout, the attacker could attack other systems.

EXAMPLE: It may be helpful to know the type of operating system from which the intruder is entering. A passive monitor can examine settings of the TCP and IP headers of incoming connections to generate a signature. For example, some systems change the window size field more often, and in different ways, than others. This signature can be compared with known signatures of operating systems, and the analyst may be able to draw some conclusions about the type of the remote system from which the packets have been generated [484].

The other approach, in which steps are taken to constrain the actions of the attacker, is considerably more difficult. The goal is to minimize the protection domain of the attacker while preventing the attacker from achieving her goal. But the system defenders may not know what the goal of the attacker is, and thus may misdirect the confinement so that the data or resources that the attacker seeks lie within the minimal protection domain of the attacker.

EXAMPLE: Stoll [975] detected an attacker in a computer system at the Lawrence Berkeley Laboratory. After a period of monitoring, Stoll concluded that the attacker was looking for documents related to nuclear weaponry. He arranged for a trace over network and telephone lines, but the tracing ended at the attacker's point of entry into the United States. The foreign authorities reported that they would need a longer connection to trace the attacker to his point of origin in Europe. Stoll created a very large file containing some of the keywords for which the attacker had been searching. When the attacker next entered, he found the file and began to upload it. The time required for the upload was more than ample for the trace to be completed, and the attacker was identified and subsequently arrested.

The document that Stoll wrote is an example of a honeypot. The file was carefully designed to entice the attacker to upload it but in fact contained false and meaningless information. This technique can be extended to systems and networks. Honeypots, sometimes called decoy servers, are servers that offer many targets for attackers. The targets are designed to entice attackers to take actions that indicate their goals. Honeypots are also instrumented and closely monitored. When a system detects an attack, it takes actions to shift the attacker onto a honeypot system. The defenders can then analyze the attack without disrupting legitimate work or systems. Two good examples are the Deception Tool Kit and the Honeynet Project.

EXAMPLE: Cohen's Deception Tool Kit (DTK) [209] creates a false network interface that allows the user of the tool kit to present any desired configuration to incoming connections. When an attacker probes the putative network, the DTK returns a wide range of vulnerabilities. The attacker may then choose some subset of the presented network addresses to attack. The defender can configure illusionary systems and servers, and monitor the attacks, so while the attacker is probing nonexistent systems the defender can analyze the attacks to determine the goals and abilities of the attacker. Based on experiments using the DTK, Cohen concluded that this technique of deception was an effective response to keep attackers from targeting real systems.

EXAMPLE: The Honeynet Project was created to learn about the "black hat" (attacker) community. The organizers were interested in the motives, techniques, and tools of the attackers. The honeypot work was split into two phases. The first was to identify common threats against specific operating systems and configurations. The second was to develop a honeypot network that was "easier to deploy, harder to detect, and more efficient in collecting data." The group has written several papers about the attacks and the attackers [483, 485, 486, 488, 489], and about the design of the honeynet [487].

25.6.2.2 Eradication Phase

Eradicating an attack means stopping the attack. The usual approach is to deny access to the system completely (such as by terminating the network connection) or to terminate the processes involved in the attack. An important aspect of eradication is to ensure that the attack does not immediately resume. This requires that attacks be blocked.

A common method for implementing blocking is to place wrappers around suspected targets. The wrappers implement various forms of access control. Wrappers can control access locally on systems or control network access.

EXAMPLE: Wrappers that control local access to resources are usually embedded in the kernel to make them difficult to bypass. In an experiment that used wrappers to improve the security of commercial off-the-shelf programs, Fraser, Badger, and Feldman [372] used loadable kernel modules to place wrappers in the kernels of UNIX systems. When the wrappers were invoked, they waited for some specified event (such as a system call, possibly with particular privilege settings or arguments). When the event occurred, the wrapper would take control of the process and perform a specified action. The action could be to log the call, to deny access (by returning a failure code to the caller), or to generate and process auxiliary data such as system call counts. The wrappers were specified using an extension of the C programming language. The performance impact of using the wrappers was less than 7%. The researchers noted that the wrappers' uses were varied, ranging from access control and auditing to intrusion detection and response. Others [581] focused on the latter, designing wrappers that would detect intrusions. Their mechanism accepted notifications from multiple wrappers. In one experiment, when two wrappers notified a wrapper monitoring program execution that a process appeared to be launching an attack, the monitoring wrapper terminated the process.

EXAMPLE: Wrappers can also control access from the network. Bina, McCool, Jones, and Winslett [99] describe an application in which a Web server accepts requests for database records and returns the desired records if so authorized. Access to the records is determined by the role of the requester. To determine this, the Web server obtains information from the client (including a public key for authentication) and passes the data to a script that assigns the appropriate role to the request. The role and request are given to the database engine, which returns an appropriate response. The script is a wrapper around the database. It mediates access to the database.

Firewalls (see Section 26.3.1) are systems that sit between an organization's internal network and some other external network (such as the Internet). The firewall controls access from the external network to the internal network and vice versa. The advantage of firewalls is that they can filter network traffic before it reaches the target host. They can also redirect network connections as appropriate, or throttle traffic to limit the amount of traffic that flows into (or out of) the internal network.

EXAMPLE: Because Java applets (see Section 4.5.1) come from (usually) untrusted sources, many organizations want to block the applets from entering their internal networks. A simple method of doing this is to block the applets at a firewall [662]. When an HTTP connection is made through the firewall, the firewall creates a small application (called a proxy) to reassemble the packets and determine if they contain a Java applet. The proxy then may use one of three approaches to block the applet. First, it can rewrite the HTML tag to something other than "<applet>". When the page is delivered to the browser, the browser will not recognize the applet and will not run it. This method requires the firewall to determine that the connection is indeed an HTTP connection and to parse the HTML in that connection. Both are nontrivial tasks. The second approach is to look for incoming files with the hexadecimal sequence CA FE BA BE. All Java class files must contain this four-byte signature in order to be properly recognized and interpreted. If this sequence is found, the file is immediately discarded. The danger here is a false positive. Because ActiveX and Javascript code are different, this approach cannot block those types of applets. The third approach is to block based on file name, but this is far more problematic because the names do not necessarily represent the contents of the file. Many browsers require Java class files to end in ".class". The firewall can block these applets. However, more recent browsers allow Java class files to be combined into archives. The names of these archives often end in ".zip". This is a popular format among users of MS-DOS and Windows, so it is not realistic to block all such files. Martin, Rajagopalan, and Rubin [662] conclude that the situation is rather bleak for stopping Java applets at the firewall.

An organization may have several firewalls on its perimeter, or several organizations may wish to coordinate their responses. The Intruder Detection and Isolation Protocol [887] provides a protocol for coordinated responses to attacks.

The IDIP protocol runs on a set of computer systems. A boundary controller is a system that can block connections from entering a perimeter. Typically, boundary controllers are firewalls or routers. A boundary controller and another system are neighbors if they are directly connected. If they send messages to one another, the messages go directly to their destination without traversing any other system. If two systems are not boundary controllers and can send messages to each other without the messages passing through a boundary controller, they are said to be in the same IDIP domain. This means that the boundary controllers form a perimeter for an IDIP domain.

When a connection passes through a member of an IDIP domain, the system monitors the connection for intrusion attempts. If one occurs, the system reports the attempt to its neighbors. The neighbors propagate information about the attack and proceed to trace the connection or datagrams to the appropriate boundary controllers. The boundary controllers can then coordinate their responses, usually by blocking the attack and notifying other boundary controllers to block the relevant communications.

EXAMPLE: Kahn and Zurko [535] discuss the use of IDIP to handle network flooding attacks, in which one or more sources spew large numbers of packets to a target. This effectively prevents legitimate traffic from being processed, either because the target is overwhelmed with processing the flooding packets or because the legitimate traffic cannot reach the destination (target). Consider Figure 25-8. Suppose host f launches a flooding attack against host A along the path f, Z, Y, X, W, a, A. The flood effectively stops all traffic along that path. Host a detects the flood and begins blocking traffic for host A. It also notifies its neighbor W, a boundary controller. W detects traffic targeting A, suppresses it, and notifies its neighbor X. X detects the traffic targeting A, suppresses it, and notifies its neighbors Y and C. W then notices the traffic for A has stopped, and it eliminates its suppression. At this point, A, a, W, and b can again communicate freely, because the traffic formerly saturating the links has been eliminated by X. C detects no traffic for A and so does nothing. Y does detect the traffic, and suppresses it. X detects that the traffic going through it for A has stopped, and X eliminates its suppression. Y then communicates with Z, and Z detects and suppresses the traffic. Y also communicates with D, which detects no relevant traffic. This process continues until all traffic from f to A is suppressed. Figure 25-8. Example of IDIP. C, D, W, X, Y, and Z are boundary controllers. Host a runs the IDIP protocol but is not a boundary controller. The flooding attack follows the dashed arrows from f to A. The IDIP protocol is flexible, because if multiple sources attempt to flood a host, the boundary controllers will block the traffic along each path that the sources use. Of course, if any path has no IDIP controllers, the traffic can flow freely along that path. Kahn and Zurko suggest that IDIP, or a similar protocol, should be widely deployed throughout the Internet to handle flooding attacks. They argue that economic and other incentives will encourage Internet Service Providers and other network providers to cooperate in suppressing distributed flooding attacks.

25.6.2.3 Follow-Up Phase

In the follow-up phase, the systems take some action external to the system against the attacker. The most common follow-up is to pursue some form of legal action, either criminal or civil. The requirements of the law vary among communities, and indeed vary within communities over time. So, for our purposes, we confine ourselves to the technical issue of tracing the attack through a network. Two techniques for tracing are thumbprinting and IP header marking.

Thumbprinting [460, 964] takes advantage of connections passing through several hosts. An attacker may go from one host, through many intermediate hosts, until he attacks his target. If one monitors the connections at any two hosts that the connections pass through, the contents of the connections will be the same (excluding data added at the lower layers). By comparing contents of connections passing through hosts, one can construct the chain of hosts making up the connections.

Staniford-Chen and Heberlein [964] list five characteristics of a good thumbprint.

There are several possible sources of error (see Exercise 8).

EXAMPLE: Staniford-Chen and Heberlein [964] experimented with thumbprints made up of linear combinations of character frequencies in telnet and rlogin connections. The thumbprints were computed over a set of connections drawn from normal network traffic. First, a control experiment checked that thumbprints were unlikely to match randomly paired connections. Out of 4,000 pairings, only one match was reported. On inspection, the two connections had identical contents over the period of thumbprinting (a prompt and a logout command). Next, they computed thumbprints from connections passing through multiple hosts (one thumbprint per host for each connection) on a local area network. These thumbprints were injected into a collection of thumbprints made at the same time. Comparison identified these thumbprints as belonging to the same connections. Experiments on long haul networks (across the United States and the Atlantic Ocean) also showed that the comparison procedure was able to find the connections correctly.

An alternative approach is to ignore the contents of the packets and examine the headers. IP header marking does just this. A router places extra information into the IP header of each packet to indicate the path that the packet has taken. This information may be examined in order to to trace the packet's route back through the Internet [880].

The keys to IP header marking are selection of the packets to mark, and marking of the packets. Packet selection may be deterministic or probabilistic. Packet marking may be internal or expansive.

Deterministic packet selection means that packets are selected on the basis of a deterministic, nonrandom algorithm. For example, every second packet may have the router's IP address inserted as the marking. In general, deterministic packet selection is too expensive and unreliable. It is unreliable because an attacker can enter false data into the header area and prevent the marking (see Exercise 10). In probabilistic packet selection, packets are selected with some given probability.

Internal packet marking places the router's marking in the packet header without expanding it. For example, Dean, Franklin, and Stubblefield [263] have identified several bits in an IPv4 header that could be used for marking. Expansive packet marking means that the packet header is expanded to include extra space for the marking.

EXAMPLE: Doeppner, Klein, and Koyfman [305] suggest a probabilistic, expansive packet marking scheme. They propose adding space ("slots") for s markings. The probability that a packet will be marked is p. The following algorithm describes how the router handles an incoming packet. /* generate random number between 0, 1 */ x = random(0, 1); /* stamp the packet appropriately */ if x < s * p then slot[x/p] = router's stamp; Note that the slot into which the router's marking is placed depends on the random number generated. The markings can enable one to trace certain attacks back to their sources. As an example, consider the SYN flood attack [255], in which an attacker generates a large number of TCP SYN packets (see Section 26.4). The target receives them and sends the SYN/ACK packet in the second step of the three-way handshake (see Section 24.4.2). The attacker never sends the third packet, so the connection is pending until it times out. At that point, the target grabs the next incoming SYN packet and repeats the cycle. Because the attacker is flooding the target with SYN packets, the probability of any other host's SYN packet initiating the three-way handshake is very low. The attack therefore denies service to all other hosts. Consider the network in Figure 25-9. Host E experiences a SYN flood. The administrators identify 3,150 packets that could be a result of the attack. Of these packets, 600 have the route (A, B, D), 200 have the route (A, D), 150 have the route (B, D), 1,500 have the route (D), 400 have the route (A, C), and 300 have the route (C). Counting, there are 1,200 packets with A's mark, 750 with B's mark, 700 with C's mark, and 2,450 with D's mark. Assuming that the probability of marking is the same on each router, the number of packets that D received is more than three times as great as the number that B received. Thus, B is probably the source of the flooding. Figure 25-9. Example network. Routers A, B, C, and D mark packets. The destination host E is experiencing a SYN flood. The administrators at host E use the markings to identify the flooder.

EXAMPLE: Dean, Franklin, and Stubblefield [263] employ an algebraic technique to encode suffixes of the paths taken by packets. Evaluating an nth-degree polynomial requires n + 1 data points. Consider the packets being sent from host A to host B along path P. The first router in path P labels the jth packet with the integer xj. Let the routers' IP addresses on path P be a0, ..., an. Then, when the jth packet arrives at B, its marking will be the value of the polynomial f(x) = a0xn + ... + an at xj. When n + 1 packets along that path arrive, the coefficients of the polynomial can be determined. Note that the routers can use Horner's rule [578] to evaluate the polynomial, so they need not know the path P. Router ai needs to know a0 xji + ... + ai–1. It then computes (a0 xji + ... + ai–1)xj + ai. This approach has several problems. First, recording the entire path requires that a router know it is the first router on path P so that it can assign xj and begin computing the polynomial. This is infeasible given the current structure of the Internet. One approach is to select a number from a weighted random distribution of integers to determine if the router is the first on P. If so, it assigns xj; if not, the router evaluates the polynomial as described above. Furthermore, if xj is assigned, the router assigns similar values to the next k packets (k being a tunable parameter). This increases the number of packets needed to reconstruct the full path. Moreover, an attacker can place arbitrary information into the marking, so if the router does not select the packet for marking, the erroneous information is passed along. Ultimately, the destination will not be able to distinguish the erroneous information from legitimate packet markings. An alternative approach modifies the scheme so that at most l routers mark the packet. The first router sets this parameter; each marking router decrements it by 1. This scheme records subsequences of P, rather than suffixes of P or the entire path. This keeps the degree of the polynomials small (of degree l – 1 at most). Dean, Franklin, and Stubblefield used this last scheme to mark packets. The bits of the values of the polynomials were distributed over 11 bits in the IP header. In their simulations, they analyzed 20,000 packets and recovered paths of length 25 more than 98% of the time, demonstrating the efficacy of their scheme.

Counterattacking, or attacking the attacker, takes two forms. The first form involves legal mechanisms, such as filing criminal complaints. This requires protecting a "chain of evidence" so that legal authorities can establish that the attack was real (in other words, that the attacked site did not invent evidence) and that the evidence can be used in court. The precise requirements of the law change over time and jurisdictions, so this first form of counterattacking lies outside the scope of this discussion. The second form is a technical attack, in which the goal is to damage the attacker seriously enough to stop the current attack and discourage future attacks. This approach has several important consequences that must be considered.

Under exceptional circumstances, counterattacking may be appropriate. In general, it should be avoided, and legal avenues of prosecution (either civil or criminal) should be pursued. Improving defenses will also hinder attacks. The efforts used to develop and launch counterattacks could be spent far more effectively in that way.

EXAMPLE: Recall the example of the two versions of the animal game (see page 586). In that case, the new version of animal targeted a specific, older version written by the same authors, and it was unlikely that any organization depended on the existence of that game. Consider moving this example into the world of distributed systems and networks. Imagine a computer worm that enters systems through a widely used network server. The worm spreads rapidly, and despite attempts to eradicate it, systems continue to be reinfected. One company designs a "counterworm." Whenever a break-in comes from a remote site, the "counterworm" detects the break-in, deletes the connection, and uses the same infection technique as that of the original worm to enter the attacking host. On that host, it deletes all worm processes (except its own). It then waits until that system is attacked, and the cycle repeats. This response raises several questions. First, how can the "counterworm" be set to ensure that it deletes only those processes belonging to the original worm? Second, what if the invaded machine is gathering data for research or countermeasures? Third, how can the originators of the "counterworm" ensure that it does no damage to any system it is sent to? Fourth, can they be held legally liable for any problems that a site encounters if that site is sent the "counterworm"? The answers to these questions are complex, and illustrate clearly why one needs informed, full consent of a remote site before sending an automated response.