17.2 Isolation

Systems isolate processes in two ways. In the first, the process is presented with an environment that appears to be a computer running only that process or those processes to be isolated. In the second, an environment is provided in which process actions are analyzed to determine if they leak information. The first type of environment prevents the process from accessing the underlying computer system and any processes or resources that are not part of that environment. The second type of environment does not emulate a computer. It merely alters the interface between the existing computer and the process(es).

17.2.1 Virtual Machines

The first type of environment is called a virtual machine.

Definition 17–4. A virtual machine is a program that simulates the hardware of a (possibly abstract) computer system.

A virtual machine uses a special operating system called a virtual machine monitor to provide a virtual machine on which conventional operating systems can run. Chapter 33 discusses virtual machines in more detail.

The primary advantage of a virtual machine is that existing operating systems do not need to be modified. They run on the virtual machine monitor. The virtual machine monitor enforces the desired security policy. This is transparent to the user. The virtual machine monitor functions as a security kernel.

In terms of policy, the virtual machine monitor deals with subjects (the subjects being the virtual machines). Even if one virtual machine is running hundreds of processes, the virtual machine monitor knows only about the virtual machine. Thus, it can apply security checks to its subjects, and those controls apply to the processes that those subjects are running. This satisfies the rule of transitive confinement.

EXAMPLE: The KVM/370 was a security-enhanced version of the IBM VM/370 virtual machine monitor [401]. This system provided virtual machines for its users, and one of its goals was to prevent communications between virtual machines of different security classes, so users in different security classes could use the system at the same time. Like VM/370, it provided virtual machines with minidisks and allowed systems to share some areas of disk. Unlike VM/370, it used a security policy to mediate access to shared areas of the disk to limit communications between systems.

EXAMPLE: Karger and colleagues at Digital Equipment Corporation developed a virtual machine monitor (VMM) for the DEC VAX [554]. The monitor is a security kernel and can run either the VMS or the Ultrix operating system. The VMM runs on the native VAX hardware and is invoked whenever the virtual machine executes a privileged instruction. Its structure is typical of virtual machines designed to provide security. The VAX has four levels of privilege: user, supervisor, executive, and kernel modes. In order to provide a compatible virtual machine, the virtual machines must also have four levels of privilege. However, the kernel mode allows a process to access privileged instructions on the VAX hardware directly. Only the VMM is allowed to do this. The virtual machines cannot access kernel mode. The solution is to provide virtual modes. These modes are VM user (corresponding to user mode), VM supervisor mode, and VM executive and VM kernel modes (both actually executive mode).[1] The VMM subjects are users and virtual machines. VMM has a basic, flat file system for its own use and partitions the remaining disk space among the virtual machines. Those machines may use any file structure they desire, and each virtual machine has its own set of file systems. Each subject and object has a multilevel security and integrity label, and the security and integrity levels form an access class. Two entities have the same access class if and only if their security and integrity labels are the same, and one entity dominates another if and only if both the security and integrity classes dominate. An integral component of the VMM is an auditing mechanism. This mechanism records actions for later analysis.

^[1] Chapter 33, "Virtual Machines," discusses this approach in more detail.

Because virtual machines provide the same interface for communication with other virtual machines that computers provide, those channels of communication can be controlled or severed. As mentioned earlier, if a single host runs multiple virtual machines, those virtual machines share the physical resources of the host on which they run. (They may also share logical resources, depending on how the virtualizing kernel is implemented.) This provides a fertile ground for covert channels, a subject explored in Section 17.3.

17.2.2 Sandboxes

A playground sandbox provides a safe environment for children to stay in. If the children leave the sandbox without supervision, they may do things they are not supposed to do. The computer sandbox is similar. It provides a safe environment for programs to execute in. If the programs "leave" the sandbox, they may do things that they are not supposed to do. Both types of sandboxes restrict the actions of their occupants.

Definition 17–5. A sandbox is an environment in which the actions of a process are restricted according to a security policy.

Systems may enforce restrictions in two ways. First, the sandbox can limit the execution environment as needed. This is usually done by adding extra security-checking mechanisms to the libraries or kernel. The program itself is not modified. For example, the VMM kernel discussed earlier is a sandbox because it constrains the accesses of the (unmodified) operating systems that run on it. The Java virtual machine is a sandbox because its security manager limits access of downloaded programs to system resources as dictated by a security policy [170].

EXAMPLE: The operational kernel of the Sidewinder firewall [996] uses type enforcement to confine processes (see the example on page 398 in Section 15.3.1). This is an example of a sandbox built into a kernel, and it has the property that the sandbox is defined by the vendor. It is not intended to be altered at the site. Such a design is typical for a turnkey system, which is the intended use for a Sidewinder firewall. The Java virtual machine, in which downloaded applets are executed, is another example of a sandbox. The sandbox restricts the set of files that the applet can access and the hosts to which the applet can connect. Other security mechanisms enhance the sandbox [407]. DTE, the type enforcement mechanism for DTEL (see page 106 in Section 4.5), is an example in which kernel modifications enable system administrators to configure their own sandboxes. The kernel enforces the constraints.

The second enforcement method is to modify the program (or process) to be executed. Dynamic debuggers [8, 326, 442, 962] and some profilers [108] use this technique by adding breakpoints to the code and, when the trap occurs, analyzing the state of the running process. A variant, known as software fault isolation [934, 1028], adds instructions that perform memory access checks or other checks as the program runs, so any attempt to violate the security policy causes an error.

EXAMPLE: Janus [402] implements a sandbox. It is an execution environment in which system calls are trapped and checked. Users execute it to restrict the objects and modes of access of an untrusted program. Janus consists of a framework, which does the runtime checking, and modules, which determine which accesses are to be allowed. Janus first reads a configuration file. This file instructs it to load certain modules. Along with the module identification is a list of constraints. The following example configuration file defines the environment variable IFS for the child and restricts the child's access to the file system. The child cannot access any files except those that are named below (this meets the principle of fail-safe defaults discussed in Section 13.2.2). The child can read or write to any file in the /usr file system except for those in the /usr/lib and /usr/local/lib directories (which are read only) and in /usr/bin (read and execute). The child can read any file in the /lib directory and can read and execute any file in the /sbin and /bin directories. In the configuration file below, the first word in each instruction line is the name of the module and the other words are the arguments passed to the modules ("#" begins a comment). # basic module basic # define subprocess environment variables putenv IFS="\t\n " PATH=/sbin:/bin:/usr/bin TZ=PST8PDT # allow access to everything except files under /usr path deny read,write * path allow read,write /usr/* # allow subprocess to read files in library directories # needed for dynamic loading path allow read /lib/* /usr/lib/* /usr/local/lib/* # needed so child can execute programs path allow read,exec /sbin/* /bin/* /usr/bin/* Each module constrains system calls. The framework uses the modules to build a linked list for each monitored system call. The list defines allowed and disallowed actions. Once this list has been constructed, the Janus framework invokes the program in such a way that all monitored system calls are trapped. When the program executes a monitored system call, the program traps and the Janus framework is invoked. It has access to the arguments supplied to the sysem call. It validates that the system call, with these specific parameters, is allowed. If the system call is not allowed, the framework sets the child's environment so that the system call appears to have failed. If the system call is allowed, the framework returns control to the child, which in turn passes control to the kernel. On return, control goes to the framework, which updates any internal state and returns the results to the child. An example use would be in reading MIME mail. One could have set the mail reading program to pass control to a Postscript display engine. Some such engines have a mechanism for executing system-level commands embedded in the Postscript file. Hence, an attacker could put a file deletion command in the Postscript file. The recipient would run the display engine to read the file, and some of her files would be deleted [215]. However, the user (or system administrator) can set up the Janus configuration file to disallow execution of any subprograms. Then the embedded command will be detected (on the system call to execute it) and rejected.

Like a virtual machine monitor, a sandbox forms part of the trusted computing base. If the sandbox fails, it provides less protection than it is believed to provide. Hence, ensuring that the sandbox correctly implements a desired security policy is critical to the security of the system.