Automated Methods for Creating Diversity in Computer Systems

The pervasive homogeneity of computer systems attached to the Internet, combined with the ease of attacking multiple identical systems once one machine is compromised, represents a serious security threat. A possible response to this situation can be found using biological diversity as inspiration. In nature, diversity provides a defense against unpredictable threats by maximizing the probability that some individuals will survive and replenish the population with a defense against that particular threat. Diversity in computer systems could confer security benefits by protecting against attacks that rely on known regularities.

A diversity defense can render a standard attack ineffective or slow it down, depending on its placement and implementation. Diminishing the uniformity in existing systems is, however, a non-trivial task, as standardization must be maintained at many interface points in any given system. This dissertation intends to assess the costs and benefits of adding diversity to existing computer systems by implementing diversity at different levels. Diversification in computer systems can be accomplished at the interface or the implementation level. In general, an interface diversification changes function labels making them unique to a given system. In contrast, an implementation diversification modifies function behavior to prevent locking into idiosyncratic states. Three techniques to introduce automated diversity in existing systems are presented: one at the interface and two at the implementation levels. Their effectiveness at stopping or slowing down attacks is studied.

The first diversity scheme presented is an interface diversification: a machine language randomization, named Randomized Instruction Set Emulation (RISE). RISE is intended as a protection against the threat of code-injection attacks, which insert malicious machinelanguage code into programs. Code-injection attacks constitute one of the most prevalent threats on current networks, and its most famous examples are the so-called buffer overflow attacks. RISE protects against all code-injection attacks regardless of their point of entry by creating a unique machine code per process. The current RISE implementation runs over an emulator, and maps all executable bytes of the process to a random mask. When injected code attempts to execute, its code is also mapped to the mask, but given that it was not correctly encoded, it is ‘decoded’ to random bytes. Though the attack will not execute as intended, there is a small probability that the attack will manage to execute some random instructions. This work also offers an analysis of the risks associated with the execution of random instructions.

Many Denial of Service (DoS) attacks exploit a system’s implementation rather than its interface. Two approaches to diversify an implementation are explored. The first approach randomizes internal protocol parameters within acceptable ranges. It is tested against an attack targeting one of the TCP congestion control parameters. The diversification achieves the objective of keeping a portion of the hosts in the attacked network operating at larger bandwidths than if they were all using the same standard parameter values.

The second implementation diversification targets Denial of Service attacks by re source exhaustion. The diversity solution used creates a unique filter per host that, for a given attack traffic pattern, passes most of the legitimate traffic and blocks attacking requests. Filters are created using Genetic Programming on different attack patterns. Current results suggest that it could be effective when used as a front-end for resource managers that are non-preemptive in nature.