Updated Computer System Testing Tool Speeds Process, Reduces Costs

NIST’s free Advanced Combinatorial Testing System undergoing further study with Lockheed Martin

Published: Wednesday, November 24, 2010 - 06:00

(NIST: Gaithersburg, MD) -- Researchers at the National Institute of Standards and Technology (NIST) have released an updated version of a computer system testing tool that can cut costs by more efficiently finding flaws.

Catching software “bugs” before a program is released enhances computer security because hackers often exploit these flaws to introduce malware, including viruses, to disrupt or take control of computer systems. But this preventive action is difficult. A widely cited May 2002 study prepared for NIST reported that even though 50 percent of software development budgets go to testing, flaws in software still cost the U.S. economy $59.5 billion annually.

Exhaustive checking of all possible combinations of input actions that could cause software failure is not practical, explains NIST’s Raghu Kacker, because of the huge number of possibilities. It’s also not necessary. Based on studies of software crashes in applications, including medical devices and web browsers, NIST’s Rick Kuhn and other researchers determined that between 70 and 95 percent of software failures are triggered by only two interacting variables, and practically 100 percent of software failures are triggered by no more than six. “Testing every combination up to six variables can be as good as exhaustive testing,” says Kacker.

Working with researcher Jeff Yu Lei and his students from the University of Texas at Arlington, NIST designed Advanced Combinatorial Testing System (ACTS), a freely distributed software tool to generate plans for efficiently testing combinations of two to six interacting variables. The method goes beyond the commonly used “pairwise” approach to software testing, which tests combinations of two variables, so it can detect more obscure flaws.

NIST’s software for testing computer systems—called ACTS—takes advantage of research showing that virtually all software failures appear to be caused by six or fewer interactions.
Credit: NIST

Kuhn describes the process “as packing as many combinations into a set of tests as efficiently as we know how.” For example, testing all possible interactions for a product with 34 on and off switches would require 17 billion tests. Using ACTS, all three-way interactions can be evaluated using only 33 tests, and all six-way combinations with just 522 tests.

The first version of ACTS was released in 2008. Since then, it has been distributed at no cost to 465 organizations and individuals in industry, academia, and government. “About half of our users are in IT, but other heavy users are in the financial, defense, and telecommunications sectors,” says Kuhn. In August, NIST and Lockheed Martin initiated a cooperative research and development agreement to study the application of ACTS in the company’s large and complex software applications. The two groups will jointly publish the results.

NIST released the latest update of ACTS in October. The new version includes an improved user interface and a better method of specifying relationships between parameters for testing. This can eliminate the problem, for example, of spending time on tests for invalid combinations, such as using Internet Explorer on a Linux system. Information for requesting ACTS is available by scrolling down this NIST web page.

Just released is a new tutorial, Practical Combinatorial Testing, which introduces key concepts and methods along with explaining the use of software tools for generating combinatorial tests. Cost and other practical considerations are addressed. The tutorial is designed to be accessible to undergraduate students in computer science or engineering and includes extensive references. NIST Special Publication 800-142 pdf is available at http://csrc.nist.gov.

About The Author

NIST

Founded in 1901, the National Institute of Standards and Technology (NIST) is a nonregulatory federal agency within the U.S. Department of Commerce. Headquartered in Gaithersburg, Maryland, NIST’s mission is to promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life.