OUSPG

[This page is CSS2 enabled. Your browser might not fully support it]

Test-suite releases in Theory and Practice

$RCSfile: index.html,v $ $Revision: 1.43 $ $Date: 2004/01/16 21:07:44 $

ABSTRACT

Trivial vulnerabilities foster in protocol implementations. Same kinds of bugs keep reappearing over and over again. The PROTOS test-suite release is an attempt to raise the bar by establishing a baseline. Test-suites aimed against a specific set of vulnerabilities are created for the chosen protocol implementations. Stand-alone test-suites are created for utilization in customer evaluation, during vendor development or as regression tests. Test-suites are published with supporting background material.

Table of Contents

• ABSTRACT
• Table of Contents
• Introduction
• Release policy and process
  • Goals
  • Limitations
  • Release process
  • Test-material package and naming
    • Cycles 01-09
  • License and copyright

Introduction

The PROTOS test-suite releases aim to provide material to both customers and vendors for evaluating software implementations for some trivial information security related vulnerabilities and for robustness.

The test-suite releases are a byproduct of the "PROTOS - Security Testing of Protocol Implementations" project. The PROTOS project is a government funded research partnership between University of Oulu and VTT Electronics. The PROTOS project is supported by two partner companies from the telecommunication industry.

A test-suite is created for a subset of a specific protocol. The test-suite consists of test-cases and code needed for evaluating the system under test against the test-suite. Protocol data units have been constructed using the principles described in "Software vulnerability analysis through syntax testing".

The software vulnerabilities that are likely to be found with the test-suites have been caused by implementation level mistakes. A typical category of these types of vulnerabilities is reviewed in "Running Malicious Code By Exploiting Buffer Overflows: A Survey Of Publicly Available Exploits"

The terminology used herein and in the test-suite releases has been adopted from definitions available from the "Glossary of Vulnerability Testing Terminology" document.

The procedures for handling and reporting any internally found vulnerabilities have been documented in "The Vulnerability Process: a tiger team approach to resolving vulnerability cases". We recommend following these guidelines with any externally found vulnerabilities as well.

The PROTOS test-suite concept follows the constructive disclosure model as illustrated in the "Introducing constructive vulnerability disclosures" conference paper.

Release policy and process

Goals

A baseline for new implementations of the protocol

A test-suite aims to set a baseline that eliminates a specific subset of errors that may infest a new implementation of the protocol. The test-suite may be used early in the development process for early elimination of some of the trivial pitfalls.

Acceptance testing

When customers evaluate a commercial off-the-shelf product that implements the protocol, they may use the test-suite as a criteria for accepting the product. The same applies to a custom made product, although a more effective net outcome may be achieved by requiring the ability to survive the test-suite in the requirements specification phase.

Product evaluation

Test-suite may be used as a supplementary metric for evaluating several competing products in process of purchase decision making.

Regression testing

If the test-suite is used in the development process, it may be adopted as a part of the regression test-suite to ensure that the errors covered by the test-suite do not emerge during code maintenance.

Security community impact

The test-suite should be able to eliminate some of the most trivial vulnerabilities that may otherwise be found only by manual trial and error. This may reduce the time spent on the most trivial vulnerabilities and free up some resources of security community and end users, to be spent on more fundamental (information security related) issues.

Vendor awareness

A test-suite, prerelease process and the background material aims to help the vendors that are new to vulnerability scene in the early phases of the learning curve in a more controlled way than a full disclosure on the mailing lists might provide.

Remedy on the field

Eventually customers themselves and consultants acting for them will be able to check for vulnerability residual in their live configurations.

Limitations

Usual limitations for black-box testing and testing in general apply. Passing the test-suite is in no way a certificate for a vulnerability-free system.

Test-suites are provided as a proof-of-concept only. They are in no way supported, sold as a service, or otherwise guaranteed to fill any particular need.

In typical case, the test-suites are created for a very specific subset of the chosen protocol and for very specific types of errors. This means that only portion of the system under test is exercised and only a narrow subset of over-all security of the system is addressed.

Test-material contains no arbitrary code exploits. However, running the test-material against production systems is strongly discouraged. All failure modes caused by the material may not be transient.

Release process

Three major phases of the test-suite release process are sketched below:

1. Creation
   • Selection of the protocol and the focus area
   • Creating and wrapping-up the test-suite
   • Internally testing the available implementations
2. Prerelease
   • Notifying respective vendors of any vulnerabilities found
   • Distributing the test-suite to identified vendors implementing the chosen protocol
   • Vulnerability and advisory coordination
   • Grace period
3. Release
   • Deploying the test-suite for public perusal
   • Collecting feedback
   • Reiterating either with same or next protocol

Test-material package and naming

Cycles 01-09

Test-material is distributed as a JAR-package. This package comprises of the following elements:

• Individual test-cases (PDUs) as data files
• Java code (source and compiled) for feeding the test-cases against the system under test
• License and README files

Downloadable test-material packages are named as <cycle>-<test-suite>-<status><version>.jar , where:

• <cycle> is the iteration of the testing approach used in the test-suite.
• <test-suite> is the test-suite name, possibly in shortened form, identifying the subject protocol.
• <status> indicates either prerelease phase [pr] or release phase [r].
• <version> is a positive integer, starting from one incremented by one for each maintenance release made to fix bugs in the test-material package itself.

For example: c04-wap-pr1.jar

License and copyright

The test-material is licensed under GNU General Public License (GPL) version 2, at no charge. This is done in order to ensure that vendors and their customers may freely utilize the test-material. Standard GPL terms for no warranty and no liability apply.

We recommend some additional guidelines, although these do not restrict the test-material license:

During the prerelease phase and the grace period

• Vendors should concentrate on testing their own implementations
• Testing should not be conducted in or over the public network
• Refraining from redistribution is dearly recommended
• Heed to recommendations herein for handling the found vulnerabilities
• Provide feed-back to the PROTOS Working Group directly

After the grace period and public release

• If vulnerabilities are found with the test-material, please consider a professional and responsible approach to the vulnerability and advisory process
• If you modify the test-material, please consider renaming the package
• Provide feed-back to the PROTOS Working Group

A test-suite as whole contains the test-material package and related documentation from the PROTOS web pages. All material is under Copyright (C) 2000 - 2004 PROTOS Project Consortium [http://www.ee.oulu.fi/research/ouspg/protos/]. Only the test-material package is placed under the GPL license, normal restrictions apply to all other material.

[This page is CSS2 enabled. Your browser might not fully support it]