Chapter 5. HPI Conformance Test Design Guide

Hpi conformance tests are open for participation to the open source community. Anyone can join and contribute. Because of this, we need to set some guidelines to allow for easier maintenance and uniformity.

Each of the tests will be written in 'C' and only comprise of one module (file). In the comment header of the file, the module name, copyright statement, and a description of the test case is required.

(TBD) A particular format for the name and the test case description may be required in the future. This is to support a self-documenting source parsing code to add to the ease of updating this document.

SHARED TEST CODE

So far, the conformance tests don't share a library of code. There is a header file (hpitest.h) which is to be included into all the tests which includes some static inline functions for the test writer to use. Inspection of the other tests would reveal the usage and benefits of using these functions.

Also included in the header file are the test return codes for all of the tests.

TEST RETURN CODES

There are 4 return codes supported in the test suite, and these return codes are defined in the hpitest.h header file.

• HPI_TEST_PASS

  This return means that the test criteria was fully met and the test case ran without errors.

• HPI_TEST_FAIL

  This return means that the test criteria was fully met, but the test failed when running the test case. Additional information on the failure was sent to stdout. This failure information will be stored into a logfile.

• HPI_TEST_UNKNOWN

  This return means that the test was not able to run for some reason, or that the test exited without any determined reason. This should be the default return for all test. If a test quits with this return, it is probable there is a problem with the test or test conditions, but not the test case.

• HPI_TEST_NA (non-applicable)

  This return means that the test was unable to set up the proper test conditions needed for the test case to execute. The HPI target implementation may not have a valid resource or domain to test this particular test case against. This return could still reflect a problem in the HPI implementation if the test case should have been able to execute against the current target.

  Any failures generated by clean-up functions, after the test case has completed, will not change the test return status. Test cases dedicated for the clean-up functions will report the problems. The intention of this is to focus the result of the test to just the test case. Otherwise, we would have many failures, in a wide range of test cases, which don't actually relate to the cause of the problem.

TEST FAILURE INFORMATION

All tests returning HPI_TEST_FAIL must print out why the test failed. This information should be targeted to stdout. (printf are suggested). This information will be captured in the test execution process and sent to a logfile.

The makefile is the primary means for running the tests. For every source file, it will compile, link and execute. Statistics will be captured from the execution of the tests and stored in a logfile under the log directory.