roottest suite

From: Philippe Canal <pcanal_at_fnal.gov>
Date: Thu, 02 Mar 2006 01:59:27 -0600

Hi,

We have been maintaining and developing a test suite for root for some time now. It is currently run nightly on linux and macos.

The result are now yet available on the web but we plan on using QMTest to make them available.

Also fell free to help improve and extend the test suite with your own your own test.

You can download read-only it via:

   cvs -d :pserver:cvs_at_root.cern.ch:/user/cvs co roottest

To run it just do:

   cd roottest
   gmake

The output will be something like:

   Running test in ./cint/array

   Tests in ./cint/array .............................................. OK

   Running test in ./cint/const

   Tests in ./cint/const .............................................. OK

   Running test in ./cint/controls
   Warning Loopbreak has some known skipped failures (in ./controls)

   Tests in ./cint/controls ........................................... OK

   ....

   Tests in . ......................................................... OK

The message "Warning xyz has some known skipped failures" is reminder of a known failure.

An unexpected failure will look like:

   Running test in ./cint/template
   2a3
> Warning in <TClassTable::Add>: class TParameter<Long64_t> already in
TClassTable

   gmake[2]: *** [longlong] Error 1

   Test in ./cint/template ........................................... FAIL

This says that the test named 'longlong' in the directory roottest/cint/template
has failed because the script issues the unexpected message

> Warning in <TClassTable::Add>: class TParameter<Long64_t> already in
TClassTable

For a given test the expected screen output is in testname.ref (in the above example the file name is roottest/cint/template/longlong.ref). The actual is output is located in roottest/cint/template/longlong.log.

There are also .log files containing the output of the compilation of (when applicable) of auxiliary files.

To see what the are the command actually issued do:

   cd roottest; gmake V=t (or gmake VERBOSE=t) (or export V=t; gmake )

To add a new test

   cd roottest; addtest pathname Testname

For example:

   cd roottest; addtest vectorOfvector root/io/newstl

add a new test name vectorOfvector in the directory roottest/root/io/newstl. In particular this will update (or create) the makefile and generate 2 files:
runTestname.C and runTestname.ref.

To actually implement your test, just modify the file runTestname.C

If you want the runTestname.C to being compiled instead of interpreted, modify
in the makefile the line:

   Testname: testname.log
into

   Testname: testname.clog

Then the makefile will run

   root.exe -q -b -l runTestname.C > runTestname.log 2>&1 or

   root.exe -q -b -l runTestname.C+ > runTestname.clog 2>&1 And

   diff -bw runTestname.ref runTestname.log

Failure of test can be indicated by a change in the output or by an error return code.

Library can be build automatically using ACLiC. For example, if there is a Testname.C that needs to be compiled just add

   Testname: Testname_C.$(DllSuf)
To the local Makefile

There are other helpful macros in roottest/scripts/Rules.mk

A few things about the test suite:

The directory organization.

   scripts : some helper scripts for the makefile    root : ROOT specific test
   cint : CINT test
   python : PyROOT test

Design goals:
  Adding test (event compiled) must be very simple   Running subset of the test must be very simple   Running a single test or part of a test must be very simple

     It needs to be very easy to know how to run the test in gdb   Flexibility of testing

     both test return value and output can be tested   Speed of re-run

     An additional run after a successful run should 
     be very fast (i.e. only really re-run the test 
     when something has changed)
  Highly portable
     Mostly by using ACLiC for compilation, a few legacy 
     tests are still building their own executable

  Integration with QMS
  Integration of CPPUnit

Thanks,
Philippe. Received on Thu Mar 02 2006 - 09:03:01 MET