Hi all,
On Wed, 2006-03-01 at 15:38 -0600, Philippe Canal wrote:
> Hi Daniele,
>
> We absolutely agree that having a test framework is essential
> to our development. For this purpose we have a separate cvs
> module (roottest, see http://root.cern.ch/viewcvs/?cvsroot=TEST).
It's good to see that you have a testsuite set up. Even better that you advertised the URL :-)
> We attempt to add as many test as possible and made the framwork
> so that it is trivial to add new test (see the README in roottest).
As far as I can gather, most of your tests are in the form of a script (possibly compiled), and have semantics like `please test that we can fit this kind of data, or we can do this and that.' A `real' unit test suite, is more like `for this class/module/... do tests of all functionality of this class/module/...'. That is, if you have class
class Foo
{
...
bool Bar();
void Baz();
...
};
make sure that `Foo::Bar' evaluates to `true' in a representative number of cases where it should do that, and in cases where it should return `false' make sure it returns that, and similar for `Foo::Baz'.
So, in essence, a unit test suite tests each `function' separately as well as collections of `functions'. The idea is to factor the problem of testing into small easily understood chunks. More on all this at f. ex. [1]
[1] http://www.codeproject.com/csharp/autp1.asp
> > The easy solutin to the situation is to have unit tests that cover
> > all the functionalities of root and run them often on the development trunk.
`The easy solutin ... all the functionalities of root ...' - Whow, `easy' and `all the functionalities' in one sentence!? That's got to be an oxymoron. Writing a good test suite is _never_ easy.
> The part that is not easy in the case of ROOT is the 'all the functionality'.
> The phase space of the cases we have need to test is quasy illimited
> since we are supporting any (almost :) ) arbitrary C++ structure.
`... supporting any (almost :) ) arbitrary C++ structure' - this is why Daniele suggests a unit testing framework. With a unit test, the idea is to break down the test to small, well understood, tests, and then combine these tests to form larger, more complex tests. In principle a good idea, but still darn hard to implement.
One of the problems with defining a unit test for ROOT, is that many of the classes/modules of ROOT are very tightly interlinked. Sure, you can define a general test case for SQL database clients, and apply that to the MySQL, PostGreSQL, MaxDB, Oracle, ... clients in ROOT, but there are other more crucial pieces where such tests would not only test one `unit' but several units - because of the relatively tight coupling.
> So we do as much as possible and welcome in additional help
> in developing further the test suite.
Have you considered using some of the testing frameworks out there? Say DejaGNU, cunit, cxxunit, and what not? For a big project like ROOT, it's probably not a bad idea to use some sort of time-tested system.
> Thanks for your input.
> Philippe.
>
> On Wed, 2006-03-01 at 21:57 +0100, Rene Brun wrote:
> We did not wait your suggestion to implement a test suite.
> We are running this test suite every day.
Is the results posted somewhere? That could be a really good source of information for people trying to hunt down bugs.
> This test suite
> is regurlarly incremented when bug reports are submitted.
Cool. But, it'd be even better if this was publicly visible.
> Could you clarify what you mean by "regressions on precedent
> behaviour of the framework"?
The `regression' is making previously good things worse. Like, `before you could read in a data class in a TClonesArray of a TTree without the dictionary, and you'd get the right numbers. Now you get zero's instead' (semi-hypothetical example).
> and give examples illustrating
> your affirmation?
> You can check-out the roottest suite from root.cern.ch
> if you want to run the test suite, or run simpler tests
> from $ROOTSYS/test.
But, the tests in the source tree are exactly the kind of tests that Daniele suggests you avoid - that is, tests of the `whole framework', not a particular `unit' (in a very strict sense). The `tests' in the `test' directory are nice, don't get me wrong, but they are more valuable as examples and inspiration than proper tests.
> We are welcoming your contribution to this test suite.
Maybe you want to formalise the testsuite a bit, if you're going to accept 3rd party contributions. For example, one can insist that tests are evaluated using a particular class, like say
class test_suite
{
...
public:
/** Constructor. */
test_suite(const std::string& n, int& argc, char** argv);
/** Destructor. */
virtual ~test_suite() {}
static test_suite& instance() { return *_instance; }
/** Set the verbosity level.
@param v The verbosity level to use. */
/** Show a banner */
void banner() const;
//@{
/// @name Tests
/** Test a truth value.
The test is passed if @a status evaluates to true.
@param status The truth value.
@param description What you expect.
@return true if test succeded, false otherwise. */
bool check(bool status, const char* description, ...);
/** Test a return status.
The test is passed if @a status is 0.
@param status The result of the computations.
@param description What you expect.
@return true if test succeded, false otherwise. */
bool status(int status, const char* description, ...);
/** Relative error test.
The test is passed if
@f$ |result - expected|/|expected| \leq error @f$
@param result The result of the computations.
@param expected The expected result.
@param error The relatvie tolerance.
@param description What you expect.
@return true if test succeded, false otherwise. */
bool relative(double result, double expected, double error,
const char* description, ...);
/** Absolute error test.
The test is passed if
@f$ |result - expected| \leq error @f$
@param result The result of the computations.
@param expected The expected result.
@param error The absolute tolerance.
@param description What you expect.
@return true if test succeded, false otherwise. */
bool absolute(double result, double expected, double error,
const char* description, ...);
/** Factor error test.
The test is passed if
@f$ f \leq error \wedge f \leq 1 / error_at_f$, where
@f$ f = result / expected @f$
@param result The result of the computations.
@param expected The expected result.
@param error The factor tolerance.
@param description What you expect.
@return true if test succeded, false otherwise. */
bool factor(double result, double expected, double error,
const char* description, ...);
/** Range error test.
Test is passed if @f$ min < result < max_at_f$
@param result The result of the computations
@param min The lower limit
@param max The upper limit.
@param description What you expect.
@return true if test succeded, false otherwise. */
template <typename Type>
const char* description, ...);
/** Compare arbitriary types.
Type must define the @c == operator, and the put-to operator
for streams (@c operator<<(ostream&,const @c Type&)).
@param result The result of the computations.
@param expected The expected result.
@param description What you expect.
@return true if test succeded, false otherwise. */
template <typename Type, typename Type1>
bool test(const Type& result, const Type1& expected,
const char* description, ...);
//@}
//@{
/// @name Status of test suite
/** Write a summary of the tests.
@return true if all tests succeded, false otherwise. */
bool summary();
@return the number of tests done so far. */ int tests() const { return _tests; } /** Get the number of passed tests.
@return the number of passed tests done so far. */ int passed() const { return _passed; } /** Get the number of failed tests.
@return the number of failed tests done so far. */
int failed() const { return _failed; }
/** Print a message */
void message(const char* format, ...);
//@}
};
[If you like, I can send you the full class].
This would give you a uniform way of reporting results from the test-suite. Other test suite frameworks exists and are probably worth looking into, f. ex., DejaGnu [1], TuT [2], CppUnit [3], Unit++ [4], and comparisons [5]
[1] http://www.gnu.org/software/dejagnu/ [2] http://tut-framework.sourceforge.net/ [3] http://cppunit.sourceforge.net/ [4] http://unitpp.sourceforge.net/ [5] http://weblogs.asp.net/rosherove/archive/2004/12/30/343938.aspx
> Rene Brun
>
> On Wed, 1 Mar
> 2006, Daniele Nicolodi wrote:
>
> > Hello. I'm following the mailing list since one year or more and i'm
> > reading a lot of bug reports that can be described as regressions on
> > precedent behaviour of the framework. Since root is supposed to be a
> > reliable data analysis system i think is very bad to have such bugs
> > pop out now and then.
Kudos to Daniele for reading the mailing list in such detail.
> > The easy solutin to the situation is to have unit tests that cover all
> > the functionalities of root and run them often on the development trunk.
> > The best would be to run them after each commit.
I guess you mean you want to run a test on the commited stuff only, right?
> Looking quickly through
> > the code i was not able to find any test of the root functionality but
> > the examples that imho does not represent a good way of testing.
> >
> > I'm a python programmer and in python therea re plenty of test
> > frameworks and i love test driven programming.
eXtreem Programming - something that I don't think many physicist do at all well, even if it looks like it's what they are doing :-)
On the other hand, some people see XP and other things like `Design Patterns', as Buzz-words - fancy, but without content. Me, I'm somewhere in the middle.
If there was a good mathematically proved methodology of writing code, I'd subscribe to that - but then again, we wouldn't need programmers, and what should we do then with all the graduate students? ;-)
> what the other think
> > about that?
I was pleased to hear that there _was_ a test suite, but I'd have loved to know about it sooner. Ah well.
Yours,
--
___ | Christian Holm Christensen
|_| | -------------------------------------------------------------
| | Address: Sankt Hansgade 23, 1. th. Phone: (+45) 35 35 96 91
_| DK-2200 Copenhagen N Cell: (+45) 24 61 85 91
_| Denmark Office: (+45) 353 25 404
____| Email: cholm_at_nbi.dk Web: www.nbi.dk/~cholm
| |
Received on Thu Mar 02 2006 - 01:28:44 MET
This archive was generated by hypermail 2.2.0 : Mon Jan 01 2007 - 16:31:57 MET