Logo ROOT  
Reference Guide
HypoTestResult.cxx
Go to the documentation of this file.
1// @(#)root/roostats:$Id$
2// Author: Kyle Cranmer, Lorenzo Moneta, Gregory Schott, Wouter Verkerke, Sven Kreiss
3/*************************************************************************
4 * Copyright (C) 1995-2008, Rene Brun and Fons Rademakers. *
5 * All rights reserved. *
6 * *
7 * For the licensing terms see $ROOTSYS/LICENSE. *
8 * For the list of contributors see $ROOTSYS/README/CREDITS. *
9 *************************************************************************/
10
11/*****************************************************************************
12 * Project: RooStats
13 * Package: RooFit/RooStats
14 * @(#)root/roofit/roostats:$Id$
15 * Authors:
16 * Kyle Cranmer, Lorenzo Moneta, Gregory Schott, Wouter Verkerke, Sven Kreiss
17 *
18 *****************************************************************************/
19
20
21/** \class RooStats::HypoTestResult
22 \ingroup Roostats
23
24HypoTestResult is a base class for results from hypothesis tests.
25Any tool inheriting from HypoTestCalculator can return a HypoTestResult.
26As such, it stores a p-value for the null-hypothesis (eg. background-only)
27and an alternate hypothesis (eg. signal+background).
28The p-values can also be transformed into confidence levels
29(\f$CL_{b}\f$, \f$CL_{s+b}\f$) in a trivial way.
30The ratio of the \f$CL_{s+b}\f$ to \f$CL_{b}\f$ is often called
31\f$CL_{s}\f$, and is considered useful, though it is not a probability.
32Finally, the p-value of the null can be transformed into a number of
33equivalent Gaussian sigma using the Significance method.
34
35The p-value of the null for a given test statistic is rigorously defined and
36this is the starting point for the following conventions.
37
38### Conventions used in this class
39
40The p-value for the null and alternate are on the **same side** of the
41observed value of the test statistic. This is the more standard
42convention and avoids confusion when doing inverted tests.
43
44For exclusion, we also want the formula \f$CL_{s} = CL_{s+b} / CL_{b}\f$
45to hold which therefore defines our conventions for \f$CL_{s+b}\f$ and
46\f$CL_{b}\f$. \f$CL_{s}\f$ was specifically invented for exclusion
47and therefore all quantities need be related through the assignments
48as they are for exclusion: \f$CL_{s+b} = p_{s+b}\f$; \f$CL_{b} = p_{b}\f$. This
49is derived by considering the scenarios of a powerful and not powerful
50inverted test, where for the not so powerful test, \f$CL_{s}\f$ must be
51close to one.
52
53For results of Hypothesis tests,
54\f$CL_{s}\f$ has no similar direct interpretation as for exclusion and can
55be larger than one.
56
57*/
58
61#include "RooAbsReal.h"
62
64
65#include <TMath.h>
66
67#include <limits>
68#define NaN numeric_limits<float>::quiet_NaN()
69#define IsNaN(a) TMath::IsNaN(a)
70
72
73using namespace RooStats;
74using namespace std;
75
76////////////////////////////////////////////////////////////////////////////////
77/// Default constructor
78
79HypoTestResult::HypoTestResult(const char* name) :
81 fNullPValue(NaN), fAlternatePValue(NaN),
82 fNullPValueError(0), fAlternatePValueError(0),
83 fTestStatisticData(NaN),
84 fAllTestStatisticsData(nullptr),
85 fNullDistr(nullptr), fAltDistr(nullptr),
86 fNullDetailedOutput(nullptr), fAltDetailedOutput(nullptr),
87 fPValueIsRightTail(true),
88 fBackgroundIsAlt(false)
89{
90}
91
92////////////////////////////////////////////////////////////////////////////////
93/// Alternate constructor
94
95HypoTestResult::HypoTestResult(const char* name, double nullp, double altp) :
97 fNullPValue(nullp), fAlternatePValue(altp),
98 fNullPValueError(0), fAlternatePValueError(0),
99 fTestStatisticData(NaN),
100 fAllTestStatisticsData(nullptr),
101 fNullDistr(nullptr), fAltDistr(nullptr),
102 fNullDetailedOutput(nullptr), fAltDetailedOutput(nullptr),
103 fPValueIsRightTail(true),
104 fBackgroundIsAlt(false)
105{
106}
107
108////////////////////////////////////////////////////////////////////////////////
109/// copy constructor
110
112 TNamed(other),
113 fNullPValue(NaN), fAlternatePValue(NaN),
114 fNullPValueError(0), fAlternatePValueError(0),
115 fTestStatisticData(NaN),
116 fAllTestStatisticsData(nullptr),
117 fNullDistr(nullptr), fAltDistr(nullptr),
118 fNullDetailedOutput(nullptr), fAltDetailedOutput(nullptr),
119 fPValueIsRightTail( other.GetPValueIsRightTail() ),
120 fBackgroundIsAlt( other.GetBackGroundIsAlt() )
121{
122 this->Append( &other );
123}
124
125////////////////////////////////////////////////////////////////////////////////
126/// Destructor
127
129{
130 if( fNullDistr ) delete fNullDistr;
131 if( fAltDistr ) delete fAltDistr;
132
135
137}
138
139////////////////////////////////////////////////////////////////////////////////
140/// assignment operator
141
143 if (this == &other) return *this;
144 SetName(other.GetName());
145 SetTitle(other.GetTitle());
146 fNullPValue = other.fNullPValue;
151
153 fAllTestStatisticsData = nullptr;
154 if( fNullDistr ) { delete fNullDistr; fNullDistr = nullptr; }
155 if( fAltDistr ) { delete fAltDistr; fAltDistr = nullptr; }
157 if( fAltDetailedOutput ) { delete fAltDetailedOutput; fAltDetailedOutput = nullptr; }
158 fFitInfo = nullptr;
159
162
163 this->Append( &other );
164
165 return *this;
166}
167
168////////////////////////////////////////////////////////////////////////////////
169/// Add additional toy-MC experiments to the current results.
170/// Use the data test statistics of the added object if it is not already
171/// set (otherwise, ignore the new one).
172
174 if(fNullDistr)
176 else
178
179 if(fAltDistr)
181 else
183
184
185 if( fNullDetailedOutput ) {
187 }else{
189 }
190
191 if( fAltDetailedOutput ) {
193 }else{
195 }
196
197 if( fFitInfo ) {
198 if( other->GetFitInfo() ) fFitInfo->append( *other->GetFitInfo() );
199 }else{
200 if( other->GetFitInfo() ) fFitInfo.reset(new RooDataSet( *other->GetFitInfo() ));
201 }
202
203 // if no data is present use the other HypoTestResult's data
205
208}
209
210////////////////////////////////////////////////////////////////////////////////
211
213 fAltDistr = alt;
215}
216
217////////////////////////////////////////////////////////////////////////////////
218
222}
223
224////////////////////////////////////////////////////////////////////////////////
225
227 fTestStatisticData = tsd;
228
231}
232
233////////////////////////////////////////////////////////////////////////////////
234
239 }
240 if (tsd) fAllTestStatisticsData = (const RooArgList*)tsd->snapshot();
241
244 if( firstTS ) SetTestStatisticData( firstTS->getVal() );
245 }
246}
247
248////////////////////////////////////////////////////////////////////////////////
249
252
255}
256
257////////////////////////////////////////////////////////////////////////////////
258
260 return !IsNaN(fTestStatisticData);
261}
262
263////////////////////////////////////////////////////////////////////////////////
264
266 // compute error on Null pvalue
267 return fNullPValueError;
268}
269
270////////////////////////////////////////////////////////////////////////////////
271/// compute \f$CL_{b}\f$ error
272/// \f$CL_{b}\f$ = 1 - NullPValue()
273/// must use opposite condition that routine above
274
277}
278
279////////////////////////////////////////////////////////////////////////////////
280
283}
284
285////////////////////////////////////////////////////////////////////////////////
286/// Taylor expansion series approximation for standard deviation (error propagation)
287
290}
291
292////////////////////////////////////////////////////////////////////////////////
293/// Returns an estimate of the error on \f$CL_{s}\f$ through combination of the
294/// errors on \f$CL_{b}\f$ and \f$CL_{s+b}\f$:
295/// \f[
296/// \sigma_{CL_s} = CL_s
297/// \sqrt{\left( \frac{\sigma_{CL_{s+b}}}{CL_{s+b}} \right)^2 + \left( \frac{\sigma_{CL_{b}}}{CL_{b}} \right)^2}
298/// \f]
299
301 if(!fAltDistr || !fNullDistr) return 0.0;
302
303 // unsigned const int n_b = fNullDistr->GetSamplingDistribution().size();
304 // unsigned const int n_sb = fAltDistr->GetSamplingDistribution().size();
305
306 // if CLb() == 0 CLs = -1 so return a -1 error
307 if (CLb() == 0 ) return -1;
308
309 double cl_b_err2 = pow(CLbError(),2);
310 double cl_sb_err2 = pow(CLsplusbError(),2);
311
312 return TMath::Sqrt(cl_sb_err2 + cl_b_err2 * pow(CLs(),2))/CLb();
313}
314
315////////////////////////////////////////////////////////////////////////////////
316/// updates the pvalue if sufficient data is available
317
318void HypoTestResult::UpdatePValue(const SamplingDistribution* distr, double &pvalue, double &perror, bool /*isNull*/) {
319 if(IsNaN(fTestStatisticData)) return;
320 if(!distr) return;
321
322 /* Got to be careful for discrete distributions:
323 * To get the right behaviour for limits, the p-value must
324 * include the value of fTestStatistic both for Alt and Null cases
325 */
327 pvalue = distr->IntegralAndError(perror, fTestStatisticData, RooNumber::infinity(), true,
328 true , true ); // always closed interval [ fTestStatistic, inf ]
329
330 }else{
331 pvalue = distr->IntegralAndError(perror, -RooNumber::infinity(), fTestStatisticData, true,
332 true, true ); // // always closed [ -inf, fTestStatistic ]
333 }
334}
335
336////////////////////////////////////////////////////////////////////////////////
337/// Print out some information about the results
338/// Note: use Alt/Null labels for the hypotheses here as the Null
339/// might be the s+b hypothesis.
340
342{
343 bool fromToys = (fAltDistr || fNullDistr);
344
345 std::cout << std::endl << "Results " << GetName() << ": " << endl;
346 std::cout << " - Null p-value = " << NullPValue();
347 if (fromToys) std::cout << " +/- " << NullPValueError();
348 std::cout << std::endl;
349 std::cout << " - Significance = " << Significance();
350 if (fromToys) std::cout << " +/- " << SignificanceError() << " sigma";
351 std::cout << std::endl;
352 if(fAltDistr)
353 std::cout << " - Number of Alt toys: " << fAltDistr->GetSize() << std::endl;
354 if(fNullDistr)
355 std::cout << " - Number of Null toys: " << fNullDistr->GetSize() << std::endl;
356
357 if (HasTestStatisticData() ) std::cout << " - Test statistic evaluated on data: " << fTestStatisticData << std::endl;
358 std::cout << " - CL_b: " << CLb();
359 if (fromToys) std::cout << " +/- " << CLbError();
360 std::cout << std::endl;
361 std::cout << " - CL_s+b: " << CLsplusb();
362 if (fromToys) std::cout << " +/- " << CLsplusbError();
363 std::cout << std::endl;
364 std::cout << " - CL_s: " << CLs();
365 if (fromToys) std::cout << " +/- " << CLsError();
366 std::cout << std::endl;
367
368 return;
369}
#define NaN
#define IsNaN(a)
const char Option_t
Definition: RtypesCore.h:66
#define ClassImp(name)
Definition: Rtypes.h:375
char name[80]
Definition: TGX11.cxx:110
RooAbsCollection * snapshot(bool deepCopy=true) const
Take a snap shot of current collection contents.
Int_t getSize() const
Return the number of elements in the collection.
double getVal(const RooArgSet *normalisationSet=nullptr) const
Evaluate object.
Definition: RooAbsReal.h:91
RooArgList is a container object that can hold multiple RooAbsArg objects.
Definition: RooArgList.h:22
RooAbsArg * at(Int_t idx) const
Return object at given index, or nullptr if index is out of range.
Definition: RooArgList.h:110
RooDataSet is a container class to hold unbinned data.
Definition: RooDataSet.h:55
void append(RooDataSet &data)
Add all data points of given data set to this data set.
static double infinity()
Return internal infinity representation.
Definition: RooNumber.cxx:48
RooRealVar represents a variable that can be changed from the outside.
Definition: RooRealVar.h:40
HypoTestResult is a base class for results from hypothesis tests.
RooDataSet * GetFitInfo() const
RooDataSet * fAltDetailedOutput
void UpdatePValue(const SamplingDistribution *distr, double &pvalue, double &perror, bool pIsRightTail)
updates the pvalue if sufficient data is available
void Print(const Option_t *="") const override
Print out some information about the results Note: use Alt/Null labels for the hypotheses here as the...
double fNullPValue
p-value for the null hypothesis (small number means disfavoured)
bool HasTestStatisticData(void) const
double fAlternatePValueError
error of p-value for the alternate hypothesis (small number means disfavoured)
HypoTestResult & operator=(const HypoTestResult &other)
assignment operator
virtual double CLsplusb() const
Convert AlternatePValue into a "confidence level".
void SetAllTestStatisticsData(const RooArgList *tsd)
double GetTestStatisticData(void) const
virtual void Append(const HypoTestResult *other)
add values from another HypoTestResult
double NullPValueError() const
The error on the Null p-value.
double CLsError() const
The error on the ratio .
RooDataSet * GetNullDetailedOutput(void) const
virtual double Significance() const
familiar name for the Null p-value in terms of 1-sided Gaussian significance
RooDataSet * fNullDetailedOutput
SamplingDistribution * fAltDistr
bool GetPValueIsRightTail(void) const
void SetNullDistribution(SamplingDistribution *null)
~HypoTestResult() override
destructor
bool GetBackGroundIsAlt(void) const
HypoTestResult(const char *name=nullptr)
default constructor
double SignificanceError() const
The error on the significance, computed from NullPValueError via error propagation.
virtual double NullPValue() const
Return p-value for null hypothesis.
double CLbError() const
The error on the "confidence level" of the null hypothesis.
void SetTestStatisticData(const double tsd)
double CLsplusbError() const
The error on the "confidence level" of the alternative hypothesis.
double fNullPValueError
error of p-value for the null hypothesis (small number means disfavoured)
double fTestStatisticData
result of the test statistic evaluated on data
void SetAltDistribution(SamplingDistribution *alt)
void SetPValueIsRightTail(bool pr)
RooDataSet * GetAltDetailedOutput(void) const
const RooArgList * fAllTestStatisticsData
for the case of multiple test statistics, holds all the results
SamplingDistribution * GetNullDistribution(void) const
std::unique_ptr< RooDataSet > fFitInfo
virtual double CLs() const
is simply (not a method, but a quantity)
double fAlternatePValue
p-value for the alternate hypothesis (small number means disfavoured)
virtual double CLb() const
Convert NullPValue into a "confidence level".
SamplingDistribution * fNullDistr
SamplingDistribution * GetAltDistribution(void) const
This class simply holds a sampling distribution of some test statistic.
Int_t GetSize() const
size of samples
double IntegralAndError(double &error, double low, double high, bool normalize=true, bool lowClosed=true, bool highClosed=false) const
numerical integral in these limits including error estimation
void Add(const SamplingDistribution *other)
merge two sampling distributions
The TNamed class is the base class for all named ROOT classes.
Definition: TNamed.h:29
virtual void SetTitle(const char *title="")
Set the title of the TNamed.
Definition: TNamed.cxx:164
const char * GetName() const override
Returns name of object.
Definition: TNamed.h:47
const char * GetTitle() const override
Returns title of object.
Definition: TNamed.h:48
virtual void SetName(const char *name)
Set the name of the TNamed.
Definition: TNamed.cxx:140
double normal_pdf(double x, double sigma=1, double x0=0)
Probability density function of the normal (Gaussian) distribution.
RVec< PromoteTypes< T0, T1 > > pow(const T0 &x, const RVec< T1 > &v)
Definition: RVec.hxx:1792
Namespace for the RooStats classes.
Definition: Asimov.h:19
null_t< F > null()
Double_t Sqrt(Double_t x)
Returns the square root of x.
Definition: TMath.h:660