ROOT   Reference Guide
Searching...
No Matches
HypoTestResult.cxx
Go to the documentation of this file.
1// @(#)root/roostats:$Id$
2// Author: Kyle Cranmer, Lorenzo Moneta, Gregory Schott, Wouter Verkerke, Sven Kreiss
3/*************************************************************************
6 * *
7 * For the licensing terms see $ROOTSYS/LICENSE. * 8 * For the list of contributors see$ROOTSYS/README/CREDITS. *
9 *************************************************************************/
10
11/*****************************************************************************
12 * Project: RooStats
13 * Package: RooFit/RooStats
14 * @(#)root/roofit/roostats:$Id$
15 * Authors:
16 * Kyle Cranmer, Lorenzo Moneta, Gregory Schott, Wouter Verkerke, Sven Kreiss
17 *
18 *****************************************************************************/
19
20
21/** \class RooStats::HypoTestResult
22 \ingroup Roostats
23
24HypoTestResult is a base class for results from hypothesis tests.
25Any tool inheriting from HypoTestCalculator can return a HypoTestResult.
26As such, it stores a p-value for the null-hypothesis (eg. background-only)
27and an alternate hypothesis (eg. signal+background).
28The p-values can also be transformed into confidence levels
29(\f$CL_{b}\f$, \f$CL_{s+b}\f$) in a trivial way.
30The ratio of the \f$CL_{s+b}\f$ to \f$CL_{b}\f$ is often called
31\f$CL_{s}\f$, and is considered useful, though it is not a probability.
32Finally, the p-value of the null can be transformed into a number of
33equivalent Gaussian sigma using the Significance method.
34
35The p-value of the null for a given test statistic is rigorously defined and
36this is the starting point for the following conventions.
37
38### Conventions used in this class
39
40The p-value for the null and alternate are on the **same side** of the
41observed value of the test statistic. This is the more standard
42convention and avoids confusion when doing inverted tests.
43
44For exclusion, we also want the formula \f$CL_{s} = CL_{s+b} / CL_{b}\f$
45to hold which therefore defines our conventions for \f$CL_{s+b}\f$ and
46\f$CL_{b}\f$. \f$CL_{s}\f$ was specifically invented for exclusion
47and therefore all quantities need be related through the assignments
48as they are for exclusion: \f$CL_{s+b} = p_{s+b}\f$; \f$CL_{b} = p_{b}\f$. This
49is derived by considering the scenarios of a powerful and not powerful
50inverted test, where for the not so powerful test, \f$CL_{s}\f$ must be
51close to one.
52
53For results of Hypothesis tests,
54\f$CL_{s}\f$ has no similar direct interpretation as for exclusion and can
55be larger than one.
56
57*/
58
61#include "RooAbsReal.h"
62
64
65#include <TMath.h>
66
67#include <limits>
68#define NaN numeric_limits<float>::quiet_NaN()
69#define IsNaN(a) TMath::IsNaN(a)
70
72
73using namespace RooStats;
74using namespace std;
75
76////////////////////////////////////////////////////////////////////////////////
77/// Default constructor
78
81 fNullPValue(NaN), fAlternatePValue(NaN),
82 fNullPValueError(0), fAlternatePValueError(0),
83 fTestStatisticData(NaN),
84 fAllTestStatisticsData(nullptr),
85 fNullDistr(nullptr), fAltDistr(nullptr),
86 fNullDetailedOutput(nullptr), fAltDetailedOutput(nullptr),
87 fPValueIsRightTail(true),
88 fBackgroundIsAlt(false)
89{
90}
91
92////////////////////////////////////////////////////////////////////////////////
93/// Alternate constructor
94
95HypoTestResult::HypoTestResult(const char* name, double nullp, double altp) :
97 fNullPValue(nullp), fAlternatePValue(altp),
98 fNullPValueError(0), fAlternatePValueError(0),
99 fTestStatisticData(NaN),
100 fAllTestStatisticsData(nullptr),
101 fNullDistr(nullptr), fAltDistr(nullptr),
102 fNullDetailedOutput(nullptr), fAltDetailedOutput(nullptr),
103 fPValueIsRightTail(true),
104 fBackgroundIsAlt(false)
105{
106}
107
108////////////////////////////////////////////////////////////////////////////////
109/// copy constructor
110
112 TNamed(other),
113 fNullPValue(other.fNullPValue), fAlternatePValue(other.fAlternatePValue),
114 fNullPValueError(other.fNullPValueError), fAlternatePValueError(other.fAlternatePValueError),
115 fTestStatisticData(other.fTestStatisticData),
116 fAllTestStatisticsData(nullptr),
117 fNullDistr(nullptr), fAltDistr(nullptr),
118 fNullDetailedOutput(nullptr), fAltDetailedOutput(nullptr),
119 fPValueIsRightTail( other.GetPValueIsRightTail() ),
120 fBackgroundIsAlt( other.GetBackGroundIsAlt() )
121{
122 this->Append( &other );
123}
124
125////////////////////////////////////////////////////////////////////////////////
126/// Destructor
127
129{
130 if( fNullDistr ) delete fNullDistr;
131 if( fAltDistr ) delete fAltDistr;
132
135
137}
138
139////////////////////////////////////////////////////////////////////////////////
140/// assignment operator
141
143 if (this == &other) return *this;
144 SetName(other.GetName());
145 SetTitle(other.GetTitle());
146 fNullPValue = other.fNullPValue;
151
153 fAllTestStatisticsData = nullptr;
154 if( fNullDistr ) { delete fNullDistr; fNullDistr = nullptr; }
155 if( fAltDistr ) { delete fAltDistr; fAltDistr = nullptr; }
157 if( fAltDetailedOutput ) { delete fAltDetailedOutput; fAltDetailedOutput = nullptr; }
158 fFitInfo = nullptr;
159
162
163 this->Append( &other );
164
165 return *this;
166}
167
168////////////////////////////////////////////////////////////////////////////////
170/// Use the data test statistics of the added object if it is not already
171/// set (otherwise, ignore the new one).
172
174 if (fNullDistr) {
176 } else if (other->GetNullDistribution()) {
178 }
179
180 if (fAltDistr) {
182 } else if (other->GetAltDistribution()) {
184 }
185
186 if( fNullDetailedOutput ) {
188 }else{
190 }
191
192 if( fAltDetailedOutput ) {
194 }else{
196 }
197
198 if( fFitInfo ) {
199 if( other->GetFitInfo() ) fFitInfo->append( *other->GetFitInfo() );
200 }else{
201 if( other->GetFitInfo() ) fFitInfo = std::make_unique<RooDataSet>( *other->GetFitInfo());
202 }
203
204 // if no data is present use the other HypoTestResult's data
206
209}
210
211////////////////////////////////////////////////////////////////////////////////
212
214 fAltDistr = alt;
216}
217
218////////////////////////////////////////////////////////////////////////////////
219
221 fNullDistr = null;
223}
224
225////////////////////////////////////////////////////////////////////////////////
226
228 fTestStatisticData = tsd;
229
232}
233
234////////////////////////////////////////////////////////////////////////////////
235
239 fAllTestStatisticsData = nullptr;
240 }
241 if (tsd) fAllTestStatisticsData = static_cast<const RooArgList*>(tsd->snapshot());
242
244 RooRealVar* firstTS = static_cast<RooRealVar*>(fAllTestStatisticsData->at(0));
245 if( firstTS ) SetTestStatisticData( firstTS->getVal() );
246 }
247}
248
249////////////////////////////////////////////////////////////////////////////////
250
253
256}
257
258////////////////////////////////////////////////////////////////////////////////
259
261 return !IsNaN(fTestStatisticData);
262}
263
264////////////////////////////////////////////////////////////////////////////////
265
267 // compute error on Null pvalue
268 return fNullPValueError;
269}
270
271////////////////////////////////////////////////////////////////////////////////
272/// compute \f$CL_{b}\f$ error
273/// \f$CL_{b}\f$ = 1 - NullPValue()
274/// must use opposite condition that routine above
275
278}
279
280////////////////////////////////////////////////////////////////////////////////
281
284}
285
286////////////////////////////////////////////////////////////////////////////////
287/// Taylor expansion series approximation for standard deviation (error propagation)
288
291}
292
293////////////////////////////////////////////////////////////////////////////////
294/// Returns an estimate of the error on \f$CL_{s}\f$ through combination of the
295/// errors on \f$CL_{b}\f$ and \f$CL_{s+b}\f$:
296/// \f[
297/// \sigma_{CL_s} = CL_s
298/// \sqrt{\left( \frac{\sigma_{CL_{s+b}}}{CL_{s+b}} \right)^2 + \left( \frac{\sigma_{CL_{b}}}{CL_{b}} \right)^2}
299/// \f]
300
302 if(!fAltDistr || !fNullDistr) return 0.0;
303
304 // unsigned const int n_b = fNullDistr->GetSamplingDistribution().size();
305 // unsigned const int n_sb = fAltDistr->GetSamplingDistribution().size();
306
307 // if CLb() == 0 CLs = -1 so return a -1 error
308 if (CLb() == 0 ) return -1;
309
310 double cl_b_err2 = pow(CLbError(),2);
311 double cl_sb_err2 = pow(CLsplusbError(),2);
312
313 return TMath::Sqrt(cl_sb_err2 + cl_b_err2 * pow(CLs(),2))/CLb();
314}
315
316////////////////////////////////////////////////////////////////////////////////
317/// updates the pvalue if sufficient data is available
318
319void HypoTestResult::UpdatePValue(const SamplingDistribution* distr, double &pvalue, double &perror, bool /*isNull*/) {
320 if(IsNaN(fTestStatisticData)) return;
321 if(!distr) return;
322
323 /* Got to be careful for discrete distributions:
324 * To get the right behaviour for limits, the p-value must
325 * include the value of fTestStatistic both for Alt and Null cases
326 */
328 pvalue = distr->IntegralAndError(perror, fTestStatisticData, RooNumber::infinity(), true,
329 true , true ); // always closed interval [ fTestStatistic, inf ]
330
331 }else{
332 pvalue = distr->IntegralAndError(perror, -RooNumber::infinity(), fTestStatisticData, true,
333 true, true ); // // always closed [ -inf, fTestStatistic ]
334 }
335}
336
337////////////////////////////////////////////////////////////////////////////////
338/// Print out some information about the results
339/// Note: use Alt/Null labels for the hypotheses here as the Null
340/// might be the s+b hypothesis.
341
343{
344 bool fromToys = (fAltDistr || fNullDistr);
345
346 std::cout << std::endl << "Results " << GetName() << ": " << endl;
347 std::cout << " - Null p-value = " << NullPValue();
348 if (fromToys) std::cout << " +/- " << NullPValueError();
349 std::cout << std::endl;
350 std::cout << " - Significance = " << Significance();
351 if (fromToys) std::cout << " +/- " << SignificanceError() << " sigma";
352 std::cout << std::endl;
353 if(fAltDistr)
354 std::cout << " - Number of Alt toys: " << fAltDistr->GetSize() << std::endl;
355 if(fNullDistr)
356 std::cout << " - Number of Null toys: " << fNullDistr->GetSize() << std::endl;
357
358 if (HasTestStatisticData() ) std::cout << " - Test statistic evaluated on data: " << fTestStatisticData << std::endl;
359 std::cout << " - CL_b: " << CLb();
360 if (fromToys) std::cout << " +/- " << CLbError();
361 std::cout << std::endl;
362 std::cout << " - CL_s+b: " << CLsplusb();
363 if (fromToys) std::cout << " +/- " << CLsplusbError();
364 std::cout << std::endl;
365 std::cout << " - CL_s: " << CLs();
366 if (fromToys) std::cout << " +/- " << CLsError();
367 std::cout << std::endl;
368
369 return;
370}
#define NaN
#define IsNaN(a)
#define NaN
const char Option_t
Definition RtypesCore.h:66
#define ClassImp(name)
Definition Rtypes.h:377
char name[80]
Definition TGX11.cxx:110
RooAbsCollection * snapshot(bool deepCopy=true) const
Take a snap shot of current collection contents.
double getVal(const RooArgSet *normalisationSet=nullptr) const
Evaluate object.
Definition RooAbsReal.h:103
RooArgList is a container object that can hold multiple RooAbsArg objects.
Definition RooArgList.h:22
RooAbsArg * at(Int_t idx) const
Return object at given index, or nullptr if index is out of range.
Definition RooArgList.h:110
Container class to hold unbinned data.
Definition RooDataSet.h:57
void append(RooDataSet &data)
Add all data points of given data set to this data set.
static constexpr double infinity()
Return internal infinity representation.
Definition RooNumber.h:25
Variable that can be changed from the outside.
Definition RooRealVar.h:37
HypoTestResult is a base class for results from hypothesis tests.
RooDataSet * GetFitInfo() const
void UpdatePValue(const SamplingDistribution *distr, double &pvalue, double &perror, bool pIsRightTail)
updates the pvalue if sufficient data is available
void Print(const Option_t *="") const override
Print out some information about the results Note: use Alt/Null labels for the hypotheses here as the...
double fNullPValue
p-value for the null hypothesis (small number means disfavoured)
bool HasTestStatisticData(void) const
double fAlternatePValueError
error of p-value for the alternate hypothesis (small number means disfavoured)
HypoTestResult & operator=(const HypoTestResult &other)
assignment operator
virtual double CLsplusb() const
Convert AlternatePValue into a "confidence level".
void SetAllTestStatisticsData(const RooArgList *tsd)
double GetTestStatisticData(void) const
virtual void Append(const HypoTestResult *other)
double NullPValueError() const
The error on the Null p-value.
double CLsError() const
The error on the ratio .
RooDataSet * GetNullDetailedOutput(void) const
virtual double Significance() const
familiar name for the Null p-value in terms of 1-sided Gaussian significance
SamplingDistribution * fAltDistr
bool GetPValueIsRightTail(void) const
void SetNullDistribution(SamplingDistribution *null)
~HypoTestResult() override
destructor
bool GetBackGroundIsAlt(void) const
HypoTestResult(const char *name=nullptr)
default constructor
double SignificanceError() const
The error on the significance, computed from NullPValueError via error propagation.
virtual double NullPValue() const
Return p-value for null hypothesis.
double CLbError() const
The error on the "confidence level" of the null hypothesis.
void SetTestStatisticData(const double tsd)
double CLsplusbError() const
The error on the "confidence level" of the alternative hypothesis.
double fNullPValueError
error of p-value for the null hypothesis (small number means disfavoured)
double fTestStatisticData
result of the test statistic evaluated on data
void SetAltDistribution(SamplingDistribution *alt)
RooDataSet * GetAltDetailedOutput(void) const
const RooArgList * fAllTestStatisticsData
for the case of multiple test statistics, holds all the results
SamplingDistribution * GetNullDistribution(void) const
std::unique_ptr< RooDataSet > fFitInfo
virtual double CLs() const
is simply (not a method, but a quantity)
double fAlternatePValue
p-value for the alternate hypothesis (small number means disfavoured)
virtual double CLb() const
Convert NullPValue into a "confidence level".
SamplingDistribution * fNullDistr
SamplingDistribution * GetAltDistribution(void) const
This class simply holds a sampling distribution of some test statistic.
Int_t GetSize() const
size of samples
double IntegralAndError(double &error, double low, double high, bool normalize=true, bool lowClosed=true, bool highClosed=false) const
numerical integral in these limits including error estimation
merge two sampling distributions
The TNamed class is the base class for all named ROOT classes.
Definition TNamed.h:29
virtual void SetTitle(const char *title="")
Set the title of the TNamed.
Definition TNamed.cxx:164
const char * GetName() const override
Returns name of object.
Definition TNamed.h:47
const char * GetTitle() const override
Returns title of object.
Definition TNamed.h:48
virtual void SetName(const char *name)
Set the name of the TNamed.
Definition TNamed.cxx:140
double normal_pdf(double x, double sigma=1, double x0=0)
Probability density function of the normal (Gaussian) distribution.
Namespace for the RooStats classes.
Definition Asimov.h:19
Double_t Sqrt(Double_t x)
Returns the square root of x.
Definition TMath.h:662