Logo ROOT   6.21/01
Reference Guide
TMVA::CrossValidation Class Reference

Class to perform cross validation, splitting the dataloader into folds.


Use html for explicit line breaking
Markdown links? class reference?

ce->BookMethod(dataloader, options);
ce->Evaluate();

Cross-evaluation will generate a new training and a test set dynamically from from K folds. These K folds are generated by splitting the input training set. The input test set is currently ignored.

This means that when you specify your DataSet you should include all events in your training set. One way of doing this would be the following:

dataloader->AddTree( signalTree, "cls1" );
dataloader->AddTree( background, "cls2" );
dataloader->PrepareTrainingAndTestTree( "", "", "nTest_cls1=1:nTest_cls2=1" );

Split Expression

See CVSplit documentation?

Definition at line 122 of file CrossValidation.h.

Public Member Functions

 CrossValidation (TString jobName, TMVA::DataLoader *dataloader, TString options)
 
 CrossValidation (TString jobName, TMVA::DataLoader *dataloader, TFile *outputFile, TString options)
 
 ~CrossValidation ()
 
void Evaluate ()
 Does training, test set evaluation and performance evaluation of using cross-evalution. More...
 
FactoryGetFactory ()
 
UInt_t GetNumFolds ()
 
const std::vector< CrossValidationResult > & GetResults () const
 
TString GetSplitExpr ()
 
void InitOptions ()
 
void ParseOptions ()
 
Method to parse the internal option string. More...
 
void SetNumFolds (UInt_t i)
 
void SetSplitExpr (TString splitExpr)
 
- Public Member Functions inherited from TMVA::Envelope
 ~Envelope ()
 
Default destructor More...
 
virtual void BookMethod (TString methodname, TString methodtitle, TString options="")
 
Method to book the machine learning method to perform the algorithm. More...
 
virtual void BookMethod (Types::EMVA method, TString methodtitle, TString options="")
 
Method to book the machine learning method to perform the algorithm. More...
 
DataLoaderGetDataLoader ()
 
Method to get the pointer to TMVA::DataLoader object. More...
 
TFileGetFile ()
 
Method to get the pointer to TFile object. More...
 
std::vector< OptionMap > & GetMethods ()
 
Method get the Booked methods in a option map object. More...
 
Bool_t HasMethod (TString methodname, TString methodtitle)
 function to check methods booked More...
 
Bool_t IsModelPersistence ()
 
Method to see if the algorithm model is saved in xml or serialized files. More...
 
Bool_t IsSilentFile ()
 
Method to see if a file is available to save results More...
 
Bool_t IsVerbose ()
 
Method to see if the algorithm should print extra information. More...
 
void SetDataLoader (DataLoader *dalaloader)
 
Method to set the pointer to TMVA::DataLoader object. More...
 
void SetFile (TFile *file)
 
Method to set the pointer to TFile object, with a writable file. More...
 
void SetModelPersistence (Bool_t status=kTRUE)
 
Method enable model persistence, then algorithms model is saved in xml or serialized files. More...
 
void SetVerbose (Bool_t status)
 
Method enable print extra information in the algorithms. More...
 
- Public Member Functions inherited from TMVA::Configurable
 Configurable (const TString &theOption="")
 constructor More...
 
virtual ~Configurable ()
 default destructor More...
 
void AddOptionsXMLTo (void *parent) const
 write options to XML file More...
 
template<class T >
void AddPreDefVal (const T &)
 
template<class T >
void AddPreDefVal (const TString &optname, const T &)
 
void CheckForUnusedOptions () const
 checks for unused options in option string More...
 
template<class T >
OptionBaseDeclareOptionRef (T &ref, const TString &name, const TString &desc="")
 
template<class T >
OptionBaseDeclareOptionRef (T *&ref, Int_t size, const TString &name, const TString &desc="")
 
template<class T >
TMVA::OptionBaseDeclareOptionRef (T &ref, const TString &name, const TString &desc)
 
template<class T >
TMVA::OptionBaseDeclareOptionRef (T *&ref, Int_t size, const TString &name, const TString &desc)
 
const char * GetConfigDescription () const
 
const char * GetConfigName () const
 
const TStringGetOptions () const
 
MsgLoggerLog () const
 
void PrintOptions () const
 prints out the options set in the options string and the defaults More...
 
void ReadOptionsFromStream (std::istream &istr)
 read option back from the weight file More...
 
void ReadOptionsFromXML (void *node)
 
void SetConfigDescription (const char *d)
 
void SetConfigName (const char *n)
 
void SetMsgType (EMsgType t)
 
void SetOptions (const TString &s)
 
void WriteOptionsToStream (std::ostream &o, const TString &prefix) const
 write options to output stream (e.g. in writing the MVA weight files More...
 
- Public Member Functions inherited from TNamed
 TNamed ()
 
 TNamed (const char *name, const char *title)
 
 TNamed (const TString &name, const TString &title)
 
 TNamed (const TNamed &named)
 TNamed copy ctor. More...
 
virtual ~TNamed ()
 TNamed destructor. More...
 
virtual void Clear (Option_t *option="")
 Set name and title to empty strings (""). More...
 
virtual TObjectClone (const char *newname="") const
 Make a clone of an object using the Streamer facility. More...
 
virtual Int_t Compare (const TObject *obj) const
 Compare two TNamed objects. More...
 
virtual void Copy (TObject &named) const
 Copy this to obj. More...
 
virtual void FillBuffer (char *&buffer)
 Encode TNamed into output buffer. More...
 
virtual const char * GetName () const
 Returns name of object. More...
 
virtual const char * GetTitle () const
 Returns title of object. More...
 
virtual ULong_t Hash () const
 Return hash value for this object. More...
 
virtual Bool_t IsSortable () const
 
virtual void ls (Option_t *option="") const
 List TNamed name and title. More...
 
TNamedoperator= (const TNamed &rhs)
 TNamed assignment operator. More...
 
virtual void Print (Option_t *option="") const
 Print TNamed name and title. More...
 
virtual void SetName (const char *name)
 Set the name of the TNamed. More...
 
virtual void SetNameTitle (const char *name, const char *title)
 Set all the TNamed parameters (name and title). More...
 
virtual void SetTitle (const char *title="")
 Set the title of the TNamed. More...
 
virtual Int_t Sizeof () const
 Return size of the TNamed part of the TObject. More...
 
- Public Member Functions inherited from TObject
 TObject ()
 TObject constructor. More...
 
 TObject (const TObject &object)
 TObject copy ctor. More...
 
virtual ~TObject ()
 TObject destructor. More...
 
void AbstractMethod (const char *method) const
 Use this method to implement an "abstract" method that you don't want to leave purely abstract. More...
 
virtual void AppendPad (Option_t *option="")
 Append graphics object to current pad. More...
 
virtual void Browse (TBrowser *b)
 Browse object. May be overridden for another default action. More...
 
ULong_t CheckedHash ()
 Check and record whether this class has a consistent Hash/RecursiveRemove setup (*) and then return the regular Hash value for this object. More...
 
virtual const char * ClassName () const
 Returns name of class to which the object belongs. More...
 
virtual void Delete (Option_t *option="")
 Delete this object. More...
 
virtual Int_t DistancetoPrimitive (Int_t px, Int_t py)
 Computes distance from point (px,py) to the object. More...
 
virtual void Draw (Option_t *option="")
 Default Draw method for all objects. More...
 
virtual void DrawClass () const
 Draw class inheritance tree of the class to which this object belongs. More...
 
virtual TObjectDrawClone (Option_t *option="") const
 Draw a clone of this object in the current selected pad for instance with: gROOT->SetSelectedPad(gPad). More...
 
virtual void Dump () const
 Dump contents of object on stdout. More...
 
virtual void Error (const char *method, const char *msgfmt,...) const
 Issue error message. More...
 
virtual void Execute (const char *method, const char *params, Int_t *error=0)
 Execute method on this object with the given parameter string, e.g. More...
 
virtual void Execute (TMethod *method, TObjArray *params, Int_t *error=0)
 Execute method on this object with parameters stored in the TObjArray. More...
 
virtual void ExecuteEvent (Int_t event, Int_t px, Int_t py)
 Execute action corresponding to an event at (px,py). More...
 
virtual void Fatal (const char *method, const char *msgfmt,...) const
 Issue fatal error message. More...
 
virtual TObjectFindObject (const char *name) const
 Must be redefined in derived classes. More...
 
virtual TObjectFindObject (const TObject *obj) const
 Must be redefined in derived classes. More...
 
virtual Option_tGetDrawOption () const
 Get option used by the graphics system to draw this object. More...
 
virtual const char * GetIconName () const
 Returns mime type name of object. More...
 
virtual char * GetObjectInfo (Int_t px, Int_t py) const
 Returns string containing info about the object at position (px,py). More...
 
virtual Option_tGetOption () const
 
virtual UInt_t GetUniqueID () const
 Return the unique object id. More...
 
virtual Bool_t HandleTimer (TTimer *timer)
 Execute action in response of a timer timing out. More...
 
Bool_t HasInconsistentHash () const
 Return true is the type of this object is known to have an inconsistent setup for Hash and RecursiveRemove (i.e. More...
 
virtual void Info (const char *method, const char *msgfmt,...) const
 Issue info message. More...
 
virtual Bool_t InheritsFrom (const char *classname) const
 Returns kTRUE if object inherits from class "classname". More...
 
virtual Bool_t InheritsFrom (const TClass *cl) const
 Returns kTRUE if object inherits from TClass cl. More...
 
virtual void Inspect () const
 Dump contents of this object in a graphics canvas. More...
 
void InvertBit (UInt_t f)
 
virtual Bool_t IsEqual (const TObject *obj) const
 Default equal comparison (objects are equal if they have the same address in memory). More...
 
virtual Bool_t IsFolder () const
 Returns kTRUE in case object contains browsable objects (like containers or lists of other objects). More...
 
R__ALWAYS_INLINE Bool_t IsOnHeap () const
 
R__ALWAYS_INLINE Bool_t IsZombie () const
 
void MayNotUse (const char *method) const
 Use this method to signal that a method (defined in a base class) may not be called in a derived class (in principle against good design since a child class should not provide less functionality than its parent, however, sometimes it is necessary). More...
 
virtual Bool_t Notify ()
 This method must be overridden to handle object notification. More...
 
void Obsolete (const char *method, const char *asOfVers, const char *removedFromVers) const
 Use this method to declare a method obsolete. More...
 
void operator delete (void *ptr)
 Operator delete. More...
 
void operator delete[] (void *ptr)
 Operator delete []. More...
 
voidoperator new (size_t sz)
 
voidoperator new (size_t sz, void *vp)
 
voidoperator new[] (size_t sz)
 
voidoperator new[] (size_t sz, void *vp)
 
TObjectoperator= (const TObject &rhs)
 TObject assignment operator. More...
 
virtual void Paint (Option_t *option="")
 This method must be overridden if a class wants to paint itself. More...
 
virtual void Pop ()
 Pop on object drawn in a pad to the top of the display list. More...
 
virtual Int_t Read (const char *name)
 Read contents of object with specified name from the current directory. More...
 
virtual void RecursiveRemove (TObject *obj)
 Recursively remove this object from a list. More...
 
void ResetBit (UInt_t f)
 
virtual void SaveAs (const char *filename="", Option_t *option="") const
 Save this object in the file specified by filename. More...
 
virtual void SavePrimitive (std::ostream &out, Option_t *option="")
 Save a primitive as a C++ statement(s) on output stream "out". More...
 
void SetBit (UInt_t f, Bool_t set)
 Set or unset the user status bits as specified in f. More...
 
void SetBit (UInt_t f)
 
virtual void SetDrawOption (Option_t *option="")
 Set drawing option for object. More...
 
virtual void SetUniqueID (UInt_t uid)
 Set the unique object id. More...
 
virtual void SysError (const char *method, const char *msgfmt,...) const
 Issue system error message. More...
 
R__ALWAYS_INLINE Bool_t TestBit (UInt_t f) const
 
Int_t TestBits (UInt_t f) const
 
virtual void UseCurrentStyle ()
 Set current style settings in this object This function is called when either TCanvas::UseCurrentStyle or TROOT::ForceStyle have been invoked. More...
 
virtual void Warning (const char *method, const char *msgfmt,...) const
 Issue warning message. More...
 
virtual Int_t Write (const char *name=0, Int_t option=0, Int_t bufsize=0)
 Write this object to the current directory. More...
 
virtual Int_t Write (const char *name=0, Int_t option=0, Int_t bufsize=0) const
 Write this object to the current directory. More...
 

Private Member Functions

CrossValidationFoldResult ProcessFold (UInt_t iFold, const OptionMap &methodInfo)
 Evaluates each fold in turn. More...
 

Private Attributes

Types::EAnalysisType fAnalysisType
 
TString fAnalysisTypeStr
 
Bool_t fCorrelations
 
TString fCvFactoryOptions
 
Bool_t fDrawProgressBar
 
std::unique_ptr< FactoryfFactory
 
std::unique_ptr< FactoryfFoldFactory
 
Bool_t fFoldFileOutput
 
Bool_t fFoldStatus
 If true: generate output file for each fold. More...
 
TString fJobName
 If true: dataset is prepared. More...
 
UInt_t fNumFolds
 
UInt_t fNumWorkerProcs
 Number of folds to prepare. More...
 
TString fOutputEnsembling
 
TString fOutputFactoryOptions
 Number of processes to use for fold evaluation. More...
 
TFilefOutputFile
 How to combine output of individual folds. More...
 
std::vector< CrossValidationResultfResults
 
Bool_t fROC
 
Bool_t fSilent
 
std::unique_ptr< CvSplitKFoldsfSplit
 
TString fSplitExprString
 
TString fSplitTypeStr
 
TString fTransformations
 
Bool_t fVerbose
 
TString fVerboseLevel
 

Additional Inherited Members

- Public Types inherited from TObject
enum  {
  kIsOnHeap = 0x01000000, kNotDeleted = 0x02000000, kZombie = 0x04000000, kInconsistent = 0x08000000,
  kBitMask = 0x00ffffff
}
 
enum  { kSingleKey = BIT(0), kOverwrite = BIT(1), kWriteDelete = BIT(2) }
 
enum  EDeprecatedStatusBits { kObjInCanvas = BIT(3) }
 
enum  EStatusBits {
  kCanDelete = BIT(0), kMustCleanup = BIT(3), kIsReferenced = BIT(4), kHasUUID = BIT(5),
  kCannotPick = BIT(6), kNoContextMenu = BIT(8), kInvalidObject = BIT(13)
}
 
- Static Public Member Functions inherited from TObject
static Long_t GetDtorOnly ()
 Return destructor only flag. More...
 
static Bool_t GetObjectStat ()
 Get status of object stat flag. More...
 
static void SetDtorOnly (void *obj)
 Set destructor only flag. More...
 
static void SetObjectStat (Bool_t stat)
 Turn on/off tracking of objects in the TObjectTable. More...
 
- Protected Member Functions inherited from TMVA::Envelope
 Envelope (const TString &name, DataLoader *dataloader=nullptr, TFile *file=nullptr, const TString options="")
 timer to measute the time. More...
 
DataInputHandlerGetDataLoaderDataInput ()
 Utility method to get TMVA::DataInputHandler reference from the DataLoader. More...
 
DataSetInfoGetDataLoaderDataSetInfo ()
 Utility method to get TMVA::DataSetInfo reference from the DataLoader. More...
 
DataSetManagerGetDataLoaderDataSetManager ()
 Utility method to get TMVA::DataSetManager pointer from the DataLoader. More...
 
TDirectoryRootBaseDir ()
 Utility method to get base dir directory from current file. More...
 
void WriteDataInformation (TMVA::DataSetInfo &fDataSetInfo, TMVA::Types::EAnalysisType fAnalysisType)
 method to save Train/Test information into the output file. More...
 
- Protected Member Functions inherited from TMVA::Configurable
void EnableLooseOptions (Bool_t b=kTRUE)
 
const TStringGetReferenceFile () const
 
Bool_t LooseOptionCheckingEnabled () const
 
void ResetSetFlag ()
 resets the IsSet flag for all declare options to be called before options are read from stream More...
 
void WriteOptionsReferenceToFile ()
 write complete options to output stream More...
 
- Protected Member Functions inherited from TObject
virtual void DoError (int level, const char *location, const char *fmt, va_list va) const
 Interface to ErrorHandler (protected). More...
 
void MakeZombie ()
 
- Protected Attributes inherited from TMVA::Envelope
std::shared_ptr< DataLoaderfDataLoader
 Booked method information. More...
 
std::shared_ptr< TFilefFile
 data More...
 
UInt_t fJobs
 procpool object More...
 
std::vector< OptionMapfMethods
 
Bool_t fModelPersistence
 file to save the results More...
 
Bool_t fSilentFile
 List of transformations to test. More...
 
TStopwatch fTimer
 number of jobs to run some high level algorithm in parallel More...
 
TString fTransformations
 flag for extra information More...
 
Bool_t fVerbose
 flag to save the trained model More...
 
TProcPool fWorkers
 if true dont produce file output More...
 
- Protected Attributes inherited from TMVA::Configurable
MsgLoggerfLogger
 
- Protected Attributes inherited from TNamed
TString fName
 
TString fTitle
 

#include <TMVA/CrossValidation.h>

Inheritance diagram for TMVA::CrossValidation:
[legend]

Constructor & Destructor Documentation

◆ CrossValidation() [1/2]

TMVA::CrossValidation::CrossValidation ( TString  jobName,
TMVA::DataLoader dataloader,
TString  options 
)
explicit

Definition at line 307 of file CrossValidation.cxx.

◆ CrossValidation() [2/2]

TMVA::CrossValidation::CrossValidation ( TString  jobName,
TMVA::DataLoader dataloader,
TFile outputFile,
TString  options 
)
explicit

Definition at line 276 of file CrossValidation.cxx.

◆ ~CrossValidation()

TMVA::CrossValidation::~CrossValidation ( )
default

Member Function Documentation

◆ Evaluate()

void TMVA::CrossValidation::Evaluate ( )
virtual

Does training, test set evaluation and performance evaluation of using cross-evalution.

Implements TMVA::Envelope.

Definition at line 585 of file CrossValidation.cxx.

◆ GetFactory()

Factory& TMVA::CrossValidation::GetFactory ( )
inline

Definition at line 138 of file CrossValidation.h.

◆ GetNumFolds()

UInt_t TMVA::CrossValidation::GetNumFolds ( )
inline

Definition at line 135 of file CrossValidation.h.

◆ GetResults()

const std::vector< TMVA::CrossValidationResult > & TMVA::CrossValidation::GetResults ( ) const

Definition at line 697 of file CrossValidation.cxx.

◆ GetSplitExpr()

TString TMVA::CrossValidation::GetSplitExpr ( )
inline

Definition at line 136 of file CrossValidation.h.

◆ InitOptions()

void TMVA::CrossValidation::InitOptions ( )

Definition at line 320 of file CrossValidation.cxx.

◆ ParseOptions()

void TMVA::CrossValidation::ParseOptions ( )
virtual


Method to parse the internal option string.

Reimplemented from TMVA::Envelope.

Definition at line 377 of file CrossValidation.cxx.

◆ ProcessFold()

TMVA::CrossValidationFoldResult TMVA::CrossValidation::ProcessFold ( UInt_t  iFold,
const OptionMap methodInfo 
)
private

Evaluates each fold in turn.

  • Prepares train and test data sets
  • Trains method
  • Evalutes on test set
  • Stores the evaluation internally
Parameters
iFoldfold to evaluate

Definition at line 504 of file CrossValidation.cxx.

◆ SetNumFolds()

void TMVA::CrossValidation::SetNumFolds ( UInt_t  i)

Definition at line 471 of file CrossValidation.cxx.

◆ SetSplitExpr()

void TMVA::CrossValidation::SetSplitExpr ( TString  splitExpr)

Definition at line 484 of file CrossValidation.cxx.

Member Data Documentation

◆ fAnalysisType

Types::EAnalysisType TMVA::CrossValidation::fAnalysisType
private

Definition at line 147 of file CrossValidation.h.

◆ fAnalysisTypeStr

TString TMVA::CrossValidation::fAnalysisTypeStr
private

Definition at line 148 of file CrossValidation.h.

◆ fCorrelations

Bool_t TMVA::CrossValidation::fCorrelations
private

Definition at line 150 of file CrossValidation.h.

◆ fCvFactoryOptions

TString TMVA::CrossValidation::fCvFactoryOptions
private

Definition at line 151 of file CrossValidation.h.

◆ fDrawProgressBar

Bool_t TMVA::CrossValidation::fDrawProgressBar
private

Definition at line 152 of file CrossValidation.h.

◆ fFactory

std::unique_ptr<Factory> TMVA::CrossValidation::fFactory
private

Definition at line 171 of file CrossValidation.h.

◆ fFoldFactory

std::unique_ptr<Factory> TMVA::CrossValidation::fFoldFactory
private

Definition at line 170 of file CrossValidation.h.

◆ fFoldFileOutput

Bool_t TMVA::CrossValidation::fFoldFileOutput
private

Definition at line 153 of file CrossValidation.h.

◆ fFoldStatus

Bool_t TMVA::CrossValidation::fFoldStatus
private

If true: generate output file for each fold.

Definition at line 154 of file CrossValidation.h.

◆ fJobName

TString TMVA::CrossValidation::fJobName
private

If true: dataset is prepared.

Definition at line 155 of file CrossValidation.h.

◆ fNumFolds

UInt_t TMVA::CrossValidation::fNumFolds
private

Definition at line 156 of file CrossValidation.h.

◆ fNumWorkerProcs

UInt_t TMVA::CrossValidation::fNumWorkerProcs
private

Number of folds to prepare.

Definition at line 157 of file CrossValidation.h.

◆ fOutputEnsembling

TString TMVA::CrossValidation::fOutputEnsembling
private

Definition at line 160 of file CrossValidation.h.

◆ fOutputFactoryOptions

TString TMVA::CrossValidation::fOutputFactoryOptions
private

Number of processes to use for fold evaluation.

(Default, no parallel evaluation)

Definition at line 159 of file CrossValidation.h.

◆ fOutputFile

TFile* TMVA::CrossValidation::fOutputFile
private

How to combine output of individual folds.

Definition at line 161 of file CrossValidation.h.

◆ fResults

std::vector<CrossValidationResult> TMVA::CrossValidation::fResults
private

Definition at line 164 of file CrossValidation.h.

◆ fROC

Bool_t TMVA::CrossValidation::fROC
private

Definition at line 165 of file CrossValidation.h.

◆ fSilent

Bool_t TMVA::CrossValidation::fSilent
private

Definition at line 162 of file CrossValidation.h.

◆ fSplit

std::unique_ptr<CvSplitKFolds> TMVA::CrossValidation::fSplit
private

Definition at line 172 of file CrossValidation.h.

◆ fSplitExprString

TString TMVA::CrossValidation::fSplitExprString
private

Definition at line 163 of file CrossValidation.h.

◆ fSplitTypeStr

TString TMVA::CrossValidation::fSplitTypeStr
private

Definition at line 149 of file CrossValidation.h.

◆ fTransformations

TString TMVA::CrossValidation::fTransformations
private

Definition at line 166 of file CrossValidation.h.

◆ fVerbose

Bool_t TMVA::CrossValidation::fVerbose
private

Definition at line 167 of file CrossValidation.h.

◆ fVerboseLevel

TString TMVA::CrossValidation::fVerboseLevel
private

Definition at line 168 of file CrossValidation.h.

Libraries for TMVA::CrossValidation:
[legend]

The documentation for this class was generated from the following files: