Logo ROOT  
Reference Guide
MethodBDT.cxx
Go to the documentation of this file.
1// Author: Andreas Hoecker, Joerg Stelzer, Helge Voss, Kai Voss, Eckhard v. Toerne, Jan Therhaag
2
3/**********************************************************************************
4 * Project: TMVA - a Root-integrated toolkit for multivariate data analysis *
5 * Package: TMVA *
6 * Class : MethodBDT (BDT = Boosted Decision Trees) *
7 * Web : http://tmva.sourceforge.net *
8 * *
9 * Description: *
10 * Analysis of Boosted Decision Trees *
11 * *
12 * Authors (alphabetical): *
13 * Andreas Hoecker <Andreas.Hocker@cern.ch> - CERN, Switzerland *
14 * Helge Voss <Helge.Voss@cern.ch> - MPI-K Heidelberg, Germany *
15 * Kai Voss <Kai.Voss@cern.ch> - U. of Victoria, Canada *
16 * Doug Schouten <dschoute@sfu.ca> - Simon Fraser U., Canada *
17 * Jan Therhaag <jan.therhaag@cern.ch> - U. of Bonn, Germany *
18 * Eckhard v. Toerne <evt@uni-bonn.de> - U of Bonn, Germany *
19 * *
20 * Copyright (c) 2005-2011: *
21 * CERN, Switzerland *
22 * U. of Victoria, Canada *
23 * MPI-K Heidelberg, Germany *
24 * U. of Bonn, Germany *
25 * *
26 * Redistribution and use in source and binary forms, with or without *
27 * modification, are permitted according to the terms listed in LICENSE *
28 * (http://tmva.sourceforge.net/LICENSE) *
29 **********************************************************************************/
30
31/*! \class TMVA::MethodBDT
32\ingroup TMVA
33
34Analysis of Boosted Decision Trees
35
36Boosted decision trees have been successfully used in High Energy
37Physics analysis for example by the MiniBooNE experiment
38(Yang-Roe-Zhu, physics/0508045). In Boosted Decision Trees, the
39selection is done on a majority vote on the result of several decision
40trees, which are all derived from the same training sample by
41supplying different event weights during the training.
42
43### Decision trees:
44
45Successive decision nodes are used to categorize the
46events out of the sample as either signal or background. Each node
47uses only a single discriminating variable to decide if the event is
48signal-like ("goes right") or background-like ("goes left"). This
49forms a tree like structure with "baskets" at the end (leave nodes),
50and an event is classified as either signal or background according to
51whether the basket where it ends up has been classified signal or
52background during the training. Training of a decision tree is the
53process to define the "cut criteria" for each node. The training
54starts with the root node. Here one takes the full training event
55sample and selects the variable and corresponding cut value that gives
56the best separation between signal and background at this stage. Using
57this cut criterion, the sample is then divided into two subsamples, a
58signal-like (right) and a background-like (left) sample. Two new nodes
59are then created for each of the two sub-samples and they are
60constructed using the same mechanism as described for the root
61node. The devision is stopped once a certain node has reached either a
62minimum number of events, or a minimum or maximum signal purity. These
63leave nodes are then called "signal" or "background" if they contain
64more signal respective background events from the training sample.
65
66### Boosting:
67
68The idea behind adaptive boosting (AdaBoost) is, that signal events
69from the training sample, that end up in a background node
70(and vice versa) are given a larger weight than events that are in
71the correct leave node. This results in a re-weighed training event
72sample, with which then a new decision tree can be developed.
73The boosting can be applied several times (typically 100-500 times)
74and one ends up with a set of decision trees (a forest).
75Gradient boosting works more like a function expansion approach, where
76each tree corresponds to a summand. The parameters for each summand (tree)
77are determined by the minimization of a error function (binomial log-
78likelihood for classification and Huber loss for regression).
79A greedy algorithm is used, which means, that only one tree is modified
80at a time, while the other trees stay fixed.
81
82### Bagging:
83
84In this particular variant of the Boosted Decision Trees the boosting
85is not done on the basis of previous training results, but by a simple
86stochastic re-sampling of the initial training event sample.
87
88### Random Trees:
89
90Similar to the "Random Forests" from Leo Breiman and Adele Cutler, it
91uses the bagging algorithm together and bases the determination of the
92best node-split during the training on a random subset of variables only
93which is individually chosen for each split.
94
95### Analysis:
96
97Applying an individual decision tree to a test event results in a
98classification of the event as either signal or background. For the
99boosted decision tree selection, an event is successively subjected to
100the whole set of decision trees and depending on how often it is
101classified as signal, a "likelihood" estimator is constructed for the
102event being signal or background. The value of this estimator is the
103one which is then used to select the events from an event sample, and
104the cut value on this estimator defines the efficiency and purity of
105the selection.
106
107*/
108
109
110#include "TMVA/MethodBDT.h"
111#include "TMVA/Config.h"
112
113#include "TMVA/BDTEventWrapper.h"
116#include "TMVA/Configurable.h"
117#include "TMVA/CrossEntropy.h"
118#include "TMVA/DecisionTree.h"
119#include "TMVA/DataSet.h"
120#include "TMVA/GiniIndex.h"
122#include "TMVA/Interval.h"
123#include "TMVA/IMethod.h"
124#include "TMVA/LogInterval.h"
125#include "TMVA/MethodBase.h"
127#include "TMVA/MsgLogger.h"
129#include "TMVA/PDF.h"
130#include "TMVA/Ranking.h"
131#include "TMVA/Results.h"
133#include "TMVA/SdivSqrtSplusB.h"
134#include "TMVA/SeparationBase.h"
135#include "TMVA/Timer.h"
136#include "TMVA/Tools.h"
137#include "TMVA/Types.h"
138
139#include "Riostream.h"
140#include "TDirectory.h"
141#include "TRandom3.h"
142#include "TMath.h"
143#include "TMatrixTSym.h"
144#include "TGraph.h"
145
146#include <algorithm>
147#include <cmath>
148#include <fstream>
149#include <numeric>
150#include <unordered_map>
151
152using std::vector;
153using std::make_pair;
154
156
158
160
161////////////////////////////////////////////////////////////////////////////////
162/// The standard constructor for the "boosted decision trees".
163
165 const TString& methodTitle,
166 DataSetInfo& theData,
167 const TString& theOption ) :
168 TMVA::MethodBase( jobName, Types::kBDT, methodTitle, theData, theOption)
169 , fTrainSample(0)
170 , fNTrees(0)
171 , fSigToBkgFraction(0)
172 , fAdaBoostBeta(0)
173// , fTransitionPoint(0)
174 , fShrinkage(0)
175 , fBaggedBoost(kFALSE)
176 , fBaggedGradBoost(kFALSE)
177// , fSumOfWeights(0)
178 , fMinNodeEvents(0)
179 , fMinNodeSize(5)
180 , fMinNodeSizeS("5%")
181 , fNCuts(0)
182 , fUseFisherCuts(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
183 , fMinLinCorrForFisher(.8) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
184 , fUseExclusiveVars(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
185 , fUseYesNoLeaf(kFALSE)
186 , fNodePurityLimit(0)
187 , fNNodesMax(0)
188 , fMaxDepth(0)
189 , fPruneMethod(DecisionTree::kNoPruning)
190 , fPruneStrength(0)
191 , fFValidationEvents(0)
192 , fAutomatic(kFALSE)
193 , fRandomisedTrees(kFALSE)
194 , fUseNvars(0)
195 , fUsePoissonNvars(0) // don't use this initialisation, only here to make Coverity happy. Is set in Init()
196 , fUseNTrainEvents(0)
197 , fBaggedSampleFraction(0)
198 , fNoNegWeightsInTraining(kFALSE)
199 , fInverseBoostNegWeights(kFALSE)
200 , fPairNegWeightsGlobal(kFALSE)
201 , fTrainWithNegWeights(kFALSE)
202 , fDoBoostMonitor(kFALSE)
203 , fITree(0)
204 , fBoostWeight(0)
205 , fErrorFraction(0)
206 , fCss(0)
207 , fCts_sb(0)
208 , fCtb_ss(0)
209 , fCbb(0)
210 , fDoPreselection(kFALSE)
211 , fSkipNormalization(kFALSE)
212 , fHistoricBool(kFALSE)
213{
214 fMonitorNtuple = NULL;
215 fSepType = NULL;
217}
218
219////////////////////////////////////////////////////////////////////////////////
220
222 const TString& theWeightFile)
223 : TMVA::MethodBase( Types::kBDT, theData, theWeightFile)
224 , fTrainSample(0)
225 , fNTrees(0)
226 , fSigToBkgFraction(0)
227 , fAdaBoostBeta(0)
228// , fTransitionPoint(0)
229 , fShrinkage(0)
230 , fBaggedBoost(kFALSE)
231 , fBaggedGradBoost(kFALSE)
232// , fSumOfWeights(0)
233 , fMinNodeEvents(0)
234 , fMinNodeSize(5)
235 , fMinNodeSizeS("5%")
236 , fNCuts(0)
237 , fUseFisherCuts(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
238 , fMinLinCorrForFisher(.8) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
239 , fUseExclusiveVars(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
240 , fUseYesNoLeaf(kFALSE)
241 , fNodePurityLimit(0)
242 , fNNodesMax(0)
243 , fMaxDepth(0)
244 , fPruneMethod(DecisionTree::kNoPruning)
245 , fPruneStrength(0)
246 , fFValidationEvents(0)
247 , fAutomatic(kFALSE)
248 , fRandomisedTrees(kFALSE)
249 , fUseNvars(0)
250 , fUsePoissonNvars(0) // don't use this initialisation, only here to make Coverity happy. Is set in Init()
251 , fUseNTrainEvents(0)
252 , fBaggedSampleFraction(0)
253 , fNoNegWeightsInTraining(kFALSE)
254 , fInverseBoostNegWeights(kFALSE)
255 , fPairNegWeightsGlobal(kFALSE)
256 , fTrainWithNegWeights(kFALSE)
257 , fDoBoostMonitor(kFALSE)
258 , fITree(0)
259 , fBoostWeight(0)
260 , fErrorFraction(0)
261 , fCss(0)
262 , fCts_sb(0)
263 , fCtb_ss(0)
264 , fCbb(0)
265 , fDoPreselection(kFALSE)
266 , fSkipNormalization(kFALSE)
267 , fHistoricBool(kFALSE)
268{
269 fMonitorNtuple = NULL;
270 fSepType = NULL;
272 // constructor for calculating BDT-MVA using previously generated decision trees
273 // the result of the previous training (the decision trees) are read in via the
274 // weight file. Make sure the the variables correspond to the ones used in
275 // creating the "weight"-file
276}
277
278////////////////////////////////////////////////////////////////////////////////
279/// BDT can handle classification with multiple classes and regression with one regression-target.
280
282{
283 if (type == Types::kClassification && numberClasses == 2) return kTRUE;
284 if (type == Types::kMulticlass ) return kTRUE;
285 if( type == Types::kRegression && numberTargets == 1 ) return kTRUE;
286 return kFALSE;
287}
288
289////////////////////////////////////////////////////////////////////////////////
290/// Define the options (their key words). That can be set in the option string.
291///
292/// know options:
293///
294/// - nTrees number of trees in the forest to be created
295/// - BoostType the boosting type for the trees in the forest (AdaBoost e.t.c..).
296/// Known:
297/// - AdaBoost
298/// - AdaBoostR2 (Adaboost for regression)
299/// - Bagging
300/// - GradBoost
301/// - AdaBoostBeta the boosting parameter, beta, for AdaBoost
302/// - UseRandomisedTrees choose at each node splitting a random set of variables
303/// - UseNvars use UseNvars variables in randomised trees
304/// - UsePoisson Nvars use UseNvars not as fixed number but as mean of a poisson distribution
305/// - SeparationType the separation criterion applied in the node splitting.
306/// Known:
307/// - GiniIndex
308/// - MisClassificationError
309/// - CrossEntropy
310/// - SDivSqrtSPlusB
311/// - MinNodeSize: minimum percentage of training events in a leaf node (leaf criteria, stop splitting)
312/// - nCuts: the number of steps in the optimisation of the cut for a node (if < 0, then
313/// step size is determined by the events)
314/// - UseFisherCuts: use multivariate splits using the Fisher criterion
315/// - UseYesNoLeaf decide if the classification is done simply by the node type, or the S/B
316/// (from the training) in the leaf node
317/// - NodePurityLimit the minimum purity to classify a node as a signal node (used in pruning and boosting to determine
318/// misclassification error rate)
319/// - PruneMethod The Pruning method.
320/// Known:
321/// - NoPruning // switch off pruning completely
322/// - ExpectedError
323/// - CostComplexity
324/// - PruneStrength a parameter to adjust the amount of pruning. Should be large enough such that overtraining is avoided.
325/// - PruningValFraction number of events to use for optimizing pruning (only if PruneStrength < 0, i.e. automatic pruning)
326/// - NegWeightTreatment
327/// - IgnoreNegWeightsInTraining Ignore negative weight events in the training.
328/// - DecreaseBoostWeight Boost ev. with neg. weight with 1/boostweight instead of boostweight
329/// - PairNegWeightsGlobal Pair ev. with neg. and pos. weights in training sample and "annihilate" them
330/// - MaxDepth maximum depth of the decision tree allowed before further splitting is stopped
331/// - SkipNormalization Skip normalization at initialization, to keep expectation value of BDT output
332/// according to the fraction of events
333
335{
336 DeclareOptionRef(fNTrees, "NTrees", "Number of trees in the forest");
337 if (DoRegression()) {
338 DeclareOptionRef(fMaxDepth=50,"MaxDepth","Max depth of the decision tree allowed");
339 }else{
340 DeclareOptionRef(fMaxDepth=3,"MaxDepth","Max depth of the decision tree allowed");
341 }
342
343 TString tmp="5%"; if (DoRegression()) tmp="0.2%";
344 DeclareOptionRef(fMinNodeSizeS=tmp, "MinNodeSize", "Minimum percentage of training events required in a leaf node (default: Classification: 5%, Regression: 0.2%)");
345 // MinNodeSize: minimum percentage of training events in a leaf node (leaf criteria, stop splitting)
346 DeclareOptionRef(fNCuts, "nCuts", "Number of grid points in variable range used in finding optimal cut in node splitting");
347
348 DeclareOptionRef(fBoostType, "BoostType", "Boosting type for the trees in the forest (note: AdaCost is still experimental)");
349
350 AddPreDefVal(TString("AdaBoost"));
351 AddPreDefVal(TString("RealAdaBoost"));
352 AddPreDefVal(TString("AdaCost"));
353 AddPreDefVal(TString("Bagging"));
354 // AddPreDefVal(TString("RegBoost"));
355 AddPreDefVal(TString("AdaBoostR2"));
356 AddPreDefVal(TString("Grad"));
357 if (DoRegression()) {
358 fBoostType = "AdaBoostR2";
359 }else{
360 fBoostType = "AdaBoost";
361 }
362 DeclareOptionRef(fAdaBoostR2Loss="Quadratic", "AdaBoostR2Loss", "Type of Loss function in AdaBoostR2");
363 AddPreDefVal(TString("Linear"));
364 AddPreDefVal(TString("Quadratic"));
365 AddPreDefVal(TString("Exponential"));
366
367 DeclareOptionRef(fBaggedBoost=kFALSE, "UseBaggedBoost","Use only a random subsample of all events for growing the trees in each boost iteration.");
368 DeclareOptionRef(fShrinkage = 1.0, "Shrinkage", "Learning rate for BoostType=Grad algorithm");
369 DeclareOptionRef(fAdaBoostBeta=.5, "AdaBoostBeta", "Learning rate for AdaBoost algorithm");
370 DeclareOptionRef(fRandomisedTrees,"UseRandomisedTrees","Determine at each node splitting the cut variable only as the best out of a random subset of variables (like in RandomForests)");
371 DeclareOptionRef(fUseNvars,"UseNvars","Size of the subset of variables used with RandomisedTree option");
372 DeclareOptionRef(fUsePoissonNvars,"UsePoissonNvars", "Interpret \"UseNvars\" not as fixed number but as mean of a Poisson distribution in each split with RandomisedTree option");
373 DeclareOptionRef(fBaggedSampleFraction=.6,"BaggedSampleFraction","Relative size of bagged event sample to original size of the data sample (used whenever bagging is used (i.e. UseBaggedBoost, Bagging,)" );
374
375 DeclareOptionRef(fUseYesNoLeaf=kTRUE, "UseYesNoLeaf",
376 "Use Sig or Bkg categories, or the purity=S/(S+B) as classification of the leaf node -> Real-AdaBoost");
377 if (DoRegression()) {
378 fUseYesNoLeaf = kFALSE;
379 }
380
381 DeclareOptionRef(fNegWeightTreatment="InverseBoostNegWeights","NegWeightTreatment","How to treat events with negative weights in the BDT training (particular the boosting) : IgnoreInTraining; Boost With inverse boostweight; Pair events with negative and positive weights in training sample and *annihilate* them (experimental!)");
382 AddPreDefVal(TString("InverseBoostNegWeights"));
383 AddPreDefVal(TString("IgnoreNegWeightsInTraining"));
384 AddPreDefVal(TString("NoNegWeightsInTraining")); // well, let's be nice to users and keep at least this old name anyway ..
385 AddPreDefVal(TString("PairNegWeightsGlobal"));
386 AddPreDefVal(TString("Pray"));
387
388
389
390 DeclareOptionRef(fCss=1., "Css", "AdaCost: cost of true signal selected signal");
391 DeclareOptionRef(fCts_sb=1.,"Cts_sb","AdaCost: cost of true signal selected bkg");
392 DeclareOptionRef(fCtb_ss=1.,"Ctb_ss","AdaCost: cost of true bkg selected signal");
393 DeclareOptionRef(fCbb=1., "Cbb", "AdaCost: cost of true bkg selected bkg ");
394
395 DeclareOptionRef(fNodePurityLimit=0.5, "NodePurityLimit", "In boosting/pruning, nodes with purity > NodePurityLimit are signal; background otherwise.");
396
397
398 DeclareOptionRef(fSepTypeS, "SeparationType", "Separation criterion for node splitting");
399 AddPreDefVal(TString("CrossEntropy"));
400 AddPreDefVal(TString("GiniIndex"));
401 AddPreDefVal(TString("GiniIndexWithLaplace"));
402 AddPreDefVal(TString("MisClassificationError"));
403 AddPreDefVal(TString("SDivSqrtSPlusB"));
404 AddPreDefVal(TString("RegressionVariance"));
405 if (DoRegression()) {
406 fSepTypeS = "RegressionVariance";
407 }else{
408 fSepTypeS = "GiniIndex";
409 }
410
411 DeclareOptionRef(fRegressionLossFunctionBDTGS = "Huber", "RegressionLossFunctionBDTG", "Loss function for BDTG regression.");
412 AddPreDefVal(TString("Huber"));
413 AddPreDefVal(TString("AbsoluteDeviation"));
414 AddPreDefVal(TString("LeastSquares"));
415
416 DeclareOptionRef(fHuberQuantile = 0.7, "HuberQuantile", "In the Huber loss function this is the quantile that separates the core from the tails in the residuals distribution.");
417
418 DeclareOptionRef(fDoBoostMonitor=kFALSE,"DoBoostMonitor","Create control plot with ROC integral vs tree number");
419
420 DeclareOptionRef(fUseFisherCuts=kFALSE, "UseFisherCuts", "Use multivariate splits using the Fisher criterion");
421 DeclareOptionRef(fMinLinCorrForFisher=.8,"MinLinCorrForFisher", "The minimum linear correlation between two variables demanded for use in Fisher criterion in node splitting");
422 DeclareOptionRef(fUseExclusiveVars=kFALSE,"UseExclusiveVars","Variables already used in fisher criterion are not anymore analysed individually for node splitting");
423
424
425 DeclareOptionRef(fDoPreselection=kFALSE,"DoPreselection","and and apply automatic pre-selection for 100% efficient signal (bkg) cuts prior to training");
426
427
428 DeclareOptionRef(fSigToBkgFraction=1,"SigToBkgFraction","Sig to Bkg ratio used in Training (similar to NodePurityLimit, which cannot be used in real adaboost");
429
430 DeclareOptionRef(fPruneMethodS, "PruneMethod", "Note: for BDTs use small trees (e.g.MaxDepth=3) and NoPruning: Pruning: Method used for pruning (removal) of statistically insignificant branches ");
431 AddPreDefVal(TString("NoPruning"));
432 AddPreDefVal(TString("ExpectedError"));
433 AddPreDefVal(TString("CostComplexity"));
434
435 DeclareOptionRef(fPruneStrength, "PruneStrength", "Pruning strength");
436
437 DeclareOptionRef(fFValidationEvents=0.5, "PruningValFraction", "Fraction of events to use for optimizing automatic pruning.");
438
439 DeclareOptionRef(fSkipNormalization=kFALSE, "SkipNormalization", "Skip normalization at initialization, to keep expectation value of BDT output according to the fraction of events");
440
441 // deprecated options, still kept for the moment:
442 DeclareOptionRef(fMinNodeEvents=0, "nEventsMin", "deprecated: Use MinNodeSize (in % of training events) instead");
443
444 DeclareOptionRef(fBaggedGradBoost=kFALSE, "UseBaggedGrad","deprecated: Use *UseBaggedBoost* instead: Use only a random subsample of all events for growing the trees in each iteration.");
445 DeclareOptionRef(fBaggedSampleFraction, "GradBaggingFraction","deprecated: Use *BaggedSampleFraction* instead: Defines the fraction of events to be used in each iteration, e.g. when UseBaggedGrad=kTRUE. ");
446 DeclareOptionRef(fUseNTrainEvents,"UseNTrainEvents","deprecated: Use *BaggedSampleFraction* instead: Number of randomly picked training events used in randomised (and bagged) trees");
447 DeclareOptionRef(fNNodesMax,"NNodesMax","deprecated: Use MaxDepth instead to limit the tree size" );
448
449
450}
451
452////////////////////////////////////////////////////////////////////////////////
453/// Options that are used ONLY for the READER to ensure backward compatibility.
454
457
458
459 DeclareOptionRef(fHistoricBool=kTRUE, "UseWeightedTrees",
460 "Use weighted trees or simple average in classification from the forest");
461 DeclareOptionRef(fHistoricBool=kFALSE, "PruneBeforeBoost", "Flag to prune the tree before applying boosting algorithm");
462 DeclareOptionRef(fHistoricBool=kFALSE,"RenormByClass","Individually re-normalize each event class to the original size after boosting");
463
464 AddPreDefVal(TString("NegWeightTreatment"),TString("IgnoreNegWeights"));
465
466}
467
468////////////////////////////////////////////////////////////////////////////////
469/// The option string is decoded, for available options see "DeclareOptions".
470
472{
473 fSepTypeS.ToLower();
474 if (fSepTypeS == "misclassificationerror") fSepType = new MisClassificationError();
475 else if (fSepTypeS == "giniindex") fSepType = new GiniIndex();
476 else if (fSepTypeS == "giniindexwithlaplace") fSepType = new GiniIndexWithLaplace();
477 else if (fSepTypeS == "crossentropy") fSepType = new CrossEntropy();
478 else if (fSepTypeS == "sdivsqrtsplusb") fSepType = new SdivSqrtSplusB();
479 else if (fSepTypeS == "regressionvariance") fSepType = NULL;
480 else {
481 Log() << kINFO << GetOptions() << Endl;
482 Log() << kFATAL << "<ProcessOptions> unknown Separation Index option " << fSepTypeS << " called" << Endl;
483 }
484
485 if(!(fHuberQuantile >= 0.0 && fHuberQuantile <= 1.0)){
486 Log() << kINFO << GetOptions() << Endl;
487 Log() << kFATAL << "<ProcessOptions> Huber Quantile must be in range [0,1]. Value given, " << fHuberQuantile << ", does not match this criteria" << Endl;
488 }
489
490
491 fRegressionLossFunctionBDTGS.ToLower();
492 if (fRegressionLossFunctionBDTGS == "huber") fRegressionLossFunctionBDTG = new HuberLossFunctionBDT(fHuberQuantile);
493 else if (fRegressionLossFunctionBDTGS == "leastsquares") fRegressionLossFunctionBDTG = new LeastSquaresLossFunctionBDT();
494 else if (fRegressionLossFunctionBDTGS == "absolutedeviation") fRegressionLossFunctionBDTG = new AbsoluteDeviationLossFunctionBDT();
495 else {
496 Log() << kINFO << GetOptions() << Endl;
497 Log() << kFATAL << "<ProcessOptions> unknown Regression Loss Function BDT option " << fRegressionLossFunctionBDTGS << " called" << Endl;
498 }
499
500 fPruneMethodS.ToLower();
501 if (fPruneMethodS == "expectederror") fPruneMethod = DecisionTree::kExpectedErrorPruning;
502 else if (fPruneMethodS == "costcomplexity") fPruneMethod = DecisionTree::kCostComplexityPruning;
503 else if (fPruneMethodS == "nopruning") fPruneMethod = DecisionTree::kNoPruning;
504 else {
505 Log() << kINFO << GetOptions() << Endl;
506 Log() << kFATAL << "<ProcessOptions> unknown PruneMethod " << fPruneMethodS << " option called" << Endl;
507 }
508 if (fPruneStrength < 0 && (fPruneMethod != DecisionTree::kNoPruning) && fBoostType!="Grad") fAutomatic = kTRUE;
509 else fAutomatic = kFALSE;
510 if (fAutomatic && fPruneMethod==DecisionTree::kExpectedErrorPruning){
511 Log() << kFATAL
512 << "Sorry automatic pruning strength determination is not implemented yet for ExpectedErrorPruning" << Endl;
513 }
514
515
516 if (fMinNodeEvents > 0){
517 fMinNodeSize = Double_t(fMinNodeEvents*100.) / Data()->GetNTrainingEvents();
518 Log() << kWARNING << "You have explicitly set ** nEventsMin = " << fMinNodeEvents<<" ** the min absolute number \n"
519 << "of events in a leaf node. This is DEPRECATED, please use the option \n"
520 << "*MinNodeSize* giving the relative number as percentage of training \n"
521 << "events instead. \n"
522 << "nEventsMin="<<fMinNodeEvents<< "--> MinNodeSize="<<fMinNodeSize<<"%"
523 << Endl;
524 Log() << kWARNING << "Note also that explicitly setting *nEventsMin* so far OVERWRITES the option recommended \n"
525 << " *MinNodeSize* = " << fMinNodeSizeS << " option !!" << Endl ;
526 fMinNodeSizeS = Form("%F3.2",fMinNodeSize);
527
528 }else{
529 SetMinNodeSize(fMinNodeSizeS);
530 }
531
532
533 fAdaBoostR2Loss.ToLower();
534
535 if (fBoostType=="Grad") {
536 fPruneMethod = DecisionTree::kNoPruning;
537 if (fNegWeightTreatment=="InverseBoostNegWeights"){
538 Log() << kINFO << "the option NegWeightTreatment=InverseBoostNegWeights does"
539 << " not exist for BoostType=Grad" << Endl;
540 Log() << kINFO << "--> change to new default NegWeightTreatment=Pray" << Endl;
541 Log() << kDEBUG << "i.e. simply keep them as if which should work fine for Grad Boost" << Endl;
542 fNegWeightTreatment="Pray";
543 fNoNegWeightsInTraining=kFALSE;
544 }
545 } else if (fBoostType=="RealAdaBoost"){
546 fBoostType = "AdaBoost";
547 fUseYesNoLeaf = kFALSE;
548 } else if (fBoostType=="AdaCost"){
549 fUseYesNoLeaf = kFALSE;
550 }
551
552 if (fFValidationEvents < 0.0) fFValidationEvents = 0.0;
553 if (fAutomatic && fFValidationEvents > 0.5) {
554 Log() << kWARNING << "You have chosen to use more than half of your training sample "
555 << "to optimize the automatic pruning algorithm. This is probably wasteful "
556 << "and your overall results will be degraded. Are you sure you want this?"
557 << Endl;
558 }
559
560
561 if (this->Data()->HasNegativeEventWeights()){
562 Log() << kINFO << " You are using a Monte Carlo that has also negative weights. "
563 << "That should in principle be fine as long as on average you end up with "
564 << "something positive. For this you have to make sure that the minimal number "
565 << "of (un-weighted) events demanded for a tree node (currently you use: MinNodeSize="
566 << fMinNodeSizeS << " ("<< fMinNodeSize << "%)"
567 <<", (or the deprecated equivalent nEventsMin) you can set this via the "
568 <<"BDT option string when booking the "
569 << "classifier) is large enough to allow for reasonable averaging!!! "
570 << " If this does not help.. maybe you want to try the option: IgnoreNegWeightsInTraining "
571 << "which ignores events with negative weight in the training. " << Endl
572 << Endl << "Note: You'll get a WARNING message during the training if that should ever happen" << Endl;
573 }
574
575 if (DoRegression()) {
576 if (fUseYesNoLeaf && !IsConstructedFromWeightFile()){
577 Log() << kWARNING << "Regression Trees do not work with fUseYesNoLeaf=TRUE --> I will set it to FALSE" << Endl;
578 fUseYesNoLeaf = kFALSE;
579 }
580
581 if (fSepType != NULL){
582 Log() << kWARNING << "Regression Trees do not work with Separation type other than <RegressionVariance> --> I will use it instead" << Endl;
583 fSepType = NULL;
584 }
585 if (fUseFisherCuts){
586 Log() << kWARNING << "Sorry, UseFisherCuts is not available for regression analysis, I will ignore it!" << Endl;
587 fUseFisherCuts = kFALSE;
588 }
589 if (fNCuts < 0) {
590 Log() << kWARNING << "Sorry, the option of nCuts<0 using a more elaborate node splitting algorithm " << Endl;
591 Log() << kWARNING << "is not implemented for regression analysis ! " << Endl;
592 Log() << kWARNING << "--> I switch do default nCuts = 20 and use standard node splitting"<<Endl;
593 fNCuts=20;
594 }
595 }
596 if (fRandomisedTrees){
597 Log() << kINFO << " Randomised trees use no pruning" << Endl;
598 fPruneMethod = DecisionTree::kNoPruning;
599 // fBoostType = "Bagging";
600 }
601
602 if (fUseFisherCuts) {
603 Log() << kWARNING << "When using the option UseFisherCuts, the other option nCuts<0 (i.e. using" << Endl;
604 Log() << " a more elaborate node splitting algorithm) is not implemented. " << Endl;
605 //I will switch o " << Endl;
606 //Log() << "--> I switch do default nCuts = 20 and use standard node splitting WITH possible Fisher criteria"<<Endl;
607 fNCuts=20;
608 }
609
610 if (fNTrees==0){
611 Log() << kERROR << " Zero Decision Trees demanded... that does not work !! "
612 << " I set it to 1 .. just so that the program does not crash"
613 << Endl;
614 fNTrees = 1;
615 }
616
617 fNegWeightTreatment.ToLower();
618 if (fNegWeightTreatment == "ignorenegweightsintraining") fNoNegWeightsInTraining = kTRUE;
619 else if (fNegWeightTreatment == "nonegweightsintraining") fNoNegWeightsInTraining = kTRUE;
620 else if (fNegWeightTreatment == "inverseboostnegweights") fInverseBoostNegWeights = kTRUE;
621 else if (fNegWeightTreatment == "pairnegweightsglobal") fPairNegWeightsGlobal = kTRUE;
622 else if (fNegWeightTreatment == "pray") Log() << kDEBUG << "Yes, good luck with praying " << Endl;
623 else {
624 Log() << kINFO << GetOptions() << Endl;
625 Log() << kFATAL << "<ProcessOptions> unknown option for treating negative event weights during training " << fNegWeightTreatment << " requested" << Endl;
626 }
627
628 if (fNegWeightTreatment == "pairnegweightsglobal")
629 Log() << kWARNING << " you specified the option NegWeightTreatment=PairNegWeightsGlobal : This option is still considered EXPERIMENTAL !! " << Endl;
630
631
632 // dealing with deprecated options !
633 if (fNNodesMax>0) {
634 UInt_t tmp=1; // depth=0 == 1 node
635 fMaxDepth=0;
636 while (tmp < fNNodesMax){
637 tmp+=2*tmp;
638 fMaxDepth++;
639 }
640 Log() << kWARNING << "You have specified a deprecated option *NNodesMax="<<fNNodesMax
641 << "* \n this has been translated to MaxDepth="<<fMaxDepth<<Endl;
642 }
643
644
645 if (fUseNTrainEvents>0){
646 fBaggedSampleFraction = (Double_t) fUseNTrainEvents/Data()->GetNTrainingEvents();
647 Log() << kWARNING << "You have specified a deprecated option *UseNTrainEvents="<<fUseNTrainEvents
648 << "* \n this has been translated to BaggedSampleFraction="<<fBaggedSampleFraction<<"(%)"<<Endl;
649 }
650
651 if (fBoostType=="Bagging") fBaggedBoost = kTRUE;
652 if (fBaggedGradBoost){
653 fBaggedBoost = kTRUE;
654 Log() << kWARNING << "You have specified a deprecated option *UseBaggedGrad* --> please use *UseBaggedBoost* instead" << Endl;
655 }
656
657}
658
659////////////////////////////////////////////////////////////////////////////////
660
662 if (sizeInPercent > 0 && sizeInPercent < 50){
663 fMinNodeSize=sizeInPercent;
664
665 } else {
666 Log() << kFATAL << "you have demanded a minimal node size of "
667 << sizeInPercent << "% of the training events.. \n"
668 << " that somehow does not make sense "<<Endl;
669 }
670
671}
672
673////////////////////////////////////////////////////////////////////////////////
674
676 sizeInPercent.ReplaceAll("%","");
677 sizeInPercent.ReplaceAll(" ","");
678 if (sizeInPercent.IsFloat()) SetMinNodeSize(sizeInPercent.Atof());
679 else {
680 Log() << kFATAL << "I had problems reading the option MinNodeEvents, which "
681 << "after removing a possible % sign now reads " << sizeInPercent << Endl;
682 }
683}
684
685////////////////////////////////////////////////////////////////////////////////
686/// Common initialisation with defaults for the BDT-Method.
687
689{
690 fNTrees = 800;
691 if (fAnalysisType == Types::kClassification || fAnalysisType == Types::kMulticlass ) {
692 fMaxDepth = 3;
693 fBoostType = "AdaBoost";
694 if(DataInfo().GetNClasses()!=0) //workaround for multiclass application
695 fMinNodeSize = 5.;
696 }else {
697 fMaxDepth = 50;
698 fBoostType = "AdaBoostR2";
699 fAdaBoostR2Loss = "Quadratic";
700 if(DataInfo().GetNClasses()!=0) //workaround for multiclass application
701 fMinNodeSize = .2;
702 }
703
704
705 fNCuts = 20;
706 fPruneMethodS = "NoPruning";
707 fPruneMethod = DecisionTree::kNoPruning;
708 fPruneStrength = 0;
709 fAutomatic = kFALSE;
710 fFValidationEvents = 0.5;
711 fRandomisedTrees = kFALSE;
712 // fUseNvars = (GetNvar()>12) ? UInt_t(GetNvar()/8) : TMath::Max(UInt_t(2),UInt_t(GetNvar()/3));
713 fUseNvars = UInt_t(TMath::Sqrt(GetNvar())+0.6);
714 fUsePoissonNvars = kTRUE;
715 fShrinkage = 1.0;
716// fSumOfWeights = 0.0;
717
718 // reference cut value to distinguish signal-like from background-like events
719 SetSignalReferenceCut( 0 );
720}
721
722
723////////////////////////////////////////////////////////////////////////////////
724/// Reset the method, as if it had just been instantiated (forget all training etc.).
725
727{
728 // I keep the BDT EventSample and its Validation sample (eventually they should all
729 // disappear and just use the DataSet samples ..
730
731 // remove all the trees
732 for (UInt_t i=0; i<fForest.size(); i++) delete fForest[i];
733 fForest.clear();
734
735 fBoostWeights.clear();
736 if (fMonitorNtuple) { fMonitorNtuple->Delete(); fMonitorNtuple=NULL; }
737 fVariableImportance.clear();
738 fResiduals.clear();
739 fLossFunctionEventInfo.clear();
740 // now done in "InitEventSample" which is called in "Train"
741 // reset all previously stored/accumulated BOOST weights in the event sample
742 //for (UInt_t iev=0; iev<fEventSample.size(); iev++) fEventSample[iev]->SetBoostWeight(1.);
743 if (Data()) Data()->DeleteResults(GetMethodName(), Types::kTraining, GetAnalysisType());
744 Log() << kDEBUG << " successfully(?) reset the method " << Endl;
745}
746
747
748////////////////////////////////////////////////////////////////////////////////
749/// Destructor.
750///
751/// - Note: fEventSample and ValidationSample are already deleted at the end of TRAIN
752/// When they are not used anymore
753
755{
756 for (UInt_t i=0; i<fForest.size(); i++) delete fForest[i];
757}
758
759////////////////////////////////////////////////////////////////////////////////
760/// Initialize the event sample (i.e. reset the boost-weights... etc).
761
763{
764 if (!HasTrainingTree()) Log() << kFATAL << "<Init> Data().TrainingTree() is zero pointer" << Endl;
765
766 if (fEventSample.size() > 0) { // do not re-initialise the event sample, just set all boostweights to 1. as if it were untouched
767 // reset all previously stored/accumulated BOOST weights in the event sample
768 for (UInt_t iev=0; iev<fEventSample.size(); iev++) fEventSample[iev]->SetBoostWeight(1.);
769 } else {
770 Data()->SetCurrentType(Types::kTraining);
771 UInt_t nevents = Data()->GetNTrainingEvents();
772
773 std::vector<const TMVA::Event*> tmpEventSample;
774 for (Long64_t ievt=0; ievt<nevents; ievt++) {
775 // const Event *event = new Event(*(GetEvent(ievt)));
776 Event* event = new Event( *GetTrainingEvent(ievt) );
777 tmpEventSample.push_back(event);
778 }
779
780 if (!DoRegression()) DeterminePreselectionCuts(tmpEventSample);
781 else fDoPreselection = kFALSE; // just to make sure...
782
783 for (UInt_t i=0; i<tmpEventSample.size(); i++) delete tmpEventSample[i];
784
785
786 Bool_t firstNegWeight=kTRUE;
787 Bool_t firstZeroWeight=kTRUE;
788 for (Long64_t ievt=0; ievt<nevents; ievt++) {
789 // const Event *event = new Event(*(GetEvent(ievt)));
790 // const Event* event = new Event( *GetTrainingEvent(ievt) );
791 Event* event = new Event( *GetTrainingEvent(ievt) );
792 if (fDoPreselection){
793 if (TMath::Abs(ApplyPreselectionCuts(event)) > 0.05) {
794 delete event;
795 continue;
796 }
797 }
798
799 if (event->GetWeight() < 0 && (IgnoreEventsWithNegWeightsInTraining() || fNoNegWeightsInTraining)){
800 if (firstNegWeight) {
801 Log() << kWARNING << " Note, you have events with negative event weight in the sample, but you've chosen to ignore them" << Endl;
802 firstNegWeight=kFALSE;
803 }
804 delete event;
805 }else if (event->GetWeight()==0){
806 if (firstZeroWeight) {
807 firstZeroWeight = kFALSE;
808 Log() << "Events with weight == 0 are going to be simply ignored " << Endl;
809 }
810 delete event;
811 }else{
812 if (event->GetWeight() < 0) {
813 fTrainWithNegWeights=kTRUE;
814 if (firstNegWeight){
815 firstNegWeight = kFALSE;
816 if (fPairNegWeightsGlobal){
817 Log() << kWARNING << "Events with negative event weights are found and "
818 << " will be removed prior to the actual BDT training by global "
819 << " paring (and subsequent annihilation) with positiv weight events"
820 << Endl;
821 }else{
822 Log() << kWARNING << "Events with negative event weights are USED during "
823 << "the BDT training. This might cause problems with small node sizes "
824 << "or with the boosting. Please remove negative events from training "
825 << "using the option *IgnoreEventsWithNegWeightsInTraining* in case you "
826 << "observe problems with the boosting"
827 << Endl;
828 }
829 }
830 }
831 // if fAutomatic == true you need a validation sample to optimize pruning
832 if (fAutomatic) {
833 Double_t modulo = 1.0/(fFValidationEvents);
834 Int_t imodulo = static_cast<Int_t>( fmod(modulo,1.0) > 0.5 ? ceil(modulo) : floor(modulo) );
835 if (ievt % imodulo == 0) fValidationSample.push_back( event );
836 else fEventSample.push_back( event );
837 }
838 else {
839 fEventSample.push_back(event);
840 }
841 }
842 }
843
844 if (fAutomatic) {
845 Log() << kINFO << "<InitEventSample> Internally I use " << fEventSample.size()
846 << " for Training and " << fValidationSample.size()
847 << " for Pruning Validation (" << ((Float_t)fValidationSample.size())/((Float_t)fEventSample.size()+fValidationSample.size())*100.0
848 << "% of training used for validation)" << Endl;
849 }
850
851 // some pre-processing for events with negative weights
852 if (fPairNegWeightsGlobal) PreProcessNegativeEventWeights();
853 }
854
855 if (DoRegression()) {
856 // Regression, no reweighting to do
857 } else if (DoMulticlass()) {
858 // Multiclass, only gradboost is supported. No reweighting.
859 } else if (!fSkipNormalization) {
860 // Binary classification.
861 Log() << kDEBUG << "\t<InitEventSample> For classification trees, "<< Endl;
862 Log() << kDEBUG << " \tthe effective number of backgrounds is scaled to match "<<Endl;
863 Log() << kDEBUG << " \tthe signal. Otherwise the first boosting step would do 'just that'!"<<Endl;
864 // it does not make sense in decision trees to start with unequal number of signal/background
865 // events (weights) .. hence normalize them now (happens otherwise in first 'boosting step'
866 // anyway..
867 // Also make sure, that the sum_of_weights == sample.size() .. as this is assumed in
868 // the DecisionTree to derive a sensible number for "fMinSize" (min.#events in node)
869 // that currently is an OR between "weighted" and "unweighted number"
870 // I want:
871 // nS + nB = n
872 // a*SW + b*BW = n
873 // (a*SW)/(b*BW) = fSigToBkgFraction
874 //
875 // ==> b = n/((1+f)BW) and a = (nf/(1+f))/SW
876
877 Double_t nevents = fEventSample.size();
878 Double_t sumSigW=0, sumBkgW=0;
879 Int_t sumSig=0, sumBkg=0;
880 for (UInt_t ievt=0; ievt<fEventSample.size(); ievt++) {
881 if ((DataInfo().IsSignal(fEventSample[ievt])) ) {
882 sumSigW += fEventSample[ievt]->GetWeight();
883 sumSig++;
884 } else {
885 sumBkgW += fEventSample[ievt]->GetWeight();
886 sumBkg++;
887 }
888 }
889 if (sumSigW && sumBkgW){
890 Double_t normSig = nevents/((1+fSigToBkgFraction)*sumSigW)*fSigToBkgFraction;
891 Double_t normBkg = nevents/((1+fSigToBkgFraction)*sumBkgW); ;
892 Log() << kDEBUG << "\tre-normalise events such that Sig and Bkg have respective sum of weights = "
893 << fSigToBkgFraction << Endl;
894 Log() << kDEBUG << " \tsig->sig*"<<normSig << "ev. bkg->bkg*"<<normBkg << "ev." <<Endl;
895 Log() << kHEADER << "#events: (reweighted) sig: "<< sumSigW*normSig << " bkg: " << sumBkgW*normBkg << Endl;
896 Log() << kINFO << "#events: (unweighted) sig: "<< sumSig << " bkg: " << sumBkg << Endl;
897 for (Long64_t ievt=0; ievt<nevents; ievt++) {
898 if ((DataInfo().IsSignal(fEventSample[ievt])) ) fEventSample[ievt]->SetBoostWeight(normSig);
899 else fEventSample[ievt]->SetBoostWeight(normBkg);
900 }
901 }else{
902 Log() << kINFO << "--> could not determine scaling factors as either there are " << Endl;
903 Log() << kINFO << " no signal events (sumSigW="<<sumSigW<<") or no bkg ev. (sumBkgW="<<sumBkgW<<")"<<Endl;
904 }
905
906 }
907
908 fTrainSample = &fEventSample;
909 if (fBaggedBoost){
910 GetBaggedSubSample(fEventSample);
911 fTrainSample = &fSubSample;
912 }
913
914 //just for debug purposes..
915 /*
916 sumSigW=0;
917 sumBkgW=0;
918 for (UInt_t ievt=0; ievt<fEventSample.size(); ievt++) {
919 if ((DataInfo().IsSignal(fEventSample[ievt])) ) sumSigW += fEventSample[ievt]->GetWeight();
920 else sumBkgW += fEventSample[ievt]->GetWeight();
921 }
922 Log() << kWARNING << "sigSumW="<<sumSigW<<"bkgSumW="<<sumBkgW<< Endl;
923 */
924}
925
926////////////////////////////////////////////////////////////////////////////////
927/// O.k. you know there are events with negative event weights. This routine will remove
928/// them by pairing them with the closest event(s) of the same event class with positive
929/// weights
930/// A first attempt is "brute force", I dont' try to be clever using search trees etc,
931/// just quick and dirty to see if the result is any good
932
934 Double_t totalNegWeights = 0;
935 Double_t totalPosWeights = 0;
936 Double_t totalWeights = 0;
937 std::vector<const Event*> negEvents;
938 for (UInt_t iev = 0; iev < fEventSample.size(); iev++){
939 if (fEventSample[iev]->GetWeight() < 0) {
940 totalNegWeights += fEventSample[iev]->GetWeight();
941 negEvents.push_back(fEventSample[iev]);
942 } else {
943 totalPosWeights += fEventSample[iev]->GetWeight();
944 }
945 totalWeights += fEventSample[iev]->GetWeight();
946 }
947 if (totalNegWeights == 0 ) {
948 Log() << kINFO << "no negative event weights found .. no preprocessing necessary" << Endl;
949 return;
950 } else {
951 Log() << kINFO << "found a total of " << totalNegWeights << " of negative event weights which I am going to try to pair with positive events to annihilate them" << Endl;
952 Log() << kINFO << "found a total of " << totalPosWeights << " of events with positive weights" << Endl;
953 Log() << kINFO << "--> total sum of weights = " << totalWeights << " = " << totalNegWeights+totalPosWeights << Endl;
954 }
955
956 std::vector<TMatrixDSym*>* cov = gTools().CalcCovarianceMatrices( fEventSample, 2);
957
958 TMatrixDSym *invCov;
959
960 for (Int_t i=0; i<2; i++){
961 invCov = ((*cov)[i]);
962 if ( TMath::Abs(invCov->Determinant()) < 10E-24 ) {
963 std::cout << "<MethodBDT::PreProcessNeg...> matrix is almost singular with determinant="
964 << TMath::Abs(invCov->Determinant())
965 << " did you use the variables that are linear combinations or highly correlated?"
966 << std::endl;
967 }
968 if ( TMath::Abs(invCov->Determinant()) < 10E-120 ) {
969 std::cout << "<MethodBDT::PreProcessNeg...> matrix is singular with determinant="
970 << TMath::Abs(invCov->Determinant())
971 << " did you use the variables that are linear combinations?"
972 << std::endl;
973 }
974
975 invCov->Invert();
976 }
977
978
979
980 Log() << kINFO << "Found a total of " << totalNegWeights << " in negative weights out of " << fEventSample.size() << " training events " << Endl;
981 Timer timer(negEvents.size(),"Negative Event paired");
982 for (UInt_t nev = 0; nev < negEvents.size(); nev++){
983 timer.DrawProgressBar( nev );
984 Double_t weight = negEvents[nev]->GetWeight();
985 UInt_t iClassID = negEvents[nev]->GetClass();
986 invCov = ((*cov)[iClassID]);
987 while (weight < 0){
988 // find closest event with positive event weight and "pair" it with the negative event
989 // (add their weight) until there is no negative weight anymore
990 Int_t iMin=-1;
991 Double_t dist, minDist=10E270;
992 for (UInt_t iev = 0; iev < fEventSample.size(); iev++){
993 if (iClassID==fEventSample[iev]->GetClass() && fEventSample[iev]->GetWeight() > 0){
994 dist=0;
995 for (UInt_t ivar=0; ivar < GetNvar(); ivar++){
996 for (UInt_t jvar=0; jvar<GetNvar(); jvar++){
997 dist += (negEvents[nev]->GetValue(ivar)-fEventSample[iev]->GetValue(ivar))*
998 (*invCov)[ivar][jvar]*
999 (negEvents[nev]->GetValue(jvar)-fEventSample[iev]->GetValue(jvar));
1000 }
1001 }
1002 if (dist < minDist) { iMin=iev; minDist=dist;}
1003 }
1004 }
1005
1006 if (iMin > -1) {
1007 // std::cout << "Happily pairing .. weight before : " << negEvents[nev]->GetWeight() << " and " << fEventSample[iMin]->GetWeight();
1008 Double_t newWeight = (negEvents[nev]->GetWeight() + fEventSample[iMin]->GetWeight());
1009 if (newWeight > 0){
1010 negEvents[nev]->SetBoostWeight( 0 );
1011 fEventSample[iMin]->SetBoostWeight( newWeight/fEventSample[iMin]->GetOriginalWeight() ); // note the weight*boostweight should be "newWeight"
1012 } else {
1013 negEvents[nev]->SetBoostWeight( newWeight/negEvents[nev]->GetOriginalWeight() ); // note the weight*boostweight should be "newWeight"
1014 fEventSample[iMin]->SetBoostWeight( 0 );
1015 }
1016 // std::cout << " and afterwards " << negEvents[nev]->GetWeight() << " and the paired " << fEventSample[iMin]->GetWeight() << " dist="<<minDist<< std::endl;
1017 } else Log() << kFATAL << "preprocessing didn't find event to pair with the negative weight ... probably a bug" << Endl;
1018 weight = negEvents[nev]->GetWeight();
1019 }
1020 }
1021 Log() << kINFO << "<Negative Event Pairing> took: " << timer.GetElapsedTime()
1022 << " " << Endl;
1023
1024 // just check.. now there should be no negative event weight left anymore
1025 totalNegWeights = 0;
1026 totalPosWeights = 0;
1027 totalWeights = 0;
1028 Double_t sigWeight=0;
1029 Double_t bkgWeight=0;
1030 Int_t nSig=0;
1031 Int_t nBkg=0;
1032
1033 std::vector<const Event*> newEventSample;
1034
1035 for (UInt_t iev = 0; iev < fEventSample.size(); iev++){
1036 if (fEventSample[iev]->GetWeight() < 0) {
1037 totalNegWeights += fEventSample[iev]->GetWeight();
1038 totalWeights += fEventSample[iev]->GetWeight();
1039 } else {
1040 totalPosWeights += fEventSample[iev]->GetWeight();
1041 totalWeights += fEventSample[iev]->GetWeight();
1042 }
1043 if (fEventSample[iev]->GetWeight() > 0) {
1044 newEventSample.push_back(new Event(*fEventSample[iev]));
1045 if (fEventSample[iev]->GetClass() == fSignalClass){
1046 sigWeight += fEventSample[iev]->GetWeight();
1047 nSig+=1;
1048 }else{
1049 bkgWeight += fEventSample[iev]->GetWeight();
1050 nBkg+=1;
1051 }
1052 }
1053 }
1054 if (totalNegWeights < 0) Log() << kFATAL << " compensation of negative event weights with positive ones did not work " << totalNegWeights << Endl;
1055
1056 for (UInt_t i=0; i<fEventSample.size(); i++) delete fEventSample[i];
1057 fEventSample = newEventSample;
1058
1059 Log() << kINFO << " after PreProcessing, the Event sample is left with " << fEventSample.size() << " events (unweighted), all with positive weights, adding up to " << totalWeights << Endl;
1060 Log() << kINFO << " nSig="<<nSig << " sigWeight="<<sigWeight << " nBkg="<<nBkg << " bkgWeight="<<bkgWeight << Endl;
1061
1062
1063}
1064
1065////////////////////////////////////////////////////////////////////////////////
1066/// Call the Optimizer with the set of parameters and ranges that
1067/// are meant to be tuned.
1068
1069std::map<TString,Double_t> TMVA::MethodBDT::OptimizeTuningParameters(TString fomType, TString fitType)
1070{
1071 // fill all the tuning parameters that should be optimized into a map:
1072 std::map<TString,TMVA::Interval*> tuneParameters;
1073 std::map<TString,Double_t> tunedParameters;
1074
1075 // note: the 3rd parameter in the interval is the "number of bins", NOT the stepsize !!
1076 // the actual VALUES at (at least for the scan, guess also in GA) are always
1077 // read from the middle of the bins. Hence.. the choice of Intervals e.g. for the
1078 // MaxDepth, in order to make nice integer values!!!
1079
1080 // find some reasonable ranges for the optimisation of MinNodeEvents:
1081
1082 tuneParameters.insert(std::pair<TString,Interval*>("NTrees", new Interval(10,1000,5))); // stepsize 50
1083 tuneParameters.insert(std::pair<TString,Interval*>("MaxDepth", new Interval(2,4,3))); // stepsize 1
1084 tuneParameters.insert(std::pair<TString,Interval*>("MinNodeSize", new LogInterval(1,30,30))); //
1085 //tuneParameters.insert(std::pair<TString,Interval*>("NodePurityLimit",new Interval(.4,.6,3))); // stepsize .1
1086 //tuneParameters.insert(std::pair<TString,Interval*>("BaggedSampleFraction",new Interval(.4,.9,6))); // stepsize .1
1087
1088 // method-specific parameters
1089 if (fBoostType=="AdaBoost"){
1090 tuneParameters.insert(std::pair<TString,Interval*>("AdaBoostBeta", new Interval(.2,1.,5)));
1091
1092 }else if (fBoostType=="Grad"){
1093 tuneParameters.insert(std::pair<TString,Interval*>("Shrinkage", new Interval(0.05,0.50,5)));
1094
1095 }else if (fBoostType=="Bagging" && fRandomisedTrees){
1096 Int_t min_var = TMath::FloorNint( GetNvar() * .25 );
1097 Int_t max_var = TMath::CeilNint( GetNvar() * .75 );
1098 tuneParameters.insert(std::pair<TString,Interval*>("UseNvars", new Interval(min_var,max_var,4)));
1099
1100 }
1101
1102 Log()<<kINFO << " the following BDT parameters will be tuned on the respective *grid*\n"<<Endl;
1103 std::map<TString,TMVA::Interval*>::iterator it;
1104 for(it=tuneParameters.begin(); it!= tuneParameters.end(); ++it){
1105 Log() << kWARNING << it->first << Endl;
1106 std::ostringstream oss;
1107 (it->second)->Print(oss);
1108 Log()<<oss.str();
1109 Log()<<Endl;
1110 }
1111
1112 OptimizeConfigParameters optimize(this, tuneParameters, fomType, fitType);
1113 tunedParameters=optimize.optimize();
1114
1115 return tunedParameters;
1116
1117}
1118
1119////////////////////////////////////////////////////////////////////////////////
1120/// Set the tuning parameters according to the argument.
1121
1122void TMVA::MethodBDT::SetTuneParameters(std::map<TString,Double_t> tuneParameters)
1123{
1124 std::map<TString,Double_t>::iterator it;
1125 for(it=tuneParameters.begin(); it!= tuneParameters.end(); ++it){
1126 Log() << kWARNING << it->first << " = " << it->second << Endl;
1127 if (it->first == "MaxDepth" ) SetMaxDepth ((Int_t)it->second);
1128 else if (it->first == "MinNodeSize" ) SetMinNodeSize (it->second);
1129 else if (it->first == "NTrees" ) SetNTrees ((Int_t)it->second);
1130 else if (it->first == "NodePurityLimit") SetNodePurityLimit (it->second);
1131 else if (it->first == "AdaBoostBeta" ) SetAdaBoostBeta (it->second);
1132 else if (it->first == "Shrinkage" ) SetShrinkage (it->second);
1133 else if (it->first == "UseNvars" ) SetUseNvars ((Int_t)it->second);
1134 else if (it->first == "BaggedSampleFraction" ) SetBaggedSampleFraction (it->second);
1135 else Log() << kFATAL << " SetParameter for " << it->first << " not yet implemented " <<Endl;
1136 }
1137}
1138
1139////////////////////////////////////////////////////////////////////////////////
1140/// BDT training.
1141
1142
1144{
1146
1147 // fill the STL Vector with the event sample
1148 // (needs to be done here and cannot be done in "init" as the options need to be
1149 // known).
1150 InitEventSample();
1151
1152 if (fNTrees==0){
1153 Log() << kERROR << " Zero Decision Trees demanded... that does not work !! "
1154 << " I set it to 1 .. just so that the program does not crash"
1155 << Endl;
1156 fNTrees = 1;
1157 }
1158
1159 if (fInteractive && fInteractive->NotInitialized()){
1160 std::vector<TString> titles = {"Boost weight", "Error Fraction"};
1161 fInteractive->Init(titles);
1162 }
1163 fIPyMaxIter = fNTrees;
1164 fExitFromTraining = false;
1165
1166 // HHV (it's been here since looong but I really don't know why we cannot handle
1167 // normalized variables in BDTs... todo
1168 if (IsNormalised()) Log() << kFATAL << "\"Normalise\" option cannot be used with BDT; "
1169 << "please remove the option from the configuration string, or "
1170 << "use \"!Normalise\""
1171 << Endl;
1172
1173 if(DoRegression())
1174 Log() << kINFO << "Regression Loss Function: "<< fRegressionLossFunctionBDTG->Name() << Endl;
1175
1176 Log() << kINFO << "Training "<< fNTrees << " Decision Trees ... patience please" << Endl;
1177
1178 Log() << kDEBUG << "Training with maximal depth = " <<fMaxDepth
1179 << ", MinNodeEvents=" << fMinNodeEvents
1180 << ", NTrees="<<fNTrees
1181 << ", NodePurityLimit="<<fNodePurityLimit
1182 << ", AdaBoostBeta="<<fAdaBoostBeta
1183 << Endl;
1184
1185 // weights applied in boosting
1186 Int_t nBins;
1187 Double_t xMin,xMax;
1188 TString hname = "AdaBooost weight distribution";
1189
1190 nBins= 100;
1191 xMin = 0;
1192 xMax = 30;
1193
1194 if (DoRegression()) {
1195 nBins= 100;
1196 xMin = 0;
1197 xMax = 1;
1198 hname="Boost event weights distribution";
1199 }
1200
1201 // book monitoring histograms (for AdaBost only)
1202
1203 TH1* h = new TH1F(Form("%s_BoostWeight",DataInfo().GetName()),hname,nBins,xMin,xMax);
1204 TH1* nodesBeforePruningVsTree = new TH1I(Form("%s_NodesBeforePruning",DataInfo().GetName()),"nodes before pruning",fNTrees,0,fNTrees);
1205 TH1* nodesAfterPruningVsTree = new TH1I(Form("%s_NodesAfterPruning",DataInfo().GetName()),"nodes after pruning",fNTrees,0,fNTrees);
1206
1207
1208
1209 if(!DoMulticlass()){
1210 Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, GetAnalysisType());
1211
1212 h->SetXTitle("boost weight");
1213 results->Store(h, "BoostWeights");
1214
1215
1216 // Monitor the performance (on TEST sample) versus number of trees
1217 if (fDoBoostMonitor){
1218 TH2* boostMonitor = new TH2F("BoostMonitor","ROC Integral Vs iTree",2,0,fNTrees,2,0,1.05);
1219 boostMonitor->SetXTitle("#tree");
1220 boostMonitor->SetYTitle("ROC Integral");
1221 results->Store(boostMonitor, "BoostMonitor");
1222 TGraph *boostMonitorGraph = new TGraph();
1223 boostMonitorGraph->SetName("BoostMonitorGraph");
1224 boostMonitorGraph->SetTitle("ROCIntegralVsNTrees");
1225 results->Store(boostMonitorGraph, "BoostMonitorGraph");
1226 }
1227
1228 // weights applied in boosting vs tree number
1229 h = new TH1F("BoostWeightVsTree","Boost weights vs tree",fNTrees,0,fNTrees);
1230 h->SetXTitle("#tree");
1231 h->SetYTitle("boost weight");
1232 results->Store(h, "BoostWeightsVsTree");
1233
1234 // error fraction vs tree number
1235 h = new TH1F("ErrFractHist","error fraction vs tree number",fNTrees,0,fNTrees);
1236 h->SetXTitle("#tree");
1237 h->SetYTitle("error fraction");
1238 results->Store(h, "ErrorFrac");
1239
1240 // nNodesBeforePruning vs tree number
1241 nodesBeforePruningVsTree->SetXTitle("#tree");
1242 nodesBeforePruningVsTree->SetYTitle("#tree nodes");
1243 results->Store(nodesBeforePruningVsTree);
1244
1245 // nNodesAfterPruning vs tree number
1246 nodesAfterPruningVsTree->SetXTitle("#tree");
1247 nodesAfterPruningVsTree->SetYTitle("#tree nodes");
1248 results->Store(nodesAfterPruningVsTree);
1249
1250 }
1251
1252 fMonitorNtuple= new TTree("MonitorNtuple","BDT variables");
1253 fMonitorNtuple->Branch("iTree",&fITree,"iTree/I");
1254 fMonitorNtuple->Branch("boostWeight",&fBoostWeight,"boostWeight/D");
1255 fMonitorNtuple->Branch("errorFraction",&fErrorFraction,"errorFraction/D");
1256
1257 Timer timer( fNTrees, GetName() );
1258 Int_t nNodesBeforePruningCount = 0;
1259 Int_t nNodesAfterPruningCount = 0;
1260
1261 Int_t nNodesBeforePruning = 0;
1262 Int_t nNodesAfterPruning = 0;
1263
1264 if(fBoostType=="Grad"){
1265 InitGradBoost(fEventSample);
1266 }
1267
1268 Int_t itree=0;
1269 Bool_t continueBoost=kTRUE;
1270 //for (int itree=0; itree<fNTrees; itree++) {
1271
1272 while (itree < fNTrees && continueBoost){
1273 if (fExitFromTraining) break;
1274 fIPyCurrentIter = itree;
1275 timer.DrawProgressBar( itree );
1276 // Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, GetAnalysisType());
1277 // TH1 *hxx = new TH1F(Form("swdist%d",itree),Form("swdist%d",itree),10000,0,15);
1278 // results->Store(hxx,Form("swdist%d",itree));
1279 // TH1 *hxy = new TH1F(Form("bwdist%d",itree),Form("bwdist%d",itree),10000,0,15);
1280 // results->Store(hxy,Form("bwdist%d",itree));
1281 // for (Int_t iev=0; iev<fEventSample.size(); iev++) {
1282 // if (fEventSample[iev]->GetClass()!=0) hxy->Fill((fEventSample[iev])->GetWeight());
1283 // else hxx->Fill((fEventSample[iev])->GetWeight());
1284 // }
1285
1286 if(DoMulticlass()){
1287 if (fBoostType!="Grad"){
1288 Log() << kFATAL << "Multiclass is currently only supported by gradient boost. "
1289 << "Please change boost option accordingly (BoostType=Grad)." << Endl;
1290 }
1291
1292 UInt_t nClasses = DataInfo().GetNClasses();
1293 for (UInt_t i=0;i<nClasses;i++){
1294 // Careful: If fSepType is nullptr, the tree will be considered a regression tree and
1295 // use the correct output for gradboost (response rather than yesnoleaf) in checkEvent.
1296 // See TMVA::MethodBDT::InitGradBoost.
1297 fForest.push_back( new DecisionTree( fSepType, fMinNodeSize, fNCuts, &(DataInfo()), i,
1298 fRandomisedTrees, fUseNvars, fUsePoissonNvars, fMaxDepth,
1299 itree*nClasses+i, fNodePurityLimit, itree*nClasses+1));
1300 fForest.back()->SetNVars(GetNvar());
1301 if (fUseFisherCuts) {
1302 fForest.back()->SetUseFisherCuts();
1303 fForest.back()->SetMinLinCorrForFisher(fMinLinCorrForFisher);
1304 fForest.back()->SetUseExclusiveVars(fUseExclusiveVars);
1305 }
1306 // the minimum linear correlation between two variables demanded for use in fisher criterion in node splitting
1307
1308 nNodesBeforePruning = fForest.back()->BuildTree(*fTrainSample);
1309 Double_t bw = this->Boost(*fTrainSample, fForest.back(),i);
1310 if (bw > 0) {
1311 fBoostWeights.push_back(bw);
1312 }else{
1313 fBoostWeights.push_back(0);
1314 Log() << kWARNING << "stopped boosting at itree="<<itree << Endl;
1315 // fNTrees = itree+1; // that should stop the boosting
1316 continueBoost=kFALSE;
1317 }
1318 }
1319 }
1320 else{
1321
1322 DecisionTree* dt = new DecisionTree( fSepType, fMinNodeSize, fNCuts, &(DataInfo()), fSignalClass,
1323 fRandomisedTrees, fUseNvars, fUsePoissonNvars, fMaxDepth,
1324 itree, fNodePurityLimit, itree);
1325
1326 fForest.push_back(dt);
1327 fForest.back()->SetNVars(GetNvar());
1328 if (fUseFisherCuts) {
1329 fForest.back()->SetUseFisherCuts();
1330 fForest.back()->SetMinLinCorrForFisher(fMinLinCorrForFisher);
1331 fForest.back()->SetUseExclusiveVars(fUseExclusiveVars);
1332 }
1333
1334 nNodesBeforePruning = fForest.back()->BuildTree(*fTrainSample);
1335
1336 if (fUseYesNoLeaf && !DoRegression() && fBoostType!="Grad") { // remove leaf nodes where both daughter nodes are of same type
1337 nNodesBeforePruning = fForest.back()->CleanTree();
1338 }
1339
1340 nNodesBeforePruningCount += nNodesBeforePruning;
1341 nodesBeforePruningVsTree->SetBinContent(itree+1,nNodesBeforePruning);
1342
1343 fForest.back()->SetPruneMethod(fPruneMethod); // set the pruning method for the tree
1344 fForest.back()->SetPruneStrength(fPruneStrength); // set the strength parameter
1345
1346 std::vector<const Event*> * validationSample = NULL;
1347 if(fAutomatic) validationSample = &fValidationSample;
1348 Double_t bw = this->Boost(*fTrainSample, fForest.back());
1349 if (bw > 0) {
1350 fBoostWeights.push_back(bw);
1351 }else{
1352 fBoostWeights.push_back(0);
1353 Log() << kWARNING << "stopped boosting at itree="<<itree << Endl;
1354 continueBoost=kFALSE;
1355 }
1356
1357 // if fAutomatic == true, pruneStrength will be the optimal pruning strength
1358 // determined by the pruning algorithm; otherwise, it is simply the strength parameter
1359 // set by the user
1360 if (fPruneMethod != DecisionTree::kNoPruning) fForest.back()->PruneTree(validationSample);
1361
1362 if (fUseYesNoLeaf && !DoRegression() && fBoostType!="Grad"){ // remove leaf nodes where both daughter nodes are of same type
1363 fForest.back()->CleanTree();
1364 }
1365 nNodesAfterPruning = fForest.back()->GetNNodes();
1366 nNodesAfterPruningCount += nNodesAfterPruning;
1367 nodesAfterPruningVsTree->SetBinContent(itree+1,nNodesAfterPruning);
1368
1369 if (fInteractive){
1370 fInteractive->AddPoint(itree, fBoostWeight, fErrorFraction);
1371 }
1372 fITree = itree;
1373 fMonitorNtuple->Fill();
1374 if (fDoBoostMonitor){
1375 if (! DoRegression() ){
1376 if ( itree==fNTrees-1 || (!(itree%500)) ||
1377 (!(itree%250) && itree <1000)||
1378 (!(itree%100) && itree < 500)||
1379 (!(itree%50) && itree < 250)||
1380 (!(itree%25) && itree < 150)||
1381 (!(itree%10) && itree < 50)||
1382 (!(itree%5) && itree < 20)
1383 ) BoostMonitor(itree);
1384 }
1385 }
1386 }
1387 itree++;
1388 }
1389
1390 // get elapsed time
1391 Log() << kDEBUG << "\t<Train> elapsed time: " << timer.GetElapsedTime()
1392 << " " << Endl;
1393 if (fPruneMethod == DecisionTree::kNoPruning) {
1394 Log() << kDEBUG << "\t<Train> average number of nodes (w/o pruning) : "
1395 << nNodesBeforePruningCount/GetNTrees() << Endl;
1396 }
1397 else {
1398 Log() << kDEBUG << "\t<Train> average number of nodes before/after pruning : "
1399 << nNodesBeforePruningCount/GetNTrees() << " / "
1400 << nNodesAfterPruningCount/GetNTrees()
1401 << Endl;
1402 }
1404
1405
1406 // reset all previously stored/accumulated BOOST weights in the event sample
1407 // for (UInt_t iev=0; iev<fEventSample.size(); iev++) fEventSample[iev]->SetBoostWeight(1.);
1408 Log() << kDEBUG << "Now I delete the privat data sample"<< Endl;
1409 for (UInt_t i=0; i<fEventSample.size(); i++) delete fEventSample[i];
1410 for (UInt_t i=0; i<fValidationSample.size(); i++) delete fValidationSample[i];
1411 fEventSample.clear();
1412 fValidationSample.clear();
1413
1414 if (!fExitFromTraining) fIPyMaxIter = fIPyCurrentIter;
1415 ExitFromTraining();
1416}
1417
1418
1419////////////////////////////////////////////////////////////////////////////////
1420/// Returns MVA value: -1 for background, 1 for signal.
1421
1423{
1424 Double_t sum=0;
1425 for (UInt_t itree=0; itree<nTrees; itree++) {
1426 //loop over all trees in forest
1427 sum += fForest[itree]->CheckEvent(e,kFALSE);
1428
1429 }
1430 return 2.0/(1.0+exp(-2.0*sum))-1; //MVA output between -1 and 1
1431}
1432
1433////////////////////////////////////////////////////////////////////////////////
1434/// Calculate residual for all events.
1435
1436void TMVA::MethodBDT::UpdateTargets(std::vector<const TMVA::Event*>& eventSample, UInt_t cls)
1437{
1438 if (DoMulticlass()) {
1439 UInt_t nClasses = DataInfo().GetNClasses();
1440 Bool_t isLastClass = (cls == nClasses - 1);
1441
1442 #ifdef R__USE_IMT
1443 //
1444 // This is the multi-threaded multiclass version
1445 //
1446 // Note: we only need to update the predicted probabilities every
1447 // `nClasses` tree. Let's call a set of `nClasses` trees a "round". Thus
1448 // the algortihm is split in two parts `update_residuals` and
1449 // `update_residuals_last` where the latter is inteded to be run instead
1450 // of the former for the last tree in a "round".
1451 //
1452 std::map<const TMVA::Event *, std::vector<double>> & residuals = this->fResiduals;
1453 DecisionTree & lastTree = *(this->fForest.back());
1454
1455 auto update_residuals = [&residuals, &lastTree, cls](const TMVA::Event * e) {
1456 residuals[e].at(cls) += lastTree.CheckEvent(e, kFALSE);
1457 };
1458
1459 auto update_residuals_last = [&residuals, &lastTree, cls, nClasses](const TMVA::Event * e) {
1460 residuals[e].at(cls) += lastTree.CheckEvent(e, kFALSE);
1461
1462 auto &residualsThisEvent = residuals[e];
1463
1464 std::vector<Double_t> expCache(nClasses, 0.0);
1465 std::transform(residualsThisEvent.begin(),
1466 residualsThisEvent.begin() + nClasses,
1467 expCache.begin(), [](Double_t d) { return exp(d); });
1468
1469 Double_t exp_sum = std::accumulate(expCache.begin(),
1470 expCache.begin() + nClasses,
1471 0.0);
1472
1473 for (UInt_t i = 0; i < nClasses; i++) {
1474 Double_t p_cls = expCache[i] / exp_sum;
1475
1476 Double_t res = (e->GetClass() == i) ? (1.0 - p_cls) : (-p_cls);
1477 const_cast<TMVA::Event *>(e)->SetTarget(i, res);
1478 }
1479 };
1480
1481 if (isLastClass) {
1483 .Foreach(update_residuals_last, eventSample);
1484 } else {
1486 .Foreach(update_residuals, eventSample);
1487 }
1488 #else
1489 //
1490 // Single-threaded multiclass version
1491 //
1492 std::vector<Double_t> expCache;
1493 if (isLastClass) {
1494 expCache.resize(nClasses);
1495 }
1496
1497 for (auto e : eventSample) {
1498 fResiduals[e].at(cls) += fForest.back()->CheckEvent(e, kFALSE);
1499 if (isLastClass) {
1500 auto &residualsThisEvent = fResiduals[e];
1501 std::transform(residualsThisEvent.begin(),
1502 residualsThisEvent.begin() + nClasses,
1503 expCache.begin(), [](Double_t d) { return exp(d); });
1504
1505 Double_t exp_sum = std::accumulate(expCache.begin(),
1506 expCache.begin() + nClasses,
1507 0.0);
1508
1509 for (UInt_t i = 0; i < nClasses; i++) {
1510 Double_t p_cls = expCache[i] / exp_sum;
1511
1512 Double_t res = (e->GetClass() == i) ? (1.0 - p_cls) : (-p_cls);
1513 const_cast<TMVA::Event *>(e)->SetTarget(i, res);
1514 }
1515 }
1516 }
1517 #endif
1518 } else {
1519 std::map<const TMVA::Event *, std::vector<double>> & residuals = this->fResiduals;
1520 DecisionTree & lastTree = *(this->fForest.back());
1521
1522 UInt_t signalClass = DataInfo().GetSignalClassIndex();
1523
1524 #ifdef R__USE_IMT
1525 auto update_residuals = [&residuals, &lastTree, signalClass](const TMVA::Event * e) {
1526 double & residualAt0 = residuals[e].at(0);
1527 residualAt0 += lastTree.CheckEvent(e, kFALSE);
1528
1529 Double_t p_sig = 1.0 / (1.0 + exp(-2.0 * residualAt0));
1530 Double_t res = ((e->GetClass() == signalClass) ? (1.0 - p_sig) : (-p_sig));
1531
1532 const_cast<TMVA::Event *>(e)->SetTarget(0, res);
1533 };
1534
1536 .Foreach(update_residuals, eventSample);
1537 #else
1538 for (auto e : eventSample) {
1539 double & residualAt0 = residuals[e].at(0);
1540 residualAt0 += lastTree.CheckEvent(e, kFALSE);
1541
1542 Double_t p_sig = 1.0 / (1.0 + exp(-2.0 * residualAt0));
1543 Double_t res = ((e->GetClass() == signalClass) ? (1.0 - p_sig) : (-p_sig));
1544
1545 const_cast<TMVA::Event *>(e)->SetTarget(0, res);
1546 }
1547 #endif
1548 }
1549}
1550
1551////////////////////////////////////////////////////////////////////////////////
1552/// \brief Calculate residuals for all events and update targets for next iter.
1553///
1554/// \param[in] eventSample The collection of events currently under training.
1555/// \param[in] first Should be true when called before the first boosting
1556/// iteration has been run
1557///
1558void TMVA::MethodBDT::UpdateTargetsRegression(std::vector<const TMVA::Event*>& eventSample, Bool_t first)
1559{
1560 if (!first) {
1561#ifdef R__USE_IMT
1563 auto seeds = ROOT::TSeqU(nPartitions);
1564
1565 // need a lambda function to pass to TThreadExecutor::MapReduce
1566 auto f = [this, &nPartitions](UInt_t partition = 0) -> Int_t {
1567 Int_t start = 1.0 * partition / nPartitions * this->fEventSample.size();
1568 Int_t end = (partition + 1.0) / nPartitions * this->fEventSample.size();
1569
1570 for (Int_t i = start; i < end; ++i) {
1571 const TMVA::Event *e = fEventSample[i];
1572 LossFunctionEventInfo & lossInfo = fLossFunctionEventInfo.at(e);
1573 lossInfo.predictedValue += fForest.back()->CheckEvent(e, kFALSE);
1574 }
1575
1576 return 0;
1577 };
1578
1580#else
1581 for (const TMVA::Event *e : fEventSample) {
1582 LossFunctionEventInfo & lossInfo = fLossFunctionEventInfo.at(e);
1583 lossInfo.predictedValue += fForest.back()->CheckEvent(e, kFALSE);
1584 }
1585#endif
1586 }
1587
1588 // NOTE: Set targets are also parallelised internally
1589 fRegressionLossFunctionBDTG->SetTargets(eventSample, fLossFunctionEventInfo);
1590
1591}
1592
1593////////////////////////////////////////////////////////////////////////////////
1594/// Calculate the desired response value for each region.
1595
1596Double_t TMVA::MethodBDT::GradBoost(std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt, UInt_t cls)
1597{
1598 struct LeafInfo {
1599 Double_t sumWeightTarget = 0;
1600 Double_t sum2 = 0;
1601 };
1602
1603 std::unordered_map<TMVA::DecisionTreeNode*, LeafInfo> leaves;
1604 for (auto e : eventSample) {
1605 Double_t weight = e->GetWeight();
1607 auto &v = leaves[node];
1608 auto target = e->GetTarget(cls);
1609 v.sumWeightTarget += target * weight;
1610 v.sum2 += fabs(target) * (1.0 - fabs(target)) * weight;
1611 }
1612 for (auto &iLeave : leaves) {
1613 constexpr auto minValue = 1e-30;
1614 if (iLeave.second.sum2 < minValue) {
1615 iLeave.second.sum2 = minValue;
1616 }
1617 const Double_t K = DataInfo().GetNClasses();
1618 iLeave.first->SetResponse(fShrinkage * (K - 1) / K * iLeave.second.sumWeightTarget / iLeave.second.sum2);
1619 }
1620
1621 //call UpdateTargets before next tree is grown
1622
1623 DoMulticlass() ? UpdateTargets(fEventSample, cls) : UpdateTargets(fEventSample);
1624 return 1; //trees all have the same weight
1625}
1626
1627////////////////////////////////////////////////////////////////////////////////
1628/// Implementation of M_TreeBoost using any loss function as described by Friedman 1999.
1629
1630Double_t TMVA::MethodBDT::GradBoostRegression(std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
1631{
1632 // get the vector of events for each terminal so that we can calculate the constant fit value in each
1633 // terminal node
1634 // #### Not sure how many events are in each node in advance, so I can't parallelize this easily
1635 std::map<TMVA::DecisionTreeNode*,vector< TMVA::LossFunctionEventInfo > > leaves;
1636 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1637 TMVA::DecisionTreeNode* node = dt->GetEventNode(*(*e));
1638 (leaves[node]).push_back(fLossFunctionEventInfo[*e]);
1639 }
1640
1641 // calculate the constant fit for each terminal node based upon the events in the node
1642 // node (iLeave->first), vector of event information (iLeave->second)
1643 // #### could parallelize this and do the leaves at the same time, but this doesn't take very long compared
1644 // #### to the other processes
1645 for (std::map<TMVA::DecisionTreeNode*,vector< TMVA::LossFunctionEventInfo > >::iterator iLeave=leaves.begin();
1646 iLeave!=leaves.end();++iLeave){
1647 Double_t fit = fRegressionLossFunctionBDTG->Fit(iLeave->second);
1648 (iLeave->first)->SetResponse(fShrinkage*fit);
1649 }
1650
1651 UpdateTargetsRegression(*fTrainSample);
1652
1653 return 1;
1654}
1655
1656////////////////////////////////////////////////////////////////////////////////
1657/// Initialize targets for first tree.
1658
1659void TMVA::MethodBDT::InitGradBoost( std::vector<const TMVA::Event*>& eventSample)
1660{
1661 // Should get rid of this line. It's just for debugging.
1662 //std::sort(eventSample.begin(), eventSample.end(), [](const TMVA::Event* a, const TMVA::Event* b){
1663 // return (a->GetTarget(0) < b->GetTarget(0)); });
1664 fSepType=NULL; //set fSepType to NULL (regression trees are used for both classification an regression)
1665 if(DoRegression()){
1666 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1667 fLossFunctionEventInfo[*e]= TMVA::LossFunctionEventInfo((*e)->GetTarget(0), 0, (*e)->GetWeight());
1668 }
1669
1670 fRegressionLossFunctionBDTG->Init(fLossFunctionEventInfo, fBoostWeights);
1671 UpdateTargetsRegression(*fTrainSample,kTRUE);
1672
1673 return;
1674 }
1675 else if(DoMulticlass()){
1676 UInt_t nClasses = DataInfo().GetNClasses();
1677 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1678 for (UInt_t i=0;i<nClasses;i++){
1679 //Calculate initial residua, assuming equal probability for all classes
1680 Double_t r = (*e)->GetClass()==i?(1-1.0/nClasses):(-1.0/nClasses);
1681 const_cast<TMVA::Event*>(*e)->SetTarget(i,r);
1682 fResiduals[*e].push_back(0);
1683 }
1684 }
1685 }
1686 else{
1687 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1688 Double_t r = (DataInfo().IsSignal(*e)?1:0)-0.5; //Calculate initial residua
1689 const_cast<TMVA::Event*>(*e)->SetTarget(0,r);
1690 fResiduals[*e].push_back(0);
1691 }
1692 }
1693
1694}
1695////////////////////////////////////////////////////////////////////////////////
1696/// Test the tree quality.. in terms of Misclassification.
1697
1699{
1700 Double_t ncorrect=0, nfalse=0;
1701 for (UInt_t ievt=0; ievt<fValidationSample.size(); ievt++) {
1702 Bool_t isSignalType= (dt->CheckEvent(fValidationSample[ievt]) > fNodePurityLimit ) ? 1 : 0;
1703
1704 if (isSignalType == (DataInfo().IsSignal(fValidationSample[ievt])) ) {
1705 ncorrect += fValidationSample[ievt]->GetWeight();
1706 }
1707 else{
1708 nfalse += fValidationSample[ievt]->GetWeight();
1709 }
1710 }
1711
1712 return ncorrect / (ncorrect + nfalse);
1713}
1714
1715////////////////////////////////////////////////////////////////////////////////
1716/// Apply the boosting algorithm (the algorithm is selecte via the the "option" given
1717/// in the constructor. The return value is the boosting weight.
1718
1719Double_t TMVA::MethodBDT::Boost( std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt, UInt_t cls )
1720{
1721 Double_t returnVal=-1;
1722
1723 if (fBoostType=="AdaBoost") returnVal = this->AdaBoost (eventSample, dt);
1724 else if (fBoostType=="AdaCost") returnVal = this->AdaCost (eventSample, dt);
1725 else if (fBoostType=="Bagging") returnVal = this->Bagging ( );
1726 else if (fBoostType=="RegBoost") returnVal = this->RegBoost (eventSample, dt);
1727 else if (fBoostType=="AdaBoostR2") returnVal = this->AdaBoostR2(eventSample, dt);
1728 else if (fBoostType=="Grad"){
1729 if(DoRegression())
1730 returnVal = this->GradBoostRegression(eventSample, dt);
1731 else if(DoMulticlass())
1732 returnVal = this->GradBoost (eventSample, dt, cls);
1733 else
1734 returnVal = this->GradBoost (eventSample, dt);
1735 }
1736 else {
1737 Log() << kINFO << GetOptions() << Endl;
1738 Log() << kFATAL << "<Boost> unknown boost option " << fBoostType<< " called" << Endl;
1739 }
1740
1741 if (fBaggedBoost){
1742 GetBaggedSubSample(fEventSample);
1743 }
1744
1745
1746 return returnVal;
1747}
1748
1749////////////////////////////////////////////////////////////////////////////////
1750/// Fills the ROCIntegral vs Itree from the testSample for the monitoring plots
1751/// during the training .. but using the testing events
1752
1754{
1755 Results* results = Data()->GetResults(GetMethodName(),Types::kTraining, Types::kMaxAnalysisType);
1756
1757 TH1F *tmpS = new TH1F( "tmpS", "", 100 , -1., 1.00001 );
1758 TH1F *tmpB = new TH1F( "tmpB", "", 100 , -1., 1.00001 );
1759 TH1F *tmp;
1760
1761
1762 UInt_t signalClassNr = DataInfo().GetClassInfo("Signal")->GetNumber();
1763
1764 // const std::vector<Event*> events=Data()->GetEventCollection(Types::kTesting);
1765 // // fMethod->GetTransformationHandler().CalcTransformations(fMethod->Data()->GetEventCollection(Types::kTesting));
1766 // for (UInt_t iev=0; iev < events.size() ; iev++){
1767 // if (events[iev]->GetClass() == signalClassNr) tmp=tmpS;
1768 // else tmp=tmpB;
1769 // tmp->Fill(PrivateGetMvaValue(*(events[iev])),events[iev]->GetWeight());
1770 // }
1771
1772 UInt_t nevents = Data()->GetNTestEvents();
1773 for (UInt_t iev=0; iev < nevents; iev++){
1774 const Event* event = GetTestingEvent(iev);
1775
1776 if (event->GetClass() == signalClassNr) {tmp=tmpS;}
1777 else {tmp=tmpB;}
1778 tmp->Fill(PrivateGetMvaValue(event),event->GetWeight());
1779 }
1780 Double_t max=1;
1781
1782 std::vector<TH1F*> hS;
1783 std::vector<TH1F*> hB;
1784 for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
1785 hS.push_back(new TH1F(Form("SigVar%dAtTree%d",ivar,iTree),Form("SigVar%dAtTree%d",ivar,iTree),100,DataInfo().GetVariableInfo(ivar).GetMin(),DataInfo().GetVariableInfo(ivar).GetMax()));
1786 hB.push_back(new TH1F(Form("BkgVar%dAtTree%d",ivar,iTree),Form("BkgVar%dAtTree%d",ivar,iTree),100,DataInfo().GetVariableInfo(ivar).GetMin(),DataInfo().GetVariableInfo(ivar).GetMax()));
1787 results->Store(hS.back(),hS.back()->GetTitle());
1788 results->Store(hB.back(),hB.back()->GetTitle());
1789 }
1790
1791
1792 for (UInt_t iev=0; iev < fEventSample.size(); iev++){
1793 if (fEventSample[iev]->GetBoostWeight() > max) max = 1.01*fEventSample[iev]->GetBoostWeight();
1794 }
1795 TH1F *tmpBoostWeightsS = new TH1F(Form("BoostWeightsInTreeS%d",iTree),Form("BoostWeightsInTreeS%d",iTree),100,0.,max);
1796 TH1F *tmpBoostWeightsB = new TH1F(Form("BoostWeightsInTreeB%d",iTree),Form("BoostWeightsInTreeB%d",iTree),100,0.,max);
1797 results->Store(tmpBoostWeightsS,tmpBoostWeightsS->GetTitle());
1798 results->Store(tmpBoostWeightsB,tmpBoostWeightsB->GetTitle());
1799
1800 TH1F *tmpBoostWeights;
1801 std::vector<TH1F*> *h;
1802
1803 for (UInt_t iev=0; iev < fEventSample.size(); iev++){
1804 if (fEventSample[iev]->GetClass() == signalClassNr) {
1805 tmpBoostWeights=tmpBoostWeightsS;
1806 h=&hS;
1807 }else{
1808 tmpBoostWeights=tmpBoostWeightsB;
1809 h=&hB;
1810 }
1811 tmpBoostWeights->Fill(fEventSample[iev]->GetBoostWeight());
1812 for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
1813 (*h)[ivar]->Fill(fEventSample[iev]->GetValue(ivar),fEventSample[iev]->GetWeight());
1814 }
1815 }
1816
1817
1818 TMVA::PDF *sig = new TMVA::PDF( " PDF Sig", tmpS, TMVA::PDF::kSpline3 );
1819 TMVA::PDF *bkg = new TMVA::PDF( " PDF Bkg", tmpB, TMVA::PDF::kSpline3 );
1820
1821
1822 TGraph* gr=results->GetGraph("BoostMonitorGraph");
1823 Int_t nPoints = gr->GetN();
1824 gr->Set(nPoints+1);
1825 gr->SetPoint(nPoints,(Double_t)iTree+1,GetROCIntegral(sig,bkg));
1826
1827 tmpS->Delete();
1828 tmpB->Delete();
1829
1830 delete sig;
1831 delete bkg;
1832
1833 return;
1834}
1835
1836////////////////////////////////////////////////////////////////////////////////
1837/// The AdaBoost implementation.
1838/// a new training sample is generated by weighting
1839/// events that are misclassified by the decision tree. The weight
1840/// applied is \f$ w = \frac{(1-err)}{err} \f$ or more general:
1841/// \f$ w = (\frac{(1-err)}{err})^\beta \f$
1842/// where \f$err\f$ is the fraction of misclassified events in the tree ( <0.5 assuming
1843/// demanding the that previous selection was better than random guessing)
1844/// and "beta" being a free parameter (standard: beta = 1) that modifies the
1845/// boosting.
1846
1847Double_t TMVA::MethodBDT::AdaBoost( std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
1848{
1849 Double_t err=0, sumGlobalw=0, sumGlobalwfalse=0, sumGlobalwfalse2=0;
1850
1851 std::vector<Double_t> sumw(DataInfo().GetNClasses(),0); //for individually re-scaling each class
1852
1853 Double_t maxDev=0;
1854 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1855 Double_t w = (*e)->GetWeight();
1856 sumGlobalw += w;
1857 UInt_t iclass=(*e)->GetClass();
1858 sumw[iclass] += w;
1859
1860 if ( DoRegression() ) {
1861 Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
1862 sumGlobalwfalse += w * tmpDev;
1863 sumGlobalwfalse2 += w * tmpDev*tmpDev;
1864 if (tmpDev > maxDev) maxDev = tmpDev;
1865 }else{
1866
1867 if (fUseYesNoLeaf){
1868 Bool_t isSignalType = (dt->CheckEvent(*e,fUseYesNoLeaf) > fNodePurityLimit );
1869 if (!(isSignalType == DataInfo().IsSignal(*e))) {
1870 sumGlobalwfalse+= w;
1871 }
1872 }else{
1873 Double_t dtoutput = (dt->CheckEvent(*e,fUseYesNoLeaf) - 0.5)*2.;
1874 Int_t trueType;
1875 if (DataInfo().IsSignal(*e)) trueType = 1;
1876 else trueType = -1;
1877 sumGlobalwfalse+= w*trueType*dtoutput;
1878 }
1879 }
1880 }
1881
1882 err = sumGlobalwfalse/sumGlobalw ;
1883 if ( DoRegression() ) {
1884 //if quadratic loss:
1885 if (fAdaBoostR2Loss=="linear"){
1886 err = sumGlobalwfalse/maxDev/sumGlobalw ;
1887 }
1888 else if (fAdaBoostR2Loss=="quadratic"){
1889 err = sumGlobalwfalse2/maxDev/maxDev/sumGlobalw ;
1890 }
1891 else if (fAdaBoostR2Loss=="exponential"){
1892 err = 0;
1893 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1894 Double_t w = (*e)->GetWeight();
1895 Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
1896 err += w * (1 - exp (-tmpDev/maxDev)) / sumGlobalw;
1897 }
1898
1899 }
1900 else {
1901 Log() << kFATAL << " you've chosen a Loss type for Adaboost other than linear, quadratic or exponential "
1902 << " namely " << fAdaBoostR2Loss << "\n"
1903 << "and this is not implemented... a typo in the options ??" <<Endl;
1904 }
1905 }
1906
1907 Log() << kDEBUG << "BDT AdaBoos wrong/all: " << sumGlobalwfalse << "/" << sumGlobalw << Endl;
1908
1909
1910 Double_t newSumGlobalw=0;
1911 std::vector<Double_t> newSumw(sumw.size(),0);
1912
1913 Double_t boostWeight=1.;
1914 if (err >= 0.5 && fUseYesNoLeaf) { // sanity check ... should never happen as otherwise there is apparently
1915 // something odd with the assignment of the leaf nodes (rem: you use the training
1916 // events for this determination of the error rate)
1917 if (dt->GetNNodes() == 1){
1918 Log() << kERROR << " YOUR tree has only 1 Node... kind of a funny *tree*. I cannot "
1919 << "boost such a thing... if after 1 step the error rate is == 0.5"
1920 << Endl
1921 << "please check why this happens, maybe too many events per node requested ?"
1922 << Endl;
1923
1924 }else{
1925 Log() << kERROR << " The error rate in the BDT boosting is > 0.5. ("<< err
1926 << ") That should not happen, please check your code (i.e... the BDT code), I "
1927 << " stop boosting here" << Endl;
1928 return -1;
1929 }
1930 err = 0.5;
1931 } else if (err < 0) {
1932 Log() << kERROR << " The error rate in the BDT boosting is < 0. That can happen"
1933 << " due to improper treatment of negative weights in a Monte Carlo.. (if you have"
1934 << " an idea on how to do it in a better way, please let me know (Helge.Voss@cern.ch)"
1935 << " for the time being I set it to its absolute value.. just to continue.." << Endl;
1936 err = TMath::Abs(err);
1937 }
1938 if (fUseYesNoLeaf)
1939 boostWeight = TMath::Log((1.-err)/err)*fAdaBoostBeta;
1940 else
1941 boostWeight = TMath::Log((1.+err)/(1-err))*fAdaBoostBeta;
1942
1943
1944 Log() << kDEBUG << "BDT AdaBoos wrong/all: " << sumGlobalwfalse << "/" << sumGlobalw << " 1-err/err="<<boostWeight<< " log.."<<TMath::Log(boostWeight)<<Endl;
1945
1946 Results* results = Data()->GetResults(GetMethodName(),Types::kTraining, Types::kMaxAnalysisType);
1947
1948
1949 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1950
1951 if (fUseYesNoLeaf||DoRegression()){
1952 if ((!( (dt->CheckEvent(*e,fUseYesNoLeaf) > fNodePurityLimit ) == DataInfo().IsSignal(*e))) || DoRegression()) {
1953 Double_t boostfactor = TMath::Exp(boostWeight);
1954
1955 if (DoRegression()) boostfactor = TMath::Power(1/boostWeight,(1.-TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) )/maxDev ) );
1956 if ( (*e)->GetWeight() > 0 ){
1957 (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1958 // Helge change back (*e)->ScaleBoostWeight(boostfactor);
1959 if (DoRegression()) results->GetHist("BoostWeights")->Fill(boostfactor);
1960 } else {
1961 if ( fInverseBoostNegWeights )(*e)->ScaleBoostWeight( 1. / boostfactor); // if the original event weight is negative, and you want to "increase" the events "positive" influence, you'd rather make the event weight "smaller" in terms of it's absolute value while still keeping it something "negative"
1962 else (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1963
1964 }
1965 }
1966
1967 }else{
1968 Double_t dtoutput = (dt->CheckEvent(*e,fUseYesNoLeaf) - 0.5)*2.;
1969 Int_t trueType;
1970 if (DataInfo().IsSignal(*e)) trueType = 1;
1971 else trueType = -1;
1972 Double_t boostfactor = TMath::Exp(-1*boostWeight*trueType*dtoutput);
1973
1974 if ( (*e)->GetWeight() > 0 ){
1975 (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1976 // Helge change back (*e)->ScaleBoostWeight(boostfactor);
1977 if (DoRegression()) results->GetHist("BoostWeights")->Fill(boostfactor);
1978 } else {
1979 if ( fInverseBoostNegWeights )(*e)->ScaleBoostWeight( 1. / boostfactor); // if the original event weight is negative, and you want to "increase" the events "positive" influence, you'd rather make the event weight "smaller" in terms of it's absolute value while still keeping it something "negative"
1980 else (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1981 }
1982 }
1983 newSumGlobalw+=(*e)->GetWeight();
1984 newSumw[(*e)->GetClass()] += (*e)->GetWeight();
1985 }
1986
1987
1988 // Double_t globalNormWeight=sumGlobalw/newSumGlobalw;
1989 Double_t globalNormWeight=( (Double_t) eventSample.size())/newSumGlobalw;
1990 Log() << kDEBUG << "new Nsig="<<newSumw[0]*globalNormWeight << " new Nbkg="<<newSumw[1]*globalNormWeight << Endl;
1991
1992
1993 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1994 // if (fRenormByClass) (*e)->ScaleBoostWeight( normWeightByClass[(*e)->GetClass()] );
1995 // else (*e)->ScaleBoostWeight( globalNormWeight );
1996 // else (*e)->ScaleBoostWeight( globalNormWeight );
1997 if (DataInfo().IsSignal(*e))(*e)->ScaleBoostWeight( globalNormWeight * fSigToBkgFraction );
1998 else (*e)->ScaleBoostWeight( globalNormWeight );
1999 }
2000
2001 if (!(DoRegression()))results->GetHist("BoostWeights")->Fill(boostWeight);
2002 results->GetHist("BoostWeightsVsTree")->SetBinContent(fForest.size(),boostWeight);
2003 results->GetHist("ErrorFrac")->SetBinContent(fForest.size(),err);
2004
2005 fBoostWeight = boostWeight;
2006 fErrorFraction = err;
2007
2008 return boostWeight;
2009}
2010
2011////////////////////////////////////////////////////////////////////////////////
2012/// The AdaCost boosting algorithm takes a simple cost Matrix (currently fixed for
2013/// all events... later could be modified to use individual cost matrices for each
2014/// events as in the original paper...
2015///
2016/// true_signal true_bkg
2017/// ----------------------------------
2018/// sel_signal | Css Ctb_ss Cxx.. in the range [0,1]
2019/// sel_bkg | Cts_sb Cbb
2020///
2021/// and takes this into account when calculating the mis class. cost (former: error fraction):
2022///
2023/// err = sum_events ( weight* y_true*y_sel * beta(event)
2024
2025Double_t TMVA::MethodBDT::AdaCost( vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
2026{
2027 Double_t Css = fCss;
2028 Double_t Cbb = fCbb;
2029 Double_t Cts_sb = fCts_sb;
2030 Double_t Ctb_ss = fCtb_ss;
2031
2032 Double_t err=0, sumGlobalWeights=0, sumGlobalCost=0;
2033
2034 std::vector<Double_t> sumw(DataInfo().GetNClasses(),0); //for individually re-scaling each class
2035
2036 for (vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2037 Double_t w = (*e)->GetWeight();
2038 sumGlobalWeights += w;
2039 UInt_t iclass=(*e)->GetClass();
2040
2041 sumw[iclass] += w;
2042
2043 if ( DoRegression() ) {
2044 Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2045 }else{
2046
2047 Double_t dtoutput = (dt->CheckEvent(*e,false) - 0.5)*2.;
2048 Int_t trueType;
2049 Bool_t isTrueSignal = DataInfo().IsSignal(*e);
2050 Bool_t isSelectedSignal = (dtoutput>0);
2051 if (isTrueSignal) trueType = 1;
2052 else trueType = -1;
2053
2054 Double_t cost=0;
2055 if (isTrueSignal && isSelectedSignal) cost=Css;
2056 else if (isTrueSignal && !isSelectedSignal) cost=Cts_sb;
2057 else if (!isTrueSignal && isSelectedSignal) cost=Ctb_ss;
2058 else if (!isTrueSignal && !isSelectedSignal) cost=Cbb;
2059 else Log() << kERROR << "something went wrong in AdaCost" << Endl;
2060
2061 sumGlobalCost+= w*trueType*dtoutput*cost;
2062
2063 }
2064 }
2065
2066 if ( DoRegression() ) {
2067 Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2068 }
2069
2070 // Log() << kDEBUG << "BDT AdaBoos wrong/all: " << sumGlobalCost << "/" << sumGlobalWeights << Endl;
2071 // Log() << kWARNING << "BDT AdaBoos wrong/all: " << sumGlobalCost << "/" << sumGlobalWeights << Endl;
2072 sumGlobalCost /= sumGlobalWeights;
2073 // Log() << kWARNING << "BDT AdaBoos wrong/all: " << sumGlobalCost << "/" << sumGlobalWeights << Endl;
2074
2075
2076 Double_t newSumGlobalWeights=0;
2077 vector<Double_t> newSumClassWeights(sumw.size(),0);
2078
2079 Double_t boostWeight = TMath::Log((1+sumGlobalCost)/(1-sumGlobalCost)) * fAdaBoostBeta;
2080
2081 Results* results = Data()->GetResults(GetMethodName(),Types::kTraining, Types::kMaxAnalysisType);
2082
2083 for (vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2084 Double_t dtoutput = (dt->CheckEvent(*e,false) - 0.5)*2.;
2085 Int_t trueType;
2086 Bool_t isTrueSignal = DataInfo().IsSignal(*e);
2087 Bool_t isSelectedSignal = (dtoutput>0);
2088 if (isTrueSignal) trueType = 1;
2089 else trueType = -1;
2090
2091 Double_t cost=0;
2092 if (isTrueSignal && isSelectedSignal) cost=Css;
2093 else if (isTrueSignal && !isSelectedSignal) cost=Cts_sb;
2094 else if (!isTrueSignal && isSelectedSignal) cost=Ctb_ss;
2095 else if (!isTrueSignal && !isSelectedSignal) cost=Cbb;
2096 else Log() << kERROR << "something went wrong in AdaCost" << Endl;
2097
2098 Double_t boostfactor = TMath::Exp(-1*boostWeight*trueType*dtoutput*cost);
2099 if (DoRegression())Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2100 if ( (*e)->GetWeight() > 0 ){
2101 (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
2102 // Helge change back (*e)->ScaleBoostWeight(boostfactor);
2103 if (DoRegression())Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2104 } else {
2105 if ( fInverseBoostNegWeights )(*e)->ScaleBoostWeight( 1. / boostfactor); // if the original event weight is negative, and you want to "increase" the events "positive" influence, you'd rather make the event weight "smaller" in terms of it's absolute value while still keeping it something "negative"
2106 }
2107
2108 newSumGlobalWeights+=(*e)->GetWeight();
2109 newSumClassWeights[(*e)->GetClass()] += (*e)->GetWeight();
2110 }
2111
2112
2113 // Double_t globalNormWeight=sumGlobalWeights/newSumGlobalWeights;
2114 Double_t globalNormWeight=Double_t(eventSample.size())/newSumGlobalWeights;
2115 Log() << kDEBUG << "new Nsig="<<newSumClassWeights[0]*globalNormWeight << " new Nbkg="<<newSumClassWeights[1]*globalNormWeight << Endl;
2116
2117
2118 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2119 // if (fRenormByClass) (*e)->ScaleBoostWeight( normWeightByClass[(*e)->GetClass()] );
2120 // else (*e)->ScaleBoostWeight( globalNormWeight );
2121 if (DataInfo().IsSignal(*e))(*e)->ScaleBoostWeight( globalNormWeight * fSigToBkgFraction );
2122 else (*e)->ScaleBoostWeight( globalNormWeight );
2123 }
2124
2125
2126 if (!(DoRegression()))results->GetHist("BoostWeights")->Fill(boostWeight);
2127 results->GetHist("BoostWeightsVsTree")->SetBinContent(fForest.size(),boostWeight);
2128 results->GetHist("ErrorFrac")->SetBinContent(fForest.size(),err);
2129
2130 fBoostWeight = boostWeight;
2131 fErrorFraction = err;
2132
2133
2134 return boostWeight;
2135}
2136
2137////////////////////////////////////////////////////////////////////////////////
2138/// Call it boot-strapping, re-sampling or whatever you like, in the end it is nothing
2139/// else but applying "random" poisson weights to each event.
2140
2142{
2143 // this is now done in "MethodBDT::Boost as it might be used by other boost methods, too
2144 // GetBaggedSample(eventSample);
2145
2146 return 1.; //here as there are random weights for each event, just return a constant==1;
2147}
2148
2149////////////////////////////////////////////////////////////////////////////////
2150/// Fills fEventSample with fBaggedSampleFraction*NEvents random training events.
2151
2152void TMVA::MethodBDT::GetBaggedSubSample(std::vector<const TMVA::Event*>& eventSample)
2153{
2154
2155 Double_t n;
2156 TRandom3 *trandom = new TRandom3(100*fForest.size()+1234);
2157
2158 if (!fSubSample.empty()) fSubSample.clear();
2159
2160 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2161 n = trandom->PoissonD(fBaggedSampleFraction);
2162 for (Int_t i=0;i<n;i++) fSubSample.push_back(*e);
2163 }
2164
2165 delete trandom;
2166 return;
2167
2168 /*
2169 UInt_t nevents = fEventSample.size();
2170
2171 if (!fSubSample.empty()) fSubSample.clear();
2172 TRandom3 *trandom = new TRandom3(fForest.size()+1);
2173
2174 for (UInt_t ievt=0; ievt<nevents; ievt++) { // recreate new random subsample
2175 if(trandom->Rndm()<fBaggedSampleFraction)
2176 fSubSample.push_back(fEventSample[ievt]);
2177 }
2178 delete trandom;
2179 */
2180
2181}
2182
2183////////////////////////////////////////////////////////////////////////////////
2184/// A special boosting only for Regression (not implemented).
2185
2186Double_t TMVA::MethodBDT::RegBoost( std::vector<const TMVA::Event*>& /* eventSample */, DecisionTree* /* dt */ )
2187{
2188 return 1;
2189}
2190
2191////////////////////////////////////////////////////////////////////////////////
2192/// Adaption of the AdaBoost to regression problems (see H.Drucker 1997).
2193
2194Double_t TMVA::MethodBDT::AdaBoostR2( std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
2195{
2196 if ( !DoRegression() ) Log() << kFATAL << "Somehow you chose a regression boost method for a classification job" << Endl;
2197
2198 Double_t err=0, sumw=0, sumwfalse=0, sumwfalse2=0;
2199 Double_t maxDev=0;
2200 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2201 Double_t w = (*e)->GetWeight();
2202 sumw += w;
2203
2204 Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
2205 sumwfalse += w * tmpDev;
2206 sumwfalse2 += w * tmpDev*tmpDev;
2207 if (tmpDev > maxDev) maxDev = tmpDev;
2208 }
2209
2210 //if quadratic loss:
2211 if (fAdaBoostR2Loss=="linear"){
2212 err = sumwfalse/maxDev/sumw ;
2213 }
2214 else if (fAdaBoostR2Loss=="quadratic"){
2215 err = sumwfalse2/maxDev/maxDev/sumw ;
2216 }
2217 else if (fAdaBoostR2Loss=="exponential"){
2218 err = 0;
2219 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2220 Double_t w = (*e)->GetWeight();
2221 Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
2222 err += w * (1 - exp (-tmpDev/maxDev)) / sumw;
2223 }
2224
2225 }
2226 else {
2227 Log() << kFATAL << " you've chosen a Loss type for Adaboost other than linear, quadratic or exponential "
2228 << " namely " << fAdaBoostR2Loss << "\n"
2229 << "and this is not implemented... a typo in the options ??" <<Endl;
2230 }
2231
2232
2233 if (err >= 0.5) { // sanity check ... should never happen as otherwise there is apparently
2234 // something odd with the assignment of the leaf nodes (rem: you use the training
2235 // events for this determination of the error rate)
2236 if (dt->GetNNodes() == 1){
2237 Log() << kERROR << " YOUR tree has only 1 Node... kind of a funny *tree*. I cannot "
2238 << "boost such a thing... if after 1 step the error rate is == 0.5"
2239 << Endl
2240 << "please check why this happens, maybe too many events per node requested ?"
2241 << Endl;
2242
2243 }else{
2244 Log() << kERROR << " The error rate in the BDT boosting is > 0.5. ("<< err
2245 << ") That should not happen, but is possible for regression trees, and"
2246 << " should trigger a stop for the boosting. please check your code (i.e... the BDT code), I "
2247 << " stop boosting " << Endl;
2248 return -1;
2249 }
2250 err = 0.5;
2251 } else if (err < 0) {
2252 Log() << kERROR << " The error rate in the BDT boosting is < 0. That can happen"
2253 << " due to improper treatment of negative weights in a Monte Carlo.. (if you have"
2254 << " an idea on how to do it in a better way, please let me know (Helge.Voss@cern.ch)"
2255 << " for the time being I set it to its absolute value.. just to continue.." << Endl;
2256 err = TMath::Abs(err);
2257 }
2258
2259 Double_t boostWeight = err / (1.-err);
2260 Double_t newSumw=0;
2261
2262 Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, Types::kMaxAnalysisType);
2263
2264 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2265 Double_t boostfactor = TMath::Power(boostWeight,(1.-TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) )/maxDev ) );
2266 results->GetHist("BoostWeights")->Fill(boostfactor);
2267 // std::cout << "R2 " << boostfactor << " " << boostWeight << " " << (1.-TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) )/maxDev) << std::endl;
2268 if ( (*e)->GetWeight() > 0 ){
2269 Float_t newBoostWeight = (*e)->GetBoostWeight() * boostfactor;
2270 Float_t newWeight = (*e)->GetWeight() * (*e)->GetBoostWeight() * boostfactor;
2271 if (newWeight == 0) {
2272 Log() << kINFO << "Weight= " << (*e)->GetWeight() << Endl;
2273 Log() << kINFO << "BoostWeight= " << (*e)->GetBoostWeight() << Endl;
2274 Log() << kINFO << "boostweight="<<boostWeight << " err= " <<err << Endl;
2275 Log() << kINFO << "NewBoostWeight= " << newBoostWeight << Endl;
2276 Log() << kINFO << "boostfactor= " << boostfactor << Endl;
2277 Log() << kINFO << "maxDev = " << maxDev << Endl;
2278 Log() << kINFO << "tmpDev = " << TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) ) << Endl;
2279 Log() << kINFO << "target = " << (*e)->GetTarget(0) << Endl;
2280 Log() << kINFO << "estimate = " << dt->CheckEvent(*e,kFALSE) << Endl;
2281 }
2282 (*e)->SetBoostWeight( newBoostWeight );
2283 // (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
2284 } else {
2285 (*e)->SetBoostWeight( (*e)->GetBoostWeight() / boostfactor);
2286 }
2287 newSumw+=(*e)->GetWeight();
2288 }
2289
2290 // re-normalise the weights
2291 Double_t normWeight = sumw / newSumw;
2292 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2293 //Helge (*e)->ScaleBoostWeight( sumw/newSumw);
2294 // (*e)->ScaleBoostWeight( normWeight);
2295 (*e)->SetBoostWeight( (*e)->GetBoostWeight() * normWeight );
2296 }
2297
2298
2299 results->GetHist("BoostWeightsVsTree")->SetBinContent(fForest.size(),1./boostWeight);
2300 results->GetHist("ErrorFrac")->SetBinContent(fForest.size(),err);
2301
2302 fBoostWeight = boostWeight;
2303 fErrorFraction = err;
2304
2305 return TMath::Log(1./boostWeight);
2306}
2307
2308////////////////////////////////////////////////////////////////////////////////
2309/// Write weights to XML.
2310
2311void TMVA::MethodBDT::AddWeightsXMLTo( void* parent ) const
2312{
2313 void* wght = gTools().AddChild(parent, "Weights");
2314
2315 if (fDoPreselection){
2316 for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
2317 gTools().AddAttr( wght, Form("PreselectionLowBkgVar%d",ivar), fIsLowBkgCut[ivar]);
2318 gTools().AddAttr( wght, Form("PreselectionLowBkgVar%dValue",ivar), fLowBkgCut[ivar]);
2319 gTools().AddAttr( wght, Form("PreselectionLowSigVar%d",ivar), fIsLowSigCut[ivar]);
2320 gTools().AddAttr( wght, Form("PreselectionLowSigVar%dValue",ivar), fLowSigCut[ivar]);
2321 gTools().AddAttr( wght, Form("PreselectionHighBkgVar%d",ivar), fIsHighBkgCut[ivar]);
2322 gTools().AddAttr( wght, Form("PreselectionHighBkgVar%dValue",ivar),fHighBkgCut[ivar]);
2323 gTools().AddAttr( wght, Form("PreselectionHighSigVar%d",ivar), fIsHighSigCut[ivar]);
2324 gTools().AddAttr( wght, Form("PreselectionHighSigVar%dValue",ivar),fHighSigCut[ivar]);
2325 }
2326 }
2327
2328
2329 gTools().AddAttr( wght, "NTrees", fForest.size() );
2330 gTools().AddAttr( wght, "AnalysisType", fForest.back()->GetAnalysisType() );
2331
2332 for (UInt_t i=0; i< fForest.size(); i++) {
2333 void* trxml = fForest[i]->AddXMLTo(wght);
2334 gTools().AddAttr( trxml, "boostWeight", fBoostWeights[i] );
2335 gTools().AddAttr( trxml, "itree", i );
2336 }
2337}
2338
2339////////////////////////////////////////////////////////////////////////////////
2340/// Reads the BDT from the xml file.
2341
2343 UInt_t i;
2344 for (i=0; i<fForest.size(); i++) delete fForest[i];
2345 fForest.clear();
2346 fBoostWeights.clear();
2347
2348 UInt_t ntrees;
2349 UInt_t analysisType;
2350 Float_t boostWeight;
2351
2352
2353 if (gTools().HasAttr( parent, Form("PreselectionLowBkgVar%d",0))) {
2354 fIsLowBkgCut.resize(GetNvar());
2355 fLowBkgCut.resize(GetNvar());
2356 fIsLowSigCut.resize(GetNvar());
2357 fLowSigCut.resize(GetNvar());
2358 fIsHighBkgCut.resize(GetNvar());
2359 fHighBkgCut.resize(GetNvar());
2360 fIsHighSigCut.resize(GetNvar());
2361 fHighSigCut.resize(GetNvar());
2362
2363 Bool_t tmpBool;
2364 Double_t tmpDouble;
2365 for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
2366 gTools().ReadAttr( parent, Form("PreselectionLowBkgVar%d",ivar), tmpBool);
2367 fIsLowBkgCut[ivar]=tmpBool;
2368 gTools().ReadAttr( parent, Form("PreselectionLowBkgVar%dValue",ivar), tmpDouble);
2369 fLowBkgCut[ivar]=tmpDouble;
2370 gTools().ReadAttr( parent, Form("PreselectionLowSigVar%d",ivar), tmpBool);
2371 fIsLowSigCut[ivar]=tmpBool;
2372 gTools().ReadAttr( parent, Form("PreselectionLowSigVar%dValue",ivar), tmpDouble);
2373 fLowSigCut[ivar]=tmpDouble;
2374 gTools().ReadAttr( parent, Form("PreselectionHighBkgVar%d",ivar), tmpBool);
2375 fIsHighBkgCut[ivar]=tmpBool;
2376 gTools().ReadAttr( parent, Form("PreselectionHighBkgVar%dValue",ivar), tmpDouble);
2377 fHighBkgCut[ivar]=tmpDouble;
2378 gTools().ReadAttr( parent, Form("PreselectionHighSigVar%d",ivar),tmpBool);
2379 fIsHighSigCut[ivar]=tmpBool;
2380 gTools().ReadAttr( parent, Form("PreselectionHighSigVar%dValue",ivar), tmpDouble);
2381 fHighSigCut[ivar]=tmpDouble;
2382 }
2383 }
2384
2385
2386 gTools().ReadAttr( parent, "NTrees", ntrees );
2387
2388 if(gTools().HasAttr(parent, "TreeType")) { // pre 4.1.0 version
2389 gTools().ReadAttr( parent, "TreeType", analysisType );
2390 } else { // from 4.1.0 onwards
2391 gTools().ReadAttr( parent, "AnalysisType", analysisType );
2392 }
2393
2394 void* ch = gTools().GetChild(parent);
2395 i=0;
2396 while(ch) {
2397 fForest.push_back( dynamic_cast<DecisionTree*>( DecisionTree::CreateFromXML(ch, GetTrainingTMVAVersionCode()) ) );
2398 fForest.back()->SetAnalysisType(Types::EAnalysisType(analysisType));
2399 fForest.back()->SetTreeID(i++);
2400 gTools().ReadAttr(ch,"boostWeight",boostWeight);
2401 fBoostWeights.push_back(boostWeight);
2402 ch = gTools().GetNextChild(ch);
2403 }
2404}
2405
2406////////////////////////////////////////////////////////////////////////////////
2407/// Read the weights (BDT coefficients).
2408
2410{
2411 TString dummy;
2412 // Types::EAnalysisType analysisType;
2413 Int_t analysisType(0);
2414
2415 // coverity[tainted_data_argument]
2416 istr >> dummy >> fNTrees;
2417 Log() << kINFO << "Read " << fNTrees << " Decision trees" << Endl;
2418
2419 for (UInt_t i=0;i<fForest.size();i++) delete fForest[i];
2420 fForest.clear();
2421 fBoostWeights.clear();
2422 Int_t iTree;
2423 Double_t boostWeight;
2424 for (int i=0;i<fNTrees;i++) {
2425 istr >> dummy >> iTree >> dummy >> boostWeight;
2426 if (iTree != i) {
2427 fForest.back()->Print( std::cout );
2428 Log() << kFATAL << "Error while reading weight file; mismatch iTree="
2429 << iTree << " i=" << i
2430 << " dummy " << dummy
2431 << " boostweight " << boostWeight
2432 << Endl;
2433 }
2434 fForest.push_back( new DecisionTree() );
2435 fForest.back()->SetAnalysisType(Types::EAnalysisType(analysisType));
2436 fForest.back()->SetTreeID(i);
2437 fForest.back()->Read(istr, GetTrainingTMVAVersionCode());
2438 fBoostWeights.push_back(boostWeight);
2439 }
2440}
2441
2442////////////////////////////////////////////////////////////////////////////////
2443
2445 return this->GetMvaValue( err, errUpper, 0 );
2446}
2447
2448////////////////////////////////////////////////////////////////////////////////
2449/// Return the MVA value (range [-1;1]) that classifies the
2450/// event according to the majority vote from the total number of
2451/// decision trees.
2452
2454{
2455 const Event* ev = GetEvent();
2456 if (fDoPreselection) {
2457 Double_t val = ApplyPreselectionCuts(ev);
2458 if (TMath::Abs(val)>0.05) return val;
2459 }
2460 return PrivateGetMvaValue(ev, err, errUpper, useNTrees);
2461
2462}
2463
2464////////////////////////////////////////////////////////////////////////////////
2465/// Return the MVA value (range [-1;1]) that classifies the
2466/// event according to the majority vote from the total number of
2467/// decision trees.
2468
2470{
2471 // cannot determine error
2472 NoErrorCalc(err, errUpper);
2473
2474 // allow for the possibility to use less trees in the actual MVA calculation
2475 // than have been originally trained.
2476 UInt_t nTrees = fForest.size();
2477
2478 if (useNTrees > 0 ) nTrees = useNTrees;
2479
2480 if (fBoostType=="Grad") return GetGradBoostMVA(ev,nTrees);
2481
2482 Double_t myMVA = 0;
2483 Double_t norm = 0;
2484 for (UInt_t itree=0; itree<nTrees; itree++) {
2485 //
2486 myMVA += fBoostWeights[itree] * fForest[itree]->CheckEvent(ev,fUseYesNoLeaf);
2487 norm += fBoostWeights[itree];
2488 }
2489 return ( norm > std::numeric_limits<double>::epsilon() ) ? myMVA /= norm : 0 ;
2490}
2491
2492
2493////////////////////////////////////////////////////////////////////////////////
2494/// Get the multiclass MVA response for the BDT classifier.
2495
2496const std::vector<Float_t>& TMVA::MethodBDT::GetMulticlassValues()
2497{
2498 const TMVA::Event *e = GetEvent();
2499 if (fMulticlassReturnVal == NULL) fMulticlassReturnVal = new std::vector<Float_t>();
2500 fMulticlassReturnVal->clear();
2501
2502 UInt_t nClasses = DataInfo().GetNClasses();
2503 std::vector<Double_t> temp(nClasses);
2504 auto forestSize = fForest.size();
2505
2506 #ifdef R__USE_IMT
2507 std::vector<TMVA::DecisionTree *> forest = fForest;
2508 auto get_output = [&e, &forest, &temp, forestSize, nClasses](UInt_t iClass) {
2509 for (UInt_t itree = iClass; itree < forestSize; itree += nClasses) {
2510 temp[iClass] += forest[itree]->CheckEvent(e, kFALSE);
2511 }
2512 };
2513
2515 .Foreach(get_output, ROOT::TSeqU(nClasses));
2516 #else
2517 // trees 0, nClasses, 2*nClasses, ... belong to class 0
2518 // trees 1, nClasses+1, 2*nClasses+1, ... belong to class 1 and so forth
2519 UInt_t classOfTree = 0;
2520 for (UInt_t itree = 0; itree < forestSize; ++itree) {
2521 temp[classOfTree] += fForest[itree]->CheckEvent(e, kFALSE);
2522 if (++classOfTree == nClasses) classOfTree = 0; // cheap modulo
2523 }
2524 #endif
2525
2526 // we want to calculate sum of exp(temp[j] - temp[i]) for all i,j (i!=j)
2527 // first calculate exp(), then replace minus with division.
2528 std::transform(temp.begin(), temp.end(), temp.begin(), [](Double_t d){return exp(d);});
2529
2530 Double_t exp_sum = std::accumulate(temp.begin(), temp.end(), 0.0);
2531
2532 for (UInt_t i = 0; i < nClasses; i++) {
2533 Double_t p_cls = temp[i] / exp_sum;
2534 (*fMulticlassReturnVal).push_back(p_cls);
2535 }
2536
2537 return *fMulticlassReturnVal;
2538}
2539
2540////////////////////////////////////////////////////////////////////////////////
2541/// Get the regression value generated by the BDTs.
2542
2543const std::vector<Float_t> & TMVA::MethodBDT::GetRegressionValues()
2544{
2545
2546 if (fRegressionReturnVal == NULL) fRegressionReturnVal = new std::vector<Float_t>();
2547 fRegressionReturnVal->clear();
2548
2549 const Event * ev = GetEvent();
2550 Event * evT = new Event(*ev);
2551
2552 Double_t myMVA = 0;
2553 Double_t norm = 0;
2554 if (fBoostType=="AdaBoostR2") {
2555 // rather than using the weighted average of the tree respones in the forest
2556 // H.Decker(1997) proposed to use the "weighted median"
2557
2558 // sort all individual tree responses according to the prediction value
2559 // (keep the association to their tree weight)
2560 // the sum up all the associated weights (starting from the one whose tree
2561 // yielded the smalles response) up to the tree "t" at which you've
2562 // added enough tree weights to have more than half of the sum of all tree weights.
2563 // choose as response of the forest that one which belongs to this "t"
2564
2565 vector< Double_t > response(fForest.size());
2566 vector< Double_t > weight(fForest.size());
2567 Double_t totalSumOfWeights = 0;
2568
2569 for (UInt_t itree=0; itree<fForest.size(); itree++) {
2570 response[itree] = fForest[itree]->CheckEvent(ev,kFALSE);
2571 weight[itree] = fBoostWeights[itree];
2572 totalSumOfWeights += fBoostWeights[itree];
2573 }
2574
2575 std::vector< std::vector<Double_t> > vtemp;
2576 vtemp.push_back( response ); // this is the vector that will get sorted
2577 vtemp.push_back( weight );
2578 gTools().UsefulSortAscending( vtemp );
2579
2580 Int_t t=0;
2581 Double_t sumOfWeights = 0;
2582 while (sumOfWeights <= totalSumOfWeights/2.) {
2583 sumOfWeights += vtemp[1][t];
2584 t++;
2585 }
2586
2587 Double_t rVal=0;
2588 Int_t count=0;
2589 for (UInt_t i= TMath::Max(UInt_t(0),UInt_t(t-(fForest.size()/6)-0.5));
2590 i< TMath::Min(UInt_t(fForest.size()),UInt_t(t+(fForest.size()/6)+0.5)); i++) {
2591 count++;
2592 rVal+=vtemp[0][i];
2593 }
2594 // fRegressionReturnVal->push_back( rVal/Double_t(count));
2595 evT->SetTarget(0, rVal/Double_t(count) );
2596 }
2597 else if(fBoostType=="Grad"){
2598 for (UInt_t itree=0; itree<fForest.size(); itree++) {
2599 myMVA += fForest[itree]->CheckEvent(ev,kFALSE);
2600 }
2601 // fRegressionReturnVal->push_back( myMVA+fBoostWeights[0]);
2602 evT->SetTarget(0, myMVA+fBoostWeights[0] );
2603 }
2604 else{
2605 for (UInt_t itree=0; itree<fForest.size(); itree++) {
2606 //
2607 myMVA += fBoostWeights[itree] * fForest[itree]->CheckEvent(ev,kFALSE);
2608 norm += fBoostWeights[itree];
2609 }
2610 // fRegressionReturnVal->push_back( ( norm > std::numeric_limits<double>::epsilon() ) ? myMVA /= norm : 0 );
2611 evT->SetTarget(0, ( norm > std::numeric_limits<double>::epsilon() ) ? myMVA /= norm : 0 );
2612 }
2613
2614
2615
2616 const Event* evT2 = GetTransformationHandler().InverseTransform( evT );
2617 fRegressionReturnVal->push_back( evT2->GetTarget(0) );
2618
2619 delete evT;
2620
2621
2622 return *fRegressionReturnVal;
2623}
2624
2625////////////////////////////////////////////////////////////////////////////////
2626/// Here we could write some histograms created during the processing
2627/// to the output file.
2628
2630{
2631 Log() << kDEBUG << "\tWrite monitoring histograms to file: " << BaseDir()->GetPath() << Endl;
2632
2633 //Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, Types::kMaxAnalysisType);
2634 //results->GetStorage()->Write();
2635 fMonitorNtuple->Write();
2636}
2637
2638////////////////////////////////////////////////////////////////////////////////
2639/// Return the relative variable importance, normalized to all
2640/// variables together having the importance 1. The importance in
2641/// evaluated as the total separation-gain that this variable had in
2642/// the decision trees (weighted by the number of events)
2643
2645{
2646 fVariableImportance.resize(GetNvar());
2647 for (UInt_t ivar = 0; ivar < GetNvar(); ivar++) {
2648 fVariableImportance[ivar]=0;
2649 }
2650 Double_t sum=0;
2651 for (UInt_t itree = 0; itree < GetNTrees(); itree++) {
2652 std::vector<Double_t> relativeImportance(fForest[itree]->GetVariableImportance());
2653 for (UInt_t i=0; i< relativeImportance.size(); i++) {
2654 fVariableImportance[i] += fBoostWeights[itree] * relativeImportance[i];
2655 }
2656 }
2657
2658 for (UInt_t ivar=0; ivar< fVariableImportance.size(); ivar++){
2659 fVariableImportance[ivar] = TMath::Sqrt(fVariableImportance[ivar]);
2660 sum += fVariableImportance[ivar];
2661 }
2662 for (UInt_t ivar=0; ivar< fVariableImportance.size(); ivar++) fVariableImportance[ivar] /= sum;
2663
2664 return fVariableImportance;
2665}
2666
2667////////////////////////////////////////////////////////////////////////////////
2668/// Returns the measure for the variable importance of variable "ivar"
2669/// which is later used in GetVariableImportance() to calculate the
2670/// relative variable importances.
2671
2673{
2674 std::vector<Double_t> relativeImportance = this->GetVariableImportance();
2675 if (ivar < (UInt_t)relativeImportance.size()) return relativeImportance[ivar];
2676 else Log() << kFATAL << "<GetVariableImportance> ivar = " << ivar << " is out of range " << Endl;
2677
2678 return -1;
2679}
2680
2681////////////////////////////////////////////////////////////////////////////////
2682/// Compute ranking of input variables
2683
2685{
2686 // create the ranking object
2687 fRanking = new Ranking( GetName(), "Variable Importance" );
2688 vector< Double_t> importance(this->GetVariableImportance());
2689
2690 for (UInt_t ivar=0; ivar<GetNvar(); ivar++) {
2691
2692 fRanking->AddRank( Rank( GetInputLabel(ivar), importance[ivar] ) );
2693 }
2694
2695 return fRanking;
2696}
2697
2698////////////////////////////////////////////////////////////////////////////////
2699/// Get help message text.
2700
2702{
2703 Log() << Endl;
2704 Log() << gTools().Color("bold") << "--- Short description:" << gTools().Color("reset") << Endl;
2705 Log() << Endl;
2706 Log() << "Boosted Decision Trees are a collection of individual decision" << Endl;
2707 Log() << "trees which form a multivariate classifier by (weighted) majority " << Endl;
2708 Log() << "vote of the individual trees. Consecutive decision trees are " << Endl;
2709 Log() << "trained using the original training data set with re-weighted " << Endl;
2710 Log() << "events. By default, the AdaBoost method is employed, which gives " << Endl;
2711 Log() << "events that were misclassified in the previous tree a larger " << Endl;
2712 Log() << "weight in the training of the following tree." << Endl;
2713 Log() << Endl;
2714 Log() << "Decision trees are a sequence of binary splits of the data sample" << Endl;
2715 Log() << "using a single discriminant variable at a time. A test event " << Endl;
2716 Log() << "ending up after the sequence of left-right splits in a final " << Endl;
2717 Log() << "(\"leaf\") node is classified as either signal or background" << Endl;
2718 Log() << "depending on the majority type of training events in that node." << Endl;
2719 Log() << Endl;
2720 Log() << gTools().Color("bold") << "--- Performance optimisation:" << gTools().Color("reset") << Endl;
2721 Log() << Endl;
2722 Log() << "By the nature of the binary splits performed on the individual" << Endl;
2723 Log() << "variables, decision trees do not deal well with linear correlations" << Endl;
2724 Log() << "between variables (they need to approximate the linear split in" << Endl;
2725 Log() << "the two dimensional space by a sequence of splits on the two " << Endl;
2726 Log() << "variables individually). Hence decorrelation could be useful " << Endl;
2727 Log() << "to optimise the BDT performance." << Endl;
2728 Log() << Endl;
2729 Log() << gTools().Color("bold") << "--- Performance tuning via configuration options:" << gTools().Color("reset") << Endl;
2730 Log() << Endl;
2731 Log() << "The two most important parameters in the configuration are the " << Endl;
2732 Log() << "minimal number of events requested by a leaf node as percentage of the " <<Endl;
2733 Log() << " number of training events (option \"MinNodeSize\" replacing the actual number " << Endl;
2734 Log() << " of events \"nEventsMin\" as given in earlier versions" << Endl;
2735 Log() << "If this number is too large, detailed features " << Endl;
2736 Log() << "in the parameter space are hard to be modelled. If it is too small, " << Endl;
2737 Log() << "the risk to overtrain rises and boosting seems to be less effective" << Endl;
2738 Log() << " typical values from our current experience for best performance " << Endl;
2739 Log() << " are between 0.5(%) and 10(%) " << Endl;
2740 Log() << Endl;
2741 Log() << "The default minimal number is currently set to " << Endl;
2742 Log() << " max(20, (N_training_events / N_variables^2 / 10)) " << Endl;
2743 Log() << "and can be changed by the user." << Endl;
2744 Log() << Endl;
2745 Log() << "The other crucial parameter, the pruning strength (\"PruneStrength\")," << Endl;
2746 Log() << "is also related to overtraining. It is a regularisation parameter " << Endl;
2747 Log() << "that is used when determining after the training which splits " << Endl;
2748 Log() << "are considered statistically insignificant and are removed. The" << Endl;
2749 Log() << "user is advised to carefully watch the BDT screen output for" << Endl;
2750 Log() << "the comparison between efficiencies obtained on the training and" << Endl;
2751 Log() << "the independent test sample. They should be equal within statistical" << Endl;
2752 Log() << "errors, in order to minimize statistical fluctuations in different samples." << Endl;
2753}
2754
2755////////////////////////////////////////////////////////////////////////////////
2756/// Make ROOT-independent C++ class for classifier response (classifier-specific implementation).
2757
2758void TMVA::MethodBDT::MakeClassSpecific( std::ostream& fout, const TString& className ) const
2759{
2760 TString nodeName = className;
2761 nodeName.ReplaceAll("Read","");
2762 nodeName.Append("Node");
2763 // write BDT-specific classifier response
2764 fout << " std::vector<"<<nodeName<<"*> fForest; // i.e. root nodes of decision trees" << std::endl;
2765 fout << " std::vector<double> fBoostWeights; // the weights applied in the individual boosts" << std::endl;
2766 fout << "};" << std::endl << std::endl;
2767
2768 if(GetAnalysisType() == Types::kMulticlass) {
2769 fout << "std::vector<double> ReadBDTG::GetMulticlassValues__( const std::vector<double>& inputValues ) const" << std::endl;
2770 fout << "{" << std::endl;
2771 fout << " uint nClasses = " << DataInfo().GetNClasses() << ";" << std::endl;
2772 fout << " std::vector<double> fMulticlassReturnVal;" << std::endl;
2773 fout << " fMulticlassReturnVal.reserve(nClasses);" << std::endl;
2774 fout << std::endl;
2775 fout << " std::vector<double> temp(nClasses);" << std::endl;
2776 fout << " auto forestSize = fForest.size();" << std::endl;
2777 fout << " // trees 0, nClasses, 2*nClasses, ... belong to class 0" << std::endl;
2778 fout << " // trees 1, nClasses+1, 2*nClasses+1, ... belong to class 1 and so forth" << std::endl;
2779 fout << " uint classOfTree = 0;" << std::endl;
2780 fout << " for (uint itree = 0; itree < forestSize; ++itree) {" << std::endl;
2781 fout << " BDTGNode *current = fForest[itree];" << std::endl;
2782 fout << " while (current->GetNodeType() == 0) { //intermediate node" << std::endl;
2783 fout << " if (current->GoesRight(inputValues)) current=(BDTGNode*)current->GetRight();" << std::endl;
2784 fout << " else current=(BDTGNode*)current->GetLeft();" << std::endl;
2785 fout << " }" << std::endl;
2786 fout << " temp[classOfTree] += current->GetResponse();" << std::endl;
2787 fout << " if (++classOfTree == nClasses) classOfTree = 0; // cheap modulo" << std::endl;
2788 fout << " }" << std::endl;
2789 fout << std::endl;
2790 fout << " // we want to calculate sum of exp(temp[j] - temp[i]) for all i,j (i!=j)" << std::endl;
2791 fout << " // first calculate exp(), then replace minus with division." << std::endl;
2792 fout << " std::transform(temp.begin(), temp.end(), temp.begin(), [](double d){return exp(d);});" << std::endl;
2793 fout << std::endl;
2794 fout << " for(uint iClass=0; iClass<nClasses; iClass++){" << std::endl;
2795 fout << " double norm = 0.0;" << std::endl;
2796 fout << " for(uint j=0;j<nClasses;j++){" << std::endl;
2797 fout << " if(iClass!=j)" << std::endl;
2798 fout << " norm += temp[j] / temp[iClass];" << std::endl;
2799 fout << " }" << std::endl;
2800 fout << " fMulticlassReturnVal.push_back(1.0/(1.0+norm));" << std::endl;
2801 fout << " }" << std::endl;
2802 fout << std::endl;
2803 fout << " return fMulticlassReturnVal;" << std::endl;
2804 fout << "}" << std::endl;
2805 } else {
2806 fout << "double " << className << "::GetMvaValue__( const std::vector<double>& inputValues ) const" << std::endl;
2807 fout << "{" << std::endl;
2808 fout << " double myMVA = 0;" << std::endl;
2809 if (fDoPreselection){
2810 for (UInt_t ivar = 0; ivar< fIsLowBkgCut.size(); ivar++){
2811 if (fIsLowBkgCut[ivar]){
2812 fout << " if (inputValues["<<ivar<<"] < " << fLowBkgCut[ivar] << ") return -1; // is background preselection cut" << std::endl;
2813 }
2814 if (fIsLowSigCut[ivar]){
2815 fout << " if (inputValues["<<ivar<<"] < "<< fLowSigCut[ivar] << ") return 1; // is signal preselection cut" << std::endl;
2816 }
2817 if (fIsHighBkgCut[ivar]){
2818 fout << " if (inputValues["<<ivar<<"] > "<<fHighBkgCut[ivar] <<") return -1; // is background preselection cut" << std::endl;
2819 }
2820 if (fIsHighSigCut[ivar]){
2821 fout << " if (inputValues["<<ivar<<"] > "<<fHighSigCut[ivar]<<") return 1; // is signal preselection cut" << std::endl;
2822 }
2823 }
2824 }
2825
2826 if (fBoostType!="Grad"){
2827 fout << " double norm = 0;" << std::endl;
2828 }
2829 fout << " for (unsigned int itree=0; itree<fForest.size(); itree++){" << std::endl;
2830 fout << " "<<nodeName<<" *current = fForest[itree];" << std::endl;
2831 fout << " while (current->GetNodeType() == 0) { //intermediate node" << std::endl;
2832 fout << " if (current->GoesRight(inputValues)) current=("<<nodeName<<"*)current->GetRight();" << std::endl;
2833 fout << " else current=("<<nodeName<<"*)current->GetLeft();" << std::endl;
2834 fout << " }" << std::endl;
2835 if (fBoostType=="Grad"){
2836 fout << " myMVA += current->GetResponse();" << std::endl;
2837 }else{
2838 if (fUseYesNoLeaf) fout << " myMVA += fBoostWeights[itree] * current->GetNodeType();" << std::endl;
2839 else fout << " myMVA += fBoostWeights[itree] * current->GetPurity();" << std::endl;
2840 fout << " norm += fBoostWeights[itree];" << std::endl;
2841 }
2842 fout << " }" << std::endl;
2843 if (fBoostType=="Grad"){
2844 fout << " return 2.0/(1.0+exp(-2.0*myMVA))-1.0;" << std::endl;
2845 }
2846 else fout << " return myMVA /= norm;" << std::endl;
2847 fout << "}" << std::endl << std::endl;
2848 }
2849
2850 fout << "void " << className << "::Initialize()" << std::endl;
2851 fout << "{" << std::endl;
2852 fout << " double inf = std::numeric_limits<double>::infinity();" << std::endl;
2853 fout << " double nan = std::numeric_limits<double>::quiet_NaN();" << std::endl;
2854 //Now for each decision tree, write directly the constructors of the nodes in the tree structure
2855 for (UInt_t itree=0; itree<GetNTrees(); itree++) {
2856 fout << " // itree = " << itree << std::endl;
2857 fout << " fBoostWeights.push_back(" << fBoostWeights[itree] << ");" << std::endl;
2858 fout << " fForest.push_back( " << std::endl;
2859 this->MakeClassInstantiateNode((DecisionTreeNode*)fForest[itree]->GetRoot(), fout, className);
2860 fout <<" );" << std::endl;
2861 }
2862 fout << " return;" << std::endl;
2863 fout << "};" << std::endl;
2864 fout << std::endl;
2865 fout << "// Clean up" << std::endl;
2866 fout << "inline void " << className << "::Clear() " << std::endl;
2867 fout << "{" << std::endl;
2868 fout << " for (unsigned int itree=0; itree<fForest.size(); itree++) { " << std::endl;
2869 fout << " delete fForest[itree]; " << std::endl;
2870 fout << " }" << std::endl;
2871 fout << "}" << std::endl;
2872 fout << std::endl;
2873}
2874
2875////////////////////////////////////////////////////////////////////////////////
2876/// Specific class header.
2877
2878void TMVA::MethodBDT::MakeClassSpecificHeader( std::ostream& fout, const TString& className) const
2879{
2880 TString nodeName = className;
2881 nodeName.ReplaceAll("Read","");
2882 nodeName.Append("Node");
2883 fout << "#include <algorithm>" << std::endl;
2884 fout << "#include <limits>" << std::endl;
2885 fout << std::endl;
2886 //fout << "#ifndef NN" << std::endl; commented out on purpose see next line
2887 fout << "#define NN new "<<nodeName << std::endl; // NN definition depends on individual methods. Important to have NO #ifndef if several BDT methods compile together
2888 //fout << "#endif" << std::endl; commented out on purpose see previous line
2889 fout << std::endl;
2890 fout << "#ifndef "<<nodeName<<"__def" << std::endl;
2891 fout << "#define "<<nodeName<<"__def" << std::endl;
2892 fout << std::endl;
2893 fout << "class "<<nodeName<<" {" << std::endl;
2894 fout << std::endl;
2895 fout << "public:" << std::endl;
2896 fout << std::endl;
2897 fout << " // constructor of an essentially \"empty\" node floating in space" << std::endl;
2898 fout << " "<<nodeName<<" ( "<<nodeName<<"* left,"<<nodeName<<"* right," << std::endl;
2899 if (fUseFisherCuts){
2900 fout << " int nFisherCoeff," << std::endl;
2901 for (UInt_t i=0;i<GetNVariables()+1;i++){
2902 fout << " double fisherCoeff"<<i<<"," << std::endl;
2903 }
2904 }
2905 fout << " int selector, double cutValue, bool cutType, " << std::endl;
2906 fout << " int nodeType, double purity, double response ) :" << std::endl;
2907 fout << " fLeft ( left )," << std::endl;
2908 fout << " fRight ( right )," << std::endl;
2909 if (fUseFisherCuts) fout << " fNFisherCoeff ( nFisherCoeff )," << std::endl;
2910 fout << " fSelector ( selector )," << std::endl;
2911 fout << " fCutValue ( cutValue )," << std::endl;
2912 fout << " fCutType ( cutType )," << std::endl;
2913 fout << " fNodeType ( nodeType )," << std::endl;
2914 fout << " fPurity ( purity )," << std::endl;
2915 fout << " fResponse ( response ){" << std::endl;
2916 if (fUseFisherCuts){
2917 for (UInt_t i=0;i<GetNVariables()+1;i++){
2918 fout << " fFisherCoeff.push_back(fisherCoeff"<<i<<");" << std::endl;
2919 }
2920 }
2921 fout << " }" << std::endl << std::endl;
2922 fout << " virtual ~"<<nodeName<<"();" << std::endl << std::endl;
2923 fout << " // test event if it descends the tree at this node to the right" << std::endl;
2924 fout << " virtual bool GoesRight( const std::vector<double>& inputValues ) const;" << std::endl;
2925 fout << " "<<nodeName<<"* GetRight( void ) {return fRight; };" << std::endl << std::endl;
2926 fout << " // test event if it descends the tree at this node to the left " << std::endl;
2927 fout << " virtual bool GoesLeft ( const std::vector<double>& inputValues ) const;" << std::endl;
2928 fout << " "<<nodeName<<"* GetLeft( void ) { return fLeft; }; " << std::endl << std::endl;
2929 fout << " // return S/(S+B) (purity) at this node (from training)" << std::endl << std::endl;
2930 fout << " double GetPurity( void ) const { return fPurity; } " << std::endl;
2931 fout << " // return the node type" << std::endl;
2932 fout << " int GetNodeType( void ) const { return fNodeType; }" << std::endl;
2933 fout << " double GetResponse(void) const {return fResponse;}" << std::endl << std::endl;
2934 fout << "private:" << std::endl << std::endl;
2935 fout << " "<<nodeName<<"* fLeft; // pointer to the left daughter node" << std::endl;
2936 fout << " "<<nodeName<<"* fRight; // pointer to the right daughter node" << std::endl;
2937 if (fUseFisherCuts){
2938 fout << " int fNFisherCoeff; // =0 if this node doesn't use fisher, else =nvar+1 " << std::endl;
2939 fout << " std::vector<double> fFisherCoeff; // the fisher coeff (offset at the last element)" << std::endl;
2940 }
2941 fout << " int fSelector; // index of variable used in node selection (decision tree) " << std::endl;
2942 fout << " double fCutValue; // cut value applied on this node to discriminate bkg against sig" << std::endl;
2943 fout << " bool fCutType; // true: if event variable > cutValue ==> signal , false otherwise" << std::endl;
2944 fout << " int fNodeType; // Type of node: -1 == Bkg-leaf, 1 == Signal-leaf, 0 = internal " << std::endl;
2945 fout << " double fPurity; // Purity of node from training"<< std::endl;
2946 fout << " double fResponse; // Regression response value of node" << std::endl;
2947 fout << "}; " << std::endl;
2948 fout << std::endl;
2949 fout << "//_______________________________________________________________________" << std::endl;
2950 fout << " "<<nodeName<<"::~"<<nodeName<<"()" << std::endl;
2951 fout << "{" << std::endl;
2952 fout << " if (fLeft != NULL) delete fLeft;" << std::endl;
2953 fout << " if (fRight != NULL) delete fRight;" << std::endl;
2954 fout << "}; " << std::endl;
2955 fout << std::endl;
2956 fout << "//_______________________________________________________________________" << std::endl;
2957 fout << "bool "<<nodeName<<"::GoesRight( const std::vector<double>& inputValues ) const" << std::endl;
2958 fout << "{" << std::endl;
2959 fout << " // test event if it descends the tree at this node to the right" << std::endl;
2960 fout << " bool result;" << std::endl;
2961 if (fUseFisherCuts){
2962 fout << " if (fNFisherCoeff == 0){" << std::endl;
2963 fout << " result = (inputValues[fSelector] >= fCutValue );" << std::endl;
2964 fout << " }else{" << std::endl;
2965 fout << " double fisher = fFisherCoeff.at(fFisherCoeff.size()-1);" << std::endl;
2966 fout << " for (unsigned int ivar=0; ivar<fFisherCoeff.size()-1; ivar++)" << std::endl;
2967 fout << " fisher += fFisherCoeff.at(ivar)*inputValues.at(ivar);" << std::endl;
2968 fout << " result = fisher > fCutValue;" << std::endl;
2969 fout << " }" << std::endl;
2970 }else{
2971 fout << " result = (inputValues[fSelector] >= fCutValue );" << std::endl;
2972 }
2973 fout << " if (fCutType == true) return result; //the cuts are selecting Signal ;" << std::endl;
2974 fout << " else return !result;" << std::endl;
2975 fout << "}" << std::endl;
2976 fout << std::endl;
2977 fout << "//_______________________________________________________________________" << std::endl;
2978 fout << "bool "<<nodeName<<"::GoesLeft( const std::vector<double>& inputValues ) const" << std::endl;
2979 fout << "{" << std::endl;
2980 fout << " // test event if it descends the tree at this node to the left" << std::endl;
2981 fout << " if (!this->GoesRight(inputValues)) return true;" << std::endl;
2982 fout << " else return false;" << std::endl;
2983 fout << "}" << std::endl;
2984 fout << std::endl;
2985 fout << "#endif" << std::endl;
2986 fout << std::endl;
2987}
2988
2989////////////////////////////////////////////////////////////////////////////////
2990/// Recursively descends a tree and writes the node instance to the output stream.
2991
2992void TMVA::MethodBDT::MakeClassInstantiateNode( DecisionTreeNode *n, std::ostream& fout, const TString& className ) const
2993{
2994 if (n == NULL) {
2995 Log() << kFATAL << "MakeClassInstantiateNode: started with undefined node" <<Endl;
2996 return ;
2997 }
2998 fout << "NN("<<std::endl;
2999 if (n->GetLeft() != NULL){
3000 this->MakeClassInstantiateNode( (DecisionTreeNode*)n->GetLeft() , fout, className);
3001 }
3002 else {
3003 fout << "0";
3004 }
3005 fout << ", " <<std::endl;
3006 if (n->GetRight() != NULL){
3007 this->MakeClassInstantiateNode( (DecisionTreeNode*)n->GetRight(), fout, className );
3008 }
3009 else {
3010 fout << "0";
3011 }
3012 fout << ", " << std::endl
3013 << std::setprecision(6);
3014 if (fUseFisherCuts){
3015 fout << n->GetNFisherCoeff() << ", ";
3016 for (UInt_t i=0; i< GetNVariables()+1; i++) {
3017 if (n->GetNFisherCoeff() == 0 ){
3018 fout << "0, ";
3019 }else{
3020 fout << n->GetFisherCoeff(i) << ", ";
3021 }
3022 }
3023 }
3024 fout << n->GetSelector() << ", "
3025 << n->GetCutValue() << ", "
3026 << n->GetCutType() << ", "
3027 << n->GetNodeType() << ", "
3028 << n->GetPurity() << ","
3029 << n->GetResponse() << ") ";
3030}
3031
3032////////////////////////////////////////////////////////////////////////////////
3033/// Find useful preselection cuts that will be applied before
3034/// and Decision Tree training.. (and of course also applied
3035/// in the GetMVA .. --> -1 for background +1 for Signal)
3036
3037void TMVA::MethodBDT::DeterminePreselectionCuts(const std::vector<const TMVA::Event*>& eventSample)
3038{
3039 Double_t nTotS = 0.0, nTotB = 0.0;
3040 Int_t nTotS_unWeighted = 0, nTotB_unWeighted = 0;
3041
3042 std::vector<TMVA::BDTEventWrapper> bdtEventSample;
3043
3044 fIsLowSigCut.assign(GetNvar(),kFALSE);
3045 fIsLowBkgCut.assign(GetNvar(),kFALSE);
3046 fIsHighSigCut.assign(GetNvar(),kFALSE);
3047 fIsHighBkgCut.assign(GetNvar(),kFALSE);
3048
3049 fLowSigCut.assign(GetNvar(),0.); // ---------------| --> in var is signal (accept all above lower cut)
3050 fLowBkgCut.assign(GetNvar(),0.); // ---------------| --> in var is bkg (accept all above lower cut)
3051 fHighSigCut.assign(GetNvar(),0.); // <-- | -------------- in var is signal (accept all blow cut)
3052 fHighBkgCut.assign(GetNvar(),0.); // <-- | -------------- in var is blg (accept all blow cut)
3053
3054
3055 // Initialize (un)weighted counters for signal & background
3056 // Construct a list of event wrappers that point to the original data
3057 for( std::vector<const TMVA::Event*>::const_iterator it = eventSample.begin(); it != eventSample.end(); ++it ) {
3058 if (DataInfo().IsSignal(*it)){
3059 nTotS += (*it)->GetWeight();
3060 ++nTotS_unWeighted;
3061 }
3062 else {
3063 nTotB += (*it)->GetWeight();
3064 ++nTotB_unWeighted;
3065 }
3066 bdtEventSample.push_back(TMVA::BDTEventWrapper(*it));
3067 }
3068
3069 for( UInt_t ivar = 0; ivar < GetNvar(); ivar++ ) { // loop over all discriminating variables
3070 TMVA::BDTEventWrapper::SetVarIndex(ivar); // select the variable to sort by
3071 std::sort( bdtEventSample.begin(),bdtEventSample.end() ); // sort the event data
3072
3073 Double_t bkgWeightCtr = 0.0, sigWeightCtr = 0.0;
3074 std::vector<TMVA::BDTEventWrapper>::iterator it = bdtEventSample.begin(), it_end = bdtEventSample.end();
3075 for( ; it != it_end; ++it ) {
3076 if (DataInfo().IsSignal(**it))
3077 sigWeightCtr += (**it)->GetWeight();
3078 else
3079 bkgWeightCtr += (**it)->GetWeight();
3080 // Store the accumulated signal (background) weights
3081 it->SetCumulativeWeight(false,bkgWeightCtr);
3082 it->SetCumulativeWeight(true,sigWeightCtr);
3083 }
3084
3085 //variable that determines how "exact" you cut on the preselection found in the training data. Here I chose
3086 //1% of the variable range...
3087 Double_t dVal = (DataInfo().GetVariableInfo(ivar).GetMax() - DataInfo().GetVariableInfo(ivar).GetMin())/100. ;
3088 Double_t nSelS, nSelB, effS=0.05, effB=0.05, rejS=0.05, rejB=0.05;
3089 Double_t tmpEffS, tmpEffB, tmpRejS, tmpRejB;
3090 // Locate the optimal cut for this (ivar-th) variable
3091
3092
3093
3094 for(UInt_t iev = 1; iev < bdtEventSample.size(); iev++) {
3095 //dVal = bdtEventSample[iev].GetVal() - bdtEventSample[iev-1].GetVal();
3096
3097 nSelS = bdtEventSample[iev].GetCumulativeWeight(true);
3098 nSelB = bdtEventSample[iev].GetCumulativeWeight(false);
3099 // you look for some 100% efficient pre-selection cut to remove background.. i.e. nSelS=0 && nSelB>5%nTotB or ( nSelB=0 nSelS>5%nTotS)
3100 tmpEffS=nSelS/nTotS;
3101 tmpEffB=nSelB/nTotB;
3102 tmpRejS=1-tmpEffS;
3103 tmpRejB=1-tmpEffB;
3104 if (nSelS==0 && tmpEffB>effB) {effB=tmpEffB; fLowBkgCut[ivar] = bdtEventSample[iev].GetVal() - dVal; fIsLowBkgCut[ivar]=kTRUE;}
3105 else if (nSelB==0 && tmpEffS>effS) {effS=tmpEffS; fLowSigCut[ivar] = bdtEventSample[iev].GetVal() - dVal; fIsLowSigCut[ivar]=kTRUE;}
3106 else if (nSelB==nTotB && tmpRejS>rejS) {rejS=tmpRejS; fHighSigCut[ivar] = bdtEventSample[iev].GetVal() + dVal; fIsHighSigCut[ivar]=kTRUE;}
3107 else if (nSelS==nTotS && tmpRejB>rejB) {rejB=tmpRejB; fHighBkgCut[ivar] = bdtEventSample[iev].GetVal() + dVal; fIsHighBkgCut[ivar]=kTRUE;}
3108
3109 }
3110 }
3111
3112 Log() << kDEBUG << " \tfound and suggest the following possible pre-selection cuts " << Endl;
3113 if (fDoPreselection) Log() << kDEBUG << "\tthe training will be done after these cuts... and GetMVA value returns +1, (-1) for a signal (bkg) event that passes these cuts" << Endl;
3114 else Log() << kDEBUG << "\tas option DoPreselection was not used, these cuts however will not be performed, but the training will see the full sample"<<Endl;
3115 for (UInt_t ivar=0; ivar < GetNvar(); ivar++ ) { // loop over all discriminating variables
3116 if (fIsLowBkgCut[ivar]){
3117 Log() << kDEBUG << " \tfound cut: Bkg if var " << ivar << " < " << fLowBkgCut[ivar] << Endl;
3118 }
3119 if (fIsLowSigCut[ivar]){
3120 Log() << kDEBUG << " \tfound cut: Sig if var " << ivar << " < " << fLowSigCut[ivar] << Endl;
3121 }
3122 if (fIsHighBkgCut[ivar]){
3123 Log() << kDEBUG << " \tfound cut: Bkg if var " << ivar << " > " << fHighBkgCut[ivar] << Endl;
3124 }
3125 if (fIsHighSigCut[ivar]){
3126 Log() << kDEBUG << " \tfound cut: Sig if var " << ivar << " > " << fHighSigCut[ivar] << Endl;
3127 }
3128 }
3129
3130 return;
3131}
3132
3133////////////////////////////////////////////////////////////////////////////////
3134/// Apply the preselection cuts before even bothering about any
3135/// Decision Trees in the GetMVA .. --> -1 for background +1 for Signal
3136
3138{
3139 Double_t result=0;
3140
3141 for (UInt_t ivar=0; ivar < GetNvar(); ivar++ ) { // loop over all discriminating variables
3142 if (fIsLowBkgCut[ivar]){
3143 if (ev->GetValue(ivar) < fLowBkgCut[ivar]) result = -1; // is background
3144 }
3145 if (fIsLowSigCut[ivar]){
3146 if (ev->GetValue(ivar) < fLowSigCut[ivar]) result = 1; // is signal
3147 }
3148 if (fIsHighBkgCut[ivar]){
3149 if (ev->GetValue(ivar) > fHighBkgCut[ivar]) result = -1; // is background
3150 }
3151 if (fIsHighSigCut[ivar]){
3152 if (ev->GetValue(ivar) > fHighSigCut[ivar]) result = 1; // is signal
3153 }
3154 }
3155
3156 return result;
3157}
3158
#define REGISTER_METHOD(CLASS)
for example
ROOT::R::TRInterface & r
Definition: Object.C:4
#define d(i)
Definition: RSha256.hxx:102
#define f(i)
Definition: RSha256.hxx:104
#define h(i)
Definition: RSha256.hxx:106
#define e(i)
Definition: RSha256.hxx:103
static RooMathCoreReg dummy
int Int_t
Definition: RtypesCore.h:43
unsigned int UInt_t
Definition: RtypesCore.h:44
const Bool_t kFALSE
Definition: RtypesCore.h:90
bool Bool_t
Definition: RtypesCore.h:61
double Double_t
Definition: RtypesCore.h:57
long long Long64_t
Definition: RtypesCore.h:71
float Float_t
Definition: RtypesCore.h:55
const Bool_t kTRUE
Definition: RtypesCore.h:89
#define ClassImp(name)
Definition: Rtypes.h:361
int type
Definition: TGX11.cxx:120
double ceil(double)
double floor(double)
double exp(double)
char * Form(const char *fmt,...)
A pseudo container class which is a generator of indices.
Definition: TSeq.hxx:66
A TGraph is an object made of two arrays X and Y with npoints each.
Definition: TGraph.h:41
virtual void SetPoint(Int_t i, Double_t x, Double_t y)
Set x and y values for point number i.
Definition: TGraph.cxx:2269
virtual void SetName(const char *name="")
Set graph name.
Definition: TGraph.cxx:2308
Int_t GetN() const
Definition: TGraph.h:123
virtual void SetTitle(const char *title="")
Change (i.e.
Definition: TGraph.cxx:2324
virtual void Set(Int_t n)
Set number of points in the graph Existing coordinates are preserved New coordinates above fNpoints a...
Definition: TGraph.cxx:2204
1-D histogram with a float per channel (see TH1 documentation)}
Definition: TH1.h:571
1-D histogram with an int per channel (see TH1 documentation)}
Definition: TH1.h:530
The TH1 histogram class.
Definition: TH1.h:56
virtual void SetXTitle(const char *title)
Definition: TH1.h:409
virtual Int_t Fill(Double_t x)
Increment bin with abscissa X by 1.
Definition: TH1.cxx:3275
virtual void SetBinContent(Int_t bin, Double_t content)
Set bin content see convention for numbering bins in TH1::GetBin In case the bin number is greater th...
Definition: TH1.cxx:8678
virtual void SetYTitle(const char *title)
Definition: TH1.h:410
2-D histogram with a float per channel (see TH1 documentation)}
Definition: TH2.h:251
Service class for 2-Dim histogram classes.
Definition: TH2.h:30
Absolute Deviation BDT Loss Function.
Definition: LossFunction.h:261
static void SetVarIndex(Int_t iVar)
UInt_t GetNNodes() const
Definition: BinaryTree.h:86
Executor & GetThreadExecutor()
Get executor class for multi-thread usage In case when MT is not enabled will return a serial executo...
Definition: Config.h:83
static Config & Instance()
static function: returns TMVA instance
Definition: Config.cxx:107
Implementation of the CrossEntropy as separation criterion.
Definition: CrossEntropy.h:43
Class that contains all the data information.
Definition: DataSetInfo.h:60
Implementation of a Decision Tree.
Definition: DecisionTree.h:64
TMVA::DecisionTreeNode * GetEventNode(const TMVA::Event &e) const
get the pointer to the leaf node where a particular event ends up in... (used in gradient boosting)
Double_t CheckEvent(const TMVA::Event *, Bool_t UseYesNoLeaf=kFALSE) const
the event e is put into the decision tree (starting at the root node) and the output is NodeType (sig...
static DecisionTree * CreateFromXML(void *node, UInt_t tmva_Version_Code=TMVA_VERSION_CODE)
re-create a new tree (decision tree or search tree) from XML
Float_t GetValue(UInt_t ivar) const
return value of i'th variable
Definition: Event.cxx:236
void SetTarget(UInt_t itgt, Float_t value)
set the target value (dimension itgt) to value
Definition: Event.cxx:359
Float_t GetTarget(UInt_t itgt) const
Definition: Event.h:102
void Foreach(Function func, unsigned int nTimes, unsigned nChunks=0)
wrap TExecutor::Foreach
Definition: Executor.h:110
unsigned int GetPoolSize() const
Definition: Executor.h:99
auto Map(F func, unsigned nTimes) -> std::vector< typename std::result_of< F()>::type >
Wrap TExecutor::Map functions.
Definition: Executor.h:133
Implementation of the GiniIndex With Laplace correction as separation criterion.
Implementation of the GiniIndex as separation criterion.
Definition: GiniIndex.h:63
Huber BDT Loss Function.
Definition: LossFunction.h:179
The TMVA::Interval Class.
Definition: Interval.h:61
Least Squares BDT Loss Function.
Definition: LossFunction.h:222
The TMVA::Interval Class.
Definition: LogInterval.h:83
Analysis of Boosted Decision Trees.
Definition: MethodBDT.h:61
void Init(void)
Common initialisation with defaults for the BDT-Method.
Definition: MethodBDT.cxx:688
static const Int_t fgDebugLevel
Definition: MethodBDT.h:300
MethodBDT(const TString &jobName, const TString &methodTitle, DataSetInfo &theData, const TString &theOption="")
The standard constructor for the "boosted decision trees".
Definition: MethodBDT.cxx:164
void BoostMonitor(Int_t iTree)
Fills the ROCIntegral vs Itree from the testSample for the monitoring plots during the training .
Definition: MethodBDT.cxx:1753
const std::vector< Float_t > & GetMulticlassValues()
Get the multiclass MVA response for the BDT classifier.
Definition: MethodBDT.cxx:2496
Double_t AdaBoostR2(std::vector< const TMVA::Event * > &, DecisionTree *dt)
Adaption of the AdaBoost to regression problems (see H.Drucker 1997).
Definition: MethodBDT.cxx:2194
void MakeClassSpecific(std::ostream &, const TString &) const
Make ROOT-independent C++ class for classifier response (classifier-specific implementation).
Definition: MethodBDT.cxx:2758
void GetHelpMessage() const
Get help message text.
Definition: MethodBDT.cxx:2701
LossFunctionBDT * fRegressionLossFunctionBDTG
Definition: MethodBDT.h:297
void DeterminePreselectionCuts(const std::vector< const TMVA::Event * > &eventSample)
Find useful preselection cuts that will be applied before and Decision Tree training.
Definition: MethodBDT.cxx:3037
Double_t GradBoost(std::vector< const TMVA::Event * > &, DecisionTree *dt, UInt_t cls=0)
Calculate the desired response value for each region.
Definition: MethodBDT.cxx:1596
const Ranking * CreateRanking()
Compute ranking of input variables.
Definition: MethodBDT.cxx:2684
virtual void SetTuneParameters(std::map< TString, Double_t > tuneParameters)
Set the tuning parameters according to the argument.
Definition: MethodBDT.cxx:1122
Double_t AdaCost(std::vector< const TMVA::Event * > &, DecisionTree *dt)
The AdaCost boosting algorithm takes a simple cost Matrix (currently fixed for all events....
Definition: MethodBDT.cxx:2025
void DeclareOptions()
Define the options (their key words).
Definition: MethodBDT.cxx:334
virtual std::map< TString, Double_t > OptimizeTuningParameters(TString fomType="ROCIntegral", TString fitType="FitGA")
Call the Optimizer with the set of parameters and ranges that are meant to be tuned.
Definition: MethodBDT.cxx:1069
Double_t Boost(std::vector< const TMVA::Event * > &, DecisionTree *dt, UInt_t cls=0)
Apply the boosting algorithm (the algorithm is selecte via the the "option" given in the constructor.
Definition: MethodBDT.cxx:1719
Double_t TestTreeQuality(DecisionTree *dt)
Test the tree quality.. in terms of Misclassification.
Definition: MethodBDT.cxx:1698
Double_t Bagging()
Call it boot-strapping, re-sampling or whatever you like, in the end it is nothing else but applying ...
Definition: MethodBDT.cxx:2141
void UpdateTargets(std::vector< const TMVA::Event * > &, UInt_t cls=0)
Calculate residual for all events.
Definition: MethodBDT.cxx:1436
void UpdateTargetsRegression(std::vector< const TMVA::Event * > &, Bool_t first=kFALSE)
Calculate residuals for all events and update targets for next iter.
Definition: MethodBDT.cxx:1558
Double_t GradBoostRegression(std::vector< const TMVA::Event * > &, DecisionTree *dt)
Implementation of M_TreeBoost using any loss function as described by Friedman 1999.
Definition: MethodBDT.cxx:1630
void WriteMonitoringHistosToFile(void) const
Here we could write some histograms created during the processing to the output file.
Definition: MethodBDT.cxx:2629
virtual ~MethodBDT(void)
Destructor.
Definition: MethodBDT.cxx:754
void AddWeightsXMLTo(void *parent) const
Write weights to XML.
Definition: MethodBDT.cxx:2311
Double_t GetGradBoostMVA(const TMVA::Event *e, UInt_t nTrees)
Returns MVA value: -1 for background, 1 for signal.
Definition: MethodBDT.cxx:1422
virtual Bool_t HasAnalysisType(Types::EAnalysisType type, UInt_t numberClasses, UInt_t numberTargets)
BDT can handle classification with multiple classes and regression with one regression-target.
Definition: MethodBDT.cxx:281
Double_t RegBoost(std::vector< const TMVA::Event * > &, DecisionTree *dt)
A special boosting only for Regression (not implemented).
Definition: MethodBDT.cxx:2186
void InitEventSample()
Initialize the event sample (i.e. reset the boost-weights... etc).
Definition: MethodBDT.cxx:762
Double_t ApplyPreselectionCuts(const Event *ev)
Apply the preselection cuts before even bothering about any Decision Trees in the GetMVA .
Definition: MethodBDT.cxx:3137
void SetMinNodeSize(Double_t sizeInPercent)
Definition: MethodBDT.cxx:661
void ProcessOptions()
The option string is decoded, for available options see "DeclareOptions".
Definition: MethodBDT.cxx:471
void PreProcessNegativeEventWeights()
O.k.
Definition: MethodBDT.cxx:933
void MakeClassInstantiateNode(DecisionTreeNode *n, std::ostream &fout, const TString &className) const
Recursively descends a tree and writes the node instance to the output stream.
Definition: MethodBDT.cxx:2992
Double_t AdaBoost(std::vector< const TMVA::Event * > &, DecisionTree *dt)
The AdaBoost implementation.
Definition: MethodBDT.cxx:1847
TTree * fMonitorNtuple
Definition: MethodBDT.h:262
std::vector< Double_t > GetVariableImportance()
Return the relative variable importance, normalized to all variables together having the importance 1...
Definition: MethodBDT.cxx:2644
Double_t GetMvaValue(Double_t *err=0, Double_t *errUpper=0)
Definition: MethodBDT.cxx:2444
Double_t PrivateGetMvaValue(const TMVA::Event *ev, Double_t *err=0, Double_t *errUpper=0, UInt_t useNTrees=0)
Return the MVA value (range [-1;1]) that classifies the event according to the majority vote from the...
Definition: MethodBDT.cxx:2469
void InitGradBoost(std::vector< const TMVA::Event * > &)
Initialize targets for first tree.
Definition: MethodBDT.cxx:1659
void Train(void)
BDT training.
Definition: MethodBDT.cxx:1143
void GetBaggedSubSample(std::vector< const TMVA::Event * > &)
Fills fEventSample with fBaggedSampleFraction*NEvents random training events.
Definition: MethodBDT.cxx:2152
const std::vector< Float_t > & GetRegressionValues()
Get the regression value generated by the BDTs.
Definition: MethodBDT.cxx:2543
SeparationBase * fSepType
Definition: MethodBDT.h:227
void ReadWeightsFromXML(void *parent)
Reads the BDT from the xml file.
Definition: MethodBDT.cxx:2342
void ReadWeightsFromStream(std::istream &istr)
Read the weights (BDT coefficients).
Definition: MethodBDT.cxx:2409
void Reset(void)
Reset the method, as if it had just been instantiated (forget all training etc.).
Definition: MethodBDT.cxx:726
void MakeClassSpecificHeader(std::ostream &, const TString &) const
Specific class header.
Definition: MethodBDT.cxx:2878
void DeclareCompatibilityOptions()
Options that are used ONLY for the READER to ensure backward compatibility.
Definition: MethodBDT.cxx:455
Virtual base Class for all MVA method.
Definition: MethodBase.h:111
virtual void DeclareCompatibilityOptions()
options that are used ONLY for the READER to ensure backward compatibility they are hence without any...
Definition: MethodBase.cxx:598
Implementation of the MisClassificationError as separation criterion.
std::map< TString, Double_t > optimize()
PDF wrapper for histograms; uses user-defined spline interpolation.
Definition: PDF.h:63
@ kSpline3
Definition: PDF.h:70
Ranking for variables in method (implementation)
Definition: Ranking.h:48
Class that is the base-class for a vector of result.
Definition: Results.h:57
TGraph * GetGraph(const TString &alias) const
Definition: Results.cxx:153
TH1 * GetHist(const TString &alias) const
Definition: Results.cxx:136
void Store(TObject *obj, const char *alias=0)
Definition: Results.cxx:86
Implementation of the SdivSqrtSplusB as separation criterion.
Timing information for training and evaluation of MVA methods.
Definition: Timer.h:58
TString GetElapsedTime(Bool_t Scientific=kTRUE)
returns pretty string with elapsed time
Definition: Timer.cxx:147
void DrawProgressBar(Int_t, const TString &comment="")
draws progress bar in color or B&W caution:
Definition: Timer.cxx:203
void * GetNextChild(void *prevchild, const char *childname=0)
XML helpers.
Definition: Tools.cxx:1173
void * AddChild(void *parent, const char *childname, const char *content=0, bool isRootNode=false)
add child node
Definition: Tools.cxx:1135
const TString & Color(const TString &)
human readable color strings
Definition: Tools.cxx:839
void * GetChild(void *parent, const char *childname=0)
get child node
Definition: Tools.cxx:1161
void ReadAttr(void *node, const char *, T &value)
read attribute from xml
Definition: Tools.h:335
void AddAttr(void *node, const char *, const T &value, Int_t precision=16)
add attribute to xml
Definition: Tools.h:353
std::vector< TMatrixDSym * > * CalcCovarianceMatrices(const std::vector< Event * > &events, Int_t maxCls, VariableTransformBase *transformBase=0)
compute covariance matrices
Definition: Tools.cxx:1525
void UsefulSortAscending(std::vector< std::vector< Double_t > > &, std::vector< TString > *vs=0)
sort 2D vector (AND in parallel a TString vector) in such a way that the "first vector is sorted" and...
Definition: Tools.cxx:549
Singleton class for Global types used by TMVA.
Definition: Types.h:73
EAnalysisType
Definition: Types.h:127
@ kMulticlass
Definition: Types.h:130
@ kClassification
Definition: Types.h:128
@ kMaxAnalysisType
Definition: Types.h:132
@ kRegression
Definition: Types.h:129
@ kTraining
Definition: Types.h:144
virtual Double_t Determinant() const
TMatrixTSym< Element > & Invert(Double_t *det=0)
Invert the matrix and calculate its determinant Notice that the LU decomposition is used instead of B...
virtual const char * GetTitle() const
Returns title of object.
Definition: TNamed.h:48
virtual Int_t Write(const char *name=0, Int_t option=0, Int_t bufsize=0)
Write this object to the current directory.
Definition: TObject.cxx:796
virtual void Delete(Option_t *option="")
Delete this object.
Definition: TObject.cxx:169
virtual Int_t Read(const char *name)
Read contents of object with specified name from the current directory.
Definition: TObject.cxx:562
Random number generator class based on M.
Definition: TRandom3.h:27
virtual Double_t PoissonD(Double_t mean)
Generates a random number according to a Poisson law.
Definition: TRandom.cxx:443
Basic string class.
Definition: TString.h:131
Double_t Atof() const
Return floating-point value contained in string.
Definition: TString.cxx:1987
Bool_t IsFloat() const
Returns kTRUE if string contains a floating point or integer number.
Definition: TString.cxx:1791
TString & ReplaceAll(const TString &s1, const TString &s2)
Definition: TString.h:687
TString & Append(const char *cs)
Definition: TString.h:559
A TTree represents a columnar dataset.
Definition: TTree.h:78
const Int_t n
Definition: legend1.C:16
TGraphErrors * gr
Definition: legend1.C:25
static const uint32_t K[64]
Definition: RSha256.hxx:148
void Print(std::ostream &os, const OptionType &opt)
double dist(Rotation3D const &r1, Rotation3D const &r2)
Definition: 3DDistances.cxx:48
VecExpr< UnaryOp< Fabs< T >, VecExpr< A, T, D >, T >, T, D > fabs(const VecExpr< A, T, D > &rhs)
TClass * GetClass(T *)
Definition: TClass.h:658
TSeq< unsigned int > TSeqU
Definition: TSeq.hxx:195
void GetMethodName(TString &name, TKey *mkey)
Definition: tmvaglob.cxx:335
create variable transformations
Tools & gTools()
void BDT(TString dataset, const TString &fin="TMVA.root")
MsgLogger & Endl(MsgLogger &ml)
Definition: MsgLogger.h:158
Short_t Max(Short_t a, Short_t b)
Definition: TMathBase.h:212
Double_t Exp(Double_t x)
Definition: TMath.h:717
Int_t FloorNint(Double_t x)
Definition: TMath.h:697
constexpr Double_t E()
Base of natural log:
Definition: TMath.h:97
Double_t Log(Double_t x)
Definition: TMath.h:750
Double_t Sqrt(Double_t x)
Definition: TMath.h:681
LongDouble_t Power(LongDouble_t x, LongDouble_t y)
Definition: TMath.h:725
Int_t CeilNint(Double_t x)
Definition: TMath.h:689
Short_t Min(Short_t a, Short_t b)
Definition: TMathBase.h:180
Short_t Abs(Short_t d)
Definition: TMathBase.h:120
Definition: first.py:1
static long int sum(long int i)
Definition: Factory.cxx:2275
REAL epsilon
Definition: triangle.c:617