Logo ROOT  
Reference Guide
MethodBDT.cxx
Go to the documentation of this file.
1// Author: Andreas Hoecker, Joerg Stelzer, Helge Voss, Kai Voss, Eckhard v. Toerne, Jan Therhaag
2
3/**********************************************************************************
4 * Project: TMVA - a Root-integrated toolkit for multivariate data analysis *
5 * Package: TMVA *
6 * Class : MethodBDT (BDT = Boosted Decision Trees) *
7 * Web : http://tmva.sourceforge.net *
8 * *
9 * Description: *
10 * Analysis of Boosted Decision Trees *
11 * *
12 * Authors (alphabetical): *
13 * Andreas Hoecker <Andreas.Hocker@cern.ch> - CERN, Switzerland *
14 * Helge Voss <Helge.Voss@cern.ch> - MPI-K Heidelberg, Germany *
15 * Kai Voss <Kai.Voss@cern.ch> - U. of Victoria, Canada *
16 * Doug Schouten <dschoute@sfu.ca> - Simon Fraser U., Canada *
17 * Jan Therhaag <jan.therhaag@cern.ch> - U. of Bonn, Germany *
18 * Eckhard v. Toerne <evt@uni-bonn.de> - U of Bonn, Germany *
19 * *
20 * Copyright (c) 2005-2011: *
21 * CERN, Switzerland *
22 * U. of Victoria, Canada *
23 * MPI-K Heidelberg, Germany *
24 * U. of Bonn, Germany *
25 * *
26 * Redistribution and use in source and binary forms, with or without *
27 * modification, are permitted according to the terms listed in LICENSE *
28 * (http://tmva.sourceforge.net/LICENSE) *
29 **********************************************************************************/
30
31/*! \class TMVA::MethodBDT
32\ingroup TMVA
33
34Analysis of Boosted Decision Trees
35
36Boosted decision trees have been successfully used in High Energy
37Physics analysis for example by the MiniBooNE experiment
38(Yang-Roe-Zhu, physics/0508045). In Boosted Decision Trees, the
39selection is done on a majority vote on the result of several decision
40trees, which are all derived from the same training sample by
41supplying different event weights during the training.
42
43### Decision trees:
44
45Successive decision nodes are used to categorize the
46events out of the sample as either signal or background. Each node
47uses only a single discriminating variable to decide if the event is
48signal-like ("goes right") or background-like ("goes left"). This
49forms a tree like structure with "baskets" at the end (leave nodes),
50and an event is classified as either signal or background according to
51whether the basket where it ends up has been classified signal or
52background during the training. Training of a decision tree is the
53process to define the "cut criteria" for each node. The training
54starts with the root node. Here one takes the full training event
55sample and selects the variable and corresponding cut value that gives
56the best separation between signal and background at this stage. Using
57this cut criterion, the sample is then divided into two subsamples, a
58signal-like (right) and a background-like (left) sample. Two new nodes
59are then created for each of the two sub-samples and they are
60constructed using the same mechanism as described for the root
61node. The devision is stopped once a certain node has reached either a
62minimum number of events, or a minimum or maximum signal purity. These
63leave nodes are then called "signal" or "background" if they contain
64more signal respective background events from the training sample.
65
66### Boosting:
67
68The idea behind adaptive boosting (AdaBoost) is, that signal events
69from the training sample, that end up in a background node
70(and vice versa) are given a larger weight than events that are in
71the correct leave node. This results in a re-weighed training event
72sample, with which then a new decision tree can be developed.
73The boosting can be applied several times (typically 100-500 times)
74and one ends up with a set of decision trees (a forest).
75Gradient boosting works more like a function expansion approach, where
76each tree corresponds to a summand. The parameters for each summand (tree)
77are determined by the minimization of a error function (binomial log-
78likelihood for classification and Huber loss for regression).
79A greedy algorithm is used, which means, that only one tree is modified
80at a time, while the other trees stay fixed.
81
82### Bagging:
83
84In this particular variant of the Boosted Decision Trees the boosting
85is not done on the basis of previous training results, but by a simple
86stochastic re-sampling of the initial training event sample.
87
88### Random Trees:
89
90Similar to the "Random Forests" from Leo Breiman and Adele Cutler, it
91uses the bagging algorithm together and bases the determination of the
92best node-split during the training on a random subset of variables only
93which is individually chosen for each split.
94
95### Analysis:
96
97Applying an individual decision tree to a test event results in a
98classification of the event as either signal or background. For the
99boosted decision tree selection, an event is successively subjected to
100the whole set of decision trees and depending on how often it is
101classified as signal, a "likelihood" estimator is constructed for the
102event being signal or background. The value of this estimator is the
103one which is then used to select the events from an event sample, and
104the cut value on this estimator defines the efficiency and purity of
105the selection.
106
107*/
108
109
110#include "TMVA/MethodBDT.h"
111#include "TMVA/Config.h"
112
113#include "TMVA/BDTEventWrapper.h"
116#include "TMVA/Configurable.h"
117#include "TMVA/CrossEntropy.h"
118#include "TMVA/DecisionTree.h"
119#include "TMVA/DataSet.h"
120#include "TMVA/GiniIndex.h"
122#include "TMVA/Interval.h"
123#include "TMVA/IMethod.h"
124#include "TMVA/LogInterval.h"
125#include "TMVA/MethodBase.h"
127#include "TMVA/MsgLogger.h"
129#include "TMVA/PDF.h"
130#include "TMVA/Ranking.h"
131#include "TMVA/Results.h"
133#include "TMVA/SdivSqrtSplusB.h"
134#include "TMVA/SeparationBase.h"
135#include "TMVA/Timer.h"
136#include "TMVA/Tools.h"
137#include "TMVA/Types.h"
138
139#include "TRandom3.h"
140#include "TMath.h"
141#include "TMatrixTSym.h"
142#include "TGraph.h"
143
144#include <iostream>
145#include <iomanip>
146#include <algorithm>
147#include <cmath>
148#include <numeric>
149#include <unordered_map>
150
151using std::vector;
152using std::make_pair;
153
155
157
159
160////////////////////////////////////////////////////////////////////////////////
161/// The standard constructor for the "boosted decision trees".
162
164 const TString& methodTitle,
165 DataSetInfo& theData,
166 const TString& theOption ) :
167 TMVA::MethodBase( jobName, Types::kBDT, methodTitle, theData, theOption)
168 , fTrainSample(0)
169 , fNTrees(0)
170 , fSigToBkgFraction(0)
171 , fAdaBoostBeta(0)
172// , fTransitionPoint(0)
173 , fShrinkage(0)
174 , fBaggedBoost(kFALSE)
175 , fBaggedGradBoost(kFALSE)
176// , fSumOfWeights(0)
177 , fMinNodeEvents(0)
178 , fMinNodeSize(5)
179 , fMinNodeSizeS("5%")
180 , fNCuts(0)
181 , fUseFisherCuts(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
182 , fMinLinCorrForFisher(.8) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
183 , fUseExclusiveVars(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
184 , fUseYesNoLeaf(kFALSE)
185 , fNodePurityLimit(0)
186 , fNNodesMax(0)
187 , fMaxDepth(0)
188 , fPruneMethod(DecisionTree::kNoPruning)
189 , fPruneStrength(0)
190 , fFValidationEvents(0)
191 , fAutomatic(kFALSE)
192 , fRandomisedTrees(kFALSE)
193 , fUseNvars(0)
194 , fUsePoissonNvars(0) // don't use this initialisation, only here to make Coverity happy. Is set in Init()
195 , fUseNTrainEvents(0)
196 , fBaggedSampleFraction(0)
197 , fNoNegWeightsInTraining(kFALSE)
198 , fInverseBoostNegWeights(kFALSE)
199 , fPairNegWeightsGlobal(kFALSE)
200 , fTrainWithNegWeights(kFALSE)
201 , fDoBoostMonitor(kFALSE)
202 , fITree(0)
203 , fBoostWeight(0)
204 , fErrorFraction(0)
205 , fCss(0)
206 , fCts_sb(0)
207 , fCtb_ss(0)
208 , fCbb(0)
209 , fDoPreselection(kFALSE)
210 , fSkipNormalization(kFALSE)
211 , fHistoricBool(kFALSE)
212{
213 fMonitorNtuple = NULL;
214 fSepType = NULL;
216}
217
218////////////////////////////////////////////////////////////////////////////////
219
221 const TString& theWeightFile)
222 : TMVA::MethodBase( Types::kBDT, theData, theWeightFile)
223 , fTrainSample(0)
224 , fNTrees(0)
225 , fSigToBkgFraction(0)
226 , fAdaBoostBeta(0)
227// , fTransitionPoint(0)
228 , fShrinkage(0)
229 , fBaggedBoost(kFALSE)
230 , fBaggedGradBoost(kFALSE)
231// , fSumOfWeights(0)
232 , fMinNodeEvents(0)
233 , fMinNodeSize(5)
234 , fMinNodeSizeS("5%")
235 , fNCuts(0)
236 , fUseFisherCuts(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
237 , fMinLinCorrForFisher(.8) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
238 , fUseExclusiveVars(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
239 , fUseYesNoLeaf(kFALSE)
240 , fNodePurityLimit(0)
241 , fNNodesMax(0)
242 , fMaxDepth(0)
243 , fPruneMethod(DecisionTree::kNoPruning)
244 , fPruneStrength(0)
245 , fFValidationEvents(0)
246 , fAutomatic(kFALSE)
247 , fRandomisedTrees(kFALSE)
248 , fUseNvars(0)
249 , fUsePoissonNvars(0) // don't use this initialisation, only here to make Coverity happy. Is set in Init()
250 , fUseNTrainEvents(0)
251 , fBaggedSampleFraction(0)
252 , fNoNegWeightsInTraining(kFALSE)
253 , fInverseBoostNegWeights(kFALSE)
254 , fPairNegWeightsGlobal(kFALSE)
255 , fTrainWithNegWeights(kFALSE)
256 , fDoBoostMonitor(kFALSE)
257 , fITree(0)
258 , fBoostWeight(0)
259 , fErrorFraction(0)
260 , fCss(0)
261 , fCts_sb(0)
262 , fCtb_ss(0)
263 , fCbb(0)
264 , fDoPreselection(kFALSE)
265 , fSkipNormalization(kFALSE)
266 , fHistoricBool(kFALSE)
267{
268 fMonitorNtuple = NULL;
269 fSepType = NULL;
271 // constructor for calculating BDT-MVA using previously generated decision trees
272 // the result of the previous training (the decision trees) are read in via the
273 // weight file. Make sure the variables correspond to the ones used in
274 // creating the "weight"-file
275}
276
277////////////////////////////////////////////////////////////////////////////////
278/// BDT can handle classification with multiple classes and regression with one regression-target.
279
281{
282 if (type == Types::kClassification && numberClasses == 2) return kTRUE;
283 if (type == Types::kMulticlass ) return kTRUE;
284 if( type == Types::kRegression && numberTargets == 1 ) return kTRUE;
285 return kFALSE;
286}
287
288////////////////////////////////////////////////////////////////////////////////
289/// Define the options (their key words). That can be set in the option string.
290///
291/// know options:
292///
293/// - nTrees number of trees in the forest to be created
294/// - BoostType the boosting type for the trees in the forest (AdaBoost e.t.c..).
295/// Known:
296/// - AdaBoost
297/// - AdaBoostR2 (Adaboost for regression)
298/// - Bagging
299/// - GradBoost
300/// - AdaBoostBeta the boosting parameter, beta, for AdaBoost
301/// - UseRandomisedTrees choose at each node splitting a random set of variables
302/// - UseNvars use UseNvars variables in randomised trees
303/// - UsePoisson Nvars use UseNvars not as fixed number but as mean of a poisson distribution
304/// - SeparationType the separation criterion applied in the node splitting.
305/// Known:
306/// - GiniIndex
307/// - MisClassificationError
308/// - CrossEntropy
309/// - SDivSqrtSPlusB
310/// - MinNodeSize: minimum percentage of training events in a leaf node (leaf criteria, stop splitting)
311/// - nCuts: the number of steps in the optimisation of the cut for a node (if < 0, then
312/// step size is determined by the events)
313/// - UseFisherCuts: use multivariate splits using the Fisher criterion
314/// - UseYesNoLeaf decide if the classification is done simply by the node type, or the S/B
315/// (from the training) in the leaf node
316/// - NodePurityLimit the minimum purity to classify a node as a signal node (used in pruning and boosting to determine
317/// misclassification error rate)
318/// - PruneMethod The Pruning method.
319/// Known:
320/// - NoPruning // switch off pruning completely
321/// - ExpectedError
322/// - CostComplexity
323/// - PruneStrength a parameter to adjust the amount of pruning. Should be large enough such that overtraining is avoided.
324/// - PruningValFraction number of events to use for optimizing pruning (only if PruneStrength < 0, i.e. automatic pruning)
325/// - NegWeightTreatment
326/// - IgnoreNegWeightsInTraining Ignore negative weight events in the training.
327/// - DecreaseBoostWeight Boost ev. with neg. weight with 1/boostweight instead of boostweight
328/// - PairNegWeightsGlobal Pair ev. with neg. and pos. weights in training sample and "annihilate" them
329/// - MaxDepth maximum depth of the decision tree allowed before further splitting is stopped
330/// - SkipNormalization Skip normalization at initialization, to keep expectation value of BDT output
331/// according to the fraction of events
332
334{
335 DeclareOptionRef(fNTrees, "NTrees", "Number of trees in the forest");
336 if (DoRegression()) {
337 DeclareOptionRef(fMaxDepth=50,"MaxDepth","Max depth of the decision tree allowed");
338 }else{
339 DeclareOptionRef(fMaxDepth=3,"MaxDepth","Max depth of the decision tree allowed");
340 }
341
342 TString tmp="5%"; if (DoRegression()) tmp="0.2%";
343 DeclareOptionRef(fMinNodeSizeS=tmp, "MinNodeSize", "Minimum percentage of training events required in a leaf node (default: Classification: 5%, Regression: 0.2%)");
344 // MinNodeSize: minimum percentage of training events in a leaf node (leaf criteria, stop splitting)
345 DeclareOptionRef(fNCuts, "nCuts", "Number of grid points in variable range used in finding optimal cut in node splitting");
346
347 DeclareOptionRef(fBoostType, "BoostType", "Boosting type for the trees in the forest (note: AdaCost is still experimental)");
348
349 AddPreDefVal(TString("AdaBoost"));
350 AddPreDefVal(TString("RealAdaBoost"));
351 AddPreDefVal(TString("AdaCost"));
352 AddPreDefVal(TString("Bagging"));
353 // AddPreDefVal(TString("RegBoost"));
354 AddPreDefVal(TString("AdaBoostR2"));
355 AddPreDefVal(TString("Grad"));
356 if (DoRegression()) {
357 fBoostType = "AdaBoostR2";
358 }else{
359 fBoostType = "AdaBoost";
360 }
361 DeclareOptionRef(fAdaBoostR2Loss="Quadratic", "AdaBoostR2Loss", "Type of Loss function in AdaBoostR2");
362 AddPreDefVal(TString("Linear"));
363 AddPreDefVal(TString("Quadratic"));
364 AddPreDefVal(TString("Exponential"));
365
366 DeclareOptionRef(fBaggedBoost=kFALSE, "UseBaggedBoost","Use only a random subsample of all events for growing the trees in each boost iteration.");
367 DeclareOptionRef(fShrinkage = 1.0, "Shrinkage", "Learning rate for BoostType=Grad algorithm");
368 DeclareOptionRef(fAdaBoostBeta=.5, "AdaBoostBeta", "Learning rate for AdaBoost algorithm");
369 DeclareOptionRef(fRandomisedTrees,"UseRandomisedTrees","Determine at each node splitting the cut variable only as the best out of a random subset of variables (like in RandomForests)");
370 DeclareOptionRef(fUseNvars,"UseNvars","Size of the subset of variables used with RandomisedTree option");
371 DeclareOptionRef(fUsePoissonNvars,"UsePoissonNvars", "Interpret \"UseNvars\" not as fixed number but as mean of a Poisson distribution in each split with RandomisedTree option");
372 DeclareOptionRef(fBaggedSampleFraction=.6,"BaggedSampleFraction","Relative size of bagged event sample to original size of the data sample (used whenever bagging is used (i.e. UseBaggedBoost, Bagging,)" );
373
374 DeclareOptionRef(fUseYesNoLeaf=kTRUE, "UseYesNoLeaf",
375 "Use Sig or Bkg categories, or the purity=S/(S+B) as classification of the leaf node -> Real-AdaBoost");
376 if (DoRegression()) {
377 fUseYesNoLeaf = kFALSE;
378 }
379
380 DeclareOptionRef(fNegWeightTreatment="InverseBoostNegWeights","NegWeightTreatment","How to treat events with negative weights in the BDT training (particular the boosting) : IgnoreInTraining; Boost With inverse boostweight; Pair events with negative and positive weights in training sample and *annihilate* them (experimental!)");
381 AddPreDefVal(TString("InverseBoostNegWeights"));
382 AddPreDefVal(TString("IgnoreNegWeightsInTraining"));
383 AddPreDefVal(TString("NoNegWeightsInTraining")); // well, let's be nice to users and keep at least this old name anyway ..
384 AddPreDefVal(TString("PairNegWeightsGlobal"));
385 AddPreDefVal(TString("Pray"));
386
387
388
389 DeclareOptionRef(fCss=1., "Css", "AdaCost: cost of true signal selected signal");
390 DeclareOptionRef(fCts_sb=1.,"Cts_sb","AdaCost: cost of true signal selected bkg");
391 DeclareOptionRef(fCtb_ss=1.,"Ctb_ss","AdaCost: cost of true bkg selected signal");
392 DeclareOptionRef(fCbb=1., "Cbb", "AdaCost: cost of true bkg selected bkg ");
393
394 DeclareOptionRef(fNodePurityLimit=0.5, "NodePurityLimit", "In boosting/pruning, nodes with purity > NodePurityLimit are signal; background otherwise.");
395
396
397 DeclareOptionRef(fSepTypeS, "SeparationType", "Separation criterion for node splitting");
398 AddPreDefVal(TString("CrossEntropy"));
399 AddPreDefVal(TString("GiniIndex"));
400 AddPreDefVal(TString("GiniIndexWithLaplace"));
401 AddPreDefVal(TString("MisClassificationError"));
402 AddPreDefVal(TString("SDivSqrtSPlusB"));
403 AddPreDefVal(TString("RegressionVariance"));
404 if (DoRegression()) {
405 fSepTypeS = "RegressionVariance";
406 }else{
407 fSepTypeS = "GiniIndex";
408 }
409
410 DeclareOptionRef(fRegressionLossFunctionBDTGS = "Huber", "RegressionLossFunctionBDTG", "Loss function for BDTG regression.");
411 AddPreDefVal(TString("Huber"));
412 AddPreDefVal(TString("AbsoluteDeviation"));
413 AddPreDefVal(TString("LeastSquares"));
414
415 DeclareOptionRef(fHuberQuantile = 0.7, "HuberQuantile", "In the Huber loss function this is the quantile that separates the core from the tails in the residuals distribution.");
416
417 DeclareOptionRef(fDoBoostMonitor=kFALSE,"DoBoostMonitor","Create control plot with ROC integral vs tree number");
418
419 DeclareOptionRef(fUseFisherCuts=kFALSE, "UseFisherCuts", "Use multivariate splits using the Fisher criterion");
420 DeclareOptionRef(fMinLinCorrForFisher=.8,"MinLinCorrForFisher", "The minimum linear correlation between two variables demanded for use in Fisher criterion in node splitting");
421 DeclareOptionRef(fUseExclusiveVars=kFALSE,"UseExclusiveVars","Variables already used in fisher criterion are not anymore analysed individually for node splitting");
422
423
424 DeclareOptionRef(fDoPreselection=kFALSE,"DoPreselection","and and apply automatic pre-selection for 100% efficient signal (bkg) cuts prior to training");
425
426
427 DeclareOptionRef(fSigToBkgFraction=1,"SigToBkgFraction","Sig to Bkg ratio used in Training (similar to NodePurityLimit, which cannot be used in real adaboost");
428
429 DeclareOptionRef(fPruneMethodS, "PruneMethod", "Note: for BDTs use small trees (e.g.MaxDepth=3) and NoPruning: Pruning: Method used for pruning (removal) of statistically insignificant branches ");
430 AddPreDefVal(TString("NoPruning"));
431 AddPreDefVal(TString("ExpectedError"));
432 AddPreDefVal(TString("CostComplexity"));
433
434 DeclareOptionRef(fPruneStrength, "PruneStrength", "Pruning strength");
435
436 DeclareOptionRef(fFValidationEvents=0.5, "PruningValFraction", "Fraction of events to use for optimizing automatic pruning.");
437
438 DeclareOptionRef(fSkipNormalization=kFALSE, "SkipNormalization", "Skip normalization at initialization, to keep expectation value of BDT output according to the fraction of events");
439
440 // deprecated options, still kept for the moment:
441 DeclareOptionRef(fMinNodeEvents=0, "nEventsMin", "deprecated: Use MinNodeSize (in % of training events) instead");
442
443 DeclareOptionRef(fBaggedGradBoost=kFALSE, "UseBaggedGrad","deprecated: Use *UseBaggedBoost* instead: Use only a random subsample of all events for growing the trees in each iteration.");
444 DeclareOptionRef(fBaggedSampleFraction, "GradBaggingFraction","deprecated: Use *BaggedSampleFraction* instead: Defines the fraction of events to be used in each iteration, e.g. when UseBaggedGrad=kTRUE. ");
445 DeclareOptionRef(fUseNTrainEvents,"UseNTrainEvents","deprecated: Use *BaggedSampleFraction* instead: Number of randomly picked training events used in randomised (and bagged) trees");
446 DeclareOptionRef(fNNodesMax,"NNodesMax","deprecated: Use MaxDepth instead to limit the tree size" );
447
448
449}
450
451////////////////////////////////////////////////////////////////////////////////
452/// Options that are used ONLY for the READER to ensure backward compatibility.
453
456
457
458 DeclareOptionRef(fHistoricBool=kTRUE, "UseWeightedTrees",
459 "Use weighted trees or simple average in classification from the forest");
460 DeclareOptionRef(fHistoricBool=kFALSE, "PruneBeforeBoost", "Flag to prune the tree before applying boosting algorithm");
461 DeclareOptionRef(fHistoricBool=kFALSE,"RenormByClass","Individually re-normalize each event class to the original size after boosting");
462
463 AddPreDefVal(TString("NegWeightTreatment"),TString("IgnoreNegWeights"));
464
465}
466
467////////////////////////////////////////////////////////////////////////////////
468/// The option string is decoded, for available options see "DeclareOptions".
469
471{
472 fSepTypeS.ToLower();
473 if (fSepTypeS == "misclassificationerror") fSepType = new MisClassificationError();
474 else if (fSepTypeS == "giniindex") fSepType = new GiniIndex();
475 else if (fSepTypeS == "giniindexwithlaplace") fSepType = new GiniIndexWithLaplace();
476 else if (fSepTypeS == "crossentropy") fSepType = new CrossEntropy();
477 else if (fSepTypeS == "sdivsqrtsplusb") fSepType = new SdivSqrtSplusB();
478 else if (fSepTypeS == "regressionvariance") fSepType = NULL;
479 else {
480 Log() << kINFO << GetOptions() << Endl;
481 Log() << kFATAL << "<ProcessOptions> unknown Separation Index option " << fSepTypeS << " called" << Endl;
482 }
483
484 if(!(fHuberQuantile >= 0.0 && fHuberQuantile <= 1.0)){
485 Log() << kINFO << GetOptions() << Endl;
486 Log() << kFATAL << "<ProcessOptions> Huber Quantile must be in range [0,1]. Value given, " << fHuberQuantile << ", does not match this criteria" << Endl;
487 }
488
489
490 fRegressionLossFunctionBDTGS.ToLower();
491 if (fRegressionLossFunctionBDTGS == "huber") fRegressionLossFunctionBDTG = new HuberLossFunctionBDT(fHuberQuantile);
492 else if (fRegressionLossFunctionBDTGS == "leastsquares") fRegressionLossFunctionBDTG = new LeastSquaresLossFunctionBDT();
493 else if (fRegressionLossFunctionBDTGS == "absolutedeviation") fRegressionLossFunctionBDTG = new AbsoluteDeviationLossFunctionBDT();
494 else {
495 Log() << kINFO << GetOptions() << Endl;
496 Log() << kFATAL << "<ProcessOptions> unknown Regression Loss Function BDT option " << fRegressionLossFunctionBDTGS << " called" << Endl;
497 }
498
499 fPruneMethodS.ToLower();
500 if (fPruneMethodS == "expectederror") fPruneMethod = DecisionTree::kExpectedErrorPruning;
501 else if (fPruneMethodS == "costcomplexity") fPruneMethod = DecisionTree::kCostComplexityPruning;
502 else if (fPruneMethodS == "nopruning") fPruneMethod = DecisionTree::kNoPruning;
503 else {
504 Log() << kINFO << GetOptions() << Endl;
505 Log() << kFATAL << "<ProcessOptions> unknown PruneMethod " << fPruneMethodS << " option called" << Endl;
506 }
507 if (fPruneStrength < 0 && (fPruneMethod != DecisionTree::kNoPruning) && fBoostType!="Grad") fAutomatic = kTRUE;
508 else fAutomatic = kFALSE;
509 if (fAutomatic && fPruneMethod==DecisionTree::kExpectedErrorPruning){
510 Log() << kFATAL
511 << "Sorry automatic pruning strength determination is not implemented yet for ExpectedErrorPruning" << Endl;
512 }
513
514
515 if (fMinNodeEvents > 0){
516 fMinNodeSize = Double_t(fMinNodeEvents*100.) / Data()->GetNTrainingEvents();
517 Log() << kWARNING << "You have explicitly set ** nEventsMin = " << fMinNodeEvents<<" ** the min absolute number \n"
518 << "of events in a leaf node. This is DEPRECATED, please use the option \n"
519 << "*MinNodeSize* giving the relative number as percentage of training \n"
520 << "events instead. \n"
521 << "nEventsMin="<<fMinNodeEvents<< "--> MinNodeSize="<<fMinNodeSize<<"%"
522 << Endl;
523 Log() << kWARNING << "Note also that explicitly setting *nEventsMin* so far OVERWRITES the option recommended \n"
524 << " *MinNodeSize* = " << fMinNodeSizeS << " option !!" << Endl ;
525 fMinNodeSizeS = Form("%F3.2",fMinNodeSize);
526
527 }else{
528 SetMinNodeSize(fMinNodeSizeS);
529 }
530
531
532 fAdaBoostR2Loss.ToLower();
533
534 if (fBoostType=="Grad") {
535 fPruneMethod = DecisionTree::kNoPruning;
536 if (fNegWeightTreatment=="InverseBoostNegWeights"){
537 Log() << kINFO << "the option NegWeightTreatment=InverseBoostNegWeights does"
538 << " not exist for BoostType=Grad" << Endl;
539 Log() << kINFO << "--> change to new default NegWeightTreatment=Pray" << Endl;
540 Log() << kDEBUG << "i.e. simply keep them as if which should work fine for Grad Boost" << Endl;
541 fNegWeightTreatment="Pray";
542 fNoNegWeightsInTraining=kFALSE;
543 }
544 } else if (fBoostType=="RealAdaBoost"){
545 fBoostType = "AdaBoost";
546 fUseYesNoLeaf = kFALSE;
547 } else if (fBoostType=="AdaCost"){
548 fUseYesNoLeaf = kFALSE;
549 }
550
551 if (fFValidationEvents < 0.0) fFValidationEvents = 0.0;
552 if (fAutomatic && fFValidationEvents > 0.5) {
553 Log() << kWARNING << "You have chosen to use more than half of your training sample "
554 << "to optimize the automatic pruning algorithm. This is probably wasteful "
555 << "and your overall results will be degraded. Are you sure you want this?"
556 << Endl;
557 }
558
559
560 if (this->Data()->HasNegativeEventWeights()){
561 Log() << kINFO << " You are using a Monte Carlo that has also negative weights. "
562 << "That should in principle be fine as long as on average you end up with "
563 << "something positive. For this you have to make sure that the minimal number "
564 << "of (un-weighted) events demanded for a tree node (currently you use: MinNodeSize="
565 << fMinNodeSizeS << " ("<< fMinNodeSize << "%)"
566 <<", (or the deprecated equivalent nEventsMin) you can set this via the "
567 <<"BDT option string when booking the "
568 << "classifier) is large enough to allow for reasonable averaging!!! "
569 << " If this does not help.. maybe you want to try the option: IgnoreNegWeightsInTraining "
570 << "which ignores events with negative weight in the training. " << Endl
571 << Endl << "Note: You'll get a WARNING message during the training if that should ever happen" << Endl;
572 }
573
574 if (DoRegression()) {
575 if (fUseYesNoLeaf && !IsConstructedFromWeightFile()){
576 Log() << kWARNING << "Regression Trees do not work with fUseYesNoLeaf=TRUE --> I will set it to FALSE" << Endl;
577 fUseYesNoLeaf = kFALSE;
578 }
579
580 if (fSepType != NULL){
581 Log() << kWARNING << "Regression Trees do not work with Separation type other than <RegressionVariance> --> I will use it instead" << Endl;
582 fSepType = NULL;
583 }
584 if (fUseFisherCuts){
585 Log() << kWARNING << "Sorry, UseFisherCuts is not available for regression analysis, I will ignore it!" << Endl;
586 fUseFisherCuts = kFALSE;
587 }
588 if (fNCuts < 0) {
589 Log() << kWARNING << "Sorry, the option of nCuts<0 using a more elaborate node splitting algorithm " << Endl;
590 Log() << kWARNING << "is not implemented for regression analysis ! " << Endl;
591 Log() << kWARNING << "--> I switch do default nCuts = 20 and use standard node splitting"<<Endl;
592 fNCuts=20;
593 }
594 }
595 if (fRandomisedTrees){
596 Log() << kINFO << " Randomised trees use no pruning" << Endl;
597 fPruneMethod = DecisionTree::kNoPruning;
598 // fBoostType = "Bagging";
599 }
600
601 if (fUseFisherCuts) {
602 Log() << kWARNING << "When using the option UseFisherCuts, the other option nCuts<0 (i.e. using" << Endl;
603 Log() << " a more elaborate node splitting algorithm) is not implemented. " << Endl;
604 //I will switch o " << Endl;
605 //Log() << "--> I switch do default nCuts = 20 and use standard node splitting WITH possible Fisher criteria"<<Endl;
606 fNCuts=20;
607 }
608
609 if (fNTrees==0){
610 Log() << kERROR << " Zero Decision Trees demanded... that does not work !! "
611 << " I set it to 1 .. just so that the program does not crash"
612 << Endl;
613 fNTrees = 1;
614 }
615
616 fNegWeightTreatment.ToLower();
617 if (fNegWeightTreatment == "ignorenegweightsintraining") fNoNegWeightsInTraining = kTRUE;
618 else if (fNegWeightTreatment == "nonegweightsintraining") fNoNegWeightsInTraining = kTRUE;
619 else if (fNegWeightTreatment == "inverseboostnegweights") fInverseBoostNegWeights = kTRUE;
620 else if (fNegWeightTreatment == "pairnegweightsglobal") fPairNegWeightsGlobal = kTRUE;
621 else if (fNegWeightTreatment == "pray") Log() << kDEBUG << "Yes, good luck with praying " << Endl;
622 else {
623 Log() << kINFO << GetOptions() << Endl;
624 Log() << kFATAL << "<ProcessOptions> unknown option for treating negative event weights during training " << fNegWeightTreatment << " requested" << Endl;
625 }
626
627 if (fNegWeightTreatment == "pairnegweightsglobal")
628 Log() << kWARNING << " you specified the option NegWeightTreatment=PairNegWeightsGlobal : This option is still considered EXPERIMENTAL !! " << Endl;
629
630
631 // dealing with deprecated options !
632 if (fNNodesMax>0) {
633 UInt_t tmp=1; // depth=0 == 1 node
634 fMaxDepth=0;
635 while (tmp < fNNodesMax){
636 tmp+=2*tmp;
637 fMaxDepth++;
638 }
639 Log() << kWARNING << "You have specified a deprecated option *NNodesMax="<<fNNodesMax
640 << "* \n this has been translated to MaxDepth="<<fMaxDepth<<Endl;
641 }
642
643
644 if (fUseNTrainEvents>0){
645 fBaggedSampleFraction = (Double_t) fUseNTrainEvents/Data()->GetNTrainingEvents();
646 Log() << kWARNING << "You have specified a deprecated option *UseNTrainEvents="<<fUseNTrainEvents
647 << "* \n this has been translated to BaggedSampleFraction="<<fBaggedSampleFraction<<"(%)"<<Endl;
648 }
649
650 if (fBoostType=="Bagging") fBaggedBoost = kTRUE;
651 if (fBaggedGradBoost){
652 fBaggedBoost = kTRUE;
653 Log() << kWARNING << "You have specified a deprecated option *UseBaggedGrad* --> please use *UseBaggedBoost* instead" << Endl;
654 }
655
656}
657
658////////////////////////////////////////////////////////////////////////////////
659
661 if (sizeInPercent > 0 && sizeInPercent < 50){
662 fMinNodeSize=sizeInPercent;
663
664 } else {
665 Log() << kFATAL << "you have demanded a minimal node size of "
666 << sizeInPercent << "% of the training events.. \n"
667 << " that somehow does not make sense "<<Endl;
668 }
669
670}
671
672////////////////////////////////////////////////////////////////////////////////
673
675 sizeInPercent.ReplaceAll("%","");
676 sizeInPercent.ReplaceAll(" ","");
677 if (sizeInPercent.IsFloat()) SetMinNodeSize(sizeInPercent.Atof());
678 else {
679 Log() << kFATAL << "I had problems reading the option MinNodeEvents, which "
680 << "after removing a possible % sign now reads " << sizeInPercent << Endl;
681 }
682}
683
684////////////////////////////////////////////////////////////////////////////////
685/// Common initialisation with defaults for the BDT-Method.
686
688{
689 fNTrees = 800;
690 if (fAnalysisType == Types::kClassification || fAnalysisType == Types::kMulticlass ) {
691 fMaxDepth = 3;
692 fBoostType = "AdaBoost";
693 if(DataInfo().GetNClasses()!=0) //workaround for multiclass application
694 fMinNodeSize = 5.;
695 }else {
696 fMaxDepth = 50;
697 fBoostType = "AdaBoostR2";
698 fAdaBoostR2Loss = "Quadratic";
699 if(DataInfo().GetNClasses()!=0) //workaround for multiclass application
700 fMinNodeSize = .2;
701 }
702
703
704 fNCuts = 20;
705 fPruneMethodS = "NoPruning";
706 fPruneMethod = DecisionTree::kNoPruning;
707 fPruneStrength = 0;
708 fAutomatic = kFALSE;
709 fFValidationEvents = 0.5;
710 fRandomisedTrees = kFALSE;
711 // fUseNvars = (GetNvar()>12) ? UInt_t(GetNvar()/8) : TMath::Max(UInt_t(2),UInt_t(GetNvar()/3));
712 fUseNvars = UInt_t(TMath::Sqrt(GetNvar())+0.6);
713 fUsePoissonNvars = kTRUE;
714 fShrinkage = 1.0;
715// fSumOfWeights = 0.0;
716
717 // reference cut value to distinguish signal-like from background-like events
718 SetSignalReferenceCut( 0 );
719}
720
721
722////////////////////////////////////////////////////////////////////////////////
723/// Reset the method, as if it had just been instantiated (forget all training etc.).
724
726{
727 // I keep the BDT EventSample and its Validation sample (eventually they should all
728 // disappear and just use the DataSet samples ..
729
730 // remove all the trees
731 for (UInt_t i=0; i<fForest.size(); i++) delete fForest[i];
732 fForest.clear();
733
734 fBoostWeights.clear();
735 if (fMonitorNtuple) { fMonitorNtuple->Delete(); fMonitorNtuple=NULL; }
736 fVariableImportance.clear();
737 fResiduals.clear();
738 fLossFunctionEventInfo.clear();
739 // now done in "InitEventSample" which is called in "Train"
740 // reset all previously stored/accumulated BOOST weights in the event sample
741 //for (UInt_t iev=0; iev<fEventSample.size(); iev++) fEventSample[iev]->SetBoostWeight(1.);
742 if (Data()) Data()->DeleteResults(GetMethodName(), Types::kTraining, GetAnalysisType());
743 Log() << kDEBUG << " successfully(?) reset the method " << Endl;
744}
745
746
747////////////////////////////////////////////////////////////////////////////////
748/// Destructor.
749///
750/// - Note: fEventSample and ValidationSample are already deleted at the end of TRAIN
751/// When they are not used anymore
752
754{
755 for (UInt_t i=0; i<fForest.size(); i++) delete fForest[i];
756}
757
758////////////////////////////////////////////////////////////////////////////////
759/// Initialize the event sample (i.e. reset the boost-weights... etc).
760
762{
763 if (!HasTrainingTree()) Log() << kFATAL << "<Init> Data().TrainingTree() is zero pointer" << Endl;
764
765 if (fEventSample.size() > 0) { // do not re-initialise the event sample, just set all boostweights to 1. as if it were untouched
766 // reset all previously stored/accumulated BOOST weights in the event sample
767 for (UInt_t iev=0; iev<fEventSample.size(); iev++) fEventSample[iev]->SetBoostWeight(1.);
768 } else {
769 Data()->SetCurrentType(Types::kTraining);
770 UInt_t nevents = Data()->GetNTrainingEvents();
771
772 std::vector<const TMVA::Event*> tmpEventSample;
773 for (Long64_t ievt=0; ievt<nevents; ievt++) {
774 // const Event *event = new Event(*(GetEvent(ievt)));
775 Event* event = new Event( *GetTrainingEvent(ievt) );
776 tmpEventSample.push_back(event);
777 }
778
779 if (!DoRegression()) DeterminePreselectionCuts(tmpEventSample);
780 else fDoPreselection = kFALSE; // just to make sure...
781
782 for (UInt_t i=0; i<tmpEventSample.size(); i++) delete tmpEventSample[i];
783
784
785 Bool_t firstNegWeight=kTRUE;
786 Bool_t firstZeroWeight=kTRUE;
787 for (Long64_t ievt=0; ievt<nevents; ievt++) {
788 // const Event *event = new Event(*(GetEvent(ievt)));
789 // const Event* event = new Event( *GetTrainingEvent(ievt) );
790 Event* event = new Event( *GetTrainingEvent(ievt) );
791 if (fDoPreselection){
792 if (TMath::Abs(ApplyPreselectionCuts(event)) > 0.05) {
793 delete event;
794 continue;
795 }
796 }
797
798 if (event->GetWeight() < 0 && (IgnoreEventsWithNegWeightsInTraining() || fNoNegWeightsInTraining)){
799 if (firstNegWeight) {
800 Log() << kWARNING << " Note, you have events with negative event weight in the sample, but you've chosen to ignore them" << Endl;
801 firstNegWeight=kFALSE;
802 }
803 delete event;
804 }else if (event->GetWeight()==0){
805 if (firstZeroWeight) {
806 firstZeroWeight = kFALSE;
807 Log() << "Events with weight == 0 are going to be simply ignored " << Endl;
808 }
809 delete event;
810 }else{
811 if (event->GetWeight() < 0) {
812 fTrainWithNegWeights=kTRUE;
813 if (firstNegWeight){
814 firstNegWeight = kFALSE;
815 if (fPairNegWeightsGlobal){
816 Log() << kWARNING << "Events with negative event weights are found and "
817 << " will be removed prior to the actual BDT training by global "
818 << " paring (and subsequent annihilation) with positiv weight events"
819 << Endl;
820 }else{
821 Log() << kWARNING << "Events with negative event weights are USED during "
822 << "the BDT training. This might cause problems with small node sizes "
823 << "or with the boosting. Please remove negative events from training "
824 << "using the option *IgnoreEventsWithNegWeightsInTraining* in case you "
825 << "observe problems with the boosting"
826 << Endl;
827 }
828 }
829 }
830 // if fAutomatic == true you need a validation sample to optimize pruning
831 if (fAutomatic) {
832 Double_t modulo = 1.0/(fFValidationEvents);
833 Int_t imodulo = static_cast<Int_t>( fmod(modulo,1.0) > 0.5 ? ceil(modulo) : floor(modulo) );
834 if (ievt % imodulo == 0) fValidationSample.push_back( event );
835 else fEventSample.push_back( event );
836 }
837 else {
838 fEventSample.push_back(event);
839 }
840 }
841 }
842
843 if (fAutomatic) {
844 Log() << kINFO << "<InitEventSample> Internally I use " << fEventSample.size()
845 << " for Training and " << fValidationSample.size()
846 << " for Pruning Validation (" << ((Float_t)fValidationSample.size())/((Float_t)fEventSample.size()+fValidationSample.size())*100.0
847 << "% of training used for validation)" << Endl;
848 }
849
850 // some pre-processing for events with negative weights
851 if (fPairNegWeightsGlobal) PreProcessNegativeEventWeights();
852 }
853
854 if (DoRegression()) {
855 // Regression, no reweighting to do
856 } else if (DoMulticlass()) {
857 // Multiclass, only gradboost is supported. No reweighting.
858 } else if (!fSkipNormalization) {
859 // Binary classification.
860 Log() << kDEBUG << "\t<InitEventSample> For classification trees, "<< Endl;
861 Log() << kDEBUG << " \tthe effective number of backgrounds is scaled to match "<<Endl;
862 Log() << kDEBUG << " \tthe signal. Otherwise the first boosting step would do 'just that'!"<<Endl;
863 // it does not make sense in decision trees to start with unequal number of signal/background
864 // events (weights) .. hence normalize them now (happens otherwise in first 'boosting step'
865 // anyway..
866 // Also make sure, that the sum_of_weights == sample.size() .. as this is assumed in
867 // the DecisionTree to derive a sensible number for "fMinSize" (min.#events in node)
868 // that currently is an OR between "weighted" and "unweighted number"
869 // I want:
870 // nS + nB = n
871 // a*SW + b*BW = n
872 // (a*SW)/(b*BW) = fSigToBkgFraction
873 //
874 // ==> b = n/((1+f)BW) and a = (nf/(1+f))/SW
875
876 Double_t nevents = fEventSample.size();
877 Double_t sumSigW=0, sumBkgW=0;
878 Int_t sumSig=0, sumBkg=0;
879 for (UInt_t ievt=0; ievt<fEventSample.size(); ievt++) {
880 if ((DataInfo().IsSignal(fEventSample[ievt])) ) {
881 sumSigW += fEventSample[ievt]->GetWeight();
882 sumSig++;
883 } else {
884 sumBkgW += fEventSample[ievt]->GetWeight();
885 sumBkg++;
886 }
887 }
888 if (sumSigW && sumBkgW){
889 Double_t normSig = nevents/((1+fSigToBkgFraction)*sumSigW)*fSigToBkgFraction;
890 Double_t normBkg = nevents/((1+fSigToBkgFraction)*sumBkgW); ;
891 Log() << kDEBUG << "\tre-normalise events such that Sig and Bkg have respective sum of weights = "
892 << fSigToBkgFraction << Endl;
893 Log() << kDEBUG << " \tsig->sig*"<<normSig << "ev. bkg->bkg*"<<normBkg << "ev." <<Endl;
894 Log() << kHEADER << "#events: (reweighted) sig: "<< sumSigW*normSig << " bkg: " << sumBkgW*normBkg << Endl;
895 Log() << kINFO << "#events: (unweighted) sig: "<< sumSig << " bkg: " << sumBkg << Endl;
896 for (Long64_t ievt=0; ievt<nevents; ievt++) {
897 if ((DataInfo().IsSignal(fEventSample[ievt])) ) fEventSample[ievt]->SetBoostWeight(normSig);
898 else fEventSample[ievt]->SetBoostWeight(normBkg);
899 }
900 }else{
901 Log() << kINFO << "--> could not determine scaling factors as either there are " << Endl;
902 Log() << kINFO << " no signal events (sumSigW="<<sumSigW<<") or no bkg ev. (sumBkgW="<<sumBkgW<<")"<<Endl;
903 }
904
905 }
906
907 fTrainSample = &fEventSample;
908 if (fBaggedBoost){
909 GetBaggedSubSample(fEventSample);
910 fTrainSample = &fSubSample;
911 }
912
913 //just for debug purposes..
914 /*
915 sumSigW=0;
916 sumBkgW=0;
917 for (UInt_t ievt=0; ievt<fEventSample.size(); ievt++) {
918 if ((DataInfo().IsSignal(fEventSample[ievt])) ) sumSigW += fEventSample[ievt]->GetWeight();
919 else sumBkgW += fEventSample[ievt]->GetWeight();
920 }
921 Log() << kWARNING << "sigSumW="<<sumSigW<<"bkgSumW="<<sumBkgW<< Endl;
922 */
923}
924
925////////////////////////////////////////////////////////////////////////////////
926/// O.k. you know there are events with negative event weights. This routine will remove
927/// them by pairing them with the closest event(s) of the same event class with positive
928/// weights
929/// A first attempt is "brute force", I dont' try to be clever using search trees etc,
930/// just quick and dirty to see if the result is any good
931
933 Double_t totalNegWeights = 0;
934 Double_t totalPosWeights = 0;
935 Double_t totalWeights = 0;
936 std::vector<const Event*> negEvents;
937 for (UInt_t iev = 0; iev < fEventSample.size(); iev++){
938 if (fEventSample[iev]->GetWeight() < 0) {
939 totalNegWeights += fEventSample[iev]->GetWeight();
940 negEvents.push_back(fEventSample[iev]);
941 } else {
942 totalPosWeights += fEventSample[iev]->GetWeight();
943 }
944 totalWeights += fEventSample[iev]->GetWeight();
945 }
946 if (totalNegWeights == 0 ) {
947 Log() << kINFO << "no negative event weights found .. no preprocessing necessary" << Endl;
948 return;
949 } else {
950 Log() << kINFO << "found a total of " << totalNegWeights << " of negative event weights which I am going to try to pair with positive events to annihilate them" << Endl;
951 Log() << kINFO << "found a total of " << totalPosWeights << " of events with positive weights" << Endl;
952 Log() << kINFO << "--> total sum of weights = " << totalWeights << " = " << totalNegWeights+totalPosWeights << Endl;
953 }
954
955 std::vector<TMatrixDSym*>* cov = gTools().CalcCovarianceMatrices( fEventSample, 2);
956
957 TMatrixDSym *invCov;
958
959 for (Int_t i=0; i<2; i++){
960 invCov = ((*cov)[i]);
961 if ( TMath::Abs(invCov->Determinant()) < 10E-24 ) {
962 std::cout << "<MethodBDT::PreProcessNeg...> matrix is almost singular with determinant="
963 << TMath::Abs(invCov->Determinant())
964 << " did you use the variables that are linear combinations or highly correlated?"
965 << std::endl;
966 }
967 if ( TMath::Abs(invCov->Determinant()) < 10E-120 ) {
968 std::cout << "<MethodBDT::PreProcessNeg...> matrix is singular with determinant="
969 << TMath::Abs(invCov->Determinant())
970 << " did you use the variables that are linear combinations?"
971 << std::endl;
972 }
973
974 invCov->Invert();
975 }
976
977
978
979 Log() << kINFO << "Found a total of " << totalNegWeights << " in negative weights out of " << fEventSample.size() << " training events " << Endl;
980 Timer timer(negEvents.size(),"Negative Event paired");
981 for (UInt_t nev = 0; nev < negEvents.size(); nev++){
982 timer.DrawProgressBar( nev );
983 Double_t weight = negEvents[nev]->GetWeight();
984 UInt_t iClassID = negEvents[nev]->GetClass();
985 invCov = ((*cov)[iClassID]);
986 while (weight < 0){
987 // find closest event with positive event weight and "pair" it with the negative event
988 // (add their weight) until there is no negative weight anymore
989 Int_t iMin=-1;
990 Double_t dist, minDist=10E270;
991 for (UInt_t iev = 0; iev < fEventSample.size(); iev++){
992 if (iClassID==fEventSample[iev]->GetClass() && fEventSample[iev]->GetWeight() > 0){
993 dist=0;
994 for (UInt_t ivar=0; ivar < GetNvar(); ivar++){
995 for (UInt_t jvar=0; jvar<GetNvar(); jvar++){
996 dist += (negEvents[nev]->GetValue(ivar)-fEventSample[iev]->GetValue(ivar))*
997 (*invCov)[ivar][jvar]*
998 (negEvents[nev]->GetValue(jvar)-fEventSample[iev]->GetValue(jvar));
999 }
1000 }
1001 if (dist < minDist) { iMin=iev; minDist=dist;}
1002 }
1003 }
1004
1005 if (iMin > -1) {
1006 // std::cout << "Happily pairing .. weight before : " << negEvents[nev]->GetWeight() << " and " << fEventSample[iMin]->GetWeight();
1007 Double_t newWeight = (negEvents[nev]->GetWeight() + fEventSample[iMin]->GetWeight());
1008 if (newWeight > 0){
1009 negEvents[nev]->SetBoostWeight( 0 );
1010 fEventSample[iMin]->SetBoostWeight( newWeight/fEventSample[iMin]->GetOriginalWeight() ); // note the weight*boostweight should be "newWeight"
1011 } else {
1012 negEvents[nev]->SetBoostWeight( newWeight/negEvents[nev]->GetOriginalWeight() ); // note the weight*boostweight should be "newWeight"
1013 fEventSample[iMin]->SetBoostWeight( 0 );
1014 }
1015 // std::cout << " and afterwards " << negEvents[nev]->GetWeight() << " and the paired " << fEventSample[iMin]->GetWeight() << " dist="<<minDist<< std::endl;
1016 } else Log() << kFATAL << "preprocessing didn't find event to pair with the negative weight ... probably a bug" << Endl;
1017 weight = negEvents[nev]->GetWeight();
1018 }
1019 }
1020 Log() << kINFO << "<Negative Event Pairing> took: " << timer.GetElapsedTime()
1021 << " " << Endl;
1022
1023 // just check.. now there should be no negative event weight left anymore
1024 totalNegWeights = 0;
1025 totalPosWeights = 0;
1026 totalWeights = 0;
1027 Double_t sigWeight=0;
1028 Double_t bkgWeight=0;
1029 Int_t nSig=0;
1030 Int_t nBkg=0;
1031
1032 std::vector<const Event*> newEventSample;
1033
1034 for (UInt_t iev = 0; iev < fEventSample.size(); iev++){
1035 if (fEventSample[iev]->GetWeight() < 0) {
1036 totalNegWeights += fEventSample[iev]->GetWeight();
1037 totalWeights += fEventSample[iev]->GetWeight();
1038 } else {
1039 totalPosWeights += fEventSample[iev]->GetWeight();
1040 totalWeights += fEventSample[iev]->GetWeight();
1041 }
1042 if (fEventSample[iev]->GetWeight() > 0) {
1043 newEventSample.push_back(new Event(*fEventSample[iev]));
1044 if (fEventSample[iev]->GetClass() == fSignalClass){
1045 sigWeight += fEventSample[iev]->GetWeight();
1046 nSig+=1;
1047 }else{
1048 bkgWeight += fEventSample[iev]->GetWeight();
1049 nBkg+=1;
1050 }
1051 }
1052 }
1053 if (totalNegWeights < 0) Log() << kFATAL << " compensation of negative event weights with positive ones did not work " << totalNegWeights << Endl;
1054
1055 for (UInt_t i=0; i<fEventSample.size(); i++) delete fEventSample[i];
1056 fEventSample = newEventSample;
1057
1058 Log() << kINFO << " after PreProcessing, the Event sample is left with " << fEventSample.size() << " events (unweighted), all with positive weights, adding up to " << totalWeights << Endl;
1059 Log() << kINFO << " nSig="<<nSig << " sigWeight="<<sigWeight << " nBkg="<<nBkg << " bkgWeight="<<bkgWeight << Endl;
1060
1061
1062}
1063
1064////////////////////////////////////////////////////////////////////////////////
1065/// Call the Optimizer with the set of parameters and ranges that
1066/// are meant to be tuned.
1067
1068std::map<TString,Double_t> TMVA::MethodBDT::OptimizeTuningParameters(TString fomType, TString fitType)
1069{
1070 // fill all the tuning parameters that should be optimized into a map:
1071 std::map<TString,TMVA::Interval*> tuneParameters;
1072 std::map<TString,Double_t> tunedParameters;
1073
1074 // note: the 3rd parameter in the interval is the "number of bins", NOT the stepsize !!
1075 // the actual VALUES at (at least for the scan, guess also in GA) are always
1076 // read from the middle of the bins. Hence.. the choice of Intervals e.g. for the
1077 // MaxDepth, in order to make nice integer values!!!
1078
1079 // find some reasonable ranges for the optimisation of MinNodeEvents:
1080
1081 tuneParameters.insert(std::pair<TString,Interval*>("NTrees", new Interval(10,1000,5))); // stepsize 50
1082 tuneParameters.insert(std::pair<TString,Interval*>("MaxDepth", new Interval(2,4,3))); // stepsize 1
1083 tuneParameters.insert(std::pair<TString,Interval*>("MinNodeSize", new LogInterval(1,30,30))); //
1084 //tuneParameters.insert(std::pair<TString,Interval*>("NodePurityLimit",new Interval(.4,.6,3))); // stepsize .1
1085 //tuneParameters.insert(std::pair<TString,Interval*>("BaggedSampleFraction",new Interval(.4,.9,6))); // stepsize .1
1086
1087 // method-specific parameters
1088 if (fBoostType=="AdaBoost"){
1089 tuneParameters.insert(std::pair<TString,Interval*>("AdaBoostBeta", new Interval(.2,1.,5)));
1090
1091 }else if (fBoostType=="Grad"){
1092 tuneParameters.insert(std::pair<TString,Interval*>("Shrinkage", new Interval(0.05,0.50,5)));
1093
1094 }else if (fBoostType=="Bagging" && fRandomisedTrees){
1095 Int_t min_var = TMath::FloorNint( GetNvar() * .25 );
1096 Int_t max_var = TMath::CeilNint( GetNvar() * .75 );
1097 tuneParameters.insert(std::pair<TString,Interval*>("UseNvars", new Interval(min_var,max_var,4)));
1098
1099 }
1100
1101 Log()<<kINFO << " the following BDT parameters will be tuned on the respective *grid*\n"<<Endl;
1102 std::map<TString,TMVA::Interval*>::iterator it;
1103 for(it=tuneParameters.begin(); it!= tuneParameters.end(); ++it){
1104 Log() << kWARNING << it->first << Endl;
1105 std::ostringstream oss;
1106 (it->second)->Print(oss);
1107 Log()<<oss.str();
1108 Log()<<Endl;
1109 }
1110
1111 OptimizeConfigParameters optimize(this, tuneParameters, fomType, fitType);
1112 tunedParameters=optimize.optimize();
1113
1114 return tunedParameters;
1115
1116}
1117
1118////////////////////////////////////////////////////////////////////////////////
1119/// Set the tuning parameters according to the argument.
1120
1121void TMVA::MethodBDT::SetTuneParameters(std::map<TString,Double_t> tuneParameters)
1122{
1123 std::map<TString,Double_t>::iterator it;
1124 for(it=tuneParameters.begin(); it!= tuneParameters.end(); ++it){
1125 Log() << kWARNING << it->first << " = " << it->second << Endl;
1126 if (it->first == "MaxDepth" ) SetMaxDepth ((Int_t)it->second);
1127 else if (it->first == "MinNodeSize" ) SetMinNodeSize (it->second);
1128 else if (it->first == "NTrees" ) SetNTrees ((Int_t)it->second);
1129 else if (it->first == "NodePurityLimit") SetNodePurityLimit (it->second);
1130 else if (it->first == "AdaBoostBeta" ) SetAdaBoostBeta (it->second);
1131 else if (it->first == "Shrinkage" ) SetShrinkage (it->second);
1132 else if (it->first == "UseNvars" ) SetUseNvars ((Int_t)it->second);
1133 else if (it->first == "BaggedSampleFraction" ) SetBaggedSampleFraction (it->second);
1134 else Log() << kFATAL << " SetParameter for " << it->first << " not yet implemented " <<Endl;
1135 }
1136}
1137
1138////////////////////////////////////////////////////////////////////////////////
1139/// BDT training.
1140
1141
1143{
1145
1146 // fill the STL Vector with the event sample
1147 // (needs to be done here and cannot be done in "init" as the options need to be
1148 // known).
1149 InitEventSample();
1150
1151 if (fNTrees==0){
1152 Log() << kERROR << " Zero Decision Trees demanded... that does not work !! "
1153 << " I set it to 1 .. just so that the program does not crash"
1154 << Endl;
1155 fNTrees = 1;
1156 }
1157
1158 if (fInteractive && fInteractive->NotInitialized()){
1159 std::vector<TString> titles = {"Boost weight", "Error Fraction"};
1160 fInteractive->Init(titles);
1161 }
1162 fIPyMaxIter = fNTrees;
1163 fExitFromTraining = false;
1164
1165 // HHV (it's been here since looong but I really don't know why we cannot handle
1166 // normalized variables in BDTs... todo
1167 if (IsNormalised()) Log() << kFATAL << "\"Normalise\" option cannot be used with BDT; "
1168 << "please remove the option from the configuration string, or "
1169 << "use \"!Normalise\""
1170 << Endl;
1171
1172 if(DoRegression())
1173 Log() << kINFO << "Regression Loss Function: "<< fRegressionLossFunctionBDTG->Name() << Endl;
1174
1175 Log() << kINFO << "Training "<< fNTrees << " Decision Trees ... patience please" << Endl;
1176
1177 Log() << kDEBUG << "Training with maximal depth = " <<fMaxDepth
1178 << ", MinNodeEvents=" << fMinNodeEvents
1179 << ", NTrees="<<fNTrees
1180 << ", NodePurityLimit="<<fNodePurityLimit
1181 << ", AdaBoostBeta="<<fAdaBoostBeta
1182 << Endl;
1183
1184 // weights applied in boosting
1185 Int_t nBins;
1186 Double_t xMin,xMax;
1187 TString hname = "AdaBooost weight distribution";
1188
1189 nBins= 100;
1190 xMin = 0;
1191 xMax = 30;
1192
1193 if (DoRegression()) {
1194 nBins= 100;
1195 xMin = 0;
1196 xMax = 1;
1197 hname="Boost event weights distribution";
1198 }
1199
1200 // book monitoring histograms (for AdaBost only)
1201
1202 TH1* h = new TH1F(Form("%s_BoostWeight",DataInfo().GetName()),hname,nBins,xMin,xMax);
1203 TH1* nodesBeforePruningVsTree = new TH1I(Form("%s_NodesBeforePruning",DataInfo().GetName()),"nodes before pruning",fNTrees,0,fNTrees);
1204 TH1* nodesAfterPruningVsTree = new TH1I(Form("%s_NodesAfterPruning",DataInfo().GetName()),"nodes after pruning",fNTrees,0,fNTrees);
1205
1206
1207
1208 if(!DoMulticlass()){
1209 Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, GetAnalysisType());
1210
1211 h->SetXTitle("boost weight");
1212 results->Store(h, "BoostWeights");
1213
1214
1215 // Monitor the performance (on TEST sample) versus number of trees
1216 if (fDoBoostMonitor){
1217 TH2* boostMonitor = new TH2F("BoostMonitor","ROC Integral Vs iTree",2,0,fNTrees,2,0,1.05);
1218 boostMonitor->SetXTitle("#tree");
1219 boostMonitor->SetYTitle("ROC Integral");
1220 results->Store(boostMonitor, "BoostMonitor");
1221 TGraph *boostMonitorGraph = new TGraph();
1222 boostMonitorGraph->SetName("BoostMonitorGraph");
1223 boostMonitorGraph->SetTitle("ROCIntegralVsNTrees");
1224 results->Store(boostMonitorGraph, "BoostMonitorGraph");
1225 }
1226
1227 // weights applied in boosting vs tree number
1228 h = new TH1F("BoostWeightVsTree","Boost weights vs tree",fNTrees,0,fNTrees);
1229 h->SetXTitle("#tree");
1230 h->SetYTitle("boost weight");
1231 results->Store(h, "BoostWeightsVsTree");
1232
1233 // error fraction vs tree number
1234 h = new TH1F("ErrFractHist","error fraction vs tree number",fNTrees,0,fNTrees);
1235 h->SetXTitle("#tree");
1236 h->SetYTitle("error fraction");
1237 results->Store(h, "ErrorFrac");
1238
1239 // nNodesBeforePruning vs tree number
1240 nodesBeforePruningVsTree->SetXTitle("#tree");
1241 nodesBeforePruningVsTree->SetYTitle("#tree nodes");
1242 results->Store(nodesBeforePruningVsTree);
1243
1244 // nNodesAfterPruning vs tree number
1245 nodesAfterPruningVsTree->SetXTitle("#tree");
1246 nodesAfterPruningVsTree->SetYTitle("#tree nodes");
1247 results->Store(nodesAfterPruningVsTree);
1248
1249 }
1250
1251 fMonitorNtuple= new TTree("MonitorNtuple","BDT variables");
1252 fMonitorNtuple->Branch("iTree",&fITree,"iTree/I");
1253 fMonitorNtuple->Branch("boostWeight",&fBoostWeight,"boostWeight/D");
1254 fMonitorNtuple->Branch("errorFraction",&fErrorFraction,"errorFraction/D");
1255
1256 Timer timer( fNTrees, GetName() );
1257 Int_t nNodesBeforePruningCount = 0;
1258 Int_t nNodesAfterPruningCount = 0;
1259
1260 Int_t nNodesBeforePruning = 0;
1261 Int_t nNodesAfterPruning = 0;
1262
1263 if(fBoostType=="Grad"){
1264 InitGradBoost(fEventSample);
1265 }
1266
1267 Int_t itree=0;
1268 Bool_t continueBoost=kTRUE;
1269 //for (int itree=0; itree<fNTrees; itree++) {
1270
1271 while (itree < fNTrees && continueBoost){
1272 if (fExitFromTraining) break;
1273 fIPyCurrentIter = itree;
1274 timer.DrawProgressBar( itree );
1275 // Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, GetAnalysisType());
1276 // TH1 *hxx = new TH1F(Form("swdist%d",itree),Form("swdist%d",itree),10000,0,15);
1277 // results->Store(hxx,Form("swdist%d",itree));
1278 // TH1 *hxy = new TH1F(Form("bwdist%d",itree),Form("bwdist%d",itree),10000,0,15);
1279 // results->Store(hxy,Form("bwdist%d",itree));
1280 // for (Int_t iev=0; iev<fEventSample.size(); iev++) {
1281 // if (fEventSample[iev]->GetClass()!=0) hxy->Fill((fEventSample[iev])->GetWeight());
1282 // else hxx->Fill((fEventSample[iev])->GetWeight());
1283 // }
1284
1285 if(DoMulticlass()){
1286 if (fBoostType!="Grad"){
1287 Log() << kFATAL << "Multiclass is currently only supported by gradient boost. "
1288 << "Please change boost option accordingly (BoostType=Grad)." << Endl;
1289 }
1290
1291 UInt_t nClasses = DataInfo().GetNClasses();
1292 for (UInt_t i=0;i<nClasses;i++){
1293 // Careful: If fSepType is nullptr, the tree will be considered a regression tree and
1294 // use the correct output for gradboost (response rather than yesnoleaf) in checkEvent.
1295 // See TMVA::MethodBDT::InitGradBoost.
1296 fForest.push_back( new DecisionTree( fSepType, fMinNodeSize, fNCuts, &(DataInfo()), i,
1297 fRandomisedTrees, fUseNvars, fUsePoissonNvars, fMaxDepth,
1298 itree*nClasses+i, fNodePurityLimit, itree*nClasses+1));
1299 fForest.back()->SetNVars(GetNvar());
1300 if (fUseFisherCuts) {
1301 fForest.back()->SetUseFisherCuts();
1302 fForest.back()->SetMinLinCorrForFisher(fMinLinCorrForFisher);
1303 fForest.back()->SetUseExclusiveVars(fUseExclusiveVars);
1304 }
1305 // the minimum linear correlation between two variables demanded for use in fisher criterion in node splitting
1306
1307 nNodesBeforePruning = fForest.back()->BuildTree(*fTrainSample);
1308 Double_t bw = this->Boost(*fTrainSample, fForest.back(),i);
1309 if (bw > 0) {
1310 fBoostWeights.push_back(bw);
1311 }else{
1312 fBoostWeights.push_back(0);
1313 Log() << kWARNING << "stopped boosting at itree="<<itree << Endl;
1314 // fNTrees = itree+1; // that should stop the boosting
1315 continueBoost=kFALSE;
1316 }
1317 }
1318 }
1319 else{
1320
1321 DecisionTree* dt = new DecisionTree( fSepType, fMinNodeSize, fNCuts, &(DataInfo()), fSignalClass,
1322 fRandomisedTrees, fUseNvars, fUsePoissonNvars, fMaxDepth,
1323 itree, fNodePurityLimit, itree);
1324
1325 fForest.push_back(dt);
1326 fForest.back()->SetNVars(GetNvar());
1327 if (fUseFisherCuts) {
1328 fForest.back()->SetUseFisherCuts();
1329 fForest.back()->SetMinLinCorrForFisher(fMinLinCorrForFisher);
1330 fForest.back()->SetUseExclusiveVars(fUseExclusiveVars);
1331 }
1332
1333 nNodesBeforePruning = fForest.back()->BuildTree(*fTrainSample);
1334
1335 if (fUseYesNoLeaf && !DoRegression() && fBoostType!="Grad") { // remove leaf nodes where both daughter nodes are of same type
1336 nNodesBeforePruning = fForest.back()->CleanTree();
1337 }
1338
1339 nNodesBeforePruningCount += nNodesBeforePruning;
1340 nodesBeforePruningVsTree->SetBinContent(itree+1,nNodesBeforePruning);
1341
1342 fForest.back()->SetPruneMethod(fPruneMethod); // set the pruning method for the tree
1343 fForest.back()->SetPruneStrength(fPruneStrength); // set the strength parameter
1344
1345 std::vector<const Event*> * validationSample = NULL;
1346 if(fAutomatic) validationSample = &fValidationSample;
1347 Double_t bw = this->Boost(*fTrainSample, fForest.back());
1348 if (bw > 0) {
1349 fBoostWeights.push_back(bw);
1350 }else{
1351 fBoostWeights.push_back(0);
1352 Log() << kWARNING << "stopped boosting at itree="<<itree << Endl;
1353 continueBoost=kFALSE;
1354 }
1355
1356 // if fAutomatic == true, pruneStrength will be the optimal pruning strength
1357 // determined by the pruning algorithm; otherwise, it is simply the strength parameter
1358 // set by the user
1359 if (fPruneMethod != DecisionTree::kNoPruning) fForest.back()->PruneTree(validationSample);
1360
1361 if (fUseYesNoLeaf && !DoRegression() && fBoostType!="Grad"){ // remove leaf nodes where both daughter nodes are of same type
1362 fForest.back()->CleanTree();
1363 }
1364 nNodesAfterPruning = fForest.back()->GetNNodes();
1365 nNodesAfterPruningCount += nNodesAfterPruning;
1366 nodesAfterPruningVsTree->SetBinContent(itree+1,nNodesAfterPruning);
1367
1368 if (fInteractive){
1369 fInteractive->AddPoint(itree, fBoostWeight, fErrorFraction);
1370 }
1371 fITree = itree;
1372 fMonitorNtuple->Fill();
1373 if (fDoBoostMonitor){
1374 if (! DoRegression() ){
1375 if ( itree==fNTrees-1 || (!(itree%500)) ||
1376 (!(itree%250) && itree <1000)||
1377 (!(itree%100) && itree < 500)||
1378 (!(itree%50) && itree < 250)||
1379 (!(itree%25) && itree < 150)||
1380 (!(itree%10) && itree < 50)||
1381 (!(itree%5) && itree < 20)
1382 ) BoostMonitor(itree);
1383 }
1384 }
1385 }
1386 itree++;
1387 }
1388
1389 // get elapsed time
1390 Log() << kDEBUG << "\t<Train> elapsed time: " << timer.GetElapsedTime()
1391 << " " << Endl;
1392 if (fPruneMethod == DecisionTree::kNoPruning) {
1393 Log() << kDEBUG << "\t<Train> average number of nodes (w/o pruning) : "
1394 << nNodesBeforePruningCount/GetNTrees() << Endl;
1395 }
1396 else {
1397 Log() << kDEBUG << "\t<Train> average number of nodes before/after pruning : "
1398 << nNodesBeforePruningCount/GetNTrees() << " / "
1399 << nNodesAfterPruningCount/GetNTrees()
1400 << Endl;
1401 }
1403
1404
1405 // reset all previously stored/accumulated BOOST weights in the event sample
1406 // for (UInt_t iev=0; iev<fEventSample.size(); iev++) fEventSample[iev]->SetBoostWeight(1.);
1407 Log() << kDEBUG << "Now I delete the privat data sample"<< Endl;
1408 for (UInt_t i=0; i<fEventSample.size(); i++) delete fEventSample[i];
1409 for (UInt_t i=0; i<fValidationSample.size(); i++) delete fValidationSample[i];
1410 fEventSample.clear();
1411 fValidationSample.clear();
1412
1413 if (!fExitFromTraining) fIPyMaxIter = fIPyCurrentIter;
1414 ExitFromTraining();
1415}
1416
1417
1418////////////////////////////////////////////////////////////////////////////////
1419/// Returns MVA value: -1 for background, 1 for signal.
1420
1422{
1423 Double_t sum=0;
1424 for (UInt_t itree=0; itree<nTrees; itree++) {
1425 //loop over all trees in forest
1426 sum += fForest[itree]->CheckEvent(e,kFALSE);
1427
1428 }
1429 return 2.0/(1.0+exp(-2.0*sum))-1; //MVA output between -1 and 1
1430}
1431
1432////////////////////////////////////////////////////////////////////////////////
1433/// Calculate residual for all events.
1434
1435void TMVA::MethodBDT::UpdateTargets(std::vector<const TMVA::Event*>& eventSample, UInt_t cls)
1436{
1437 if (DoMulticlass()) {
1438 UInt_t nClasses = DataInfo().GetNClasses();
1439 Bool_t isLastClass = (cls == nClasses - 1);
1440
1441 #ifdef R__USE_IMT
1442 //
1443 // This is the multi-threaded multiclass version
1444 //
1445 // Note: we only need to update the predicted probabilities every
1446 // `nClasses` tree. Let's call a set of `nClasses` trees a "round". Thus
1447 // the algortihm is split in two parts `update_residuals` and
1448 // `update_residuals_last` where the latter is inteded to be run instead
1449 // of the former for the last tree in a "round".
1450 //
1451 std::map<const TMVA::Event *, std::vector<double>> & residuals = this->fResiduals;
1452 DecisionTree & lastTree = *(this->fForest.back());
1453
1454 auto update_residuals = [&residuals, &lastTree, cls](const TMVA::Event * e) {
1455 residuals[e].at(cls) += lastTree.CheckEvent(e, kFALSE);
1456 };
1457
1458 auto update_residuals_last = [&residuals, &lastTree, cls, nClasses](const TMVA::Event * e) {
1459 residuals[e].at(cls) += lastTree.CheckEvent(e, kFALSE);
1460
1461 auto &residualsThisEvent = residuals[e];
1462
1463 std::vector<Double_t> expCache(nClasses, 0.0);
1464 std::transform(residualsThisEvent.begin(),
1465 residualsThisEvent.begin() + nClasses,
1466 expCache.begin(), [](Double_t d) { return exp(d); });
1467
1468 Double_t exp_sum = std::accumulate(expCache.begin(),
1469 expCache.begin() + nClasses,
1470 0.0);
1471
1472 for (UInt_t i = 0; i < nClasses; i++) {
1473 Double_t p_cls = expCache[i] / exp_sum;
1474
1475 Double_t res = (e->GetClass() == i) ? (1.0 - p_cls) : (-p_cls);
1476 const_cast<TMVA::Event *>(e)->SetTarget(i, res);
1477 }
1478 };
1479
1480 if (isLastClass) {
1482 .Foreach(update_residuals_last, eventSample);
1483 } else {
1485 .Foreach(update_residuals, eventSample);
1486 }
1487 #else
1488 //
1489 // Single-threaded multiclass version
1490 //
1491 std::vector<Double_t> expCache;
1492 if (isLastClass) {
1493 expCache.resize(nClasses);
1494 }
1495
1496 for (auto e : eventSample) {
1497 fResiduals[e].at(cls) += fForest.back()->CheckEvent(e, kFALSE);
1498 if (isLastClass) {
1499 auto &residualsThisEvent = fResiduals[e];
1500 std::transform(residualsThisEvent.begin(),
1501 residualsThisEvent.begin() + nClasses,
1502 expCache.begin(), [](Double_t d) { return exp(d); });
1503
1504 Double_t exp_sum = std::accumulate(expCache.begin(),
1505 expCache.begin() + nClasses,
1506 0.0);
1507
1508 for (UInt_t i = 0; i < nClasses; i++) {
1509 Double_t p_cls = expCache[i] / exp_sum;
1510
1511 Double_t res = (e->GetClass() == i) ? (1.0 - p_cls) : (-p_cls);
1512 const_cast<TMVA::Event *>(e)->SetTarget(i, res);
1513 }
1514 }
1515 }
1516 #endif
1517 } else {
1518 std::map<const TMVA::Event *, std::vector<double>> & residuals = this->fResiduals;
1519 DecisionTree & lastTree = *(this->fForest.back());
1520
1521 UInt_t signalClass = DataInfo().GetSignalClassIndex();
1522
1523 #ifdef R__USE_IMT
1524 auto update_residuals = [&residuals, &lastTree, signalClass](const TMVA::Event * e) {
1525 double & residualAt0 = residuals[e].at(0);
1526 residualAt0 += lastTree.CheckEvent(e, kFALSE);
1527
1528 Double_t p_sig = 1.0 / (1.0 + exp(-2.0 * residualAt0));
1529 Double_t res = ((e->GetClass() == signalClass) ? (1.0 - p_sig) : (-p_sig));
1530
1531 const_cast<TMVA::Event *>(e)->SetTarget(0, res);
1532 };
1533
1535 .Foreach(update_residuals, eventSample);
1536 #else
1537 for (auto e : eventSample) {
1538 double & residualAt0 = residuals[e].at(0);
1539 residualAt0 += lastTree.CheckEvent(e, kFALSE);
1540
1541 Double_t p_sig = 1.0 / (1.0 + exp(-2.0 * residualAt0));
1542 Double_t res = ((e->GetClass() == signalClass) ? (1.0 - p_sig) : (-p_sig));
1543
1544 const_cast<TMVA::Event *>(e)->SetTarget(0, res);
1545 }
1546 #endif
1547 }
1548}
1549
1550////////////////////////////////////////////////////////////////////////////////
1551/// \brief Calculate residuals for all events and update targets for next iter.
1552///
1553/// \param[in] eventSample The collection of events currently under training.
1554/// \param[in] first Should be true when called before the first boosting
1555/// iteration has been run
1556///
1557void TMVA::MethodBDT::UpdateTargetsRegression(std::vector<const TMVA::Event*>& eventSample, Bool_t first)
1558{
1559 if (!first) {
1560#ifdef R__USE_IMT
1562 auto seeds = ROOT::TSeqU(nPartitions);
1563
1564 // need a lambda function to pass to TThreadExecutor::MapReduce
1565 auto f = [this, &nPartitions](UInt_t partition = 0) -> Int_t {
1566 Int_t start = 1.0 * partition / nPartitions * this->fEventSample.size();
1567 Int_t end = (partition + 1.0) / nPartitions * this->fEventSample.size();
1568
1569 for (Int_t i = start; i < end; ++i) {
1570 const TMVA::Event *e = fEventSample[i];
1571 LossFunctionEventInfo & lossInfo = fLossFunctionEventInfo.at(e);
1572 lossInfo.predictedValue += fForest.back()->CheckEvent(e, kFALSE);
1573 }
1574
1575 return 0;
1576 };
1577
1579#else
1580 for (const TMVA::Event *e : fEventSample) {
1581 LossFunctionEventInfo & lossInfo = fLossFunctionEventInfo.at(e);
1582 lossInfo.predictedValue += fForest.back()->CheckEvent(e, kFALSE);
1583 }
1584#endif
1585 }
1586
1587 // NOTE: Set targets are also parallelised internally
1588 fRegressionLossFunctionBDTG->SetTargets(eventSample, fLossFunctionEventInfo);
1589
1590}
1591
1592////////////////////////////////////////////////////////////////////////////////
1593/// Calculate the desired response value for each region.
1594
1595Double_t TMVA::MethodBDT::GradBoost(std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt, UInt_t cls)
1596{
1597 struct LeafInfo {
1598 Double_t sumWeightTarget = 0;
1599 Double_t sum2 = 0;
1600 };
1601
1602 std::unordered_map<TMVA::DecisionTreeNode*, LeafInfo> leaves;
1603 for (auto e : eventSample) {
1604 Double_t weight = e->GetWeight();
1606 auto &v = leaves[node];
1607 auto target = e->GetTarget(cls);
1608 v.sumWeightTarget += target * weight;
1609 v.sum2 += fabs(target) * (1.0 - fabs(target)) * weight;
1610 }
1611 for (auto &iLeave : leaves) {
1612 constexpr auto minValue = 1e-30;
1613 if (iLeave.second.sum2 < minValue) {
1614 iLeave.second.sum2 = minValue;
1615 }
1616 const Double_t K = DataInfo().GetNClasses();
1617 iLeave.first->SetResponse(fShrinkage * (K - 1) / K * iLeave.second.sumWeightTarget / iLeave.second.sum2);
1618 }
1619
1620 //call UpdateTargets before next tree is grown
1621
1622 DoMulticlass() ? UpdateTargets(fEventSample, cls) : UpdateTargets(fEventSample);
1623 return 1; //trees all have the same weight
1624}
1625
1626////////////////////////////////////////////////////////////////////////////////
1627/// Implementation of M_TreeBoost using any loss function as described by Friedman 1999.
1628
1629Double_t TMVA::MethodBDT::GradBoostRegression(std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
1630{
1631 // get the vector of events for each terminal so that we can calculate the constant fit value in each
1632 // terminal node
1633 // #### Not sure how many events are in each node in advance, so I can't parallelize this easily
1634 std::map<TMVA::DecisionTreeNode*,vector< TMVA::LossFunctionEventInfo > > leaves;
1635 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1636 TMVA::DecisionTreeNode* node = dt->GetEventNode(*(*e));
1637 (leaves[node]).push_back(fLossFunctionEventInfo[*e]);
1638 }
1639
1640 // calculate the constant fit for each terminal node based upon the events in the node
1641 // node (iLeave->first), vector of event information (iLeave->second)
1642 // #### could parallelize this and do the leaves at the same time, but this doesn't take very long compared
1643 // #### to the other processes
1644 for (std::map<TMVA::DecisionTreeNode*,vector< TMVA::LossFunctionEventInfo > >::iterator iLeave=leaves.begin();
1645 iLeave!=leaves.end();++iLeave){
1646 Double_t fit = fRegressionLossFunctionBDTG->Fit(iLeave->second);
1647 (iLeave->first)->SetResponse(fShrinkage*fit);
1648 }
1649
1650 UpdateTargetsRegression(*fTrainSample);
1651
1652 return 1;
1653}
1654
1655////////////////////////////////////////////////////////////////////////////////
1656/// Initialize targets for first tree.
1657
1658void TMVA::MethodBDT::InitGradBoost( std::vector<const TMVA::Event*>& eventSample)
1659{
1660 // Should get rid of this line. It's just for debugging.
1661 //std::sort(eventSample.begin(), eventSample.end(), [](const TMVA::Event* a, const TMVA::Event* b){
1662 // return (a->GetTarget(0) < b->GetTarget(0)); });
1663 fSepType=NULL; //set fSepType to NULL (regression trees are used for both classification an regression)
1664 if(DoRegression()){
1665 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1666 fLossFunctionEventInfo[*e]= TMVA::LossFunctionEventInfo((*e)->GetTarget(0), 0, (*e)->GetWeight());
1667 }
1668
1669 fRegressionLossFunctionBDTG->Init(fLossFunctionEventInfo, fBoostWeights);
1670 UpdateTargetsRegression(*fTrainSample,kTRUE);
1671
1672 return;
1673 }
1674 else if(DoMulticlass()){
1675 UInt_t nClasses = DataInfo().GetNClasses();
1676 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1677 for (UInt_t i=0;i<nClasses;i++){
1678 //Calculate initial residua, assuming equal probability for all classes
1679 Double_t r = (*e)->GetClass()==i?(1-1.0/nClasses):(-1.0/nClasses);
1680 const_cast<TMVA::Event*>(*e)->SetTarget(i,r);
1681 fResiduals[*e].push_back(0);
1682 }
1683 }
1684 }
1685 else{
1686 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1687 Double_t r = (DataInfo().IsSignal(*e)?1:0)-0.5; //Calculate initial residua
1688 const_cast<TMVA::Event*>(*e)->SetTarget(0,r);
1689 fResiduals[*e].push_back(0);
1690 }
1691 }
1692
1693}
1694////////////////////////////////////////////////////////////////////////////////
1695/// Test the tree quality.. in terms of Misclassification.
1696
1698{
1699 Double_t ncorrect=0, nfalse=0;
1700 for (UInt_t ievt=0; ievt<fValidationSample.size(); ievt++) {
1701 Bool_t isSignalType= (dt->CheckEvent(fValidationSample[ievt]) > fNodePurityLimit ) ? 1 : 0;
1702
1703 if (isSignalType == (DataInfo().IsSignal(fValidationSample[ievt])) ) {
1704 ncorrect += fValidationSample[ievt]->GetWeight();
1705 }
1706 else{
1707 nfalse += fValidationSample[ievt]->GetWeight();
1708 }
1709 }
1710
1711 return ncorrect / (ncorrect + nfalse);
1712}
1713
1714////////////////////////////////////////////////////////////////////////////////
1715/// Apply the boosting algorithm (the algorithm is selecte via the "option" given
1716/// in the constructor. The return value is the boosting weight.
1717
1718Double_t TMVA::MethodBDT::Boost( std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt, UInt_t cls )
1719{
1720 Double_t returnVal=-1;
1721
1722 if (fBoostType=="AdaBoost") returnVal = this->AdaBoost (eventSample, dt);
1723 else if (fBoostType=="AdaCost") returnVal = this->AdaCost (eventSample, dt);
1724 else if (fBoostType=="Bagging") returnVal = this->Bagging ( );
1725 else if (fBoostType=="RegBoost") returnVal = this->RegBoost (eventSample, dt);
1726 else if (fBoostType=="AdaBoostR2") returnVal = this->AdaBoostR2(eventSample, dt);
1727 else if (fBoostType=="Grad"){
1728 if(DoRegression())
1729 returnVal = this->GradBoostRegression(eventSample, dt);
1730 else if(DoMulticlass())
1731 returnVal = this->GradBoost (eventSample, dt, cls);
1732 else
1733 returnVal = this->GradBoost (eventSample, dt);
1734 }
1735 else {
1736 Log() << kINFO << GetOptions() << Endl;
1737 Log() << kFATAL << "<Boost> unknown boost option " << fBoostType<< " called" << Endl;
1738 }
1739
1740 if (fBaggedBoost){
1741 GetBaggedSubSample(fEventSample);
1742 }
1743
1744
1745 return returnVal;
1746}
1747
1748////////////////////////////////////////////////////////////////////////////////
1749/// Fills the ROCIntegral vs Itree from the testSample for the monitoring plots
1750/// during the training .. but using the testing events
1751
1753{
1754 Results* results = Data()->GetResults(GetMethodName(),Types::kTraining, Types::kMaxAnalysisType);
1755
1756 TH1F *tmpS = new TH1F( "tmpS", "", 100 , -1., 1.00001 );
1757 TH1F *tmpB = new TH1F( "tmpB", "", 100 , -1., 1.00001 );
1758 TH1F *tmp;
1759
1760
1761 UInt_t signalClassNr = DataInfo().GetClassInfo("Signal")->GetNumber();
1762
1763 // const std::vector<Event*> events=Data()->GetEventCollection(Types::kTesting);
1764 // // fMethod->GetTransformationHandler().CalcTransformations(fMethod->Data()->GetEventCollection(Types::kTesting));
1765 // for (UInt_t iev=0; iev < events.size() ; iev++){
1766 // if (events[iev]->GetClass() == signalClassNr) tmp=tmpS;
1767 // else tmp=tmpB;
1768 // tmp->Fill(PrivateGetMvaValue(*(events[iev])),events[iev]->GetWeight());
1769 // }
1770
1771 UInt_t nevents = Data()->GetNTestEvents();
1772 for (UInt_t iev=0; iev < nevents; iev++){
1773 const Event* event = GetTestingEvent(iev);
1774
1775 if (event->GetClass() == signalClassNr) {tmp=tmpS;}
1776 else {tmp=tmpB;}
1777 tmp->Fill(PrivateGetMvaValue(event),event->GetWeight());
1778 }
1779 Double_t max=1;
1780
1781 std::vector<TH1F*> hS;
1782 std::vector<TH1F*> hB;
1783 for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
1784 hS.push_back(new TH1F(Form("SigVar%dAtTree%d",ivar,iTree),Form("SigVar%dAtTree%d",ivar,iTree),100,DataInfo().GetVariableInfo(ivar).GetMin(),DataInfo().GetVariableInfo(ivar).GetMax()));
1785 hB.push_back(new TH1F(Form("BkgVar%dAtTree%d",ivar,iTree),Form("BkgVar%dAtTree%d",ivar,iTree),100,DataInfo().GetVariableInfo(ivar).GetMin(),DataInfo().GetVariableInfo(ivar).GetMax()));
1786 results->Store(hS.back(),hS.back()->GetTitle());
1787 results->Store(hB.back(),hB.back()->GetTitle());
1788 }
1789
1790
1791 for (UInt_t iev=0; iev < fEventSample.size(); iev++){
1792 if (fEventSample[iev]->GetBoostWeight() > max) max = 1.01*fEventSample[iev]->GetBoostWeight();
1793 }
1794 TH1F *tmpBoostWeightsS = new TH1F(Form("BoostWeightsInTreeS%d",iTree),Form("BoostWeightsInTreeS%d",iTree),100,0.,max);
1795 TH1F *tmpBoostWeightsB = new TH1F(Form("BoostWeightsInTreeB%d",iTree),Form("BoostWeightsInTreeB%d",iTree),100,0.,max);
1796 results->Store(tmpBoostWeightsS,tmpBoostWeightsS->GetTitle());
1797 results->Store(tmpBoostWeightsB,tmpBoostWeightsB->GetTitle());
1798
1799 TH1F *tmpBoostWeights;
1800 std::vector<TH1F*> *h;
1801
1802 for (UInt_t iev=0; iev < fEventSample.size(); iev++){
1803 if (fEventSample[iev]->GetClass() == signalClassNr) {
1804 tmpBoostWeights=tmpBoostWeightsS;
1805 h=&hS;
1806 }else{
1807 tmpBoostWeights=tmpBoostWeightsB;
1808 h=&hB;
1809 }
1810 tmpBoostWeights->Fill(fEventSample[iev]->GetBoostWeight());
1811 for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
1812 (*h)[ivar]->Fill(fEventSample[iev]->GetValue(ivar),fEventSample[iev]->GetWeight());
1813 }
1814 }
1815
1816
1817 TMVA::PDF *sig = new TMVA::PDF( " PDF Sig", tmpS, TMVA::PDF::kSpline3 );
1818 TMVA::PDF *bkg = new TMVA::PDF( " PDF Bkg", tmpB, TMVA::PDF::kSpline3 );
1819
1820
1821 TGraph* gr=results->GetGraph("BoostMonitorGraph");
1822 Int_t nPoints = gr->GetN();
1823 gr->Set(nPoints+1);
1824 gr->SetPoint(nPoints,(Double_t)iTree+1,GetROCIntegral(sig,bkg));
1825
1826 tmpS->Delete();
1827 tmpB->Delete();
1828
1829 delete sig;
1830 delete bkg;
1831
1832 return;
1833}
1834
1835////////////////////////////////////////////////////////////////////////////////
1836/// The AdaBoost implementation.
1837/// a new training sample is generated by weighting
1838/// events that are misclassified by the decision tree. The weight
1839/// applied is \f$ w = \frac{(1-err)}{err} \f$ or more general:
1840/// \f$ w = (\frac{(1-err)}{err})^\beta \f$
1841/// where \f$err\f$ is the fraction of misclassified events in the tree ( <0.5 assuming
1842/// demanding the that previous selection was better than random guessing)
1843/// and "beta" being a free parameter (standard: beta = 1) that modifies the
1844/// boosting.
1845
1846Double_t TMVA::MethodBDT::AdaBoost( std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
1847{
1848 Double_t err=0, sumGlobalw=0, sumGlobalwfalse=0, sumGlobalwfalse2=0;
1849
1850 std::vector<Double_t> sumw(DataInfo().GetNClasses(),0); //for individually re-scaling each class
1851
1852 Double_t maxDev=0;
1853 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1854 Double_t w = (*e)->GetWeight();
1855 sumGlobalw += w;
1856 UInt_t iclass=(*e)->GetClass();
1857 sumw[iclass] += w;
1858
1859 if ( DoRegression() ) {
1860 Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
1861 sumGlobalwfalse += w * tmpDev;
1862 sumGlobalwfalse2 += w * tmpDev*tmpDev;
1863 if (tmpDev > maxDev) maxDev = tmpDev;
1864 }else{
1865
1866 if (fUseYesNoLeaf){
1867 Bool_t isSignalType = (dt->CheckEvent(*e,fUseYesNoLeaf) > fNodePurityLimit );
1868 if (!(isSignalType == DataInfo().IsSignal(*e))) {
1869 sumGlobalwfalse+= w;
1870 }
1871 }else{
1872 Double_t dtoutput = (dt->CheckEvent(*e,fUseYesNoLeaf) - 0.5)*2.;
1873 Int_t trueType;
1874 if (DataInfo().IsSignal(*e)) trueType = 1;
1875 else trueType = -1;
1876 sumGlobalwfalse+= w*trueType*dtoutput;
1877 }
1878 }
1879 }
1880
1881 err = sumGlobalwfalse/sumGlobalw ;
1882 if ( DoRegression() ) {
1883 //if quadratic loss:
1884 if (fAdaBoostR2Loss=="linear"){
1885 err = sumGlobalwfalse/maxDev/sumGlobalw ;
1886 }
1887 else if (fAdaBoostR2Loss=="quadratic"){
1888 err = sumGlobalwfalse2/maxDev/maxDev/sumGlobalw ;
1889 }
1890 else if (fAdaBoostR2Loss=="exponential"){
1891 err = 0;
1892 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1893 Double_t w = (*e)->GetWeight();
1894 Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
1895 err += w * (1 - exp (-tmpDev/maxDev)) / sumGlobalw;
1896 }
1897
1898 }
1899 else {
1900 Log() << kFATAL << " you've chosen a Loss type for Adaboost other than linear, quadratic or exponential "
1901 << " namely " << fAdaBoostR2Loss << "\n"
1902 << "and this is not implemented... a typo in the options ??" <<Endl;
1903 }
1904 }
1905
1906 Log() << kDEBUG << "BDT AdaBoos wrong/all: " << sumGlobalwfalse << "/" << sumGlobalw << Endl;
1907
1908
1909 Double_t newSumGlobalw=0;
1910 std::vector<Double_t> newSumw(sumw.size(),0);
1911
1912 Double_t boostWeight=1.;
1913 if (err >= 0.5 && fUseYesNoLeaf) { // sanity check ... should never happen as otherwise there is apparently
1914 // something odd with the assignment of the leaf nodes (rem: you use the training
1915 // events for this determination of the error rate)
1916 if (dt->GetNNodes() == 1){
1917 Log() << kERROR << " YOUR tree has only 1 Node... kind of a funny *tree*. I cannot "
1918 << "boost such a thing... if after 1 step the error rate is == 0.5"
1919 << Endl
1920 << "please check why this happens, maybe too many events per node requested ?"
1921 << Endl;
1922
1923 }else{
1924 Log() << kERROR << " The error rate in the BDT boosting is > 0.5. ("<< err
1925 << ") That should not happen, please check your code (i.e... the BDT code), I "
1926 << " stop boosting here" << Endl;
1927 return -1;
1928 }
1929 err = 0.5;
1930 } else if (err < 0) {
1931 Log() << kERROR << " The error rate in the BDT boosting is < 0. That can happen"
1932 << " due to improper treatment of negative weights in a Monte Carlo.. (if you have"
1933 << " an idea on how to do it in a better way, please let me know (Helge.Voss@cern.ch)"
1934 << " for the time being I set it to its absolute value.. just to continue.." << Endl;
1935 err = TMath::Abs(err);
1936 }
1937 if (fUseYesNoLeaf)
1938 boostWeight = TMath::Log((1.-err)/err)*fAdaBoostBeta;
1939 else
1940 boostWeight = TMath::Log((1.+err)/(1-err))*fAdaBoostBeta;
1941
1942
1943 Log() << kDEBUG << "BDT AdaBoos wrong/all: " << sumGlobalwfalse << "/" << sumGlobalw << " 1-err/err="<<boostWeight<< " log.."<<TMath::Log(boostWeight)<<Endl;
1944
1945 Results* results = Data()->GetResults(GetMethodName(),Types::kTraining, Types::kMaxAnalysisType);
1946
1947
1948 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1949
1950 if (fUseYesNoLeaf||DoRegression()){
1951 if ((!( (dt->CheckEvent(*e,fUseYesNoLeaf) > fNodePurityLimit ) == DataInfo().IsSignal(*e))) || DoRegression()) {
1952 Double_t boostfactor = TMath::Exp(boostWeight);
1953
1954 if (DoRegression()) boostfactor = TMath::Power(1/boostWeight,(1.-TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) )/maxDev ) );
1955 if ( (*e)->GetWeight() > 0 ){
1956 (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1957 // Helge change back (*e)->ScaleBoostWeight(boostfactor);
1958 if (DoRegression()) results->GetHist("BoostWeights")->Fill(boostfactor);
1959 } else {
1960 if ( fInverseBoostNegWeights )(*e)->ScaleBoostWeight( 1. / boostfactor); // if the original event weight is negative, and you want to "increase" the events "positive" influence, you'd rather make the event weight "smaller" in terms of it's absolute value while still keeping it something "negative"
1961 else (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1962
1963 }
1964 }
1965
1966 }else{
1967 Double_t dtoutput = (dt->CheckEvent(*e,fUseYesNoLeaf) - 0.5)*2.;
1968 Int_t trueType;
1969 if (DataInfo().IsSignal(*e)) trueType = 1;
1970 else trueType = -1;
1971 Double_t boostfactor = TMath::Exp(-1*boostWeight*trueType*dtoutput);
1972
1973 if ( (*e)->GetWeight() > 0 ){
1974 (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1975 // Helge change back (*e)->ScaleBoostWeight(boostfactor);
1976 if (DoRegression()) results->GetHist("BoostWeights")->Fill(boostfactor);
1977 } else {
1978 if ( fInverseBoostNegWeights )(*e)->ScaleBoostWeight( 1. / boostfactor); // if the original event weight is negative, and you want to "increase" the events "positive" influence, you'd rather make the event weight "smaller" in terms of it's absolute value while still keeping it something "negative"
1979 else (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1980 }
1981 }
1982 newSumGlobalw+=(*e)->GetWeight();
1983 newSumw[(*e)->GetClass()] += (*e)->GetWeight();
1984 }
1985
1986
1987 // Double_t globalNormWeight=sumGlobalw/newSumGlobalw;
1988 Double_t globalNormWeight=( (Double_t) eventSample.size())/newSumGlobalw;
1989 Log() << kDEBUG << "new Nsig="<<newSumw[0]*globalNormWeight << " new Nbkg="<<newSumw[1]*globalNormWeight << Endl;
1990
1991
1992 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1993 // if (fRenormByClass) (*e)->ScaleBoostWeight( normWeightByClass[(*e)->GetClass()] );
1994 // else (*e)->ScaleBoostWeight( globalNormWeight );
1995 // else (*e)->ScaleBoostWeight( globalNormWeight );
1996 if (DataInfo().IsSignal(*e))(*e)->ScaleBoostWeight( globalNormWeight * fSigToBkgFraction );
1997 else (*e)->ScaleBoostWeight( globalNormWeight );
1998 }
1999
2000 if (!(DoRegression()))results->GetHist("BoostWeights")->Fill(boostWeight);
2001 results->GetHist("BoostWeightsVsTree")->SetBinContent(fForest.size(),boostWeight);
2002 results->GetHist("ErrorFrac")->SetBinContent(fForest.size(),err);
2003
2004 fBoostWeight = boostWeight;
2005 fErrorFraction = err;
2006
2007 return boostWeight;
2008}
2009
2010////////////////////////////////////////////////////////////////////////////////
2011/// The AdaCost boosting algorithm takes a simple cost Matrix (currently fixed for
2012/// all events... later could be modified to use individual cost matrices for each
2013/// events as in the original paper...
2014///
2015/// true_signal true_bkg
2016/// ----------------------------------
2017/// sel_signal | Css Ctb_ss Cxx.. in the range [0,1]
2018/// sel_bkg | Cts_sb Cbb
2019///
2020/// and takes this into account when calculating the mis class. cost (former: error fraction):
2021///
2022/// err = sum_events ( weight* y_true*y_sel * beta(event)
2023
2024Double_t TMVA::MethodBDT::AdaCost( vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
2025{
2026 Double_t Css = fCss;
2027 Double_t Cbb = fCbb;
2028 Double_t Cts_sb = fCts_sb;
2029 Double_t Ctb_ss = fCtb_ss;
2030
2031 Double_t err=0, sumGlobalWeights=0, sumGlobalCost=0;
2032
2033 std::vector<Double_t> sumw(DataInfo().GetNClasses(),0); //for individually re-scaling each class
2034
2035 for (vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2036 Double_t w = (*e)->GetWeight();
2037 sumGlobalWeights += w;
2038 UInt_t iclass=(*e)->GetClass();
2039
2040 sumw[iclass] += w;
2041
2042 if ( DoRegression() ) {
2043 Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2044 }else{
2045
2046 Double_t dtoutput = (dt->CheckEvent(*e,false) - 0.5)*2.;
2047 Int_t trueType;
2048 Bool_t isTrueSignal = DataInfo().IsSignal(*e);
2049 Bool_t isSelectedSignal = (dtoutput>0);
2050 if (isTrueSignal) trueType = 1;
2051 else trueType = -1;
2052
2053 Double_t cost=0;
2054 if (isTrueSignal && isSelectedSignal) cost=Css;
2055 else if (isTrueSignal && !isSelectedSignal) cost=Cts_sb;
2056 else if (!isTrueSignal && isSelectedSignal) cost=Ctb_ss;
2057 else if (!isTrueSignal && !isSelectedSignal) cost=Cbb;
2058 else Log() << kERROR << "something went wrong in AdaCost" << Endl;
2059
2060 sumGlobalCost+= w*trueType*dtoutput*cost;
2061
2062 }
2063 }
2064
2065 if ( DoRegression() ) {
2066 Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2067 }
2068
2069 // Log() << kDEBUG << "BDT AdaBoos wrong/all: " << sumGlobalCost << "/" << sumGlobalWeights << Endl;
2070 // Log() << kWARNING << "BDT AdaBoos wrong/all: " << sumGlobalCost << "/" << sumGlobalWeights << Endl;
2071 sumGlobalCost /= sumGlobalWeights;
2072 // Log() << kWARNING << "BDT AdaBoos wrong/all: " << sumGlobalCost << "/" << sumGlobalWeights << Endl;
2073
2074
2075 Double_t newSumGlobalWeights=0;
2076 vector<Double_t> newSumClassWeights(sumw.size(),0);
2077
2078 Double_t boostWeight = TMath::Log((1+sumGlobalCost)/(1-sumGlobalCost)) * fAdaBoostBeta;
2079
2080 Results* results = Data()->GetResults(GetMethodName(),Types::kTraining, Types::kMaxAnalysisType);
2081
2082 for (vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2083 Double_t dtoutput = (dt->CheckEvent(*e,false) - 0.5)*2.;
2084 Int_t trueType;
2085 Bool_t isTrueSignal = DataInfo().IsSignal(*e);
2086 Bool_t isSelectedSignal = (dtoutput>0);
2087 if (isTrueSignal) trueType = 1;
2088 else trueType = -1;
2089
2090 Double_t cost=0;
2091 if (isTrueSignal && isSelectedSignal) cost=Css;
2092 else if (isTrueSignal && !isSelectedSignal) cost=Cts_sb;
2093 else if (!isTrueSignal && isSelectedSignal) cost=Ctb_ss;
2094 else if (!isTrueSignal && !isSelectedSignal) cost=Cbb;
2095 else Log() << kERROR << "something went wrong in AdaCost" << Endl;
2096
2097 Double_t boostfactor = TMath::Exp(-1*boostWeight*trueType*dtoutput*cost);
2098 if (DoRegression())Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2099 if ( (*e)->GetWeight() > 0 ){
2100 (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
2101 // Helge change back (*e)->ScaleBoostWeight(boostfactor);
2102 if (DoRegression())Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2103 } else {
2104 if ( fInverseBoostNegWeights )(*e)->ScaleBoostWeight( 1. / boostfactor); // if the original event weight is negative, and you want to "increase" the events "positive" influence, you'd rather make the event weight "smaller" in terms of it's absolute value while still keeping it something "negative"
2105 }
2106
2107 newSumGlobalWeights+=(*e)->GetWeight();
2108 newSumClassWeights[(*e)->GetClass()] += (*e)->GetWeight();
2109 }
2110
2111
2112 // Double_t globalNormWeight=sumGlobalWeights/newSumGlobalWeights;
2113 Double_t globalNormWeight=Double_t(eventSample.size())/newSumGlobalWeights;
2114 Log() << kDEBUG << "new Nsig="<<newSumClassWeights[0]*globalNormWeight << " new Nbkg="<<newSumClassWeights[1]*globalNormWeight << Endl;
2115
2116
2117 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2118 // if (fRenormByClass) (*e)->ScaleBoostWeight( normWeightByClass[(*e)->GetClass()] );
2119 // else (*e)->ScaleBoostWeight( globalNormWeight );
2120 if (DataInfo().IsSignal(*e))(*e)->ScaleBoostWeight( globalNormWeight * fSigToBkgFraction );
2121 else (*e)->ScaleBoostWeight( globalNormWeight );
2122 }
2123
2124
2125 if (!(DoRegression()))results->GetHist("BoostWeights")->Fill(boostWeight);
2126 results->GetHist("BoostWeightsVsTree")->SetBinContent(fForest.size(),boostWeight);
2127 results->GetHist("ErrorFrac")->SetBinContent(fForest.size(),err);
2128
2129 fBoostWeight = boostWeight;
2130 fErrorFraction = err;
2131
2132
2133 return boostWeight;
2134}
2135
2136////////////////////////////////////////////////////////////////////////////////
2137/// Call it boot-strapping, re-sampling or whatever you like, in the end it is nothing
2138/// else but applying "random" poisson weights to each event.
2139
2141{
2142 // this is now done in "MethodBDT::Boost as it might be used by other boost methods, too
2143 // GetBaggedSample(eventSample);
2144
2145 return 1.; //here as there are random weights for each event, just return a constant==1;
2146}
2147
2148////////////////////////////////////////////////////////////////////////////////
2149/// Fills fEventSample with fBaggedSampleFraction*NEvents random training events.
2150
2151void TMVA::MethodBDT::GetBaggedSubSample(std::vector<const TMVA::Event*>& eventSample)
2152{
2153
2154 Double_t n;
2155 TRandom3 *trandom = new TRandom3(100*fForest.size()+1234);
2156
2157 if (!fSubSample.empty()) fSubSample.clear();
2158
2159 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2160 n = trandom->PoissonD(fBaggedSampleFraction);
2161 for (Int_t i=0;i<n;i++) fSubSample.push_back(*e);
2162 }
2163
2164 delete trandom;
2165 return;
2166
2167 /*
2168 UInt_t nevents = fEventSample.size();
2169
2170 if (!fSubSample.empty()) fSubSample.clear();
2171 TRandom3 *trandom = new TRandom3(fForest.size()+1);
2172
2173 for (UInt_t ievt=0; ievt<nevents; ievt++) { // recreate new random subsample
2174 if(trandom->Rndm()<fBaggedSampleFraction)
2175 fSubSample.push_back(fEventSample[ievt]);
2176 }
2177 delete trandom;
2178 */
2179
2180}
2181
2182////////////////////////////////////////////////////////////////////////////////
2183/// A special boosting only for Regression (not implemented).
2184
2185Double_t TMVA::MethodBDT::RegBoost( std::vector<const TMVA::Event*>& /* eventSample */, DecisionTree* /* dt */ )
2186{
2187 return 1;
2188}
2189
2190////////////////////////////////////////////////////////////////////////////////
2191/// Adaption of the AdaBoost to regression problems (see H.Drucker 1997).
2192
2193Double_t TMVA::MethodBDT::AdaBoostR2( std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
2194{
2195 if ( !DoRegression() ) Log() << kFATAL << "Somehow you chose a regression boost method for a classification job" << Endl;
2196
2197 Double_t err=0, sumw=0, sumwfalse=0, sumwfalse2=0;
2198 Double_t maxDev=0;
2199 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2200 Double_t w = (*e)->GetWeight();
2201 sumw += w;
2202
2203 Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
2204 sumwfalse += w * tmpDev;
2205 sumwfalse2 += w * tmpDev*tmpDev;
2206 if (tmpDev > maxDev) maxDev = tmpDev;
2207 }
2208
2209 //if quadratic loss:
2210 if (fAdaBoostR2Loss=="linear"){
2211 err = sumwfalse/maxDev/sumw ;
2212 }
2213 else if (fAdaBoostR2Loss=="quadratic"){
2214 err = sumwfalse2/maxDev/maxDev/sumw ;
2215 }
2216 else if (fAdaBoostR2Loss=="exponential"){
2217 err = 0;
2218 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2219 Double_t w = (*e)->GetWeight();
2220 Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
2221 err += w * (1 - exp (-tmpDev/maxDev)) / sumw;
2222 }
2223
2224 }
2225 else {
2226 Log() << kFATAL << " you've chosen a Loss type for Adaboost other than linear, quadratic or exponential "
2227 << " namely " << fAdaBoostR2Loss << "\n"
2228 << "and this is not implemented... a typo in the options ??" <<Endl;
2229 }
2230
2231
2232 if (err >= 0.5) { // sanity check ... should never happen as otherwise there is apparently
2233 // something odd with the assignment of the leaf nodes (rem: you use the training
2234 // events for this determination of the error rate)
2235 if (dt->GetNNodes() == 1){
2236 Log() << kERROR << " YOUR tree has only 1 Node... kind of a funny *tree*. I cannot "
2237 << "boost such a thing... if after 1 step the error rate is == 0.5"
2238 << Endl
2239 << "please check why this happens, maybe too many events per node requested ?"
2240 << Endl;
2241
2242 }else{
2243 Log() << kERROR << " The error rate in the BDT boosting is > 0.5. ("<< err
2244 << ") That should not happen, but is possible for regression trees, and"
2245 << " should trigger a stop for the boosting. please check your code (i.e... the BDT code), I "
2246 << " stop boosting " << Endl;
2247 return -1;
2248 }
2249 err = 0.5;
2250 } else if (err < 0) {
2251 Log() << kERROR << " The error rate in the BDT boosting is < 0. That can happen"
2252 << " due to improper treatment of negative weights in a Monte Carlo.. (if you have"
2253 << " an idea on how to do it in a better way, please let me know (Helge.Voss@cern.ch)"
2254 << " for the time being I set it to its absolute value.. just to continue.." << Endl;
2255 err = TMath::Abs(err);
2256 }
2257
2258 Double_t boostWeight = err / (1.-err);
2259 Double_t newSumw=0;
2260
2261 Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, Types::kMaxAnalysisType);
2262
2263 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2264 Double_t boostfactor = TMath::Power(boostWeight,(1.-TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) )/maxDev ) );
2265 results->GetHist("BoostWeights")->Fill(boostfactor);
2266 // std::cout << "R2 " << boostfactor << " " << boostWeight << " " << (1.-TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) )/maxDev) << std::endl;
2267 if ( (*e)->GetWeight() > 0 ){
2268 Float_t newBoostWeight = (*e)->GetBoostWeight() * boostfactor;
2269 Float_t newWeight = (*e)->GetWeight() * (*e)->GetBoostWeight() * boostfactor;
2270 if (newWeight == 0) {
2271 Log() << kINFO << "Weight= " << (*e)->GetWeight() << Endl;
2272 Log() << kINFO << "BoostWeight= " << (*e)->GetBoostWeight() << Endl;
2273 Log() << kINFO << "boostweight="<<boostWeight << " err= " <<err << Endl;
2274 Log() << kINFO << "NewBoostWeight= " << newBoostWeight << Endl;
2275 Log() << kINFO << "boostfactor= " << boostfactor << Endl;
2276 Log() << kINFO << "maxDev = " << maxDev << Endl;
2277 Log() << kINFO << "tmpDev = " << TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) ) << Endl;
2278 Log() << kINFO << "target = " << (*e)->GetTarget(0) << Endl;
2279 Log() << kINFO << "estimate = " << dt->CheckEvent(*e,kFALSE) << Endl;
2280 }
2281 (*e)->SetBoostWeight( newBoostWeight );
2282 // (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
2283 } else {
2284 (*e)->SetBoostWeight( (*e)->GetBoostWeight() / boostfactor);
2285 }
2286 newSumw+=(*e)->GetWeight();
2287 }
2288
2289 // re-normalise the weights
2290 Double_t normWeight = sumw / newSumw;
2291 for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2292 //Helge (*e)->ScaleBoostWeight( sumw/newSumw);
2293 // (*e)->ScaleBoostWeight( normWeight);
2294 (*e)->SetBoostWeight( (*e)->GetBoostWeight() * normWeight );
2295 }
2296
2297
2298 results->GetHist("BoostWeightsVsTree")->SetBinContent(fForest.size(),1./boostWeight);
2299 results->GetHist("ErrorFrac")->SetBinContent(fForest.size(),err);
2300
2301 fBoostWeight = boostWeight;
2302 fErrorFraction = err;
2303
2304 return TMath::Log(1./boostWeight);
2305}
2306
2307////////////////////////////////////////////////////////////////////////////////
2308/// Write weights to XML.
2309
2310void TMVA::MethodBDT::AddWeightsXMLTo( void* parent ) const
2311{
2312 void* wght = gTools().AddChild(parent, "Weights");
2313
2314 if (fDoPreselection){
2315 for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
2316 gTools().AddAttr( wght, Form("PreselectionLowBkgVar%d",ivar), fIsLowBkgCut[ivar]);
2317 gTools().AddAttr( wght, Form("PreselectionLowBkgVar%dValue",ivar), fLowBkgCut[ivar]);
2318 gTools().AddAttr( wght, Form("PreselectionLowSigVar%d",ivar), fIsLowSigCut[ivar]);
2319 gTools().AddAttr( wght, Form("PreselectionLowSigVar%dValue",ivar), fLowSigCut[ivar]);
2320 gTools().AddAttr( wght, Form("PreselectionHighBkgVar%d",ivar), fIsHighBkgCut[ivar]);
2321 gTools().AddAttr( wght, Form("PreselectionHighBkgVar%dValue",ivar),fHighBkgCut[ivar]);
2322 gTools().AddAttr( wght, Form("PreselectionHighSigVar%d",ivar), fIsHighSigCut[ivar]);
2323 gTools().AddAttr( wght, Form("PreselectionHighSigVar%dValue",ivar),fHighSigCut[ivar]);
2324 }
2325 }
2326
2327
2328 gTools().AddAttr( wght, "NTrees", fForest.size() );
2329 gTools().AddAttr( wght, "AnalysisType", fForest.back()->GetAnalysisType() );
2330
2331 for (UInt_t i=0; i< fForest.size(); i++) {
2332 void* trxml = fForest[i]->AddXMLTo(wght);
2333 gTools().AddAttr( trxml, "boostWeight", fBoostWeights[i] );
2334 gTools().AddAttr( trxml, "itree", i );
2335 }
2336}
2337
2338////////////////////////////////////////////////////////////////////////////////
2339/// Reads the BDT from the xml file.
2340
2342 UInt_t i;
2343 for (i=0; i<fForest.size(); i++) delete fForest[i];
2344 fForest.clear();
2345 fBoostWeights.clear();
2346
2347 UInt_t ntrees;
2348 UInt_t analysisType;
2349 Float_t boostWeight;
2350
2351
2352 if (gTools().HasAttr( parent, Form("PreselectionLowBkgVar%d",0))) {
2353 fIsLowBkgCut.resize(GetNvar());
2354 fLowBkgCut.resize(GetNvar());
2355 fIsLowSigCut.resize(GetNvar());
2356 fLowSigCut.resize(GetNvar());
2357 fIsHighBkgCut.resize(GetNvar());
2358 fHighBkgCut.resize(GetNvar());
2359 fIsHighSigCut.resize(GetNvar());
2360 fHighSigCut.resize(GetNvar());
2361
2362 Bool_t tmpBool;
2363 Double_t tmpDouble;
2364 for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
2365 gTools().ReadAttr( parent, Form("PreselectionLowBkgVar%d",ivar), tmpBool);
2366 fIsLowBkgCut[ivar]=tmpBool;
2367 gTools().ReadAttr( parent, Form("PreselectionLowBkgVar%dValue",ivar), tmpDouble);
2368 fLowBkgCut[ivar]=tmpDouble;
2369 gTools().ReadAttr( parent, Form("PreselectionLowSigVar%d",ivar), tmpBool);
2370 fIsLowSigCut[ivar]=tmpBool;
2371 gTools().ReadAttr( parent, Form("PreselectionLowSigVar%dValue",ivar), tmpDouble);
2372 fLowSigCut[ivar]=tmpDouble;
2373 gTools().ReadAttr( parent, Form("PreselectionHighBkgVar%d",ivar), tmpBool);
2374 fIsHighBkgCut[ivar]=tmpBool;
2375 gTools().ReadAttr( parent, Form("PreselectionHighBkgVar%dValue",ivar), tmpDouble);
2376 fHighBkgCut[ivar]=tmpDouble;
2377 gTools().ReadAttr( parent, Form("PreselectionHighSigVar%d",ivar),tmpBool);
2378 fIsHighSigCut[ivar]=tmpBool;
2379 gTools().ReadAttr( parent, Form("PreselectionHighSigVar%dValue",ivar), tmpDouble);
2380 fHighSigCut[ivar]=tmpDouble;
2381 }
2382 }
2383
2384
2385 gTools().ReadAttr( parent, "NTrees", ntrees );
2386
2387 if(gTools().HasAttr(parent, "TreeType")) { // pre 4.1.0 version
2388 gTools().ReadAttr( parent, "TreeType", analysisType );
2389 } else { // from 4.1.0 onwards
2390 gTools().ReadAttr( parent, "AnalysisType", analysisType );
2391 }
2392
2393 void* ch = gTools().GetChild(parent);
2394 i=0;
2395 while(ch) {
2396 fForest.push_back( dynamic_cast<DecisionTree*>( DecisionTree::CreateFromXML(ch, GetTrainingTMVAVersionCode()) ) );
2397 fForest.back()->SetAnalysisType(Types::EAnalysisType(analysisType));
2398 fForest.back()->SetTreeID(i++);
2399 gTools().ReadAttr(ch,"boostWeight",boostWeight);
2400 fBoostWeights.push_back(boostWeight);
2401 ch = gTools().GetNextChild(ch);
2402 }
2403}
2404
2405////////////////////////////////////////////////////////////////////////////////
2406/// Read the weights (BDT coefficients).
2407
2409{
2410 TString dummy;
2411 // Types::EAnalysisType analysisType;
2412 Int_t analysisType(0);
2413
2414 // coverity[tainted_data_argument]
2415 istr >> dummy >> fNTrees;
2416 Log() << kINFO << "Read " << fNTrees << " Decision trees" << Endl;
2417
2418 for (UInt_t i=0;i<fForest.size();i++) delete fForest[i];
2419 fForest.clear();
2420 fBoostWeights.clear();
2421 Int_t iTree;
2422 Double_t boostWeight;
2423 for (int i=0;i<fNTrees;i++) {
2424 istr >> dummy >> iTree >> dummy >> boostWeight;
2425 if (iTree != i) {
2426 fForest.back()->Print( std::cout );
2427 Log() << kFATAL << "Error while reading weight file; mismatch iTree="
2428 << iTree << " i=" << i
2429 << " dummy " << dummy
2430 << " boostweight " << boostWeight
2431 << Endl;
2432 }
2433 fForest.push_back( new DecisionTree() );
2434 fForest.back()->SetAnalysisType(Types::EAnalysisType(analysisType));
2435 fForest.back()->SetTreeID(i);
2436 fForest.back()->Read(istr, GetTrainingTMVAVersionCode());
2437 fBoostWeights.push_back(boostWeight);
2438 }
2439}
2440
2441////////////////////////////////////////////////////////////////////////////////
2442
2444 return this->GetMvaValue( err, errUpper, 0 );
2445}
2446
2447////////////////////////////////////////////////////////////////////////////////
2448/// Return the MVA value (range [-1;1]) that classifies the
2449/// event according to the majority vote from the total number of
2450/// decision trees.
2451
2453{
2454 const Event* ev = GetEvent();
2455 if (fDoPreselection) {
2456 Double_t val = ApplyPreselectionCuts(ev);
2457 if (TMath::Abs(val)>0.05) return val;
2458 }
2459 return PrivateGetMvaValue(ev, err, errUpper, useNTrees);
2460
2461}
2462
2463////////////////////////////////////////////////////////////////////////////////
2464/// Return the MVA value (range [-1;1]) that classifies the
2465/// event according to the majority vote from the total number of
2466/// decision trees.
2467
2469{
2470 // cannot determine error
2471 NoErrorCalc(err, errUpper);
2472
2473 // allow for the possibility to use less trees in the actual MVA calculation
2474 // than have been originally trained.
2475 UInt_t nTrees = fForest.size();
2476
2477 if (useNTrees > 0 ) nTrees = useNTrees;
2478
2479 if (fBoostType=="Grad") return GetGradBoostMVA(ev,nTrees);
2480
2481 Double_t myMVA = 0;
2482 Double_t norm = 0;
2483 for (UInt_t itree=0; itree<nTrees; itree++) {
2484 //
2485 myMVA += fBoostWeights[itree] * fForest[itree]->CheckEvent(ev,fUseYesNoLeaf);
2486 norm += fBoostWeights[itree];
2487 }
2488 return ( norm > std::numeric_limits<double>::epsilon() ) ? myMVA /= norm : 0 ;
2489}
2490
2491
2492////////////////////////////////////////////////////////////////////////////////
2493/// Get the multiclass MVA response for the BDT classifier.
2494
2495const std::vector<Float_t>& TMVA::MethodBDT::GetMulticlassValues()
2496{
2497 const TMVA::Event *e = GetEvent();
2498 if (fMulticlassReturnVal == NULL) fMulticlassReturnVal = new std::vector<Float_t>();
2499 fMulticlassReturnVal->clear();
2500
2501 UInt_t nClasses = DataInfo().GetNClasses();
2502 std::vector<Double_t> temp(nClasses);
2503 auto forestSize = fForest.size();
2504
2505 #ifdef R__USE_IMT
2506 std::vector<TMVA::DecisionTree *> forest = fForest;
2507 auto get_output = [&e, &forest, &temp, forestSize, nClasses](UInt_t iClass) {
2508 for (UInt_t itree = iClass; itree < forestSize; itree += nClasses) {
2509 temp[iClass] += forest[itree]->CheckEvent(e, kFALSE);
2510 }
2511 };
2512
2514 .Foreach(get_output, ROOT::TSeqU(nClasses));
2515 #else
2516 // trees 0, nClasses, 2*nClasses, ... belong to class 0
2517 // trees 1, nClasses+1, 2*nClasses+1, ... belong to class 1 and so forth
2518 UInt_t classOfTree = 0;
2519 for (UInt_t itree = 0; itree < forestSize; ++itree) {
2520 temp[classOfTree] += fForest[itree]->CheckEvent(e, kFALSE);
2521 if (++classOfTree == nClasses) classOfTree = 0; // cheap modulo
2522 }
2523 #endif
2524
2525 // we want to calculate sum of exp(temp[j] - temp[i]) for all i,j (i!=j)
2526 // first calculate exp(), then replace minus with division.
2527 std::transform(temp.begin(), temp.end(), temp.begin(), [](Double_t d){return exp(d);});
2528
2529 Double_t exp_sum = std::accumulate(temp.begin(), temp.end(), 0.0);
2530
2531 for (UInt_t i = 0; i < nClasses; i++) {
2532 Double_t p_cls = temp[i] / exp_sum;
2533 (*fMulticlassReturnVal).push_back(p_cls);
2534 }
2535
2536 return *fMulticlassReturnVal;
2537}
2538
2539////////////////////////////////////////////////////////////////////////////////
2540/// Get the regression value generated by the BDTs.
2541
2542const std::vector<Float_t> & TMVA::MethodBDT::GetRegressionValues()
2543{
2544
2545 if (fRegressionReturnVal == NULL) fRegressionReturnVal = new std::vector<Float_t>();
2546 fRegressionReturnVal->clear();
2547
2548 const Event * ev = GetEvent();
2549 Event * evT = new Event(*ev);
2550
2551 Double_t myMVA = 0;
2552 Double_t norm = 0;
2553 if (fBoostType=="AdaBoostR2") {
2554 // rather than using the weighted average of the tree respones in the forest
2555 // H.Decker(1997) proposed to use the "weighted median"
2556
2557 // sort all individual tree responses according to the prediction value
2558 // (keep the association to their tree weight)
2559 // the sum up all the associated weights (starting from the one whose tree
2560 // yielded the smalles response) up to the tree "t" at which you've
2561 // added enough tree weights to have more than half of the sum of all tree weights.
2562 // choose as response of the forest that one which belongs to this "t"
2563
2564 vector< Double_t > response(fForest.size());
2565 vector< Double_t > weight(fForest.size());
2566 Double_t totalSumOfWeights = 0;
2567
2568 for (UInt_t itree=0; itree<fForest.size(); itree++) {
2569 response[itree] = fForest[itree]->CheckEvent(ev,kFALSE);
2570 weight[itree] = fBoostWeights[itree];
2571 totalSumOfWeights += fBoostWeights[itree];
2572 }
2573
2574 std::vector< std::vector<Double_t> > vtemp;
2575 vtemp.push_back( response ); // this is the vector that will get sorted
2576 vtemp.push_back( weight );
2577 gTools().UsefulSortAscending( vtemp );
2578
2579 Int_t t=0;
2580 Double_t sumOfWeights = 0;
2581 while (sumOfWeights <= totalSumOfWeights/2.) {
2582 sumOfWeights += vtemp[1][t];
2583 t++;
2584 }
2585
2586 Double_t rVal=0;
2587 Int_t count=0;
2588 for (UInt_t i= TMath::Max(UInt_t(0),UInt_t(t-(fForest.size()/6)-0.5));
2589 i< TMath::Min(UInt_t(fForest.size()),UInt_t(t+(fForest.size()/6)+0.5)); i++) {
2590 count++;
2591 rVal+=vtemp[0][i];
2592 }
2593 // fRegressionReturnVal->push_back( rVal/Double_t(count));
2594 evT->SetTarget(0, rVal/Double_t(count) );
2595 }
2596 else if(fBoostType=="Grad"){
2597 for (UInt_t itree=0; itree<fForest.size(); itree++) {
2598 myMVA += fForest[itree]->CheckEvent(ev,kFALSE);
2599 }
2600 // fRegressionReturnVal->push_back( myMVA+fBoostWeights[0]);
2601 evT->SetTarget(0, myMVA+fBoostWeights[0] );
2602 }
2603 else{
2604 for (UInt_t itree=0; itree<fForest.size(); itree++) {
2605 //
2606 myMVA += fBoostWeights[itree] * fForest[itree]->CheckEvent(ev,kFALSE);
2607 norm += fBoostWeights[itree];
2608 }
2609 // fRegressionReturnVal->push_back( ( norm > std::numeric_limits<double>::epsilon() ) ? myMVA /= norm : 0 );
2610 evT->SetTarget(0, ( norm > std::numeric_limits<double>::epsilon() ) ? myMVA /= norm : 0 );
2611 }
2612
2613
2614
2615 const Event* evT2 = GetTransformationHandler().InverseTransform( evT );
2616 fRegressionReturnVal->push_back( evT2->GetTarget(0) );
2617
2618 delete evT;
2619
2620
2621 return *fRegressionReturnVal;
2622}
2623
2624////////////////////////////////////////////////////////////////////////////////
2625/// Here we could write some histograms created during the processing
2626/// to the output file.
2627
2629{
2630 Log() << kDEBUG << "\tWrite monitoring histograms to file: " << BaseDir()->GetPath() << Endl;
2631
2632 //Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, Types::kMaxAnalysisType);
2633 //results->GetStorage()->Write();
2634 fMonitorNtuple->Write();
2635}
2636
2637////////////////////////////////////////////////////////////////////////////////
2638/// Return the relative variable importance, normalized to all
2639/// variables together having the importance 1. The importance in
2640/// evaluated as the total separation-gain that this variable had in
2641/// the decision trees (weighted by the number of events)
2642
2644{
2645 fVariableImportance.resize(GetNvar());
2646 for (UInt_t ivar = 0; ivar < GetNvar(); ivar++) {
2647 fVariableImportance[ivar]=0;
2648 }
2649 Double_t sum=0;
2650 for (UInt_t itree = 0; itree < GetNTrees(); itree++) {
2651 std::vector<Double_t> relativeImportance(fForest[itree]->GetVariableImportance());
2652 for (UInt_t i=0; i< relativeImportance.size(); i++) {
2653 fVariableImportance[i] += fBoostWeights[itree] * relativeImportance[i];
2654 }
2655 }
2656
2657 for (UInt_t ivar=0; ivar< fVariableImportance.size(); ivar++){
2658 fVariableImportance[ivar] = TMath::Sqrt(fVariableImportance[ivar]);
2659 sum += fVariableImportance[ivar];
2660 }
2661 for (UInt_t ivar=0; ivar< fVariableImportance.size(); ivar++) fVariableImportance[ivar] /= sum;
2662
2663 return fVariableImportance;
2664}
2665
2666////////////////////////////////////////////////////////////////////////////////
2667/// Returns the measure for the variable importance of variable "ivar"
2668/// which is later used in GetVariableImportance() to calculate the
2669/// relative variable importances.
2670
2672{
2673 std::vector<Double_t> relativeImportance = this->GetVariableImportance();
2674 if (ivar < (UInt_t)relativeImportance.size()) return relativeImportance[ivar];
2675 else Log() << kFATAL << "<GetVariableImportance> ivar = " << ivar << " is out of range " << Endl;
2676
2677 return -1;
2678}
2679
2680////////////////////////////////////////////////////////////////////////////////
2681/// Compute ranking of input variables
2682
2684{
2685 // create the ranking object
2686 fRanking = new Ranking( GetName(), "Variable Importance" );
2687 vector< Double_t> importance(this->GetVariableImportance());
2688
2689 for (UInt_t ivar=0; ivar<GetNvar(); ivar++) {
2690
2691 fRanking->AddRank( Rank( GetInputLabel(ivar), importance[ivar] ) );
2692 }
2693
2694 return fRanking;
2695}
2696
2697////////////////////////////////////////////////////////////////////////////////
2698/// Get help message text.
2699
2701{
2702 Log() << Endl;
2703 Log() << gTools().Color("bold") << "--- Short description:" << gTools().Color("reset") << Endl;
2704 Log() << Endl;
2705 Log() << "Boosted Decision Trees are a collection of individual decision" << Endl;
2706 Log() << "trees which form a multivariate classifier by (weighted) majority " << Endl;
2707 Log() << "vote of the individual trees. Consecutive decision trees are " << Endl;
2708 Log() << "trained using the original training data set with re-weighted " << Endl;
2709 Log() << "events. By default, the AdaBoost method is employed, which gives " << Endl;
2710 Log() << "events that were misclassified in the previous tree a larger " << Endl;
2711 Log() << "weight in the training of the following tree." << Endl;
2712 Log() << Endl;
2713 Log() << "Decision trees are a sequence of binary splits of the data sample" << Endl;
2714 Log() << "using a single discriminant variable at a time. A test event " << Endl;
2715 Log() << "ending up after the sequence of left-right splits in a final " << Endl;
2716 Log() << "(\"leaf\") node is classified as either signal or background" << Endl;
2717 Log() << "depending on the majority type of training events in that node." << Endl;
2718 Log() << Endl;
2719 Log() << gTools().Color("bold") << "--- Performance optimisation:" << gTools().Color("reset") << Endl;
2720 Log() << Endl;
2721 Log() << "By the nature of the binary splits performed on the individual" << Endl;
2722 Log() << "variables, decision trees do not deal well with linear correlations" << Endl;
2723 Log() << "between variables (they need to approximate the linear split in" << Endl;
2724 Log() << "the two dimensional space by a sequence of splits on the two " << Endl;
2725 Log() << "variables individually). Hence decorrelation could be useful " << Endl;
2726 Log() << "to optimise the BDT performance." << Endl;
2727 Log() << Endl;
2728 Log() << gTools().Color("bold") << "--- Performance tuning via configuration options:" << gTools().Color("reset") << Endl;
2729 Log() << Endl;
2730 Log() << "The two most important parameters in the configuration are the " << Endl;
2731 Log() << "minimal number of events requested by a leaf node as percentage of the " <<Endl;
2732 Log() << " number of training events (option \"MinNodeSize\" replacing the actual number " << Endl;
2733 Log() << " of events \"nEventsMin\" as given in earlier versions" << Endl;
2734 Log() << "If this number is too large, detailed features " << Endl;
2735 Log() << "in the parameter space are hard to be modelled. If it is too small, " << Endl;
2736 Log() << "the risk to overtrain rises and boosting seems to be less effective" << Endl;
2737 Log() << " typical values from our current experience for best performance " << Endl;
2738 Log() << " are between 0.5(%) and 10(%) " << Endl;
2739 Log() << Endl;
2740 Log() << "The default minimal number is currently set to " << Endl;
2741 Log() << " max(20, (N_training_events / N_variables^2 / 10)) " << Endl;
2742 Log() << "and can be changed by the user." << Endl;
2743 Log() << Endl;
2744 Log() << "The other crucial parameter, the pruning strength (\"PruneStrength\")," << Endl;
2745 Log() << "is also related to overtraining. It is a regularisation parameter " << Endl;
2746 Log() << "that is used when determining after the training which splits " << Endl;
2747 Log() << "are considered statistically insignificant and are removed. The" << Endl;
2748 Log() << "user is advised to carefully watch the BDT screen output for" << Endl;
2749 Log() << "the comparison between efficiencies obtained on the training and" << Endl;
2750 Log() << "the independent test sample. They should be equal within statistical" << Endl;
2751 Log() << "errors, in order to minimize statistical fluctuations in different samples." << Endl;
2752}
2753
2754////////////////////////////////////////////////////////////////////////////////
2755/// Make ROOT-independent C++ class for classifier response (classifier-specific implementation).
2756
2757void TMVA::MethodBDT::MakeClassSpecific( std::ostream& fout, const TString& className ) const
2758{
2759 TString nodeName = className;
2760 nodeName.ReplaceAll("Read","");
2761 nodeName.Append("Node");
2762 // write BDT-specific classifier response
2763 fout << " std::vector<"<<nodeName<<"*> fForest; // i.e. root nodes of decision trees" << std::endl;
2764 fout << " std::vector<double> fBoostWeights; // the weights applied in the individual boosts" << std::endl;
2765 fout << "};" << std::endl << std::endl;
2766
2767 if(GetAnalysisType() == Types::kMulticlass) {
2768 fout << "std::vector<double> ReadBDTG::GetMulticlassValues__( const std::vector<double>& inputValues ) const" << std::endl;
2769 fout << "{" << std::endl;
2770 fout << " uint nClasses = " << DataInfo().GetNClasses() << ";" << std::endl;
2771 fout << " std::vector<double> fMulticlassReturnVal;" << std::endl;
2772 fout << " fMulticlassReturnVal.reserve(nClasses);" << std::endl;
2773 fout << std::endl;
2774 fout << " std::vector<double> temp(nClasses);" << std::endl;
2775 fout << " auto forestSize = fForest.size();" << std::endl;
2776 fout << " // trees 0, nClasses, 2*nClasses, ... belong to class 0" << std::endl;
2777 fout << " // trees 1, nClasses+1, 2*nClasses+1, ... belong to class 1 and so forth" << std::endl;
2778 fout << " uint classOfTree = 0;" << std::endl;
2779 fout << " for (uint itree = 0; itree < forestSize; ++itree) {" << std::endl;
2780 fout << " BDTGNode *current = fForest[itree];" << std::endl;
2781 fout << " while (current->GetNodeType() == 0) { //intermediate node" << std::endl;
2782 fout << " if (current->GoesRight(inputValues)) current=(BDTGNode*)current->GetRight();" << std::endl;
2783 fout << " else current=(BDTGNode*)current->GetLeft();" << std::endl;
2784 fout << " }" << std::endl;
2785 fout << " temp[classOfTree] += current->GetResponse();" << std::endl;
2786 fout << " if (++classOfTree == nClasses) classOfTree = 0; // cheap modulo" << std::endl;
2787 fout << " }" << std::endl;
2788 fout << std::endl;
2789 fout << " // we want to calculate sum of exp(temp[j] - temp[i]) for all i,j (i!=j)" << std::endl;
2790 fout << " // first calculate exp(), then replace minus with division." << std::endl;
2791 fout << " std::transform(temp.begin(), temp.end(), temp.begin(), [](double d){return exp(d);});" << std::endl;
2792 fout << std::endl;
2793 fout << " for(uint iClass=0; iClass<nClasses; iClass++){" << std::endl;
2794 fout << " double norm = 0.0;" << std::endl;
2795 fout << " for(uint j=0;j<nClasses;j++){" << std::endl;
2796 fout << " if(iClass!=j)" << std::endl;
2797 fout << " norm += temp[j] / temp[iClass];" << std::endl;
2798 fout << " }" << std::endl;
2799 fout << " fMulticlassReturnVal.push_back(1.0/(1.0+norm));" << std::endl;
2800 fout << " }" << std::endl;
2801 fout << std::endl;
2802 fout << " return fMulticlassReturnVal;" << std::endl;
2803 fout << "}" << std::endl;
2804 } else {
2805 fout << "double " << className << "::GetMvaValue__( const std::vector<double>& inputValues ) const" << std::endl;
2806 fout << "{" << std::endl;
2807 fout << " double myMVA = 0;" << std::endl;
2808 if (fDoPreselection){
2809 for (UInt_t ivar = 0; ivar< fIsLowBkgCut.size(); ivar++){
2810 if (fIsLowBkgCut[ivar]){
2811 fout << " if (inputValues["<<ivar<<"] < " << fLowBkgCut[ivar] << ") return -1; // is background preselection cut" << std::endl;
2812 }
2813 if (fIsLowSigCut[ivar]){
2814 fout << " if (inputValues["<<ivar<<"] < "<< fLowSigCut[ivar] << ") return 1; // is signal preselection cut" << std::endl;
2815 }
2816 if (fIsHighBkgCut[ivar]){
2817 fout << " if (inputValues["<<ivar<<"] > "<<fHighBkgCut[ivar] <<") return -1; // is background preselection cut" << std::endl;
2818 }
2819 if (fIsHighSigCut[ivar]){
2820 fout << " if (inputValues["<<ivar<<"] > "<<fHighSigCut[ivar]<<") return 1; // is signal preselection cut" << std::endl;
2821 }
2822 }
2823 }
2824
2825 if (fBoostType!="Grad"){
2826 fout << " double norm = 0;" << std::endl;
2827 }
2828 fout << " for (unsigned int itree=0; itree<fForest.size(); itree++){" << std::endl;
2829 fout << " "<<nodeName<<" *current = fForest[itree];" << std::endl;
2830 fout << " while (current->GetNodeType() == 0) { //intermediate node" << std::endl;
2831 fout << " if (current->GoesRight(inputValues)) current=("<<nodeName<<"*)current->GetRight();" << std::endl;
2832 fout << " else current=("<<nodeName<<"*)current->GetLeft();" << std::endl;
2833 fout << " }" << std::endl;
2834 if (fBoostType=="Grad"){
2835 fout << " myMVA += current->GetResponse();" << std::endl;
2836 }else{
2837 if (fUseYesNoLeaf) fout << " myMVA += fBoostWeights[itree] * current->GetNodeType();" << std::endl;
2838 else fout << " myMVA += fBoostWeights[itree] * current->GetPurity();" << std::endl;
2839 fout << " norm += fBoostWeights[itree];" << std::endl;
2840 }
2841 fout << " }" << std::endl;
2842 if (fBoostType=="Grad"){
2843 fout << " return 2.0/(1.0+exp(-2.0*myMVA))-1.0;" << std::endl;
2844 }
2845 else fout << " return myMVA /= norm;" << std::endl;
2846 fout << "}" << std::endl << std::endl;
2847 }
2848
2849 fout << "void " << className << "::Initialize()" << std::endl;
2850 fout << "{" << std::endl;
2851 fout << " double inf = std::numeric_limits<double>::infinity();" << std::endl;
2852 fout << " double nan = std::numeric_limits<double>::quiet_NaN();" << std::endl;
2853 //Now for each decision tree, write directly the constructors of the nodes in the tree structure
2854 for (UInt_t itree=0; itree<GetNTrees(); itree++) {
2855 fout << " // itree = " << itree << std::endl;
2856 fout << " fBoostWeights.push_back(" << fBoostWeights[itree] << ");" << std::endl;
2857 fout << " fForest.push_back( " << std::endl;
2858 this->MakeClassInstantiateNode((DecisionTreeNode*)fForest[itree]->GetRoot(), fout, className);
2859 fout <<" );" << std::endl;
2860 }
2861 fout << " return;" << std::endl;
2862 fout << "};" << std::endl;
2863 fout << std::endl;
2864 fout << "// Clean up" << std::endl;
2865 fout << "inline void " << className << "::Clear() " << std::endl;
2866 fout << "{" << std::endl;
2867 fout << " for (unsigned int itree=0; itree<fForest.size(); itree++) { " << std::endl;
2868 fout << " delete fForest[itree]; " << std::endl;
2869 fout << " }" << std::endl;
2870 fout << "}" << std::endl;
2871 fout << std::endl;
2872}
2873
2874////////////////////////////////////////////////////////////////////////////////
2875/// Specific class header.
2876
2877void TMVA::MethodBDT::MakeClassSpecificHeader( std::ostream& fout, const TString& className) const
2878{
2879 TString nodeName = className;
2880 nodeName.ReplaceAll("Read","");
2881 nodeName.Append("Node");
2882 fout << "#include <algorithm>" << std::endl;
2883 fout << "#include <limits>" << std::endl;
2884 fout << std::endl;
2885 //fout << "#ifndef NN" << std::endl; commented out on purpose see next line
2886 fout << "#define NN new "<<nodeName << std::endl; // NN definition depends on individual methods. Important to have NO #ifndef if several BDT methods compile together
2887 //fout << "#endif" << std::endl; commented out on purpose see previous line
2888 fout << std::endl;
2889 fout << "#ifndef "<<nodeName<<"__def" << std::endl;
2890 fout << "#define "<<nodeName<<"__def" << std::endl;
2891 fout << std::endl;
2892 fout << "class "<<nodeName<<" {" << std::endl;
2893 fout << std::endl;
2894 fout << "public:" << std::endl;
2895 fout << std::endl;
2896 fout << " // constructor of an essentially \"empty\" node floating in space" << std::endl;
2897 fout << " "<<nodeName<<" ( "<<nodeName<<"* left,"<<nodeName<<"* right," << std::endl;
2898 if (fUseFisherCuts){
2899 fout << " int nFisherCoeff," << std::endl;
2900 for (UInt_t i=0;i<GetNVariables()+1;i++){
2901 fout << " double fisherCoeff"<<i<<"," << std::endl;
2902 }
2903 }
2904 fout << " int selector, double cutValue, bool cutType, " << std::endl;
2905 fout << " int nodeType, double purity, double response ) :" << std::endl;
2906 fout << " fLeft ( left )," << std::endl;
2907 fout << " fRight ( right )," << std::endl;
2908 if (fUseFisherCuts) fout << " fNFisherCoeff ( nFisherCoeff )," << std::endl;
2909 fout << " fSelector ( selector )," << std::endl;
2910 fout << " fCutValue ( cutValue )," << std::endl;
2911 fout << " fCutType ( cutType )," << std::endl;
2912 fout << " fNodeType ( nodeType )," << std::endl;
2913 fout << " fPurity ( purity )," << std::endl;
2914 fout << " fResponse ( response ){" << std::endl;
2915 if (fUseFisherCuts){
2916 for (UInt_t i=0;i<GetNVariables()+1;i++){
2917 fout << " fFisherCoeff.push_back(fisherCoeff"<<i<<");" << std::endl;
2918 }
2919 }
2920 fout << " }" << std::endl << std::endl;
2921 fout << " virtual ~"<<nodeName<<"();" << std::endl << std::endl;
2922 fout << " // test event if it descends the tree at this node to the right" << std::endl;
2923 fout << " virtual bool GoesRight( const std::vector<double>& inputValues ) const;" << std::endl;
2924 fout << " "<<nodeName<<"* GetRight( void ) {return fRight; };" << std::endl << std::endl;
2925 fout << " // test event if it descends the tree at this node to the left " << std::endl;
2926 fout << " virtual bool GoesLeft ( const std::vector<double>& inputValues ) const;" << std::endl;
2927 fout << " "<<nodeName<<"* GetLeft( void ) { return fLeft; }; " << std::endl << std::endl;
2928 fout << " // return S/(S+B) (purity) at this node (from training)" << std::endl << std::endl;
2929 fout << " double GetPurity( void ) const { return fPurity; } " << std::endl;
2930 fout << " // return the node type" << std::endl;
2931 fout << " int GetNodeType( void ) const { return fNodeType; }" << std::endl;
2932 fout << " double GetResponse(void) const {return fResponse;}" << std::endl << std::endl;
2933 fout << "private:" << std::endl << std::endl;
2934 fout << " "<<nodeName<<"* fLeft; // pointer to the left daughter node" << std::endl;
2935 fout << " "<<nodeName<<"* fRight; // pointer to the right daughter node" << std::endl;
2936 if (fUseFisherCuts){
2937 fout << " int fNFisherCoeff; // =0 if this node doesn't use fisher, else =nvar+1 " << std::endl;
2938 fout << " std::vector<double> fFisherCoeff; // the fisher coeff (offset at the last element)" << std::endl;
2939 }
2940 fout << " int fSelector; // index of variable used in node selection (decision tree) " << std::endl;
2941 fout << " double fCutValue; // cut value applied on this node to discriminate bkg against sig" << std::endl;
2942 fout << " bool fCutType; // true: if event variable > cutValue ==> signal , false otherwise" << std::endl;
2943 fout << " int fNodeType; // Type of node: -1 == Bkg-leaf, 1 == Signal-leaf, 0 = internal " << std::endl;
2944 fout << " double fPurity; // Purity of node from training"<< std::endl;
2945 fout << " double fResponse; // Regression response value of node" << std::endl;
2946 fout << "}; " << std::endl;
2947 fout << std::endl;
2948 fout << "//_______________________________________________________________________" << std::endl;
2949 fout << " "<<nodeName<<"::~"<<nodeName<<"()" << std::endl;
2950 fout << "{" << std::endl;
2951 fout << " if (fLeft != NULL) delete fLeft;" << std::endl;
2952 fout << " if (fRight != NULL) delete fRight;" << std::endl;
2953 fout << "}; " << std::endl;
2954 fout << std::endl;
2955 fout << "//_______________________________________________________________________" << std::endl;
2956 fout << "bool "<<nodeName<<"::GoesRight( const std::vector<double>& inputValues ) const" << std::endl;
2957 fout << "{" << std::endl;
2958 fout << " // test event if it descends the tree at this node to the right" << std::endl;
2959 fout << " bool result;" << std::endl;
2960 if (fUseFisherCuts){
2961 fout << " if (fNFisherCoeff == 0){" << std::endl;
2962 fout << " result = (inputValues[fSelector] >= fCutValue );" << std::endl;
2963 fout << " }else{" << std::endl;
2964 fout << " double fisher = fFisherCoeff.at(fFisherCoeff.size()-1);" << std::endl;
2965 fout << " for (unsigned int ivar=0; ivar<fFisherCoeff.size()-1; ivar++)" << std::endl;
2966 fout << " fisher += fFisherCoeff.at(ivar)*inputValues.at(ivar);" << std::endl;
2967 fout << " result = fisher > fCutValue;" << std::endl;
2968 fout << " }" << std::endl;
2969 }else{
2970 fout << " result = (inputValues[fSelector] >= fCutValue );" << std::endl;
2971 }
2972 fout << " if (fCutType == true) return result; //the cuts are selecting Signal ;" << std::endl;
2973 fout << " else return !result;" << std::endl;
2974 fout << "}" << std::endl;
2975 fout << std::endl;
2976 fout << "//_______________________________________________________________________" << std::endl;
2977 fout << "bool "<<nodeName<<"::GoesLeft( const std::vector<double>& inputValues ) const" << std::endl;
2978 fout << "{" << std::endl;
2979 fout << " // test event if it descends the tree at this node to the left" << std::endl;
2980 fout << " if (!this->GoesRight(inputValues)) return true;" << std::endl;
2981 fout << " else return false;" << std::endl;
2982 fout << "}" << std::endl;
2983 fout << std::endl;
2984 fout << "#endif" << std::endl;
2985 fout << std::endl;
2986}
2987
2988////////////////////////////////////////////////////////////////////////////////
2989/// Recursively descends a tree and writes the node instance to the output stream.
2990
2991void TMVA::MethodBDT::MakeClassInstantiateNode( DecisionTreeNode *n, std::ostream& fout, const TString& className ) const
2992{
2993 if (n == NULL) {
2994 Log() << kFATAL << "MakeClassInstantiateNode: started with undefined node" <<Endl;
2995 return ;
2996 }
2997 fout << "NN("<<std::endl;
2998 if (n->GetLeft() != NULL){
2999 this->MakeClassInstantiateNode( (DecisionTreeNode*)n->GetLeft() , fout, className);
3000 }
3001 else {
3002 fout << "0";
3003 }
3004 fout << ", " <<std::endl;
3005 if (n->GetRight() != NULL){
3006 this->MakeClassInstantiateNode( (DecisionTreeNode*)n->GetRight(), fout, className );
3007 }
3008 else {
3009 fout << "0";
3010 }
3011 fout << ", " << std::endl
3012 << std::setprecision(6);
3013 if (fUseFisherCuts){
3014 fout << n->GetNFisherCoeff() << ", ";
3015 for (UInt_t i=0; i< GetNVariables()+1; i++) {
3016 if (n->GetNFisherCoeff() == 0 ){
3017 fout << "0, ";
3018 }else{
3019 fout << n->GetFisherCoeff(i) << ", ";
3020 }
3021 }
3022 }
3023 fout << n->GetSelector() << ", "
3024 << n->GetCutValue() << ", "
3025 << n->GetCutType() << ", "
3026 << n->GetNodeType() << ", "
3027 << n->GetPurity() << ","
3028 << n->GetResponse() << ") ";
3029}
3030
3031////////////////////////////////////////////////////////////////////////////////
3032/// Find useful preselection cuts that will be applied before
3033/// and Decision Tree training.. (and of course also applied
3034/// in the GetMVA .. --> -1 for background +1 for Signal)
3035
3036void TMVA::MethodBDT::DeterminePreselectionCuts(const std::vector<const TMVA::Event*>& eventSample)
3037{
3038 Double_t nTotS = 0.0, nTotB = 0.0;
3039
3040 std::vector<TMVA::BDTEventWrapper> bdtEventSample;
3041
3042 fIsLowSigCut.assign(GetNvar(),kFALSE);
3043 fIsLowBkgCut.assign(GetNvar(),kFALSE);
3044 fIsHighSigCut.assign(GetNvar(),kFALSE);
3045 fIsHighBkgCut.assign(GetNvar(),kFALSE);
3046
3047 fLowSigCut.assign(GetNvar(),0.); // ---------------| --> in var is signal (accept all above lower cut)
3048 fLowBkgCut.assign(GetNvar(),0.); // ---------------| --> in var is bkg (accept all above lower cut)
3049 fHighSigCut.assign(GetNvar(),0.); // <-- | -------------- in var is signal (accept all blow cut)
3050 fHighBkgCut.assign(GetNvar(),0.); // <-- | -------------- in var is blg (accept all blow cut)
3051
3052
3053 // Initialize (un)weighted counters for signal & background
3054 // Construct a list of event wrappers that point to the original data
3055 for( std::vector<const TMVA::Event*>::const_iterator it = eventSample.begin(); it != eventSample.end(); ++it ) {
3056 if (DataInfo().IsSignal(*it)){
3057 nTotS += (*it)->GetWeight();
3058 }
3059 else {
3060 nTotB += (*it)->GetWeight();
3061 }
3062 bdtEventSample.push_back(TMVA::BDTEventWrapper(*it));
3063 }
3064
3065 for( UInt_t ivar = 0; ivar < GetNvar(); ivar++ ) { // loop over all discriminating variables
3066 TMVA::BDTEventWrapper::SetVarIndex(ivar); // select the variable to sort by
3067 std::sort( bdtEventSample.begin(),bdtEventSample.end() ); // sort the event data
3068
3069 Double_t bkgWeightCtr = 0.0, sigWeightCtr = 0.0;
3070 std::vector<TMVA::BDTEventWrapper>::iterator it = bdtEventSample.begin(), it_end = bdtEventSample.end();
3071 for( ; it != it_end; ++it ) {
3072 if (DataInfo().IsSignal(**it))
3073 sigWeightCtr += (**it)->GetWeight();
3074 else
3075 bkgWeightCtr += (**it)->GetWeight();
3076 // Store the accumulated signal (background) weights
3077 it->SetCumulativeWeight(false,bkgWeightCtr);
3078 it->SetCumulativeWeight(true,sigWeightCtr);
3079 }
3080
3081 //variable that determines how "exact" you cut on the preselection found in the training data. Here I chose
3082 //1% of the variable range...
3083 Double_t dVal = (DataInfo().GetVariableInfo(ivar).GetMax() - DataInfo().GetVariableInfo(ivar).GetMin())/100. ;
3084 Double_t nSelS, nSelB, effS=0.05, effB=0.05, rejS=0.05, rejB=0.05;
3085 Double_t tmpEffS, tmpEffB, tmpRejS, tmpRejB;
3086 // Locate the optimal cut for this (ivar-th) variable
3087
3088
3089
3090 for(UInt_t iev = 1; iev < bdtEventSample.size(); iev++) {
3091 //dVal = bdtEventSample[iev].GetVal() - bdtEventSample[iev-1].GetVal();
3092
3093 nSelS = bdtEventSample[iev].GetCumulativeWeight(true);
3094 nSelB = bdtEventSample[iev].GetCumulativeWeight(false);
3095 // you look for some 100% efficient pre-selection cut to remove background.. i.e. nSelS=0 && nSelB>5%nTotB or ( nSelB=0 nSelS>5%nTotS)
3096 tmpEffS=nSelS/nTotS;
3097 tmpEffB=nSelB/nTotB;
3098 tmpRejS=1-tmpEffS;
3099 tmpRejB=1-tmpEffB;
3100 if (nSelS==0 && tmpEffB>effB) {effB=tmpEffB; fLowBkgCut[ivar] = bdtEventSample[iev].GetVal() - dVal; fIsLowBkgCut[ivar]=kTRUE;}
3101 else if (nSelB==0 && tmpEffS>effS) {effS=tmpEffS; fLowSigCut[ivar] = bdtEventSample[iev].GetVal() - dVal; fIsLowSigCut[ivar]=kTRUE;}
3102 else if (nSelB==nTotB && tmpRejS>rejS) {rejS=tmpRejS; fHighSigCut[ivar] = bdtEventSample[iev].GetVal() + dVal; fIsHighSigCut[ivar]=kTRUE;}
3103 else if (nSelS==nTotS && tmpRejB>rejB) {rejB=tmpRejB; fHighBkgCut[ivar] = bdtEventSample[iev].GetVal() + dVal; fIsHighBkgCut[ivar]=kTRUE;}
3104
3105 }
3106 }
3107
3108 Log() << kDEBUG << " \tfound and suggest the following possible pre-selection cuts " << Endl;
3109 if (fDoPreselection) Log() << kDEBUG << "\tthe training will be done after these cuts... and GetMVA value returns +1, (-1) for a signal (bkg) event that passes these cuts" << Endl;
3110 else Log() << kDEBUG << "\tas option DoPreselection was not used, these cuts however will not be performed, but the training will see the full sample"<<Endl;
3111 for (UInt_t ivar=0; ivar < GetNvar(); ivar++ ) { // loop over all discriminating variables
3112 if (fIsLowBkgCut[ivar]){
3113 Log() << kDEBUG << " \tfound cut: Bkg if var " << ivar << " < " << fLowBkgCut[ivar] << Endl;
3114 }
3115 if (fIsLowSigCut[ivar]){
3116 Log() << kDEBUG << " \tfound cut: Sig if var " << ivar << " < " << fLowSigCut[ivar] << Endl;
3117 }
3118 if (fIsHighBkgCut[ivar]){
3119 Log() << kDEBUG << " \tfound cut: Bkg if var " << ivar << " > " << fHighBkgCut[ivar] << Endl;
3120 }
3121 if (fIsHighSigCut[ivar]){
3122 Log() << kDEBUG << " \tfound cut: Sig if var " << ivar << " > " << fHighSigCut[ivar] << Endl;
3123 }
3124 }
3125
3126 return;
3127}
3128
3129////////////////////////////////////////////////////////////////////////////////
3130/// Apply the preselection cuts before even bothering about any
3131/// Decision Trees in the GetMVA .. --> -1 for background +1 for Signal
3132
3134{
3135 Double_t result=0;
3136
3137 for (UInt_t ivar=0; ivar < GetNvar(); ivar++ ) { // loop over all discriminating variables
3138 if (fIsLowBkgCut[ivar]){
3139 if (ev->GetValue(ivar) < fLowBkgCut[ivar]) result = -1; // is background
3140 }
3141 if (fIsLowSigCut[ivar]){
3142 if (ev->GetValue(ivar) < fLowSigCut[ivar]) result = 1; // is signal
3143 }
3144 if (fIsHighBkgCut[ivar]){
3145 if (ev->GetValue(ivar) > fHighBkgCut[ivar]) result = -1; // is background
3146 }
3147 if (fIsHighSigCut[ivar]){
3148 if (ev->GetValue(ivar) > fHighSigCut[ivar]) result = 1; // is signal
3149 }
3150 }
3151
3152 return result;
3153}
3154
#define REGISTER_METHOD(CLASS)
for example
#define d(i)
Definition: RSha256.hxx:102
#define f(i)
Definition: RSha256.hxx:104
#define h(i)
Definition: RSha256.hxx:106
#define e(i)
Definition: RSha256.hxx:103
const Bool_t kFALSE
Definition: RtypesCore.h:101
unsigned int UInt_t
Definition: RtypesCore.h:46
float Float_t
Definition: RtypesCore.h:57
double Double_t
Definition: RtypesCore.h:59
long long Long64_t
Definition: RtypesCore.h:80
const Bool_t kTRUE
Definition: RtypesCore.h:100
#define ClassImp(name)
Definition: Rtypes.h:375
Option_t Option_t TPoint TPoint const char GetTextMagnitude GetFillStyle GetLineColor GetLineWidth GetMarkerStyle GetTextAlign GetTextColor GetTextSize void char Point_t Rectangle_t WindowAttributes_t Float_t Float_t Float_t Int_t Int_t UInt_t UInt_t Rectangle_t Int_t Int_t Window_t TString Int_t GCValues_t GetPrimarySelectionOwner GetDisplay GetScreen GetColormap GetNativeEvent const char const char dpyName wid window const char font_name cursor keysym reg const char only_if_exist regb h Point_t winding char text const char depth char const char Int_t count const char ColorStruct_t color const char Pixmap_t Pixmap_t PictureAttributes_t attr const char char ret_data h unsigned char height h Atom_t Int_t ULong_t ULong_t unsigned char prop_list Atom_t Atom_t target
Option_t Option_t TPoint TPoint const char GetTextMagnitude GetFillStyle GetLineColor GetLineWidth GetMarkerStyle GetTextAlign GetTextColor GetTextSize void char Point_t Rectangle_t WindowAttributes_t Float_t r
Option_t Option_t TPoint TPoint const char GetTextMagnitude GetFillStyle GetLineColor GetLineWidth GetMarkerStyle GetTextAlign GetTextColor GetTextSize void char Point_t Rectangle_t WindowAttributes_t Float_t Float_t Float_t Int_t Int_t UInt_t UInt_t Rectangle_t result
Option_t Option_t TPoint TPoint const char GetTextMagnitude GetFillStyle GetLineColor GetLineWidth GetMarkerStyle GetTextAlign GetTextColor GetTextSize void char Point_t Rectangle_t WindowAttributes_t Float_t Float_t Float_t Int_t Int_t UInt_t UInt_t Rectangle_t Int_t Int_t Window_t TString Int_t GCValues_t GetPrimarySelectionOwner GetDisplay GetScreen GetColormap GetNativeEvent const char const char dpyName wid window const char font_name cursor keysym reg const char only_if_exist regb h Point_t winding char text const char depth char const char Int_t count const char ColorStruct_t color const char Pixmap_t Pixmap_t PictureAttributes_t attr const char char ret_data h unsigned char height h Atom_t Int_t ULong_t ULong_t unsigned char prop_list Atom_t Atom_t Atom_t Time_t type
char * Form(const char *fmt,...)
Formats a string in a circular formatting buffer.
Definition: TString.cxx:2452
A pseudo container class which is a generator of indices.
Definition: TSeq.hxx:67
A TGraph is an object made of two arrays X and Y with npoints each.
Definition: TGraph.h:41
virtual void SetPoint(Int_t i, Double_t x, Double_t y)
Set x and y values for point number i.
Definition: TGraph.cxx:2281
Int_t GetN() const
Definition: TGraph.h:126
void SetName(const char *name="") override
Set graph name.
Definition: TGraph.cxx:2320
void SetTitle(const char *title="") override
Change (i.e.
Definition: TGraph.cxx:2336
virtual void Set(Int_t n)
Set number of points in the graph Existing coordinates are preserved New coordinates above fNpoints a...
Definition: TGraph.cxx:2216
1-D histogram with a float per channel (see TH1 documentation)}
Definition: TH1.h:574
1-D histogram with an int per channel (see TH1 documentation)}
Definition: TH1.h:533
TH1 is the base class of all histogram classes in ROOT.
Definition: TH1.h:58
virtual void SetXTitle(const char *title)
Definition: TH1.h:412
virtual Int_t Fill(Double_t x)
Increment bin with abscissa X by 1.
Definition: TH1.cxx:3338
virtual void SetBinContent(Int_t bin, Double_t content)
Set bin content see convention for numbering bins in TH1::GetBin In case the bin number is greater th...
Definition: TH1.cxx:9089
virtual void SetYTitle(const char *title)
Definition: TH1.h:413
2-D histogram with a float per channel (see TH1 documentation)}
Definition: TH2.h:257
Service class for 2-D histogram classes.
Definition: TH2.h:30
Absolute Deviation BDT Loss Function.
Definition: LossFunction.h:261
static void SetVarIndex(Int_t iVar)
UInt_t GetNNodes() const
Definition: BinaryTree.h:86
Executor & GetThreadExecutor()
Get executor class for multi-thread usage In case when MT is not enabled will return a serial executo...
Definition: Config.h:81
static Config & Instance()
static function: returns TMVA instance
Definition: Config.cxx:98
Implementation of the CrossEntropy as separation criterion.
Definition: CrossEntropy.h:43
Class that contains all the data information.
Definition: DataSetInfo.h:62
static void SetIsTraining(bool on)
Implementation of a Decision Tree.
Definition: DecisionTree.h:65
TMVA::DecisionTreeNode * GetEventNode(const TMVA::Event &e) const
get the pointer to the leaf node where a particular event ends up in... (used in gradient boosting)
static DecisionTree * CreateFromXML(void *node, UInt_t tmva_Version_Code=262657)
re-create a new tree (decision tree or search tree) from XML
Double_t CheckEvent(const TMVA::Event *, Bool_t UseYesNoLeaf=kFALSE) const
the event e is put into the decision tree (starting at the root node) and the output is NodeType (sig...
Float_t GetValue(UInt_t ivar) const
return value of i'th variable
Definition: Event.cxx:236
void SetTarget(UInt_t itgt, Float_t value)
set the target value (dimension itgt) to value
Definition: Event.cxx:367
Float_t GetTarget(UInt_t itgt) const
Definition: Event.h:102
void Foreach(Function func, unsigned int nTimes, unsigned nChunks=0)
wrap TExecutor::Foreach
Definition: Executor.h:111
auto Map(F func, unsigned nTimes) -> std::vector< InvokeResult_t< F > >
Wrap TExecutor::Map functions.
Definition: Executor.h:134
unsigned int GetPoolSize() const
Definition: Executor.h:100
Implementation of the GiniIndex With Laplace correction as separation criterion.
Implementation of the GiniIndex as separation criterion.
Definition: GiniIndex.h:63
Huber BDT Loss Function.
Definition: LossFunction.h:179
The TMVA::Interval Class.
Definition: Interval.h:61
Least Squares BDT Loss Function.
Definition: LossFunction.h:222
The TMVA::Interval Class.
Definition: LogInterval.h:83
Analysis of Boosted Decision Trees.
Definition: MethodBDT.h:63
void Init(void)
Common initialisation with defaults for the BDT-Method.
Definition: MethodBDT.cxx:687
static const Int_t fgDebugLevel
debug level determining some printout/control plots etc.
Definition: MethodBDT.h:302
MethodBDT(const TString &jobName, const TString &methodTitle, DataSetInfo &theData, const TString &theOption="")
The standard constructor for the "boosted decision trees".
Definition: MethodBDT.cxx:163
void BoostMonitor(Int_t iTree)
Fills the ROCIntegral vs Itree from the testSample for the monitoring plots during the training .
Definition: MethodBDT.cxx:1752
const std::vector< Float_t > & GetMulticlassValues()
Get the multiclass MVA response for the BDT classifier.
Definition: MethodBDT.cxx:2495
Double_t AdaBoostR2(std::vector< const TMVA::Event * > &, DecisionTree *dt)
Adaption of the AdaBoost to regression problems (see H.Drucker 1997).
Definition: MethodBDT.cxx:2193
Double_t PrivateGetMvaValue(const TMVA::Event *ev, Double_t *err=nullptr, Double_t *errUpper=nullptr, UInt_t useNTrees=0)
Return the MVA value (range [-1;1]) that classifies the event according to the majority vote from the...
Definition: MethodBDT.cxx:2468
void MakeClassSpecific(std::ostream &, const TString &) const
Make ROOT-independent C++ class for classifier response (classifier-specific implementation).
Definition: MethodBDT.cxx:2757
void GetHelpMessage() const
Get help message text.
Definition: MethodBDT.cxx:2700
LossFunctionBDT * fRegressionLossFunctionBDTG
Definition: MethodBDT.h:299
void DeterminePreselectionCuts(const std::vector< const TMVA::Event * > &eventSample)
Find useful preselection cuts that will be applied before and Decision Tree training.
Definition: MethodBDT.cxx:3036
Double_t GradBoost(std::vector< const TMVA::Event * > &, DecisionTree *dt, UInt_t cls=0)
Calculate the desired response value for each region.
Definition: MethodBDT.cxx:1595
const Ranking * CreateRanking()
Compute ranking of input variables.
Definition: MethodBDT.cxx:2683
virtual void SetTuneParameters(std::map< TString, Double_t > tuneParameters)
Set the tuning parameters according to the argument.
Definition: MethodBDT.cxx:1121
Double_t AdaCost(std::vector< const TMVA::Event * > &, DecisionTree *dt)
The AdaCost boosting algorithm takes a simple cost Matrix (currently fixed for all events....
Definition: MethodBDT.cxx:2024
void DeclareOptions()
Define the options (their key words).
Definition: MethodBDT.cxx:333
Double_t GetMvaValue(Double_t *err=nullptr, Double_t *errUpper=nullptr)
Definition: MethodBDT.cxx:2443
virtual std::map< TString, Double_t > OptimizeTuningParameters(TString fomType="ROCIntegral", TString fitType="FitGA")
Call the Optimizer with the set of parameters and ranges that are meant to be tuned.
Definition: MethodBDT.cxx:1068
Double_t Boost(std::vector< const TMVA::Event * > &, DecisionTree *dt, UInt_t cls=0)
Apply the boosting algorithm (the algorithm is selecte via the "option" given in the constructor.
Definition: MethodBDT.cxx:1718
Double_t TestTreeQuality(DecisionTree *dt)
Test the tree quality.. in terms of Misclassification.
Definition: MethodBDT.cxx:1697
Double_t Bagging()
Call it boot-strapping, re-sampling or whatever you like, in the end it is nothing else but applying ...
Definition: MethodBDT.cxx:2140
void UpdateTargets(std::vector< const TMVA::Event * > &, UInt_t cls=0)
Calculate residual for all events.
Definition: MethodBDT.cxx:1435
void UpdateTargetsRegression(std::vector< const TMVA::Event * > &, Bool_t first=kFALSE)
Calculate residuals for all events and update targets for next iter.
Definition: MethodBDT.cxx:1557
Double_t GradBoostRegression(std::vector< const TMVA::Event * > &, DecisionTree *dt)
Implementation of M_TreeBoost using any loss function as described by Friedman 1999.
Definition: MethodBDT.cxx:1629
void WriteMonitoringHistosToFile(void) const
Here we could write some histograms created during the processing to the output file.
Definition: MethodBDT.cxx:2628
virtual ~MethodBDT(void)
Destructor.
Definition: MethodBDT.cxx:753
void AddWeightsXMLTo(void *parent) const
Write weights to XML.
Definition: MethodBDT.cxx:2310
Double_t GetGradBoostMVA(const TMVA::Event *e, UInt_t nTrees)
Returns MVA value: -1 for background, 1 for signal.
Definition: MethodBDT.cxx:1421
virtual Bool_t HasAnalysisType(Types::EAnalysisType type, UInt_t numberClasses, UInt_t numberTargets)
BDT can handle classification with multiple classes and regression with one regression-target.
Definition: MethodBDT.cxx:280
Double_t RegBoost(std::vector< const TMVA::Event * > &, DecisionTree *dt)
A special boosting only for Regression (not implemented).
Definition: MethodBDT.cxx:2185
void InitEventSample()
Initialize the event sample (i.e. reset the boost-weights... etc).
Definition: MethodBDT.cxx:761
Double_t ApplyPreselectionCuts(const Event *ev)
Apply the preselection cuts before even bothering about any Decision Trees in the GetMVA .
Definition: MethodBDT.cxx:3133
void SetMinNodeSize(Double_t sizeInPercent)
Definition: MethodBDT.cxx:660
void ProcessOptions()
The option string is decoded, for available options see "DeclareOptions".
Definition: MethodBDT.cxx:470
void PreProcessNegativeEventWeights()
O.k.
Definition: MethodBDT.cxx:932
void MakeClassInstantiateNode(DecisionTreeNode *n, std::ostream &fout, const TString &className) const
Recursively descends a tree and writes the node instance to the output stream.
Definition: MethodBDT.cxx:2991
Double_t AdaBoost(std::vector< const TMVA::Event * > &, DecisionTree *dt)
The AdaBoost implementation.
Definition: MethodBDT.cxx:1846
TTree * fMonitorNtuple
monitoring ntuple
Definition: MethodBDT.h:264
std::vector< Double_t > GetVariableImportance()
Return the relative variable importance, normalized to all variables together having the importance 1...
Definition: MethodBDT.cxx:2643
void InitGradBoost(std::vector< const TMVA::Event * > &)
Initialize targets for first tree.
Definition: MethodBDT.cxx:1658
void Train(void)
BDT training.
Definition: MethodBDT.cxx:1142
void GetBaggedSubSample(std::vector< const TMVA::Event * > &)
Fills fEventSample with fBaggedSampleFraction*NEvents random training events.
Definition: MethodBDT.cxx:2151
const std::vector< Float_t > & GetRegressionValues()
Get the regression value generated by the BDTs.
Definition: MethodBDT.cxx:2542
SeparationBase * fSepType
the separation used in node splitting
Definition: MethodBDT.h:229
void ReadWeightsFromXML(void *parent)
Reads the BDT from the xml file.
Definition: MethodBDT.cxx:2341
void ReadWeightsFromStream(std::istream &istr)
Read the weights (BDT coefficients).
Definition: MethodBDT.cxx:2408
void Reset(void)
Reset the method, as if it had just been instantiated (forget all training etc.).
Definition: MethodBDT.cxx:725
void MakeClassSpecificHeader(std::ostream &, const TString &) const
Specific class header.
Definition: MethodBDT.cxx:2877
void DeclareCompatibilityOptions()
Options that are used ONLY for the READER to ensure backward compatibility.
Definition: MethodBDT.cxx:454
Virtual base Class for all MVA method.
Definition: MethodBase.h:111
virtual void DeclareCompatibilityOptions()
options that are used ONLY for the READER to ensure backward compatibility they are hence without any...
Definition: MethodBase.cxx:596
Implementation of the MisClassificationError as separation criterion.
std::map< TString, Double_t > optimize()
PDF wrapper for histograms; uses user-defined spline interpolation.
Definition: PDF.h:63
@ kSpline3
Definition: PDF.h:70
Ranking for variables in method (implementation)
Definition: Ranking.h:48
Class that is the base-class for a vector of result.
Definition: Results.h:57
TGraph * GetGraph(const TString &alias) const
Definition: Results.cxx:153
void Store(TObject *obj, const char *alias=nullptr)
Definition: Results.cxx:86
TH1 * GetHist(const TString &alias) const
Definition: Results.cxx:136
Implementation of the SdivSqrtSplusB as separation criterion.
Timing information for training and evaluation of MVA methods.
Definition: Timer.h:58
TString GetElapsedTime(Bool_t Scientific=kTRUE)
returns pretty string with elapsed time
Definition: Timer.cxx:146
void DrawProgressBar(Int_t, const TString &comment="")
draws progress bar in color or B&W caution:
Definition: Timer.cxx:202
const TString & Color(const TString &)
human readable color strings
Definition: Tools.cxx:828
void ReadAttr(void *node, const char *, T &value)
read attribute from xml
Definition: Tools.h:329
void * GetChild(void *parent, const char *childname=nullptr)
get child node
Definition: Tools.cxx:1150
void AddAttr(void *node, const char *, const T &value, Int_t precision=16)
add attribute to xml
Definition: Tools.h:347
void * AddChild(void *parent, const char *childname, const char *content=nullptr, bool isRootNode=false)
add child node
Definition: Tools.cxx:1124
void UsefulSortAscending(std::vector< std::vector< Double_t > > &, std::vector< TString > *vs=nullptr)
sort 2D vector (AND in parallel a TString vector) in such a way that the "first vector is sorted" and...
Definition: Tools.cxx:538
std::vector< TMatrixDSym * > * CalcCovarianceMatrices(const std::vector< Event * > &events, Int_t maxCls, VariableTransformBase *transformBase=nullptr)
compute covariance matrices
Definition: Tools.cxx:1514
void * GetNextChild(void *prevchild, const char *childname=nullptr)
XML helpers.
Definition: Tools.cxx:1162
Singleton class for Global types used by TMVA.
Definition: Types.h:71
EAnalysisType
Definition: Types.h:126
@ kMulticlass
Definition: Types.h:129
@ kClassification
Definition: Types.h:127
@ kMaxAnalysisType
Definition: Types.h:131
@ kRegression
Definition: Types.h:128
@ kTraining
Definition: Types.h:143
@ kDEBUG
Definition: Types.h:56
@ kHEADER
Definition: Types.h:63
@ kERROR
Definition: Types.h:60
@ kINFO
Definition: Types.h:58
@ kWARNING
Definition: Types.h:59
@ kFATAL
Definition: Types.h:61
Double_t Determinant() const override
TMatrixTSym< Element > & Invert(Double_t *det=nullptr)
Invert the matrix and calculate its determinant Notice that the LU decomposition is used instead of B...
const char * GetTitle() const override
Returns title of object.
Definition: TNamed.h:48
virtual Int_t Write(const char *name=nullptr, Int_t option=0, Int_t bufsize=0)
Write this object to the current directory.
Definition: TObject.cxx:798
virtual void Delete(Option_t *option="")
Delete this object.
Definition: TObject.cxx:171
virtual Int_t Read(const char *name)
Read contents of object with specified name from the current directory.
Definition: TObject.cxx:566
Random number generator class based on M.
Definition: TRandom3.h:27
virtual Double_t PoissonD(Double_t mean)
Generates a random number according to a Poisson law.
Definition: TRandom.cxx:454
Basic string class.
Definition: TString.h:136
Double_t Atof() const
Return floating-point value contained in string.
Definition: TString.cxx:2017
Bool_t IsFloat() const
Returns kTRUE if string contains a floating point or integer number.
Definition: TString.cxx:1821
TString & ReplaceAll(const TString &s1, const TString &s2)
Definition: TString.h:692
TString & Append(const char *cs)
Definition: TString.h:564
A TTree represents a columnar dataset.
Definition: TTree.h:79
RVec< PromoteType< T > > floor(const RVec< T > &v)
Definition: RVec.hxx:1790
RVec< PromoteType< T > > ceil(const RVec< T > &v)
Definition: RVec.hxx:1791
RVec< PromoteTypes< T0, T1 > > fmod(const T0 &x, const RVec< T1 > &v)
Definition: RVec.hxx:1758
RVec< PromoteType< T > > exp(const RVec< T > &v)
Definition: RVec.hxx:1761
const Int_t n
Definition: legend1.C:16
TGraphErrors * gr
Definition: legend1.C:25
def fit(model, train_loader, val_loader, num_epochs, batch_size, optimizer, criterion, save_best, scheduler)
static const uint32_t K[64]
Definition: RSha256.hxx:148
void Print(std::ostream &os, const OptionType &opt)
double dist(Rotation3D const &r1, Rotation3D const &r2)
Definition: 3DDistances.cxx:48
VecExpr< UnaryOp< Fabs< T >, VecExpr< A, T, D >, T >, T, D > fabs(const VecExpr< A, T, D > &rhs)
TClass * GetClass(T *)
Definition: TClass.h:659
TSeq< unsigned int > TSeqU
Definition: TSeq.hxx:204
void GetMethodName(TString &name, TKey *mkey)
Definition: tmvaglob.cxx:342
create variable transformations
Tools & gTools()
void BDT(TString dataset, const TString &fin="TMVA.root")
MsgLogger & Endl(MsgLogger &ml)
Definition: MsgLogger.h:148
Short_t Max(Short_t a, Short_t b)
Returns the largest of a and b.
Definition: TMathBase.h:250
Double_t Exp(Double_t x)
Returns the base-e exponential function of x, which is e raised to the power x.
Definition: TMath.h:707
Int_t FloorNint(Double_t x)
Returns the nearest integer of TMath::Floor(x).
Definition: TMath.h:684
constexpr Double_t E()
Base of natural log: .
Definition: TMath.h:93
Double_t Log(Double_t x)
Returns the natural logarithm of x.
Definition: TMath.h:754
Double_t Sqrt(Double_t x)
Returns the square root of x.
Definition: TMath.h:660
LongDouble_t Power(LongDouble_t x, LongDouble_t y)
Returns x raised to the power y.
Definition: TMath.h:719
Int_t CeilNint(Double_t x)
Returns the nearest integer of TMath::Ceil(x).
Definition: TMath.h:672
Short_t Min(Short_t a, Short_t b)
Returns the smallest of a and b.
Definition: TMathBase.h:198
Short_t Abs(Short_t d)
Returns the absolute value of parameter Short_t d.
Definition: TMathBase.h:123
Definition: first.py:1
double epsilon
Definition: triangle.c:618