Logo ROOT  
Reference Guide
MethodBDT.cxx
Go to the documentation of this file.
1 // Author: Andreas Hoecker, Joerg Stelzer, Helge Voss, Kai Voss, Eckhard v. Toerne, Jan Therhaag
2 
3 /**********************************************************************************
4  * Project: TMVA - a Root-integrated toolkit for multivariate data analysis *
5  * Package: TMVA *
6  * Class : MethodBDT (BDT = Boosted Decision Trees) *
7  * Web : http://tmva.sourceforge.net *
8  * *
9  * Description: *
10  * Analysis of Boosted Decision Trees *
11  * *
12  * Authors (alphabetical): *
13  * Andreas Hoecker <Andreas.Hocker@cern.ch> - CERN, Switzerland *
14  * Helge Voss <Helge.Voss@cern.ch> - MPI-K Heidelberg, Germany *
15  * Kai Voss <Kai.Voss@cern.ch> - U. of Victoria, Canada *
16  * Doug Schouten <dschoute@sfu.ca> - Simon Fraser U., Canada *
17  * Jan Therhaag <jan.therhaag@cern.ch> - U. of Bonn, Germany *
18  * Eckhard v. Toerne <evt@uni-bonn.de> - U of Bonn, Germany *
19  * *
20  * Copyright (c) 2005-2011: *
21  * CERN, Switzerland *
22  * U. of Victoria, Canada *
23  * MPI-K Heidelberg, Germany *
24  * U. of Bonn, Germany *
25  * *
26  * Redistribution and use in source and binary forms, with or without *
27  * modification, are permitted according to the terms listed in LICENSE *
28  * (http://tmva.sourceforge.net/LICENSE) *
29  **********************************************************************************/
30 
31 /*! \class TMVA::MethodBDT
32 \ingroup TMVA
33 
34 Analysis of Boosted Decision Trees
35 
36 Boosted decision trees have been successfully used in High Energy
37 Physics analysis for example by the MiniBooNE experiment
38 (Yang-Roe-Zhu, physics/0508045). In Boosted Decision Trees, the
39 selection is done on a majority vote on the result of several decision
40 trees, which are all derived from the same training sample by
41 supplying different event weights during the training.
42 
43 ### Decision trees:
44 
45 Successive decision nodes are used to categorize the
46 events out of the sample as either signal or background. Each node
47 uses only a single discriminating variable to decide if the event is
48 signal-like ("goes right") or background-like ("goes left"). This
49 forms a tree like structure with "baskets" at the end (leave nodes),
50 and an event is classified as either signal or background according to
51 whether the basket where it ends up has been classified signal or
52 background during the training. Training of a decision tree is the
53 process to define the "cut criteria" for each node. The training
54 starts with the root node. Here one takes the full training event
55 sample and selects the variable and corresponding cut value that gives
56 the best separation between signal and background at this stage. Using
57 this cut criterion, the sample is then divided into two subsamples, a
58 signal-like (right) and a background-like (left) sample. Two new nodes
59 are then created for each of the two sub-samples and they are
60 constructed using the same mechanism as described for the root
61 node. The devision is stopped once a certain node has reached either a
62 minimum number of events, or a minimum or maximum signal purity. These
63 leave nodes are then called "signal" or "background" if they contain
64 more signal respective background events from the training sample.
65 
66 ### Boosting:
67 
68 The idea behind adaptive boosting (AdaBoost) is, that signal events
69 from the training sample, that end up in a background node
70 (and vice versa) are given a larger weight than events that are in
71 the correct leave node. This results in a re-weighed training event
72 sample, with which then a new decision tree can be developed.
73 The boosting can be applied several times (typically 100-500 times)
74 and one ends up with a set of decision trees (a forest).
75 Gradient boosting works more like a function expansion approach, where
76 each tree corresponds to a summand. The parameters for each summand (tree)
77 are determined by the minimization of a error function (binomial log-
78 likelihood for classification and Huber loss for regression).
79 A greedy algorithm is used, which means, that only one tree is modified
80 at a time, while the other trees stay fixed.
81 
82 ### Bagging:
83 
84 In this particular variant of the Boosted Decision Trees the boosting
85 is not done on the basis of previous training results, but by a simple
86 stochastic re-sampling of the initial training event sample.
87 
88 ### Random Trees:
89 
90 Similar to the "Random Forests" from Leo Breiman and Adele Cutler, it
91 uses the bagging algorithm together and bases the determination of the
92 best node-split during the training on a random subset of variables only
93 which is individually chosen for each split.
94 
95 ### Analysis:
96 
97 Applying an individual decision tree to a test event results in a
98 classification of the event as either signal or background. For the
99 boosted decision tree selection, an event is successively subjected to
100 the whole set of decision trees and depending on how often it is
101 classified as signal, a "likelihood" estimator is constructed for the
102 event being signal or background. The value of this estimator is the
103 one which is then used to select the events from an event sample, and
104 the cut value on this estimator defines the efficiency and purity of
105 the selection.
106 
107 */
108 
109 
110 #include "TMVA/MethodBDT.h"
111 #include "TMVA/Config.h"
112 
113 #include "TMVA/BDTEventWrapper.h"
114 #include "TMVA/BinarySearchTree.h"
115 #include "TMVA/ClassifierFactory.h"
116 #include "TMVA/Configurable.h"
117 #include "TMVA/CrossEntropy.h"
118 #include "TMVA/DecisionTree.h"
119 #include "TMVA/DataSet.h"
120 #include "TMVA/GiniIndex.h"
122 #include "TMVA/Interval.h"
123 #include "TMVA/IMethod.h"
124 #include "TMVA/LogInterval.h"
125 #include "TMVA/MethodBase.h"
127 #include "TMVA/MsgLogger.h"
129 #include "TMVA/PDF.h"
130 #include "TMVA/Ranking.h"
131 #include "TMVA/Results.h"
132 #include "TMVA/ResultsMulticlass.h"
133 #include "TMVA/SdivSqrtSplusB.h"
134 #include "TMVA/SeparationBase.h"
135 #include "TMVA/Timer.h"
136 #include "TMVA/Tools.h"
137 #include "TMVA/Types.h"
138 
139 #include "TRandom3.h"
140 #include "TMath.h"
141 #include "TMatrixTSym.h"
142 #include "TGraph.h"
143 
144 #include <iostream>
145 #include <iomanip>
146 #include <algorithm>
147 #include <cmath>
148 #include <numeric>
149 #include <unordered_map>
150 
151 using std::vector;
152 using std::make_pair;
153 
155 
157 
159 
160 ////////////////////////////////////////////////////////////////////////////////
161 /// The standard constructor for the "boosted decision trees".
162 
164  const TString& methodTitle,
165  DataSetInfo& theData,
166  const TString& theOption ) :
167  TMVA::MethodBase( jobName, Types::kBDT, methodTitle, theData, theOption)
168  , fTrainSample(0)
169  , fNTrees(0)
170  , fSigToBkgFraction(0)
171  , fAdaBoostBeta(0)
172 // , fTransitionPoint(0)
173  , fShrinkage(0)
174  , fBaggedBoost(kFALSE)
175  , fBaggedGradBoost(kFALSE)
176 // , fSumOfWeights(0)
177  , fMinNodeEvents(0)
178  , fMinNodeSize(5)
179  , fMinNodeSizeS("5%")
180  , fNCuts(0)
181  , fUseFisherCuts(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
182  , fMinLinCorrForFisher(.8) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
183  , fUseExclusiveVars(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
184  , fUseYesNoLeaf(kFALSE)
185  , fNodePurityLimit(0)
186  , fNNodesMax(0)
187  , fMaxDepth(0)
188  , fPruneMethod(DecisionTree::kNoPruning)
189  , fPruneStrength(0)
190  , fFValidationEvents(0)
191  , fAutomatic(kFALSE)
192  , fRandomisedTrees(kFALSE)
193  , fUseNvars(0)
194  , fUsePoissonNvars(0) // don't use this initialisation, only here to make Coverity happy. Is set in Init()
195  , fUseNTrainEvents(0)
196  , fBaggedSampleFraction(0)
197  , fNoNegWeightsInTraining(kFALSE)
198  , fInverseBoostNegWeights(kFALSE)
199  , fPairNegWeightsGlobal(kFALSE)
200  , fTrainWithNegWeights(kFALSE)
201  , fDoBoostMonitor(kFALSE)
202  , fITree(0)
203  , fBoostWeight(0)
204  , fErrorFraction(0)
205  , fCss(0)
206  , fCts_sb(0)
207  , fCtb_ss(0)
208  , fCbb(0)
209  , fDoPreselection(kFALSE)
210  , fSkipNormalization(kFALSE)
211  , fHistoricBool(kFALSE)
212 {
213  fMonitorNtuple = NULL;
214  fSepType = NULL;
215  fRegressionLossFunctionBDTG = nullptr;
216 }
217 
218 ////////////////////////////////////////////////////////////////////////////////
219 
221  const TString& theWeightFile)
222  : TMVA::MethodBase( Types::kBDT, theData, theWeightFile)
223  , fTrainSample(0)
224  , fNTrees(0)
225  , fSigToBkgFraction(0)
226  , fAdaBoostBeta(0)
227 // , fTransitionPoint(0)
228  , fShrinkage(0)
229  , fBaggedBoost(kFALSE)
230  , fBaggedGradBoost(kFALSE)
231 // , fSumOfWeights(0)
232  , fMinNodeEvents(0)
233  , fMinNodeSize(5)
234  , fMinNodeSizeS("5%")
235  , fNCuts(0)
236  , fUseFisherCuts(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
237  , fMinLinCorrForFisher(.8) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
238  , fUseExclusiveVars(0) // don't use this initialisation, only here to make Coverity happy. Is set in DeclarOptions()
239  , fUseYesNoLeaf(kFALSE)
240  , fNodePurityLimit(0)
241  , fNNodesMax(0)
242  , fMaxDepth(0)
243  , fPruneMethod(DecisionTree::kNoPruning)
244  , fPruneStrength(0)
245  , fFValidationEvents(0)
246  , fAutomatic(kFALSE)
247  , fRandomisedTrees(kFALSE)
248  , fUseNvars(0)
249  , fUsePoissonNvars(0) // don't use this initialisation, only here to make Coverity happy. Is set in Init()
250  , fUseNTrainEvents(0)
251  , fBaggedSampleFraction(0)
252  , fNoNegWeightsInTraining(kFALSE)
253  , fInverseBoostNegWeights(kFALSE)
254  , fPairNegWeightsGlobal(kFALSE)
255  , fTrainWithNegWeights(kFALSE)
256  , fDoBoostMonitor(kFALSE)
257  , fITree(0)
258  , fBoostWeight(0)
259  , fErrorFraction(0)
260  , fCss(0)
261  , fCts_sb(0)
262  , fCtb_ss(0)
263  , fCbb(0)
264  , fDoPreselection(kFALSE)
265  , fSkipNormalization(kFALSE)
266  , fHistoricBool(kFALSE)
267 {
268  fMonitorNtuple = NULL;
269  fSepType = NULL;
270  fRegressionLossFunctionBDTG = nullptr;
271  // constructor for calculating BDT-MVA using previously generated decision trees
272  // the result of the previous training (the decision trees) are read in via the
273  // weight file. Make sure the the variables correspond to the ones used in
274  // creating the "weight"-file
275 }
276 
277 ////////////////////////////////////////////////////////////////////////////////
278 /// BDT can handle classification with multiple classes and regression with one regression-target.
279 
281 {
282  if (type == Types::kClassification && numberClasses == 2) return kTRUE;
283  if (type == Types::kMulticlass ) return kTRUE;
284  if( type == Types::kRegression && numberTargets == 1 ) return kTRUE;
285  return kFALSE;
286 }
287 
288 ////////////////////////////////////////////////////////////////////////////////
289 /// Define the options (their key words). That can be set in the option string.
290 ///
291 /// know options:
292 ///
293 /// - nTrees number of trees in the forest to be created
294 /// - BoostType the boosting type for the trees in the forest (AdaBoost e.t.c..).
295 /// Known:
296 /// - AdaBoost
297 /// - AdaBoostR2 (Adaboost for regression)
298 /// - Bagging
299 /// - GradBoost
300 /// - AdaBoostBeta the boosting parameter, beta, for AdaBoost
301 /// - UseRandomisedTrees choose at each node splitting a random set of variables
302 /// - UseNvars use UseNvars variables in randomised trees
303 /// - UsePoisson Nvars use UseNvars not as fixed number but as mean of a poisson distribution
304 /// - SeparationType the separation criterion applied in the node splitting.
305 /// Known:
306 /// - GiniIndex
307 /// - MisClassificationError
308 /// - CrossEntropy
309 /// - SDivSqrtSPlusB
310 /// - MinNodeSize: minimum percentage of training events in a leaf node (leaf criteria, stop splitting)
311 /// - nCuts: the number of steps in the optimisation of the cut for a node (if < 0, then
312 /// step size is determined by the events)
313 /// - UseFisherCuts: use multivariate splits using the Fisher criterion
314 /// - UseYesNoLeaf decide if the classification is done simply by the node type, or the S/B
315 /// (from the training) in the leaf node
316 /// - NodePurityLimit the minimum purity to classify a node as a signal node (used in pruning and boosting to determine
317 /// misclassification error rate)
318 /// - PruneMethod The Pruning method.
319 /// Known:
320 /// - NoPruning // switch off pruning completely
321 /// - ExpectedError
322 /// - CostComplexity
323 /// - PruneStrength a parameter to adjust the amount of pruning. Should be large enough such that overtraining is avoided.
324 /// - PruningValFraction number of events to use for optimizing pruning (only if PruneStrength < 0, i.e. automatic pruning)
325 /// - NegWeightTreatment
326 /// - IgnoreNegWeightsInTraining Ignore negative weight events in the training.
327 /// - DecreaseBoostWeight Boost ev. with neg. weight with 1/boostweight instead of boostweight
328 /// - PairNegWeightsGlobal Pair ev. with neg. and pos. weights in training sample and "annihilate" them
329 /// - MaxDepth maximum depth of the decision tree allowed before further splitting is stopped
330 /// - SkipNormalization Skip normalization at initialization, to keep expectation value of BDT output
331 /// according to the fraction of events
332 
334 {
335  DeclareOptionRef(fNTrees, "NTrees", "Number of trees in the forest");
336  if (DoRegression()) {
337  DeclareOptionRef(fMaxDepth=50,"MaxDepth","Max depth of the decision tree allowed");
338  }else{
339  DeclareOptionRef(fMaxDepth=3,"MaxDepth","Max depth of the decision tree allowed");
340  }
341 
342  TString tmp="5%"; if (DoRegression()) tmp="0.2%";
343  DeclareOptionRef(fMinNodeSizeS=tmp, "MinNodeSize", "Minimum percentage of training events required in a leaf node (default: Classification: 5%, Regression: 0.2%)");
344  // MinNodeSize: minimum percentage of training events in a leaf node (leaf criteria, stop splitting)
345  DeclareOptionRef(fNCuts, "nCuts", "Number of grid points in variable range used in finding optimal cut in node splitting");
346 
347  DeclareOptionRef(fBoostType, "BoostType", "Boosting type for the trees in the forest (note: AdaCost is still experimental)");
348 
349  AddPreDefVal(TString("AdaBoost"));
350  AddPreDefVal(TString("RealAdaBoost"));
351  AddPreDefVal(TString("AdaCost"));
352  AddPreDefVal(TString("Bagging"));
353  // AddPreDefVal(TString("RegBoost"));
354  AddPreDefVal(TString("AdaBoostR2"));
355  AddPreDefVal(TString("Grad"));
356  if (DoRegression()) {
357  fBoostType = "AdaBoostR2";
358  }else{
359  fBoostType = "AdaBoost";
360  }
361  DeclareOptionRef(fAdaBoostR2Loss="Quadratic", "AdaBoostR2Loss", "Type of Loss function in AdaBoostR2");
362  AddPreDefVal(TString("Linear"));
363  AddPreDefVal(TString("Quadratic"));
364  AddPreDefVal(TString("Exponential"));
365 
366  DeclareOptionRef(fBaggedBoost=kFALSE, "UseBaggedBoost","Use only a random subsample of all events for growing the trees in each boost iteration.");
367  DeclareOptionRef(fShrinkage = 1.0, "Shrinkage", "Learning rate for BoostType=Grad algorithm");
368  DeclareOptionRef(fAdaBoostBeta=.5, "AdaBoostBeta", "Learning rate for AdaBoost algorithm");
369  DeclareOptionRef(fRandomisedTrees,"UseRandomisedTrees","Determine at each node splitting the cut variable only as the best out of a random subset of variables (like in RandomForests)");
370  DeclareOptionRef(fUseNvars,"UseNvars","Size of the subset of variables used with RandomisedTree option");
371  DeclareOptionRef(fUsePoissonNvars,"UsePoissonNvars", "Interpret \"UseNvars\" not as fixed number but as mean of a Poisson distribution in each split with RandomisedTree option");
372  DeclareOptionRef(fBaggedSampleFraction=.6,"BaggedSampleFraction","Relative size of bagged event sample to original size of the data sample (used whenever bagging is used (i.e. UseBaggedBoost, Bagging,)" );
373 
374  DeclareOptionRef(fUseYesNoLeaf=kTRUE, "UseYesNoLeaf",
375  "Use Sig or Bkg categories, or the purity=S/(S+B) as classification of the leaf node -> Real-AdaBoost");
376  if (DoRegression()) {
377  fUseYesNoLeaf = kFALSE;
378  }
379 
380  DeclareOptionRef(fNegWeightTreatment="InverseBoostNegWeights","NegWeightTreatment","How to treat events with negative weights in the BDT training (particular the boosting) : IgnoreInTraining; Boost With inverse boostweight; Pair events with negative and positive weights in training sample and *annihilate* them (experimental!)");
381  AddPreDefVal(TString("InverseBoostNegWeights"));
382  AddPreDefVal(TString("IgnoreNegWeightsInTraining"));
383  AddPreDefVal(TString("NoNegWeightsInTraining")); // well, let's be nice to users and keep at least this old name anyway ..
384  AddPreDefVal(TString("PairNegWeightsGlobal"));
385  AddPreDefVal(TString("Pray"));
386 
387 
388 
389  DeclareOptionRef(fCss=1., "Css", "AdaCost: cost of true signal selected signal");
390  DeclareOptionRef(fCts_sb=1.,"Cts_sb","AdaCost: cost of true signal selected bkg");
391  DeclareOptionRef(fCtb_ss=1.,"Ctb_ss","AdaCost: cost of true bkg selected signal");
392  DeclareOptionRef(fCbb=1., "Cbb", "AdaCost: cost of true bkg selected bkg ");
393 
394  DeclareOptionRef(fNodePurityLimit=0.5, "NodePurityLimit", "In boosting/pruning, nodes with purity > NodePurityLimit are signal; background otherwise.");
395 
396 
397  DeclareOptionRef(fSepTypeS, "SeparationType", "Separation criterion for node splitting");
398  AddPreDefVal(TString("CrossEntropy"));
399  AddPreDefVal(TString("GiniIndex"));
400  AddPreDefVal(TString("GiniIndexWithLaplace"));
401  AddPreDefVal(TString("MisClassificationError"));
402  AddPreDefVal(TString("SDivSqrtSPlusB"));
403  AddPreDefVal(TString("RegressionVariance"));
404  if (DoRegression()) {
405  fSepTypeS = "RegressionVariance";
406  }else{
407  fSepTypeS = "GiniIndex";
408  }
409 
410  DeclareOptionRef(fRegressionLossFunctionBDTGS = "Huber", "RegressionLossFunctionBDTG", "Loss function for BDTG regression.");
411  AddPreDefVal(TString("Huber"));
412  AddPreDefVal(TString("AbsoluteDeviation"));
413  AddPreDefVal(TString("LeastSquares"));
414 
415  DeclareOptionRef(fHuberQuantile = 0.7, "HuberQuantile", "In the Huber loss function this is the quantile that separates the core from the tails in the residuals distribution.");
416 
417  DeclareOptionRef(fDoBoostMonitor=kFALSE,"DoBoostMonitor","Create control plot with ROC integral vs tree number");
418 
419  DeclareOptionRef(fUseFisherCuts=kFALSE, "UseFisherCuts", "Use multivariate splits using the Fisher criterion");
420  DeclareOptionRef(fMinLinCorrForFisher=.8,"MinLinCorrForFisher", "The minimum linear correlation between two variables demanded for use in Fisher criterion in node splitting");
421  DeclareOptionRef(fUseExclusiveVars=kFALSE,"UseExclusiveVars","Variables already used in fisher criterion are not anymore analysed individually for node splitting");
422 
423 
424  DeclareOptionRef(fDoPreselection=kFALSE,"DoPreselection","and and apply automatic pre-selection for 100% efficient signal (bkg) cuts prior to training");
425 
426 
427  DeclareOptionRef(fSigToBkgFraction=1,"SigToBkgFraction","Sig to Bkg ratio used in Training (similar to NodePurityLimit, which cannot be used in real adaboost");
428 
429  DeclareOptionRef(fPruneMethodS, "PruneMethod", "Note: for BDTs use small trees (e.g.MaxDepth=3) and NoPruning: Pruning: Method used for pruning (removal) of statistically insignificant branches ");
430  AddPreDefVal(TString("NoPruning"));
431  AddPreDefVal(TString("ExpectedError"));
432  AddPreDefVal(TString("CostComplexity"));
433 
434  DeclareOptionRef(fPruneStrength, "PruneStrength", "Pruning strength");
435 
436  DeclareOptionRef(fFValidationEvents=0.5, "PruningValFraction", "Fraction of events to use for optimizing automatic pruning.");
437 
438  DeclareOptionRef(fSkipNormalization=kFALSE, "SkipNormalization", "Skip normalization at initialization, to keep expectation value of BDT output according to the fraction of events");
439 
440  // deprecated options, still kept for the moment:
441  DeclareOptionRef(fMinNodeEvents=0, "nEventsMin", "deprecated: Use MinNodeSize (in % of training events) instead");
442 
443  DeclareOptionRef(fBaggedGradBoost=kFALSE, "UseBaggedGrad","deprecated: Use *UseBaggedBoost* instead: Use only a random subsample of all events for growing the trees in each iteration.");
444  DeclareOptionRef(fBaggedSampleFraction, "GradBaggingFraction","deprecated: Use *BaggedSampleFraction* instead: Defines the fraction of events to be used in each iteration, e.g. when UseBaggedGrad=kTRUE. ");
445  DeclareOptionRef(fUseNTrainEvents,"UseNTrainEvents","deprecated: Use *BaggedSampleFraction* instead: Number of randomly picked training events used in randomised (and bagged) trees");
446  DeclareOptionRef(fNNodesMax,"NNodesMax","deprecated: Use MaxDepth instead to limit the tree size" );
447 
448 
449 }
450 
451 ////////////////////////////////////////////////////////////////////////////////
452 /// Options that are used ONLY for the READER to ensure backward compatibility.
453 
456 
457 
458  DeclareOptionRef(fHistoricBool=kTRUE, "UseWeightedTrees",
459  "Use weighted trees or simple average in classification from the forest");
460  DeclareOptionRef(fHistoricBool=kFALSE, "PruneBeforeBoost", "Flag to prune the tree before applying boosting algorithm");
461  DeclareOptionRef(fHistoricBool=kFALSE,"RenormByClass","Individually re-normalize each event class to the original size after boosting");
462 
463  AddPreDefVal(TString("NegWeightTreatment"),TString("IgnoreNegWeights"));
464 
465 }
466 
467 ////////////////////////////////////////////////////////////////////////////////
468 /// The option string is decoded, for available options see "DeclareOptions".
469 
471 {
472  fSepTypeS.ToLower();
473  if (fSepTypeS == "misclassificationerror") fSepType = new MisClassificationError();
474  else if (fSepTypeS == "giniindex") fSepType = new GiniIndex();
475  else if (fSepTypeS == "giniindexwithlaplace") fSepType = new GiniIndexWithLaplace();
476  else if (fSepTypeS == "crossentropy") fSepType = new CrossEntropy();
477  else if (fSepTypeS == "sdivsqrtsplusb") fSepType = new SdivSqrtSplusB();
478  else if (fSepTypeS == "regressionvariance") fSepType = NULL;
479  else {
480  Log() << kINFO << GetOptions() << Endl;
481  Log() << kFATAL << "<ProcessOptions> unknown Separation Index option " << fSepTypeS << " called" << Endl;
482  }
483 
484  if(!(fHuberQuantile >= 0.0 && fHuberQuantile <= 1.0)){
485  Log() << kINFO << GetOptions() << Endl;
486  Log() << kFATAL << "<ProcessOptions> Huber Quantile must be in range [0,1]. Value given, " << fHuberQuantile << ", does not match this criteria" << Endl;
487  }
488 
489 
490  fRegressionLossFunctionBDTGS.ToLower();
491  if (fRegressionLossFunctionBDTGS == "huber") fRegressionLossFunctionBDTG = new HuberLossFunctionBDT(fHuberQuantile);
492  else if (fRegressionLossFunctionBDTGS == "leastsquares") fRegressionLossFunctionBDTG = new LeastSquaresLossFunctionBDT();
493  else if (fRegressionLossFunctionBDTGS == "absolutedeviation") fRegressionLossFunctionBDTG = new AbsoluteDeviationLossFunctionBDT();
494  else {
495  Log() << kINFO << GetOptions() << Endl;
496  Log() << kFATAL << "<ProcessOptions> unknown Regression Loss Function BDT option " << fRegressionLossFunctionBDTGS << " called" << Endl;
497  }
498 
499  fPruneMethodS.ToLower();
500  if (fPruneMethodS == "expectederror") fPruneMethod = DecisionTree::kExpectedErrorPruning;
501  else if (fPruneMethodS == "costcomplexity") fPruneMethod = DecisionTree::kCostComplexityPruning;
502  else if (fPruneMethodS == "nopruning") fPruneMethod = DecisionTree::kNoPruning;
503  else {
504  Log() << kINFO << GetOptions() << Endl;
505  Log() << kFATAL << "<ProcessOptions> unknown PruneMethod " << fPruneMethodS << " option called" << Endl;
506  }
507  if (fPruneStrength < 0 && (fPruneMethod != DecisionTree::kNoPruning) && fBoostType!="Grad") fAutomatic = kTRUE;
508  else fAutomatic = kFALSE;
509  if (fAutomatic && fPruneMethod==DecisionTree::kExpectedErrorPruning){
510  Log() << kFATAL
511  << "Sorry automatic pruning strength determination is not implemented yet for ExpectedErrorPruning" << Endl;
512  }
513 
514 
515  if (fMinNodeEvents > 0){
516  fMinNodeSize = Double_t(fMinNodeEvents*100.) / Data()->GetNTrainingEvents();
517  Log() << kWARNING << "You have explicitly set ** nEventsMin = " << fMinNodeEvents<<" ** the min absolute number \n"
518  << "of events in a leaf node. This is DEPRECATED, please use the option \n"
519  << "*MinNodeSize* giving the relative number as percentage of training \n"
520  << "events instead. \n"
521  << "nEventsMin="<<fMinNodeEvents<< "--> MinNodeSize="<<fMinNodeSize<<"%"
522  << Endl;
523  Log() << kWARNING << "Note also that explicitly setting *nEventsMin* so far OVERWRITES the option recommended \n"
524  << " *MinNodeSize* = " << fMinNodeSizeS << " option !!" << Endl ;
525  fMinNodeSizeS = Form("%F3.2",fMinNodeSize);
526 
527  }else{
528  SetMinNodeSize(fMinNodeSizeS);
529  }
530 
531 
532  fAdaBoostR2Loss.ToLower();
533 
534  if (fBoostType=="Grad") {
535  fPruneMethod = DecisionTree::kNoPruning;
536  if (fNegWeightTreatment=="InverseBoostNegWeights"){
537  Log() << kINFO << "the option NegWeightTreatment=InverseBoostNegWeights does"
538  << " not exist for BoostType=Grad" << Endl;
539  Log() << kINFO << "--> change to new default NegWeightTreatment=Pray" << Endl;
540  Log() << kDEBUG << "i.e. simply keep them as if which should work fine for Grad Boost" << Endl;
541  fNegWeightTreatment="Pray";
542  fNoNegWeightsInTraining=kFALSE;
543  }
544  } else if (fBoostType=="RealAdaBoost"){
545  fBoostType = "AdaBoost";
546  fUseYesNoLeaf = kFALSE;
547  } else if (fBoostType=="AdaCost"){
548  fUseYesNoLeaf = kFALSE;
549  }
550 
551  if (fFValidationEvents < 0.0) fFValidationEvents = 0.0;
552  if (fAutomatic && fFValidationEvents > 0.5) {
553  Log() << kWARNING << "You have chosen to use more than half of your training sample "
554  << "to optimize the automatic pruning algorithm. This is probably wasteful "
555  << "and your overall results will be degraded. Are you sure you want this?"
556  << Endl;
557  }
558 
559 
560  if (this->Data()->HasNegativeEventWeights()){
561  Log() << kINFO << " You are using a Monte Carlo that has also negative weights. "
562  << "That should in principle be fine as long as on average you end up with "
563  << "something positive. For this you have to make sure that the minimal number "
564  << "of (un-weighted) events demanded for a tree node (currently you use: MinNodeSize="
565  << fMinNodeSizeS << " ("<< fMinNodeSize << "%)"
566  <<", (or the deprecated equivalent nEventsMin) you can set this via the "
567  <<"BDT option string when booking the "
568  << "classifier) is large enough to allow for reasonable averaging!!! "
569  << " If this does not help.. maybe you want to try the option: IgnoreNegWeightsInTraining "
570  << "which ignores events with negative weight in the training. " << Endl
571  << Endl << "Note: You'll get a WARNING message during the training if that should ever happen" << Endl;
572  }
573 
574  if (DoRegression()) {
575  if (fUseYesNoLeaf && !IsConstructedFromWeightFile()){
576  Log() << kWARNING << "Regression Trees do not work with fUseYesNoLeaf=TRUE --> I will set it to FALSE" << Endl;
577  fUseYesNoLeaf = kFALSE;
578  }
579 
580  if (fSepType != NULL){
581  Log() << kWARNING << "Regression Trees do not work with Separation type other than <RegressionVariance> --> I will use it instead" << Endl;
582  fSepType = NULL;
583  }
584  if (fUseFisherCuts){
585  Log() << kWARNING << "Sorry, UseFisherCuts is not available for regression analysis, I will ignore it!" << Endl;
586  fUseFisherCuts = kFALSE;
587  }
588  if (fNCuts < 0) {
589  Log() << kWARNING << "Sorry, the option of nCuts<0 using a more elaborate node splitting algorithm " << Endl;
590  Log() << kWARNING << "is not implemented for regression analysis ! " << Endl;
591  Log() << kWARNING << "--> I switch do default nCuts = 20 and use standard node splitting"<<Endl;
592  fNCuts=20;
593  }
594  }
595  if (fRandomisedTrees){
596  Log() << kINFO << " Randomised trees use no pruning" << Endl;
597  fPruneMethod = DecisionTree::kNoPruning;
598  // fBoostType = "Bagging";
599  }
600 
601  if (fUseFisherCuts) {
602  Log() << kWARNING << "When using the option UseFisherCuts, the other option nCuts<0 (i.e. using" << Endl;
603  Log() << " a more elaborate node splitting algorithm) is not implemented. " << Endl;
604  //I will switch o " << Endl;
605  //Log() << "--> I switch do default nCuts = 20 and use standard node splitting WITH possible Fisher criteria"<<Endl;
606  fNCuts=20;
607  }
608 
609  if (fNTrees==0){
610  Log() << kERROR << " Zero Decision Trees demanded... that does not work !! "
611  << " I set it to 1 .. just so that the program does not crash"
612  << Endl;
613  fNTrees = 1;
614  }
615 
616  fNegWeightTreatment.ToLower();
617  if (fNegWeightTreatment == "ignorenegweightsintraining") fNoNegWeightsInTraining = kTRUE;
618  else if (fNegWeightTreatment == "nonegweightsintraining") fNoNegWeightsInTraining = kTRUE;
619  else if (fNegWeightTreatment == "inverseboostnegweights") fInverseBoostNegWeights = kTRUE;
620  else if (fNegWeightTreatment == "pairnegweightsglobal") fPairNegWeightsGlobal = kTRUE;
621  else if (fNegWeightTreatment == "pray") Log() << kDEBUG << "Yes, good luck with praying " << Endl;
622  else {
623  Log() << kINFO << GetOptions() << Endl;
624  Log() << kFATAL << "<ProcessOptions> unknown option for treating negative event weights during training " << fNegWeightTreatment << " requested" << Endl;
625  }
626 
627  if (fNegWeightTreatment == "pairnegweightsglobal")
628  Log() << kWARNING << " you specified the option NegWeightTreatment=PairNegWeightsGlobal : This option is still considered EXPERIMENTAL !! " << Endl;
629 
630 
631  // dealing with deprecated options !
632  if (fNNodesMax>0) {
633  UInt_t tmp=1; // depth=0 == 1 node
634  fMaxDepth=0;
635  while (tmp < fNNodesMax){
636  tmp+=2*tmp;
637  fMaxDepth++;
638  }
639  Log() << kWARNING << "You have specified a deprecated option *NNodesMax="<<fNNodesMax
640  << "* \n this has been translated to MaxDepth="<<fMaxDepth<<Endl;
641  }
642 
643 
644  if (fUseNTrainEvents>0){
645  fBaggedSampleFraction = (Double_t) fUseNTrainEvents/Data()->GetNTrainingEvents();
646  Log() << kWARNING << "You have specified a deprecated option *UseNTrainEvents="<<fUseNTrainEvents
647  << "* \n this has been translated to BaggedSampleFraction="<<fBaggedSampleFraction<<"(%)"<<Endl;
648  }
649 
650  if (fBoostType=="Bagging") fBaggedBoost = kTRUE;
651  if (fBaggedGradBoost){
652  fBaggedBoost = kTRUE;
653  Log() << kWARNING << "You have specified a deprecated option *UseBaggedGrad* --> please use *UseBaggedBoost* instead" << Endl;
654  }
655 
656 }
657 
658 ////////////////////////////////////////////////////////////////////////////////
659 
661  if (sizeInPercent > 0 && sizeInPercent < 50){
662  fMinNodeSize=sizeInPercent;
663 
664  } else {
665  Log() << kFATAL << "you have demanded a minimal node size of "
666  << sizeInPercent << "% of the training events.. \n"
667  << " that somehow does not make sense "<<Endl;
668  }
669 
670 }
671 
672 ////////////////////////////////////////////////////////////////////////////////
673 
675  sizeInPercent.ReplaceAll("%","");
676  sizeInPercent.ReplaceAll(" ","");
677  if (sizeInPercent.IsFloat()) SetMinNodeSize(sizeInPercent.Atof());
678  else {
679  Log() << kFATAL << "I had problems reading the option MinNodeEvents, which "
680  << "after removing a possible % sign now reads " << sizeInPercent << Endl;
681  }
682 }
683 
684 ////////////////////////////////////////////////////////////////////////////////
685 /// Common initialisation with defaults for the BDT-Method.
686 
688 {
689  fNTrees = 800;
690  if (fAnalysisType == Types::kClassification || fAnalysisType == Types::kMulticlass ) {
691  fMaxDepth = 3;
692  fBoostType = "AdaBoost";
693  if(DataInfo().GetNClasses()!=0) //workaround for multiclass application
694  fMinNodeSize = 5.;
695  }else {
696  fMaxDepth = 50;
697  fBoostType = "AdaBoostR2";
698  fAdaBoostR2Loss = "Quadratic";
699  if(DataInfo().GetNClasses()!=0) //workaround for multiclass application
700  fMinNodeSize = .2;
701  }
702 
703 
704  fNCuts = 20;
705  fPruneMethodS = "NoPruning";
706  fPruneMethod = DecisionTree::kNoPruning;
707  fPruneStrength = 0;
708  fAutomatic = kFALSE;
709  fFValidationEvents = 0.5;
710  fRandomisedTrees = kFALSE;
711  // fUseNvars = (GetNvar()>12) ? UInt_t(GetNvar()/8) : TMath::Max(UInt_t(2),UInt_t(GetNvar()/3));
712  fUseNvars = UInt_t(TMath::Sqrt(GetNvar())+0.6);
713  fUsePoissonNvars = kTRUE;
714  fShrinkage = 1.0;
715 // fSumOfWeights = 0.0;
716 
717  // reference cut value to distinguish signal-like from background-like events
718  SetSignalReferenceCut( 0 );
719 }
720 
721 
722 ////////////////////////////////////////////////////////////////////////////////
723 /// Reset the method, as if it had just been instantiated (forget all training etc.).
724 
726 {
727  // I keep the BDT EventSample and its Validation sample (eventually they should all
728  // disappear and just use the DataSet samples ..
729 
730  // remove all the trees
731  for (UInt_t i=0; i<fForest.size(); i++) delete fForest[i];
732  fForest.clear();
733 
734  fBoostWeights.clear();
735  if (fMonitorNtuple) { fMonitorNtuple->Delete(); fMonitorNtuple=NULL; }
736  fVariableImportance.clear();
737  fResiduals.clear();
738  fLossFunctionEventInfo.clear();
739  // now done in "InitEventSample" which is called in "Train"
740  // reset all previously stored/accumulated BOOST weights in the event sample
741  //for (UInt_t iev=0; iev<fEventSample.size(); iev++) fEventSample[iev]->SetBoostWeight(1.);
742  if (Data()) Data()->DeleteResults(GetMethodName(), Types::kTraining, GetAnalysisType());
743  Log() << kDEBUG << " successfully(?) reset the method " << Endl;
744 }
745 
746 
747 ////////////////////////////////////////////////////////////////////////////////
748 /// Destructor.
749 ///
750 /// - Note: fEventSample and ValidationSample are already deleted at the end of TRAIN
751 /// When they are not used anymore
752 
754 {
755  for (UInt_t i=0; i<fForest.size(); i++) delete fForest[i];
756 }
757 
758 ////////////////////////////////////////////////////////////////////////////////
759 /// Initialize the event sample (i.e. reset the boost-weights... etc).
760 
762 {
763  if (!HasTrainingTree()) Log() << kFATAL << "<Init> Data().TrainingTree() is zero pointer" << Endl;
764 
765  if (fEventSample.size() > 0) { // do not re-initialise the event sample, just set all boostweights to 1. as if it were untouched
766  // reset all previously stored/accumulated BOOST weights in the event sample
767  for (UInt_t iev=0; iev<fEventSample.size(); iev++) fEventSample[iev]->SetBoostWeight(1.);
768  } else {
769  Data()->SetCurrentType(Types::kTraining);
770  UInt_t nevents = Data()->GetNTrainingEvents();
771 
772  std::vector<const TMVA::Event*> tmpEventSample;
773  for (Long64_t ievt=0; ievt<nevents; ievt++) {
774  // const Event *event = new Event(*(GetEvent(ievt)));
775  Event* event = new Event( *GetTrainingEvent(ievt) );
776  tmpEventSample.push_back(event);
777  }
778 
779  if (!DoRegression()) DeterminePreselectionCuts(tmpEventSample);
780  else fDoPreselection = kFALSE; // just to make sure...
781 
782  for (UInt_t i=0; i<tmpEventSample.size(); i++) delete tmpEventSample[i];
783 
784 
785  Bool_t firstNegWeight=kTRUE;
786  Bool_t firstZeroWeight=kTRUE;
787  for (Long64_t ievt=0; ievt<nevents; ievt++) {
788  // const Event *event = new Event(*(GetEvent(ievt)));
789  // const Event* event = new Event( *GetTrainingEvent(ievt) );
790  Event* event = new Event( *GetTrainingEvent(ievt) );
791  if (fDoPreselection){
792  if (TMath::Abs(ApplyPreselectionCuts(event)) > 0.05) {
793  delete event;
794  continue;
795  }
796  }
797 
798  if (event->GetWeight() < 0 && (IgnoreEventsWithNegWeightsInTraining() || fNoNegWeightsInTraining)){
799  if (firstNegWeight) {
800  Log() << kWARNING << " Note, you have events with negative event weight in the sample, but you've chosen to ignore them" << Endl;
801  firstNegWeight=kFALSE;
802  }
803  delete event;
804  }else if (event->GetWeight()==0){
805  if (firstZeroWeight) {
806  firstZeroWeight = kFALSE;
807  Log() << "Events with weight == 0 are going to be simply ignored " << Endl;
808  }
809  delete event;
810  }else{
811  if (event->GetWeight() < 0) {
812  fTrainWithNegWeights=kTRUE;
813  if (firstNegWeight){
814  firstNegWeight = kFALSE;
815  if (fPairNegWeightsGlobal){
816  Log() << kWARNING << "Events with negative event weights are found and "
817  << " will be removed prior to the actual BDT training by global "
818  << " paring (and subsequent annihilation) with positiv weight events"
819  << Endl;
820  }else{
821  Log() << kWARNING << "Events with negative event weights are USED during "
822  << "the BDT training. This might cause problems with small node sizes "
823  << "or with the boosting. Please remove negative events from training "
824  << "using the option *IgnoreEventsWithNegWeightsInTraining* in case you "
825  << "observe problems with the boosting"
826  << Endl;
827  }
828  }
829  }
830  // if fAutomatic == true you need a validation sample to optimize pruning
831  if (fAutomatic) {
832  Double_t modulo = 1.0/(fFValidationEvents);
833  Int_t imodulo = static_cast<Int_t>( fmod(modulo,1.0) > 0.5 ? ceil(modulo) : floor(modulo) );
834  if (ievt % imodulo == 0) fValidationSample.push_back( event );
835  else fEventSample.push_back( event );
836  }
837  else {
838  fEventSample.push_back(event);
839  }
840  }
841  }
842 
843  if (fAutomatic) {
844  Log() << kINFO << "<InitEventSample> Internally I use " << fEventSample.size()
845  << " for Training and " << fValidationSample.size()
846  << " for Pruning Validation (" << ((Float_t)fValidationSample.size())/((Float_t)fEventSample.size()+fValidationSample.size())*100.0
847  << "% of training used for validation)" << Endl;
848  }
849 
850  // some pre-processing for events with negative weights
851  if (fPairNegWeightsGlobal) PreProcessNegativeEventWeights();
852  }
853 
854  if (DoRegression()) {
855  // Regression, no reweighting to do
856  } else if (DoMulticlass()) {
857  // Multiclass, only gradboost is supported. No reweighting.
858  } else if (!fSkipNormalization) {
859  // Binary classification.
860  Log() << kDEBUG << "\t<InitEventSample> For classification trees, "<< Endl;
861  Log() << kDEBUG << " \tthe effective number of backgrounds is scaled to match "<<Endl;
862  Log() << kDEBUG << " \tthe signal. Otherwise the first boosting step would do 'just that'!"<<Endl;
863  // it does not make sense in decision trees to start with unequal number of signal/background
864  // events (weights) .. hence normalize them now (happens otherwise in first 'boosting step'
865  // anyway..
866  // Also make sure, that the sum_of_weights == sample.size() .. as this is assumed in
867  // the DecisionTree to derive a sensible number for "fMinSize" (min.#events in node)
868  // that currently is an OR between "weighted" and "unweighted number"
869  // I want:
870  // nS + nB = n
871  // a*SW + b*BW = n
872  // (a*SW)/(b*BW) = fSigToBkgFraction
873  //
874  // ==> b = n/((1+f)BW) and a = (nf/(1+f))/SW
875 
876  Double_t nevents = fEventSample.size();
877  Double_t sumSigW=0, sumBkgW=0;
878  Int_t sumSig=0, sumBkg=0;
879  for (UInt_t ievt=0; ievt<fEventSample.size(); ievt++) {
880  if ((DataInfo().IsSignal(fEventSample[ievt])) ) {
881  sumSigW += fEventSample[ievt]->GetWeight();
882  sumSig++;
883  } else {
884  sumBkgW += fEventSample[ievt]->GetWeight();
885  sumBkg++;
886  }
887  }
888  if (sumSigW && sumBkgW){
889  Double_t normSig = nevents/((1+fSigToBkgFraction)*sumSigW)*fSigToBkgFraction;
890  Double_t normBkg = nevents/((1+fSigToBkgFraction)*sumBkgW); ;
891  Log() << kDEBUG << "\tre-normalise events such that Sig and Bkg have respective sum of weights = "
892  << fSigToBkgFraction << Endl;
893  Log() << kDEBUG << " \tsig->sig*"<<normSig << "ev. bkg->bkg*"<<normBkg << "ev." <<Endl;
894  Log() << kHEADER << "#events: (reweighted) sig: "<< sumSigW*normSig << " bkg: " << sumBkgW*normBkg << Endl;
895  Log() << kINFO << "#events: (unweighted) sig: "<< sumSig << " bkg: " << sumBkg << Endl;
896  for (Long64_t ievt=0; ievt<nevents; ievt++) {
897  if ((DataInfo().IsSignal(fEventSample[ievt])) ) fEventSample[ievt]->SetBoostWeight(normSig);
898  else fEventSample[ievt]->SetBoostWeight(normBkg);
899  }
900  }else{
901  Log() << kINFO << "--> could not determine scaling factors as either there are " << Endl;
902  Log() << kINFO << " no signal events (sumSigW="<<sumSigW<<") or no bkg ev. (sumBkgW="<<sumBkgW<<")"<<Endl;
903  }
904 
905  }
906 
907  fTrainSample = &fEventSample;
908  if (fBaggedBoost){
909  GetBaggedSubSample(fEventSample);
910  fTrainSample = &fSubSample;
911  }
912 
913  //just for debug purposes..
914  /*
915  sumSigW=0;
916  sumBkgW=0;
917  for (UInt_t ievt=0; ievt<fEventSample.size(); ievt++) {
918  if ((DataInfo().IsSignal(fEventSample[ievt])) ) sumSigW += fEventSample[ievt]->GetWeight();
919  else sumBkgW += fEventSample[ievt]->GetWeight();
920  }
921  Log() << kWARNING << "sigSumW="<<sumSigW<<"bkgSumW="<<sumBkgW<< Endl;
922  */
923 }
924 
925 ////////////////////////////////////////////////////////////////////////////////
926 /// O.k. you know there are events with negative event weights. This routine will remove
927 /// them by pairing them with the closest event(s) of the same event class with positive
928 /// weights
929 /// A first attempt is "brute force", I dont' try to be clever using search trees etc,
930 /// just quick and dirty to see if the result is any good
931 
933  Double_t totalNegWeights = 0;
934  Double_t totalPosWeights = 0;
935  Double_t totalWeights = 0;
936  std::vector<const Event*> negEvents;
937  for (UInt_t iev = 0; iev < fEventSample.size(); iev++){
938  if (fEventSample[iev]->GetWeight() < 0) {
939  totalNegWeights += fEventSample[iev]->GetWeight();
940  negEvents.push_back(fEventSample[iev]);
941  } else {
942  totalPosWeights += fEventSample[iev]->GetWeight();
943  }
944  totalWeights += fEventSample[iev]->GetWeight();
945  }
946  if (totalNegWeights == 0 ) {
947  Log() << kINFO << "no negative event weights found .. no preprocessing necessary" << Endl;
948  return;
949  } else {
950  Log() << kINFO << "found a total of " << totalNegWeights << " of negative event weights which I am going to try to pair with positive events to annihilate them" << Endl;
951  Log() << kINFO << "found a total of " << totalPosWeights << " of events with positive weights" << Endl;
952  Log() << kINFO << "--> total sum of weights = " << totalWeights << " = " << totalNegWeights+totalPosWeights << Endl;
953  }
954 
955  std::vector<TMatrixDSym*>* cov = gTools().CalcCovarianceMatrices( fEventSample, 2);
956 
957  TMatrixDSym *invCov;
958 
959  for (Int_t i=0; i<2; i++){
960  invCov = ((*cov)[i]);
961  if ( TMath::Abs(invCov->Determinant()) < 10E-24 ) {
962  std::cout << "<MethodBDT::PreProcessNeg...> matrix is almost singular with determinant="
963  << TMath::Abs(invCov->Determinant())
964  << " did you use the variables that are linear combinations or highly correlated?"
965  << std::endl;
966  }
967  if ( TMath::Abs(invCov->Determinant()) < 10E-120 ) {
968  std::cout << "<MethodBDT::PreProcessNeg...> matrix is singular with determinant="
969  << TMath::Abs(invCov->Determinant())
970  << " did you use the variables that are linear combinations?"
971  << std::endl;
972  }
973 
974  invCov->Invert();
975  }
976 
977 
978 
979  Log() << kINFO << "Found a total of " << totalNegWeights << " in negative weights out of " << fEventSample.size() << " training events " << Endl;
980  Timer timer(negEvents.size(),"Negative Event paired");
981  for (UInt_t nev = 0; nev < negEvents.size(); nev++){
982  timer.DrawProgressBar( nev );
983  Double_t weight = negEvents[nev]->GetWeight();
984  UInt_t iClassID = negEvents[nev]->GetClass();
985  invCov = ((*cov)[iClassID]);
986  while (weight < 0){
987  // find closest event with positive event weight and "pair" it with the negative event
988  // (add their weight) until there is no negative weight anymore
989  Int_t iMin=-1;
990  Double_t dist, minDist=10E270;
991  for (UInt_t iev = 0; iev < fEventSample.size(); iev++){
992  if (iClassID==fEventSample[iev]->GetClass() && fEventSample[iev]->GetWeight() > 0){
993  dist=0;
994  for (UInt_t ivar=0; ivar < GetNvar(); ivar++){
995  for (UInt_t jvar=0; jvar<GetNvar(); jvar++){
996  dist += (negEvents[nev]->GetValue(ivar)-fEventSample[iev]->GetValue(ivar))*
997  (*invCov)[ivar][jvar]*
998  (negEvents[nev]->GetValue(jvar)-fEventSample[iev]->GetValue(jvar));
999  }
1000  }
1001  if (dist < minDist) { iMin=iev; minDist=dist;}
1002  }
1003  }
1004 
1005  if (iMin > -1) {
1006  // std::cout << "Happily pairing .. weight before : " << negEvents[nev]->GetWeight() << " and " << fEventSample[iMin]->GetWeight();
1007  Double_t newWeight = (negEvents[nev]->GetWeight() + fEventSample[iMin]->GetWeight());
1008  if (newWeight > 0){
1009  negEvents[nev]->SetBoostWeight( 0 );
1010  fEventSample[iMin]->SetBoostWeight( newWeight/fEventSample[iMin]->GetOriginalWeight() ); // note the weight*boostweight should be "newWeight"
1011  } else {
1012  negEvents[nev]->SetBoostWeight( newWeight/negEvents[nev]->GetOriginalWeight() ); // note the weight*boostweight should be "newWeight"
1013  fEventSample[iMin]->SetBoostWeight( 0 );
1014  }
1015  // std::cout << " and afterwards " << negEvents[nev]->GetWeight() << " and the paired " << fEventSample[iMin]->GetWeight() << " dist="<<minDist<< std::endl;
1016  } else Log() << kFATAL << "preprocessing didn't find event to pair with the negative weight ... probably a bug" << Endl;
1017  weight = negEvents[nev]->GetWeight();
1018  }
1019  }
1020  Log() << kINFO << "<Negative Event Pairing> took: " << timer.GetElapsedTime()
1021  << " " << Endl;
1022 
1023  // just check.. now there should be no negative event weight left anymore
1024  totalNegWeights = 0;
1025  totalPosWeights = 0;
1026  totalWeights = 0;
1027  Double_t sigWeight=0;
1028  Double_t bkgWeight=0;
1029  Int_t nSig=0;
1030  Int_t nBkg=0;
1031 
1032  std::vector<const Event*> newEventSample;
1033 
1034  for (UInt_t iev = 0; iev < fEventSample.size(); iev++){
1035  if (fEventSample[iev]->GetWeight() < 0) {
1036  totalNegWeights += fEventSample[iev]->GetWeight();
1037  totalWeights += fEventSample[iev]->GetWeight();
1038  } else {
1039  totalPosWeights += fEventSample[iev]->GetWeight();
1040  totalWeights += fEventSample[iev]->GetWeight();
1041  }
1042  if (fEventSample[iev]->GetWeight() > 0) {
1043  newEventSample.push_back(new Event(*fEventSample[iev]));
1044  if (fEventSample[iev]->GetClass() == fSignalClass){
1045  sigWeight += fEventSample[iev]->GetWeight();
1046  nSig+=1;
1047  }else{
1048  bkgWeight += fEventSample[iev]->GetWeight();
1049  nBkg+=1;
1050  }
1051  }
1052  }
1053  if (totalNegWeights < 0) Log() << kFATAL << " compensation of negative event weights with positive ones did not work " << totalNegWeights << Endl;
1054 
1055  for (UInt_t i=0; i<fEventSample.size(); i++) delete fEventSample[i];
1056  fEventSample = newEventSample;
1057 
1058  Log() << kINFO << " after PreProcessing, the Event sample is left with " << fEventSample.size() << " events (unweighted), all with positive weights, adding up to " << totalWeights << Endl;
1059  Log() << kINFO << " nSig="<<nSig << " sigWeight="<<sigWeight << " nBkg="<<nBkg << " bkgWeight="<<bkgWeight << Endl;
1060 
1061 
1062 }
1063 
1064 ////////////////////////////////////////////////////////////////////////////////
1065 /// Call the Optimizer with the set of parameters and ranges that
1066 /// are meant to be tuned.
1067 
1068 std::map<TString,Double_t> TMVA::MethodBDT::OptimizeTuningParameters(TString fomType, TString fitType)
1069 {
1070  // fill all the tuning parameters that should be optimized into a map:
1071  std::map<TString,TMVA::Interval*> tuneParameters;
1072  std::map<TString,Double_t> tunedParameters;
1073 
1074  // note: the 3rd parameter in the interval is the "number of bins", NOT the stepsize !!
1075  // the actual VALUES at (at least for the scan, guess also in GA) are always
1076  // read from the middle of the bins. Hence.. the choice of Intervals e.g. for the
1077  // MaxDepth, in order to make nice integer values!!!
1078 
1079  // find some reasonable ranges for the optimisation of MinNodeEvents:
1080 
1081  tuneParameters.insert(std::pair<TString,Interval*>("NTrees", new Interval(10,1000,5))); // stepsize 50
1082  tuneParameters.insert(std::pair<TString,Interval*>("MaxDepth", new Interval(2,4,3))); // stepsize 1
1083  tuneParameters.insert(std::pair<TString,Interval*>("MinNodeSize", new LogInterval(1,30,30))); //
1084  //tuneParameters.insert(std::pair<TString,Interval*>("NodePurityLimit",new Interval(.4,.6,3))); // stepsize .1
1085  //tuneParameters.insert(std::pair<TString,Interval*>("BaggedSampleFraction",new Interval(.4,.9,6))); // stepsize .1
1086 
1087  // method-specific parameters
1088  if (fBoostType=="AdaBoost"){
1089  tuneParameters.insert(std::pair<TString,Interval*>("AdaBoostBeta", new Interval(.2,1.,5)));
1090 
1091  }else if (fBoostType=="Grad"){
1092  tuneParameters.insert(std::pair<TString,Interval*>("Shrinkage", new Interval(0.05,0.50,5)));
1093 
1094  }else if (fBoostType=="Bagging" && fRandomisedTrees){
1095  Int_t min_var = TMath::FloorNint( GetNvar() * .25 );
1096  Int_t max_var = TMath::CeilNint( GetNvar() * .75 );
1097  tuneParameters.insert(std::pair<TString,Interval*>("UseNvars", new Interval(min_var,max_var,4)));
1098 
1099  }
1100 
1101  Log()<<kINFO << " the following BDT parameters will be tuned on the respective *grid*\n"<<Endl;
1102  std::map<TString,TMVA::Interval*>::iterator it;
1103  for(it=tuneParameters.begin(); it!= tuneParameters.end(); ++it){
1104  Log() << kWARNING << it->first << Endl;
1105  std::ostringstream oss;
1106  (it->second)->Print(oss);
1107  Log()<<oss.str();
1108  Log()<<Endl;
1109  }
1110 
1111  OptimizeConfigParameters optimize(this, tuneParameters, fomType, fitType);
1112  tunedParameters=optimize.optimize();
1113 
1114  return tunedParameters;
1115 
1116 }
1117 
1118 ////////////////////////////////////////////////////////////////////////////////
1119 /// Set the tuning parameters according to the argument.
1120 
1121 void TMVA::MethodBDT::SetTuneParameters(std::map<TString,Double_t> tuneParameters)
1122 {
1123  std::map<TString,Double_t>::iterator it;
1124  for(it=tuneParameters.begin(); it!= tuneParameters.end(); ++it){
1125  Log() << kWARNING << it->first << " = " << it->second << Endl;
1126  if (it->first == "MaxDepth" ) SetMaxDepth ((Int_t)it->second);
1127  else if (it->first == "MinNodeSize" ) SetMinNodeSize (it->second);
1128  else if (it->first == "NTrees" ) SetNTrees ((Int_t)it->second);
1129  else if (it->first == "NodePurityLimit") SetNodePurityLimit (it->second);
1130  else if (it->first == "AdaBoostBeta" ) SetAdaBoostBeta (it->second);
1131  else if (it->first == "Shrinkage" ) SetShrinkage (it->second);
1132  else if (it->first == "UseNvars" ) SetUseNvars ((Int_t)it->second);
1133  else if (it->first == "BaggedSampleFraction" ) SetBaggedSampleFraction (it->second);
1134  else Log() << kFATAL << " SetParameter for " << it->first << " not yet implemented " <<Endl;
1135  }
1136 }
1137 
1138 ////////////////////////////////////////////////////////////////////////////////
1139 /// BDT training.
1140 
1141 
1143 {
1145 
1146  // fill the STL Vector with the event sample
1147  // (needs to be done here and cannot be done in "init" as the options need to be
1148  // known).
1149  InitEventSample();
1150 
1151  if (fNTrees==0){
1152  Log() << kERROR << " Zero Decision Trees demanded... that does not work !! "
1153  << " I set it to 1 .. just so that the program does not crash"
1154  << Endl;
1155  fNTrees = 1;
1156  }
1157 
1158  if (fInteractive && fInteractive->NotInitialized()){
1159  std::vector<TString> titles = {"Boost weight", "Error Fraction"};
1160  fInteractive->Init(titles);
1161  }
1162  fIPyMaxIter = fNTrees;
1163  fExitFromTraining = false;
1164 
1165  // HHV (it's been here since looong but I really don't know why we cannot handle
1166  // normalized variables in BDTs... todo
1167  if (IsNormalised()) Log() << kFATAL << "\"Normalise\" option cannot be used with BDT; "
1168  << "please remove the option from the configuration string, or "
1169  << "use \"!Normalise\""
1170  << Endl;
1171 
1172  if(DoRegression())
1173  Log() << kINFO << "Regression Loss Function: "<< fRegressionLossFunctionBDTG->Name() << Endl;
1174 
1175  Log() << kINFO << "Training "<< fNTrees << " Decision Trees ... patience please" << Endl;
1176 
1177  Log() << kDEBUG << "Training with maximal depth = " <<fMaxDepth
1178  << ", MinNodeEvents=" << fMinNodeEvents
1179  << ", NTrees="<<fNTrees
1180  << ", NodePurityLimit="<<fNodePurityLimit
1181  << ", AdaBoostBeta="<<fAdaBoostBeta
1182  << Endl;
1183 
1184  // weights applied in boosting
1185  Int_t nBins;
1186  Double_t xMin,xMax;
1187  TString hname = "AdaBooost weight distribution";
1188 
1189  nBins= 100;
1190  xMin = 0;
1191  xMax = 30;
1192 
1193  if (DoRegression()) {
1194  nBins= 100;
1195  xMin = 0;
1196  xMax = 1;
1197  hname="Boost event weights distribution";
1198  }
1199 
1200  // book monitoring histograms (for AdaBost only)
1201 
1202  TH1* h = new TH1F(Form("%s_BoostWeight",DataInfo().GetName()),hname,nBins,xMin,xMax);
1203  TH1* nodesBeforePruningVsTree = new TH1I(Form("%s_NodesBeforePruning",DataInfo().GetName()),"nodes before pruning",fNTrees,0,fNTrees);
1204  TH1* nodesAfterPruningVsTree = new TH1I(Form("%s_NodesAfterPruning",DataInfo().GetName()),"nodes after pruning",fNTrees,0,fNTrees);
1205 
1206 
1207 
1208  if(!DoMulticlass()){
1209  Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, GetAnalysisType());
1210 
1211  h->SetXTitle("boost weight");
1212  results->Store(h, "BoostWeights");
1213 
1214 
1215  // Monitor the performance (on TEST sample) versus number of trees
1216  if (fDoBoostMonitor){
1217  TH2* boostMonitor = new TH2F("BoostMonitor","ROC Integral Vs iTree",2,0,fNTrees,2,0,1.05);
1218  boostMonitor->SetXTitle("#tree");
1219  boostMonitor->SetYTitle("ROC Integral");
1220  results->Store(boostMonitor, "BoostMonitor");
1221  TGraph *boostMonitorGraph = new TGraph();
1222  boostMonitorGraph->SetName("BoostMonitorGraph");
1223  boostMonitorGraph->SetTitle("ROCIntegralVsNTrees");
1224  results->Store(boostMonitorGraph, "BoostMonitorGraph");
1225  }
1226 
1227  // weights applied in boosting vs tree number
1228  h = new TH1F("BoostWeightVsTree","Boost weights vs tree",fNTrees,0,fNTrees);
1229  h->SetXTitle("#tree");
1230  h->SetYTitle("boost weight");
1231  results->Store(h, "BoostWeightsVsTree");
1232 
1233  // error fraction vs tree number
1234  h = new TH1F("ErrFractHist","error fraction vs tree number",fNTrees,0,fNTrees);
1235  h->SetXTitle("#tree");
1236  h->SetYTitle("error fraction");
1237  results->Store(h, "ErrorFrac");
1238 
1239  // nNodesBeforePruning vs tree number
1240  nodesBeforePruningVsTree->SetXTitle("#tree");
1241  nodesBeforePruningVsTree->SetYTitle("#tree nodes");
1242  results->Store(nodesBeforePruningVsTree);
1243 
1244  // nNodesAfterPruning vs tree number
1245  nodesAfterPruningVsTree->SetXTitle("#tree");
1246  nodesAfterPruningVsTree->SetYTitle("#tree nodes");
1247  results->Store(nodesAfterPruningVsTree);
1248 
1249  }
1250 
1251  fMonitorNtuple= new TTree("MonitorNtuple","BDT variables");
1252  fMonitorNtuple->Branch("iTree",&fITree,"iTree/I");
1253  fMonitorNtuple->Branch("boostWeight",&fBoostWeight,"boostWeight/D");
1254  fMonitorNtuple->Branch("errorFraction",&fErrorFraction,"errorFraction/D");
1255 
1256  Timer timer( fNTrees, GetName() );
1257  Int_t nNodesBeforePruningCount = 0;
1258  Int_t nNodesAfterPruningCount = 0;
1259 
1260  Int_t nNodesBeforePruning = 0;
1261  Int_t nNodesAfterPruning = 0;
1262 
1263  if(fBoostType=="Grad"){
1264  InitGradBoost(fEventSample);
1265  }
1266 
1267  Int_t itree=0;
1268  Bool_t continueBoost=kTRUE;
1269  //for (int itree=0; itree<fNTrees; itree++) {
1270 
1271  while (itree < fNTrees && continueBoost){
1272  if (fExitFromTraining) break;
1273  fIPyCurrentIter = itree;
1274  timer.DrawProgressBar( itree );
1275  // Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, GetAnalysisType());
1276  // TH1 *hxx = new TH1F(Form("swdist%d",itree),Form("swdist%d",itree),10000,0,15);
1277  // results->Store(hxx,Form("swdist%d",itree));
1278  // TH1 *hxy = new TH1F(Form("bwdist%d",itree),Form("bwdist%d",itree),10000,0,15);
1279  // results->Store(hxy,Form("bwdist%d",itree));
1280  // for (Int_t iev=0; iev<fEventSample.size(); iev++) {
1281  // if (fEventSample[iev]->GetClass()!=0) hxy->Fill((fEventSample[iev])->GetWeight());
1282  // else hxx->Fill((fEventSample[iev])->GetWeight());
1283  // }
1284 
1285  if(DoMulticlass()){
1286  if (fBoostType!="Grad"){
1287  Log() << kFATAL << "Multiclass is currently only supported by gradient boost. "
1288  << "Please change boost option accordingly (BoostType=Grad)." << Endl;
1289  }
1290 
1291  UInt_t nClasses = DataInfo().GetNClasses();
1292  for (UInt_t i=0;i<nClasses;i++){
1293  // Careful: If fSepType is nullptr, the tree will be considered a regression tree and
1294  // use the correct output for gradboost (response rather than yesnoleaf) in checkEvent.
1295  // See TMVA::MethodBDT::InitGradBoost.
1296  fForest.push_back( new DecisionTree( fSepType, fMinNodeSize, fNCuts, &(DataInfo()), i,
1297  fRandomisedTrees, fUseNvars, fUsePoissonNvars, fMaxDepth,
1298  itree*nClasses+i, fNodePurityLimit, itree*nClasses+1));
1299  fForest.back()->SetNVars(GetNvar());
1300  if (fUseFisherCuts) {
1301  fForest.back()->SetUseFisherCuts();
1302  fForest.back()->SetMinLinCorrForFisher(fMinLinCorrForFisher);
1303  fForest.back()->SetUseExclusiveVars(fUseExclusiveVars);
1304  }
1305  // the minimum linear correlation between two variables demanded for use in fisher criterion in node splitting
1306 
1307  nNodesBeforePruning = fForest.back()->BuildTree(*fTrainSample);
1308  Double_t bw = this->Boost(*fTrainSample, fForest.back(),i);
1309  if (bw > 0) {
1310  fBoostWeights.push_back(bw);
1311  }else{
1312  fBoostWeights.push_back(0);
1313  Log() << kWARNING << "stopped boosting at itree="<<itree << Endl;
1314  // fNTrees = itree+1; // that should stop the boosting
1315  continueBoost=kFALSE;
1316  }
1317  }
1318  }
1319  else{
1320 
1321  DecisionTree* dt = new DecisionTree( fSepType, fMinNodeSize, fNCuts, &(DataInfo()), fSignalClass,
1322  fRandomisedTrees, fUseNvars, fUsePoissonNvars, fMaxDepth,
1323  itree, fNodePurityLimit, itree);
1324 
1325  fForest.push_back(dt);
1326  fForest.back()->SetNVars(GetNvar());
1327  if (fUseFisherCuts) {
1328  fForest.back()->SetUseFisherCuts();
1329  fForest.back()->SetMinLinCorrForFisher(fMinLinCorrForFisher);
1330  fForest.back()->SetUseExclusiveVars(fUseExclusiveVars);
1331  }
1332 
1333  nNodesBeforePruning = fForest.back()->BuildTree(*fTrainSample);
1334 
1335  if (fUseYesNoLeaf && !DoRegression() && fBoostType!="Grad") { // remove leaf nodes where both daughter nodes are of same type
1336  nNodesBeforePruning = fForest.back()->CleanTree();
1337  }
1338 
1339  nNodesBeforePruningCount += nNodesBeforePruning;
1340  nodesBeforePruningVsTree->SetBinContent(itree+1,nNodesBeforePruning);
1341 
1342  fForest.back()->SetPruneMethod(fPruneMethod); // set the pruning method for the tree
1343  fForest.back()->SetPruneStrength(fPruneStrength); // set the strength parameter
1344 
1345  std::vector<const Event*> * validationSample = NULL;
1346  if(fAutomatic) validationSample = &fValidationSample;
1347  Double_t bw = this->Boost(*fTrainSample, fForest.back());
1348  if (bw > 0) {
1349  fBoostWeights.push_back(bw);
1350  }else{
1351  fBoostWeights.push_back(0);
1352  Log() << kWARNING << "stopped boosting at itree="<<itree << Endl;
1353  continueBoost=kFALSE;
1354  }
1355 
1356  // if fAutomatic == true, pruneStrength will be the optimal pruning strength
1357  // determined by the pruning algorithm; otherwise, it is simply the strength parameter
1358  // set by the user
1359  if (fPruneMethod != DecisionTree::kNoPruning) fForest.back()->PruneTree(validationSample);
1360 
1361  if (fUseYesNoLeaf && !DoRegression() && fBoostType!="Grad"){ // remove leaf nodes where both daughter nodes are of same type
1362  fForest.back()->CleanTree();
1363  }
1364  nNodesAfterPruning = fForest.back()->GetNNodes();
1365  nNodesAfterPruningCount += nNodesAfterPruning;
1366  nodesAfterPruningVsTree->SetBinContent(itree+1,nNodesAfterPruning);
1367 
1368  if (fInteractive){
1369  fInteractive->AddPoint(itree, fBoostWeight, fErrorFraction);
1370  }
1371  fITree = itree;
1372  fMonitorNtuple->Fill();
1373  if (fDoBoostMonitor){
1374  if (! DoRegression() ){
1375  if ( itree==fNTrees-1 || (!(itree%500)) ||
1376  (!(itree%250) && itree <1000)||
1377  (!(itree%100) && itree < 500)||
1378  (!(itree%50) && itree < 250)||
1379  (!(itree%25) && itree < 150)||
1380  (!(itree%10) && itree < 50)||
1381  (!(itree%5) && itree < 20)
1382  ) BoostMonitor(itree);
1383  }
1384  }
1385  }
1386  itree++;
1387  }
1388 
1389  // get elapsed time
1390  Log() << kDEBUG << "\t<Train> elapsed time: " << timer.GetElapsedTime()
1391  << " " << Endl;
1392  if (fPruneMethod == DecisionTree::kNoPruning) {
1393  Log() << kDEBUG << "\t<Train> average number of nodes (w/o pruning) : "
1394  << nNodesBeforePruningCount/GetNTrees() << Endl;
1395  }
1396  else {
1397  Log() << kDEBUG << "\t<Train> average number of nodes before/after pruning : "
1398  << nNodesBeforePruningCount/GetNTrees() << " / "
1399  << nNodesAfterPruningCount/GetNTrees()
1400  << Endl;
1401  }
1403 
1404 
1405  // reset all previously stored/accumulated BOOST weights in the event sample
1406  // for (UInt_t iev=0; iev<fEventSample.size(); iev++) fEventSample[iev]->SetBoostWeight(1.);
1407  Log() << kDEBUG << "Now I delete the privat data sample"<< Endl;
1408  for (UInt_t i=0; i<fEventSample.size(); i++) delete fEventSample[i];
1409  for (UInt_t i=0; i<fValidationSample.size(); i++) delete fValidationSample[i];
1410  fEventSample.clear();
1411  fValidationSample.clear();
1412 
1413  if (!fExitFromTraining) fIPyMaxIter = fIPyCurrentIter;
1414  ExitFromTraining();
1415 }
1416 
1417 
1418 ////////////////////////////////////////////////////////////////////////////////
1419 /// Returns MVA value: -1 for background, 1 for signal.
1420 
1422 {
1423  Double_t sum=0;
1424  for (UInt_t itree=0; itree<nTrees; itree++) {
1425  //loop over all trees in forest
1426  sum += fForest[itree]->CheckEvent(e,kFALSE);
1427 
1428  }
1429  return 2.0/(1.0+exp(-2.0*sum))-1; //MVA output between -1 and 1
1430 }
1431 
1432 ////////////////////////////////////////////////////////////////////////////////
1433 /// Calculate residual for all events.
1434 
1435 void TMVA::MethodBDT::UpdateTargets(std::vector<const TMVA::Event*>& eventSample, UInt_t cls)
1436 {
1437  if (DoMulticlass()) {
1438  UInt_t nClasses = DataInfo().GetNClasses();
1439  Bool_t isLastClass = (cls == nClasses - 1);
1440 
1441  #ifdef R__USE_IMT
1442  //
1443  // This is the multi-threaded multiclass version
1444  //
1445  // Note: we only need to update the predicted probabilities every
1446  // `nClasses` tree. Let's call a set of `nClasses` trees a "round". Thus
1447  // the algortihm is split in two parts `update_residuals` and
1448  // `update_residuals_last` where the latter is inteded to be run instead
1449  // of the former for the last tree in a "round".
1450  //
1451  std::map<const TMVA::Event *, std::vector<double>> & residuals = this->fResiduals;
1452  DecisionTree & lastTree = *(this->fForest.back());
1453 
1454  auto update_residuals = [&residuals, &lastTree, cls](const TMVA::Event * e) {
1455  residuals[e].at(cls) += lastTree.CheckEvent(e, kFALSE);
1456  };
1457 
1458  auto update_residuals_last = [&residuals, &lastTree, cls, nClasses](const TMVA::Event * e) {
1459  residuals[e].at(cls) += lastTree.CheckEvent(e, kFALSE);
1460 
1461  auto &residualsThisEvent = residuals[e];
1462 
1463  std::vector<Double_t> expCache(nClasses, 0.0);
1464  std::transform(residualsThisEvent.begin(),
1465  residualsThisEvent.begin() + nClasses,
1466  expCache.begin(), [](Double_t d) { return exp(d); });
1467 
1468  Double_t exp_sum = std::accumulate(expCache.begin(),
1469  expCache.begin() + nClasses,
1470  0.0);
1471 
1472  for (UInt_t i = 0; i < nClasses; i++) {
1473  Double_t p_cls = expCache[i] / exp_sum;
1474 
1475  Double_t res = (e->GetClass() == i) ? (1.0 - p_cls) : (-p_cls);
1476  const_cast<TMVA::Event *>(e)->SetTarget(i, res);
1477  }
1478  };
1479 
1480  if (isLastClass) {
1482  .Foreach(update_residuals_last, eventSample);
1483  } else {
1485  .Foreach(update_residuals, eventSample);
1486  }
1487  #else
1488  //
1489  // Single-threaded multiclass version
1490  //
1491  std::vector<Double_t> expCache;
1492  if (isLastClass) {
1493  expCache.resize(nClasses);
1494  }
1495 
1496  for (auto e : eventSample) {
1497  fResiduals[e].at(cls) += fForest.back()->CheckEvent(e, kFALSE);
1498  if (isLastClass) {
1499  auto &residualsThisEvent = fResiduals[e];
1500  std::transform(residualsThisEvent.begin(),
1501  residualsThisEvent.begin() + nClasses,
1502  expCache.begin(), [](Double_t d) { return exp(d); });
1503 
1504  Double_t exp_sum = std::accumulate(expCache.begin(),
1505  expCache.begin() + nClasses,
1506  0.0);
1507 
1508  for (UInt_t i = 0; i < nClasses; i++) {
1509  Double_t p_cls = expCache[i] / exp_sum;
1510 
1511  Double_t res = (e->GetClass() == i) ? (1.0 - p_cls) : (-p_cls);
1512  const_cast<TMVA::Event *>(e)->SetTarget(i, res);
1513  }
1514  }
1515  }
1516  #endif
1517  } else {
1518  std::map<const TMVA::Event *, std::vector<double>> & residuals = this->fResiduals;
1519  DecisionTree & lastTree = *(this->fForest.back());
1520 
1521  UInt_t signalClass = DataInfo().GetSignalClassIndex();
1522 
1523  #ifdef R__USE_IMT
1524  auto update_residuals = [&residuals, &lastTree, signalClass](const TMVA::Event * e) {
1525  double & residualAt0 = residuals[e].at(0);
1526  residualAt0 += lastTree.CheckEvent(e, kFALSE);
1527 
1528  Double_t p_sig = 1.0 / (1.0 + exp(-2.0 * residualAt0));
1529  Double_t res = ((e->GetClass() == signalClass) ? (1.0 - p_sig) : (-p_sig));
1530 
1531  const_cast<TMVA::Event *>(e)->SetTarget(0, res);
1532  };
1533 
1535  .Foreach(update_residuals, eventSample);
1536  #else
1537  for (auto e : eventSample) {
1538  double & residualAt0 = residuals[e].at(0);
1539  residualAt0 += lastTree.CheckEvent(e, kFALSE);
1540 
1541  Double_t p_sig = 1.0 / (1.0 + exp(-2.0 * residualAt0));
1542  Double_t res = ((e->GetClass() == signalClass) ? (1.0 - p_sig) : (-p_sig));
1543 
1544  const_cast<TMVA::Event *>(e)->SetTarget(0, res);
1545  }
1546  #endif
1547  }
1548 }
1549 
1550 ////////////////////////////////////////////////////////////////////////////////
1551 /// \brief Calculate residuals for all events and update targets for next iter.
1552 ///
1553 /// \param[in] eventSample The collection of events currently under training.
1554 /// \param[in] first Should be true when called before the first boosting
1555 /// iteration has been run
1556 ///
1557 void TMVA::MethodBDT::UpdateTargetsRegression(std::vector<const TMVA::Event*>& eventSample, Bool_t first)
1558 {
1559  if (!first) {
1560 #ifdef R__USE_IMT
1562  auto seeds = ROOT::TSeqU(nPartitions);
1563 
1564  // need a lambda function to pass to TThreadExecutor::MapReduce
1565  auto f = [this, &nPartitions](UInt_t partition = 0) -> Int_t {
1566  Int_t start = 1.0 * partition / nPartitions * this->fEventSample.size();
1567  Int_t end = (partition + 1.0) / nPartitions * this->fEventSample.size();
1568 
1569  for (Int_t i = start; i < end; ++i) {
1570  const TMVA::Event *e = fEventSample[i];
1571  LossFunctionEventInfo & lossInfo = fLossFunctionEventInfo.at(e);
1572  lossInfo.predictedValue += fForest.back()->CheckEvent(e, kFALSE);
1573  }
1574 
1575  return 0;
1576  };
1577 
1579 #else
1580  for (const TMVA::Event *e : fEventSample) {
1581  LossFunctionEventInfo & lossInfo = fLossFunctionEventInfo.at(e);
1582  lossInfo.predictedValue += fForest.back()->CheckEvent(e, kFALSE);
1583  }
1584 #endif
1585  }
1586 
1587  // NOTE: Set targets are also parallelised internally
1588  fRegressionLossFunctionBDTG->SetTargets(eventSample, fLossFunctionEventInfo);
1589 
1590 }
1591 
1592 ////////////////////////////////////////////////////////////////////////////////
1593 /// Calculate the desired response value for each region.
1594 
1595 Double_t TMVA::MethodBDT::GradBoost(std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt, UInt_t cls)
1596 {
1597  struct LeafInfo {
1598  Double_t sumWeightTarget = 0;
1599  Double_t sum2 = 0;
1600  };
1601 
1602  std::unordered_map<TMVA::DecisionTreeNode*, LeafInfo> leaves;
1603  for (auto e : eventSample) {
1604  Double_t weight = e->GetWeight();
1605  TMVA::DecisionTreeNode* node = dt->GetEventNode(*e);
1606  auto &v = leaves[node];
1607  auto target = e->GetTarget(cls);
1608  v.sumWeightTarget += target * weight;
1609  v.sum2 += fabs(target) * (1.0 - fabs(target)) * weight;
1610  }
1611  for (auto &iLeave : leaves) {
1612  constexpr auto minValue = 1e-30;
1613  if (iLeave.second.sum2 < minValue) {
1614  iLeave.second.sum2 = minValue;
1615  }
1616  const Double_t K = DataInfo().GetNClasses();
1617  iLeave.first->SetResponse(fShrinkage * (K - 1) / K * iLeave.second.sumWeightTarget / iLeave.second.sum2);
1618  }
1619 
1620  //call UpdateTargets before next tree is grown
1621 
1622  DoMulticlass() ? UpdateTargets(fEventSample, cls) : UpdateTargets(fEventSample);
1623  return 1; //trees all have the same weight
1624 }
1625 
1626 ////////////////////////////////////////////////////////////////////////////////
1627 /// Implementation of M_TreeBoost using any loss function as described by Friedman 1999.
1628 
1629 Double_t TMVA::MethodBDT::GradBoostRegression(std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
1630 {
1631  // get the vector of events for each terminal so that we can calculate the constant fit value in each
1632  // terminal node
1633  // #### Not sure how many events are in each node in advance, so I can't parallelize this easily
1634  std::map<TMVA::DecisionTreeNode*,vector< TMVA::LossFunctionEventInfo > > leaves;
1635  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1636  TMVA::DecisionTreeNode* node = dt->GetEventNode(*(*e));
1637  (leaves[node]).push_back(fLossFunctionEventInfo[*e]);
1638  }
1639 
1640  // calculate the constant fit for each terminal node based upon the events in the node
1641  // node (iLeave->first), vector of event information (iLeave->second)
1642  // #### could parallelize this and do the leaves at the same time, but this doesn't take very long compared
1643  // #### to the other processes
1644  for (std::map<TMVA::DecisionTreeNode*,vector< TMVA::LossFunctionEventInfo > >::iterator iLeave=leaves.begin();
1645  iLeave!=leaves.end();++iLeave){
1646  Double_t fit = fRegressionLossFunctionBDTG->Fit(iLeave->second);
1647  (iLeave->first)->SetResponse(fShrinkage*fit);
1648  }
1649 
1650  UpdateTargetsRegression(*fTrainSample);
1651 
1652  return 1;
1653 }
1654 
1655 ////////////////////////////////////////////////////////////////////////////////
1656 /// Initialize targets for first tree.
1657 
1658 void TMVA::MethodBDT::InitGradBoost( std::vector<const TMVA::Event*>& eventSample)
1659 {
1660  // Should get rid of this line. It's just for debugging.
1661  //std::sort(eventSample.begin(), eventSample.end(), [](const TMVA::Event* a, const TMVA::Event* b){
1662  // return (a->GetTarget(0) < b->GetTarget(0)); });
1663  fSepType=NULL; //set fSepType to NULL (regression trees are used for both classification an regression)
1664  if(DoRegression()){
1665  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1666  fLossFunctionEventInfo[*e]= TMVA::LossFunctionEventInfo((*e)->GetTarget(0), 0, (*e)->GetWeight());
1667  }
1668 
1669  fRegressionLossFunctionBDTG->Init(fLossFunctionEventInfo, fBoostWeights);
1670  UpdateTargetsRegression(*fTrainSample,kTRUE);
1671 
1672  return;
1673  }
1674  else if(DoMulticlass()){
1675  UInt_t nClasses = DataInfo().GetNClasses();
1676  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1677  for (UInt_t i=0;i<nClasses;i++){
1678  //Calculate initial residua, assuming equal probability for all classes
1679  Double_t r = (*e)->GetClass()==i?(1-1.0/nClasses):(-1.0/nClasses);
1680  const_cast<TMVA::Event*>(*e)->SetTarget(i,r);
1681  fResiduals[*e].push_back(0);
1682  }
1683  }
1684  }
1685  else{
1686  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1687  Double_t r = (DataInfo().IsSignal(*e)?1:0)-0.5; //Calculate initial residua
1688  const_cast<TMVA::Event*>(*e)->SetTarget(0,r);
1689  fResiduals[*e].push_back(0);
1690  }
1691  }
1692 
1693 }
1694 ////////////////////////////////////////////////////////////////////////////////
1695 /// Test the tree quality.. in terms of Misclassification.
1696 
1698 {
1699  Double_t ncorrect=0, nfalse=0;
1700  for (UInt_t ievt=0; ievt<fValidationSample.size(); ievt++) {
1701  Bool_t isSignalType= (dt->CheckEvent(fValidationSample[ievt]) > fNodePurityLimit ) ? 1 : 0;
1702 
1703  if (isSignalType == (DataInfo().IsSignal(fValidationSample[ievt])) ) {
1704  ncorrect += fValidationSample[ievt]->GetWeight();
1705  }
1706  else{
1707  nfalse += fValidationSample[ievt]->GetWeight();
1708  }
1709  }
1710 
1711  return ncorrect / (ncorrect + nfalse);
1712 }
1713 
1714 ////////////////////////////////////////////////////////////////////////////////
1715 /// Apply the boosting algorithm (the algorithm is selecte via the the "option" given
1716 /// in the constructor. The return value is the boosting weight.
1717 
1718 Double_t TMVA::MethodBDT::Boost( std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt, UInt_t cls )
1719 {
1720  Double_t returnVal=-1;
1721 
1722  if (fBoostType=="AdaBoost") returnVal = this->AdaBoost (eventSample, dt);
1723  else if (fBoostType=="AdaCost") returnVal = this->AdaCost (eventSample, dt);
1724  else if (fBoostType=="Bagging") returnVal = this->Bagging ( );
1725  else if (fBoostType=="RegBoost") returnVal = this->RegBoost (eventSample, dt);
1726  else if (fBoostType=="AdaBoostR2") returnVal = this->AdaBoostR2(eventSample, dt);
1727  else if (fBoostType=="Grad"){
1728  if(DoRegression())
1729  returnVal = this->GradBoostRegression(eventSample, dt);
1730  else if(DoMulticlass())
1731  returnVal = this->GradBoost (eventSample, dt, cls);
1732  else
1733  returnVal = this->GradBoost (eventSample, dt);
1734  }
1735  else {
1736  Log() << kINFO << GetOptions() << Endl;
1737  Log() << kFATAL << "<Boost> unknown boost option " << fBoostType<< " called" << Endl;
1738  }
1739 
1740  if (fBaggedBoost){
1741  GetBaggedSubSample(fEventSample);
1742  }
1743 
1744 
1745  return returnVal;
1746 }
1747 
1748 ////////////////////////////////////////////////////////////////////////////////
1749 /// Fills the ROCIntegral vs Itree from the testSample for the monitoring plots
1750 /// during the training .. but using the testing events
1751 
1753 {
1754  Results* results = Data()->GetResults(GetMethodName(),Types::kTraining, Types::kMaxAnalysisType);
1755 
1756  TH1F *tmpS = new TH1F( "tmpS", "", 100 , -1., 1.00001 );
1757  TH1F *tmpB = new TH1F( "tmpB", "", 100 , -1., 1.00001 );
1758  TH1F *tmp;
1759 
1760 
1761  UInt_t signalClassNr = DataInfo().GetClassInfo("Signal")->GetNumber();
1762 
1763  // const std::vector<Event*> events=Data()->GetEventCollection(Types::kTesting);
1764  // // fMethod->GetTransformationHandler().CalcTransformations(fMethod->Data()->GetEventCollection(Types::kTesting));
1765  // for (UInt_t iev=0; iev < events.size() ; iev++){
1766  // if (events[iev]->GetClass() == signalClassNr) tmp=tmpS;
1767  // else tmp=tmpB;
1768  // tmp->Fill(PrivateGetMvaValue(*(events[iev])),events[iev]->GetWeight());
1769  // }
1770 
1771  UInt_t nevents = Data()->GetNTestEvents();
1772  for (UInt_t iev=0; iev < nevents; iev++){
1773  const Event* event = GetTestingEvent(iev);
1774 
1775  if (event->GetClass() == signalClassNr) {tmp=tmpS;}
1776  else {tmp=tmpB;}
1777  tmp->Fill(PrivateGetMvaValue(event),event->GetWeight());
1778  }
1779  Double_t max=1;
1780 
1781  std::vector<TH1F*> hS;
1782  std::vector<TH1F*> hB;
1783  for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
1784  hS.push_back(new TH1F(Form("SigVar%dAtTree%d",ivar,iTree),Form("SigVar%dAtTree%d",ivar,iTree),100,DataInfo().GetVariableInfo(ivar).GetMin(),DataInfo().GetVariableInfo(ivar).GetMax()));
1785  hB.push_back(new TH1F(Form("BkgVar%dAtTree%d",ivar,iTree),Form("BkgVar%dAtTree%d",ivar,iTree),100,DataInfo().GetVariableInfo(ivar).GetMin(),DataInfo().GetVariableInfo(ivar).GetMax()));
1786  results->Store(hS.back(),hS.back()->GetTitle());
1787  results->Store(hB.back(),hB.back()->GetTitle());
1788  }
1789 
1790 
1791  for (UInt_t iev=0; iev < fEventSample.size(); iev++){
1792  if (fEventSample[iev]->GetBoostWeight() > max) max = 1.01*fEventSample[iev]->GetBoostWeight();
1793  }
1794  TH1F *tmpBoostWeightsS = new TH1F(Form("BoostWeightsInTreeS%d",iTree),Form("BoostWeightsInTreeS%d",iTree),100,0.,max);
1795  TH1F *tmpBoostWeightsB = new TH1F(Form("BoostWeightsInTreeB%d",iTree),Form("BoostWeightsInTreeB%d",iTree),100,0.,max);
1796  results->Store(tmpBoostWeightsS,tmpBoostWeightsS->GetTitle());
1797  results->Store(tmpBoostWeightsB,tmpBoostWeightsB->GetTitle());
1798 
1799  TH1F *tmpBoostWeights;
1800  std::vector<TH1F*> *h;
1801 
1802  for (UInt_t iev=0; iev < fEventSample.size(); iev++){
1803  if (fEventSample[iev]->GetClass() == signalClassNr) {
1804  tmpBoostWeights=tmpBoostWeightsS;
1805  h=&hS;
1806  }else{
1807  tmpBoostWeights=tmpBoostWeightsB;
1808  h=&hB;
1809  }
1810  tmpBoostWeights->Fill(fEventSample[iev]->GetBoostWeight());
1811  for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
1812  (*h)[ivar]->Fill(fEventSample[iev]->GetValue(ivar),fEventSample[iev]->GetWeight());
1813  }
1814  }
1815 
1816 
1817  TMVA::PDF *sig = new TMVA::PDF( " PDF Sig", tmpS, TMVA::PDF::kSpline3 );
1818  TMVA::PDF *bkg = new TMVA::PDF( " PDF Bkg", tmpB, TMVA::PDF::kSpline3 );
1819 
1820 
1821  TGraph* gr=results->GetGraph("BoostMonitorGraph");
1822  Int_t nPoints = gr->GetN();
1823  gr->Set(nPoints+1);
1824  gr->SetPoint(nPoints,(Double_t)iTree+1,GetROCIntegral(sig,bkg));
1825 
1826  tmpS->Delete();
1827  tmpB->Delete();
1828 
1829  delete sig;
1830  delete bkg;
1831 
1832  return;
1833 }
1834 
1835 ////////////////////////////////////////////////////////////////////////////////
1836 /// The AdaBoost implementation.
1837 /// a new training sample is generated by weighting
1838 /// events that are misclassified by the decision tree. The weight
1839 /// applied is \f$ w = \frac{(1-err)}{err} \f$ or more general:
1840 /// \f$ w = (\frac{(1-err)}{err})^\beta \f$
1841 /// where \f$err\f$ is the fraction of misclassified events in the tree ( <0.5 assuming
1842 /// demanding the that previous selection was better than random guessing)
1843 /// and "beta" being a free parameter (standard: beta = 1) that modifies the
1844 /// boosting.
1845 
1846 Double_t TMVA::MethodBDT::AdaBoost( std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
1847 {
1848  Double_t err=0, sumGlobalw=0, sumGlobalwfalse=0, sumGlobalwfalse2=0;
1849 
1850  std::vector<Double_t> sumw(DataInfo().GetNClasses(),0); //for individually re-scaling each class
1851 
1852  Double_t maxDev=0;
1853  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1854  Double_t w = (*e)->GetWeight();
1855  sumGlobalw += w;
1856  UInt_t iclass=(*e)->GetClass();
1857  sumw[iclass] += w;
1858 
1859  if ( DoRegression() ) {
1860  Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
1861  sumGlobalwfalse += w * tmpDev;
1862  sumGlobalwfalse2 += w * tmpDev*tmpDev;
1863  if (tmpDev > maxDev) maxDev = tmpDev;
1864  }else{
1865 
1866  if (fUseYesNoLeaf){
1867  Bool_t isSignalType = (dt->CheckEvent(*e,fUseYesNoLeaf) > fNodePurityLimit );
1868  if (!(isSignalType == DataInfo().IsSignal(*e))) {
1869  sumGlobalwfalse+= w;
1870  }
1871  }else{
1872  Double_t dtoutput = (dt->CheckEvent(*e,fUseYesNoLeaf) - 0.5)*2.;
1873  Int_t trueType;
1874  if (DataInfo().IsSignal(*e)) trueType = 1;
1875  else trueType = -1;
1876  sumGlobalwfalse+= w*trueType*dtoutput;
1877  }
1878  }
1879  }
1880 
1881  err = sumGlobalwfalse/sumGlobalw ;
1882  if ( DoRegression() ) {
1883  //if quadratic loss:
1884  if (fAdaBoostR2Loss=="linear"){
1885  err = sumGlobalwfalse/maxDev/sumGlobalw ;
1886  }
1887  else if (fAdaBoostR2Loss=="quadratic"){
1888  err = sumGlobalwfalse2/maxDev/maxDev/sumGlobalw ;
1889  }
1890  else if (fAdaBoostR2Loss=="exponential"){
1891  err = 0;
1892  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1893  Double_t w = (*e)->GetWeight();
1894  Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
1895  err += w * (1 - exp (-tmpDev/maxDev)) / sumGlobalw;
1896  }
1897 
1898  }
1899  else {
1900  Log() << kFATAL << " you've chosen a Loss type for Adaboost other than linear, quadratic or exponential "
1901  << " namely " << fAdaBoostR2Loss << "\n"
1902  << "and this is not implemented... a typo in the options ??" <<Endl;
1903  }
1904  }
1905 
1906  Log() << kDEBUG << "BDT AdaBoos wrong/all: " << sumGlobalwfalse << "/" << sumGlobalw << Endl;
1907 
1908 
1909  Double_t newSumGlobalw=0;
1910  std::vector<Double_t> newSumw(sumw.size(),0);
1911 
1912  Double_t boostWeight=1.;
1913  if (err >= 0.5 && fUseYesNoLeaf) { // sanity check ... should never happen as otherwise there is apparently
1914  // something odd with the assignment of the leaf nodes (rem: you use the training
1915  // events for this determination of the error rate)
1916  if (dt->GetNNodes() == 1){
1917  Log() << kERROR << " YOUR tree has only 1 Node... kind of a funny *tree*. I cannot "
1918  << "boost such a thing... if after 1 step the error rate is == 0.5"
1919  << Endl
1920  << "please check why this happens, maybe too many events per node requested ?"
1921  << Endl;
1922 
1923  }else{
1924  Log() << kERROR << " The error rate in the BDT boosting is > 0.5. ("<< err
1925  << ") That should not happen, please check your code (i.e... the BDT code), I "
1926  << " stop boosting here" << Endl;
1927  return -1;
1928  }
1929  err = 0.5;
1930  } else if (err < 0) {
1931  Log() << kERROR << " The error rate in the BDT boosting is < 0. That can happen"
1932  << " due to improper treatment of negative weights in a Monte Carlo.. (if you have"
1933  << " an idea on how to do it in a better way, please let me know (Helge.Voss@cern.ch)"
1934  << " for the time being I set it to its absolute value.. just to continue.." << Endl;
1935  err = TMath::Abs(err);
1936  }
1937  if (fUseYesNoLeaf)
1938  boostWeight = TMath::Log((1.-err)/err)*fAdaBoostBeta;
1939  else
1940  boostWeight = TMath::Log((1.+err)/(1-err))*fAdaBoostBeta;
1941 
1942 
1943  Log() << kDEBUG << "BDT AdaBoos wrong/all: " << sumGlobalwfalse << "/" << sumGlobalw << " 1-err/err="<<boostWeight<< " log.."<<TMath::Log(boostWeight)<<Endl;
1944 
1945  Results* results = Data()->GetResults(GetMethodName(),Types::kTraining, Types::kMaxAnalysisType);
1946 
1947 
1948  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1949 
1950  if (fUseYesNoLeaf||DoRegression()){
1951  if ((!( (dt->CheckEvent(*e,fUseYesNoLeaf) > fNodePurityLimit ) == DataInfo().IsSignal(*e))) || DoRegression()) {
1952  Double_t boostfactor = TMath::Exp(boostWeight);
1953 
1954  if (DoRegression()) boostfactor = TMath::Power(1/boostWeight,(1.-TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) )/maxDev ) );
1955  if ( (*e)->GetWeight() > 0 ){
1956  (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1957  // Helge change back (*e)->ScaleBoostWeight(boostfactor);
1958  if (DoRegression()) results->GetHist("BoostWeights")->Fill(boostfactor);
1959  } else {
1960  if ( fInverseBoostNegWeights )(*e)->ScaleBoostWeight( 1. / boostfactor); // if the original event weight is negative, and you want to "increase" the events "positive" influence, you'd rather make the event weight "smaller" in terms of it's absolute value while still keeping it something "negative"
1961  else (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1962 
1963  }
1964  }
1965 
1966  }else{
1967  Double_t dtoutput = (dt->CheckEvent(*e,fUseYesNoLeaf) - 0.5)*2.;
1968  Int_t trueType;
1969  if (DataInfo().IsSignal(*e)) trueType = 1;
1970  else trueType = -1;
1971  Double_t boostfactor = TMath::Exp(-1*boostWeight*trueType*dtoutput);
1972 
1973  if ( (*e)->GetWeight() > 0 ){
1974  (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1975  // Helge change back (*e)->ScaleBoostWeight(boostfactor);
1976  if (DoRegression()) results->GetHist("BoostWeights")->Fill(boostfactor);
1977  } else {
1978  if ( fInverseBoostNegWeights )(*e)->ScaleBoostWeight( 1. / boostfactor); // if the original event weight is negative, and you want to "increase" the events "positive" influence, you'd rather make the event weight "smaller" in terms of it's absolute value while still keeping it something "negative"
1979  else (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
1980  }
1981  }
1982  newSumGlobalw+=(*e)->GetWeight();
1983  newSumw[(*e)->GetClass()] += (*e)->GetWeight();
1984  }
1985 
1986 
1987  // Double_t globalNormWeight=sumGlobalw/newSumGlobalw;
1988  Double_t globalNormWeight=( (Double_t) eventSample.size())/newSumGlobalw;
1989  Log() << kDEBUG << "new Nsig="<<newSumw[0]*globalNormWeight << " new Nbkg="<<newSumw[1]*globalNormWeight << Endl;
1990 
1991 
1992  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
1993  // if (fRenormByClass) (*e)->ScaleBoostWeight( normWeightByClass[(*e)->GetClass()] );
1994  // else (*e)->ScaleBoostWeight( globalNormWeight );
1995  // else (*e)->ScaleBoostWeight( globalNormWeight );
1996  if (DataInfo().IsSignal(*e))(*e)->ScaleBoostWeight( globalNormWeight * fSigToBkgFraction );
1997  else (*e)->ScaleBoostWeight( globalNormWeight );
1998  }
1999 
2000  if (!(DoRegression()))results->GetHist("BoostWeights")->Fill(boostWeight);
2001  results->GetHist("BoostWeightsVsTree")->SetBinContent(fForest.size(),boostWeight);
2002  results->GetHist("ErrorFrac")->SetBinContent(fForest.size(),err);
2003 
2004  fBoostWeight = boostWeight;
2005  fErrorFraction = err;
2006 
2007  return boostWeight;
2008 }
2009 
2010 ////////////////////////////////////////////////////////////////////////////////
2011 /// The AdaCost boosting algorithm takes a simple cost Matrix (currently fixed for
2012 /// all events... later could be modified to use individual cost matrices for each
2013 /// events as in the original paper...
2014 ///
2015 /// true_signal true_bkg
2016 /// ----------------------------------
2017 /// sel_signal | Css Ctb_ss Cxx.. in the range [0,1]
2018 /// sel_bkg | Cts_sb Cbb
2019 ///
2020 /// and takes this into account when calculating the mis class. cost (former: error fraction):
2021 ///
2022 /// err = sum_events ( weight* y_true*y_sel * beta(event)
2023 
2024 Double_t TMVA::MethodBDT::AdaCost( vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
2025 {
2026  Double_t Css = fCss;
2027  Double_t Cbb = fCbb;
2028  Double_t Cts_sb = fCts_sb;
2029  Double_t Ctb_ss = fCtb_ss;
2030 
2031  Double_t err=0, sumGlobalWeights=0, sumGlobalCost=0;
2032 
2033  std::vector<Double_t> sumw(DataInfo().GetNClasses(),0); //for individually re-scaling each class
2034 
2035  for (vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2036  Double_t w = (*e)->GetWeight();
2037  sumGlobalWeights += w;
2038  UInt_t iclass=(*e)->GetClass();
2039 
2040  sumw[iclass] += w;
2041 
2042  if ( DoRegression() ) {
2043  Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2044  }else{
2045 
2046  Double_t dtoutput = (dt->CheckEvent(*e,false) - 0.5)*2.;
2047  Int_t trueType;
2048  Bool_t isTrueSignal = DataInfo().IsSignal(*e);
2049  Bool_t isSelectedSignal = (dtoutput>0);
2050  if (isTrueSignal) trueType = 1;
2051  else trueType = -1;
2052 
2053  Double_t cost=0;
2054  if (isTrueSignal && isSelectedSignal) cost=Css;
2055  else if (isTrueSignal && !isSelectedSignal) cost=Cts_sb;
2056  else if (!isTrueSignal && isSelectedSignal) cost=Ctb_ss;
2057  else if (!isTrueSignal && !isSelectedSignal) cost=Cbb;
2058  else Log() << kERROR << "something went wrong in AdaCost" << Endl;
2059 
2060  sumGlobalCost+= w*trueType*dtoutput*cost;
2061 
2062  }
2063  }
2064 
2065  if ( DoRegression() ) {
2066  Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2067  }
2068 
2069  // Log() << kDEBUG << "BDT AdaBoos wrong/all: " << sumGlobalCost << "/" << sumGlobalWeights << Endl;
2070  // Log() << kWARNING << "BDT AdaBoos wrong/all: " << sumGlobalCost << "/" << sumGlobalWeights << Endl;
2071  sumGlobalCost /= sumGlobalWeights;
2072  // Log() << kWARNING << "BDT AdaBoos wrong/all: " << sumGlobalCost << "/" << sumGlobalWeights << Endl;
2073 
2074 
2075  Double_t newSumGlobalWeights=0;
2076  vector<Double_t> newSumClassWeights(sumw.size(),0);
2077 
2078  Double_t boostWeight = TMath::Log((1+sumGlobalCost)/(1-sumGlobalCost)) * fAdaBoostBeta;
2079 
2080  Results* results = Data()->GetResults(GetMethodName(),Types::kTraining, Types::kMaxAnalysisType);
2081 
2082  for (vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2083  Double_t dtoutput = (dt->CheckEvent(*e,false) - 0.5)*2.;
2084  Int_t trueType;
2085  Bool_t isTrueSignal = DataInfo().IsSignal(*e);
2086  Bool_t isSelectedSignal = (dtoutput>0);
2087  if (isTrueSignal) trueType = 1;
2088  else trueType = -1;
2089 
2090  Double_t cost=0;
2091  if (isTrueSignal && isSelectedSignal) cost=Css;
2092  else if (isTrueSignal && !isSelectedSignal) cost=Cts_sb;
2093  else if (!isTrueSignal && isSelectedSignal) cost=Ctb_ss;
2094  else if (!isTrueSignal && !isSelectedSignal) cost=Cbb;
2095  else Log() << kERROR << "something went wrong in AdaCost" << Endl;
2096 
2097  Double_t boostfactor = TMath::Exp(-1*boostWeight*trueType*dtoutput*cost);
2098  if (DoRegression())Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2099  if ( (*e)->GetWeight() > 0 ){
2100  (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
2101  // Helge change back (*e)->ScaleBoostWeight(boostfactor);
2102  if (DoRegression())Log() << kFATAL << " AdaCost not implemented for regression"<<Endl;
2103  } else {
2104  if ( fInverseBoostNegWeights )(*e)->ScaleBoostWeight( 1. / boostfactor); // if the original event weight is negative, and you want to "increase" the events "positive" influence, you'd rather make the event weight "smaller" in terms of it's absolute value while still keeping it something "negative"
2105  }
2106 
2107  newSumGlobalWeights+=(*e)->GetWeight();
2108  newSumClassWeights[(*e)->GetClass()] += (*e)->GetWeight();
2109  }
2110 
2111 
2112  // Double_t globalNormWeight=sumGlobalWeights/newSumGlobalWeights;
2113  Double_t globalNormWeight=Double_t(eventSample.size())/newSumGlobalWeights;
2114  Log() << kDEBUG << "new Nsig="<<newSumClassWeights[0]*globalNormWeight << " new Nbkg="<<newSumClassWeights[1]*globalNormWeight << Endl;
2115 
2116 
2117  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2118  // if (fRenormByClass) (*e)->ScaleBoostWeight( normWeightByClass[(*e)->GetClass()] );
2119  // else (*e)->ScaleBoostWeight( globalNormWeight );
2120  if (DataInfo().IsSignal(*e))(*e)->ScaleBoostWeight( globalNormWeight * fSigToBkgFraction );
2121  else (*e)->ScaleBoostWeight( globalNormWeight );
2122  }
2123 
2124 
2125  if (!(DoRegression()))results->GetHist("BoostWeights")->Fill(boostWeight);
2126  results->GetHist("BoostWeightsVsTree")->SetBinContent(fForest.size(),boostWeight);
2127  results->GetHist("ErrorFrac")->SetBinContent(fForest.size(),err);
2128 
2129  fBoostWeight = boostWeight;
2130  fErrorFraction = err;
2131 
2132 
2133  return boostWeight;
2134 }
2135 
2136 ////////////////////////////////////////////////////////////////////////////////
2137 /// Call it boot-strapping, re-sampling or whatever you like, in the end it is nothing
2138 /// else but applying "random" poisson weights to each event.
2139 
2141 {
2142  // this is now done in "MethodBDT::Boost as it might be used by other boost methods, too
2143  // GetBaggedSample(eventSample);
2144 
2145  return 1.; //here as there are random weights for each event, just return a constant==1;
2146 }
2147 
2148 ////////////////////////////////////////////////////////////////////////////////
2149 /// Fills fEventSample with fBaggedSampleFraction*NEvents random training events.
2150 
2151 void TMVA::MethodBDT::GetBaggedSubSample(std::vector<const TMVA::Event*>& eventSample)
2152 {
2153 
2154  Double_t n;
2155  TRandom3 *trandom = new TRandom3(100*fForest.size()+1234);
2156 
2157  if (!fSubSample.empty()) fSubSample.clear();
2158 
2159  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2160  n = trandom->PoissonD(fBaggedSampleFraction);
2161  for (Int_t i=0;i<n;i++) fSubSample.push_back(*e);
2162  }
2163 
2164  delete trandom;
2165  return;
2166 
2167  /*
2168  UInt_t nevents = fEventSample.size();
2169 
2170  if (!fSubSample.empty()) fSubSample.clear();
2171  TRandom3 *trandom = new TRandom3(fForest.size()+1);
2172 
2173  for (UInt_t ievt=0; ievt<nevents; ievt++) { // recreate new random subsample
2174  if(trandom->Rndm()<fBaggedSampleFraction)
2175  fSubSample.push_back(fEventSample[ievt]);
2176  }
2177  delete trandom;
2178  */
2179 
2180 }
2181 
2182 ////////////////////////////////////////////////////////////////////////////////
2183 /// A special boosting only for Regression (not implemented).
2184 
2185 Double_t TMVA::MethodBDT::RegBoost( std::vector<const TMVA::Event*>& /* eventSample */, DecisionTree* /* dt */ )
2186 {
2187  return 1;
2188 }
2189 
2190 ////////////////////////////////////////////////////////////////////////////////
2191 /// Adaption of the AdaBoost to regression problems (see H.Drucker 1997).
2192 
2193 Double_t TMVA::MethodBDT::AdaBoostR2( std::vector<const TMVA::Event*>& eventSample, DecisionTree *dt )
2194 {
2195  if ( !DoRegression() ) Log() << kFATAL << "Somehow you chose a regression boost method for a classification job" << Endl;
2196 
2197  Double_t err=0, sumw=0, sumwfalse=0, sumwfalse2=0;
2198  Double_t maxDev=0;
2199  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2200  Double_t w = (*e)->GetWeight();
2201  sumw += w;
2202 
2203  Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
2204  sumwfalse += w * tmpDev;
2205  sumwfalse2 += w * tmpDev*tmpDev;
2206  if (tmpDev > maxDev) maxDev = tmpDev;
2207  }
2208 
2209  //if quadratic loss:
2210  if (fAdaBoostR2Loss=="linear"){
2211  err = sumwfalse/maxDev/sumw ;
2212  }
2213  else if (fAdaBoostR2Loss=="quadratic"){
2214  err = sumwfalse2/maxDev/maxDev/sumw ;
2215  }
2216  else if (fAdaBoostR2Loss=="exponential"){
2217  err = 0;
2218  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2219  Double_t w = (*e)->GetWeight();
2220  Double_t tmpDev = TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) );
2221  err += w * (1 - exp (-tmpDev/maxDev)) / sumw;
2222  }
2223 
2224  }
2225  else {
2226  Log() << kFATAL << " you've chosen a Loss type for Adaboost other than linear, quadratic or exponential "
2227  << " namely " << fAdaBoostR2Loss << "\n"
2228  << "and this is not implemented... a typo in the options ??" <<Endl;
2229  }
2230 
2231 
2232  if (err >= 0.5) { // sanity check ... should never happen as otherwise there is apparently
2233  // something odd with the assignment of the leaf nodes (rem: you use the training
2234  // events for this determination of the error rate)
2235  if (dt->GetNNodes() == 1){
2236  Log() << kERROR << " YOUR tree has only 1 Node... kind of a funny *tree*. I cannot "
2237  << "boost such a thing... if after 1 step the error rate is == 0.5"
2238  << Endl
2239  << "please check why this happens, maybe too many events per node requested ?"
2240  << Endl;
2241 
2242  }else{
2243  Log() << kERROR << " The error rate in the BDT boosting is > 0.5. ("<< err
2244  << ") That should not happen, but is possible for regression trees, and"
2245  << " should trigger a stop for the boosting. please check your code (i.e... the BDT code), I "
2246  << " stop boosting " << Endl;
2247  return -1;
2248  }
2249  err = 0.5;
2250  } else if (err < 0) {
2251  Log() << kERROR << " The error rate in the BDT boosting is < 0. That can happen"
2252  << " due to improper treatment of negative weights in a Monte Carlo.. (if you have"
2253  << " an idea on how to do it in a better way, please let me know (Helge.Voss@cern.ch)"
2254  << " for the time being I set it to its absolute value.. just to continue.." << Endl;
2255  err = TMath::Abs(err);
2256  }
2257 
2258  Double_t boostWeight = err / (1.-err);
2259  Double_t newSumw=0;
2260 
2261  Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, Types::kMaxAnalysisType);
2262 
2263  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2264  Double_t boostfactor = TMath::Power(boostWeight,(1.-TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) )/maxDev ) );
2265  results->GetHist("BoostWeights")->Fill(boostfactor);
2266  // std::cout << "R2 " << boostfactor << " " << boostWeight << " " << (1.-TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) )/maxDev) << std::endl;
2267  if ( (*e)->GetWeight() > 0 ){
2268  Float_t newBoostWeight = (*e)->GetBoostWeight() * boostfactor;
2269  Float_t newWeight = (*e)->GetWeight() * (*e)->GetBoostWeight() * boostfactor;
2270  if (newWeight == 0) {
2271  Log() << kINFO << "Weight= " << (*e)->GetWeight() << Endl;
2272  Log() << kINFO << "BoostWeight= " << (*e)->GetBoostWeight() << Endl;
2273  Log() << kINFO << "boostweight="<<boostWeight << " err= " <<err << Endl;
2274  Log() << kINFO << "NewBoostWeight= " << newBoostWeight << Endl;
2275  Log() << kINFO << "boostfactor= " << boostfactor << Endl;
2276  Log() << kINFO << "maxDev = " << maxDev << Endl;
2277  Log() << kINFO << "tmpDev = " << TMath::Abs(dt->CheckEvent(*e,kFALSE) - (*e)->GetTarget(0) ) << Endl;
2278  Log() << kINFO << "target = " << (*e)->GetTarget(0) << Endl;
2279  Log() << kINFO << "estimate = " << dt->CheckEvent(*e,kFALSE) << Endl;
2280  }
2281  (*e)->SetBoostWeight( newBoostWeight );
2282  // (*e)->SetBoostWeight( (*e)->GetBoostWeight() * boostfactor);
2283  } else {
2284  (*e)->SetBoostWeight( (*e)->GetBoostWeight() / boostfactor);
2285  }
2286  newSumw+=(*e)->GetWeight();
2287  }
2288 
2289  // re-normalise the weights
2290  Double_t normWeight = sumw / newSumw;
2291  for (std::vector<const TMVA::Event*>::const_iterator e=eventSample.begin(); e!=eventSample.end();++e) {
2292  //Helge (*e)->ScaleBoostWeight( sumw/newSumw);
2293  // (*e)->ScaleBoostWeight( normWeight);
2294  (*e)->SetBoostWeight( (*e)->GetBoostWeight() * normWeight );
2295  }
2296 
2297 
2298  results->GetHist("BoostWeightsVsTree")->SetBinContent(fForest.size(),1./boostWeight);
2299  results->GetHist("ErrorFrac")->SetBinContent(fForest.size(),err);
2300 
2301  fBoostWeight = boostWeight;
2302  fErrorFraction = err;
2303 
2304  return TMath::Log(1./boostWeight);
2305 }
2306 
2307 ////////////////////////////////////////////////////////////////////////////////
2308 /// Write weights to XML.
2309 
2310 void TMVA::MethodBDT::AddWeightsXMLTo( void* parent ) const
2311 {
2312  void* wght = gTools().AddChild(parent, "Weights");
2313 
2314  if (fDoPreselection){
2315  for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
2316  gTools().AddAttr( wght, Form("PreselectionLowBkgVar%d",ivar), fIsLowBkgCut[ivar]);
2317  gTools().AddAttr( wght, Form("PreselectionLowBkgVar%dValue",ivar), fLowBkgCut[ivar]);
2318  gTools().AddAttr( wght, Form("PreselectionLowSigVar%d",ivar), fIsLowSigCut[ivar]);
2319  gTools().AddAttr( wght, Form("PreselectionLowSigVar%dValue",ivar), fLowSigCut[ivar]);
2320  gTools().AddAttr( wght, Form("PreselectionHighBkgVar%d",ivar), fIsHighBkgCut[ivar]);
2321  gTools().AddAttr( wght, Form("PreselectionHighBkgVar%dValue",ivar),fHighBkgCut[ivar]);
2322  gTools().AddAttr( wght, Form("PreselectionHighSigVar%d",ivar), fIsHighSigCut[ivar]);
2323  gTools().AddAttr( wght, Form("PreselectionHighSigVar%dValue",ivar),fHighSigCut[ivar]);
2324  }
2325  }
2326 
2327 
2328  gTools().AddAttr( wght, "NTrees", fForest.size() );
2329  gTools().AddAttr( wght, "AnalysisType", fForest.back()->GetAnalysisType() );
2330 
2331  for (UInt_t i=0; i< fForest.size(); i++) {
2332  void* trxml = fForest[i]->AddXMLTo(wght);
2333  gTools().AddAttr( trxml, "boostWeight", fBoostWeights[i] );
2334  gTools().AddAttr( trxml, "itree", i );
2335  }
2336 }
2337 
2338 ////////////////////////////////////////////////////////////////////////////////
2339 /// Reads the BDT from the xml file.
2340 
2342  UInt_t i;
2343  for (i=0; i<fForest.size(); i++) delete fForest[i];
2344  fForest.clear();
2345  fBoostWeights.clear();
2346 
2347  UInt_t ntrees;
2348  UInt_t analysisType;
2349  Float_t boostWeight;
2350 
2351 
2352  if (gTools().HasAttr( parent, Form("PreselectionLowBkgVar%d",0))) {
2353  fIsLowBkgCut.resize(GetNvar());
2354  fLowBkgCut.resize(GetNvar());
2355  fIsLowSigCut.resize(GetNvar());
2356  fLowSigCut.resize(GetNvar());
2357  fIsHighBkgCut.resize(GetNvar());
2358  fHighBkgCut.resize(GetNvar());
2359  fIsHighSigCut.resize(GetNvar());
2360  fHighSigCut.resize(GetNvar());
2361 
2362  Bool_t tmpBool;
2363  Double_t tmpDouble;
2364  for (UInt_t ivar=0; ivar<GetNvar(); ivar++){
2365  gTools().ReadAttr( parent, Form("PreselectionLowBkgVar%d",ivar), tmpBool);
2366  fIsLowBkgCut[ivar]=tmpBool;
2367  gTools().ReadAttr( parent, Form("PreselectionLowBkgVar%dValue",ivar), tmpDouble);
2368  fLowBkgCut[ivar]=tmpDouble;
2369  gTools().ReadAttr( parent, Form("PreselectionLowSigVar%d",ivar), tmpBool);
2370  fIsLowSigCut[ivar]=tmpBool;
2371  gTools().ReadAttr( parent, Form("PreselectionLowSigVar%dValue",ivar), tmpDouble);
2372  fLowSigCut[ivar]=tmpDouble;
2373  gTools().ReadAttr( parent, Form("PreselectionHighBkgVar%d",ivar), tmpBool);
2374  fIsHighBkgCut[ivar]=tmpBool;
2375  gTools().ReadAttr( parent, Form("PreselectionHighBkgVar%dValue",ivar), tmpDouble);
2376  fHighBkgCut[ivar]=tmpDouble;
2377  gTools().ReadAttr( parent, Form("PreselectionHighSigVar%d",ivar),tmpBool);
2378  fIsHighSigCut[ivar]=tmpBool;
2379  gTools().ReadAttr( parent, Form("PreselectionHighSigVar%dValue",ivar), tmpDouble);
2380  fHighSigCut[ivar]=tmpDouble;
2381  }
2382  }
2383 
2384 
2385  gTools().ReadAttr( parent, "NTrees", ntrees );
2386 
2387  if(gTools().HasAttr(parent, "TreeType")) { // pre 4.1.0 version
2388  gTools().ReadAttr( parent, "TreeType", analysisType );
2389  } else { // from 4.1.0 onwards
2390  gTools().ReadAttr( parent, "AnalysisType", analysisType );
2391  }
2392 
2393  void* ch = gTools().GetChild(parent);
2394  i=0;
2395  while(ch) {
2396  fForest.push_back( dynamic_cast<DecisionTree*>( DecisionTree::CreateFromXML(ch, GetTrainingTMVAVersionCode()) ) );
2397  fForest.back()->SetAnalysisType(Types::EAnalysisType(analysisType));
2398  fForest.back()->SetTreeID(i++);
2399  gTools().ReadAttr(ch,"boostWeight",boostWeight);
2400  fBoostWeights.push_back(boostWeight);
2401  ch = gTools().GetNextChild(ch);
2402  }
2403 }
2404 
2405 ////////////////////////////////////////////////////////////////////////////////
2406 /// Read the weights (BDT coefficients).
2407 
2408 void TMVA::MethodBDT::ReadWeightsFromStream( std::istream& istr )
2409 {
2410  TString dummy;
2411  // Types::EAnalysisType analysisType;
2412  Int_t analysisType(0);
2413 
2414  // coverity[tainted_data_argument]
2415  istr >> dummy >> fNTrees;
2416  Log() << kINFO << "Read " << fNTrees << " Decision trees" << Endl;
2417 
2418  for (UInt_t i=0;i<fForest.size();i++) delete fForest[i];
2419  fForest.clear();
2420  fBoostWeights.clear();
2421  Int_t iTree;
2422  Double_t boostWeight;
2423  for (int i=0;i<fNTrees;i++) {
2424  istr >> dummy >> iTree >> dummy >> boostWeight;
2425  if (iTree != i) {
2426  fForest.back()->Print( std::cout );
2427  Log() << kFATAL << "Error while reading weight file; mismatch iTree="
2428  << iTree << " i=" << i
2429  << " dummy " << dummy
2430  << " boostweight " << boostWeight
2431  << Endl;
2432  }
2433  fForest.push_back( new DecisionTree() );
2434  fForest.back()->SetAnalysisType(Types::EAnalysisType(analysisType));
2435  fForest.back()->SetTreeID(i);
2436  fForest.back()->Read(istr, GetTrainingTMVAVersionCode());
2437  fBoostWeights.push_back(boostWeight);
2438  }
2439 }
2440 
2441 ////////////////////////////////////////////////////////////////////////////////
2442 
2444  return this->GetMvaValue( err, errUpper, 0 );
2445 }
2446 
2447 ////////////////////////////////////////////////////////////////////////////////
2448 /// Return the MVA value (range [-1;1]) that classifies the
2449 /// event according to the majority vote from the total number of
2450 /// decision trees.
2451 
2453 {
2454  const Event* ev = GetEvent();
2455  if (fDoPreselection) {
2456  Double_t val = ApplyPreselectionCuts(ev);
2457  if (TMath::Abs(val)>0.05) return val;
2458  }
2459  return PrivateGetMvaValue(ev, err, errUpper, useNTrees);
2460 
2461 }
2462 
2463 ////////////////////////////////////////////////////////////////////////////////
2464 /// Return the MVA value (range [-1;1]) that classifies the
2465 /// event according to the majority vote from the total number of
2466 /// decision trees.
2467 
2469 {
2470  // cannot determine error
2471  NoErrorCalc(err, errUpper);
2472 
2473  // allow for the possibility to use less trees in the actual MVA calculation
2474  // than have been originally trained.
2475  UInt_t nTrees = fForest.size();
2476 
2477  if (useNTrees > 0 ) nTrees = useNTrees;
2478 
2479  if (fBoostType=="Grad") return GetGradBoostMVA(ev,nTrees);
2480 
2481  Double_t myMVA = 0;
2482  Double_t norm = 0;
2483  for (UInt_t itree=0; itree<nTrees; itree++) {
2484  //
2485  myMVA += fBoostWeights[itree] * fForest[itree]->CheckEvent(ev,fUseYesNoLeaf);
2486  norm += fBoostWeights[itree];
2487  }
2488  return ( norm > std::numeric_limits<double>::epsilon() ) ? myMVA /= norm : 0 ;
2489 }
2490 
2491 
2492 ////////////////////////////////////////////////////////////////////////////////
2493 /// Get the multiclass MVA response for the BDT classifier.
2494 
2495 const std::vector<Float_t>& TMVA::MethodBDT::GetMulticlassValues()
2496 {
2497  const TMVA::Event *e = GetEvent();
2498  if (fMulticlassReturnVal == NULL) fMulticlassReturnVal = new std::vector<Float_t>();
2499  fMulticlassReturnVal->clear();
2500 
2501  UInt_t nClasses = DataInfo().GetNClasses();
2502  std::vector<Double_t> temp(nClasses);
2503  auto forestSize = fForest.size();
2504 
2505  #ifdef R__USE_IMT
2506  std::vector<TMVA::DecisionTree *> forest = fForest;
2507  auto get_output = [&e, &forest, &temp, forestSize, nClasses](UInt_t iClass) {
2508  for (UInt_t itree = iClass; itree < forestSize; itree += nClasses) {
2509  temp[iClass] += forest[itree]->CheckEvent(e, kFALSE);
2510  }
2511  };
2512 
2514  .Foreach(get_output, ROOT::TSeqU(nClasses));
2515  #else
2516  // trees 0, nClasses, 2*nClasses, ... belong to class 0
2517  // trees 1, nClasses+1, 2*nClasses+1, ... belong to class 1 and so forth
2518  UInt_t classOfTree = 0;
2519  for (UInt_t itree = 0; itree < forestSize; ++itree) {
2520  temp[classOfTree] += fForest[itree]->CheckEvent(e, kFALSE);
2521  if (++classOfTree == nClasses) classOfTree = 0; // cheap modulo
2522  }
2523  #endif
2524 
2525  // we want to calculate sum of exp(temp[j] - temp[i]) for all i,j (i!=j)
2526  // first calculate exp(), then replace minus with division.
2527  std::transform(temp.begin(), temp.end(), temp.begin(), [](Double_t d){return exp(d);});
2528 
2529  Double_t exp_sum = std::accumulate(temp.begin(), temp.end(), 0.0);
2530 
2531  for (UInt_t i = 0; i < nClasses; i++) {
2532  Double_t p_cls = temp[i] / exp_sum;
2533  (*fMulticlassReturnVal).push_back(p_cls);
2534  }
2535 
2536  return *fMulticlassReturnVal;
2537 }
2538 
2539 ////////////////////////////////////////////////////////////////////////////////
2540 /// Get the regression value generated by the BDTs.
2541 
2542 const std::vector<Float_t> & TMVA::MethodBDT::GetRegressionValues()
2543 {
2544 
2545  if (fRegressionReturnVal == NULL) fRegressionReturnVal = new std::vector<Float_t>();
2546  fRegressionReturnVal->clear();
2547 
2548  const Event * ev = GetEvent();
2549  Event * evT = new Event(*ev);
2550 
2551  Double_t myMVA = 0;
2552  Double_t norm = 0;
2553  if (fBoostType=="AdaBoostR2") {
2554  // rather than using the weighted average of the tree respones in the forest
2555  // H.Decker(1997) proposed to use the "weighted median"
2556 
2557  // sort all individual tree responses according to the prediction value
2558  // (keep the association to their tree weight)
2559  // the sum up all the associated weights (starting from the one whose tree
2560  // yielded the smalles response) up to the tree "t" at which you've
2561  // added enough tree weights to have more than half of the sum of all tree weights.
2562  // choose as response of the forest that one which belongs to this "t"
2563 
2564  vector< Double_t > response(fForest.size());
2565  vector< Double_t > weight(fForest.size());
2566  Double_t totalSumOfWeights = 0;
2567 
2568  for (UInt_t itree=0; itree<fForest.size(); itree++) {
2569  response[itree] = fForest[itree]->CheckEvent(ev,kFALSE);
2570  weight[itree] = fBoostWeights[itree];
2571  totalSumOfWeights += fBoostWeights[itree];
2572  }
2573 
2574  std::vector< std::vector<Double_t> > vtemp;
2575  vtemp.push_back( response ); // this is the vector that will get sorted
2576  vtemp.push_back( weight );
2577  gTools().UsefulSortAscending( vtemp );
2578 
2579  Int_t t=0;
2580  Double_t sumOfWeights = 0;
2581  while (sumOfWeights <= totalSumOfWeights/2.) {
2582  sumOfWeights += vtemp[1][t];
2583  t++;
2584  }
2585 
2586  Double_t rVal=0;
2587  Int_t count=0;
2588  for (UInt_t i= TMath::Max(UInt_t(0),UInt_t(t-(fForest.size()/6)-0.5));
2589  i< TMath::Min(UInt_t(fForest.size()),UInt_t(t+(fForest.size()/6)+0.5)); i++) {
2590  count++;
2591  rVal+=vtemp[0][i];
2592  }
2593  // fRegressionReturnVal->push_back( rVal/Double_t(count));
2594  evT->SetTarget(0, rVal/Double_t(count) );
2595  }
2596  else if(fBoostType=="Grad"){
2597  for (UInt_t itree=0; itree<fForest.size(); itree++) {
2598  myMVA += fForest[itree]->CheckEvent(ev,kFALSE);
2599  }
2600  // fRegressionReturnVal->push_back( myMVA+fBoostWeights[0]);
2601  evT->SetTarget(0, myMVA+fBoostWeights[0] );
2602  }
2603  else{
2604  for (UInt_t itree=0; itree<fForest.size(); itree++) {
2605  //
2606  myMVA += fBoostWeights[itree] * fForest[itree]->CheckEvent(ev,kFALSE);
2607  norm += fBoostWeights[itree];
2608  }
2609  // fRegressionReturnVal->push_back( ( norm > std::numeric_limits<double>::epsilon() ) ? myMVA /= norm : 0 );
2610  evT->SetTarget(0, ( norm > std::numeric_limits<double>::epsilon() ) ? myMVA /= norm : 0 );
2611  }
2612 
2613 
2614 
2615  const Event* evT2 = GetTransformationHandler().InverseTransform( evT );
2616  fRegressionReturnVal->push_back( evT2->GetTarget(0) );
2617 
2618  delete evT;
2619 
2620 
2621  return *fRegressionReturnVal;
2622 }
2623 
2624 ////////////////////////////////////////////////////////////////////////////////
2625 /// Here we could write some histograms created during the processing
2626 /// to the output file.
2627 
2629 {
2630  Log() << kDEBUG << "\tWrite monitoring histograms to file: " << BaseDir()->GetPath() << Endl;
2631 
2632  //Results* results = Data()->GetResults(GetMethodName(), Types::kTraining, Types::kMaxAnalysisType);
2633  //results->GetStorage()->Write();
2634  fMonitorNtuple->Write();
2635 }
2636 
2637 ////////////////////////////////////////////////////////////////////////////////
2638 /// Return the relative variable importance, normalized to all
2639 /// variables together having the importance 1. The importance in
2640 /// evaluated as the total separation-gain that this variable had in
2641 /// the decision trees (weighted by the number of events)
2642 
2644 {
2645  fVariableImportance.resize(GetNvar());
2646  for (UInt_t ivar = 0; ivar < GetNvar(); ivar++) {
2647  fVariableImportance[ivar]=0;
2648  }
2649  Double_t sum=0;
2650  for (UInt_t itree = 0; itree < GetNTrees(); itree++) {
2651  std::vector<Double_t> relativeImportance(fForest[itree]->GetVariableImportance());
2652  for (UInt_t i=0; i< relativeImportance.size(); i++) {
2653  fVariableImportance[i] += fBoostWeights[itree] * relativeImportance[i];
2654  }
2655  }
2656 
2657  for (UInt_t ivar=0; ivar< fVariableImportance.size(); ivar++){
2658  fVariableImportance[ivar] = TMath::Sqrt(fVariableImportance[ivar]);
2659  sum += fVariableImportance[ivar];
2660  }
2661  for (UInt_t ivar=0; ivar< fVariableImportance.size(); ivar++) fVariableImportance[ivar] /= sum;
2662 
2663  return fVariableImportance;
2664 }
2665 
2666 ////////////////////////////////////////////////////////////////////////////////
2667 /// Returns the measure for the variable importance of variable "ivar"
2668 /// which is later used in GetVariableImportance() to calculate the
2669 /// relative variable importances.
2670 
2672 {
2673  std::vector<Double_t> relativeImportance = this->GetVariableImportance();
2674  if (ivar < (UInt_t)relativeImportance.size()) return relativeImportance[ivar];
2675  else Log() << kFATAL << "<GetVariableImportance> ivar = " << ivar << " is out of range " << Endl;
2676 
2677  return -1;
2678 }
2679 
2680 ////////////////////////////////////////////////////////////////////////////////
2681 /// Compute ranking of input variables
2682 
2684 {
2685  // create the ranking object
2686  fRanking = new Ranking( GetName(), "Variable Importance" );
2687  vector< Double_t> importance(this->GetVariableImportance());
2688 
2689  for (UInt_t ivar=0; ivar<GetNvar(); ivar++) {
2690 
2691  fRanking->AddRank( Rank( GetInputLabel(ivar), importance[ivar] ) );
2692  }
2693 
2694  return fRanking;
2695 }
2696 
2697 ////////////////////////////////////////////////////////////////////////////////
2698 /// Get help message text.
2699 
2701 {
2702  Log() << Endl;
2703  Log() << gTools().Color("bold") << "--- Short description:" << gTools().Color("reset") << Endl;
2704  Log() << Endl;
2705  Log() << "Boosted Decision Trees are a collection of individual decision" << Endl;
2706  Log() << "trees which form a multivariate classifier by (weighted) majority " << Endl;
2707  Log() << "vote of the individual trees. Consecutive decision trees are " << Endl;
2708  Log() << "trained using the original training data set with re-weighted " << Endl;
2709  Log() << "events. By default, the AdaBoost method is employed, which gives " << Endl;
2710  Log() << "events that were misclassified in the previous tree a larger " << Endl;
2711  Log() << "weight in the training of the following tree." << Endl;
2712  Log() << Endl;
2713  Log() << "Decision trees are a sequence of binary splits of the data sample" << Endl;
2714  Log() << "using a single discriminant variable at a time. A test event " << Endl;
2715  Log() << "ending up after the sequence of left-right splits in a final " << Endl;
2716  Log() << "(\"leaf\") node is classified as either signal or background" << Endl;
2717  Log() << "depending on the majority type of training events in that node." << Endl;
2718  Log() << Endl;
2719  Log() << gTools().Color("bold") << "--- Performance optimisation:" << gTools().Color("reset") << Endl;
2720  Log() << Endl;
2721  Log() << "By the nature of the binary splits performed on the individual" << Endl;
2722  Log() << "variables, decision trees do not deal well with linear correlations" << Endl;
2723  Log() << "between variables (they need to approximate the linear split in" << Endl;
2724  Log() << "the two dimensional space by a sequence of splits on the two " << Endl;
2725  Log() << "variables individually). Hence decorrelation could be useful " << Endl;
2726  Log() << "to optimise the BDT performance." << Endl;
2727  Log() << Endl;
2728  Log() << gTools().Color("bold") << "--- Performance tuning via configuration options:" << gTools().Color("reset") << Endl;
2729  Log() << Endl;
2730  Log() << "The two most important parameters in the configuration are the " << Endl;
2731  Log() << "minimal number of events requested by a leaf node as percentage of the " <<Endl;
2732  Log() << " number of training events (option \"MinNodeSize\" replacing the actual number " << Endl;
2733  Log() << " of events \"nEventsMin\" as given in earlier versions" << Endl;
2734  Log() << "If this number is too large, detailed features " << Endl;
2735  Log() << "in the parameter space are hard to be modelled. If it is too small, " << Endl;
2736  Log() << "the risk to overtrain rises and boosting seems to be less effective" << Endl;
2737  Log() << " typical values from our current experience for best performance " << Endl;
2738  Log() << " are between 0.5(%) and 10(%) " << Endl;
2739  Log() << Endl;
2740  Log() << "The default minimal number is currently set to " << Endl;
2741  Log() << " max(20, (N_training_events / N_variables^2 / 10)) " << Endl;
2742  Log() << "and can be changed by the user." << Endl;
2743  Log() << Endl;
2744  Log() << "The other crucial parameter, the pruning strength (\"PruneStrength\")," << Endl;
2745  Log() << "is also related to overtraining. It is a regularisation parameter " << Endl;
2746  Log() << "that is used when determining after the training which splits " << Endl;
2747  Log() << "are considered statistically insignificant and are removed. The" << Endl;
2748  Log() << "user is advised to carefully watch the BDT screen output for" << Endl;
2749  Log() << "the comparison between efficiencies obtained on the training and" << Endl;
2750  Log() << "the independent test sample. They should be equal within statistical" << Endl;
2751  Log() << "errors, in order to minimize statistical fluctuations in different samples." << Endl;
2752 }
2753 
2754 ////////////////////////////////////////////////////////////////////////////////
2755 /// Make ROOT-independent C++ class for classifier response (classifier-specific implementation).
2756 
2757 void TMVA::MethodBDT::MakeClassSpecific( std::ostream& fout, const TString& className ) const
2758 {
2759  TString nodeName = className;
2760  nodeName.ReplaceAll("Read","");
2761  nodeName.Append("Node");
2762  // write BDT-specific classifier response
2763  fout << " std::vector<"<<nodeName<<"*> fForest; // i.e. root nodes of decision trees" << std::endl;
2764  fout << " std::vector<double> fBoostWeights; // the weights applied in the individual boosts" << std::endl;
2765  fout << "};" << std::endl << std::endl;
2766 
2767  if(GetAnalysisType() == Types::kMulticlass) {
2768  fout << "std::vector<double> ReadBDTG::GetMulticlassValues__( const std::vector<double>& inputValues ) const" << std::endl;
2769  fout << "{" << std::endl;
2770  fout << " uint nClasses = " << DataInfo().GetNClasses() << ";" << std::endl;
2771  fout << " std::vector<double> fMulticlassReturnVal;" << std::endl;
2772  fout << " fMulticlassReturnVal.reserve(nClasses);" << std::endl;
2773  fout << std::endl;
2774  fout << " std::vector<double> temp(nClasses);" << std::endl;
2775  fout << " auto forestSize = fForest.size();" << std::endl;
2776  fout << " // trees 0, nClasses, 2*nClasses, ... belong to class 0" << std::endl;
2777  fout << " // trees 1, nClasses+1, 2*nClasses+1, ... belong to class 1 and so forth" << std::endl;
2778  fout << " uint classOfTree = 0;" << std::endl;
2779  fout << " for (uint itree = 0; itree < forestSize; ++itree) {" << std::endl;
2780  fout << " BDTGNode *current = fForest[itree];" << std::endl;
2781  fout << " while (current->GetNodeType() == 0) { //intermediate node" << std::endl;
2782  fout << " if (current->GoesRight(inputValues)) current=(BDTGNode*)current->GetRight();" << std::endl;
2783  fout << " else current=(BDTGNode*)current->GetLeft();" << std::endl;
2784  fout << " }" << std::endl;
2785  fout << " temp[classOfTree] += current->GetResponse();" << std::endl;
2786  fout << " if (++classOfTree == nClasses) classOfTree = 0; // cheap modulo" << std::endl;
2787  fout << " }" << std::endl;
2788  fout << std::endl;
2789  fout << " // we want to calculate sum of exp(temp[j] - temp[i]) for all i,j (i!=j)" << std::endl;
2790  fout << " // first calculate exp(), then replace minus with division." << std::endl;
2791  fout << " std::transform(temp.begin(), temp.end(), temp.begin(), [](double d){return exp(d);});" << std::endl;
2792  fout << std::endl;
2793  fout << " for(uint iClass=0; iClass<nClasses; iClass++){" << std::endl;
2794  fout << " double norm = 0.0;" << std::endl;
2795  fout << " for(uint j=0;j<nClasses;j++){" << std::endl;
2796  fout << " if(iClass!=j)" << std::endl;
2797  fout << " norm += temp[j] / temp[iClass];" << std::endl;
2798  fout << " }" << std::endl;
2799  fout << " fMulticlassReturnVal.push_back(1.0/(1.0+norm));" << std::endl;
2800  fout << " }" << std::endl;
2801  fout << std::endl;
2802  fout << " return fMulticlassReturnVal;" << std::endl;
2803  fout << "}" << std::endl;
2804  } else {
2805  fout << "double " << className << "::GetMvaValue__( const std::vector<double>& inputValues ) const" << std::endl;
2806  fout << "{" << std::endl;
2807  fout << " double myMVA = 0;" << std::endl;
2808  if (fDoPreselection){
2809  for (UInt_t ivar = 0; ivar< fIsLowBkgCut.size(); ivar++){
2810  if (fIsLowBkgCut[ivar]){
2811  fout << " if (inputValues["<<ivar<<"] < " << fLowBkgCut[ivar] << ") return -1; // is background preselection cut" << std::endl;
2812  }
2813  if (fIsLowSigCut[ivar]){
2814  fout << " if (inputValues["<<ivar<<"] < "<< fLowSigCut[ivar] << ") return 1; // is signal preselection cut" << std::endl;
2815  }
2816  if (fIsHighBkgCut[ivar]){
2817  fout << " if (inputValues["<<ivar<<"] > "<<fHighBkgCut[ivar] <<") return -1; // is background preselection cut" << std::endl;
2818  }
2819  if (fIsHighSigCut[ivar]){
2820  fout << " if (inputValues["<<ivar<<"] > "<<fHighSigCut[ivar]<<") return 1; // is signal preselection cut" << std::endl;
2821  }
2822  }
2823  }
2824 
2825  if (fBoostType!="Grad"){
2826  fout << " double norm = 0;" << std::endl;
2827  }
2828  fout << " for (unsigned int itree=0; itree<fForest.size(); itree++){" << std::endl;
2829  fout << " "<<nodeName<<" *current = fForest[itree];" << std::endl;
2830  fout << " while (current->GetNodeType() == 0) { //intermediate node" << std::endl;
2831  fout << " if (current->GoesRight(inputValues)) current=("<<nodeName<<"*)current->GetRight();" << std::endl;
2832  fout << " else current=("<<nodeName<<"*)current->GetLeft();" << std::endl;
2833  fout << " }" << std::endl;
2834  if (fBoostType=="Grad"){
2835  fout << " myMVA += current->GetResponse();" << std::endl;
2836  }else{
2837  if (fUseYesNoLeaf) fout << " myMVA += fBoostWeights[itree] * current->GetNodeType();" << std::endl;
2838  else fout << " myMVA += fBoostWeights[itree] * current->GetPurity();" << std::endl;
2839  fout << " norm += fBoostWeights[itree];" << std::endl;
2840  }
2841  fout << " }" << std::endl;
2842  if (fBoostType=="Grad"){
2843  fout << " return 2.0/(1.0+exp(-2.0*myMVA))-1.0;" << std::endl;
2844  }
2845  else fout << " return myMVA /= norm;" << std::endl;
2846  fout << "}" << std::endl << std::endl;
2847  }
2848 
2849  fout << "void " << className << "::Initialize()" << std::endl;
2850  fout << "{" << std::endl;
2851  fout << " double inf = std::numeric_limits<double>::infinity();" << std::endl;
2852  fout << " double nan = std::numeric_limits<double>::quiet_NaN();" << std::endl;
2853  //Now for each decision tree, write directly the constructors of the nodes in the tree structure
2854  for (UInt_t itree=0; itree<GetNTrees(); itree++) {
2855  fout << " // itree = " << itree << std::endl;
2856  fout << " fBoostWeights.push_back(" << fBoostWeights[itree] << ");" << std::endl;
2857  fout << " fForest.push_back( " << std::endl;
2858  this->MakeClassInstantiateNode((DecisionTreeNode*)fForest[itree]->GetRoot(), fout, className);
2859  fout <<" );" << std::endl;
2860  }
2861  fout << " return;" << std::endl;
2862  fout << "};" << std::endl;
2863  fout << std::endl;
2864  fout << "// Clean up" << std::endl;
2865  fout << "inline void " << className << "::Clear() " << std::endl;
2866  fout << "{" << std::endl;
2867  fout << " for (unsigned int itree=0; itree<fForest.size(); itree++) { " << std::endl;
2868  fout << " delete fForest[itree]; " << std::endl;
2869  fout << " }" << std::endl;
2870  fout << "}" << std::endl;
2871  fout << std::endl;
2872 }
2873 
2874 ////////////////////////////////////////////////////////////////////////////////
2875 /// Specific class header.
2876 
2877 void TMVA::MethodBDT::MakeClassSpecificHeader( std::ostream& fout, const TString& className) const
2878 {
2879  TString nodeName = className;
2880  nodeName.ReplaceAll("Read","");
2881  nodeName.Append("Node");
2882  fout << "#include <algorithm>" << std::endl;
2883  fout << "#include <limits>" << std::endl;
2884  fout << std::endl;
2885  //fout << "#ifndef NN" << std::endl; commented out on purpose see next line
2886  fout << "#define NN new "<<nodeName << std::endl; // NN definition depends on individual methods. Important to have NO #ifndef if several BDT methods compile together
2887  //fout << "#endif" << std::endl; commented out on purpose see previous line
2888  fout << std::endl;
2889  fout << "#ifndef "<<nodeName<<"__def" << std::endl;
2890  fout << "#define "<<nodeName<<"__def" << std::endl;
2891  fout << std::endl;
2892  fout << "class "<<nodeName<<" {" << std::endl;
2893  fout << std::endl;
2894  fout << "public:" << std::endl;
2895  fout << std::endl;
2896  fout << " // constructor of an essentially \"empty\" node floating in space" << std::endl;
2897  fout << " "<<nodeName<<" ( "<<nodeName<<"* left,"<<nodeName<<"* right," << std::endl;
2898  if (fUseFisherCuts){
2899  fout << " int nFisherCoeff," << std::endl;
2900  for (UInt_t i=0;i<GetNVariables()+1;i++){
2901  fout << " double fisherCoeff"<<i<<"," << std::endl;
2902  }
2903  }
2904  fout << " int selector, double cutValue, bool cutType, " << std::endl;
2905  fout << " int nodeType, double purity, double response ) :" << std::endl;
2906  fout << " fLeft ( left )," << std::endl;
2907  fout << " fRight ( right )," << std::endl;
2908  if (fUseFisherCuts) fout << " fNFisherCoeff ( nFisherCoeff )," << std::endl;
2909  fout << " fSelector ( selector )," << std::endl;
2910  fout << " fCutValue ( cutValue )," << std::endl;
2911  fout << " fCutType ( cutType )," << std::endl;
2912  fout << " fNodeType ( nodeType )," << std::endl;
2913  fout << " fPurity ( purity )," << std::endl;
2914  fout << " fResponse ( response ){" << std::endl;
2915  if (fUseFisherCuts){
2916  for (UInt_t i=0;i<GetNVariables()+1;i++){
2917  fout << " fFisherCoeff.push_back(fisherCoeff"<<i<<");" << std::endl;
2918  }
2919  }
2920  fout << " }" << std::endl << std::endl;
2921  fout << " virtual ~"<<nodeName<<"();" << std::endl << std::endl;
2922  fout << " // test event if it descends the tree at this node to the right" << std::endl;
2923  fout << " virtual bool GoesRight( const std::vector<double>& inputValues ) const;" << std::endl;
2924  fout << " "<<nodeName<<"* GetRight( void ) {return fRight; };" << std::endl << std::endl;
2925  fout << " // test event if it descends the tree at this node to the left " << std::endl;
2926  fout << " virtual bool GoesLeft ( const std::vector<double>& inputValues ) const;" << std::endl;
2927  fout << " "<<nodeName<<"* GetLeft( void ) { return fLeft; }; " << std::endl << std::endl;
2928  fout << " // return S/(S+B) (purity) at this node (from training)" << std::endl << std::endl;
2929  fout << " double GetPurity( void ) const { return fPurity; } " << std::endl;
2930  fout << " // return the node type" << std::endl;
2931  fout << " int GetNodeType( void ) const { return fNodeType; }" << std::endl;
2932  fout << " double GetResponse(void) const {return fResponse;}" << std::endl << std::endl;
2933  fout << "private:" << std::endl << std::endl;
2934  fout << " "<<nodeName<<"* fLeft; // pointer to the left daughter node" << std::endl;
2935  fout << " "<<nodeName<<"* fRight; // pointer to the right daughter node" << std::endl;
2936  if (fUseFisherCuts){
2937  fout << " int fNFisherCoeff; // =0 if this node doesn't use fisher, else =nvar+1 " << std::endl;
2938  fout << " std::vector<double> fFisherCoeff; // the fisher coeff (offset at the last element)" << std::endl;
2939  }
2940  fout << " int fSelector; // index of variable used in node selection (decision tree) " << std::endl;
2941  fout << " double fCutValue; // cut value applied on this node to discriminate bkg against sig" << std::endl;
2942  fout << " bool fCutType; // true: if event variable > cutValue ==> signal , false otherwise" << std::endl;
2943  fout << " int fNodeType; // Type of node: -1 == Bkg-leaf, 1 == Signal-leaf, 0 = internal " << std::endl;
2944  fout << " double fPurity; // Purity of node from training"<< std::endl;
2945  fout << " double fResponse; // Regression response value of node" << std::endl;
2946  fout << "}; " << std::endl;
2947  fout << std::endl;
2948  fout << "//_______________________________________________________________________" << std::endl;
2949  fout << " "<<nodeName<<"::~"<<nodeName<<"()" << std::endl;
2950  fout << "{" << std::endl;
2951  fout << " if (fLeft != NULL) delete fLeft;" << std::endl;
2952  fout << " if (fRight != NULL) delete fRight;" << std::endl;
2953  fout << "}; " << std::endl;
2954  fout << std::endl;
2955  fout << "//_______________________________________________________________________" << std::endl;
2956  fout << "bool "<<nodeName<<"::GoesRight( const std::vector<double>& inputValues ) const" << std::endl;
2957  fout << "{" << std::endl;
2958  fout << " // test event if it descends the tree at this node to the right" << std::endl;
2959  fout << " bool result;" << std::endl;
2960  if (fUseFisherCuts){
2961  fout << " if (fNFisherCoeff == 0){" << std::endl;
2962  fout << " result = (inputValues[fSelector] >= fCutValue );" << std::endl;
2963  fout << " }else{" << std::endl;
2964  fout << " double fisher = fFisherCoeff.at(fFisherCoeff.size()-1);" << std::endl;
2965  fout << " for (unsigned int ivar=0; ivar<fFisherCoeff.size()-1; ivar++)" << std::endl;
2966  fout << " fisher += fFisherCoeff.at(ivar)*inputValues.at(ivar);" << std::endl;
2967  fout << " result = fisher > fCutValue;" << std::endl;
2968  fout << " }" << std::endl;
2969  }else{
2970  fout << " result = (inputValues[fSelector] >= fCutValue );" << std::endl;
2971  }
2972  fout << " if (fCutType == true) return result; //the cuts are selecting Signal ;" << std::endl;
2973  fout << " else return !result;" << std::endl;
2974  fout << "}" << std::endl;
2975  fout << std::endl;
2976  fout << "//_______________________________________________________________________" << std::endl;
2977  fout << "bool "<<nodeName<<"::GoesLeft( const std::vector<double>& inputValues ) const" << std::endl;
2978  fout << "{" << std::endl;
2979  fout << " // test event if it descends the tree at this node to the left" << std::endl;
2980  fout << " if (!this->GoesRight(inputValues)) return true;" << std::endl;
2981  fout << " else return false;" << std::endl;
2982  fout << "}" << std::endl;
2983  fout << std::endl;
2984  fout << "#endif" << std::endl;
2985  fout << std::endl;
2986 }
2987 
2988 ////////////////////////////////////////////////////////////////////////////////
2989 /// Recursively descends a tree and writes the node instance to the output stream.
2990 
2991 void TMVA::MethodBDT::MakeClassInstantiateNode( DecisionTreeNode *n, std::ostream& fout, const TString& className ) const
2992 {
2993  if (n == NULL) {
2994  Log() << kFATAL << "MakeClassInstantiateNode: started with undefined node" <<Endl;
2995  return ;
2996  }
2997  fout << "NN("<<std::endl;
2998  if (n->GetLeft() != NULL){
2999  this->MakeClassInstantiateNode( (DecisionTreeNode*)n->GetLeft() , fout, className);
3000  }
3001  else {
3002  fout << "0";
3003  }
3004  fout << ", " <<std::endl;
3005  if (n->GetRight() != NULL){
3006  this->MakeClassInstantiateNode( (DecisionTreeNode*)n->GetRight(), fout, className );
3007  }
3008  else {
3009  fout << "0";
3010  }
3011  fout << ", " << std::endl
3012  << std::setprecision(6);
3013  if (fUseFisherCuts){
3014  fout << n->GetNFisherCoeff() << ", ";
3015  for (UInt_t i=0; i< GetNVariables()+1; i++) {
3016  if (n->GetNFisherCoeff() == 0 ){
3017  fout << "0, ";
3018  }else{
3019  fout << n->GetFisherCoeff(i) << ", ";
3020  }
3021  }
3022  }
3023  fout << n->GetSelector() << ", "
3024  << n->GetCutValue() << ", "
3025  << n->GetCutType() << ", "
3026  << n->GetNodeType() << ", "
3027  << n->GetPurity() << ","
3028  << n->GetResponse() << ") ";
3029 }
3030 
3031 ////////////////////////////////////////////////////////////////////////////////
3032 /// Find useful preselection cuts that will be applied before
3033 /// and Decision Tree training.. (and of course also applied
3034 /// in the GetMVA .. --> -1 for background +1 for Signal)
3035 
3036 void TMVA::MethodBDT::DeterminePreselectionCuts(const std::vector<const TMVA::Event*>& eventSample)
3037 {
3038  Double_t nTotS = 0.0, nTotB = 0.0;
3039  Int_t nTotS_unWeighted = 0, nTotB_unWeighted = 0;
3040 
3041  std::vector<TMVA::BDTEventWrapper> bdtEventSample;
3042 
3043  fIsLowSigCut.assign(GetNvar(),kFALSE);
3044  fIsLowBkgCut.assign(GetNvar(),kFALSE);
3045  fIsHighSigCut.assign(GetNvar(),kFALSE);
3046  fIsHighBkgCut.assign(GetNvar(),kFALSE);
3047 
3048  fLowSigCut.assign(GetNvar(),0.); // ---------------| --> in var is signal (accept all above lower cut)
3049  fLowBkgCut.assign(GetNvar(),0.); // ---------------| --> in var is bkg (accept all above lower cut)
3050  fHighSigCut.assign(GetNvar(),0.); // <-- | -------------- in var is signal (accept all blow cut)
3051  fHighBkgCut.assign(GetNvar(),0.); // <-- | -------------- in var is blg (accept all blow cut)
3052 
3053 
3054  // Initialize (un)weighted counters for signal & background
3055  // Construct a list of event wrappers that point to the original data
3056  for( std::vector<const TMVA::Event*>::const_iterator it = eventSample.begin(); it != eventSample.end(); ++it ) {
3057  if (DataInfo().IsSignal(*it)){
3058  nTotS += (*it)->GetWeight();
3059  ++nTotS_unWeighted;
3060  }
3061  else {
3062  nTotB += (*it)->GetWeight();
3063  ++nTotB_unWeighted;
3064  }
3065  bdtEventSample.push_back(TMVA::BDTEventWrapper(*it));
3066  }
3067 
3068  for( UInt_t ivar = 0; ivar < GetNvar(); ivar++ ) { // loop over all discriminating variables
3069  TMVA::BDTEventWrapper::SetVarIndex(ivar); // select the variable to sort by
3070  std::sort( bdtEventSample.begin(),bdtEventSample.end() ); // sort the event data
3071 
3072  Double_t bkgWeightCtr = 0.0, sigWeightCtr = 0.0;
3073  std::vector<TMVA::BDTEventWrapper>::iterator it = bdtEventSample.begin(), it_end = bdtEventSample.end();
3074  for( ; it != it_end; ++it ) {
3075  if (DataInfo().IsSignal(**it))
3076  sigWeightCtr += (**it)->GetWeight();
3077  else
3078  bkgWeightCtr += (**it)->GetWeight();
3079  // Store the accumulated signal (background) weights
3080  it->SetCumulativeWeight(false,bkgWeightCtr);
3081  it->SetCumulativeWeight(true,sigWeightCtr);
3082  }
3083 
3084  //variable that determines how "exact" you cut on the preselection found in the training data. Here I chose
3085  //1% of the variable range...
3086  Double_t dVal = (DataInfo().GetVariableInfo(ivar).GetMax() - DataInfo().GetVariableInfo(ivar).GetMin())/100. ;
3087  Double_t nSelS, nSelB, effS=0.05, effB=0.05, rejS=0.05, rejB=0.05;
3088  Double_t tmpEffS, tmpEffB, tmpRejS, tmpRejB;
3089  // Locate the optimal cut for this (ivar-th) variable
3090 
3091 
3092 
3093  for(UInt_t iev = 1; iev < bdtEventSample.size(); iev++) {
3094  //dVal = bdtEventSample[iev].GetVal() - bdtEventSample[iev-1].GetVal();
3095 
3096  nSelS = bdtEventSample[iev].GetCumulativeWeight(true);
3097  nSelB = bdtEventSample[iev].GetCumulativeWeight(false);
3098  // you look for some 100% efficient pre-selection cut to remove background.. i.e. nSelS=0 && nSelB>5%nTotB or ( nSelB=0 nSelS>5%nTotS)
3099  tmpEffS=nSelS/nTotS;
3100  tmpEffB=nSelB/nTotB;
3101  tmpRejS=1-tmpEffS;
3102  tmpRejB=1-tmpEffB;
3103  if (nSelS==0 && tmpEffB>effB) {effB=tmpEffB; fLowBkgCut[ivar] = bdtEventSample[iev].GetVal() - dVal; fIsLowBkgCut[ivar]=kTRUE;}
3104  else if (nSelB==0 && tmpEffS>effS) {effS=tmpEffS; fLowSigCut[ivar] = bdtEventSample[iev].GetVal() - dVal; fIsLowSigCut[ivar]=kTRUE;}
3105  else if (nSelB==nTotB && tmpRejS>rejS) {rejS=tmpRejS; fHighSigCut[ivar] = bdtEventSample[iev].GetVal() + dVal; fIsHighSigCut[ivar]=kTRUE;}
3106  else if (nSelS==nTotS && tmpRejB>rejB) {rejB=tmpRejB; fHighBkgCut[ivar] = bdtEventSample[iev].GetVal() + dVal; fIsHighBkgCut[ivar]=kTRUE;}
3107 
3108  }
3109  }
3110 
3111  Log() << kDEBUG << " \tfound and suggest the following possible pre-selection cuts " << Endl;
3112  if (fDoPreselection) Log() << kDEBUG << "\tthe training will be done after these cuts... and GetMVA value returns +1, (-1) for a signal (bkg) event that passes these cuts" << Endl;
3113  else Log() << kDEBUG << "\tas option DoPreselection was not used, these cuts however will not be performed, but the training will see the full sample"<<Endl;
3114  for (UInt_t ivar=0; ivar < GetNvar(); ivar++ ) { // loop over all discriminating variables
3115  if (fIsLowBkgCut[ivar]){
3116  Log() << kDEBUG << " \tfound cut: Bkg if var " << ivar << " < " << fLowBkgCut[ivar] << Endl;
3117  }
3118  if (fIsLowSigCut[ivar]){
3119  Log() << kDEBUG << " \tfound cut: Sig if var " << ivar << " < " << fLowSigCut[ivar] << Endl;
3120  }
3121  if (fIsHighBkgCut[ivar]){
3122  Log() << kDEBUG << " \tfound cut: Bkg if var " << ivar << " > " << fHighBkgCut[ivar] << Endl;
3123  }
3124  if (fIsHighSigCut[ivar]){
3125  Log() << kDEBUG << " \tfound cut: Sig if var " << ivar << " > " << fHighSigCut[ivar] << Endl;
3126  }
3127  }
3128 
3129  return;
3130 }
3131 
3132 ////////////////////////////////////////////////////////////////////////////////
3133 /// Apply the preselection cuts before even bothering about any
3134 /// Decision Trees in the GetMVA .. --> -1 for background +1 for Signal
3135 
3137 {
3138  Double_t result=0;
3139 
3140  for (UInt_t ivar=0; ivar < GetNvar(); ivar++ ) { // loop over all discriminating variables
3141  if (fIsLowBkgCut[ivar]){
3142  if (ev->GetValue(ivar) < fLowBkgCut[ivar]) result = -1; // is background
3143  }
3144  if (fIsLowSigCut[ivar]){
3145  if (ev->GetValue(ivar) < fLowSigCut[ivar]) result = 1; // is signal
3146  }
3147  if (fIsHighBkgCut[ivar]){
3148  if (ev->GetValue(ivar) > fHighBkgCut[ivar]) result = -1; // is background
3149  }
3150  if (fIsHighSigCut[ivar]){
3151  if (ev->GetValue(ivar) > fHighSigCut[ivar]) result = 1; // is signal
3152  }
3153  }
3154 
3155  return result;
3156 }
3157 
LogInterval.h
TMVA::GiniIndexWithLaplace
Implementation of the GiniIndex With Laplace correction as separation criterion.
Definition: GiniIndexWithLaplace.h:59
TMVA::MethodBDT::MakeClassInstantiateNode
void MakeClassInstantiateNode(DecisionTreeNode *n, std::ostream &fout, const TString &className) const
Recursively descends a tree and writes the node instance to the output stream.
Definition: MethodBDT.cxx:2991
n
const Int_t n
Definition: legend1.C:16
TMVA::MethodBDT::GradBoost
Double_t GradBoost(std::vector< const TMVA::Event * > &, DecisionTree *dt, UInt_t cls=0)
Calculate the desired response value for each region.
Definition: MethodBDT.cxx:1595
TMVA::MethodBDT::Boost
Double_t Boost(std::vector< const TMVA::Event * > &, DecisionTree *dt, UInt_t cls=0)
Apply the boosting algorithm (the algorithm is selecte via the the "option" given in the constructor.
Definition: MethodBDT.cxx:1718
TMVA::MethodBDT::fMonitorNtuple
TTree * fMonitorNtuple
Definition: MethodBDT.h:264
first
Definition: first.py:1
TMVA::MethodBDT::ProcessOptions
void ProcessOptions()
The option string is decoded, for available options see "DeclareOptions".
Definition: MethodBDT.cxx:470
ROOT::Math::IntegOptionsUtil::Print
void Print(std::ostream &os, const OptionType &opt)
Definition: IntegratorOptions.cxx:91
TMVA::SdivSqrtSplusB
Implementation of the SdivSqrtSplusB as separation criterion.
Definition: SdivSqrtSplusB.h:44
kTRUE
const Bool_t kTRUE
Definition: RtypesCore.h:91
TMVA::Types::kMulticlass
@ kMulticlass
Definition: Types.h:131
e
#define e(i)
Definition: RSha256.hxx:103
TMVA::Tools::GetChild
void * GetChild(void *parent, const char *childname=0)
get child node
Definition: Tools.cxx:1162
TMVA::MethodBDT::AdaCost
Double_t AdaCost(std::vector< const TMVA::Event * > &, DecisionTree *dt)
The AdaCost boosting algorithm takes a simple cost Matrix (currently fixed for all events....
Definition: MethodBDT.cxx:2024
TMVA::MethodBDT::CreateRanking
const Ranking * CreateRanking()
Compute ranking of input variables.
Definition: MethodBDT.cxx:2683
TGraph::SetTitle
virtual void SetTitle(const char *title="")
Change (i.e.
Definition: TGraph.cxx:2329
f
#define f(i)
Definition: RSha256.hxx:104
TMVA::MethodBDT::MakeClassSpecific
void MakeClassSpecific(std::ostream &, const TString &) const
Make ROOT-independent C++ class for classifier response (classifier-specific implementation).
Definition: MethodBDT.cxx:2757
TMVA::MethodBDT::ReadWeightsFromStream
virtual void ReadWeightsFromStream(std::istream &)=0
TMVA::DecisionTree::CreateFromXML
static DecisionTree * CreateFromXML(void *node, UInt_t tmva_Version_Code=TMVA_VERSION_CODE)
re-create a new tree (decision tree or search tree) from XML
Definition: DecisionTree.cxx:281
TMath::Max
Short_t Max(Short_t a, Short_t b)
Definition: TMathBase.h:212
TMVA::MethodBDT::UpdateTargetsRegression
void UpdateTargetsRegression(std::vector< const TMVA::Event * > &, Bool_t first=kFALSE)
Calculate residuals for all events and update targets for next iter.
Definition: MethodBDT.cxx:1557
TMVA::DecisionTreeNode::SetIsTraining
static void SetIsTraining(bool on)
Definition: DecisionTreeNode.cxx:546
TMVA::Types::kRegression
@ kRegression
Definition: Types.h:130
ROOT::Internal::SHA256::K
static const uint32_t K[64]
Definition: RSha256.hxx:148
TH1I
1-D histogram with an int per channel (see TH1 documentation)}
Definition: TH1.h:534
floor
double floor(double)
TH2F
2-D histogram with a float per channel (see TH1 documentation)}
Definition: TH2.h:251
TMVA::MethodBDT::DeclareCompatibilityOptions
void DeclareCompatibilityOptions()
Options that are used ONLY for the READER to ensure backward compatibility.
Definition: MethodBDT.cxx:454
ClassImp
#define ClassImp(name)
Definition: Rtypes.h:364
Form
char * Form(const char *fmt,...)
TMVA::Executor::Map
auto Map(F func, unsigned nTimes) -> std::vector< typename std::result_of< F()>::type >
Wrap TExecutor::Map functions.
Definition: Executor.h:134
TNamed::GetTitle
virtual const char * GetTitle() const
Returns title of object.
Definition: TNamed.h:48
TMVA::Ranking
Ranking for variables in method (implementation)
Definition: Ranking.h:48
TMVA::Config::GetThreadExecutor
Executor & GetThreadExecutor()
Get executor class for multi-thread usage In case when MT is not enabled will return a serial executo...
Definition: Config.h:83
TMVA::MethodBDT::SetTuneParameters
virtual void SetTuneParameters(std::map< TString, Double_t > tuneParameters)
Set the tuning parameters according to the argument.
Definition: MethodBDT.cxx:1121
TMVA::Tools::UsefulSortAscending
void UsefulSortAscending(std::vector< std::vector< Double_t > > &, std::vector< TString > *vs=0)
sort 2D vector (AND in parallel a TString vector) in such a way that the "first vector is sorted" and...
Definition: Tools.cxx:550
TMVA::MethodBDT::AdaBoostR2
Double_t AdaBoostR2(std::vector< const TMVA::Event * > &, DecisionTree *dt)
Adaption of the AdaBoost to regression problems (see H.Drucker 1997).
Definition: MethodBDT.cxx:2193
TMVA::MethodBDT::PreProcessNegativeEventWeights
void PreProcessNegativeEventWeights()
O.k.
Definition: MethodBDT.cxx:932
r
ROOT::R::TRInterface & r
Definition: Object.C:4
TGraph.h
IMethod.h
sum
static uint64_t sum(uint64_t i)
Definition: Factory.cxx:2345
Long64_t
long long Long64_t
Definition: RtypesCore.h:73
TMath::Log
Double_t Log(Double_t x)
Definition: TMath.h:760
TMVA::MethodBase::DeclareCompatibilityOptions
virtual void DeclareCompatibilityOptions()
options that are used ONLY for the READER to ensure backward compatibility they are hence without any...
Definition: MethodBase.cxx:596
TString::Atof
Double_t Atof() const
Return floating-point value contained in string.
Definition: TString.cxx:1987
TTree
A TTree represents a columnar dataset.
Definition: TTree.h:79
TMath::Sqrt
Double_t Sqrt(Double_t x)
Definition: TMath.h:691
Ranking.h
TMVA::Tools::AddChild
void * AddChild(void *parent, const char *childname, const char *content=0, bool isRootNode=false)
add child node
Definition: Tools.cxx:1136
TMVA::MethodBDT::Reset
void Reset(void)
Reset the method, as if it had just been instantiated (forget all training etc.).
Definition: MethodBDT.cxx:725
GiniIndex.h
exp
double exp(double)
TMVA::Results::GetHist
TH1 * GetHist(const TString &alias) const
Definition: Results.cxx:136
TMath::CeilNint
Int_t CeilNint(Double_t x)
Definition: TMath.h:699
TRandom::PoissonD
virtual Double_t PoissonD(Double_t mean)
Generates a random number according to a Poisson law.
Definition: TRandom.cxx:454
Float_t
float Float_t
Definition: RtypesCore.h:57
TMath::Exp
Double_t Exp(Double_t x)
Definition: TMath.h:727
TMVA::MethodBDT::GetHelpMessage
void GetHelpMessage() const
Get help message text.
Definition: MethodBDT.cxx:2700
TMVA::LossFunctionEventInfo::predictedValue
Double_t predictedValue
Definition: LossFunction.h:64
TMVA::Timer::DrawProgressBar
void DrawProgressBar(Int_t, const TString &comment="")
draws progress bar in color or B&W caution:
Definition: Timer.cxx:202
TH1::SetBinContent
virtual void SetBinContent(Int_t bin, Double_t content)
Set bin content see convention for numbering bins in TH1::GetBin In case the bin number is greater th...
Definition: TH1.cxx:8988
TMVA::MethodBDT::DeclareOptions
void DeclareOptions()
Define the options (their key words).
Definition: MethodBDT.cxx:333
TMVA::DecisionTree::CheckEvent
Double_t CheckEvent(const TMVA::Event *, Bool_t UseYesNoLeaf=kFALSE) const
the event e is put into the decision tree (starting at the root node) and the output is NodeType (sig...
Definition: DecisionTree.cxx:2690
TMVA::DecisionTree::kExpectedErrorPruning
@ kExpectedErrorPruning
Definition: DecisionTree.h:139
ROOT::GetClass
TClass * GetClass(T *)
Definition: TClass.h:601
MethodBase.h
TMVA::Event::GetTarget
Float_t GetTarget(UInt_t itgt) const
Definition: Event.h:102
TMVA::MethodBDT::MakeClassSpecificHeader
void MakeClassSpecificHeader(std::ostream &, const TString &) const
Specific class header.
Definition: MethodBDT.cxx:2877
TMatrixTSym< Double_t >
TMVA::Event::SetTarget
void SetTarget(UInt_t itgt, Float_t value)
set the target value (dimension itgt) to value
Definition: Event.cxx:359
TMVA::DecisionTreeNode
Definition: DecisionTreeNode.h:117
TMath::Abs
Short_t Abs(Short_t d)
Definition: TMathBase.h:120
TMVA::Rank
Definition: Ranking.h:76
TString::IsFloat
Bool_t IsFloat() const
Returns kTRUE if string contains a floating point or integer number.
Definition: TString.cxx:1791
TMVA::MethodBDT::GetGradBoostMVA
Double_t GetGradBoostMVA(const TMVA::Event *e, UInt_t nTrees)
Returns MVA value: -1 for background, 1 for signal.
Definition: MethodBDT.cxx:1421
TMVA::CrossEntropy
Implementation of the CrossEntropy as separation criterion.
Definition: CrossEntropy.h:43
ceil
double ceil(double)
TMVA::MethodBDT::RegBoost
Double_t RegBoost(std::vector< const TMVA::Event * > &, DecisionTree *dt)
A special boosting only for Regression (not implemented).
Definition: MethodBDT.cxx:2185
TMVA::GiniIndex
Implementation of the GiniIndex as separation criterion.
Definition: GiniIndex.h:63
TH1::SetXTitle
virtual void SetXTitle(const char *title)
Definition: TH1.h:413
TString
Basic string class.
Definition: TString.h:136
TMVA::DecisionTree::kNoPruning
@ kNoPruning
Definition: DecisionTree.h:139
TMVA::AbsoluteDeviationLossFunctionBDT
Absolute Deviation BDT Loss Function.
Definition: LossFunction.h:261
TMVA::MethodBDT::GradBoostRegression
Double_t GradBoostRegression(std::vector< const TMVA::Event * > &, DecisionTree *dt)
Implementation of M_TreeBoost using any loss function as described by Friedman 1999.
Definition: MethodBDT.cxx:1629
TH1::SetYTitle
virtual void SetYTitle(const char *title)
Definition: TH1.h:414
v
@ v
Definition: rootcling_impl.cxx:3635
TMVA::MethodBDT::InitGradBoost
void InitGradBoost(std::vector< const TMVA::Event * > &)
Initialize targets for first tree.
Definition: MethodBDT.cxx:1658
TMVA::DecisionTree
Implementation of a Decision Tree.
Definition: DecisionTree.h:65
REGISTER_METHOD
#define REGISTER_METHOD(CLASS)
for example
Definition: ClassifierFactory.h:124
TMVA::MethodBDT::Bagging
Double_t Bagging()
Call it boot-strapping, re-sampling or whatever you like, in the end it is nothing else but applying ...
Definition: MethodBDT.cxx:2140
bool
TString::ReplaceAll
TString & ReplaceAll(const TString &s1, const TString &s2)
Definition: TString.h:692
TMVA::OptimizeConfigParameters::optimize
std::map< TString, Double_t > optimize()
Definition: OptimizeConfigParameters.cxx:127
TMVA::MethodBDT::HasAnalysisType
virtual Bool_t HasAnalysisType(Types::EAnalysisType type, UInt_t numberClasses, UInt_t numberTargets)
BDT can handle classification with multiple classes and regression with one regression-target.
Definition: MethodBDT.cxx:280
TMVA::MethodBDT::~MethodBDT
virtual ~MethodBDT(void)
Destructor.
Definition: MethodBDT.cxx:753
PDF.h
TMVA::Config::Instance
static Config & Instance()
static function: returns TMVA instance
Definition: Config.cxx:106
TMVA::Tools::AddAttr
void AddAttr(void *node, const char *, const T &value, Int_t precision=16)
add attribute to xml
Definition: Tools.h:353
TMath::FloorNint
Int_t FloorNint(Double_t x)
Definition: TMath.h:707
TMVA::MethodBDT::WriteMonitoringHistosToFile
void WriteMonitoringHistosToFile(void) const
Here we could write some histograms created during the processing to the output file.
Definition: MethodBDT.cxx:2628
TMVA::MethodBDT::BoostMonitor
void BoostMonitor(Int_t iTree)
Fills the ROCIntegral vs Itree from the testSample for the monitoring plots during the training .
Definition: MethodBDT.cxx:1752
TMVA::MethodBDT::DeterminePreselectionCuts
void DeterminePreselectionCuts(const std::vector< const TMVA::Event * > &eventSample)
Find useful preselection cuts that will be applied before and Decision Tree training.
Definition: MethodBDT.cxx:3036
ROOT::Math::gv_detail::dist
double dist(Rotation3D const &r1, Rotation3D const &r2)
Definition: 3DDistances.cxx:48
TMVA::Event::GetValue
Float_t GetValue(UInt_t ivar) const
return value of i'th variable
Definition: Event.cxx:236
TMVA::MethodBDT::GetMvaValue
Double_t GetMvaValue(Double_t *err=0, Double_t *errUpper=0)
Definition: MethodBDT.cxx:2443
TMVA::MethodBDT::GetRegressionValues
const std::vector< Float_t > & GetRegressionValues()
Get the regression value generated by the BDTs.
Definition: MethodBDT.cxx:2542
TMVA::MethodBDT::PrivateGetMvaValue
Double_t PrivateGetMvaValue(const TMVA::Event *ev, Double_t *err=0, Double_t *errUpper=0, UInt_t useNTrees=0)
Return the MVA value (range [-1;1]) that classifies the event according to the majority vote from the...
Definition: MethodBDT.cxx:2468
TMVA::DataSetInfo
Class that contains all the data information.
Definition: DataSetInfo.h:62
ROOT::Math::fabs
VecExpr< UnaryOp< Fabs< T >, VecExpr< A, T, D >, T >, T, D > fabs(const VecExpr< A, T, D > &rhs)
Definition: UnaryOperators.h:131
TMVA::Results::Store
void Store(TObject *obj, const char *alias=0)
Definition: Results.cxx:86
OptimizeConfigParameters.h
SeparationBase.h
TMVA::DecisionTree::GetEventNode
TMVA::DecisionTreeNode * GetEventNode(const TMVA::Event &e) const
get the pointer to the leaf node where a particular event ends up in...
Definition: DecisionTree.cxx:2673
TMVA::Timer::GetElapsedTime
TString GetElapsedTime(Bool_t Scientific=kTRUE)
returns pretty string with elapsed time
Definition: Timer.cxx:146
TMVA::LeastSquaresLossFunctionBDT
Least Squares BDT Loss Function.
Definition: LossFunction.h:222
MsgLogger.h
DecisionTree.h
Timer.h
GiniIndexWithLaplace.h
TMVA::MethodBDT::SetMinNodeSize
void SetMinNodeSize(Double_t sizeInPercent)
Definition: MethodBDT.cxx:660
BinarySearchTree.h
TRandom3
Random number generator class based on M.
Definition: TRandom3.h:27
TMVA::Types::EAnalysisType
EAnalysisType
Definition: Types.h:128
h
#define h(i)
Definition: RSha256.hxx:106
TGraph::SetName
virtual void SetName(const char *name="")
Set graph name.
Definition: TGraph.cxx:2313
TMatrixTSym::Determinant
virtual Double_t Determinant() const
Definition: TMatrixTSym.cxx:935
TMVA::BDTEventWrapper
Definition: BDTEventWrapper.h:31
TMVA::MethodBDT
Analysis of Boosted Decision Trees.
Definition: MethodBDT.h:63
gr
TGraphErrors * gr
Definition: legend1.C:25
epsilon
REAL epsilon
Definition: triangle.c:617
TMVA::Executor::Foreach
void Foreach(Function func, unsigned int nTimes, unsigned nChunks=0)
wrap TExecutor::Foreach
Definition: Executor.h:111
kFALSE
const Bool_t kFALSE
Definition: RtypesCore.h:92
TString::Append
TString & Append(const char *cs)
Definition: TString.h:564
TMVA::MethodBDT::GetMulticlassValues
const std::vector< Float_t > & GetMulticlassValues()
Get the multiclass MVA response for the BDT classifier.
Definition: MethodBDT.cxx:2495
TMVA::Tools::ReadAttr
void ReadAttr(void *node, const char *, T &value)
read attribute from xml
Definition: Tools.h:335
TMath::Power
LongDouble_t Power(LongDouble_t x, LongDouble_t y)
Definition: TMath.h:735
TMVA::Types::kClassification
@ kClassification
Definition: Types.h:129
TMVA::MethodBDT::TestTreeQuality
Double_t TestTreeQuality(DecisionTree *dt)
Test the tree quality.. in terms of Misclassification.
Definition: MethodBDT.cxx:1697
TH1::Fill
virtual Int_t Fill(Double_t x)
Increment bin with abscissa X by 1.
Definition: TH1.cxx:3327
TH2
Service class for 2-Dim histogram classes.
Definition: TH2.h:30
TRandom3.h
TMVA::MethodBDT::InitEventSample
void InitEventSample()
Initialize the event sample (i.e. reset the boost-weights... etc).
Definition: MethodBDT.cxx:761
UInt_t
unsigned int UInt_t
Definition: RtypesCore.h:46
TMVA::MethodBase
Virtual base Class for all MVA method.
Definition: MethodBase.h:111
TMVA::Types
Singleton class for Global types used by TMVA.
Definition: Types.h:73
Types.h
Configurable.h
TMVA::Results
Class that is the base-class for a vector of result.
Definition: Results.h:57
TObject::Read
virtual Int_t Read(const char *name)
Read contents of object with specified name from the current directory.
Definition: TObject.cxx:564
TMVA::MethodBDT::AddWeightsXMLTo
void AddWeightsXMLTo(void *parent) const
Write weights to XML.
Definition: MethodBDT.cxx:2310
TMVA::MethodBDT::fRegressionLossFunctionBDTG
LossFunctionBDT * fRegressionLossFunctionBDTG
Definition: MethodBDT.h:299
TGraph::SetPoint
virtual void SetPoint(Int_t i, Double_t x, Double_t y)
Set x and y values for point number i.
Definition: TGraph.cxx:2274
TMath::Min
Short_t Min(Short_t a, Short_t b)
Definition: TMathBase.h:180
TMVA::Endl
MsgLogger & Endl(MsgLogger &ml)
Definition: MsgLogger.h:158
Config.h
unsigned int
TMVA::Timer
Timing information for training and evaluation of MVA methods.
Definition: Timer.h:58
TMVA::OptimizeConfigParameters
Definition: OptimizeConfigParameters.h:49
TMVA::TMVAGlob::GetMethodName
void GetMethodName(TString &name, TKey *mkey)
Definition: tmvaglob.cxx:342
TMVA::Tools::Color
const TString & Color(const TString &)
human readable color strings
Definition: Tools.cxx:840
TMVA::PDF
PDF wrapper for histograms; uses user-defined spline interpolation.
Definition: PDF.h:63
TMVA::Types::kTraining
@ kTraining
Definition: Types.h:145
TMVA::LogInterval
The TMVA::Interval Class.
Definition: LogInterval.h:83
TGraph::Set
virtual void Set(Int_t n)
Set number of points in the graph Existing coordinates are preserved New coordinates above fNpoints a...
Definition: TGraph.cxx:2209
TMVA::MethodBDT::GetVariableImportance
std::vector< Double_t > GetVariableImportance()
Return the relative variable importance, normalized to all variables together having the importance 1...
Definition: MethodBDT.cxx:2643
TMVA::Types::kMaxAnalysisType
@ kMaxAnalysisType
Definition: Types.h:133
TMVA::LossFunctionEventInfo
Definition: LossFunction.h:48
TMVA::MethodBDT::OptimizeTuningParameters
virtual std::map< TString, Double_t > OptimizeTuningParameters(TString fomType="ROCIntegral", TString fitType="FitGA")
Call the Optimizer with the set of parameters and ranges that are meant to be tuned.
Definition: MethodBDT.cxx:1068
Double_t
double Double_t
Definition: RtypesCore.h:59
TMVA::BDTEventWrapper::SetVarIndex
static void SetVarIndex(Int_t iVar)
Definition: BDTEventWrapper.h:59
TMVA::MethodBDT::fgDebugLevel
static const Int_t fgDebugLevel
Definition: MethodBDT.h:302
TGraph
A TGraph is an object made of two arrays X and Y with npoints each.
Definition: TGraph.h:41
BDTEventWrapper.h
MethodBDT.h
TMVA::MethodBDT::Init
void Init(void)
Common initialisation with defaults for the BDT-Method.
Definition: MethodBDT.cxx:687
TMVA::MethodBDT::ReadWeightsFromXML
void ReadWeightsFromXML(void *parent)
Reads the BDT from the xml file.
Definition: MethodBDT.cxx:2341
TMVA::Tools::GetNextChild
void * GetNextChild(void *prevchild, const char *childname=0)
XML helpers.
Definition: Tools.cxx:1174
TMVA::DecisionTree::kCostComplexityPruning
@ kCostComplexityPruning
Definition: DecisionTree.h:139
TH1F
1-D histogram with a float per channel (see TH1 documentation)}
Definition: TH1.h:575
TObject::Write
virtual Int_t Write(const char *name=0, Int_t option=0, Int_t bufsize=0)
Write this object to the current directory.
Definition: TObject.cxx:795
TMVA::MethodBDT::UpdateTargets
void UpdateTargets(std::vector< const TMVA::Event * > &, UInt_t cls=0)
Calculate residual for all events.
Definition: MethodBDT.cxx:1435
TMVA::Event
Definition: Event.h:51
TMVA::Tools::CalcCovarianceMatrices
std::vector< TMatrixDSym * > * CalcCovarianceMatrices(const std::vector< Event * > &events, Int_t maxCls, VariableTransformBase *transformBase=0)
compute covariance matrices
Definition: Tools.cxx:1526
TH1
TH1 is the base class of all histogramm classes in ROOT.
Definition: TH1.h:58
TMVA::MisClassificationError
Implementation of the MisClassificationError as separation criterion.
Definition: MisClassificationError.h:46
TMVA::BinaryTree::GetNNodes
UInt_t GetNNodes() const
Definition: BinaryTree.h:86
CrossEntropy.h
TGraph::GetN
Int_t GetN() const
Definition: TGraph.h:124
TMVA::MethodBDT::fSepType
SeparationBase * fSepType
Definition: MethodBDT.h:229
TObject::Delete
virtual void Delete(Option_t *option="")
Delete this object.
Definition: TObject.cxx:171
SdivSqrtSplusB.h
TMVA::BDT
void BDT(TString dataset, const TString &fin="TMVA.root")
d
#define d(i)
Definition: RSha256.hxx:102
TMatrixTSym.h
TMVA::MethodBDT::GetBaggedSubSample
void GetBaggedSubSample(std::vector< const TMVA::Event * > &)
Fills fEventSample with fBaggedSampleFraction*NEvents random training events.
Definition: MethodBDT.cxx:2151
TMVA::MethodBDT::AdaBoost
Double_t AdaBoost(std::vector< const TMVA::Event * > &, DecisionTree *dt)
The AdaBoost implementation.
Definition: MethodBDT.cxx:1846
MisClassificationError.h
TMVA::PDF::kSpline3
@ kSpline3
Definition: PDF.h:70
TMVA::HuberLossFunctionBDT
Huber BDT Loss Function.
Definition: LossFunction.h:179
TMVA::MethodBDT::ApplyPreselectionCuts
Double_t ApplyPreselectionCuts(const Event *ev)
Apply the preselection cuts before even bothering about any Decision Trees in the GetMVA .
Definition: MethodBDT.cxx:3136
TMVA::Results::GetGraph
TGraph * GetGraph(const TString &alias) const
Definition: Results.cxx:153
ResultsMulticlass.h
Tools.h
ROOT::TSeq
A pseudo container class which is a generator of indices.
Definition: TSeq.hxx:66
ClassifierFactory.h
type
int type
Definition: TGX11.cxx:121
Results.h
TMVA::gTools
Tools & gTools()
TMatrixTSym::Invert
TMatrixTSym< Element > & Invert(Double_t *det=0)
Invert the matrix and calculate its determinant Notice that the LU decomposition is used instead of B...
Definition: TMatrixTSym.cxx:961
TMVA::MethodBDT::MethodBDT
MethodBDT(const TString &jobName, const TString &methodTitle, DataSetInfo &theData, const TString &theOption="")
The standard constructor for the "boosted decision trees".
Definition: MethodBDT.cxx:163
TMVA::MethodBDT::Train
void Train(void)
BDT training.
Definition: MethodBDT.cxx:1142
TMath::E
constexpr Double_t E()
Base of natural log:
Definition: TMath.h:96
DataSet.h
TMVA::Executor::GetPoolSize
unsigned int GetPoolSize() const
Definition: Executor.h:100
ROOT::TSeqU
TSeq< unsigned int > TSeqU
Definition: TSeq.hxx:195
TMath.h
Interval.h
TMVA
create variable transformations
Definition: GeneticMinimizer.h:22
int
TMVA::Interval
The TMVA::Interval Class.
Definition: Interval.h:61