{ "cells": [ { "cell_type": "markdown", "id": "489c3ada", "metadata": {}, "source": [ "# TMVA_Higgs_Classification\n", "Classification example of TMVA based on public Higgs UCI dataset\n", "\n", " The UCI data set is a public HIGGS data set , see http://archive.ics.uci.edu/ml/datasets/HIGGS\n", "used in this paper: Baldi, P., P. Sadowski, and D. Whiteson. \u201cSearching for Exotic Particles in High-energy Physics\n", " with Deep Learning.\u201d Nature Communications 5 (July 2, 2014).\n", "\n", "\n", "\n", "\n", "**Author:** Lorenzo Moneta \n", "This notebook tutorial was automatically generated with ROOTBOOK-izer from the macro found in the ROOT repository on Tuesday, May 19, 2026 at 08:22 PM." ] }, { "cell_type": "markdown", "id": "bc166b75", "metadata": {}, "source": [ "options to control used methods" ] }, { "cell_type": "code", "execution_count": null, "id": "2df43ebd", "metadata": { "collapsed": false }, "outputs": [], "source": [ " bool useLikelihood = true; // likelihood based discriminant\n", " bool useLikelihoodKDE = false; // likelihood based discriminant\n", " bool useFischer = true; // Fischer discriminant\n", " bool useMLP = false; // Multi Layer Perceptron (old TMVA NN implementation)\n", " bool useBDT = true; // Boosted Decision Tree\n", " bool useDL = true; // TMVA Deep learning ( CPU or GPU)\n", " bool useKeras = true; // Keras Deep learning\n", " bool usePyTorch = true; // PyTorch Deep learning\n", "\n", " TMVA::Tools::Instance();\n", "\n", "#ifdef R__HAS_PYMVA\n", " gSystem->Setenv(\"KERAS_BACKEND\", \"tensorflow\");" ] }, { "cell_type": "markdown", "id": "dd98efe4", "metadata": {}, "source": [ "for using Keras" ] }, { "cell_type": "code", "execution_count": null, "id": "dd87313c", "metadata": { "collapsed": false }, "outputs": [], "source": [ " TMVA::PyMethodBase::PyInitialize();\n", "#else\n", " useKeras = false;\n", " usePyTorch = false;\n", "#endif\n", "\n", " auto outputFile = TFile::Open(\"Higgs_ClassificationOutput.root\", \"RECREATE\");\n", "\n", " TMVA::Factory factory(\"TMVA_Higgs_Classification\", outputFile,\n", " \"!V:ROC:!Silent:Color:AnalysisType=Classification\" );\n", "\n", "/**\n", "\n", "## Setup Dataset(s)\n", "\n", "Define now input data file and signal and background trees\n", "\n", "**/\n", "\n", " TString inputFileName = gROOT->GetTutorialDir() + \"/machine_learning/data/Higgs_data.root\";\n", "\n", " TFile *inputFile = nullptr;\n", "\n", " if (!gSystem->AccessPathName(inputFileName)) {\n", " // file exists\n", " inputFile = TFile::Open( inputFileName );\n", " }\n", " if (!inputFile) {\n", " Error(\"TMVA_Higgs_Classification\",\"Input file is not found - exit\");\n", " return;\n", " }" ] }, { "cell_type": "markdown", "id": "e10fdbb4", "metadata": {}, "source": [ "--- Register the training and test trees" ] }, { "cell_type": "code", "execution_count": null, "id": "b78ab9d3", "metadata": { "collapsed": false }, "outputs": [], "source": [ " TTree *signalTree = (TTree*)inputFile->Get(\"sig_tree\");\n", " TTree *backgroundTree = (TTree*)inputFile->Get(\"bkg_tree\");\n", "\n", " signalTree->Print();\n", "\n", "/***\n", "## Declare DataLoader(s)\n", "\n", "The next step is to declare the DataLoader class that deals with input variables\n", "\n", "Define the input variables that shall be used for the MVA training\n", "note that you may also use variable expressions, which can be parsed by TTree::Draw( \"expression\" )]\n", "\n", "***/\n", "\n", " TMVA::DataLoader * loader = new TMVA::DataLoader(\"dataset\");\n", "\n", " loader->AddVariable(\"m_jj\");\n", " loader->AddVariable(\"m_jjj\");\n", " loader->AddVariable(\"m_lv\");\n", " loader->AddVariable(\"m_jlv\");\n", " loader->AddVariable(\"m_bb\");\n", " loader->AddVariable(\"m_wbb\");\n", " loader->AddVariable(\"m_wwbb\");" ] }, { "cell_type": "markdown", "id": "20becc06", "metadata": {}, "source": [ "We set now the input data trees in the TMVA DataLoader class" ] }, { "cell_type": "markdown", "id": "058636b7", "metadata": {}, "source": [ "global event weights per tree (see below for setting event-wise weights)" ] }, { "cell_type": "code", "execution_count": null, "id": "f197915e", "metadata": { "collapsed": false }, "outputs": [], "source": [ " Double_t signalWeight = 1.0;\n", " Double_t backgroundWeight = 1.0;" ] }, { "cell_type": "markdown", "id": "88fff44a", "metadata": {}, "source": [ "You can add an arbitrary number of signal or background trees" ] }, { "cell_type": "code", "execution_count": null, "id": "6d495cd6", "metadata": { "collapsed": false }, "outputs": [], "source": [ " loader->AddSignalTree ( signalTree, signalWeight );\n", " loader->AddBackgroundTree( backgroundTree, backgroundWeight );" ] }, { "cell_type": "markdown", "id": "dc074cec", "metadata": {}, "source": [ "Set individual event weights (the variables must exist in the original TTree)\n", "for signal : factory->SetSignalWeightExpression (\"weight1*weight2\");\n", "for background: factory->SetBackgroundWeightExpression(\"weight1*weight2\");\n", "loader->SetBackgroundWeightExpression( \"weight\" );" ] }, { "cell_type": "markdown", "id": "80875e28", "metadata": {}, "source": [ "Apply additional cuts on the signal and background samples (can be different)" ] }, { "cell_type": "code", "execution_count": null, "id": "d7d6e356", "metadata": { "collapsed": false }, "outputs": [], "source": [ " TCut mycuts = \"\"; // for example: TCut mycuts = \"abs(var1)<0.5 && abs(var2-0.5)<1\";\n", " TCut mycutb = \"\"; // for example: TCut mycutb = \"abs(var1)<0.5\";" ] }, { "cell_type": "markdown", "id": "ae39fadd", "metadata": {}, "source": [ "Tell the factory how to use the training and testing events\n", "\n", "If no numbers of events are given, half of the events in the tree are used\n", "for training, and the other half for testing:\n", "loader->PrepareTrainingAndTestTree( mycut, \"SplitMode=random:!V\" );\n", "To also specify the number of testing events, use:" ] }, { "cell_type": "code", "execution_count": null, "id": "ac7295c3", "metadata": { "collapsed": false }, "outputs": [], "source": [ " loader->PrepareTrainingAndTestTree( mycuts, mycutb,\n", " \"nTrain_Signal=7000:nTrain_Background=7000:SplitMode=Random:NormMode=NumEvents:!V\" );\n", "\n", "/***\n", "## Booking Methods\n", "\n", "Here we book the TMVA methods. We book first a Likelihood based on KDE (Kernel Density Estimation), a Fischer discriminant, a BDT\n", "and a shallow neural network\n", "\n", "*/" ] }, { "cell_type": "markdown", "id": "a9d135d0", "metadata": {}, "source": [ "Likelihood (\"naive Bayes estimator\")" ] }, { "cell_type": "code", "execution_count": null, "id": "c04b339b", "metadata": { "collapsed": false }, "outputs": [], "source": [ "if (useLikelihood) {\n", " factory.BookMethod(loader, TMVA::Types::kLikelihood, \"Likelihood\",\n", " \"H:!V:TransformOutput:PDFInterpol=Spline2:NSmoothSig[0]=20:NSmoothBkg[0]=20:NSmoothBkg[1]=10:NSmooth=1:NAvEvtPerBin=50\" );\n", "}" ] }, { "cell_type": "markdown", "id": "1025f179", "metadata": {}, "source": [ "Use a kernel density estimator to approximate the PDFs" ] }, { "cell_type": "code", "execution_count": null, "id": "b9ee4b94", "metadata": { "collapsed": false }, "outputs": [], "source": [ "if (useLikelihoodKDE) {\n", " factory.BookMethod(loader, TMVA::Types::kLikelihood, \"LikelihoodKDE\",\n", " \"!H:!V:!TransformOutput:PDFInterpol=KDE:KDEtype=Gauss:KDEiter=Adaptive:KDEFineFactor=0.3:KDEborder=None:NAvEvtPerBin=50\" );\n", "\n", "}" ] }, { "cell_type": "markdown", "id": "3c593154", "metadata": {}, "source": [ "Fisher discriminant (same as LD)" ] }, { "cell_type": "code", "execution_count": null, "id": "06799424", "metadata": { "collapsed": false }, "outputs": [], "source": [ "if (useFischer) {\n", " factory.BookMethod(loader, TMVA::Types::kFisher, \"Fisher\", \"H:!V:Fisher:VarTransform=None:CreateMVAPdfs:PDFInterpolMVAPdf=Spline2:NbinsMVAPdf=50:NsmoothMVAPdf=10\" );\n", "}" ] }, { "cell_type": "markdown", "id": "2289313a", "metadata": {}, "source": [ "Boosted Decision Trees" ] }, { "cell_type": "code", "execution_count": null, "id": "908b4042", "metadata": { "collapsed": false }, "outputs": [], "source": [ "if (useBDT) {\n", " factory.BookMethod(loader,TMVA::Types::kBDT, \"BDT\",\n", " \"!V:NTrees=200:MinNodeSize=2.5%:MaxDepth=2:BoostType=AdaBoost:AdaBoostBeta=0.5:UseBaggedBoost:BaggedSampleFraction=0.5:SeparationType=GiniIndex:nCuts=20\" );\n", "}" ] }, { "cell_type": "markdown", "id": "aa9227cc", "metadata": {}, "source": [ "Multi-Layer Perceptron (Neural Network)" ] }, { "cell_type": "code", "execution_count": null, "id": "0399a5cf", "metadata": { "collapsed": false }, "outputs": [], "source": [ "if (useMLP) {\n", " factory.BookMethod(loader, TMVA::Types::kMLP, \"MLP\",\n", " \"!H:!V:NeuronType=tanh:VarTransform=N:NCycles=100:HiddenLayers=N+5:TestRate=5:!UseRegulator\" );\n", "}" ] }, { "cell_type": "markdown", "id": "81ae7476", "metadata": {}, "source": [ "Here we book the new DNN of TMVA if we have support in ROOT. We will use GPU version if ROOT is enabled with GPU" ] }, { "cell_type": "code", "execution_count": null, "id": "ed1c4f75", "metadata": { "collapsed": false }, "outputs": [], "source": [ " /***\n", "\n", "## Booking Deep Neural Network\n", "\n", "Here we define the option string for building the Deep Neural network model.\n", "\n", "#### 1. Define DNN layout\n", "\n", "The DNN configuration is defined using a string. Note that whitespaces between characters are not allowed.\n", "\n", "We define first the DNN layout:\n", "\n", "- **input layout** : this defines the input data format for the DNN as ``input depth | height | width``.\n", " In case of a dense layer as first layer the input layout should be ``1 | 1 | number of input variables`` (features)\n", "- **batch layout** : this defines how are the input batch. It is related to input layout but not the same.\n", " If the first layer is dense it should be ``1 | batch size ! number of variables`` (features)\n", "\n", " *(note the use of the character `|` as separator of input parameters for DNN layout)*\n", "\n", "note that in case of only dense layer the input layout could be omitted but it is required when defining more\n", "complex architectures\n", "\n", "- **layer layout** string defining the layer architecture. The syntax is\n", " - layer type (e.g. DENSE, CONV, RNN)\n", " - layer parameters (e.g. number of units)\n", " - activation function (e.g TANH, RELU,...)\n", "\n", " *the different layers are separated by the ``\",\"`` *\n", "\n", "#### 2. Define Training Strategy\n", "\n", "We define here the training strategy parameters for the DNN. The parameters are separated by the ``\",\"`` separator.\n", "One can then concatenate different training strategy with different parameters. The training strategy are separated by\n", "the ``\"|\"`` separator.\n", "\n", "- Optimizer\n", "- Learning rate\n", "- Momentum (valid for SGD and RMSPROP)\n", "- Regularization and Weight Decay\n", "- Dropout\n", "- Max number of epochs\n", "- Convergence steps. if the test error will not decrease after that value the training will stop\n", "- Batch size (This value must be the same specified in the input layout)\n", "- Test Repetitions (the interval when the test error will be computed)\n", "\n", "\n", "#### 3. Define general DNN options\n", "\n", "We define the general DNN options concatenating in the final string the previously defined layout and training strategy.\n", "Note we use the ``\":\"`` separator to separate the different higher level options, as in the other TMVA methods.\n", "In addition to input layout, batch layout and training strategy we add now:\n", "\n", "- Type of Loss function (e.g. CROSSENTROPY)\n", "- Weight Initizalization (e.g XAVIER, XAVIERUNIFORM, NORMAL )\n", "- Variable Transformation\n", "- Type of Architecture (e.g. CPU, GPU, Standard)\n", "\n", "We can then book the DL method using the built option string\n", "\n", " ***/\n", "\n", " if (useDL) {\n", "\n", " bool useDLGPU = false;\n", "#ifdef R__HAS_TMVAGPU\n", " useDLGPU = true;\n", "#endif\n", "\n", " // Define DNN layout\n", " TString inputLayoutString = \"InputLayout=1|1|7\";\n", " TString batchLayoutString= \"BatchLayout=1|128|7\";\n", " TString layoutString (\"Layout=DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR\");\n", " // Define Training strategies\n", " // one can catenate several training strategies\n", " TString training1(\"LearningRate=1e-3,Momentum=0.9,\"\n", " \"ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,\"\n", " \"MaxEpochs=30,WeightDecay=1e-4,Regularization=None,\"\n", " \"Optimizer=ADAM,ADAM_beta1=0.9,ADAM_beta2=0.999,ADAM_eps=1.E-7,\" // ADAM default parameters\n", " \"DropConfig=0.0+0.0+0.0+0.\");\n", " // TString training2(\"LearningRate=1e-3,Momentum=0.9\"\n", " // \"ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,\"\n", " // \"MaxEpochs=20,WeightDecay=1e-4,Regularization=None,\"\n", " // \"Optimizer=SGD,DropConfig=0.0+0.0+0.0+0.\");\n", "\n", " TString trainingStrategyString (\"TrainingStrategy=\");\n", " trainingStrategyString += training1; // + \"|\" + training2;\n", "\n", " // General Options.\n", "\n", " TString dnnOptions (\"!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=G:\"\n", " \"WeightInitialization=XAVIER\");\n", " dnnOptions.Append (\":\"); dnnOptions.Append (inputLayoutString);\n", " dnnOptions.Append (\":\"); dnnOptions.Append (batchLayoutString);\n", " dnnOptions.Append (\":\"); dnnOptions.Append (layoutString);\n", " dnnOptions.Append (\":\"); dnnOptions.Append (trainingStrategyString);\n", "\n", " TString dnnMethodName = \"DNN_CPU\";\n", " if (useDLGPU) {\n", " dnnOptions += \":Architecture=GPU\";\n", " dnnMethodName = \"DNN_GPU\";\n", " } else {\n", " dnnOptions += \":Architecture=CPU\";\n", " }\n", "\n", " factory.BookMethod(loader, TMVA::Types::kDL, dnnMethodName, dnnOptions);\n", " }" ] }, { "cell_type": "markdown", "id": "c238ec5e", "metadata": {}, "source": [ "Keras deep learning" ] }, { "cell_type": "code", "execution_count": null, "id": "6d494f4f", "metadata": { "collapsed": false }, "outputs": [], "source": [ " if (useKeras) {\n", "\n", " Info(\"TMVA_Higgs_Classification\", \"Building deep neural network with keras \");\n", " // create python script which can be executed\n", " // create 2 conv2d layer + maxpool + dense\n", " TMacro m;\n", " m.AddLine(\"import tensorflow\");\n", " m.AddLine(\"from tensorflow.keras.models import Sequential\");\n", " m.AddLine(\"from tensorflow.keras.optimizers import Adam\");\n", " m.AddLine(\"from tensorflow.keras.layers import Input, Dense\");\n", " m.AddLine(\"\");\n", " m.AddLine(\"model = Sequential() \");\n", " m.AddLine(\"model.add(Dense(64, activation='relu',input_dim=7))\");\n", " m.AddLine(\"model.add(Dense(64, activation='relu'))\");\n", " m.AddLine(\"model.add(Dense(64, activation='relu'))\");\n", " m.AddLine(\"model.add(Dense(64, activation='relu'))\");\n", " m.AddLine(\"model.add(Dense(2, activation='sigmoid'))\");\n", " m.AddLine(\"model.compile(loss = 'binary_crossentropy', optimizer = Adam(learning_rate = 0.001), weighted_metrics = ['accuracy'])\");\n", " m.AddLine(\"model.save('Higgs_model.keras')\");\n", " m.AddLine(\"model.summary()\");\n", "\n", " m.SaveSource(\"make_higgs_model.py\");\n", " // execute\n", " auto ret = (TString *)gROOT->ProcessLine(\"TMVA::Python_Executable()\");\n", " TString python_exe = (ret) ? *(ret) : \"python\";\n", " gSystem->Exec(python_exe + \" make_higgs_model.py\");\n", "\n", " if (gSystem->AccessPathName(\"Higgs_model.keras\")) {\n", " Warning(\"TMVA_Higgs_Classification\", \"Error creating Keras model file - skip using Keras\");\n", " } else {\n", " // book PyKeras method only if Keras model could be created\n", " Info(\"TMVA_Higgs_Classification\", \"Booking tf.Keras Dense model\");\n", " factory.BookMethod(\n", " loader, TMVA::Types::kPyKeras, \"PyKeras\",\n", " \"H:!V:VarTransform=None:FilenameModel=Higgs_model.keras:tf.keras:\"\n", " \"FilenameTrainedModel=Higgs_trained_model.keras:NumEpochs=20:BatchSize=100:\"\n", " ); // needed for RTX NVidia card and to avoid TF allocates all GPU memory\n", " }\n", " }\n", "\n", " /**\n", "## Train Methods\n", "\n", "Here we train all the previously booked methods.\n", "\n", " */\n", "\n", " factory.TrainAllMethods();\n", "\n", "/**\n", " ## Test all methods\n", "\n", "Now we test and evaluate all methods using the test data set\n", "*/\n", "\n", " factory.TestAllMethods();\n", "\n", " factory.EvaluateAllMethods();" ] }, { "cell_type": "markdown", "id": "25d34e1e", "metadata": {}, "source": [ "after we get the ROC curve and we display" ] }, { "cell_type": "code", "execution_count": null, "id": "dbf1bf5a", "metadata": { "collapsed": false }, "outputs": [], "source": [ " auto c1 = factory.GetROCCurve(loader);\n", " c1->Draw();" ] }, { "cell_type": "markdown", "id": "7f4e5227", "metadata": {}, "source": [ "at the end we close the output file which contains the evaluation result of all methods and it can be used by TMVAGUI\n", "to display additional plots" ] }, { "cell_type": "code", "execution_count": null, "id": "de915a50", "metadata": { "collapsed": false }, "outputs": [], "source": [ " outputFile->Close();" ] }, { "cell_type": "markdown", "id": "5f5cf137", "metadata": {}, "source": [ "Draw all canvases " ] }, { "cell_type": "code", "execution_count": null, "id": "8ad490cf", "metadata": { "collapsed": false }, "outputs": [], "source": [ "gROOT->GetListOfCanvases()->Draw()" ] } ], "metadata": { "kernelspec": { "display_name": "ROOT C++", "language": "c++", "name": "root" }, "language_info": { "codemirror_mode": "text/x-c++src", "file_extension": ".C", "mimetype": " text/x-c++src", "name": "c++" } }, "nbformat": 4, "nbformat_minor": 5 }