{ "cells": [ { "cell_type": "markdown", "id": "73842ae4", "metadata": {}, "source": [ "# rf611_weightedfits\n", "Likelihood and minimization: Parameter uncertainties for weighted unbinned ML fits\n", "\n", "## Parameter uncertainties for weighted unbinned ML fits\n", "\n", "Based on example from https://arxiv.org/abs/1911.01303\n", "\n", "This example compares different approaches to determining parameter uncertainties in weighted unbinned maximum\n", "likelihood fits. Performing a weighted unbinned maximum likelihood fits can be useful to account for acceptance\n", "effects and to statistically subtract background events using the sPlot formalism. It is however well known that the\n", "inverse Hessian matrix does not yield parameter uncertainties with correct coverage in the presence of event\n", "weights. Three approaches to the determination of parameter uncertainties are compared in this example:\n", "\n", "1. Using the inverse weighted Hessian matrix [`SumW2Error(false)`]\n", "\n", "2. Using the expression [`SumW2Error(true)`]\n", " $$\n", " V_{ij} = H_{ik}^{-1} C_{kl} H_{lj}^{-1}\n", " $$\n", " where H is the weighted Hessian matrix and C is the Hessian matrix with squared weights\n", "\n", "3. The asymptotically correct approach (for details please see https://arxiv.org/abs/1911.01303)\n", "[`Asymptotic(true)`]\n", " $$\n", " V_{ij} = H_{ik}^{-1} D_{kl} H_{lj}^{-1}\n", " $$\n", " where H is the weighted Hessian matrix and D is given by\n", " $$\n", " D_{kl} = \\sum_{e=1}^{N} w_e^2 \\frac{\\partial \\log(P)}{\\partial \\lambda_k}\\frac{\\partial \\log(P)}{\\partial\n", " \\lambda_l}\n", " $$\n", " with the event weight $w_e$.\n", "\n", "The example performs the fit of a second order polynomial in the angle cos(theta) [-1,1] to a weighted data set.\n", "The polynomial is given by\n", " P = \\frac{ 1 + c_0 \\cdot \\cos(\\theta) + c_1 \\cdot \\cos(\\theta) \\cdot \\cos(\\theta) }{\\mathrm{Norm}}\n", "The two coefficients $ c_0 $ and $ c_1 $ and their uncertainties are to be determined in the fit.\n", "\n", "The per-event weight is used to correct for an acceptance effect, two different acceptance models can be studied:\n", "- `acceptancemodel==1`: eff = $ 0.3 + 0.7 \\cdot \\cos(\\theta) \\cdot \\cos(\\theta) $\n", "- `acceptancemodel==2`: eff = $ 1.0 - 0.7 \\cdot \\cos(\\theta) \\cdot \\cos(\\theta) $\n", "The data is generated to be flat before the acceptance effect.\n", "\n", "The performance of the different approaches to determine parameter uncertainties is compared using the pull\n", "distributions from a large number of pseudoexperiments. The pull is defined as $ (\\lambda_i -\n", "its uncertainty for pseudoexperiment number i. If the fit is unbiased and the parameter uncertainties are estimated\n", "correctly, the pull distribution should be a Gaussian centered around zero with a width of one.\n", "\n", "\n", "\n", "\n", "**Author:** Christoph Langenbruch \n", "This notebook tutorial was automatically generated with ROOTBOOK-izer from the macro found in the ROOT repository on Tuesday, May 19, 2026 at 08:33 PM." ] }, { "cell_type": "code", "execution_count": 1, "id": "a4566b51", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:38.856934Z", "iopub.status.busy": "2026-05-19T20:33:38.856817Z", "iopub.status.idle": "2026-05-19T20:33:38.871909Z", "shell.execute_reply": "2026-05-19T20:33:38.871397Z" } }, "outputs": [], "source": [ "%%cpp -d\n", "#include \"TH1D.h\"\n", "#include \"TCanvas.h\"\n", "#include \"TROOT.h\"\n", "#include \"TStyle.h\"\n", "#include \"TRandom3.h\"\n", "#include \"TLegend.h\"\n", "#include \"RooRealVar.h\"\n", "#include \"RooFitResult.h\"\n", "#include \"RooDataSet.h\"\n", "#include \"RooPolynomial.h\"\n", "\n", "using namespace RooFit;" ] }, { "cell_type": "markdown", "id": "efc19efe", "metadata": {}, "source": [ " Arguments are defined. " ] }, { "cell_type": "code", "execution_count": 2, "id": "e5a22155", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:38.873461Z", "iopub.status.busy": "2026-05-19T20:33:38.873346Z", "iopub.status.idle": "2026-05-19T20:33:39.208912Z", "shell.execute_reply": "2026-05-19T20:33:39.207841Z" } }, "outputs": [], "source": [ "int acceptancemodel = 2;" ] }, { "cell_type": "markdown", "id": "f74b1a08", "metadata": {}, "source": [ "Initialisation and Setup\n", "------------------------------------------------" ] }, { "cell_type": "markdown", "id": "0cd3e935", "metadata": {}, "source": [ "plotting options" ] }, { "cell_type": "code", "execution_count": 3, "id": "96848701", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:39.210522Z", "iopub.status.busy": "2026-05-19T20:33:39.210404Z", "iopub.status.idle": "2026-05-19T20:33:39.419454Z", "shell.execute_reply": "2026-05-19T20:33:39.418440Z" } }, "outputs": [], "source": [ "gStyle->SetPaintTextFormat(\".1f\");\n", "gStyle->SetEndErrorSize(6.0);\n", "gStyle->SetTitleSize(0.05, \"XY\");\n", "gStyle->SetLabelSize(0.05, \"XY\");\n", "gStyle->SetTitleOffset(0.9, \"XY\");\n", "gStyle->SetTextSize(0.05);\n", "gStyle->SetPadLeftMargin(0.125);\n", "gStyle->SetPadBottomMargin(0.125);\n", "gStyle->SetPadTopMargin(0.075);\n", "gStyle->SetPadRightMargin(0.075);\n", "gStyle->SetMarkerStyle(20);\n", "gStyle->SetMarkerSize(1.0);\n", "gStyle->SetHistLineWidth(2.0);\n", "gStyle->SetHistLineColor(1);" ] }, { "cell_type": "markdown", "id": "dd90496c", "metadata": {}, "source": [ "initialise TRandom3" ] }, { "cell_type": "code", "execution_count": 4, "id": "a161e86c", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:39.421040Z", "iopub.status.busy": "2026-05-19T20:33:39.420920Z", "iopub.status.idle": "2026-05-19T20:33:39.629578Z", "shell.execute_reply": "2026-05-19T20:33:39.629026Z" } }, "outputs": [], "source": [ "TRandom3 *rnd = new TRandom3();\n", "rnd->SetSeed(191101303);" ] }, { "cell_type": "markdown", "id": "24e28c59", "metadata": {}, "source": [ "accepted events and events weighted to account for the acceptance" ] }, { "cell_type": "code", "execution_count": 5, "id": "000bbc39", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:39.631547Z", "iopub.status.busy": "2026-05-19T20:33:39.631426Z", "iopub.status.idle": "2026-05-19T20:33:39.840160Z", "shell.execute_reply": "2026-05-19T20:33:39.839599Z" } }, "outputs": [], "source": [ "TH1 *haccepted = new TH1D(\"haccepted\", \"Generated events;cos(#theta);#events\", 40, -1.0, 1.0);\n", "TH1 *hweighted = new TH1D(\"hweighted\", \"Generated events;cos(#theta);#events\", 40, -1.0, 1.0);" ] }, { "cell_type": "markdown", "id": "68a4c7b8", "metadata": {}, "source": [ "histograms holding pull distributions" ] }, { "cell_type": "code", "execution_count": 6, "id": "38956c88", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:39.842127Z", "iopub.status.busy": "2026-05-19T20:33:39.842007Z", "iopub.status.idle": "2026-05-19T20:33:40.044771Z", "shell.execute_reply": "2026-05-19T20:33:40.043709Z" } }, "outputs": [], "source": [ "std::array hc0pull;\n", "std::array hc1pull;\n", "std::array hntotpull;\n", "std::array methodLabels{\"Inverse weighted Hessian matrix [SumW2Error(false)]\",\n", " \"Hessian matrix with squared weights [SumW2Error(true)]\",\n", " \"Asymptotically correct approach [Asymptotic(true)]\"};\n", "auto makePullXLabel = [](std::string const &pLabel) {\n", " return \"Pull (\" + pLabel + \"^{fit}-\" + pLabel + \"^{gen})/#sigma(\" + pLabel + \")\";\n", "};\n", "for (std::size_t i = 0; i < 3; ++i) {\n", " std::string const &iLabel = std::to_string(i);\n", " // using the inverse Hessian matrix\n", " std::string hc0XLabel = methodLabels[i] + \";\" + makePullXLabel(\"c_{0}\") + \";\";\n", " std::string hc1XLabel = methodLabels[i] + \";\" + makePullXLabel(\"c_{1}\") + \";\";\n", " std::string hntotXLabel = methodLabels[i] + \";\" + makePullXLabel(\"N_{tot}\") + \";\";\n", " hc0pull[i] = new TH1D((\"hc0pull\" + iLabel).c_str(), hc0XLabel.c_str(), 20, -5.0, 5.0);\n", " // using the correction with the Hessian matrix with squared weights\n", " hc1pull[i] = new TH1D((\"hc1pull\" + iLabel).c_str(), hc1XLabel.c_str(), 20, -5.0, 5.0);\n", " // asymptotically correct approach\n", " hntotpull[i] = new TH1D((\"hntotpull\" + iLabel).c_str(), hntotXLabel.c_str(), 20, -5.0, 5.0);\n", "}" ] }, { "cell_type": "markdown", "id": "98a29d81", "metadata": {}, "source": [ "number of pseudoexperiments (toys) and number of events per pseudoexperiment" ] }, { "cell_type": "code", "execution_count": 7, "id": "69f473bb", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:40.046394Z", "iopub.status.busy": "2026-05-19T20:33:40.046275Z", "iopub.status.idle": "2026-05-19T20:33:40.254930Z", "shell.execute_reply": "2026-05-19T20:33:40.254350Z" } }, "outputs": [], "source": [ "constexpr std::size_t ntoys = 500;\n", "constexpr std::size_t nstats = 500;" ] }, { "cell_type": "markdown", "id": "f7d0f6c7", "metadata": {}, "source": [ "parameters used in the generation" ] }, { "cell_type": "code", "execution_count": 8, "id": "4b214b67", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:40.256885Z", "iopub.status.busy": "2026-05-19T20:33:40.256764Z", "iopub.status.idle": "2026-05-19T20:33:40.465822Z", "shell.execute_reply": "2026-05-19T20:33:40.464727Z" } }, "outputs": [], "source": [ "constexpr double c0gen = 0.0;\n", "constexpr double c1gen = 0.0;" ] }, { "cell_type": "markdown", "id": "e17d534e", "metadata": {}, "source": [ "Silence fitting and minimisation messages" ] }, { "cell_type": "code", "execution_count": 9, "id": "dc76d851", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:40.467386Z", "iopub.status.busy": "2026-05-19T20:33:40.467269Z", "iopub.status.idle": "2026-05-19T20:33:40.677221Z", "shell.execute_reply": "2026-05-19T20:33:40.676311Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Running 1500 toy fits ...\n" ] } ], "source": [ "auto &msgSv = RooMsgService::instance();\n", "msgSv.getStream(1).removeTopic(RooFit::Minimization);\n", "msgSv.getStream(1).removeTopic(RooFit::Fitting);\n", "\n", "std::cout << \"Running \" << ntoys * 3 << \" toy fits ...\" << std::endl;" ] }, { "cell_type": "markdown", "id": "b0937d90", "metadata": {}, "source": [ "Main loop: run pseudoexperiments\n", "----------------------------------------------------------------" ] }, { "cell_type": "code", "execution_count": 10, "id": "e1255c7c", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:40.678846Z", "iopub.status.busy": "2026-05-19T20:33:40.678726Z", "iopub.status.idle": "2026-05-19T20:33:43.327884Z", "shell.execute_reply": "2026-05-19T20:33:43.327149Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "... done.\n" ] } ], "source": [ "for (std::size_t i = 0; i < ntoys; i++) {\n", " // S e t u p p a r a m e t e r s a n d P D F\n", " //-----------------------------------------------\n", " // angle theta and the weight to account for the acceptance effect\n", " RooRealVar costheta(\"costheta\", \"costheta\", -1.0, 1.0);\n", " RooRealVar weight(\"weight\", \"weight\", 0.0, 1000.0);\n", "\n", " // initialise parameters to fit\n", " RooRealVar c0(\"c0\", \"0th-order coefficient\", c0gen, -1.0, 1.0);\n", " RooRealVar c1(\"c1\", \"1st-order coefficient\", c1gen, -1.0, 1.0);\n", " c0.setError(0.01);\n", " c1.setError(0.01);\n", " // create simple second-order polynomial as probability density function\n", " RooPolynomial pol(\"pol\", \"pol\", costheta, {c0, c1}, 1);\n", "\n", " double ngen = nstats;\n", " if (acceptancemodel == 1)\n", " ngen *= 2.0 / (23.0 / 15.0);\n", " else\n", " ngen *= 2.0 / (16.0 / 15.0);\n", " RooRealVar ntot(\"ntot\", \"ntot\", ngen, 0.0, 2.0 * ngen);\n", " RooExtendPdf extended(\"extended\", \"extended pdf\", pol, ntot);\n", " int npoisson = rnd->Poisson(nstats);\n", "\n", " // G e n e r a t e d a t a s e t f o r p s e u d o e x p e r i m e n t i\n", " //-------------------------------------------------------------------------------\n", " RooDataSet data(\"data\", \"data\", {costheta, weight}, WeightVar(\"weight\"));\n", " // generate nstats events\n", " for (std::size_t j = 0; j < npoisson; j++) {\n", " bool finished = false;\n", " // use simple accept/reject for generation\n", " while (!finished) {\n", " costheta = 2.0 * rnd->Rndm() - 1.0;\n", " // efficiency for the specific value of cos(theta)\n", " double eff = 1.0;\n", " if (acceptancemodel == 1)\n", " eff = 1.0 - 0.7 * costheta.getVal() * costheta.getVal();\n", " else\n", " eff = 0.3 + 0.7 * costheta.getVal() * costheta.getVal();\n", " // use 1/eff as weight to account for acceptance\n", " weight = 1.0 / eff;\n", " // accept/reject\n", " if (10.0 * rnd->Rndm() < eff * pol.getVal())\n", " finished = true;\n", " }\n", " haccepted->Fill(costheta.getVal());\n", " hweighted->Fill(costheta.getVal(), weight.getVal());\n", " data.add({costheta, weight}, weight.getVal());\n", " }\n", "\n", " auto fillPulls = [&](std::size_t i) {\n", " hc0pull[i]->Fill((c0.getVal() - c0gen) / c0.getError());\n", " hc1pull[i]->Fill((c1.getVal() - c1gen) / c1.getError());\n", " hntotpull[i]->Fill((ntot.getVal() - ngen) / ntot.getError());\n", " };\n", "\n", " // F i t t o y u s i n g t h e t h r e e d i f f e r e n t a p p r o a c h e s t o u n c e r t a i\n", " // n t y d e t e r m i n a t i o n\n", " //-------------------------------------------------------------------------------------------------------------------------------------------------\n", " // this uses the inverse weighted Hessian matrix\n", " extended.fitTo(data, SumW2Error(false), PrintLevel(-1));\n", " fillPulls(0);\n", "\n", " // this uses the correction with the Hesse matrix with squared weights\n", " extended.fitTo(data, SumW2Error(true), PrintLevel(-1));\n", " fillPulls(1);\n", "\n", " // this uses the asymptotically correct approach\n", " extended.fitTo(data, AsymptoticError(true), PrintLevel(-1));\n", " fillPulls(2);\n", "}\n", "\n", "std::cout << \"... done.\" << std::endl;" ] }, { "cell_type": "markdown", "id": "967d35da", "metadata": {}, "source": [ "Plot output distributions\n", "--------------------------------------------------" ] }, { "cell_type": "markdown", "id": "6d73969d", "metadata": {}, "source": [ "plot accepted (weighted) events" ] }, { "cell_type": "code", "execution_count": 11, "id": "42712bcd", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:43.329359Z", "iopub.status.busy": "2026-05-19T20:33:43.329229Z", "iopub.status.idle": "2026-05-19T20:33:43.537535Z", "shell.execute_reply": "2026-05-19T20:33:43.537062Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "
\n", "
\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "gStyle->SetOptStat(0);\n", "gStyle->SetOptFit(0);\n", "TCanvas *cevents = new TCanvas(\"cevents\", \"cevents\", 800, 600);\n", "cevents->cd(1);\n", "hweighted->SetMinimum(0.0);\n", "hweighted->SetLineColor(2);\n", "hweighted->Draw(\"hist\");\n", "haccepted->Draw(\"same hist\");\n", "TLegend *leg = new TLegend(0.6, 0.8, 0.9, 0.9);\n", "leg->AddEntry(haccepted, \"Accepted\");\n", "leg->AddEntry(hweighted, \"Weighted\");\n", "leg->Draw();\n", "cevents->Update();" ] }, { "cell_type": "markdown", "id": "4b917213", "metadata": {}, "source": [ "plot pull distributions" ] }, { "cell_type": "code", "execution_count": 12, "id": "4efcd730", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:43.539287Z", "iopub.status.busy": "2026-05-19T20:33:43.539168Z", "iopub.status.idle": "2026-05-19T20:33:43.901899Z", "shell.execute_reply": "2026-05-19T20:33:43.901503Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "
\n", "
\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "****************************************\n", "Minimizer is Minuit2 / Migrad\n", "Chi2 = 12.2745\n", "NDf = 12\n", "Edm = 3.84566e-06\n", "NCalls = 53\n", "Constant = 82.0512 +/- 4.52865 \n", "Mean = -0.042844 +/- 0.0572716 \n", "Sigma = 1.19191 +/- 0.0392558 \t (limited)\n", "****************************************\n", "Minimizer is Minuit2 / Migrad\n", "Chi2 = 8.83881\n", "NDf = 9\n", "Edm = 6.19877e-08\n", "NCalls = 53\n", "Constant = 104.798 +/- 5.86613 \n", "Mean = -0.0132039 +/- 0.0432773 \n", "Sigma = 0.938314 +/- 0.0320986 \t (limited)\n", "****************************************\n", "Minimizer is Minuit2 / Migrad\n", "Chi2 = 7.28099\n", "NDf = 10\n", "Edm = 6.30801e-07\n", "NCalls = 53\n", "Constant = 103.063 +/- 5.55774 \n", "Mean = -0.0233563 +/- 0.0435444 \n", "Sigma = 0.954358 +/- 0.0285394 \t (limited)\n", "****************************************\n", "Minimizer is Minuit2 / Migrad\n", "Chi2 = 23.1247\n", "NDf = 14\n", "Edm = 2.27317e-06\n", "NCalls = 53\n", "Constant = 69.5383 +/- 3.95119 \n", "Mean = -0.160071 +/- 0.0652133 \n", "Sigma = 1.37036 +/- 0.0473756 \t (limited)\n", "****************************************\n", "Minimizer is Minuit2 / Migrad\n", "Chi2 = 7.18474\n", "NDf = 15\n", "Edm = 9.01145e-06\n", "NCalls = 61\n", "Constant = 51.3838 +/- 3.49309 \n", "Mean = -0.985792 +/- 0.0760226 \n", "Sigma = 1.36366 +/- 0.0605719 \t (limited)\n", "****************************************\n", "Minimizer is Minuit2 / Migrad\n", "Chi2 = 11.9456\n", "NDf = 11\n", "Edm = 4.88766e-08\n", "NCalls = 61\n", "Constant = 94.7614 +/- 5.45747 \n", "Mean = -0.0978895 +/- 0.0483575 \n", "Sigma = 1.03309 +/- 0.0382346 \t (limited)\n", "****************************************\n", "Minimizer is Minuit2 / Migrad\n", "Chi2 = 20.9563\n", "NDf = 16\n", "Edm = 8.77656e-06\n", "NCalls = 53\n", "Constant = 69.8331 +/- 3.99247 \n", "Mean = -0.0767062 +/- 0.0640105 \n", "Sigma = 1.36919 +/- 0.0474577 \t (limited)\n", "****************************************\n", "Minimizer is Minuit2 / Migrad\n", "Chi2 = 8.84994\n", "NDf = 10\n", "Edm = 1.85865e-06\n", "NCalls = 53\n", "Constant = 101.665 +/- 5.77203 \n", "Mean = -0.0558063 +/- 0.0444665 \n", "Sigma = 0.96468 +/- 0.0336688 \t (limited)\n", "****************************************\n", "Minimizer is Minuit2 / Migrad\n", "Chi2 = 9.73174\n", "NDf = 10\n", "Edm = 1.95484e-06\n", "NCalls = 53\n", "Constant = 99.7164 +/- 5.69932 \n", "Mean = -0.0653755 +/- 0.0459247 \n", "Sigma = 0.982023 +/- 0.0349408 \t (limited)\n" ] } ], "source": [ "TCanvas *cpull = new TCanvas(\"cpull\", \"cpull\", 1200, 800);\n", "cpull->Divide(3, 3);\n", "\n", "std::vector pullHistos{hc0pull[0], hc0pull[1], hc0pull[2], hc1pull[0], hc1pull[1],\n", " hc1pull[2], hntotpull[0], hntotpull[1], hntotpull[2]};\n", "\n", "gStyle->SetOptStat(1100);\n", "gStyle->SetOptFit(11);\n", "\n", "for (std::size_t i = 0; i < pullHistos.size(); ++i) {\n", " cpull->cd(i + 1);\n", " pullHistos[i]->Fit(\"gaus\");\n", " pullHistos[i]->Draw(\"ep\");\n", "}\n", "\n", "cpull->Update();" ] }, { "cell_type": "markdown", "id": "fd879277", "metadata": {}, "source": [ "Draw all canvases " ] }, { "cell_type": "code", "execution_count": 13, "id": "102df421", "metadata": { "collapsed": false, "execution": { "iopub.execute_input": "2026-05-19T20:33:43.903027Z", "iopub.status.busy": "2026-05-19T20:33:43.902919Z", "iopub.status.idle": "2026-05-19T20:33:44.118095Z", "shell.execute_reply": "2026-05-19T20:33:44.117465Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "
\n", "
\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "\n", "
\n", "
\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%jsroot on\n", "gROOT->GetListOfCanvases()->Draw()" ] } ], "metadata": { "kernelspec": { "display_name": "ROOT C++", "language": "c++", "name": "root" }, "language_info": { "codemirror_mode": "text/x-c++src", "file_extension": ".C", "mimetype": " text/x-c++src", "name": "c++" } }, "nbformat": 4, "nbformat_minor": 5 }