{
"cells": [
{
"cell_type": "markdown",
"id": "92d192a7",
"metadata": {},
"source": [
"# distrdf001_spark_connection\n",
"Configure a Spark connection and fill two histograms distributedly.\n",
"\n",
"This tutorial shows the ingredients needed to setup the connection to a Spark\n",
"cluster, namely a SparkConf object holding configuration parameters and a\n",
"SparkContext object created with the desired options. After this initial\n",
"setup, an RDataFrame with distributed capabilities is created and connected\n",
"to the SparkContext instance. Finally, a couple of histograms are drawn from\n",
"the created columns in the dataset.\n",
"\n",
"\n",
"\n",
"\n",
"**Author:** Vincenzo Eduardo Padulano \n",
"This notebook tutorial was automatically generated with ROOTBOOK-izer from the macro found in the ROOT repository on Tuesday, May 19, 2026 at 08:10 PM."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "dc9ef01a",
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2026-05-19T20:10:52.082391Z",
"iopub.status.busy": "2026-05-19T20:10:52.082275Z",
"iopub.status.idle": "2026-05-19T20:10:53.233852Z",
"shell.execute_reply": "2026-05-19T20:10:53.233442Z"
}
},
"outputs": [],
"source": [
"import pyspark\n",
"import ROOT"
]
},
{
"cell_type": "markdown",
"id": "e3007900",
"metadata": {},
"source": [
"Setup the connection to Spark\n",
"First create a dictionary with keys representing Spark specific configuration\n",
"parameters. In this tutorial we use the following configuration parameters:\n",
"\n",
" 1. spark.app.name: The name of the Spark application\n",
" 2. spark.master: The Spark endpoint responsible for running the\n",
" application. With the syntax \"local[2]\" we signal Spark we want to run\n",
" locally on the same machine with 2 cores, each running a separate\n",
" process. The default behaviour of a Spark application would run\n",
" locally on the same machine with as many concurrent processes as\n",
" available cores, that could be also written as \"local[*]\".\n",
"\n",
"If you have access to a remote cluster you should substitute the endpoint URL\n",
"of your Spark master in the form \"spark://HOST:PORT\" in the value of\n",
"`spark.master`. Depending on the availability of your cluster you may request\n",
"more computing nodes or cores per node with a similar configuration:\n",
"\n",
" sparkconf = pyspark.SparkConf().setAll(\n",
" {\"spark.master\": \"spark://HOST:PORT\",\n",
" \"spark.executor.instances\": ,\n",
" \"spark.executor.cores\" ,}.items())\n",
"\n",
"You can find all configuration options and more details in the official Spark\n",
"documentation at https://spark.apache.org/docs/latest/configuration.html ."
]
},
{
"cell_type": "markdown",
"id": "30021688",
"metadata": {},
"source": [
"Create a SparkConf object with all the desired Spark configuration parameters"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "59cb75ed",
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2026-05-19T20:10:53.245118Z",
"iopub.status.busy": "2026-05-19T20:10:53.244929Z",
"iopub.status.idle": "2026-05-19T20:10:53.358618Z",
"shell.execute_reply": "2026-05-19T20:10:53.358124Z"
}
},
"outputs": [],
"source": [
"sparkconf = pyspark.SparkConf().setAll(\n",
" {\"spark.app.name\": \"distrdf001_spark_connection\",\n",
" \"spark.master\": \"local[2]\",\n",
" \"spark.driver.memory\": \"4g\"}.items())"
]
},
{
"cell_type": "markdown",
"id": "8b7c2843",
"metadata": {},
"source": [
"Create a SparkContext with the configuration stored in `sparkconf`"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "0257b214",
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2026-05-19T20:10:53.377267Z",
"iopub.status.busy": "2026-05-19T20:10:53.377125Z",
"iopub.status.idle": "2026-05-19T20:10:57.193879Z",
"shell.execute_reply": "2026-05-19T20:10:57.173347Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: Using incubator modules: jdk.incubator.vector\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting default log level to \"WARN\".\n",
"To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"26/05/19 20:10:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n"
]
}
],
"source": [
"sparkcontext = pyspark.SparkContext(conf=sparkconf)"
]
},
{
"cell_type": "markdown",
"id": "6d239e52",
"metadata": {},
"source": [
"Create an RDataFrame that will use Spark as a backend for computations"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "9cd10e7e",
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2026-05-19T20:10:57.195911Z",
"iopub.status.busy": "2026-05-19T20:10:57.195694Z",
"iopub.status.idle": "2026-05-19T20:10:58.356988Z",
"shell.execute_reply": "2026-05-19T20:10:58.340743Z"
}
},
"outputs": [],
"source": [
"df = ROOT.RDataFrame(1000, executor=sparkcontext)"
]
},
{
"cell_type": "markdown",
"id": "90fca4fa",
"metadata": {},
"source": [
"Set the random seed and define two columns of the dataset with random numbers."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "ed28845e",
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2026-05-19T20:10:58.397263Z",
"iopub.status.busy": "2026-05-19T20:10:58.397071Z",
"iopub.status.idle": "2026-05-19T20:10:58.568038Z",
"shell.execute_reply": "2026-05-19T20:10:58.557601Z"
}
},
"outputs": [],
"source": [
"ROOT.gRandom.SetSeed(1)\n",
"df_1 = df.Define(\"gaus\", \"gRandom->Gaus(10, 1)\").Define(\"exponential\", \"gRandom->Exp(10)\")"
]
},
{
"cell_type": "markdown",
"id": "653b093e",
"metadata": {},
"source": [
"Book an histogram for each column"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "e22c541d",
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2026-05-19T20:10:58.592582Z",
"iopub.status.busy": "2026-05-19T20:10:58.592396Z",
"iopub.status.idle": "2026-05-19T20:10:58.730653Z",
"shell.execute_reply": "2026-05-19T20:10:58.710435Z"
}
},
"outputs": [],
"source": [
"h_gaus = df_1.Histo1D((\"gaus\", \"Normal distribution\", 50, 0, 30), \"gaus\")\n",
"h_exp = df_1.Histo1D((\"exponential\", \"Exponential distribution\", 50, 0, 30), \"exponential\")"
]
},
{
"cell_type": "markdown",
"id": "91f94090",
"metadata": {},
"source": [
"Plot the histograms side by side on a canvas"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "efe88335",
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2026-05-19T20:10:58.758515Z",
"iopub.status.busy": "2026-05-19T20:10:58.758342Z",
"iopub.status.idle": "2026-05-19T20:11:05.765785Z",
"shell.execute_reply": "2026-05-19T20:11:05.753628Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c = ROOT.TCanvas(\"distrdf001\", \"distrdf001\", 800, 400)\n",
"c.Divide(2, 1)\n",
"c.cd(1)\n",
"h_gaus.DrawCopy()\n",
"c.cd(2)\n",
"h_exp.DrawCopy()"
]
},
{
"cell_type": "markdown",
"id": "fc47b5fc",
"metadata": {},
"source": [
"Save the canvas"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "6f2ed46c",
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2026-05-19T20:11:05.790328Z",
"iopub.status.busy": "2026-05-19T20:11:05.790162Z",
"iopub.status.idle": "2026-05-19T20:11:05.987926Z",
"shell.execute_reply": "2026-05-19T20:11:05.976741Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Info in : png file distrdf001_spark_connection.png has been created\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Saved figure to distrdf001_spark_connection.png\n"
]
}
],
"source": [
"c.SaveAs(\"distrdf001_spark_connection.png\")\n",
"print(\"Saved figure to distrdf001_spark_connection.png\")"
]
},
{
"cell_type": "markdown",
"id": "a10e2301",
"metadata": {},
"source": [
"Draw all canvases "
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "b6f635d9",
"metadata": {
"collapsed": false,
"execution": {
"iopub.execute_input": "2026-05-19T20:11:06.014745Z",
"iopub.status.busy": "2026-05-19T20:11:06.014551Z",
"iopub.status.idle": "2026-05-19T20:11:06.227815Z",
"shell.execute_reply": "2026-05-19T20:11:06.217228Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"
\n",
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from ROOT import gROOT \n",
"gROOT.GetListOfCanvases().Draw()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}