Logo ROOT   6.19/01
Reference Guide
List of all members | Public Member Functions | Private Types | Private Member Functions | Private Attributes | List of all members
ROOT::TTreeProcessorMP Class Reference

This class provides an interface to process a TTree dataset in parallel with multi-process technology.

ROOT::TTreeProcessorMP::Process

The possible usages of the Process method are the following:

For legacy, the following signature is also supported:

For either set of signatures, the processing function is executed as many times as needed by a pool of fNWorkers workers; the number of workers can be passed to the constructor or set via SetNWorkers. It defaults to the number of cores.
A collection containing the result of each execution is returned.
Note: the user is responsible for the deletion of any object that might be created upon execution of func, returned objects included: ROOT::TTreeProcessorMP never deletes what it returns, it simply forgets it.
Note: that the usage of ROOT::TTreeProcessorMP::Process is indicated only when the task to be executed takes more than a few seconds, otherwise the overhead introduced by Process will outrun the benefits of parallel execution on most machines.

Parameters
funca lambda expression, an std::function, a loaded macro, a functor class or a function that takes zero arguments (for the first signature) or one (for the second signature).
argsa standard container (vector, list, deque), an initializer list or a pointer to a TCollection (TList*, TObjArray*, ...).

Note: the version of ROOT::TTreeProcessorMP::Process that takes a TFileCollection* as argument incurs in the overhead of copying data from the TCollection to an STL container. Only use it when absolutely necessary.
Note: in cases where the function to be executed takes more than zero/one argument but all are fixed except zero/one, the function can be wrapped in a lambda or via std::bind to give it the right signature.
Note: the user should take care of initializing random seeds differently in each process (e.g. using the process id in the seed). Otherwise several parallel executions might generate the same sequence of pseudo-random numbers.

Return value:

Methods taking 'F func' return the return type of F. Methods taking a TSelector return a 'TList *' with the selector output list; the output list content is owned by the caller.

Examples:

See tutorials/multicore/mp102_readNtuplesFillHistosAndFit.C and tutorials/multicore/mp103__processSelector.C .

Definition at line 38 of file TTreeProcessorMP.hxx.

Public Member Functions

 TTreeProcessorMP (unsigned nWorkers=0)
 
 TTreeProcessorMP (const TTreeProcessorMP &)=delete
 
 ~TTreeProcessorMP ()=default
 
unsigned GetNWorkers () const
 
TTreeProcessorMPoperator= (const TTreeProcessorMP &)=delete
 
template<class F >
auto Process (const std::vector< std::string > &fileNames, F procFunc, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 Process a TTree dataset with a functor. More...
 
template<class F >
auto Process (const std::string &fileName, F procFunc, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 
template<class F >
auto Process (TFileCollection &collection, F procFunc, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 
template<class F >
auto Process (TChain &chain, F procFunc, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 
template<class F >
auto Process (TTree &tree, F procFunc, TEntryList &entries, ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 
template<class F >
auto Process (const std::vector< std::string > &fileNames, F procFunc, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 Process a TTree dataset with a functor: version without entry list. More...
 
template<class F >
auto Process (const std::string &fileName, F procFunc, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 
template<class F >
auto Process (TFileCollection &files, F procFunc, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 
template<class F >
auto Process (TChain &files, F procFunc, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 
template<class F >
auto Process (TTree &tree, F procFunc, ULong64_t nToProcess=0, ULong64_t jFirst=0) -> typename std::result_of< F(std::reference_wrapper< TTreeReader >)>::type
 
TListProcess (const std::vector< std::string > &fileNames, TSelector &selector, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0)
 Process a TTree dataset with a selector. More...
 
TListProcess (const std::string &fileName, TSelector &selector, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0)
 TSelector-based tree processing: dataset as a single file. More...
 
TListProcess (TFileCollection &files, TSelector &selector, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0)
 TSelector-based tree processing: dataset as a TFileCollection. More...
 
TListProcess (TChain &files, TSelector &selector, TEntryList &entries, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0)
 TSelector-based tree processing: dataset as a TChain. More...
 
TListProcess (TTree &tree, TSelector &selector, TEntryList &entries, ULong64_t nToProcess=0, ULong64_t jFirst=0)
 TSelector-based tree processing: memory resident tree. More...
 
TListProcess (const std::vector< std::string > &fileNames, TSelector &selector, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0)
 Process a TTree dataset with a selector: version without entry list. More...
 
TListProcess (const std::string &fileName, TSelector &selector, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0)
 
TListProcess (TFileCollection &files, TSelector &selector, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0)
 
TListProcess (TChain &files, TSelector &selector, const std::string &treeName="", ULong64_t nToProcess=0, ULong64_t jFirst=0)
 
TListProcess (TTree &tree, TSelector &selector, ULong64_t nToProcess=0, ULong64_t jFirst=0)
 
void SetNWorkers (unsigned n)
 

Private Types

enum  ETask : unsigned char { ETask::kNoTask, ETask::kProcByRange, ETask::kProcByFile }
 A collection of the types of tasks that TTreeProcessorMP can execute. More...
 

Private Member Functions

template<class T >
void Collect (std::vector< T > &reslist)
 Listen for messages sent by the workers and call the appropriate handler function. More...
 
void FixLists (std::vector< TObject *> &lists)
 Fix list of lists before merging (to avoid errors about duplicated objects) More...
 
template<class T >
void HandlePoolCode (MPCodeBufPair &msg, TSocket *sender, std::vector< T > &reslist)
 Handle message and reply to the worker. More...
 
void ReplyToIdle (TSocket *s)
 Reply to a worker who is idle. More...
 
void Reset ()
 Reset TTreeProcessorMP's state. More...
 
- Private Member Functions inherited from TMPClient
 TMPClient (unsigned nWorkers=0)
 Class constructor. More...
 
 TMPClient (const TMPClient &)=delete
 
 ~TMPClient ()
 Class destructor. More...
 
unsigned Broadcast (unsigned code, unsigned nMessages=0)
 Send a message with the specified code to at most nMessages workers. More...
 
template<class T >
unsigned Broadcast (unsigned code, const std::vector< T > &objs)
 Send a message with a different object to each server. More...
 
template<class T >
unsigned Broadcast (unsigned code, std::initializer_list< T > &objs)
 Send a message with a different object to each server. More...
 
template<class T >
unsigned Broadcast (unsigned code, T obj, unsigned nMessages=0)
 Send a message containing code and obj to each worker, up to a maximum number of nMessages workers. More...
 
void DeActivate (TSocket *s)
 DeActivate a certain socket. More...
 
bool Fork (TMPWorker &server)
 This method forks the ROOT session into fNWorkers children processes. More...
 
bool GetIsParent () const
 
TMonitorGetMonitor ()
 
unsigned GetNWorkers () const
 
void HandleMPCode (MPCodeBufPair &msg, TSocket *sender)
 Handle messages containing an EMPCode. More...
 
TMPClientoperator= (const TMPClient &)=delete
 
void ReapWorkers ()
 Wait on worker processes and remove their pids from fWorkerPids. More...
 
void Remove (TSocket *s)
 Remove a certain socket from the monitor. More...
 
void SetNWorkers (unsigned n)
 Set the number of workers that will be spawned by the next call to Fork() More...
 

Private Attributes

unsigned fNProcessed
 number of arguments already passed to the workers More...
 
unsigned fNToProcess
 total number of arguments to pass to the workers More...
 
ETask fTaskType = ETask::kNoTask
 the kind of task that is being executed, if any More...
 

#include <ROOT/TTreeProcessorMP.hxx>

Inheritance diagram for ROOT::TTreeProcessorMP:
[legend]

Member Enumeration Documentation

◆ ETask

enum ROOT::TTreeProcessorMP::ETask : unsigned char
strongprivate

A collection of the types of tasks that TTreeProcessorMP can execute.

It is used to interpret in the right way and properly reply to the messages received (see, for example, TTreeProcessorMP::HandleInput)

Enumerator
kNoTask 

no task is being executed

kProcByRange 

a Process method is being executed and each worker will process a certain range of each file

kProcByFile 

a Process method is being executed and each worker will process a different file

Definition at line 179 of file TTreeProcessorMP.hxx.

Constructor & Destructor Documentation

◆ TTreeProcessorMP() [1/2]

ROOT::TTreeProcessorMP::TTreeProcessorMP ( unsigned  nWorkers = 0)
explicit

◆ ~TTreeProcessorMP()

ROOT::TTreeProcessorMP::~TTreeProcessorMP ( )
default

◆ TTreeProcessorMP() [2/2]

ROOT::TTreeProcessorMP::TTreeProcessorMP ( const TTreeProcessorMP )
delete

Member Function Documentation

◆ Collect()

template<class T >
void ROOT::TTreeProcessorMP::Collect ( std::vector< T > &  reslist)
private

Listen for messages sent by the workers and call the appropriate handler function.

TTreeProcessorMP::HandlePoolCode is called on messages with a code < 1000 and TMPClient::HandleMPCode is called on messages with a code >= 1000.

Definition at line 430 of file TTreeProcessorMP.hxx.

◆ FixLists()

void ROOT::TTreeProcessorMP::FixLists ( std::vector< TObject *> &  lists)
private

Fix list of lists before merging (to avoid errors about duplicated objects)

Definition at line 323 of file TTreeProcessorMP.cxx.

◆ GetNWorkers()

unsigned ROOT::TTreeProcessorMP::GetNWorkers ( ) const
inline

Definition at line 163 of file TTreeProcessorMP.hxx.

◆ HandlePoolCode()

template<class T >
void ROOT::TTreeProcessorMP::HandlePoolCode ( MPCodeBufPair msg,
TSocket sender,
std::vector< T > &  reslist 
)
private

Handle message and reply to the worker.

Definition at line 404 of file TTreeProcessorMP.hxx.

◆ operator=()

TTreeProcessorMP& ROOT::TTreeProcessorMP::operator= ( const TTreeProcessorMP )
delete

◆ Process() [1/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( const std::vector< std::string > &  fileNames,
F  procFunc,
TEntryList entries,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Process a TTree dataset with a functor.

Template Parameters
Ffunctor returning a pointer to TObject or inheriting classes and taking a TTreeReader& (both enforced at compile-time)

Dataset definition:

Parameters
[in]fileNamesvector of strings with the paths of the files with the TTree to process
[in]fileNamestring with the path of the files with the TTree to process
[in]collectionTFileCollection with the files with the TTree to process
[in]chainTChain with the files with the TTree to process
[in]treeTTree to process
[in]entriesTEntryList to filter the dataset
[in]treeNameName of the TTree to process
[in]nToProcessNumber of entries to process (0 means all)
[in]jFirstFirst entry to process (0 means the first of the first file)

Definition at line 189 of file TTreeProcessorMP.hxx.

◆ Process() [2/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( const std::string &  fileName,
F  procFunc,
TEntryList entries,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Definition at line 256 of file TTreeProcessorMP.hxx.

◆ Process() [3/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( TFileCollection collection,
F  procFunc,
TEntryList entries,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Definition at line 266 of file TTreeProcessorMP.hxx.

◆ Process() [4/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( TChain chain,
F  procFunc,
TEntryList entries,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Definition at line 280 of file TTreeProcessorMP.hxx.

◆ Process() [5/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( TTree tree,
F  procFunc,
TEntryList entries,
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Definition at line 295 of file TTreeProcessorMP.hxx.

◆ Process() [6/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( const std::vector< std::string > &  fileNames,
F  procFunc,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Process a TTree dataset with a functor: version without entry list.

No TEntryList versions of generic processor.

Template Parameters
Ffunctor returning a pointer to TObject or inheriting classes and taking a TTreeReader& (both enforced at compile-time)

Dataset definition:

Parameters
[in]fileNamesvector of strings with the paths of the files with the TTree to process
[in]fileNamestring with the path of the files with the TTree to process
[in]collectionTFileCollection with the files with the TTree to process
[in]chainTChain with the files with the TTree to process
[in]treeTTree to process
[in]treeNameName of the TTree to process
[in]nToProcessNumber of entries to process (0 means all)
[in]jFirstFirst entry to process (0 means the first of the first file)

Definition at line 353 of file TTreeProcessorMP.hxx.

◆ Process() [7/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( const std::string &  fileName,
F  procFunc,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Definition at line 363 of file TTreeProcessorMP.hxx.

◆ Process() [8/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( TFileCollection files,
F  procFunc,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Definition at line 373 of file TTreeProcessorMP.hxx.

◆ Process() [9/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( TChain files,
F  procFunc,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Definition at line 383 of file TTreeProcessorMP.hxx.

◆ Process() [10/20]

template<class F >
auto ROOT::TTreeProcessorMP::Process ( TTree tree,
F  procFunc,
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
) -> typename std::result_of<F(std::reference_wrapper<TTreeReader>)>::type

Definition at line 393 of file TTreeProcessorMP.hxx.

◆ Process() [11/20]

TList * ROOT::TTreeProcessorMP::Process ( const std::vector< std::string > &  fileNames,
TSelector selector,
TEntryList entries,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

Process a TTree dataset with a selector.

TSelector-based tree processing: dataset as a vector of files.

Dataset definition:

Parameters
[in]fileNamesvector of strings with the paths of the files with the TTree to process
[in]fileNamestring with the path of the files with the TTree to process
[in]collectionTFileCollection with the files with the TTree to process
[in]chainTChain with the files with the TTree to process
[in]treeTTree to process
[in]selectorInstance of TSelector to be applied to the dataset
[in]entriesTEntryList to filter the dataset
[in]treeNameName of the TTree to process
[in]nToProcessNumber of entries to process (0 means all)
[in]jFirstFirst entry to process (0 means the first of the first file)

Definition at line 159 of file TTreeProcessorMP.cxx.

◆ Process() [12/20]

TList * ROOT::TTreeProcessorMP::Process ( const std::string &  fileName,
TSelector selector,
TEntryList entries,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

TSelector-based tree processing: dataset as a single file.

Definition at line 277 of file TTreeProcessorMP.cxx.

◆ Process() [13/20]

TList * ROOT::TTreeProcessorMP::Process ( TFileCollection files,
TSelector selector,
TEntryList entries,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

TSelector-based tree processing: dataset as a TFileCollection.

Definition at line 249 of file TTreeProcessorMP.cxx.

◆ Process() [14/20]

TList * ROOT::TTreeProcessorMP::Process ( TChain files,
TSelector selector,
TEntryList entries,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

TSelector-based tree processing: dataset as a TChain.

Definition at line 263 of file TTreeProcessorMP.cxx.

◆ Process() [15/20]

TList * ROOT::TTreeProcessorMP::Process ( TTree tree,
TSelector selector,
TEntryList entries,
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

TSelector-based tree processing: memory resident tree.

Definition at line 96 of file TTreeProcessorMP.cxx.

◆ Process() [16/20]

TList * ROOT::TTreeProcessorMP::Process ( const std::vector< std::string > &  fileNames,
TSelector selector,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

Process a TTree dataset with a selector: version without entry list.

No TEntryList versions of selector processor.

Dataset definition:

Parameters
[in]fileNamesvector of strings with the paths of the files with the TTree to process
[in]fileNamestring with the path of the files with the TTree to process
[in]collectionTFileCollection with the files with the TTree to process
[in]chainTChain with the files with the TTree to process
[in]treeTTree to process
[in]selectorInstance of TSelector to be applied to the dataset
[in]treeNameName of the TTree to process
[in]nToProcessNumber of entries to process (0 means all)
[in]jFirstFirst entry to process (0 means the first of the first file)

Definition at line 288 of file TTreeProcessorMP.cxx.

◆ Process() [17/20]

TList * ROOT::TTreeProcessorMP::Process ( const std::string &  fileName,
TSelector selector,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

Definition at line 295 of file TTreeProcessorMP.cxx.

◆ Process() [18/20]

TList * ROOT::TTreeProcessorMP::Process ( TFileCollection files,
TSelector selector,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

Definition at line 302 of file TTreeProcessorMP.cxx.

◆ Process() [19/20]

TList * ROOT::TTreeProcessorMP::Process ( TChain files,
TSelector selector,
const std::string &  treeName = "",
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

Definition at line 309 of file TTreeProcessorMP.cxx.

◆ Process() [20/20]

TList * ROOT::TTreeProcessorMP::Process ( TTree tree,
TSelector selector,
ULong64_t  nToProcess = 0,
ULong64_t  jFirst = 0 
)

Definition at line 316 of file TTreeProcessorMP.cxx.

◆ ReplyToIdle()

void ROOT::TTreeProcessorMP::ReplyToIdle ( TSocket s)
private

Reply to a worker who is idle.

If still events to process, tell the worker. Otherwise ask for a result

Definition at line 350 of file TTreeProcessorMP.cxx.

◆ Reset()

void ROOT::TTreeProcessorMP::Reset ( void  )
private

Reset TTreeProcessorMP's state.

Definition at line 339 of file TTreeProcessorMP.cxx.

◆ SetNWorkers()

void ROOT::TTreeProcessorMP::SetNWorkers ( unsigned  n)
inline

Definition at line 162 of file TTreeProcessorMP.hxx.

Member Data Documentation

◆ fNProcessed

unsigned ROOT::TTreeProcessorMP::fNProcessed
private

number of arguments already passed to the workers

Definition at line 173 of file TTreeProcessorMP.hxx.

◆ fNToProcess

unsigned ROOT::TTreeProcessorMP::fNToProcess
private

total number of arguments to pass to the workers

Definition at line 174 of file TTreeProcessorMP.hxx.

◆ fTaskType

ETask ROOT::TTreeProcessorMP::fTaskType = ETask::kNoTask
private

the kind of task that is being executed, if any

Definition at line 185 of file TTreeProcessorMP.hxx.


The documentation for this class was generated from the following files: