Creating and saving the performance tree

As explained in the benchmark utilities page, it possible to generate some performance histograms and a tree with information which can be used to monitor the performance of a cluster. This can be used to trace possible bottlenecks and therefore it is a useful debugging tool.

The histograms contain the following information:

Name Type Description
PROOF_PacketsHist TH1D "Packets processed per Worker"
PROOF_EventsHist TH1D "Events processed per Worker"
PROOF_ProcPcktHist TH1I "Packets being processed per Worker"
PROOF_NodeHist TH1D "Workers per Fileserving Node"
PROOF_LatencyHist TH2D "GetPacket Latency per Worker"
PROOF_ProcTimeHist TH2D "Packet Processing Time per Worker"
PROOF_CpuTimeHist TH2D "Packet CPU Time per Worker"

All histograms are dynamically updated, so they can be used in feedback to monitor the worker activity. Starting with ROOT v5.34, the option "fb=stats" in TProof::Process enables feedback for the first three histograms displayed in one canvas.

The tree contains detailed information about 'packet' processing and file opening; the interpretation of the tree requires some expertise, though.

The purpose of this section is to explain how to create and save the relevant information.

  1. Creating the performance tree with ROOT >= 5.33
  2. Creating the performance tree with ROOT <= 5.32 (including 5-32 patch releases)
    1. Setting up the parameters
    2. Saving the tree

1. Creating the performance tree with ROOT >= 5.33

For ROOT >= 5.33 (SVN rev 42382) TProof provides two methods to facilitate the generation and saving of the performance tree: 

void   TProof::SetPerfTree(const char *pf = "perftree.root", Bool_t withWrks = kFALSE);
A call to SetPerfTree before the query set ups the relevant parameters to create the performance tree; when the query is over the tree is saved to the file specified by the first argument (default 'perftree.root'). The second argument controls whether the start and stop information from the workers should also be recorded (default is not); generation of the performance tree must be re-enabled before each run. 
Int_t  TProof::SavePerfTree(const char *pf = 0, const char *qref = 0);
This method saves the performance tree for query whose unique reference is 'qref' (by default last query) to be saved to the file indicated by the first argument (default is the path previously set via a call to SetPerfTree or 'perftree.root'). This method is called internally by TProof::Process when the creation and saving of the performance tree is enabled.

2. Creating the performance tree with ROOT <= 5.32 (including 5-32 patch releases)

2.1 Setting up the parameters

To create the performance tree and histograms, the following should be done before running the query:

root[] proof->SetParameter("PROOF_StatsHist", "")
root[] proof->SetParameter("PROOF_StatsTrace", "")

or

root[] gEnv->SetValue("Proof.StatsHist",1)
root[] gEnv->SetValue("Proof.StatsTrace",1)

(to save detailed information per worker the following setting needs also to be added

root[] proof->SetParameter("PROOF_SlaveStatsTrace", "")

or

root[] gEnv->SetValue("Proof.SlaveStatsTrace",1)

but this is typically not required).

After processing the query, the output list should contain, in addition to the user output objects, the following objects:

root[] proof->GetOutputList()->ls()
 ...
 OBJ: TTree     PROOF_PerfStats PROOF Statistics : 0 at: 0x163ca50
 OBJ: TH1D      PROOF_PacketsHist       Packets processed per Worker : 0 at: 0x163a180
 OBJ: TH1I      PROOF_ProcPcktHist      Packets being processed per Worker : 0 at: 0x163a180
 OBJ: TH1D      PROOF_EventsHist        Events processed per Worker : 0 at: 0x1a93b70
 OBJ: TH1D      PROOF_NodeHist  Slaves per Fileserving Node : 0 at: 0x1639b20
 OBJ: TH2D      PROOF_LatencyHist       GetPacket Latency per Worker : 0 at: 0x1a90c40
 OBJ: TH2D      PROOF_ProcTimeHist      Packet Processing Time per Worker : 0 at: 0x1b07c40
 OBJ: TH2D      PROOF_CpuTimeHist       Packet CPU Time per Worker : 0 at: 0x1b40860
 ...
root []

The tree "PROOF_PerfStats" contains detailed information about the query.

2.2 Saving the tree

To save the performance tree and histograms into a file one can execute the macro $ROOTSYS/test/ProofBench/SavePerfInfo.C with, as argument, the full path to the file where to save the information:Crea

root[] .x $ROOTSYS/test/ProofBench/SavePerfInfo.C("/tmp/perf.root")