You are here

Running queries in asynchronous (batch) mode

Running queries in asynchronous (batch) mode

 



Support for asynchronous running in PROOF has been added in ROOT version 5.04.00 . Disconnection / reconnection functionality requires a XROOTD backend and is available since ROOT version 5.08.00 .

This mode is automatically used when queries are queued by the scheduler.

Submitting a query in asynchronous mode

To have a query processed asynchronously you need to specify the option "ASYN" in the option string to TProof::Process(...)

root [] p = TProof::Open("lxb6043")
Starting master: opening connection ...
Starting master: OK
Opening connections to workers: OK (3 workers)
Setting up worker servers: OK (3 workers)
PROOF set to parallel mode (3 workers)
(class TProof*)0x82862e8
root [] .x preph1.C("http")  // defines the chain 'ch1'
root [] ch1->Process("h1analysis.C+", "ASYN")
(Long64_t)1
root [] ch1->Process("h1analysis.C+", "ASYN")
(Long64_t)2
root [] ch1->Process("h1analysis.C+", "ASYN")
(Long64_t)3

The queries are queued on the master and processed sequentially.

 

Checking the status of processing

The status of the submitted queries can be checked with TProof::ShowQueries() :

root [] p->ShowQueries()
+++
+++ Queries processed during this session: selector: 3, draw: 0
+++ #:1 ref:"session-lxb6043-1205935539-667:q2" sel:h1analysis running
+++ #:2 ref:"session-lxb6043-1205935539-667:q3" sel:h1analysis submitted
+++ #:3 ref:"session-lxb6043-1205935539-667:q1" sel:h1analysis completed evts:0-283812
+++

In this example we see that the first query has been completed and the second is being processed.

Unique query reference

Each query is uniquely identified by a reference string composed by the session tag and a submission sequential number; in the above example, 'lxb6043-1205935539-667' is the session tag (composed by the master hostname, the starting time of the master and the master process ID) and the submission number is the number following the ':q'.

Getting the output of a completed query

For each submitted query a TQueryResult object is created. This object is saved into a root file in the master sandbox and automatically sent to the attached client sessions. A pointer to the relevant result object which can be obtaibed with TProof::GetQueryResult(const char *qref):

root [] TQueryResult *qr
root [] qr = p->GetQueryResult("session-lxb6043-1205935539-667:q1")

The TQueryResult object contains all the information about the query (inputs, selector, ...) and the output list, which can be directly accessed:

root [] qr->GetOutputList()
(class TList*)0x8516610
root [] qr->GetOutputList()->ls()
OBJ: TStatus    PROOF_Status     : 0 at: 0x86fb7a0
OBJ: TH1F       hdmd    dm_d : 0 at: 0x86fb7e0
OBJ: TH2F       h2      ptD0 vs dm_d : 0 at: 0x86fbc10

Archiving the result object to a root file

The TQueryResult object associated with a query can be archived to a file using TProof::Archive(const char *qref, const char *path):

root [] p->Archive("session-lxb6043-1205935539-667:q1", "castor:/castor/cern.ch/user/g/ganis/proofresults/")

the output file will be in the form 'path/session-tag-num.root', e.g. 'castor:/castor/cern.ch/user/g/ganis/proofresults/session-lxb6043-1205935539-667-1.root' in the above example. The output file is open via TFile::Open(...), so any protocol available from the root installation can be used.

The TQueryResult object can alternatively archived via TProof::Retrieve(const char *qref, const char *path); in this case the file is first received in the client memory and then saved to 'path' (the string is unmodified in this case). Again, 'path' is open via TFile::Open(...), so any protocol available from the root installation can be used.

Disconnecting and reconnecting

After having submitted the queries in asynchronous mode the client session can quit without affecting the PROOF session. If the client re-open a session of the same PROOF cluster, it will be automatically re-attached to the processing session and notified of the progress (the progress dialog pops-up as soon as the re-connection is established). At this point the client can browse the status of the submitted queries as explained above.

The TQueryResult objects for the queries completed while disconnected are not sent back to reconnecting client automatically: they have to explicitly retrieved using TProof::Retrieve(const char *qref) before running TProof::GetQueryResult(const char *qref):

root [] p->Retrieve("session-lxb6043-1205935539-667:q2")
root [] TQueryResult *qr
root [] qr = p->GetQueryResult("session-lxb6043-1205935539-667:q2")

The full list of results available on the master get be browsed with:

root [] p->ShowQueries("A")
+++
+++ Queries processed during other sessions: 25
+++ #:1 ref:"session-lxb6043-1205777525-22784:q1" sel:AliAnalysisSelector completed evts:0-1999
+++ #:2 ref:"session-lxb6043-1205775923-16609:q2" sel:AliAnalysisSelector completed evts:0-19999
+++ #:3 ref:"session-lxb6043-1205775923-16609:q4" sel:AliAnalysisSelector completed evts:0-19999
+++ #:4 ref:"session-lxb6043-1205775923-16609:q3" sel:AliAnalysisSelector completed evts:0-1999
+++ #:5 ref:"session-lxb6043-1205775923-16609:q5" sel:AliAnalysisSelector completed evts:0-19999
+++ #:6 ref:"session-lxb6043-1205775923-16609:q1" sel:AliAnalysisSelector completed evts:0-19999
+++ #:7 ref:"session-lxb6043-1205144837-29021:q2" sel:AliAnalysisSelector completed evts:0-1999
+++ #:8 ref:"session-lxb6043-1205144837-29021:q4" sel:AliAnalysisSelector completed evts:0-199999
+++ #:9 ref:"session-lxb6043-1205144837-29021:q3" sel:AliAnalysisSelector completed evts:0-1999
+++ #:10 ref:"session-lxb6043-1205144837-29021:q5" sel:AliAnalysisSelector completed evts:0-199999
+++ #:11 ref:"session-lxb6043-1205144837-29021:q1" sel:AliAnalysisSelector completed evts:0-1999
+++ #:12 ref:"session-lxb6043-1205777661-23192:q1" sel:AliAnalysisSelector completed evts:0-1999
+++ #:13 ref:"session-lxb6043-1205777397-22180:q1" sel:AliAnalysisSelector completed evts:0-1999
+++ #:14 ref:"session-lxb6043-1205794348-22616:q1" sel:AliAnalysisSelector completed evts:0-1999
+++ #:15 ref:"session-lxb6043-1205792836-14191:q1" sel:AliAnalysisSelector completed evts:0-1999
+++ #:16 ref:"session-lxb6043-1205792318-12380:q1" sel:AliAnalysisSelector completed evts:0-1999
+++ #:17 ref:"session-lxb6043-1205935468-534:q1" sel:h1analysis completed evts:0-283812
+++ #:18 ref:"session-lxb6043-1205774860-12032:q1" sel:h1analysis completed evts:0-283812
+++ #:19 ref:"session-lxb6043-1205825635-7383:q1" sel:AliAnalysisSelector completed evts:0-19999
+++ #:20 ref:"session-lxb6043-1205935168-31354:q2" sel:h1analysis completed evts:0-283812
+++ #:22 ref:"session-lxb6043-1205935168-31354:q3" sel:h1analysis completed evts:0-283812
+++ #:23 ref:"session-lxb6043-1205935168-31354:q1" sel:h1analysis completed evts:0-283812
+++
+++ Queries processed during this session: selector: 3, draw: 0
+++ #:24 ref:"session-lxb6043-1205935539-667:q2" sel:h1analysis running
+++ #:25 ref:"session-lxb6043-1205935539-667:q3" sel:h1analysis submitted
+++ #:26 ref:"session-lxb6043-1205935539-667:q1" sel:h1analysis completed evts:0-283812
+++

and retrieved with TList *TProof::GetListOfQueries("A") . Be aware that, depending on the number and size of the result objects, this operation may be very resource-consuming .

Removing queries on the cluster

The method TProof::Remove(const char *qref) is provided to remove any reference to a query on the cluster:

root [] p->Remove("session-lxb6043-1205935539-667:q2")

If a query is not yet running it is removed from the queue of submitted queries.

This method can also be used to clean-up the queue of queries waiting to be processed:

root [] p->Remove("cleanupqueue")

Remarks

  • The default behavior is that the session will continue to run on the cluster until processing is over, if no client session has re-connected in the meantime, the session is shutdown
  • The results of the completed queries are kept in the sandbox on the master
  • Currently it is not possible to move to asynchronous a query started synchronously (Ctrl-Z functionality), except if the scheduler decides so;
  • There is currently no way to quickly refer to the queries processed during the very previous session; this will be added soon.