You are here

Accessing the sandbox

Since version 5.25/02 (more precisely, SVN trunk #29580) the PROOF manager class TProofMgr provides some functionality to access the sandbox. This page describes the functionality available in the trunk.

Available functionality:

  1. Functions to access all sandboxes
    1. TProofMgr::Ls: listing the content of the sandbox
    2. TProofMgr::Rm: removing files from the sandbox
    3. TProofMgr::More, TProofMgr::Tail, TProofMgr::Grep: accessing the content of a file
    4. TProofMgr::Stat: accesing the status of a file
    5. TProofMgr::Md5sum: calculating the MD5 check sum of a file
  2. Functions to access files on the master
    1. TProofMgr::GetFile: downloading a file from the master
    2. TProofMgr::PutFile: uploading a file to the master
    3. TProofMgr::Cp: copying a file from/to the master sandbox
  3. Directives modifying the behavior of these commands

Functionality implying modifications is limited to the sandbox. Browsing functionality can be extended by the administrator to a list of paths. Superusers can browse all the paths tey are allowed by their credentials.


1. Functions to access all the sandboxes

The functions described in this sub-section take three arguments: what, specifying the path(s), how, specifying options, and where, specifying the nodes where to run. The meaning of the first two, what and how, depend on the command and are detailed in each sub-section. The string where, instead, has the same meaning for all the functions, with some restrictions underlined where relevant: where indicates where to run the command, defaulting to master only; where="all" runs the command on all workers (the output, when relevant - e.g. ls, is prefixed by the node name); where="" runs the command on the specified worker node; wild cards are not supported.  

1.1 Listing the content of the sandbox

TProofMgr::Ls(const char *what, const char *how, const char *where)

This command runs the unix command 'ls' on the master or on the worker nodes.  The string what specifies the path to list, absolute or referred to the sandbox, e.g. "" or "~/" list the sandbox content; listing of absolute paths is restricted by default to superusers. The string how specifies the options to be passed to 'ls', e.g. '-l' to show full information about the file status (see here for where).
Example:

root [0] mgr = TProofMgr::Create("alicecaf.cern.ch:31093")
root [1] mgr->Ls()
Node: lxfsrd0506.cern.ch:31093
-----
cache                                session-lxfsrd0506-1238687519-26321
datasets                             session-lxfsrd0506-1238687684-26814
last-master-session                  session-lxfsrd0506-1238688131-28013
packages                             session-lxfsrd0506-1245399462-18523
queries                              session-lxfsrd0506-1247917607-18236
session-lxfsrd0506-1238687383-26012  session-lxfsrd0506-1247918526-22124
session-lxfsrd0506-1238687447-26173
root [2] mgr->Ls("packages","-lt")
Node: lxfsrd0506.cern.ch:31093
-----
total 344
-rw-r--r--  1 ganis sf  91245 Jul 23 12:00 STEERBase.par
drwxr-xr-x  3 ganis sf    844 Jul 18 13:25 ANALYSIS
-rw-r--r--  1 ganis sf  36575 Jul 18 13:25 ANALYSIS.par
drwxr-xr-x  3 ganis sf  16384 Jul 18 13:25 ESD
-rw-r--r--  1 ganis sf 186850 Jul 18 13:24 ESD.par
drwxr-xr-x  3 ganis sf  16384 Jul 18 13:24 STEERBase

 

1.2 Removing files from the sandbox

TProofMgr::Rm(const char *what, const char *how, const char *where)

This command runs the unix command 'rm' on the path specified by what. The meaning of the arguments is the same as for TProofMgr::Ls . The following limitations apply:

  1. Wild cards are only allowed for files in the sandbox
  2. It is not possible to delete the sandbox basic directories.

Example:

root [9] mgr->Ls("~/packages")
Node: lxfsrd0506.cern.ch:31093
-----
ANALYSIS  ANALYSIS.par  ESD  ESD.par  STEERBase  STEERBase.par
root [10] mgr->Rm("~/packages/ANALYSIS.par")
Do you really want to remove '~/packages/ANALYSIS.par'? [N/y]Y
Node: lxfsrd0506.cern.ch:31093
-----
root [11] mgr->Ls("~/packages")
Node: lxfsrd0506.cern.ch:31093
-----
ANALYSIS  ESD  ESD.par  STEERBase  STEERBase.par
root [12] mgr->Rm("~/packages","-fr")
Do you really want to remove '~/packages'? [N/y]y
Node: lxfsrd0506.cern.ch:31093
-----
lxfsrd0506.cern.ch: removing a basic sandbox directory is not allowed: /pool/proofbox-xpd/ganis/packages

1.3 Accessing the content of a file

TProofMgr::More(const char *what, const char *how, const char *where)
TProofMgr::Tail(const char *what, const char *how, const char *where)
TProofMgr::Grep(const char *what, const char *how, const char *where)

These commands run the related unix commands ('more', 'tail', 'grep') on the path specified by what. The meaning of the arguments is the same as for TProofMgr::Ls .

Grep example:

root [1] mgr->Grep("~/cache/AliAnalysisTaskPt.cxx", "-n ::")
Node: lxfsrd0506.cern.ch:31093
-----
41:AliAnalysisTaskPt::AliAnalysisTaskPt(const char *name)
48:  DefineInput(0, TChain::Class());
50:  DefineOutput(0, TH1F::Class());
54:void AliAnalysisTaskPt::ConnectInputData(Option_t *)
68:    AliESDInputHandler *esdH = dynamic_cast (AliAnalysisManager::GetAnalysisManager()->GetInputEventHandler());
78:void AliAnalysisTaskPt::CreateOutputObjects()
109:  TGrid* alien = TGrid::Connect("alien://");
110:  TGridResult* result = alien->Command("ls", 0,TAlien::kOUTPUT);
115:  //TFile::Open("alien:///alice/data/2008/LHC08d/000062564/ESDs/pass1/08000062564000.10/AliESDs.root");
129:void AliAnalysisTaskPt::Exec(Option_t *)
157:void AliAnalysisTaskPt::Terminate(Option_t *)

More example:
root [2] mgr->More("~/cache/AliAnalysisTaskPt.h")
Node: lxfsrd0506.cern.ch:31093
-----
::::::::::::::
/pool/proofbox-xpd/ganis/cache/AliAnalysisTaskPt.h
::::::::::::::
#ifndef AliAnalysisTaskPt_cxx
#define AliAnalysisTaskPt_cxx

// example of an analysis task creating a p_t spectrum
// Authors: Panos Cristakoglou, Jan Fiete Grosse-Oetringhaus, Christian Klein-Boesing

class TH1F;
class AliESDEvent;

#include "AliAnalysisTask.h"

...

Tail example:
root [3] mgr->Tail("~/cache/AliAnalysisTaskPt.cxx")
Node: lxfsrd0506.cern.ch:31093
-----
  fHistPt = dynamic_cast (GetOutputData(0));
  if (!fHistPt) {
    Printf("ERROR: fHistPt not available");
    return;
  }

  TCanvas *c1 = new TCanvas("AliAnalysisTaskPt","Pt",10,10,510,510);
  c1->cd(1)->SetLogy();
  fHistPt->DrawCopy("E");
}

1.4 Accessing the status of a file

Int_t TProofMgr::Stat(const char *what, FileStat_t &status, const char *where)

This function allows to get information about the file what on the master. The meaning of what and of where are the same as for TProofMgr::Ls, except that where="all" is not supported;the output is the same of the TSystem::GetPathInfo functions.

1.5 Calculating the MD5 check sum of a file

Int_t TProofMgr::Md5sum(const char *what, TString &sum, const char *where)

This function allows to get the MD5 check sum the file what on the master. The meaning of what and of where are the same as for TProofMgr::Ls, except that where="all" is not supported ; the output is the string version of the check sum.
Example:

root [4] TString sum
root [5] mgr->Md5sum("~/cache/AliAnalysisTaskPt.cxx", sum)
(Int_t)0
root [6] sum
(class TString)"b24e6f4ca256014b959a6ad7af0f45b4"

 

2. Functions to access files on the master

2.1 Downloading a file from the master

Int_t TProofMgr::GetFile(const char *remote, const char *local, const char *options)

This function allows to download the file remote (same conventions as for what in TProofMgr::Ls) to the local file specified by local. If local is missing or specifies a directory, the file name of remote is added to 'local'. If the 'local' file exists, a check on the MD5 sums is done; if the files are not equal the caller is asked whether the she/he wants to overwrite the file. If option="force" the file is always overwritten without asking.
A text base progress bar is displayed during downloading. The MD5 check sums are checked for consistency at the end of downloading.
Example:

root [1] mgr->GetFile("~/packages/STEERBase.par", "~/")
[GetFile] Total 0.09 MB |====================| 100.00 % [7.4 MB/s]
(Int_t)0

2.2 Uploading a file to the master

Int_t TProofMgr::PutFile(const char *local, const char *remote, const char *options)

This function allows to upload the file local to the file remote (same conventions as for what in TProofMgr::Ls). If remote is missing or specifies a directory, the file name of local is added to remote. If the remote file exists, a check on the MD5 sums is done; if the files are not equal the caller is asked whether the she/he wants to overwrite the file. If option="force" the file is always overwritten without asking.
A text base progress bar is displayed during uploading.The MD5 check sums are checked for consistency at the end of uploading.
Example:

root [2] mgr->PutFile("~/STEERBase.par", "~/packages/", "force")
[PutFile] Total 0.09 MB |====================| 100.00 % [10.1 MB/s]
(Int_t)0

2.3 Copying a file from/to the master sandbox

Int_t TProofMgr::Cp(const char *src, const char *dst, const char *fmt)

This command allows to remotely copy a file from the sandbox to another destination, possibly remote, or to copy a file from a possibly remote source to the sandbox. Either the source src or the destination dst must be in the sandbox. The file moving is done with some predefined commands specified via the card xpd.cpcmd; the command to use is chosen based on the file protocol. Protocols supported by default are

  1. root:// or xrd:// using 'xrdcp';
  2. http:// or https:// using 'wget' ; this is supported only as source.

If both source and destination are local the standard 'cp' is used. The fmt can be used to modify the format of the command, for example to add options: note that it should always contain two string converters '%s' for the source and destination files.

Example:

root [8] mgr->Cp("http://root.cern.ch/files/h1/dstarmb.root","~/cache/")
--14:52:53--  http://root.cern.ch/files/h1/dstarmb.root
           => `/pool/proofbox-xpd/ganis/cache/dstarmb.root'
Resolving root.cern.ch... 128.142.141.37
Connecting to root.cern.ch|128.142.141.37|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21,330,730 (20M) [text/plain]
    0K .......... .......... .......... .......... ..........  0%   44.84 MB/s
   50K .......... .......... .......... .......... ..........  0%   76.53 MB/s
...
20650K .......... .......... .......... .......... .......... 99%  108.99 MB/s
20700K .......... .......... .......... .......... .......... 99%   96.50 MB/s
20750K .......... .......... .......... .......... .......... 99%  111.48 MB/s
20800K .......... .......... ..........                      100%  153.42 MB/s
14:52:53 (100.61 MB/s) - `/pool/proofbox-xpd/ganis/cache/dstarmb.root' saved [21330730/21330730]
(Int_t)0

9. Directives modifying the behavior of these commands

Two directive exists to modify the behavior of these commands: xpd.exportpath to add browsable paths to the default list (the sandbox) and xpd.cpcmd to specify additional commands to be used to with TProofMgr::Cp .