You are here

The PQ2 tools

NB: The tools described in this page are not maintained any-longer and the plan is to remove them starting with version 6-16/00. In the unlikely circumstance in which this removal turns out to be disruptive for your workflow, please contact the ROOT team via the Forum, the roottalk digets or the JIRA bug/task/request tracker.



Since ROOT version 5-24/00 a set of scripts to facilitate the interaction with a dataset manager are available under $ROOTSYS/etc/proof/utils/pq2. PQ2 stands for PROOF Quick Query, and echoes the ATLAS tool dq2.

In version 5-27/02 (and 5-26-00-proof) a new version of the scripts (and some new scripts) has been introduced. The scripts are now using a new ROOT executable - pq2 - available under $ROOTSYS/bin. The new (version of the) scripts is described in detail in man pages distributed with ROOT which will be shortly available in this page.

NB: The rest of this page is not yet updated. It gives a feeling of the functionality provided by the scripts but the synatx is not always exact. This will be corrected soon.

There are currently seven scripts, four for browsing the available information and three for modifying the repository. As mentioned in the TDataSetManager page, the repository is a directory, structured per-group and pr-user, with a ROOT file per dataset, containing a TFileCollection object.

The scripts access the repository via a PROOF master. This is not strictly needed (a TDataSetManager object can instantiatiated from everywhere on any accessible directory) and future developments may allow more flexible setups.

The scripts requires the URL of the PROOF master which is taken from the command line or from the environment variable PROOFURL.

  1. Browsing tools
    1. pq2-ls
    2. pq2-ls-files
    3. pq2-ls-files-server
    4. pq2-info-server
  2. Modifiers
    1. pq2-put
    2. pq2-verify
    3. pq2-rm
       

 

1. Browsing scripts

These scripts allow to browse the content of the repository

1.2. pq2-ls

pq2-ls [what] [masterurl]
what
Selection string: use "*" (with quotes!) for all datasets; use "/mygroup/*" to show all datasets belonging to group 'mygroup'
masterurl
Entry point for the PROOF master to be used to query the dataset repository
Function
Display header information about the selected datasets
Arguments
Default
Display the datasets owned by the client and the ones under COMMON.

Example:

$~> echo $PROOFURL
alicecaf.cern.ch:31093
$~> etc/proof/utils/pq2/pq2-ls
Dataset URI                               | # Files | Default tree | # Events |   Disk   | Staged
/COMMON/COMMON/LHC08c11_10TeV_0.5T        |    1850 | /esdTree     |   277500 |   129 GB |  100 %
/COMMON/COMMON/LHC08c12_0.9TeV_0.5T       |    2126 | /esdTree     |   318900 |   101 GB |  100 %
/COMMON/COMMON/LHC08c13_10TeV_0T          |    2312 | /esdTree     |   346500 |   213 GB |   99 %
/COMMON/COMMON/LHC08c14_0.9TeV_0T         |    1646 | /esdTree     |   246900 |    94 GB |  100 %
/COMMON/COMMON/LHC08c15_10TeV_0.5T_Phojet |    1526 | /esdTree     |   222300 |    98 GB |   97 %
/COMMON/COMMON/LHC08c16_0.9TeV_0.5T_Phojet|    1415 | /esdTree     |   201750 |    71 GB |   95 %
/COMMON/COMMON/LHC08c17_10TeV_0T_Phojet   |    1314 | /esdTree     |   180750 |   109 GB |   91 %
/COMMON/COMMON/LHC08c18_0.9TeV_0T_Phojet  |    1554 | /esdTree     |   222750 |    96 GB |   95 %
/COMMON/COMMON/LHC09a4_10TeV              |    6831 | /HLTesdTree  |  1342200 |   486 GB |   98 %
/COMMON/COMMON/LHC09a4_10TeV_200k         |     989 | /HLTesdTree  |   193800 |    70 GB |   97 %
/COMMON/COMMON/LHC09a4_run8010X           |    3042 | /HLTesdTree  |   602200 |   250 GB |   98 %
/COMMON/COMMON/LHC09a4_run8100X           |    2947 | /HLTesdTree  |   586800 |   154 GB |   99 %
/COMMON/COMMON/LHC09a4_run81011           |     989 | /HLTesdTree  |   197600 |    52 GB |   99 %
/COMMON/COMMON/LHC09a4_run8101X           |    7744 | /esdTree     |  1547000 |   408 GB |   99 %
/COMMON/COMMON/LHC09a4_run8158X           |    9683 | /HLTesdTree  |  1932000 |   511 GB |   99 %
/COMMON/COMMON/LHC09a4_run8159X           |    5745 | /HLTesdTree  |  1146800 |   303 GB |   99 %
/COMMON/COMMON/LHC09a5_run9000X           |    8125 | /HLTesdTree  |  1611400 |   450 GB |   99 %
/COMMON/COMMON/LHC09a6_run9200X           |    5791 | /HLTesdTree  |  1141200 |   486 GB |   98 %
/COMMON/COMMON/tutorial_small             |     200 | /esdTree     |    20000 |     8 GB |  100 %
/proofteam/ganis/LHC09a4_10TeV-dev        |    6831 | /esdTree     |  1341000 |   486 GB |   98 %
/proofteam/ganis/LHC09a4_10TeV_a          |    3126 | /esdTree     |   614800 |   222 GB |   98 %
/proofteam/ganis/LHC09a4_run8010X_a       |    1387 | /esdTree     |   276200 |   114 GB |   99 %
/proofteam/ganis/LHC09a4_run8100X_a       |    1321 | /esdTree     |   263600 |    69 GB |   99 %
/proofteam/ganis/LHC09a4_run81011_a       |     451 | /esdTree     |    90200 |    24 GB |  100 %
/proofteam/ganis/LHC09a4_run8101X_a       |    3526 | /esdTree     |   705200 |   185 GB |  100 %
/proofteam/ganis/LHC09a4_run8158X_a       |    4005 | /esdTree     |   801000 |   211 GB |  100 %
/proofteam/ganis/LHC09a4_run8159X_a       |    2350 | /esdTree     |   470000 |   124 GB |  100 %
/proofteam/ganis/LHC09a5_run9000X_a       |    3360 | /esdTree     |   670400 |   186 GB |   99 %
/proofteam/ganis/LHC09a6_run9200X_a       |    2404 | /esdTree     |   477600 |   201 GB |   99 %
/proofteam/ganis/ds-test                  |       4 | /esdTree     |      400 |   157 MB |  100 %
/proofteam/ganis/ds-test-loc              |       4 | /esdTree     |      400 |   171 MB |  100 %
$~>

 

1.3. pq2-ls-files

pq2-ls-files datasetname [masterurl]
datasetname
Name of the dataset whose files have to be listed
masterurl
Entry point for the PROOF master to be used to query the dataset repository
Function
Display the file content of a dataset
Arguments
Default
None

Example:

$~> etc/proof/utils/pq2/pq2-ls-files ds-test
pq2-ls-files: dataset 'ds-test' has 4 files
pq2-ls-files:      #   File                                                                                    Size   #Objs Obj|Type|Entries, ...
pq2-ls-files:      1   root://lxfsrd0507.cern.ch//alien/alice/sim/PDC_08/LHC08b1/300000/1086/root_archive.zip#AliESDs.root       39 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files:      2   root://lxfsrd0507.cern.ch//alien/alice/sim/PDC_08/LHC08b1/300000/1088/root_archive.zip#AliESDs.root       40 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files:      3   root://lxfsrd0508.cern.ch//alien/alice/sim/PDC_08/LHC08b1/300000/1092/root_archive.zip#AliESDs.root       38 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files:      4   root://lxfsrd0509.cern.ch//alien/alice/sim/PDC_08/LHC08b1/300000/1094/root_archive.zip#AliESDs.root       38 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
$~>

 

1.4. pq2-ls-files-server

pq2-ls-files-server datasetname server [masterurl]
 
datasetname
Name of the dataset whose files have to be listed
server
Name of the server for which the information is wanted; can be in URL form
masterurl
Entry point for the PROOF master to be used to query the dataset repository
Function
Display information about the files of dataset available on the specified server
Arguments
Default
None

Example:

$~> etc/proof/utils/pq2/pq2-ls-files-server /COMMON/COMMON/tutorial_small lxfsrd0509
pq2-ls-files-server: dataset '/COMMON/COMMON/tutorial_small' has 15 files on server lxfsrd0509
pq2-ls-files-server:      #   File                                                                                    Size   #Objs Obj|Type|Entries, ...
pq2-ls-files-server:      1   /alien/alice/sim/PDC_08/LHC08b1/300000/022/root_archive.zip#AliESDs.root               36 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:      2   /alien/alice/sim/PDC_08/LHC08b1/300000/023/root_archive.zip#AliESDs.root               42 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:      3   /alien/alice/sim/PDC_08/LHC08b1/300000/024/root_archive.zip#AliESDs.root               41 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:      4   /alien/alice/sim/PDC_08/LHC08b1/300000/035/root_archive.zip#AliESDs.root               44 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:      5   /alien/alice/sim/PDC_08/LHC08b1/300000/056/root_archive.zip#AliESDs.root               45 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:      6   /alien/alice/sim/PDC_08/LHC08b1/300000/1001/root_archive.zip#AliESDs.root              35 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:      7   /alien/alice/sim/PDC_08/LHC08b1/300000/1011/root_archive.zip#AliESDs.root              46 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:      8   /alien/alice/sim/PDC_08/LHC08b1/300000/1020/root_archive.zip#AliESDs.root              47 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:      9   /alien/alice/sim/PDC_08/LHC08b1/300000/1027/root_archive.zip#AliESDs.root              41 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:     10   /alien/alice/sim/PDC_08/LHC08b1/300000/1081/root_archive.zip#AliESDs.root              47 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:     11   /alien/alice/sim/PDC_08/LHC08b1/300000/1087/root_archive.zip#AliESDs.root              45 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:     12   /alien/alice/sim/PDC_08/LHC08b1/300000/109/root_archive.zip#AliESDs.root               41 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:     13   /alien/alice/sim/PDC_08/LHC08b1/300000/1090/root_archive.zip#AliESDs.root              48 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:     14   /alien/alice/sim/PDC_08/LHC08b1/300000/1091/root_archive.zip#AliESDs.root              40 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server:     15   /alien/alice/sim/PDC_08/LHC08b1/300000/1094/root_archive.zip#AliESDs.root              38 MB      2  esdTree|TTree|100,HLTesdTree|TTree|100
$~>

 

1.5. pq2-info-server

pq2-info-server server [masterurl]
server
Name of the server for which the information is wanted
masterurl
Entry point for the PROOF master to be used to query the dataset repository
Function
  Display information about the datasets on a given server
Arguments
Default
None

Example:

$~> etc/proof/utils/pq2/pq2-info-server lxfsrd0509
+++
+++ Dumping: subset of 'LHC08c12_0.9TeV_0.5T' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 175 files, 8 GB, staged 100 %, tree: /esdTree, 26250 entries, 8.1 % of total
+++
+++
+++ Dumping: subset of 'LHC08c13_10TeV_0T' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 212 files, 19 GB, staged 100 %, tree: /esdTree, 31800 entries, 9.1 % of total
+++
+++
+++ Dumping: subset of 'LHC08c11_10TeV_0.5T' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 151 files, 10 GB, staged 100 %, tree: /esdTree, 22650 entries, 8.1 % of total
+++
+++
+++ Dumping: LHC09a4 10TeV Pythia produced with v4-16-Rev-08 (subset on server root://lxfsrd0509.cern.ch:1094):
+++ Summary: 221 files, 11 GB, staged 100 %, tree: /HLTesdTree, 44200 entries, 7.5 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_81011' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 75 files, 4 GB, staged 100 %, tree: /HLTesdTree, 15000 entries, 7.6 % of total
+++
+++
+++ Dumping: LHC09a4 10TeV Pythia produced with v4-16-Rev-07 (subset on server root://lxfsrd0509.cern.ch:1094):
+++ Summary: 231 files, 19 GB, staged 99 %, tree: /HLTesdTree, 45800 entries, 7.6 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_run8101X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 584 files, 31 GB, staged 100 %, tree: /esdTree, 116800 entries, 7.5 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_run8158X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 792 files, 41 GB, staged 99 %, tree: /HLTesdTree, 158000 entries, 8.1 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_run8159X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 459 files, 23 GB, staged 99 %, tree: /HLTesdTree, 91600 entries, 7.9 % of total
+++
+++
+++ Dumping: subset of 'LHC09a6_run9200X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 477 files, 40 GB, staged 98 %, tree: /HLTesdTree, 94200 entries, 8.2 % of total
+++
+++
+++ Dumping: subset of 'LHC08c14_0.9TeV_0T' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 136 files, 7 GB, staged 100 %, tree: /esdTree, 20400 entries, 8.2 % of total
+++
+++
+++ Dumping: subset of 'LHC08c15_10TeV_0.5T_Phojet' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 121 files, 7 GB, staged 100 %, tree: /esdTree, 18150 entries, 7.9 % of total
+++
+++
+++ Dumping: subset of 'tutorial_small' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 15 files, 644 MB, staged 100 %, tree: /esdTree, 1500 entries, 7.5 % of total
+++
+++
+++ Dumping: subset of 'LHC08c17_10TeV_0T_Phojet' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 90 files, 7 GB, staged 100 %, tree: /esdTree, 13500 entries, 6.8 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_10TeV' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 521 files, 37 GB, staged 98 %, tree: /HLTesdTree, 102600 entries, 7.6 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_10TeV_200k' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 75 files, 5 GB, staged 97 %, tree: /HLTesdTree, 14600 entries, 7.7 % of total
+++
+++
+++ Dumping: subset of 'LHC08c16_0.9TeV_0.5T_Phojet' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 103 files, 5 GB, staged 100 %, tree: /esdTree, 15450 entries, 7.2 % of total
+++
+++
+++ Dumping: subset of 'LHC08c18_0.9TeV_0T_Phojet' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 116 files, 7 GB, staged 100 %, tree: /esdTree, 17400 entries, 7.4 % of total
+++
+++
+++ Dumping: subset of 'LHC09a5_run9000X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 657 files, 36 GB, staged 99 %, tree: /HLTesdTree, 131000 entries, 8.1 % of total
+++
$~>

 

2. Modifiers scripts

These scripts allow to modify the content of the repository; they may fail if the remote settings do not allow modification.

2.1. pq2-put

pq2-put datasetfile [masterurl]
datasetfile
Path to the file with the list of files in the dataset or directory with the files containing the file lists of the datasets to be registered; in the first case wildcards '*' can be specified in the file name, i.e. " /fil*" is ok but " /*/file" is not. In all cases the name of the dataset is the name of the file used, e.g. " /mydset" will register a dataset called "mydset".
masterurl
Entry point for the PROOF master to be used to query the dataset repository
Function
Register one or more datasets
Arguments
Default
None

Example:

$~> etc/proof/utils/pq2/pq2-put ~/local/root/test/proof/examples/h1/h1-http.txt
pq2-put: 1 dataset(s) registered
$~> etc/proof/utils/pq2/pq2-ls
Dataset URI                               | # Files | Default tree | # Events |   Disk   | Staged
...
/proofteam/ganis/h1-http.txt              |       4 |        N/A              |   190 MB |    0 %
$~> 

 

2.2. pq2-verify

pq2-verify datasetname [masterurl]
datasetname
Name of the dataset to be removed; it accepts the '*' wild card: in such a case the full path - as shown by pq2-ls - should be given in quotes, e.g. "/default/ganis/h1-set5*
masterurl
Entry point for the PROOF master to be used to query the dataset repository
Function
Verify the content of one or more datasets
Arguments
Default
None

Example:

$~> etc/proof/utils/pq2/pq2-verify /proofteam/ganis/h1-http.txt
13:31:40 15348 Mst-0 | Info in <:scandataset>: processing 1.'new' file: http://root.cern.ch/files/h1/dstarmb.root
pq2-verify: 1 dataset(s) verified
$~> etc/proof/utils/pq2/pq2-ls
Dataset URI                               | # Files | Default tree | # Events |   Disk   | Staged
...
/proofteam/ganis/h1-http.txt              |       4 | /h42         |   283813 |   264 MB |  100 %
$~> etc/proof/utils/pq2/pq2-ls-files /proofteam/ganis/h1-http.txt
pq2-ls-files: dataset '/proofteam/ganis/h1-http.txt' has 4 files
pq2-ls-files:      #   File                                                                                    Size   #Objs Obj|Type|Entries, ...
pq2-ls-files:      1   http://root.cern.ch/files/h1/dstarmb.root                                              20 MB      1  h42|TTree|21920
pq2-ls-files:      2   http://root.cern.ch/files/h1/dstarp1a.root                                             68 MB      1  h42|TTree|73243
pq2-ls-files:      3   http://root.cern.ch/files/h1/dstarp1b.root                                             79 MB      1  h42|TTree|85597
pq2-ls-files:      4   http://root.cern.ch/files/h1/dstarp2.root                                              96 MB      1  h42|TTree|103053
$~> 

 

2.3. pq2-rm

pq2-rm datasetname [masterurl]
datasetname
Name of the dataset to be removed; it accepts the '*' wild card: in such a case the full path - as shown by pq2-ls - should be given in quotes, e.g. "/default/ganis/h1-set5*
masterurl
Entry point for the PROOF master to be used to query the dataset repository
Function
Remove one or more datasets
Arguments
Default
None

Example:

$~>  etc/proof/utils/pq2/pq2-rm /proofteam/ganis/h1-http.txt
pq2-rm: 1 dataset(s) removed
$~>