NB: The tools described in this page are not maintained any-longer and the plan is to remove them starting with version 6-16/00. In the unlikely circumstance in which this removal turns out to be disruptive for your workflow, please contact the ROOT team via the Forum, the roottalk digets or the JIRA bug/task/request tracker.
Since ROOT version 5-24/00 a set of scripts to facilitate the interaction with a dataset manager are available under $ROOTSYS/etc/proof/utils/pq2. PQ2 stands for PROOF Quick Query, and echoes the ATLAS tool dq2.
In version 5-27/02 (and 5-26-00-proof) a new version of the scripts (and some new scripts) has been introduced. The scripts are now using a new ROOT executable - pq2 - available under $ROOTSYS/bin. The new (version of the) scripts is described in detail in man pages distributed with ROOT which will be shortly available in this page.
NB: The rest of this page is not yet updated. It gives a feeling of the functionality provided by the scripts but the synatx is not always exact. This will be corrected soon.
There are currently seven scripts, four for browsing the available information and three for modifying the repository. As mentioned in the TDataSetManager page, the repository is a directory, structured per-group and pr-user, with a ROOT file per dataset, containing a TFileCollection object.
The scripts access the repository via a PROOF master. This is not strictly needed (a TDataSetManager object can instantiatiated from everywhere on any accessible directory) and future developments may allow more flexible setups.
The scripts requires the URL of the PROOF master which is taken from the command line or from the environment variable PROOFURL.
1. Browsing scripts
These scripts allow to browse the content of the repository
1.2. pq2-ls
| pq2-ls [what] [masterurl] |
- what
- Selection string: use "*" (with quotes!) for all datasets; use "/mygroup/*" to show all datasets belonging to group 'mygroup'
- masterurl
- Entry point for the PROOF master to be used to query the dataset repository
- Function
- Display header information about the selected datasets
- Arguments
- Default
- Display the datasets owned by the client and the ones under COMMON.
Example:
$~> echo $PROOFURL
alicecaf.cern.ch:31093
$~> etc/proof/utils/pq2/pq2-ls
Dataset URI | # Files | Default tree | # Events | Disk | Staged
/COMMON/COMMON/LHC08c11_10TeV_0.5T | 1850 | /esdTree | 277500 | 129 GB | 100 %
/COMMON/COMMON/LHC08c12_0.9TeV_0.5T | 2126 | /esdTree | 318900 | 101 GB | 100 %
/COMMON/COMMON/LHC08c13_10TeV_0T | 2312 | /esdTree | 346500 | 213 GB | 99 %
/COMMON/COMMON/LHC08c14_0.9TeV_0T | 1646 | /esdTree | 246900 | 94 GB | 100 %
/COMMON/COMMON/LHC08c15_10TeV_0.5T_Phojet | 1526 | /esdTree | 222300 | 98 GB | 97 %
/COMMON/COMMON/LHC08c16_0.9TeV_0.5T_Phojet| 1415 | /esdTree | 201750 | 71 GB | 95 %
/COMMON/COMMON/LHC08c17_10TeV_0T_Phojet | 1314 | /esdTree | 180750 | 109 GB | 91 %
/COMMON/COMMON/LHC08c18_0.9TeV_0T_Phojet | 1554 | /esdTree | 222750 | 96 GB | 95 %
/COMMON/COMMON/LHC09a4_10TeV | 6831 | /HLTesdTree | 1342200 | 486 GB | 98 %
/COMMON/COMMON/LHC09a4_10TeV_200k | 989 | /HLTesdTree | 193800 | 70 GB | 97 %
/COMMON/COMMON/LHC09a4_run8010X | 3042 | /HLTesdTree | 602200 | 250 GB | 98 %
/COMMON/COMMON/LHC09a4_run8100X | 2947 | /HLTesdTree | 586800 | 154 GB | 99 %
/COMMON/COMMON/LHC09a4_run81011 | 989 | /HLTesdTree | 197600 | 52 GB | 99 %
/COMMON/COMMON/LHC09a4_run8101X | 7744 | /esdTree | 1547000 | 408 GB | 99 %
/COMMON/COMMON/LHC09a4_run8158X | 9683 | /HLTesdTree | 1932000 | 511 GB | 99 %
/COMMON/COMMON/LHC09a4_run8159X | 5745 | /HLTesdTree | 1146800 | 303 GB | 99 %
/COMMON/COMMON/LHC09a5_run9000X | 8125 | /HLTesdTree | 1611400 | 450 GB | 99 %
/COMMON/COMMON/LHC09a6_run9200X | 5791 | /HLTesdTree | 1141200 | 486 GB | 98 %
/COMMON/COMMON/tutorial_small | 200 | /esdTree | 20000 | 8 GB | 100 %
/proofteam/ganis/LHC09a4_10TeV-dev | 6831 | /esdTree | 1341000 | 486 GB | 98 %
/proofteam/ganis/LHC09a4_10TeV_a | 3126 | /esdTree | 614800 | 222 GB | 98 %
/proofteam/ganis/LHC09a4_run8010X_a | 1387 | /esdTree | 276200 | 114 GB | 99 %
/proofteam/ganis/LHC09a4_run8100X_a | 1321 | /esdTree | 263600 | 69 GB | 99 %
/proofteam/ganis/LHC09a4_run81011_a | 451 | /esdTree | 90200 | 24 GB | 100 %
/proofteam/ganis/LHC09a4_run8101X_a | 3526 | /esdTree | 705200 | 185 GB | 100 %
/proofteam/ganis/LHC09a4_run8158X_a | 4005 | /esdTree | 801000 | 211 GB | 100 %
/proofteam/ganis/LHC09a4_run8159X_a | 2350 | /esdTree | 470000 | 124 GB | 100 %
/proofteam/ganis/LHC09a5_run9000X_a | 3360 | /esdTree | 670400 | 186 GB | 99 %
/proofteam/ganis/LHC09a6_run9200X_a | 2404 | /esdTree | 477600 | 201 GB | 99 %
/proofteam/ganis/ds-test | 4 | /esdTree | 400 | 157 MB | 100 %
/proofteam/ganis/ds-test-loc | 4 | /esdTree | 400 | 171 MB | 100 %
$~>
1.3. pq2-ls-files
| pq2-ls-files datasetname [masterurl] |
- datasetname
- Name of the dataset whose files have to be listed
- masterurl
- Entry point for the PROOF master to be used to query the dataset repository
- Function
- Display the file content of a dataset
- Arguments
- Default
- None
Example:
$~> etc/proof/utils/pq2/pq2-ls-files ds-test
pq2-ls-files: dataset 'ds-test' has 4 files
pq2-ls-files: # File Size #Objs Obj|Type|Entries, ...
pq2-ls-files: 1 root://lxfsrd0507.cern.ch//alien/alice/sim/PDC_08/LHC08b1/300000/1086/root_archive.zip#AliESDs.root 39 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files: 2 root://lxfsrd0507.cern.ch//alien/alice/sim/PDC_08/LHC08b1/300000/1088/root_archive.zip#AliESDs.root 40 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files: 3 root://lxfsrd0508.cern.ch//alien/alice/sim/PDC_08/LHC08b1/300000/1092/root_archive.zip#AliESDs.root 38 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files: 4 root://lxfsrd0509.cern.ch//alien/alice/sim/PDC_08/LHC08b1/300000/1094/root_archive.zip#AliESDs.root 38 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
$~>
1.4. pq2-ls-files-server
| pq2-ls-files-server datasetname server [masterurl] |
- datasetname
- Name of the dataset whose files have to be listed
- server
- Name of the server for which the information is wanted; can be in URL form
- masterurl
- Entry point for the PROOF master to be used to query the dataset repository
- Function
- Display information about the files of dataset available on the specified server
- Arguments
- Default
- None
Example:
$~> etc/proof/utils/pq2/pq2-ls-files-server /COMMON/COMMON/tutorial_small lxfsrd0509
pq2-ls-files-server: dataset '/COMMON/COMMON/tutorial_small' has 15 files on server lxfsrd0509
pq2-ls-files-server: # File Size #Objs Obj|Type|Entries, ...
pq2-ls-files-server: 1 /alien/alice/sim/PDC_08/LHC08b1/300000/022/root_archive.zip#AliESDs.root 36 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 2 /alien/alice/sim/PDC_08/LHC08b1/300000/023/root_archive.zip#AliESDs.root 42 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 3 /alien/alice/sim/PDC_08/LHC08b1/300000/024/root_archive.zip#AliESDs.root 41 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 4 /alien/alice/sim/PDC_08/LHC08b1/300000/035/root_archive.zip#AliESDs.root 44 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 5 /alien/alice/sim/PDC_08/LHC08b1/300000/056/root_archive.zip#AliESDs.root 45 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 6 /alien/alice/sim/PDC_08/LHC08b1/300000/1001/root_archive.zip#AliESDs.root 35 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 7 /alien/alice/sim/PDC_08/LHC08b1/300000/1011/root_archive.zip#AliESDs.root 46 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 8 /alien/alice/sim/PDC_08/LHC08b1/300000/1020/root_archive.zip#AliESDs.root 47 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 9 /alien/alice/sim/PDC_08/LHC08b1/300000/1027/root_archive.zip#AliESDs.root 41 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 10 /alien/alice/sim/PDC_08/LHC08b1/300000/1081/root_archive.zip#AliESDs.root 47 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 11 /alien/alice/sim/PDC_08/LHC08b1/300000/1087/root_archive.zip#AliESDs.root 45 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 12 /alien/alice/sim/PDC_08/LHC08b1/300000/109/root_archive.zip#AliESDs.root 41 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 13 /alien/alice/sim/PDC_08/LHC08b1/300000/1090/root_archive.zip#AliESDs.root 48 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 14 /alien/alice/sim/PDC_08/LHC08b1/300000/1091/root_archive.zip#AliESDs.root 40 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
pq2-ls-files-server: 15 /alien/alice/sim/PDC_08/LHC08b1/300000/1094/root_archive.zip#AliESDs.root 38 MB 2 esdTree|TTree|100,HLTesdTree|TTree|100
$~>
1.5. pq2-info-server
| pq2-info-server server [masterurl] |
- server
- Name of the server for which the information is wanted
- masterurl
- Entry point for the PROOF master to be used to query the dataset repository
- Function
- Display information about the datasets on a given server
- Arguments
- Default
- None
Example:
$~> etc/proof/utils/pq2/pq2-info-server lxfsrd0509
+++
+++ Dumping: subset of 'LHC08c12_0.9TeV_0.5T' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 175 files, 8 GB, staged 100 %, tree: /esdTree, 26250 entries, 8.1 % of total
+++
+++
+++ Dumping: subset of 'LHC08c13_10TeV_0T' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 212 files, 19 GB, staged 100 %, tree: /esdTree, 31800 entries, 9.1 % of total
+++
+++
+++ Dumping: subset of 'LHC08c11_10TeV_0.5T' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 151 files, 10 GB, staged 100 %, tree: /esdTree, 22650 entries, 8.1 % of total
+++
+++
+++ Dumping: LHC09a4 10TeV Pythia produced with v4-16-Rev-08 (subset on server root://lxfsrd0509.cern.ch:1094):
+++ Summary: 221 files, 11 GB, staged 100 %, tree: /HLTesdTree, 44200 entries, 7.5 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_81011' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 75 files, 4 GB, staged 100 %, tree: /HLTesdTree, 15000 entries, 7.6 % of total
+++
+++
+++ Dumping: LHC09a4 10TeV Pythia produced with v4-16-Rev-07 (subset on server root://lxfsrd0509.cern.ch:1094):
+++ Summary: 231 files, 19 GB, staged 99 %, tree: /HLTesdTree, 45800 entries, 7.6 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_run8101X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 584 files, 31 GB, staged 100 %, tree: /esdTree, 116800 entries, 7.5 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_run8158X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 792 files, 41 GB, staged 99 %, tree: /HLTesdTree, 158000 entries, 8.1 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_run8159X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 459 files, 23 GB, staged 99 %, tree: /HLTesdTree, 91600 entries, 7.9 % of total
+++
+++
+++ Dumping: subset of 'LHC09a6_run9200X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 477 files, 40 GB, staged 98 %, tree: /HLTesdTree, 94200 entries, 8.2 % of total
+++
+++
+++ Dumping: subset of 'LHC08c14_0.9TeV_0T' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 136 files, 7 GB, staged 100 %, tree: /esdTree, 20400 entries, 8.2 % of total
+++
+++
+++ Dumping: subset of 'LHC08c15_10TeV_0.5T_Phojet' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 121 files, 7 GB, staged 100 %, tree: /esdTree, 18150 entries, 7.9 % of total
+++
+++
+++ Dumping: subset of 'tutorial_small' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 15 files, 644 MB, staged 100 %, tree: /esdTree, 1500 entries, 7.5 % of total
+++
+++
+++ Dumping: subset of 'LHC08c17_10TeV_0T_Phojet' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 90 files, 7 GB, staged 100 %, tree: /esdTree, 13500 entries, 6.8 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_10TeV' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 521 files, 37 GB, staged 98 %, tree: /HLTesdTree, 102600 entries, 7.6 % of total
+++
+++
+++ Dumping: subset of 'LHC09a4_10TeV_200k' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 75 files, 5 GB, staged 97 %, tree: /HLTesdTree, 14600 entries, 7.7 % of total
+++
+++
+++ Dumping: subset of 'LHC08c16_0.9TeV_0.5T_Phojet' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 103 files, 5 GB, staged 100 %, tree: /esdTree, 15450 entries, 7.2 % of total
+++
+++
+++ Dumping: subset of 'LHC08c18_0.9TeV_0T_Phojet' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 116 files, 7 GB, staged 100 %, tree: /esdTree, 17400 entries, 7.4 % of total
+++
+++
+++ Dumping: subset of 'LHC09a5_run9000X' on server root://lxfsrd0509.cern.ch:1094:
+++ Summary: 657 files, 36 GB, staged 99 %, tree: /HLTesdTree, 131000 entries, 8.1 % of total
+++
$~>
2. Modifiers scripts
These scripts allow to modify the content of the repository; they may fail if the remote settings do not allow modification.
2.1. pq2-put
| pq2-put datasetfile [masterurl] |
- datasetfile
- Path to the file with the list of files in the dataset or directory with the files containing the file lists of the datasets to be registered; in the first case wildcards '*' can be specified in the file name, i.e. "
/fil*" is ok but " /*/file" is not. In all cases the name of the dataset is the name of the file used, e.g. " /mydset" will register a dataset called "mydset". - masterurl
- Entry point for the PROOF master to be used to query the dataset repository
- Function
- Register one or more datasets
- Arguments
- Default
- None
Example:
$~> etc/proof/utils/pq2/pq2-put ~/local/root/test/proof/examples/h1/h1-http.txt
pq2-put: 1 dataset(s) registered
$~> etc/proof/utils/pq2/pq2-ls
Dataset URI | # Files | Default tree | # Events | Disk | Staged
...
/proofteam/ganis/h1-http.txt | 4 | N/A | 190 MB | 0 %
$~>
2.2. pq2-verify
| pq2-verify datasetname [masterurl] |
- datasetname
- Name of the dataset to be removed; it accepts the '*' wild card: in such a case the full path - as shown by pq2-ls - should be given in quotes, e.g. "/default/ganis/h1-set5*
- masterurl
- Entry point for the PROOF master to be used to query the dataset repository
- Function
- Verify the content of one or more datasets
- Arguments
- Default
- None
Example:
$~> etc/proof/utils/pq2/pq2-verify /proofteam/ganis/h1-http.txt
13:31:40 15348 Mst-0 | Info in <:scandataset>: processing 1.'new' file: http://root.cern.ch/files/h1/dstarmb.root
pq2-verify: 1 dataset(s) verified
$~> etc/proof/utils/pq2/pq2-ls
Dataset URI | # Files | Default tree | # Events | Disk | Staged
...
/proofteam/ganis/h1-http.txt | 4 | /h42 | 283813 | 264 MB | 100 %
$~> etc/proof/utils/pq2/pq2-ls-files /proofteam/ganis/h1-http.txt
pq2-ls-files: dataset '/proofteam/ganis/h1-http.txt' has 4 files
pq2-ls-files: # File Size #Objs Obj|Type|Entries, ...
pq2-ls-files: 1 http://root.cern.ch/files/h1/dstarmb.root 20 MB 1 h42|TTree|21920
pq2-ls-files: 2 http://root.cern.ch/files/h1/dstarp1a.root 68 MB 1 h42|TTree|73243
pq2-ls-files: 3 http://root.cern.ch/files/h1/dstarp1b.root 79 MB 1 h42|TTree|85597
pq2-ls-files: 4 http://root.cern.ch/files/h1/dstarp2.root 96 MB 1 h42|TTree|103053
$~>
2.3. pq2-rm
| pq2-rm datasetname [masterurl] |
- datasetname
- Name of the dataset to be removed; it accepts the '*' wild card: in such a case the full path - as shown by pq2-ls - should be given in quotes, e.g. "/default/ganis/h1-set5*
- masterurl
- Entry point for the PROOF master to be used to query the dataset repository
- Function
- Remove one or more datasets
- Arguments
- Default
- None
Example:
$~> etc/proof/utils/pq2/pq2-rm /proofteam/ganis/h1-http.txt
pq2-rm: 1 dataset(s) removed
$~>