Log of /trunk/proof/proof/inc/TDataSetManagerFile.h
Parent Directory
Revision
48060 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Dec 15 18:53:16 2012 UTC (2 years, 1 month ago) by
ganis
File length: 5484 byte(s)
Diff to
previous 33357
From Dario: patch introducing an improved dataset management model where the PROOF (ROOT)
dataset manager is a light frontend to the experiment file catalogs; TDataSetManagerFile
is still used as local cache of the experiment information or to store the work-in-progress
status of the dataset manager daemon. This model is expected to solve the scalability issues
observed currently at AAFs.
The patch includes the new class TDataSetManagerAliEn with the first concrete implementation
of experiment catallog interface. The equivalent for ATLAS will follow.
Revision
33357 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 4 06:56:18 2010 UTC (4 years, 8 months ago) by
ganis
File length: 5353 byte(s)
Diff to
previous 33156
In proof/proof:
- Add more flexibility to TDataSetManager::ScanDataSet, the internal low-level method
used in dataset verification operations; there are now three option switches to control
- the set of files to process
- all files, the ones marked as staged, the ones marked as non staged
- which pre-action to do on these files:
- check the 'staged' status, open and/or touch the files marked as staged
- which action to do on the files
- no-action, full processing, locate only, stage request
Remarks:
- checking the stage status on 'all files' with 'no-action; can be used to fix the
'is-staged' bit on the dataset
- locate-only and 'stage-request' are done via the TFileStager interface and are fast
non-blocking actions
- Add public method TDataSetManager::ScanDataSet(const char *uri, const char *opts) to control
via a 'const char *' option the new functionality
- Separate out the extraction of the meta-information in the new method TDataSetManager::FillMetaData
which can be called recursively on the file sub-directories; this automatically adds support
for sub-directories in datasets, which was missing and now required by Atlas.
- Adapt TProofServ and TDataSetManagerFile to the new functionality in TDataSetManager .
In proof/pq2:
- Add support for the new options in dataset verification
- Fix issue with regexp parsing in dataset registration
In etc/proof/utils/pq2/pq2-verify:
- Add support for the new options in dataset verification
Revision
33156 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Apr 23 12:46:14 2010 UTC (4 years, 9 months ago) by
ganis
File length: 5116 byte(s)
Diff to
previous 33091
- Fix an issue with determining the path fo the lock file
- Fix an issue with matching wild carsd for dataset names in listing datasets
- Fix an issue with defaults in GetDataSets .
Revision
33091 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Apr 19 14:14:32 2010 UTC (4 years, 9 months ago) by
ganis
File length: 5097 byte(s)
Diff to
previous 33090
Patch implementing a cache for TDataSetManagerFile.
This mainly used to improve the speed of TDataSetManager::ShowDataSets, which is run very often
by users and may be very slow if the number of dataset is large. The cache is also used to cache
frequently received dataset objects.
The listing information (default result of TDataSetManager::ShowDataSets) is cached at {group,user}
level. A new file 'dataset.list', located in the top dataset directory, contains updated information
about the available datasets and dataset listings. This file is updated each time WriteDataSet or
RemoveDataSet are called .
The cache is automatically enabled if the dataset repository is remote (the remoteness of a directory
is determined usign the same technique used for files). It can also be enabled/disabled using the
appropriate switches.
The content of the cache can be displayed using TDataSetManager::ShowCache and cleared using
TDataSetManager::ClearCache. The listing information can be refreshed at any time (at least the one
corresponding to owned datasets) with an option to TDataSetManager::ShowDataSets .
The patch also fixes a few issues ith wildcard support in parsing the dataset URI.
Revision
33086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 18 18:15:56 2010 UTC (4 years, 9 months ago) by
ganis
File length: 3244 byte(s)
Diff to
previous 29133
- Some changes towards a parallel version of VerifyDataSet:
- Make the lower level of ScanDataSet a static in TDataSetManager and separate out
the part extracting the file meta-information into a dedicated method ScanFile
- In TDSetElement, rename the list of associated files to associated objects, so that
it can be more generally used (it was not yet used for associated files and it
was anyhow supposed to be a list of TObjString's).
- Move the parsing of the recently introduced 'DataSet.SrvMaps' env in TDataSetManager
(instead of TProof). This is more logical and it makes easier to apply the enable
the server mappings during staging requests via VerifyDataSet.
- Add a missing protection in TProof::AssertDataSet potentially causing crashes in the
case a dataset is not found.
Revision
29133 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 22 12:28:50 2009 UTC (5 years, 7 months ago) by
brun
File length: 3328 byte(s)
Copied from:
trunk/proof/proof/inc/TProofDataSetManagerFile.h revision 29121
Diff to
previous 28593
From Gerri:
Rename TProofDataSetManager and TProofDataSetManagerFile to TDataSetManager and TDataSetManagerFile.
These classes are managers of TFileCollection which do not have anything strictly dependent on PROOF.
So far they were only used internally by PROOF, but they are going to be exposed to more public for generic
file collection handling. It is the right time to give them a more proper name.
Revision
23075 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 9 09:10:25 2008 UTC (6 years, 9 months ago) by
ganis
Original Path:
trunk/proof/proof/inc/TProofDataSetManagerFile.h
File length: 3229 byte(s)
Copied from:
branches/dev/proof/proof/proof/inc/TProofDataSetManagerFile.h revision 23004
Diff to
previous 22999
From Jan-Fiete Grosse-Oetringhaus (ALICE) and me.
Patch introducing the improved dataset manager developed by ALICE.
The client side interface is basically unchanged, the biggest change is the renaming
of TProof::CreateDataSet to TProof::RegisterDataSet.
On the server side the relevant code has ben extracted from TProofServ and hidden
behind an inteface class TProofDataSetManager describing the functionality needed by
TProofServ, and an implementation class TProofDataSetManagerFile handling dataset
meta data saved into ROOT files. TProofDataSetManagerFile is instantiated via the
plugin manager and by default it handles the <sand_box>/datasets area.
This separation is needed to allow loading implementation based on different backends;
for example, ATLAS foresees to use a MySQL-based backend.
A TWiki page with information and examples in on preparation.
Revision
22999 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Mon Apr 7 08:31:22 2008 UTC (6 years, 9 months ago) by
ganis
Original Path:
branches/dev/proof/proof/proof/inc/TProofDataSetManagerFile.h
File length: 3229 byte(s)
Patch generalizing the dataset manager in preparation for merging with the trunk:
- Abstract out generic functionality in the TProofDataSetManager class
- Move file-system based implementation in TProofDataSetManagerFile
- Make the dataset manager loadable via plug-in
- Remove any dependence of XrdProofd on dataset issues (information is now
trasmitted via 'xpd.putrc' or the groupfile)
- In TXNetFile, add possibility to set the max number of retries on the fly
- Fix some issues with defaults
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
