Log of /trunk/proof/proof/src/TDataSetManagerFile.cxx
Parent Directory
Revision
48060 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Dec 15 18:53:16 2012 UTC (2 years, 1 month ago) by
ganis
File length: 75479 byte(s)
Diff to
previous 42798
From Dario: patch introducing an improved dataset management model where the PROOF (ROOT)
dataset manager is a light frontend to the experiment file catalogs; TDataSetManagerFile
is still used as local cache of the experiment information or to store the work-in-progress
status of the dataset manager daemon. This model is expected to solve the scalability issues
observed currently at AAFs.
The patch includes the new class TDataSetManagerAliEn with the first concrete implementation
of experiment catallog interface. The equivalent for ATLAS will follow.
Revision
41786 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Nov 4 17:06:37 2011 UTC (3 years, 2 months ago) by
ganis
File length: 72965 byte(s)
Diff to
previous 37934
In WriteDataSet, remove call to Sort() which modifies the order of the initial
collection, uselessly hardening any comparison and confusing any expectation.
Revision
37934 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 2 11:26:15 2011 UTC (3 years, 11 months ago) by
ganis
File length: 73019 byte(s)
Diff to
previous 36015
In TDataSetManagerFile::NotifyUpdate improve handling of the case when the global list file
does not exist yet (new dataset directory). Fixes error messages during editing dataset
operations.
Revision
33447 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon May 10 13:26:51 2010 UTC (4 years, 8 months ago) by
ganis
File length: 72457 byte(s)
Diff to
previous 33381
- Change the name of the recently introduced option bit controlling the check for the staged
status of a file from kCheckStageStatus to kNoStagedCheck (the default is to check and the
bit is used to skip the check).
- Fix an issue with checking this option
- Change the recently introduced option to ScanDataSet 'C' (or "checkstaged:") to 'I' (or
"nostagedcheck:")
- Propagate the change to 'pq2-verify'
Revision
33357 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 4 06:56:18 2010 UTC (4 years, 8 months ago) by
ganis
File length: 72578 byte(s)
Diff to
previous 33257
In proof/proof:
- Add more flexibility to TDataSetManager::ScanDataSet, the internal low-level method
used in dataset verification operations; there are now three option switches to control
- the set of files to process
- all files, the ones marked as staged, the ones marked as non staged
- which pre-action to do on these files:
- check the 'staged' status, open and/or touch the files marked as staged
- which action to do on the files
- no-action, full processing, locate only, stage request
Remarks:
- checking the stage status on 'all files' with 'no-action; can be used to fix the
'is-staged' bit on the dataset
- locate-only and 'stage-request' are done via the TFileStager interface and are fast
non-blocking actions
- Add public method TDataSetManager::ScanDataSet(const char *uri, const char *opts) to control
via a 'const char *' option the new functionality
- Separate out the extraction of the meta-information in the new method TDataSetManager::FillMetaData
which can be called recursively on the file sub-directories; this automatically adds support
for sub-directories in datasets, which was missing and now required by Atlas.
- Adapt TProofServ and TDataSetManagerFile to the new functionality in TDataSetManager .
In proof/pq2:
- Add support for the new options in dataset verification
- Fix issue with regexp parsing in dataset registration
In etc/proof/utils/pq2/pq2-verify:
- Add support for the new options in dataset verification
Revision
33257 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 28 06:48:47 2010 UTC (4 years, 8 months ago) by
ganis
File length: 71546 byte(s)
Diff to
previous 33209
- In TDataSetManagerFile::ShowCache() / ClearCache(), use the new format for the
cached files introduced to fix a problem on Windows (patch #33092)
- In TDataSetManager::ParseUri, allow again for '.' in dataset names (this was not
in the reg expression when adding support for wild cards (patch #33091)
Revision
33209 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Apr 26 10:27:40 2010 UTC (4 years, 8 months ago) by
ganis
File length: 71512 byte(s)
Diff to
previous 33156
- TDataSetManager, TDataSetManagerFile
- Add options to get a list of all dataset registered names from ScanDataSets
(option kList; the result is a TMap of {TObjString, TObjString} with the second
TObjString empty.
- Support wild cards in ScanDataSet .
- TProof, TProofServ
- Add interface to TDataSetManager::ShowCache/ClearCache (TProof::ShowDataSetCache,
TProof::ClearDataSetCache)
- If a non-default dataset directory was specified and could not be validated, do not
initialize the default dataset, as this could be misleading. An warning is issued
on the client shell.
Revision
33156 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Apr 23 12:46:14 2010 UTC (4 years, 9 months ago) by
ganis
File length: 69740 byte(s)
Diff to
previous 33126
- Fix an issue with determining the path fo the lock file
- Fix an issue with matching wild carsd for dataset names in listing datasets
- Fix an issue with defaults in GetDataSets .
Revision
33091 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Apr 19 14:14:32 2010 UTC (4 years, 9 months ago) by
ganis
File length: 68281 byte(s)
Diff to
previous 33090
Patch implementing a cache for TDataSetManagerFile.
This mainly used to improve the speed of TDataSetManager::ShowDataSets, which is run very often
by users and may be very slow if the number of dataset is large. The cache is also used to cache
frequently received dataset objects.
The listing information (default result of TDataSetManager::ShowDataSets) is cached at {group,user}
level. A new file 'dataset.list', located in the top dataset directory, contains updated information
about the available datasets and dataset listings. This file is updated each time WriteDataSet or
RemoveDataSet are called .
The cache is automatically enabled if the dataset repository is remote (the remoteness of a directory
is determined usign the same technique used for files). It can also be enabled/disabled using the
appropriate switches.
The content of the cache can be displayed using TDataSetManager::ShowCache and cleared using
TDataSetManager::ClearCache. The listing information can be refreshed at any time (at least the one
corresponding to owned datasets) with an option to TDataSetManager::ShowDataSets .
The patch also fixes a few issues ith wildcard support in parsing the dataset URI.
Revision
33086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 18 18:15:56 2010 UTC (4 years, 9 months ago) by
ganis
File length: 32345 byte(s)
Diff to
previous 32930
- Some changes towards a parallel version of VerifyDataSet:
- Make the lower level of ScanDataSet a static in TDataSetManager and separate out
the part extracting the file meta-information into a dedicated method ScanFile
- In TDSetElement, rename the list of associated files to associated objects, so that
it can be more generally used (it was not yet used for associated files and it
was anyhow supposed to be a list of TObjString's).
- Move the parsing of the recently introduced 'DataSet.SrvMaps' env in TDataSetManager
(instead of TProof). This is more logical and it makes easier to apply the enable
the server mappings during staging requests via VerifyDataSet.
- Add a missing protection in TProof::AssertDataSet potentially causing crashes in the
case a dataset is not found.
Revision
32930 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Apr 9 16:18:32 2010 UTC (4 years, 9 months ago) by
pcanal
File length: 41641 byte(s)
Diff to
previous 32217
Prefer the use of InheritsFrom(TClass*) over InheritsFrom(const char*) as long as
it does not increase the library dependencies.
Revision
32217 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Feb 4 17:39:16 2010 UTC (4 years, 11 months ago) by
ganis
File length: 41634 byte(s)
Diff to
previous 32031
Add the possibility to add files to a dataset. This is achieved with a new option
'U' (for update) to RegisterDataSet. Feature requested by ATLAS.
Revision
30421 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Sep 24 17:20:09 2009 UTC (5 years, 4 months ago) by
ganis
File length: 40816 byte(s)
Diff to
previous 30186
Some improvements in the dataset manager:
- Introduce the possibility to 'trust' the information provided by the user in the TFileCollection
object sent over upon registration (by default the status bits and the meta information are reset).
- Give the admin has the possibility to configure the dataset manager in such a way that the request
for trusting the information is ignored. A new bit TDataSetManager::kTrustInfo controls that and the
configuration option controlling it is called "Ti" (for 'Trust Info'; equivalent of "Av" for 'Allow Verification').
The current default is that the bit is set.
- Remove the unused bit TDataSetManager::kAllowStaging (staging is controlled externally, not by the
dataset manager)
- Fix a problem with browsing the existing datasets duplicating the same information in some cases.
Revision
29962 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Aug 28 09:21:16 2009 UTC (5 years, 4 months ago) by
ganis
File length: 39605 byte(s)
Diff to
previous 29171
- Correctly classify as TTree all TTree derived classes (e.g. TNtuple's)
- Fix a problem in saving the end-point URL for local files
- Improve realtime notification during 'verify'
Revision
29171 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 23 14:13:06 2009 UTC (5 years, 7 months ago) by
brun
File length: 39140 byte(s)
Diff to
previous 29133
From Gerri:
- Fix a few problems found running in valgrind:
- In XrdROOTMgr::Validate, add missing protection against inproper use of a
temporary file when in non debug mode
- Fix a problem with inproper deletion of XrdProofWorker objects when working
with a dynamically varying 'proof.conf'
- In TProof::HandleInputMessage, fix possible double delete after kPROOF_STOPPROCESS
- Fix a couple of issues on reconnection to a running session:
- make sure that the dialog buttons are in the correct state
- make sure that logs are redirected
- Fix an issue with the displaying speed of very large log files (bufferize instead of
doing line-by-line)
- In TProof::Draw, change the type of a notification message from 'error' to 'debug'
- Fix coding conventions in TDataSetManagerFile.cxx
Revision
29133 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 22 12:28:50 2009 UTC (5 years, 7 months ago) by
brun
File length: 39114 byte(s)
Copied from:
trunk/proof/proof/src/TProofDataSetManagerFile.cxx revision 29121
Diff to
previous 28735
From Gerri:
Rename TProofDataSetManager and TProofDataSetManagerFile to TDataSetManager and TDataSetManagerFile.
These classes are managers of TFileCollection which do not have anything strictly dependent on PROOF.
So far they were only used internally by PROOF, but they are going to be exposed to more public for generic
file collection handling. It is the right time to give them a more proper name.
Revision
28590 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 13 07:17:36 2009 UTC (5 years, 8 months ago) by
ganis
Original Path:
trunk/proof/proof/src/TProofDataSetManagerFile.cxx
File length: 38770 byte(s)
Diff to
previous 28576
- Fix a potential source of deadlocks due to the usage of vnsprintf
- Fixes for permission related failures during initialization of dataset managers:
- Add the possibility to define the dataset information sources via the
directive 'xpd.datasetsrc'. In this way the permissions should be
set correctly and the related problems disappear.
- In TProofDataSetManagerFile, switch in browse mode (instead of failing
construction) if the write permissions are not present.
Revision
23075 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 9 09:10:25 2008 UTC (6 years, 9 months ago) by
ganis
Original Path:
trunk/proof/proof/src/TProofDataSetManagerFile.cxx
File length: 37211 byte(s)
Copied from:
branches/dev/proof/proof/proof/src/TProofDataSetManagerFile.cxx revision 23004
Diff to
previous 22999
From Jan-Fiete Grosse-Oetringhaus (ALICE) and me.
Patch introducing the improved dataset manager developed by ALICE.
The client side interface is basically unchanged, the biggest change is the renaming
of TProof::CreateDataSet to TProof::RegisterDataSet.
On the server side the relevant code has ben extracted from TProofServ and hidden
behind an inteface class TProofDataSetManager describing the functionality needed by
TProofServ, and an implementation class TProofDataSetManagerFile handling dataset
meta data saved into ROOT files. TProofDataSetManagerFile is instantiated via the
plugin manager and by default it handles the <sand_box>/datasets area.
This separation is needed to allow loading implementation based on different backends;
for example, ATLAS foresees to use a MySQL-based backend.
A TWiki page with information and examples in on preparation.
Revision
22999 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Mon Apr 7 08:31:22 2008 UTC (6 years, 9 months ago) by
ganis
Original Path:
branches/dev/proof/proof/proof/src/TProofDataSetManagerFile.cxx
File length: 37211 byte(s)
Patch generalizing the dataset manager in preparation for merging with the trunk:
- Abstract out generic functionality in the TProofDataSetManager class
- Move file-system based implementation in TProofDataSetManagerFile
- Make the dataset manager loadable via plug-in
- Remove any dependence of XrdProofd on dataset issues (information is now
trasmitted via 'xpd.putrc' or the groupfile)
- In TXNetFile, add possibility to set the max number of retries on the fly
- Fix some issues with defaults
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
