Log of /trunk/proof/proof/src/TDataSetManager.cxx
Parent Directory
Revision
48992 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Mar 28 15:26:26 2013 UTC (21 months, 3 weeks ago) by
rdm
File length: 71969 byte(s)
Diff to
previous 48060
From Lifeng Sun:
The attached patchset fixes a bunch of typo in the source:
0001-succes-success.patch
0002-preceed-preced.patch
0003-informations-information.patch
0004-childs-children.patch
0005-avaliable-available.patch
0006-writeable-writable.patch
0007-comand-command.patch
0008-unkown-unknown.patch
0009-wierd-weird.patch
0010-wheter-whether.patch
0011-unecessary-unnecessary.patch
0012-splitted-split.patch
0013-registerd-registered.patch
0014-recieve-receive.patch
0015-processsing-processing.patch
0016-ouput-output.patch
0017-mutiple-multiple.patch
0018-lenght-length.patch
0019-interupted-interrupted.patch
0020-independant-independent.patch
0021-inconsistant-inconsistent.patch
0022-expresion-expression.patch
0023-explicitely-explicitly.patch
0024-enviroment-environment.patch
0025-deafult-default.patch
0026-continous-continuous.patch
0027-completly-completely.patch
0028-commited-committed.patch
0029-choosen-chosen.patch
0030-backgroud-background.patch
0031-auxilliary-auxiliary.patch
0032-authentification-authentication.patch
0033-appropiate-appropriate.patch
0034-an-other-another.patch
0035-environement-environment.patch
0036-targetting-targeting.patch
0037-suppported-supported.patch
0038-paramater-parameter.patch
Revision
48060 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Dec 15 18:53:16 2012 UTC (2 years, 1 month ago) by
ganis
File length: 71970 byte(s)
Diff to
previous 45740
From Dario: patch introducing an improved dataset management model where the PROOF (ROOT)
dataset manager is a light frontend to the experiment file catalogs; TDataSetManagerFile
is still used as local cache of the experiment information or to store the work-in-progress
status of the dataset manager daemon. This model is expected to solve the scalability issues
observed currently at AAFs.
The patch includes the new class TDataSetManagerAliEn with the first concrete implementation
of experiment catallog interface. The equivalent for ATLAS will follow.
Revision
45740 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 29 16:17:09 2012 UTC (2 years, 4 months ago) by
ganis
File length: 71104 byte(s)
Diff to
previous 44507
- Add support for data dirs on external servers, fully honouring URL options; this allows to
have usable paths registered in the final TFileCollection objects. One of the addressed use-case
is when local directories need to be accessed via the local xrootd server to preserve the namespace
enforced by xrootd; in such a case something like 'root://locahost//datadir?remote=1' needs to be
specified; the option 'remote=1' ensures that the final registered URL is still in 'root://' form
with 'localhost' replaced by the FQDN of the machine.
- Move the 'tests' argument to the stressProof function in the second position, to make it
more user friendly to run a specific test in the ROOT shell; CLI behevior is unchanged.
- In ProofSimple, add function to plot the histograms also in the case a dataset is created with the
worker files; the file are opened and the histograms merged on the fly.
- Fix potential segv at xproofd startup due to a double fclose in XrdProofdManager::ValidateLocalDataSetSrc
- Remove some redundant printing in TProof when handling output files.
- Make sure that when running TSelVerifyDataSet the save-to-file functionality is switched-off .
Revision
44507 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 4 12:30:41 2012 UTC (2 years, 7 months ago) by
axel
File length: 70206 byte(s)
Diff to
previous 44425
Remove
using namespace std;
from Riostream.h, which has huge consequences for all of ROOT.
Riostream.h is now a simple wrapper for fstream, iostream, iomanip for backward compatibility; Riosfwd.h simply wraps iosfwd.
Because of templates and their inline functions, Riostream.h needed to be included in headers, too (e.g. TParameter.h), which violated the assumption that Riostream.h is not exposing its using namespace std to headers.
ROOT now requires R__ANSISTREAM, R__SSTREAM, which does not change the set of supported compilers.
Without "using namespace std", several identifiers are now prefixed by std::; e.g. roofit/* source files now have a using namespace std to keep their coding style.
TFile::MakeProject() now generates "using namespace std" to convert the CINT-style class names into C++ ones.
Revision
44411 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu May 31 14:45:32 2012 UTC (2 years, 7 months ago) by
ganis
File length: 70178 byte(s)
Diff to
previous 44065
From S.Ryu: in ScanFile, do not store local-to-node end-point URLs as they are meaningless for general processing (should solve issue #94889)
Revision
43567 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Mar 30 08:59:16 2012 UTC (2 years, 9 months ago) by
ganis
File length: 69603 byte(s)
Diff to
previous 42882
In TDataSetManger::ScanFile :
- Open the file twice (the first time as 'raw') only if it is an archive (to get
the total size); this was tuned on the ALICE case, where with an xrootd back-end
the double opening was not critical; for other backends the penalty may be significant,
so it should be done only when strictly needed.
- Do not hardcode 'TIMEOUT=5' option by default when opening files; this appears to
create some problem on some back-ends, so it needs to be configurable.
In TProofServ
- In RegistersDataSets, remove the temporary dataset tags from the output list after
processing: they are not needed and generate warnings about duplicated names.
- In ProcessNext, add missing protection in the case of query abort.
Revision
42882 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Feb 6 11:10:21 2012 UTC (2 years, 11 months ago) by
ganis
File length: 68568 byte(s)
Diff to
previous 41739
From Sangsu Ryu and me.
Add support for paralell dataset verification. This is implemented via a dedicated
TSelector (TSelVerifyDataSet) which is run over the list of files in the dataset via
TPacketizerFile. The file order is preserved using the recently introduced index in
TFileInfo.
Parallel verification is the now the default. For test and comparison purposes it is
still possible to verify datasets serially by specifying 'S' in the option field of
VerifyDataSet and RegisterDataSet .
To avoid/reduce code duplication some of the static functions in TDataSetManager have
refactorized. The overall behavior should be unchanged.
Revision
33458 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 11 07:31:07 2010 UTC (4 years, 8 months ago) by
ganis
File length: 64900 byte(s)
Diff to
previous 33447
- Add support for recursive reading of group config files via the 'include sub-file'
directive. This allows to have a common part and, for example, customize differently
the quotas. See Savannah #67226.
Revision
33447 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon May 10 13:26:51 2010 UTC (4 years, 8 months ago) by
ganis
File length: 64127 byte(s)
Diff to
previous 33361
- Change the name of the recently introduced option bit controlling the check for the staged
status of a file from kCheckStageStatus to kNoStagedCheck (the default is to check and the
bit is used to skip the check).
- Fix an issue with checking this option
- Change the recently introduced option to ScanDataSet 'C' (or "checkstaged:") to 'I' (or
"nostagedcheck:")
- Propagate the change to 'pq2-verify'
Revision
33357 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 4 06:56:18 2010 UTC (4 years, 8 months ago) by
ganis
File length: 63806 byte(s)
Diff to
previous 33257
In proof/proof:
- Add more flexibility to TDataSetManager::ScanDataSet, the internal low-level method
used in dataset verification operations; there are now three option switches to control
- the set of files to process
- all files, the ones marked as staged, the ones marked as non staged
- which pre-action to do on these files:
- check the 'staged' status, open and/or touch the files marked as staged
- which action to do on the files
- no-action, full processing, locate only, stage request
Remarks:
- checking the stage status on 'all files' with 'no-action; can be used to fix the
'is-staged' bit on the dataset
- locate-only and 'stage-request' are done via the TFileStager interface and are fast
non-blocking actions
- Add public method TDataSetManager::ScanDataSet(const char *uri, const char *opts) to control
via a 'const char *' option the new functionality
- Separate out the extraction of the meta-information in the new method TDataSetManager::FillMetaData
which can be called recursively on the file sub-directories; this automatically adds support
for sub-directories in datasets, which was missing and now required by Atlas.
- Adapt TProofServ and TDataSetManagerFile to the new functionality in TDataSetManager .
In proof/pq2:
- Add support for the new options in dataset verification
- Fix issue with regexp parsing in dataset registration
In etc/proof/utils/pq2/pq2-verify:
- Add support for the new options in dataset verification
Revision
33257 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 28 06:48:47 2010 UTC (4 years, 8 months ago) by
ganis
File length: 53637 byte(s)
Diff to
previous 33099
- In TDataSetManagerFile::ShowCache() / ClearCache(), use the new format for the
cached files introduced to fix a problem on Windows (patch #33092)
- In TDataSetManager::ParseUri, allow again for '.' in dataset names (this was not
in the reg expression when adding support for wild cards (patch #33091)
Revision
33091 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Apr 19 14:14:32 2010 UTC (4 years, 9 months ago) by
ganis
File length: 53522 byte(s)
Diff to
previous 33087
Patch implementing a cache for TDataSetManagerFile.
This mainly used to improve the speed of TDataSetManager::ShowDataSets, which is run very often
by users and may be very slow if the number of dataset is large. The cache is also used to cache
frequently received dataset objects.
The listing information (default result of TDataSetManager::ShowDataSets) is cached at {group,user}
level. A new file 'dataset.list', located in the top dataset directory, contains updated information
about the available datasets and dataset listings. This file is updated each time WriteDataSet or
RemoveDataSet are called .
The cache is automatically enabled if the dataset repository is remote (the remoteness of a directory
is determined usign the same technique used for files). It can also be enabled/disabled using the
appropriate switches.
The content of the cache can be displayed using TDataSetManager::ShowCache and cleared using
TDataSetManager::ClearCache. The listing information can be refreshed at any time (at least the one
corresponding to owned datasets) with an option to TDataSetManager::ShowDataSets .
The patch also fixes a few issues ith wildcard support in parsing the dataset URI.
Revision
33086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 18 18:15:56 2010 UTC (4 years, 9 months ago) by
ganis
File length: 49490 byte(s)
Diff to
previous 30421
- Some changes towards a parallel version of VerifyDataSet:
- Make the lower level of ScanDataSet a static in TDataSetManager and separate out
the part extracting the file meta-information into a dedicated method ScanFile
- In TDSetElement, rename the list of associated files to associated objects, so that
it can be more generally used (it was not yet used for associated files and it
was anyhow supposed to be a list of TObjString's).
- Move the parsing of the recently introduced 'DataSet.SrvMaps' env in TDataSetManager
(instead of TProof). This is more logical and it makes easier to apply the enable
the server mappings during staging requests via VerifyDataSet.
- Add a missing protection in TProof::AssertDataSet potentially causing crashes in the
case a dataset is not found.
Revision
30421 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Sep 24 17:20:09 2009 UTC (5 years, 4 months ago) by
ganis
File length: 31253 byte(s)
Diff to
previous 29995
Some improvements in the dataset manager:
- Introduce the possibility to 'trust' the information provided by the user in the TFileCollection
object sent over upon registration (by default the status bits and the meta information are reset).
- Give the admin has the possibility to configure the dataset manager in such a way that the request
for trusting the information is ignored. A new bit TDataSetManager::kTrustInfo controls that and the
configuration option controlling it is called "Ti" (for 'Trust Info'; equivalent of "Av" for 'Allow Verification').
The current default is that the bit is set.
- Remove the unused bit TDataSetManager::kAllowStaging (staging is controlled externally, not by the
dataset manager)
- Fix a problem with browsing the existing datasets duplicating the same information in some cases.
Revision
29133 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 22 12:28:50 2009 UTC (5 years, 7 months ago) by
brun
File length: 31460 byte(s)
Copied from:
trunk/proof/proof/src/TProofDataSetManager.cxx revision 29121
Diff to
previous 28593
From Gerri:
Rename TProofDataSetManager and TProofDataSetManagerFile to TDataSetManager and TDataSetManagerFile.
These classes are managers of TFileCollection which do not have anything strictly dependent on PROOF.
So far they were only used internally by PROOF, but they are going to be exposed to more public for generic
file collection handling. It is the right time to give them a more proper name.
Revision
23075 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 9 09:10:25 2008 UTC (6 years, 9 months ago) by
ganis
Original Path:
trunk/proof/proof/src/TProofDataSetManager.cxx
File length: 25163 byte(s)
Copied from:
branches/dev/proof/proof/proof/src/TProofDataSetManager.cxx revision 23004
Diff to
previous 22999
From Jan-Fiete Grosse-Oetringhaus (ALICE) and me.
Patch introducing the improved dataset manager developed by ALICE.
The client side interface is basically unchanged, the biggest change is the renaming
of TProof::CreateDataSet to TProof::RegisterDataSet.
On the server side the relevant code has ben extracted from TProofServ and hidden
behind an inteface class TProofDataSetManager describing the functionality needed by
TProofServ, and an implementation class TProofDataSetManagerFile handling dataset
meta data saved into ROOT files. TProofDataSetManagerFile is instantiated via the
plugin manager and by default it handles the <sand_box>/datasets area.
This separation is needed to allow loading implementation based on different backends;
for example, ATLAS foresees to use a MySQL-based backend.
A TWiki page with information and examples in on preparation.
Revision
22999 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Apr 7 08:31:22 2008 UTC (6 years, 9 months ago) by
ganis
Original Path:
branches/dev/proof/proof/proof/src/TProofDataSetManager.cxx
File length: 25165 byte(s)
Diff to
previous 22720
Patch generalizing the dataset manager in preparation for merging with the trunk:
- Abstract out generic functionality in the TProofDataSetManager class
- Move file-system based implementation in TProofDataSetManagerFile
- Make the dataset manager loadable via plug-in
- Remove any dependence of XrdProofd on dataset issues (information is now
trasmitted via 'xpd.putrc' or the groupfile)
- In TXNetFile, add possibility to set the max number of retries on the fly
- Fix some issues with defaults
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
