Log of /trunk/proof/proof/inc/TDSet.h
Parent Directory
Revision
44010 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 29 23:07:48 2012 UTC (2 years, 8 months ago) by
ganis
File length: 12175 byte(s)
Diff to
previous 43044
Several optimizations in the low level PROOF event loop (TProofPlayer::Process):
- Per-event calls to TEventIter::GetNextEvent are replaced by per-packet calls
to a new function TEventIter::GetNextPacket and a for loop over the returned
range;
- Progress information is filled in on per-packet pace just before asking for
a new packet, instead of being filled for each event;
- In TProofProgressStatus, the relevant setters in were inlined, and the (expensive)
calls to SetLastUpdate() reduced to the minimum required;
- Status of objects is checked using bits instead of functions calls;
- Check whether to dump memory info is done before calling CheckMemUsage.
These optimizations allow to reduce dramatically the overhead introduced by the
operations PROOF needs to perform during the event loop. A measurement of the
overhead can be obtained from a very light computational task, for example,
generating one random number and filling one histogram; executing this task
within a PROOF-Lite session with 1 worker now takes only 1.8 times the time
required by a straight loop in the parent ROOT session; the same number before
was about 13.
The patch also introduces a couple of small changes enabling the use of
TProofPlayerLocal to process selectors in the local session using the PROOF
iteration mechanism; this is very useful for optimizations studies.
Finally, a couple of missing protections were added in TProofPlayer.
Revision
43044 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 17 14:34:52 2012 UTC (2 years, 11 months ago) by
ganis
File length: 12100 byte(s)
Diff to
previous 34416
From Taesoo Kim and me.
Add support for selector-by-object processing in PROOF.
The selector object, created and configured locally by the user, is added to the input
list and recuperated from there on the worker machines for processing. Any input list
setting in the selector itself is not streamed but temporarly moved to then standard
input list, so that user can use the selector input list as container of processing
information if they find convenient to do so.
Process(...) methods with the file name argument replaced by 'TSelector *' have
introduced where relevant (TProof, TProofPlayer and their derivatives, TDSet).
Two tests for the new functionality have been added to test/stressProof.cxx (tests 25
and 26).
Revision
34416 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jul 14 15:39:20 2010 UTC (4 years, 6 months ago) by
ganis
File length: 11829 byte(s)
Diff to
previous 33086
Fix problem with packet re-assignment in case of a worker death. Some packets
were processed twice or more times.
A new method MergeElement(TDSetElement *elem) has been added to TDSetElement in
order to simplify merging of contiguous or overlapping packets and avoid artificial
fragmentation of the re-assigned parts.
Revision
33086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 18 18:15:56 2010 UTC (4 years, 9 months ago) by
ganis
File length: 11774 byte(s)
Diff to
previous 33045
- Some changes towards a parallel version of VerifyDataSet:
- Make the lower level of ScanDataSet a static in TDataSetManager and separate out
the part extracting the file meta-information into a dedicated method ScanFile
- In TDSetElement, rename the list of associated files to associated objects, so that
it can be more generally used (it was not yet used for associated files and it
was anyhow supposed to be a list of TObjString's).
- Move the parsing of the recently introduced 'DataSet.SrvMaps' env in TDataSetManager
(instead of TProof). This is more logical and it makes easier to apply the enable
the server mappings during staging requests via VerifyDataSet.
- Add a missing protection in TProof::AssertDataSet potentially causing crashes in the
case a dataset is not found.
Revision
33045 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 15 14:57:32 2010 UTC (4 years, 9 months ago) by
ganis
File length: 11632 byte(s)
Diff to
previous 32204
Add the possibility to remap the server for the files in a dataset. This allows,
for example, to reuse the dataset information for the same files stored in a different
cluster.
The mapping can be defined globally by the cluster administrator using the ROOT env
variable
DataSet.SrvMaps: srv1|map1 srv2|map2 ...
where srv1 are the {protocol,host,port} of the server to be matched, and map1 are the
{protocol,host,port} to be used for those files mapping srv1, and so on.
The matching server string can contain the wildcard '*' in the host field or it can
be empty, in which case the replacement is done for all the dataset files.
If the '|' character is missing the whole string is assigned to the map part; i.e.
'map1' alone is equivalent to '*|map1' or '|map1' .
For example, the following can be used to relocate the files via 'redir.dom.ain'
DataSet.SrvMaps: root://dserv-*.dom.ain/|root://redir.dom.ain/
The user can change the default mapping(s) on the fly with the parameter PROOF_DataSetSrvMaps:
the above is equivalent to setting
p->SetParameter("PROOF_DataSetSrvMaps", "root://dserv-*.dom.ain/|root://redir.dom.ain/")
before starting a query.
Revision
32204 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 3 19:17:40 2010 UTC (4 years, 11 months ago) by
ganis
File length: 11435 byte(s)
Diff to
previous 23075
Add support for processing many datasets in one go in TProof::Process(const char *dataset, ...).
Two options are provided:
- 'grand dataset': the datasets are added up and considered as a single dataset;
syntax: "dataset1|dataset2|..."
- 'keep separated': the datasets are processed one after the other; the user is
notified in the selector of the change of dataset so she/he
has the opportunity to separate the results. A new packetizer,
TPacketizerMulti, has been developed for this case: it basically
contains a list of standard packetizers (one for each dataset) and
loops over them.
Syntax: "dataset1,dataset2,..." or dataset1 dataset2 ..."
In both cases, entry-list can be applied using the syntax "dataset<<entrylist", e.g.
"dataset1<<el1|dataset2<<el2|".
See http://root.cern.ch/drupal/content/working-data-sets#currentelem for more details.
A test for the new functionality has been added to test/stressProof.cxx .
Revision
23075 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 9 09:10:25 2008 UTC (6 years, 9 months ago) by
ganis
File length: 10749 byte(s)
Diff to
previous 22635
From Jan-Fiete Grosse-Oetringhaus (ALICE) and me.
Patch introducing the improved dataset manager developed by ALICE.
The client side interface is basically unchanged, the biggest change is the renaming
of TProof::CreateDataSet to TProof::RegisterDataSet.
On the server side the relevant code has ben extracted from TProofServ and hidden
behind an inteface class TProofDataSetManager describing the functionality needed by
TProofServ, and an implementation class TProofDataSetManagerFile handling dataset
meta data saved into ROOT files. TProofDataSetManagerFile is instantiated via the
plugin manager and by default it handles the <sand_box>/datasets area.
This separation is needed to allow loading implementation based on different backends;
for example, ATLAS foresees to use a MySQL-based backend.
A TWiki page with information and examples in on preparation.
Revision
20307 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 11 10:58:50 2007 UTC (7 years, 3 months ago) by
ganis
Original Path:
trunk/proof/inc/TDSet.h
File length: 10219 byte(s)
Diff to
previous 19826
Import branches/dev/proof r20306
Summary (see branch logs for more details):
- Set of changes related to CPU quota control including
- broadcast of centrally determined priorities
- mechanism to renice processes in a quantitative way
- New packetizer for non-tree based analysis and related API in TProof (from L. Tran-Thanh)
- Support for merging output objects saved in files on the workers (from L. Tran-Thanh and me)
- Improve version binary compatibility checks using also the SVN revision number
(when available) to define the running version.
- Extend the version binary compatibility checks also to the cached selector binaries.
- Extend TDSet::Lookup so that in case of missing files, it can remove them from the
dataset (option removeMissing must be set).
- Move the data set lookup to the TProofPlayerRemote::Process.
- Handle properly the case of incomple datasets: if the file is not found in the lookup
don't try to validate it; add it, instead, to a 'missingFiles' list returned in the
output list (fixing bug #28800 in Savannah).
Revision
18859 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 23 09:08:59 2007 UTC (7 years, 8 months ago) by
rdm
Original Path:
trunk/proof/inc/TDSet.h
File length: 10044 byte(s)
Diff to
previous 18648
From Gerri:
- Add definition of bit 16 as kWriteV3 to flag the case when the streamers
should write v3 TDSet / TDSetElements
- Add dedicated Streamer implementation to read/write v3 TDSet / TDSetElements
Revision
18648 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu May 3 15:27:40 2007 UTC (7 years, 8 months ago) by
rdm
Original Path:
trunk/proof/inc/TDSet.h
File length: 9744 byte(s)
Diff to
previous 18527
From Gerri:
This is the last patch to synchronize with the CAF:
TDSet.h, .cxx
- Use TCollection instead of TList
- Support thre object types in TDSet::Add(TCollection *): TFileInfo, TUrl
and TObjString .
- Conserve the looked-up status of TChainElements when instantiating
from a TChain.
TChainElement.h, .cxx
- Use BIT(15) to flag the looked-up status
TChain.h, .cxx
- Use TCollection instead of TList
- Support thre object types in TDSet::Add(TCollection *): TFileInfo, TUrl
and TObjString .
- Save / Test the looked-up status of elements to avoid multiple lookup;
an option is available in TChain::Lookup(Bool_t force = kFALSE) to
force a new lookup.
Revision
18527 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 19 09:33:40 2007 UTC (7 years, 9 months ago) by
rdm
Original Path:
trunk/proof/inc/TDSet.h
File length: 9719 byte(s)
Diff to
previous 18511
From Gerri:
get rid of not really needed usage of queue, map, pair and list.
- TDSet.h, .cxx
Use TList, TPair instead of std::list<std::pair<... >>
- TPacketizer.cxx, TAdaptivePacketizer.cxx, TEventIter.cxx,
TPacketizerDev.cxx, TPacketizerProgressive.cxx
Adapt to the previous change in TDSet
Revision
16969 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 28 12:10:52 2006 UTC (8 years, 1 month ago) by
rdm
Original Path:
trunk/proof/inc/TDSet.h
File length: 9992 byte(s)
Diff to
previous 16955
From Gerri:
This patch eliminates TVirtualProof, TVirtualProofMgr and TVirtualProofDesc.
The first is integrated in TProof, the second in TProofMgr and the third
becomes TProofDesc defined in TProofMgr.h.
Revision
16955 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Mon Nov 27 14:14:24 2006 UTC (8 years, 1 month ago) by
rdm
Original Path:
trunk/proof/inc/TDSet.h
File length: 10007 byte(s)
From Gerri:
This is the patch to move TDSet and TChainProof in 'proof' and remove
any dependence on PROOF in 'tree'. I have renamed TChainProof as TProofChain,
for consistency. TProofChain is now deriving from TChain and overrides only
the required methods. It is loaded via PluginManager by TChain in SetProof.
TQueryResult stores now all the input-related objects in fInputList. There
is a new method
TObject *TQueryResult::GetInputObject(const char *classname)
to retrieve the first instance of a class from the input list, so for the
TDSet:
TQueryResult *qr;
TDSet *dset = (TDSet *) qr->GetInputObject("TDSet")
The changes in TSessionViewer are needed to adapt to this new way of
accessing TDSets.
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
