Log of /trunk/proof/proofplayer/src/TPacketizer.cxx
Parent Directory
Revision
44507 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 4 12:30:41 2012 UTC (2 years, 7 months ago) by
axel
File length: 41194 byte(s)
Diff to
previous 44065
Remove
using namespace std;
from Riostream.h, which has huge consequences for all of ROOT.
Riostream.h is now a simple wrapper for fstream, iostream, iomanip for backward compatibility; Riosfwd.h simply wraps iosfwd.
Because of templates and their inline functions, Riostream.h needed to be included in headers, too (e.g. TParameter.h), which violated the assumption that Riostream.h is not exposing its using namespace std to headers.
ROOT now requires R__ANSISTREAM, R__SSTREAM, which does not change the set of supported compilers.
Without "using namespace std", several identifiers are now prefixed by std::; e.g. roofit/* source files now have a using namespace std to keep their coding style.
TFile::MakeProject() now generates "using namespace std" to convert the CINT-style class names into C++ ones.
Revision
44010 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 29 23:07:48 2012 UTC (2 years, 8 months ago) by
ganis
File length: 41150 byte(s)
Diff to
previous 41785
Several optimizations in the low level PROOF event loop (TProofPlayer::Process):
- Per-event calls to TEventIter::GetNextEvent are replaced by per-packet calls
to a new function TEventIter::GetNextPacket and a for loop over the returned
range;
- Progress information is filled in on per-packet pace just before asking for
a new packet, instead of being filled for each event;
- In TProofProgressStatus, the relevant setters in were inlined, and the (expensive)
calls to SetLastUpdate() reduced to the minimum required;
- Status of objects is checked using bits instead of functions calls;
- Check whether to dump memory info is done before calling CheckMemUsage.
These optimizations allow to reduce dramatically the overhead introduced by the
operations PROOF needs to perform during the event loop. A measurement of the
overhead can be obtained from a very light computational task, for example,
generating one random number and filling one histogram; executing this task
within a PROOF-Lite session with 1 worker now takes only 1.8 times the time
required by a straight loop in the parent ROOT session; the same number before
was about 13.
The patch also introduces a couple of small changes enabling the use of
TProofPlayerLocal to process selectors in the local session using the PROOF
iteration mechanism; this is very useful for optimizations studies.
Finally, a couple of missing protections were added in TProofPlayer.
Revision
41785 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Nov 4 17:01:33 2011 UTC (3 years, 2 months ago) by
ganis
File length: 40953 byte(s)
Diff to
previous 41619
Fix issue in TPacketizerAdaptive and TPacketizer preventi proper selection of
event sub-ranges, i.e. when processing num != -1 entries starting at first > 0 .
Revision
39835 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 20 15:37:30 2011 UTC (3 years, 7 months ago) by
ganis
File length: 40379 byte(s)
Diff to
previous 38616
- In TPacketizer and TPacketizerAdaptive, fix an issue with counter updating when a number
of events to be processed is specified (it was working up to a certain number of files and
then it was getting screwed up).
- In TPacketizerAdaptive, fix an issue with the option 'ForceLocal' on 'file:///' URLs.
- In TProofPlayer, optimize two conditional scopes.
- In TProofPlayerLite, make sure that the Progress timer is stopped when issuing STOP.
Revision
38616 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Mar 24 17:50:46 2011 UTC (3 years, 10 months ago) by
ganis
File length: 39436 byte(s)
Diff to
previous 37980
Fix a bug checking the first event, probably introduced by 'fix' #37980 .
Should fix the issue reported in Savannah #78921 .
Revision
33781 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 8 14:13:39 2010 UTC (4 years, 7 months ago) by
ganis
File length: 39075 byte(s)
Diff to
previous 32204
- Optimize the validation step in the case not all the entries are required.
The validation step is stopped as soon as the requested number of events is reached.
If the parameter "PROOF_ValidateByFile" is set to 1, the number of files is exactly what
needed; otherwise the number of files may exceed the number of fles needed by #workers-1
(this is the default because additional, serial, checks are needed to ensure that only
the files really required are open).
This feature was requested in the context of ALICE reconstruction.
This new feature is used as an example in the "eventproc" tutorial in runProof.C .
- The patch also fixes a subtle bug affecting the (possibly rare) case when not all entries
are required and # entries does not correspond to an complete subset of files (e.g.
# entries = 1001000 with files of 100000 entries each). The effect was uncomplete
processing (skipped events, magenta bar) or a session freeze.
Revision
32204 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 3 19:17:40 2010 UTC (4 years, 11 months ago) by
ganis
File length: 37618 byte(s)
Diff to
previous 31840
Add support for processing many datasets in one go in TProof::Process(const char *dataset, ...).
Two options are provided:
- 'grand dataset': the datasets are added up and considered as a single dataset;
syntax: "dataset1|dataset2|..."
- 'keep separated': the datasets are processed one after the other; the user is
notified in the selector of the change of dataset so she/he
has the opportunity to separate the results. A new packetizer,
TPacketizerMulti, has been developed for this case: it basically
contains a list of standard packetizers (one for each dataset) and
loops over them.
Syntax: "dataset1,dataset2,..." or dataset1 dataset2 ..."
In both cases, entry-list can be applied using the syntax "dataset<<entrylist", e.g.
"dataset1<<el1|dataset2<<el2|".
See http://root.cern.ch/drupal/content/working-data-sets#currentelem for more details.
A test for the new functionality has been added to test/stressProof.cxx .
Revision
30953 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 3 08:42:49 2009 UTC (5 years, 2 months ago) by
ganis
File length: 37409 byte(s)
Diff to
previous 30872
If enabled, send monitoring information from the master at each GetNextPacket
(at each call of TPerfStat::PacketEvent) to allow extrnal real-time progress
monitoring.
Revision
30870 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 26 15:16:00 2009 UTC (5 years, 2 months ago) by
ganis
File length: 37352 byte(s)
Diff to
previous 30859
Use the improved estimation of the current rate also for TPacketizer and TPacketizerUnit.
Fixes also a problem with the progress bar with these packetizers appeared after the last
check-in.
Revision
30859 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Oct 24 14:53:07 2009 UTC (5 years, 3 months ago) by
ganis
File length: 36169 byte(s)
Diff to
previous 30574
Patch for improved performance monitoring. The 'Rate Plot' button in the progress
dialog is renamed 'Performance plot' and shows up to 4 plots with the event/sec,
the average read chunck size, the number of active workers and the number of active
PROOF sessions on the cluster, all as a function of processing time.
The read chunck size plot allows to monitor the usage of the cache.
The istantaneous processing rate (event/sec) is now better estimate: a few issues
with the normalizing times have ben solved, removing the artificial structures that
were observed.
The possibility to set a max packet time length is introduced (default 30 s); this
can be changes with the parameter PROOF_MaxPacketTime.
The size of the cache is also taken into account to optimize the use of the cache.
The parameter PROOF_UseParallelUnzip has been introduced to toggle the use of the
parallel unzip (default off for now).
A page describing the new performance plots is under preparation at
http://root.cern.ch/drupal/content/progress-dialog
Revision
30574 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Oct 6 10:18:06 2009 UTC (5 years, 3 months ago) by
ganis
File length: 35685 byte(s)
Diff to
previous 29490
By default do not set any limit on the number of workers accessing a given file server;
following to recent reports, this seems a better default with current hardware.
The value can be changed/set via the PROOF_MaxSlavesPerNode parameter, as before.
Revision
26875 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 12 14:21:34 2008 UTC (6 years, 1 month ago) by
ganis
File length: 35337 byte(s)
Diff to
previous 26383
Patch to:
- Add the possibility to set upper limits on the virtual memory (changes in TProofPlayer,
TProofServ, TXProofServ and pmain (separate patch)).
If enebled, the session gets firts a warning when it reaches 80% of the limit, and then
processing is stopped whenit exceeds 95% of the limit, sending back the results.
Also, the memory footprint is notified when the session is terminated.
- Make sure the the active valuse in XrdProofWorker are always correct; this was not the
case for dynamic startup as the notification at the end of the query was not done.
This information is crucial or the scheduler.
The way the information is stored in XrdProofdProofServ had to be modified
and a new internal message type (kReleaseWorker) added.
(Changes in several proofd classes, TProof, TXsocket and TXProofServ)
- Fix problem with setting a static upper limit of the sessions that can be started,
and enable this functionality for the dynamic mode (changes in XrdProofSched)
- Remove a deleted worker from all the lists in TProof::MarkBad to avoid later attempts of use.
- Better control the use of the internal pipe for socket readiness notification in TXSocket.
Revision
26383 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Nov 22 17:58:39 2008 UTC (6 years, 2 months ago) by
ganis
File length: 35360 byte(s)
Diff to
previous 26381
- Fix the number of entries in the Draw histos for PROOF-Lite
- Set the "number of workers per node" in PROOF-Lite to the number of workers in the session
- Remove some debug messages left by mistake
Revision
26381 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Nov 22 17:15:24 2008 UTC (6 years, 2 months ago) by
ganis
File length: 35326 byte(s)
Diff to
previous 25896
- Fixes for the asynchronous mode:
- Fully localize the general handling of the input messages into two new methods
TProof::HandleInputMessage(TMessage *) and TProofServ::HandleSocketInput(TMessage *);
these methods are callable form any place that needs to intercept some messages and
do something specific on them (examples are: TProofServ::GetNextPacket, TXProofServ::Get,
TPacketizerAdaptive::ValidateFiles, ...); this allows to remove several duplications
and to make sure that no message is lost or wrongly dispatched.
- Simplify the recursive infrastructure for TProofServ::HandleSocketInput; in particular,
TProofServ::HandleSocketInputDuringProcess is removed as it is a special case of the
HandleSocketInput, saving duplications.
- Always use a kPROOF_CHECKFILE message in replies to check file operations; failures were
signaled with kPROOF_FATAL which may have some undesired side-effects, depending on the
timing
- Add support for one level of recursivity in TProof::Collect .
- Fix a problem with TProof::Finalize when called with default arguments (on the last query)
- In TProof::SendFile, send to unique workers only in the "cache" option is specified
- Remove the call to Finalize for DrawSelect queries, as it is done via the feedback
mechanisms
- Additions/fixes in test/stressProof.cxx:
- New test for the asynchronous mode;
- Fine-tuning of the progress display in batch mode;
Revision
25896 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 20 17:01:31 2008 UTC (6 years, 3 months ago) by
ganis
File length: 36408 byte(s)
Diff to
previous 25859
From Jan:
- Using consistently the recently introduced TProofProgressStatus in the kPROOF_GETPACKET
messages sent to TPacketizerUnit, TPacketizerAdaptive and TPacketizer; the message contains
the status of progress since the start of processing on a given node.
- Introduce TVirtualPacketizer::TVirtualSlaveStat as a base class of all the TSlaveStat
packetizer specific auxilliary classes.
- Full implemenation of GetProgressStatus() and AddProcessed(TProofProgressStatus *st) members
for TPacketizerUnit::TSlaveStat.
This patch should fix some consistency problems experienced after the patch introducing
TProofProgressStatus .
Revision
25859 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Oct 17 16:38:52 2008 UTC (6 years, 3 months ago) by
ganis
File length: 35589 byte(s)
Diff to
previous 25827
- Make all packetizers understand the GETPACKET messages containing the new TProofProgressStatus
structure
- Add a few missing protections
Revision
25827 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Wed Oct 15 14:02:59 2008 UTC (6 years, 3 months ago) by
ganis
File length: 35144 byte(s)
From Jan:
- Added the possibility to handle removed workers and partly processed packets. When a worker is stopped
while processing a packet it finishes, the current event and the rest of the packet is reassigned to another
worker. This is done via two interfaces:
- TVirtualPacketizer::AddProcessed(TSlave *sl, TProofProgressStatus *st, TList **)
- TVirtualPacketizer::ReassignPacket.
- New class TProofProgressStatus used to keep the query progress stauts in all the TProofPlayer objects and in
TPacketizerAdaptive::TSlaveStat. This class is also used to structure the relevant information send in
kPROOF_GETPACKET and kPROOF_STOPPROCESS messages.
- The class TPacketizerProgressive is removed completely.
- The PROOF protocol version is increased to 19: this is to handle the changes in the kPROOF_STOPPROCESS and
kPROOF_GETPACKET messages in Master - worker communication.
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
