Log of /trunk/proof/proofplayer/inc/TPacketizerUnit.h
Parent Directory
Revision
44657 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 12 08:22:13 2012 UTC (2 years, 7 months ago) by
ganis
File length: 3281 byte(s)
Diff to
previous 39088
In TVirtualPacketizer.h:
Add new virtual method AssignWork to be used to add work to an existing packetizer on
the fly. This is used while packetizing across submasters.
In TVirtualPacketizer.cxx:
Do not start the progress reporting on submaster (progress is under control of top masters).
In TPacketizerUnit:
First implementation of the new method AssignWork .
Revision
39088 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 4 11:10:47 2011 UTC (3 years, 8 months ago) by
ganis
File length: 3106 byte(s)
Diff to
previous 34748
Modify the way the packet size is calculated following a simple formula aiming at
having all workers finishing at the same time:
n_j = r_j / Sum r_i * N_left
where 'Sum r_i' is the total instantaneous rate, 'r_j' is the instantaneous rate
of worker j and Nleft is the number of cycles left to be done.
The patch also changes the way the size of the calibration packet is calculated:
a fraction is now used instead of an absolute number, which was too small in some
cases; the fraction refers to the average number per worker and defaults to 1%; it
can be changed with the Double_t parameter 'PROOF_PacketizerCalibFrac', e.g.
proof->SetParameter("PROOF_PacketizerCalibFrac", 0.05)
will set it to 5% .
This patch solves the problem observed in the ALICE CAF where the variations of
performance across machines was not correctly taken into account, and increasing
workers was not always leading to an increase of the overall perfomance.
Revision
34748 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Aug 9 10:18:05 2010 UTC (4 years, 5 months ago) by
ganis
File length: 2975 byte(s)
Diff to
previous 34086
Add the possibility to save the perfomance information shown by the dialog into a small
ntuple included in the output list. The ntuple contains 5 floats (processing time, number
of active workers, event rate, MBytes read, number of effective sessions on the cluster)
and it is filled each time the number of active workers changes or at max 100 regular
intervals at least 5 secs apart; in this way the ntuple has at most O(100 entries + number
of workers).
To enable the saving of the ntuple execute the following:
proof->SetParameter("PROOF_SaveProgressPerf", "yes");
before running the query. The ntuple is called 'PROOF_ProgressPerfNtuple'.
This patch also adds to the output list the parameters used by the active packetizer. Some
parameters of general interest (currently MinPacketTime and MaxPacketTime) have been moved
to TVirtualPacketizer and are always added to the list. Each packetizer is then responsible
of adding its relevant specific parameter to the dedicated list. The dedicated list is hosted
in TVirtualPacketizer and is transferred to the output list by TProofPlayer when finalising
the output list at the end of the query.
Revision
34086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 23 11:56:34 2010 UTC (4 years, 7 months ago) by
ganis
File length: 3029 byte(s)
Diff to
previous 31840
In TPacketizerUnit, add the possibility to exactly share the number of cycles between
the workers. This functionality was triggered by a request from roostat and it can be
useful when this packetizer is used to perform well defined actions on the workers.
The option can be enabled by calling
proof->SetParameter("PROOF_PacketizerFixedNum", (Int_t)1)
before processing.
Revision
30870 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 26 15:16:00 2009 UTC (5 years, 2 months ago) by
ganis
File length: 3577 byte(s)
Diff to
previous 30859
Use the improved estimation of the current rate also for TPacketizer and TPacketizerUnit.
Fixes also a problem with the progress bar with these packetizers appeared after the last
check-in.
Revision
30859 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Oct 24 14:53:07 2009 UTC (5 years, 3 months ago) by
ganis
File length: 3531 byte(s)
Diff to
previous 25896
Patch for improved performance monitoring. The 'Rate Plot' button in the progress
dialog is renamed 'Performance plot' and shows up to 4 plots with the event/sec,
the average read chunck size, the number of active workers and the number of active
PROOF sessions on the cluster, all as a function of processing time.
The read chunck size plot allows to monitor the usage of the cache.
The istantaneous processing rate (event/sec) is now better estimate: a few issues
with the normalizing times have ben solved, removing the artificial structures that
were observed.
The possibility to set a max packet time length is introduced (default 30 s); this
can be changes with the parameter PROOF_MaxPacketTime.
The size of the cache is also taken into account to optimize the use of the cache.
The parameter PROOF_UseParallelUnzip has been introduced to toggle the use of the
parallel unzip (default off for now).
A page describing the new performance plots is under preparation at
http://root.cern.ch/drupal/content/progress-dialog
Revision
25896 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 20 17:01:31 2008 UTC (6 years, 3 months ago) by
ganis
File length: 3428 byte(s)
Diff to
previous 25827
From Jan:
- Using consistently the recently introduced TProofProgressStatus in the kPROOF_GETPACKET
messages sent to TPacketizerUnit, TPacketizerAdaptive and TPacketizer; the message contains
the status of progress since the start of processing on a given node.
- Introduce TVirtualPacketizer::TVirtualSlaveStat as a base class of all the TSlaveStat
packetizer specific auxilliary classes.
- Full implemenation of GetProgressStatus() and AddProcessed(TProofProgressStatus *st) members
for TPacketizerUnit::TSlaveStat.
This patch should fix some consistency problems experienced after the patch introducing
TProofProgressStatus .
Revision
25827 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 15 14:02:59 2008 UTC (6 years, 3 months ago) by
ganis
File length: 3349 byte(s)
Diff to
previous 25273
From Jan:
- Added the possibility to handle removed workers and partly processed packets. When a worker is stopped
while processing a packet it finishes, the current event and the rest of the packet is reassigned to another
worker. This is done via two interfaces:
- TVirtualPacketizer::AddProcessed(TSlave *sl, TProofProgressStatus *st, TList **)
- TVirtualPacketizer::ReassignPacket.
- New class TProofProgressStatus used to keep the query progress stauts in all the TProofPlayer objects and in
TPacketizerAdaptive::TSlaveStat. This class is also used to structure the relevant information send in
kPROOF_GETPACKET and kPROOF_STOPPROCESS messages.
- The class TPacketizerProgressive is removed completely.
- The PROOF protocol version is increased to 19: this is to handle the changes in the kPROOF_STOPPROCESS and
kPROOF_GETPACKET messages in Master - worker communication.
Revision
25273 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 27 08:56:06 2008 UTC (6 years, 4 months ago) by
rdm
File length: 3446 byte(s)
Diff to
previous 22635
From Jan:
- A new optional version of PROOF with dynamic worker startup.
It can be enabled by the admin or a user with 'Proof.DynamicStartup'.
A session starts only on the master. When a query processing starts
at the master TXProofServ::GetWorkers() is called. It receives a
list of machines from the scheduler and the workers on the
machines are started. The environment is copied from the master
to the workers. It includes: the include and dynamic library paths,
the set of enabled packages as well as the macros loaded by the user.
- A new method TProof::AddWorkers(TList *workers) was added. It adds
the workers just before the query.
- A packet resubmitting mechanism. When a worker dies all the packets
that it processed are resubmitted.
- In TPacketizerAdaptive: fixing initialization of fgMaxSlaveCnt. By
default it was initialized twice.
Revision
20307 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 11 10:58:50 2007 UTC (7 years, 3 months ago) by
ganis
Original Path:
trunk/proofplayer/inc/TPacketizerUnit.h
File length: 3307 byte(s)
Copied from:
branches/dev/proof/proofplayer/inc/TPacketizerUnit.h revision 20306
Diff to
previous 20227
Import branches/dev/proof r20306
Summary (see branch logs for more details):
- Set of changes related to CPU quota control including
- broadcast of centrally determined priorities
- mechanism to renice processes in a quantitative way
- New packetizer for non-tree based analysis and related API in TProof (from L. Tran-Thanh)
- Support for merging output objects saved in files on the workers (from L. Tran-Thanh and me)
- Improve version binary compatibility checks using also the SVN revision number
(when available) to define the running version.
- Extend the version binary compatibility checks also to the cached selector binaries.
- Extend TDSet::Lookup so that in case of missing files, it can remove them from the
dataset (option removeMissing must be set).
- Move the data set lookup to the TProofPlayerRemote::Process.
- Handle properly the case of incomple datasets: if the file is not found in the lookup
don't try to validate it; add it, instead, to a 'missingFiles' list returned in the
output list (fixing bug #28800 in Savannah).
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
