Log of /trunk/proof/proofplayer/src/TPacketizerUnit.cxx
Parent Directory
Revision
48992 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Mar 28 15:26:26 2013 UTC (21 months, 3 weeks ago) by
rdm
File length: 20344 byte(s)
Diff to
previous 45614
From Lifeng Sun:
The attached patchset fixes a bunch of typo in the source:
0001-succes-success.patch
0002-preceed-preced.patch
0003-informations-information.patch
0004-childs-children.patch
0005-avaliable-available.patch
0006-writeable-writable.patch
0007-comand-command.patch
0008-unkown-unknown.patch
0009-wierd-weird.patch
0010-wheter-whether.patch
0011-unecessary-unnecessary.patch
0012-splitted-split.patch
0013-registerd-registered.patch
0014-recieve-receive.patch
0015-processsing-processing.patch
0016-ouput-output.patch
0017-mutiple-multiple.patch
0018-lenght-length.patch
0019-interupted-interrupted.patch
0020-independant-independent.patch
0021-inconsistant-inconsistent.patch
0022-expresion-expression.patch
0023-explicitely-explicitly.patch
0024-enviroment-environment.patch
0025-deafult-default.patch
0026-continous-continuous.patch
0027-completly-completely.patch
0028-commited-committed.patch
0029-choosen-chosen.patch
0030-backgroud-background.patch
0031-auxilliary-auxiliary.patch
0032-authentification-authentication.patch
0033-appropiate-appropriate.patch
0034-an-other-another.patch
0035-environement-environment.patch
0036-targetting-targeting.patch
0037-suppported-supported.patch
0038-paramater-parameter.patch
Revision
44657 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 12 08:22:13 2012 UTC (2 years, 7 months ago) by
ganis
File length: 19595 byte(s)
Diff to
previous 44010
In TVirtualPacketizer.h:
Add new virtual method AssignWork to be used to add work to an existing packetizer on
the fly. This is used while packetizing across submasters.
In TVirtualPacketizer.cxx:
Do not start the progress reporting on submaster (progress is under control of top masters).
In TPacketizerUnit:
First implementation of the new method AssignWork .
Revision
44010 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 29 23:07:48 2012 UTC (2 years, 8 months ago) by
ganis
File length: 18007 byte(s)
Diff to
previous 41801
Several optimizations in the low level PROOF event loop (TProofPlayer::Process):
- Per-event calls to TEventIter::GetNextEvent are replaced by per-packet calls
to a new function TEventIter::GetNextPacket and a for loop over the returned
range;
- Progress information is filled in on per-packet pace just before asking for
a new packet, instead of being filled for each event;
- In TProofProgressStatus, the relevant setters in were inlined, and the (expensive)
calls to SetLastUpdate() reduced to the minimum required;
- Status of objects is checked using bits instead of functions calls;
- Check whether to dump memory info is done before calling CheckMemUsage.
These optimizations allow to reduce dramatically the overhead introduced by the
operations PROOF needs to perform during the event loop. A measurement of the
overhead can be obtained from a very light computational task, for example,
generating one random number and filling one histogram; executing this task
within a PROOF-Lite session with 1 worker now takes only 1.8 times the time
required by a straight loop in the parent ROOT session; the same number before
was about 13.
The patch also introduces a couple of small changes enabling the use of
TProofPlayerLocal to process selectors in the local session using the PROOF
iteration mechanism; this is very useful for optimizations studies.
Finally, a couple of missing protections were added in TProofPlayer.
Revision
41801 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 7 19:13:11 2011 UTC (3 years, 2 months ago) by
ganis
File length: 17918 byte(s)
Diff to
previous 39088
Make sure hat the circular ntuples used internally are not atteched to any file.
Should fix a crash observed quitting PROOF-Lite when using file-resident output.
(reported on the forum).
Revision
39088 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 4 11:10:47 2011 UTC (3 years, 8 months ago) by
ganis
File length: 17888 byte(s)
Diff to
previous 35120
Modify the way the packet size is calculated following a simple formula aiming at
having all workers finishing at the same time:
n_j = r_j / Sum r_i * N_left
where 'Sum r_i' is the total instantaneous rate, 'r_j' is the instantaneous rate
of worker j and Nleft is the number of cycles left to be done.
The patch also changes the way the size of the calibration packet is calculated:
a fraction is now used instead of an absolute number, which was too small in some
cases; the fraction refers to the average number per worker and defaults to 1%; it
can be changed with the Double_t parameter 'PROOF_PacketizerCalibFrac', e.g.
proof->SetParameter("PROOF_PacketizerCalibFrac", 0.05)
will set it to 5% .
This patch solves the problem observed in the ALICE CAF where the variations of
performance across machines was not correctly taken into account, and increasing
workers was not always leading to an increase of the overall perfomance.
Revision
34748 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Aug 9 10:18:05 2010 UTC (4 years, 5 months ago) by
ganis
File length: 17486 byte(s)
Diff to
previous 34087
Add the possibility to save the perfomance information shown by the dialog into a small
ntuple included in the output list. The ntuple contains 5 floats (processing time, number
of active workers, event rate, MBytes read, number of effective sessions on the cluster)
and it is filled each time the number of active workers changes or at max 100 regular
intervals at least 5 secs apart; in this way the ntuple has at most O(100 entries + number
of workers).
To enable the saving of the ntuple execute the following:
proof->SetParameter("PROOF_SaveProgressPerf", "yes");
before running the query. The ntuple is called 'PROOF_ProgressPerfNtuple'.
This patch also adds to the output list the parameters used by the active packetizer. Some
parameters of general interest (currently MinPacketTime and MaxPacketTime) have been moved
to TVirtualPacketizer and are always added to the list. Each packetizer is then responsible
of adding its relevant specific parameter to the dedicated list. The dedicated list is hosted
in TVirtualPacketizer and is transferred to the output list by TProofPlayer when finalising
the output list at the end of the query.
Revision
34086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 23 11:56:34 2010 UTC (4 years, 7 months ago) by
ganis
File length: 16954 byte(s)
Diff to
previous 31840
In TPacketizerUnit, add the possibility to exactly share the number of cycles between
the workers. This functionality was triggered by a request from roostat and it can be
useful when this packetizer is used to perform well defined actions on the workers.
The option can be enabled by calling
proof->SetParameter("PROOF_PacketizerFixedNum", (Int_t)1)
before processing.
Revision
31438 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Nov 26 16:23:19 2009 UTC (5 years, 1 month ago) by
ganis
File length: 15365 byte(s)
Diff to
previous 30872
Make sure that the entry number passed to TSelector::Process is unique and in
increasing order for non-data driven processing (packetizer TPacketizerUnit).
This allows to give a meaning to this variable, for example to related it to
one dimension of an integration.
Revision
30870 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 26 15:16:00 2009 UTC (5 years, 2 months ago) by
ganis
File length: 15409 byte(s)
Diff to
previous 26875
Use the improved estimation of the current rate also for TPacketizer and TPacketizerUnit.
Fixes also a problem with the progress bar with these packetizers appeared after the last
check-in.
Revision
26875 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 12 14:21:34 2008 UTC (6 years, 1 month ago) by
ganis
File length: 14305 byte(s)
Diff to
previous 25917
Patch to:
- Add the possibility to set upper limits on the virtual memory (changes in TProofPlayer,
TProofServ, TXProofServ and pmain (separate patch)).
If enebled, the session gets firts a warning when it reaches 80% of the limit, and then
processing is stopped whenit exceeds 95% of the limit, sending back the results.
Also, the memory footprint is notified when the session is terminated.
- Make sure the the active valuse in XrdProofWorker are always correct; this was not the
case for dynamic startup as the notification at the end of the query was not done.
This information is crucial or the scheduler.
The way the information is stored in XrdProofdProofServ had to be modified
and a new internal message type (kReleaseWorker) added.
(Changes in several proofd classes, TProof, TXsocket and TXProofServ)
- Fix problem with setting a static upper limit of the sessions that can be started,
and enable this functionality for the dynamic mode (changes in XrdProofSched)
- Remove a deleted worker from all the lists in TProof::MarkBad to avoid later attempts of use.
- Better control the use of the internal pipe for socket readiness notification in TXSocket.
Revision
25917 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 22 14:55:20 2008 UTC (6 years, 3 months ago) by
ganis
File length: 14328 byte(s)
Diff to
previous 25896
Do not delete fProgress after a call to HandleTimer(0): this is done in ~TVirtualPacketizer
and may screw up updatign of the progress bar if done too early.
Revision
25896 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 20 17:01:31 2008 UTC (6 years, 3 months ago) by
ganis
File length: 14340 byte(s)
Diff to
previous 25867
From Jan:
- Using consistently the recently introduced TProofProgressStatus in the kPROOF_GETPACKET
messages sent to TPacketizerUnit, TPacketizerAdaptive and TPacketizer; the message contains
the status of progress since the start of processing on a given node.
- Introduce TVirtualPacketizer::TVirtualSlaveStat as a base class of all the TSlaveStat
packetizer specific auxilliary classes.
- Full implemenation of GetProgressStatus() and AddProcessed(TProofProgressStatus *st) members
for TPacketizerUnit::TSlaveStat.
This patch should fix some consistency problems experienced after the patch introducing
TProofProgressStatus .
Revision
25859 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Oct 17 16:38:52 2008 UTC (6 years, 3 months ago) by
ganis
File length: 13242 byte(s)
Diff to
previous 25827
- Make all packetizers understand the GETPACKET messages containing the new TProofProgressStatus
structure
- Add a few missing protections
Revision
25827 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 15 14:02:59 2008 UTC (6 years, 3 months ago) by
ganis
File length: 12537 byte(s)
Diff to
previous 25569
From Jan:
- Added the possibility to handle removed workers and partly processed packets. When a worker is stopped
while processing a packet it finishes, the current event and the rest of the packet is reassigned to another
worker. This is done via two interfaces:
- TVirtualPacketizer::AddProcessed(TSlave *sl, TProofProgressStatus *st, TList **)
- TVirtualPacketizer::ReassignPacket.
- New class TProofProgressStatus used to keep the query progress stauts in all the TProofPlayer objects and in
TPacketizerAdaptive::TSlaveStat. This class is also used to structure the relevant information send in
kPROOF_GETPACKET and kPROOF_STOPPROCESS messages.
- The class TPacketizerProgressive is removed completely.
- The PROOF protocol version is increased to 19: this is to handle the changes in the kPROOF_STOPPROCESS and
kPROOF_GETPACKET messages in Master - worker communication.
Revision
25273 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 27 08:56:06 2008 UTC (6 years, 4 months ago) by
rdm
File length: 12746 byte(s)
Diff to
previous 22635
From Jan:
- A new optional version of PROOF with dynamic worker startup.
It can be enabled by the admin or a user with 'Proof.DynamicStartup'.
A session starts only on the master. When a query processing starts
at the master TXProofServ::GetWorkers() is called. It receives a
list of machines from the scheduler and the workers on the
machines are started. The environment is copied from the master
to the workers. It includes: the include and dynamic library paths,
the set of enabled packages as well as the macros loaded by the user.
- A new method TProof::AddWorkers(TList *workers) was added. It adds
the workers just before the query.
- A packet resubmitting mechanism. When a worker dies all the packets
that it processed are resubmitted.
- In TPacketizerAdaptive: fixing initialization of fgMaxSlaveCnt. By
default it was initialized twice.
Revision
20758 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Nov 11 21:29:49 2007 UTC (7 years, 2 months ago) by
ganis
Original Path:
trunk/proofplayer/src/TPacketizerUnit.cxx
File length: 12753 byte(s)
Diff to
previous 20597
Several small fixes and few additions imported from 'branches/dev/proof':
- XrdProofdProtocol:
+ Fix a problem in CleanuProofServ sometimes preventing proper cleanup
+ In ReadBuffer, do not trigger an error on empty files
+ Add missing protection against an empty string while setting the ROOT version
- XrdProofdManager:
+ Define the list of unique nodes to avoid broadcasting twice to the same node
- XrdProofWorker: improve matching technique
- TProof: add setter and getter for Int_t parameter
- TProofServ:
+ do not flush the log file in SendAsynMessage: this recent change
made the logs "disappear" form the ProgressDialog
+ Activate control on the max number of query result kept
- TPacketizerUnit: fix a problem with filling the histof for PerfStats
- TProofPlayer: call Begin() also during selecter re-initialization
Revision
20307 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 11 10:58:50 2007 UTC (7 years, 3 months ago) by
ganis
Original Path:
trunk/proofplayer/src/TPacketizerUnit.cxx
File length: 12721 byte(s)
Copied from:
branches/dev/proof/proofplayer/src/TPacketizerUnit.cxx revision 20306
Diff to
previous 20227
Import branches/dev/proof r20306
Summary (see branch logs for more details):
- Set of changes related to CPU quota control including
- broadcast of centrally determined priorities
- mechanism to renice processes in a quantitative way
- New packetizer for non-tree based analysis and related API in TProof (from L. Tran-Thanh)
- Support for merging output objects saved in files on the workers (from L. Tran-Thanh and me)
- Improve version binary compatibility checks using also the SVN revision number
(when available) to define the running version.
- Extend the version binary compatibility checks also to the cached selector binaries.
- Extend TDSet::Lookup so that in case of missing files, it can remove them from the
dataset (option removeMissing must be set).
- Move the data set lookup to the TProofPlayerRemote::Process.
- Handle properly the case of incomple datasets: if the file is not found in the lookup
don't try to validate it; add it, instead, to a 'missingFiles' list returned in the
output list (fixing bug #28800 in Savannah).
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
