Log of /trunk/proof/proof/inc/TVirtualProofPlayer.h
Parent Directory
Revision
45348 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Aug 2 12:23:54 2012 UTC (2 years, 5 months ago) by
ganis
File length: 6584 byte(s)
Diff to
previous 45283
Consolidation of the automatic output saving functionality
- Fix an issue with saving output from memory on the master
- Remove some duplications and fix several small bugs
Revision
45283 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jul 30 10:42:38 2012 UTC (2 years, 5 months ago) by
ganis
File length: 6501 byte(s)
Diff to
previous 44010
Patch automatizing the usage of the file-based technology to handle outputs.
After this patch PROOF users can define as an option to Process an output file for their results
and choose if the merging goes via file or in memory. User can also decide to save theit results
in a set of files identified by a name (dataset).
Administrators can force merging-via-file or can make it conditional to the resident memory exceeding
a given threshold.
The new options are documented at
http://root.cern.ch/drupal/content/handling-outputs
Since TProofOutputFile is now used by TProof, the class has been moved from proof/proofplayer
(libProofPlayer) to proof/proof (libProof).
The selector tutorials/proof/ProofSimple.h,.C has been modified to allo testing the new options,
and a new test has been added to stressProof.
Revision
44010 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 29 23:07:48 2012 UTC (2 years, 8 months ago) by
ganis
File length: 6429 byte(s)
Diff to
previous 43044
Several optimizations in the low level PROOF event loop (TProofPlayer::Process):
- Per-event calls to TEventIter::GetNextEvent are replaced by per-packet calls
to a new function TEventIter::GetNextPacket and a for loop over the returned
range;
- Progress information is filled in on per-packet pace just before asking for
a new packet, instead of being filled for each event;
- In TProofProgressStatus, the relevant setters in were inlined, and the (expensive)
calls to SetLastUpdate() reduced to the minimum required;
- Status of objects is checked using bits instead of functions calls;
- Check whether to dump memory info is done before calling CheckMemUsage.
These optimizations allow to reduce dramatically the overhead introduced by the
operations PROOF needs to perform during the event loop. A measurement of the
overhead can be obtained from a very light computational task, for example,
generating one random number and filling one histogram; executing this task
within a PROOF-Lite session with 1 worker now takes only 1.8 times the time
required by a straight loop in the parent ROOT session; the same number before
was about 13.
The patch also introduces a couple of small changes enabling the use of
TProofPlayerLocal to process selectors in the local session using the PROOF
iteration mechanism; this is very useful for optimizations studies.
Finally, a couple of missing protections were added in TProofPlayer.
Revision
43044 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 17 14:34:52 2012 UTC (2 years, 11 months ago) by
ganis
File length: 6329 byte(s)
Diff to
previous 39171
From Taesoo Kim and me.
Add support for selector-by-object processing in PROOF.
The selector object, created and configured locally by the user, is added to the input
list and recuperated from there on the worker machines for processing. Any input list
setting in the selector itself is not streamed but temporarly moved to then standard
input list, so that user can use the selector input list as container of processing
information if they find convenient to do so.
Process(...) methods with the file name argument replaced by 'TSelector *' have
introduced where relevant (TProof, TProofPlayer and their derivatives, TDSet).
Two tests for the new functionality have been added to test/stressProof.cxx (tests 25
and 26).
Revision
36592 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Nov 11 10:43:17 2010 UTC (4 years, 2 months ago) by
ganis
File length: 6023 byte(s)
Diff to
previous 36537
- Terminate a proofserv after it has been stopped for large memory footprint and it has
sent the results back
- Notify information about which object was being merged if a memory exception happens
during merging on the master
- Allow to set the HWM and stop-process fractions via the environment
Revision
36537 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 8 16:16:41 2010 UTC (4 years, 2 months ago) by
ganis
File length: 6114 byte(s)
Diff to
previous 33638
Patch to improve the diagnostic in case of exceptions. Information about the event and file being
processed at the moment the exception was raised is sent to the client, e.g.
0.5: caught exception triggered by signal '1' while processing dset:'EventTree',
file:'http://root.cern.ch/files/data/event_3.root', event:1 - check logs for possible stacktrace
The patch also fixes a problem with submergers observed when a worker was stopped because above
the memory limits: this worker was established as merger but could not do the work, for obvious
reasons, freezing the session.
Revision
31505 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 2 13:20:06 2009 UTC (5 years, 1 month ago) by
brun
File length: 6002 byte(s)
Diff to
previous 30859
From Katerina Opocenska:
Patch implementing the new submerger functionality to speed up the merging phase.
At the end of the query, a set of workers are promoted submergers and assigned a
sub-set of workers to merge. Once each sub-merger has merged its sub-set of workers,
it sends its result to the master, which merges the partial results into the final
set of results.
The determination of the sub-mergers is always done dynamically, based on the recent
performance of workers. An optimal (i.e. giving the highest speed-up) number can be
calculated analytically under simpleassumptions.
Merging via submergers is by default disabled. To enable it, with the optimal number
of sub-mergers, one should set the integer parameter 'PROOF_UseMergers' to 0, i.e.
proof->SetParameter("PROOF_UseMergers", 0)
To force S sub-mergers (regardless of the optimal number) do
proof->SetParameter("PROOF_UseMergers", S)
Revision
30859 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Oct 24 14:53:07 2009 UTC (5 years, 3 months ago) by
ganis
File length: 5964 byte(s)
Diff to
previous 26382
Patch for improved performance monitoring. The 'Rate Plot' button in the progress
dialog is renamed 'Performance plot' and shows up to 4 plots with the event/sec,
the average read chunck size, the number of active workers and the number of active
PROOF sessions on the cluster, all as a function of processing time.
The read chunck size plot allows to monitor the usage of the cache.
The istantaneous processing rate (event/sec) is now better estimate: a few issues
with the normalizing times have ben solved, removing the artificial structures that
were observed.
The possibility to set a max packet time length is introduced (default 30 s); this
can be changes with the parameter PROOF_MaxPacketTime.
The size of the cache is also taken into account to optimize the use of the cache.
The parameter PROOF_UseParallelUnzip has been introduced to toggle the use of the
parallel unzip (default off for now).
A page describing the new performance plots is under preparation at
http://root.cern.ch/drupal/content/progress-dialog
Revision
26382 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Nov 22 17:18:42 2008 UTC (6 years, 2 months ago) by
ganis
File length: 5705 byte(s)
Diff to
previous 25827
Fixes for Draw queries in PROOF-Lite
- Do not call Finalize (it is not needed and was giving a warning about an empty query)
- Enable the progress bar
Revision
25827 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 15 14:02:59 2008 UTC (6 years, 3 months ago) by
ganis
File length: 5548 byte(s)
Diff to
previous 25273
From Jan:
- Added the possibility to handle removed workers and partly processed packets. When a worker is stopped
while processing a packet it finishes, the current event and the rest of the packet is reassigned to another
worker. This is done via two interfaces:
- TVirtualPacketizer::AddProcessed(TSlave *sl, TProofProgressStatus *st, TList **)
- TVirtualPacketizer::ReassignPacket.
- New class TProofProgressStatus used to keep the query progress stauts in all the TProofPlayer objects and in
TPacketizerAdaptive::TSlaveStat. This class is also used to structure the relevant information send in
kPROOF_GETPACKET and kPROOF_STOPPROCESS messages.
- The class TPacketizerProgressive is removed completely.
- The PROOF protocol version is increased to 19: this is to handle the changes in the kPROOF_STOPPROCESS and
kPROOF_GETPACKET messages in Master - worker communication.
Revision
25273 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 27 08:56:06 2008 UTC (6 years, 4 months ago) by
rdm
File length: 5445 byte(s)
Diff to
previous 22635
From Jan:
- A new optional version of PROOF with dynamic worker startup.
It can be enabled by the admin or a user with 'Proof.DynamicStartup'.
A session starts only on the master. When a query processing starts
at the master TXProofServ::GetWorkers() is called. It receives a
list of machines from the scheduler and the workers on the
machines are started. The environment is copied from the master
to the workers. It includes: the include and dynamic library paths,
the set of enabled packages as well as the macros loaded by the user.
- A new method TProof::AddWorkers(TList *workers) was added. It adds
the workers just before the query.
- A packet resubmitting mechanism. When a worker dies all the packets
that it processed are resubmitted.
- In TPacketizerAdaptive: fixing initialization of fgMaxSlaveCnt. By
default it was initialized twice.
Revision
20682 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 6 15:51:59 2007 UTC (7 years, 2 months ago) by
ganis
Original Path:
trunk/proof/inc/TVirtualProofPlayer.h
File length: 5135 byte(s)
Diff to
previous 19826
Import branches/dev/proof r20654
Summary:
+ Improvements
- Add support for SSH SOCKS4 tunnelling; the local port for the tunnel can be specified
in the master URL, e.g. TProof::Open("master/?tunnel:8000")
- Add the possibility to plot the estimated instantaneous rate
- Add "PROOF_ForceLocal" parameter to the !TPacketizerAdaptive; if set to 1, all the data
are processed locally.
- Add support for remote grep functionality while retrieving logs (needed by the forthcoming
memory monitor)
+ Bug fixes
- Several small fixes to revive the multi-master mode.
- XrdProofdProtocol:
- add missing lock to the client instance in SendMsg to avoid screwing up requests from
workers on the same machine
- lock the mutex of the requester when setting prorities
- add notification during Reset
- fix problem with the detection of the 'allow' directive
- fix problem with the parsing of the return value from XrdProofServProxy::TerminateProofServ()
- always use the effective user to retrieve info fom another server
(XrdProofdManager::GetProofConn is now used)
- fix possible dead-locks from debug notifications done after hard-killing a session
- re-enable the garbage collector thread of the connection manager in XrdProofConn to
fix a problem with closing physical connections;
- fix a problem with !CleanupProofServ in the case of a non-privileged daemon running
in multi-user mode
- introduce a timeout when waiting for the startup of a 'proofserv'.
- XrdProofConn: init mutex in the ctor; lock in SendRecv
- XrdProofSched: add support for using the priorities defined in the group manager to define
the number of workers for sessions
- TProof:
- Broadcast priorities to unique nodes only
- timeout after 5 mins the initial Collect to avoid clients getting stuck at this stage
- add support for generic timeout in Collect (disabled by default)
- fix a problem with SendFile.
- TXProofServ: add a call to !TProof::InterruptCurrentMonitor() in Terminate() to stop
infinite loops in Collect
- TXSocket:
- Implement a flag to interrupt a TXSocket while waiting for messages
- Split the session creation timeout in 4 attempts: the total timeout is the same but
it may circumvent occasional forking problems.
- XROOTD:
- fix a potential (possibly accademic) memory leak in the client
- fix a access permission problem with Kerberos ticket forwarding
- fix bug preventing 'locate' to work properly
- re-enable optimized 'locate'
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
