Log of /trunk/proof/proofplayer/src/TVirtualPacketizer.cxx
Parent Directory
Revision
44657 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 12 08:22:13 2012 UTC (2 years, 7 months ago) by
ganis
File length: 16292 byte(s)
Diff to
previous 44065
In TVirtualPacketizer.h:
Add new virtual method AssignWork to be used to add work to an existing packetizer on
the fly. This is used while packetizing across submasters.
In TVirtualPacketizer.cxx:
Do not start the progress reporting on submaster (progress is under control of top masters).
In TPacketizerUnit:
First implementation of the new method AssignWork .
Revision
41801 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 7 19:13:11 2011 UTC (3 years, 2 months ago) by
ganis
File length: 15969 byte(s)
Diff to
previous 41740
Make sure hat the circular ntuples used internally are not atteched to any file.
Should fix a crash observed quitting PROOF-Lite when using file-resident output.
(reported on the forum).
Revision
39406 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu May 26 13:09:39 2011 UTC (3 years, 7 months ago) by
ganis
File length: 15745 byte(s)
Diff to
previous 39152
In TVirtualPacketizer::HandleTimer, when the query is done the progress timer just needs
to be stopped. Destruction will be done by the destructor at proper time.
Revision
39152 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 10 10:14:41 2011 UTC (3 years, 8 months ago) by
ganis
File length: 15796 byte(s)
Diff to
previous 38810
- In TPacketizerFile, make sure that key names use always the host FQDN; improves
solidity of the match.
- In TVirtualPacketizer, stop and destroy the progress timer when processing is done.
Revision
38810 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Apr 12 16:22:59 2011 UTC (3 years, 9 months ago) by
ganis
File length: 15699 byte(s)
Diff to
previous 37396
Patch to correctly honour selector abort status settings in PROOF. Currently only
the TSelector::kAbortProcess was handled by stopping processing. In particular
TSelector::kAbortFile was ignored; this recently created some problems in ALICE
with corrupted files, with repeated attempts to read events eventually leading to
bad_alloc exceptions.
This patch also fixes other related issues, in particular with the reporting of the
non-processed {files, events} in the final 'MissingFiles' list. This list should
now account much more precisely of the number of events which could not be processed.
It also fixes a problem with the final update of the progress information affecting
occasionally cases with skipped events.
Revision
37396 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 8 13:12:00 2010 UTC (4 years, 1 month ago) by
rdm
File length: 15647 byte(s)
Diff to
previous 35966
From Gerri:
- Fix a problem with the registration of missing files in the 'MissingFiles'
list. The files that could not be open during processing were not properly
registered in the list (only those found missing during validation or
giving problmes durign reading were correctly added).
- Add method TProof::ShowMissingFiles() to facilitate the display of the
list of missing files.
- Add method TProof::GetMissingFiles() to get a TFileCollection (dataset)
with the missing files for further processing.
Revision
34748 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Aug 9 10:18:05 2010 UTC (4 years, 5 months ago) by
ganis
File length: 15634 byte(s)
Diff to
previous 34557
Add the possibility to save the perfomance information shown by the dialog into a small
ntuple included in the output list. The ntuple contains 5 floats (processing time, number
of active workers, event rate, MBytes read, number of effective sessions on the cluster)
and it is filled each time the number of active workers changes or at max 100 regular
intervals at least 5 secs apart; in this way the ntuple has at most O(100 entries + number
of workers).
To enable the saving of the ntuple execute the following:
proof->SetParameter("PROOF_SaveProgressPerf", "yes");
before running the query. The ntuple is called 'PROOF_ProgressPerfNtuple'.
This patch also adds to the output list the parameters used by the active packetizer. Some
parameters of general interest (currently MinPacketTime and MaxPacketTime) have been moved
to TVirtualPacketizer and are always added to the list. Each packetizer is then responsible
of adding its relevant specific parameter to the dedicated list. The dedicated list is hosted
in TVirtualPacketizer and is transferred to the output list by TProofPlayer when finalising
the output list at the end of the query.
Revision
34557 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jul 22 15:12:01 2010 UTC (4 years, 6 months ago) by
rdm
File length: 11950 byte(s)
Diff to
previous 33640
fix cases in ROOT code where we would truncate the TTime (to avoid the
new error messages in TTime operator long on 32-bit platforms).
Revision
33640 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri May 28 13:55:26 2010 UTC (4 years, 7 months ago) by
ganis
File length: 12002 byte(s)
Diff to
previous 32204
Fix a few issues affecting the usage of tree friends in PROOF (see reports on the forum)
- In TDSet:
- Add friend alias as an option to the URL ("friend_alias=<alias>|") instead of an
object string in a TPair (non streamable!)
- In TVirtualPacketizer::CreateNewPacket
- Adapt to the change in TDSet
- In TEventIter:
- Adapt to the change in TDSet
- Make sure that the file with the friend is closed only when is not used anymore.
The patch also fixes an issue with the locality check for files in TDSet.
Revision
32204 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 3 19:17:40 2010 UTC (4 years, 11 months ago) by
ganis
File length: 11994 byte(s)
Diff to
previous 31296
Add support for processing many datasets in one go in TProof::Process(const char *dataset, ...).
Two options are provided:
- 'grand dataset': the datasets are added up and considered as a single dataset;
syntax: "dataset1|dataset2|..."
- 'keep separated': the datasets are processed one after the other; the user is
notified in the selector of the change of dataset so she/he
has the opportunity to separate the results. A new packetizer,
TPacketizerMulti, has been developed for this case: it basically
contains a list of standard packetizers (one for each dataset) and
loops over them.
Syntax: "dataset1,dataset2,..." or dataset1 dataset2 ..."
In both cases, entry-list can be applied using the syntax "dataset<<entrylist", e.g.
"dataset1<<el1|dataset2<<el2|".
See http://root.cern.ch/drupal/content/working-data-sets#currentelem for more details.
A test for the new functionality has been added to test/stressProof.cxx .
Revision
30862 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Oct 25 08:26:46 2009 UTC (5 years, 3 months ago) by
ganis
File length: 11420 byte(s)
Diff to
previous 30859
- Further improvement in the estimation of the current rate
- Fix a problem preventing the chunck size to be displayed in some cases
- Adjust the scale for displaying the read bytes (use GB or TB when relevant)
Revision
30859 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Oct 24 14:53:07 2009 UTC (5 years, 3 months ago) by
ganis
File length: 11207 byte(s)
Diff to
previous 25918
Patch for improved performance monitoring. The 'Rate Plot' button in the progress
dialog is renamed 'Performance plot' and shows up to 4 plots with the event/sec,
the average read chunck size, the number of active workers and the number of active
PROOF sessions on the cluster, all as a function of processing time.
The read chunck size plot allows to monitor the usage of the cache.
The istantaneous processing rate (event/sec) is now better estimate: a few issues
with the normalizing times have ben solved, removing the artificial structures that
were observed.
The possibility to set a max packet time length is introduced (default 30 s); this
can be changes with the parameter PROOF_MaxPacketTime.
The size of the cache is also taken into account to optimize the use of the cache.
The parameter PROOF_UseParallelUnzip has been introduced to toggle the use of the
parallel unzip (default off for now).
A page describing the new performance plots is under preparation at
http://root.cern.ch/drupal/content/progress-dialog
Revision
25918 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 22 15:00:04 2008 UTC (6 years, 3 months ago) by
ganis
File length: 10674 byte(s)
Diff to
previous 25827
First version of PROOFLite, a 2-tier realization of PROOF intended for multi-core
machines. In PROOFLite the client start directly the workers; no daemon is required.
Currently this works on Unix only. Some issues with the internal connection still need
to be sorted out on Windows.
To start a session just use
TProof::Open("") or TProof::Open("lite")
From there on everything should be as in normal PROOF. To start a standard PROOF
session (i.e. via daemons) on the localhost use TProof::Open("localhost").
(The default meaning of "" in TProof::Open() is controlled via the rc value
'Proof.LocalDefault').
To run stressProof with PROOFLite pas the argument 'lite':
./stressProof lite
By default, PROOFLite sets the number of workers equal to the number of cores.
The number of workers is controlled by the rc variable 'ProofLite.Workers'; it can
also be set on the fly with the option "workers=<n>", e.g. TProof::Open("lite?workers=2").
This patch also introduces the rc variable 'Proof.Sandbox' to control the local location
of the sandboxes. The default has been changed from "~/proof" to "~/.proof" to avoid
interferences with possible users' working areas. The sandbox contains the directory
'packages' (with the clients packages; used by standard PROOF and PROOF Lite),
and some directories used only by PROOFLite: 'cache' (binary cache) and 'datasets' with
the datasets. The location of these directories is controlled by the rc variables
'Proof.PackageDir', 'Proof.CacheDir' and 'Proof.DataSetDir'.
The working directory for PROOFLite is in the form workdir=<sandbox>/path-to-current-dir where
~/path/to/current/dir is the directory from where the session in launched. The directory
<workdir> contains the session working dirs (in the form 'session-<host>-<time>-<pid>'),
with the symlink 'last-lite-session' pointing to the last started session, and the 'queries'
directory, with the results of queries started from the current directory.
Revision
25827 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 15 14:02:59 2008 UTC (6 years, 3 months ago) by
ganis
File length: 10315 byte(s)
Diff to
previous 23986
From Jan:
- Added the possibility to handle removed workers and partly processed packets. When a worker is stopped
while processing a packet it finishes, the current event and the rest of the packet is reassigned to another
worker. This is done via two interfaces:
- TVirtualPacketizer::AddProcessed(TSlave *sl, TProofProgressStatus *st, TList **)
- TVirtualPacketizer::ReassignPacket.
- New class TProofProgressStatus used to keep the query progress stauts in all the TProofPlayer objects and in
TPacketizerAdaptive::TSlaveStat. This class is also used to structure the relevant information send in
kPROOF_GETPACKET and kPROOF_STOPPROCESS messages.
- The class TPacketizerProgressive is removed completely.
- The PROOF protocol version is increased to 19: this is to handle the changes in the kPROOF_STOPPROCESS and
kPROOF_GETPACKET messages in Master - worker communication.
Revision
23986 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri May 23 09:10:18 2008 UTC (6 years, 8 months ago) by
ganis
File length: 10368 byte(s)
Diff to
previous 23632
Import fixes / new functionality from branches/dev/proof:
- proof/proofplayer/src/TPacketizerAdaptive.cxx
Implement the classic strategy of the TPacketizer in TPacketizerAdaptive.
The strategy can be changed from adaptive (default) to TPacketizer with:
"PROOF_PacketizerStrategy" parameter to PROOF
- proof/proofplayer/src/TProofPlayer.cxx
Fixed error messages for 'MissingFiles' and 'FailedPackets' lists.
Improve fault detection by creating a list of failed packets upon a mismatch
in the expected and actual number of processed events; the list is added to
the output list.
- proof/proofplayer/src/TVirtualPacketizer.cxx
Make sure that something has been processed before setting kIsDone.
Improve fault detection by creating a list of failed packets upon a mismatch
in the expected and actual number of processed events; the list is added to
the output list.
- proof/proofplayer/inc/TVirtualPacketizer.h
Improve fault detection by creating a list of failed packets upon a mismatch
in the expected and actual number of processed events; the list is added to
the output list.
- proof/proofplayer/inc/TPacketizerAdaptive.h
Implement the classic strategy of the TPacketizer in TPacketizerAdaptive
- proof/proof/src/TProofServ.cxx
Fixes:
+ option string: "stageOnly" --> "stagedOnly".
+ add parenthesis to avoid a warning after the previous patch.
+ remove the objects added to the missingFiles in TDSet::Add from the 'dataset'
before deleting it
+ fixed an error HandleCheckFile ('kPROOF_WorkDir' instead of 'kPROOF_PackDir').
+ make fCacheDir and fPackageDir controllable via directive
+ in TProofServ::ErrorHandler: do not create the related additional buffer if
not logging to syslog
Added functionality:
- add possibility to flag an "Info" message as service message using
the prefix "|Svc" in the location field; e.g.
Info("SetupCommon|Svc", "Test of SvcMsg");
will produce something like
09:28:24 6892 Mst-0 | SvcMsg in <TXProofServ::SetupCommon>: Test of SvcMsg
This is needed to be able in the future to filter-out messages needed
by some services (e.g. the forthcoming memory checker) which should not be
displayed by default.
- proof/proofd/src/XrdProofSched.cxx
Improve the calculation of the number of workers to assign by using fMinForQuery
as a minimum.
Fix the length of method separators.
Fix signed/unsigned warning.
Revision
20682 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 6 15:51:59 2007 UTC (7 years, 2 months ago) by
ganis
Original Path:
trunk/proofplayer/src/TVirtualPacketizer.cxx
File length: 10049 byte(s)
Diff to
previous 20086
Import branches/dev/proof r20654
Summary:
+ Improvements
- Add support for SSH SOCKS4 tunnelling; the local port for the tunnel can be specified
in the master URL, e.g. TProof::Open("master/?tunnel:8000")
- Add the possibility to plot the estimated instantaneous rate
- Add "PROOF_ForceLocal" parameter to the !TPacketizerAdaptive; if set to 1, all the data
are processed locally.
- Add support for remote grep functionality while retrieving logs (needed by the forthcoming
memory monitor)
+ Bug fixes
- Several small fixes to revive the multi-master mode.
- XrdProofdProtocol:
- add missing lock to the client instance in SendMsg to avoid screwing up requests from
workers on the same machine
- lock the mutex of the requester when setting prorities
- add notification during Reset
- fix problem with the detection of the 'allow' directive
- fix problem with the parsing of the return value from XrdProofServProxy::TerminateProofServ()
- always use the effective user to retrieve info fom another server
(XrdProofdManager::GetProofConn is now used)
- fix possible dead-locks from debug notifications done after hard-killing a session
- re-enable the garbage collector thread of the connection manager in XrdProofConn to
fix a problem with closing physical connections;
- fix a problem with !CleanupProofServ in the case of a non-privileged daemon running
in multi-user mode
- introduce a timeout when waiting for the startup of a 'proofserv'.
- XrdProofConn: init mutex in the ctor; lock in SendRecv
- XrdProofSched: add support for using the priorities defined in the group manager to define
the number of workers for sessions
- TProof:
- Broadcast priorities to unique nodes only
- timeout after 5 mins the initial Collect to avoid clients getting stuck at this stage
- add support for generic timeout in Collect (disabled by default)
- fix a problem with SendFile.
- TXProofServ: add a call to !TProof::InterruptCurrentMonitor() in Terminate() to stop
infinite loops in Collect
- TXSocket:
- Implement a flag to interrupt a TXSocket while waiting for messages
- Split the session creation timeout in 4 attempts: the total timeout is the same but
it may circumvent occasional forking problems.
- XROOTD:
- fix a potential (possibly accademic) memory leak in the client
- fix a access permission problem with Kerberos ticket forwarding
- fix bug preventing 'locate' to work properly
- re-enable optimized 'locate'
Revision
20086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Sep 24 18:19:34 2007 UTC (7 years, 4 months ago) by
ganis
Original Path:
trunk/proofplayer/src/TVirtualPacketizer.cxx
File length: 9450 byte(s)
Diff to
previous 19826
From Jan:
- Add a heuristic mechanism to avoid undervaluing the processing rate and to short packets.
- Add parameters: PROOF_MinPacketTime and PROOF_PacketAsAFraction to allow tuning performance.
- Change name TAdaptivePacketizer to TPacketizerAdaptive in order to follow the naming
convention for packetizers
- Slightly tuning a few parameters.
Revision
18905 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 29 16:06:55 2007 UTC (7 years, 7 months ago) by
ganis
Original Path:
trunk/proofplayer/src/TVirtualPacketizer.cxx
File length: 9179 byte(s)
Diff to
previous 18336
From Jan:
Consolidation work on similar code in 3 different packetizers.
Moving common members of TPacketizer, TAdaptivePacketizer and partly TPacketizerProgressive
up to TVirtualPacketizer:
- HandleTimer
- SplitEventList
- GetEntriesProcessed
- CreateNewPacket
- common member attributes
The progressive packetizer was also changed in order to use CreateNewPacket and fProcessed.
Revision
5717 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Dec 2 18:50:12 2002 UTC (12 years, 1 month ago) by
rdm
Original Path:
trunk/proofplayer/src/TVirtualPacketizer.cxx
File length: 3987 byte(s)
Diff to
previous 5401
mega patch to remove almost all compiler warnings on MacOS X where the
compiler is by default in pedantic mode (LHCb also like to use this option).
The following issues have been fixed:
- removal of unused arguments
- comparison between signed and unsigned integers
- not calling of base class copy ctor in copy ctor's
To be done, the TGeo classes where we get still many hundred warnings of
the above nature. List forwarded to Andrei.
Revision
4955 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Wed Jul 17 12:29:38 2002 UTC (12 years, 6 months ago) by
rdm
Original Path:
trunk/proofplayer/src/TVirtualPacketizer.cxx
File length: 3919 byte(s)
PROOF mods from Maarten. New TVirtualPacketizer that is the base class for
different packetizers. You want different packetizers for different
processing strategies (depending on files being available on disk or if
they need to be sequentially retrieved from mass storage, etc.).
Added global PROOF debug facility.
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
