Log of /trunk/proof/proof/src/TProofLite.cxx
Parent Directory
Revision
47832 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Dec 4 19:37:56 2012 UTC (2 years, 1 month ago) by
ganis
File length: 78520 byte(s)
Diff to
previous 47451
Make sure that the fSlaves list contains always ALL workers, eventhose which did not initially startup; this way GetListOfSlaveInfos can be used to find out which workers had problems strating up
Revision
47451 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 19 15:30:43 2012 UTC (2 years, 2 months ago) by
ganis
File length: 78523 byte(s)
Diff to
previous 47444
- In TProofServ::ResolveKeywords, add new keywords <rver> for the ROOT version in integer for, e.g. 53403,
and <build> for {version, architecture, compiler} string in the form '53403_linuxx8664gcc_gcc46'.
- Make sure that TProofServ::ResolveKeywords is always called for global package directories; allows to
optimize package building when working with many versions.
Revision
47444 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 19 13:48:59 2012 UTC (2 years, 2 months ago) by
ganis
File length: 78476 byte(s)
Diff to
previous 47238
- In case of non-tty, do not print all progress messages during the merging phase
to avoid filling up the logs with useless info.
- Add member TProof::fTty and getter TProof::IsTty() to centrally control the tty-ness
of the session
Revision
45718 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Aug 28 10:06:59 2012 UTC (2 years, 4 months ago) by
ganis
File length: 77024 byte(s)
Diff to
previous 45632
- Extend parallel dataset verification to PROOF-Lite
- Remove some duplication between TProofServ and TProofLite
Should solve problem observed with stressProof in CDash.
Revision
44683 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 14 09:27:40 2012 UTC (2 years, 7 months ago) by
ganis
File length: 76158 byte(s)
Diff to
previous 44065
First version of the final patch enabling multi-workers on single node using
PROOF-Lite. This is supposed to improved scalability for large number of workers
(>300-400) and is currently available only for non-data driven tasks.
The new setup configuration is enabled by the directive 'xpd.remoteplite 1' in
the xproofd configuration file.
Revision
44046 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 1 11:34:04 2012 UTC (2 years, 8 months ago) by
ganis
File length: 75036 byte(s)
Diff to
previous 43110
Simplify enabling of basic feedback.
- In TDrawFeedback::Feedback, call method Draw() of objects not identified as TH1
derivation. This allows user-defined objects implementing Draw to be displayed
via this utility class (request on the forum).
- In TPerfStats, add new histogram TH1I with the packets being processed per worker;
currently this is 0 or 1 but may change in future; it shows which workers are
still active, which may be useful for debugging at the end of query.
- Add new class TStatsFeedback to display in one canvas the distributions for
the events processed per workers, the packets processed per workers and the
packets being processed per worker.
- In TProof::Process, add support for switches "fb=name1,name2,name3,... " or
"feedback=name1,name2,name3,... " in the option field. This enables feedback
for the specified objects, creating a TDrawFeedback object attached to the session.
Feedback for the specified objects is disabled at the end of the query and the
created TDrawFeedback is destroyed.
The special name 'stats' enables the three feedback histograms required by
TStatsFeedback, and uses a TStatsFeedback instead of TDrawFeedback.
The feedback specs are removed from the option field passed to TProofPlayer::Process.
Revision
43044 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 17 14:34:52 2012 UTC (2 years, 11 months ago) by
ganis
File length: 74457 byte(s)
Diff to
previous 42382
From Taesoo Kim and me.
Add support for selector-by-object processing in PROOF.
The selector object, created and configured locally by the user, is added to the input
list and recuperated from there on the worker machines for processing. Any input list
setting in the selector itself is not streamed but temporarly moved to then standard
input list, so that user can use the selector input list as container of processing
information if they find convenient to do so.
Process(...) methods with the file name argument replaced by 'TSelector *' have
introduced where relevant (TProof, TProofPlayer and their derivatives, TDSet).
Two tests for the new functionality have been added to test/stressProof.cxx (tests 25
and 26).
Revision
42382 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 2 16:50:21 2011 UTC (3 years, 1 month ago) by
ganis
File length: 74315 byte(s)
Diff to
previous 41635
In TProof, add interface to simplify the creation of the performance tree:
- proof->SetPerfTree("<file>")
Set parameters to create the performance tree and saves it to '<file>'
(default <file> is 'perftree.root')
- proof->SavePerfTree("<file>", "<queryref>")
Saves the performance tree of query '<queryref>' (default the last one)
to '<file>' (default 'perftree.root').
The perfomance tree creation is deisble after each query, so it needs to be
enabled each time.
Revision
38819 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 13 10:17:21 2011 UTC (3 years, 9 months ago) by
ganis
File length: 73567 byte(s)
Diff to
previous 38778
Introduce method GetSandbox to parse in a common way the directives defining
the sandbox location in TProof and TProofLite. Add expansion of "." and ".."
(it was not working with input data files).
Revision
38102 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 16 18:26:13 2011 UTC (3 years, 11 months ago) by
ganis
File length: 69884 byte(s)
Diff to
previous 37976
Fix problem with enabling packages with option 'notOnClient' in PROOF-Lite (see forum):
- Implement dedicated TProofLite::FindUniqueSlaves, filling the unique worker list with
the first worker
- In TProof::BuildPackage, always issue the build request if 'notOnClient' is set.
Revision
37976 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 4 11:43:16 2011 UTC (3 years, 11 months ago) by
ganis
File length: 68650 byte(s)
Diff to
previous 36497
In TProofLite::Load
- Extend to this overload support for a comma-separated list of macros (cfr #37942)
In TSlave and related classes
- Add member for the unique session tag
- Fix worker name in TSlaveLite
Revision
34938 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Aug 23 10:22:59 2010 UTC (4 years, 5 months ago) by
ganis
File length: 68378 byte(s)
Diff to
previous 34755
In PROOF-Lite, print a '\n' after the messages about the merging status to avoid
visual screw of up the ROOT prompt at the end of the query.
In standard PROOF this is done automatically by the master.
Revision
33672 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon May 31 16:49:42 2010 UTC (4 years, 7 months ago) by
ganis
File length: 68008 byte(s)
Diff to
previous 33667
In TProofLite:
- Make sure that the default sandbox is always ~/.proof
- Make sure that the package lock file is under gSystem->TempDirectory()
Revision
33634 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri May 28 13:10:42 2010 UTC (4 years, 7 months ago) by
rdm
File length: 67827 byte(s)
Diff to
previous 33562
override TProof::Load() and implement protected method CopyMacroToCache()
(code move from TProofPlayerLite::MakeSelector()).
Revision
33562 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 18 16:28:50 2010 UTC (4 years, 8 months ago) by
ganis
File length: 58687 byte(s)
Diff to
previous 33209
Add method TProof::SetProgressDialog(Bool_t on = kTRUE) to disable the progress dialog
window while preserving graphics from the final actions in Terminate. This may be useful
when X over network is slow. To disable the dialog the call
proof->SetProgressDialog(0)
has to be placed before Process(...).
Revision
33209 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Apr 26 10:27:40 2010 UTC (4 years, 8 months ago) by
ganis
File length: 58646 byte(s)
Diff to
previous 33086
- TDataSetManager, TDataSetManagerFile
- Add options to get a list of all dataset registered names from ScanDataSets
(option kList; the result is a TMap of {TObjString, TObjString} with the second
TObjString empty.
- Support wild cards in ScanDataSet .
- TProof, TProofServ
- Add interface to TDataSetManager::ShowCache/ClearCache (TProof::ShowDataSetCache,
TProof::ClearDataSetCache)
- If a non-default dataset directory was specified and could not be validated, do not
initialize the default dataset, as this could be misleading. An warning is issued
on the client shell.
Revision
33086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 18 18:15:56 2010 UTC (4 years, 9 months ago) by
ganis
File length: 57985 byte(s)
Diff to
previous 33045
- Some changes towards a parallel version of VerifyDataSet:
- Make the lower level of ScanDataSet a static in TDataSetManager and separate out
the part extracting the file meta-information into a dedicated method ScanFile
- In TDSetElement, rename the list of associated files to associated objects, so that
it can be more generally used (it was not yet used for associated files and it
was anyhow supposed to be a list of TObjString's).
- Move the parsing of the recently introduced 'DataSet.SrvMaps' env in TDataSetManager
(instead of TProof). This is more logical and it makes easier to apply the enable
the server mappings during staging requests via VerifyDataSet.
- Add a missing protection in TProof::AssertDataSet potentially causing crashes in the
case a dataset is not found.
Revision
33045 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 15 14:57:32 2010 UTC (4 years, 9 months ago) by
ganis
File length: 58302 byte(s)
Diff to
previous 33042
Add the possibility to remap the server for the files in a dataset. This allows,
for example, to reuse the dataset information for the same files stored in a different
cluster.
The mapping can be defined globally by the cluster administrator using the ROOT env
variable
DataSet.SrvMaps: srv1|map1 srv2|map2 ...
where srv1 are the {protocol,host,port} of the server to be matched, and map1 are the
{protocol,host,port} to be used for those files mapping srv1, and so on.
The matching server string can contain the wildcard '*' in the host field or it can
be empty, in which case the replacement is done for all the dataset files.
If the '|' character is missing the whole string is assigned to the map part; i.e.
'map1' alone is equivalent to '*|map1' or '|map1' .
For example, the following can be used to relocate the files via 'redir.dom.ain'
DataSet.SrvMaps: root://dserv-*.dom.ain/|root://redir.dom.ain/
The user can change the default mapping(s) on the fly with the parameter PROOF_DataSetSrvMaps:
the above is equivalent to setting
p->SetParameter("PROOF_DataSetSrvMaps", "root://dserv-*.dom.ain/|root://redir.dom.ain/")
before starting a query.
Revision
33027 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 15 12:28:09 2010 UTC (4 years, 9 months ago) by
ganis
File length: 57879 byte(s)
Diff to
previous 32900
Do not add things to the TQueryResult object in case the query was aborted
or failed to start. Fixes a segv when processing a dataset where none of
the files had been validated.
Revision
32897 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 8 06:39:24 2010 UTC (4 years, 9 months ago) by
ganis
File length: 57742 byte(s)
Diff to
previous 31961
In TProofLite:
- use shorter name for the unix socket path; issue an error if the total length is
larger then 104 (sort of universal hardcoded limit); add variable ProofLite.SockPathDir
to control the directory where it is created.
All this should solve a problem with the crazy temp dirs on Mac.
- make sure that ~TProofLite is called when issuing ".q"
Revision
31961 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jan 5 09:52:13 2010 UTC (5 years ago) by
ganis
File length: 56870 byte(s)
Diff to
previous 31744
A few optimizations/fixes in interpreting the valgrind config string in PROOF-Lite:
- Interpret "valgrind" as "valgrind=workers#2" (instead of "valgrind=master").
- Make sure that the '#<nworkers>' specification is not ignored.
- Print a warning and ignore "valgrind=master", which does not make much sense.
Revision
31719 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 9 10:31:17 2009 UTC (5 years, 1 month ago) by
ganis
File length: 56845 byte(s)
Diff to
previous 31598
Fix issues found by Coverity:
- #7694, #7376: remove unused protected function TList *TProofLite::GetDataSet givign two defects
- #7375: use snprintf instead of sprintf in NotifyStartup
- #6230, #6229, #425, #424: consistency of pointer checking
Revision
31598 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Dec 7 15:21:47 2009 UTC (5 years, 1 month ago) by
rdm
File length: 57955 byte(s)
Diff to
previous 30421
From Gerri:
Fix a few pending issues in PROOF-Lite:
- Source the env file before running 'proofserv'; this allows to run valgrind
or other wrapper scripts (e.g. experiment-specific environment initializers)
before proofserv, as in normal PROOF
- Fix a problem with Load()
- Add the possibility for the administrator to control the number of workers
for PROOF-Lite or eventually to forbid the use of it (request on the forum).
The rootrc variable ProofLite.MaxWorkers is read out of /etc/system.rootrc
and cannot be overwritten by users.
Improve handling of Ctrl-C:
- Fix a problem in TMonitor affecting the effectiveness of interrupts.
- Do not send the results from workers to masters if the query has been
aborted.
Miscellanea:
- Make sure that the size of the cache sent back to the packetizer by the
workers is meaningful in case the defaults are used (i.e. not -1).
- Make the current processed element available to selectors via the
input list; this allows to write a more simple and effective packetizers
for tasks where the entry is a file (e.g., dataset generation in the
benchmark suite).
Revision
30421 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Sep 24 17:20:09 2009 UTC (5 years, 4 months ago) by
ganis
File length: 55679 byte(s)
Diff to
previous 29490
Some improvements in the dataset manager:
- Introduce the possibility to 'trust' the information provided by the user in the TFileCollection
object sent over upon registration (by default the status bits and the meta information are reset).
- Give the admin has the possibility to configure the dataset manager in such a way that the request
for trusting the information is ignored. A new bit TDataSetManager::kTrustInfo controls that and the
configuration option controlling it is called "Ti" (for 'Trust Info'; equivalent of "Av" for 'Allow Verification').
The current default is that the bit is set.
- Remove the unused bit TDataSetManager::kAllowStaging (staging is controlled externally, not by the
dataset manager)
- Fix a problem with browsing the existing datasets duplicating the same information in some cases.
Revision
29133 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 22 12:28:50 2009 UTC (5 years, 7 months ago) by
brun
File length: 55668 byte(s)
Diff to
previous 28746
From Gerri:
Rename TProofDataSetManager and TProofDataSetManagerFile to TDataSetManager and TDataSetManagerFile.
These classes are managers of TFileCollection which do not have anything strictly dependent on PROOF.
So far they were only used internally by PROOF, but they are going to be exposed to more public for generic
file collection handling. It is the right time to give them a more proper name.
Revision
28746 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu May 28 13:03:29 2009 UTC (5 years, 7 months ago) by
ganis
File length: 55748 byte(s)
Diff to
previous 28741
Cleanup of the TProofMgr fucntions DetachSession and ShutdownSession, and better
handling of the internal list registration, to fix potential segvs when reopening
a PROOF session inside the same ROOT session.
Now, for both normal PROOF and PROOF-Lite, it should be possible to close a
session handled by the TProof object 'p' with any of the following:
root[] delete p
or
root[] TProof::Mgr("<server>")->ShutdownSession(p)
or
root[] TProof::Mgr("<server>")->ShutdownSession(id_p)
(where id_p is the local session id given by TProof::Mgr("<server>")->QuerySessions())
and re-open a session on the same server right after.
Other fixes / changes:
- New auxilliary methods:
- TProofMgr::DiscardSession(TProof *) to remove a session from the internal
list of a manager;
- TProofDesc *TProofMgr::GetProofDesc(TProof *) to quickly get the TProofDesc
container associated with a session by object
- Make sure that the default port is always 1093 (in a couple of places it was 1094!)
- Improve some comments
Revision
28395 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 30 09:27:53 2009 UTC (5 years, 8 months ago) by
ganis
File length: 55652 byte(s)
Diff to
previous 27444
Implement two new functions:
1. TProof::SetDataSetTreeName(<dataset>,<treename>)
Sets/changes the default tree name in the dataset (TFileCollection)
2. TProof::ExistsDataSet(<dataset>)
Returns kTRUE if 'dataset' exists.
Fix a problem with attaching to an existing session in TProof::Open .
Revision
26399 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 24 01:44:18 2008 UTC (6 years, 2 months ago) by
rdm
File length: 54042 byte(s)
Diff to
previous 26388
From Gerri:
- huge simplification for PROOF-Lite, as the chain is local there is no need
to set special arguments in SetProof() to get the Tree header. The client
can just use the TChain directly to get the tree header and GetEntries().
- fix for TChain::Draw() on PROOF-Lite when doing repeated Draw() queries.
Revision
26382 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Nov 22 17:18:42 2008 UTC (6 years, 2 months ago) by
ganis
File length: 52618 byte(s)
Diff to
previous 26268
Fixes for Draw queries in PROOF-Lite
- Do not call Finalize (it is not needed and was giving a warning about an empty query)
- Enable the progress bar
Revision
26268 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 18 16:27:09 2008 UTC (6 years, 2 months ago) by
ganis
File length: 48615 byte(s)
Diff to
previous 26160
- Fix a few problems with the input data machinery (thanks to C. Cheshkov)
- Use the SingeKey option to write out containers
- Send the input data after the call to Begin() to catch up also the objects
possibly added in there
- Fully enable the input data machinery also for PROOF-Lite; this required to
move some calls from TProofServ to TProof to avoid duplications.
- Additions to test/stressProof.C
- Add a test for the input data machinery
- Add some progress info for the tests involving processing
Revision
25918 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Wed Oct 22 15:00:04 2008 UTC (6 years, 3 months ago) by
ganis
File length: 46541 byte(s)
First version of PROOFLite, a 2-tier realization of PROOF intended for multi-core
machines. In PROOFLite the client start directly the workers; no daemon is required.
Currently this works on Unix only. Some issues with the internal connection still need
to be sorted out on Windows.
To start a session just use
TProof::Open("") or TProof::Open("lite")
From there on everything should be as in normal PROOF. To start a standard PROOF
session (i.e. via daemons) on the localhost use TProof::Open("localhost").
(The default meaning of "" in TProof::Open() is controlled via the rc value
'Proof.LocalDefault').
To run stressProof with PROOFLite pas the argument 'lite':
./stressProof lite
By default, PROOFLite sets the number of workers equal to the number of cores.
The number of workers is controlled by the rc variable 'ProofLite.Workers'; it can
also be set on the fly with the option "workers=<n>", e.g. TProof::Open("lite?workers=2").
This patch also introduces the rc variable 'Proof.Sandbox' to control the local location
of the sandboxes. The default has been changed from "~/proof" to "~/.proof" to avoid
interferences with possible users' working areas. The sandbox contains the directory
'packages' (with the clients packages; used by standard PROOF and PROOF Lite),
and some directories used only by PROOFLite: 'cache' (binary cache) and 'datasets' with
the datasets. The location of these directories is controlled by the rc variables
'Proof.PackageDir', 'Proof.CacheDir' and 'Proof.DataSetDir'.
The working directory for PROOFLite is in the form workdir=<sandbox>/path-to-current-dir where
~/path/to/current/dir is the directory from where the session in launched. The directory
<workdir> contains the session working dirs (in the form 'session-<host>-<time>-<pid>'),
with the symlink 'last-lite-session' pointing to the last started session, and the 'queries'
directory, with the results of queries started from the current directory.
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
