Log of /trunk/proof/proofplayer/src/TPerfStats.cxx
Parent Directory
Revision
48046 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 14 15:35:16 2012 UTC (2 years, 1 month ago) by
rdm
File length: 29620 byte(s)
Diff to
previous 44967
From Ioan:
Replace the trunk code that uses the deprecated TH1::kCanRebin bit with
the new methods TH1::CanExtendAllAxes() and TH1::SetCanExtend() - see r47909,
r48012 and r48031. The old TH1::kCanRebin bit did not really have much to
do with histogram rebinning, but rather with extending the limits of the
histogram's axes (which can cause a rebin, but only in some cases). As such,
the word "Rebin" has been replaced with "Extend" for the new methods.
Rename TH1::RebinAxis() to TH1::ExtendAxis() for the reasons stated above.
The TH1::RebinAxis() method will be kept in v6-00 for backward compatibility
with an obsolete warning and will be removed in v6-02. The current
implementation simply forwards the call to TH1::ExtendAxis().
Revision
44683 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 14 09:27:40 2012 UTC (2 years, 7 months ago) by
ganis
File length: 29543 byte(s)
Diff to
previous 44046
First version of the final patch enabling multi-workers on single node using
PROOF-Lite. This is supposed to improved scalability for large number of workers
(>300-400) and is currently available only for non-data driven tasks.
The new setup configuration is enabled by the directive 'xpd.remoteplite 1' in
the xproofd configuration file.
Revision
44046 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 1 11:34:04 2012 UTC (2 years, 8 months ago) by
ganis
File length: 29450 byte(s)
Diff to
previous 42246
Simplify enabling of basic feedback.
- In TDrawFeedback::Feedback, call method Draw() of objects not identified as TH1
derivation. This allows user-defined objects implementing Draw to be displayed
via this utility class (request on the forum).
- In TPerfStats, add new histogram TH1I with the packets being processed per worker;
currently this is 0 or 1 but may change in future; it shows which workers are
still active, which may be useful for debugging at the end of query.
- Add new class TStatsFeedback to display in one canvas the distributions for
the events processed per workers, the packets processed per workers and the
packets being processed per worker.
- In TProof::Process, add support for switches "fb=name1,name2,name3,... " or
"feedback=name1,name2,name3,... " in the option field. This enables feedback
for the specified objects, creating a TDrawFeedback object attached to the session.
Feedback for the specified objects is disabled at the end of the query and the
created TDrawFeedback is destroyed.
The special name 'stats' enables the three feedback histograms required by
TStatsFeedback, and uses a TStatsFeedback instead of TDrawFeedback.
The feedback specs are removed from the option field passed to TProofPlayer::Process.
Revision
41837 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 9 16:04:19 2011 UTC (3 years, 2 months ago) by
rdm
File length: 27977 byte(s)
Diff to
previous 40758
From Filimon Roukoutakis and me:
Full port to AIX 7.1 and xlC Version: 09.00.0000.0017.
This port is complete (./configure --all; make) works (minus some minor
plugins that were not installed). The tutorials/benchmarks.C and test/* works.
For the list of linux-tool packages that need to be installed see:
http://root.cern.ch/drupal/content/build-prerequisites.
Port made on hardware kindly donated by IBM.
Revision
40699 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Aug 25 10:36:29 2011 UTC (3 years, 5 months ago) by
rdm
File length: 28059 byte(s)
Diff to
previous 40186
From Gerri:
Patch adding support for backend-dependent record formatting of PROOF
monitoring. This is achieved by introducing a new layer, described by
the abstract interface TProofMonSender, with the record format defined
in the backend implemenattion (currently TProofMonSenderML, for MonaLisa,
and TProofMonSenderSQL, for SQL backends).
Currently three types of records are sent: 'summary' (derived from what
was currently posted), 'dataset', with entries per dataset processed
in the query, and 'files', with entries per file processed in the query.
In SQL terms, each of this records corresponds to a different table.
Sending of any of the three records can be toggled independently.
Record format have been versionned, so that previous versions of the
'summary' record can be still sent chosing the corresponding version.
The latest version of 'summary' has been augmented with information
about the number of missing files and the ROOT version.
The headers of the relevant methods in TProofMonSenderSQL and
TProofMonSenderML contain the details about the records.
The Web doc is up-to-date.
The patch also add to TSQMonitoring::SendParameters the possibility to
do a bulk insert. This feature is used by TProofMonSenderSQL when
sending dataset or files information.
Revision
40186 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jul 11 12:00:50 2011 UTC (3 years, 6 months ago) by
ganis
File length: 27737 byte(s)
Diff to
previous 40017
- Record in TStatus also the max memory usage on the master and printed via TStatus::Print;
this allow a quick visualisation of the overall memroy usage at the end of the query:
root [] TStatus *st = (TStatus *) p->GetOutputList()->FindObject("PROOF_Status");
root [] st->Print();
OBJ: TStatus PROOF_Status OK
Max worker virtual memory: 394.90 MB Max worker resident memory: 229.47 MB
Max master virtual memory: 1104.55 MB Max master resident memory: 940.73 MB
- Fix issue in TProofPlayer::HandleHistogram preventing proper histogram cleaning right
after merging when using TH1::Add; histogram were still destroyed at the end of the
query, but there was no memory advantage in TH1::Add wrt TH1::Merge.
Revision
40017 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 27 16:23:01 2011 UTC (3 years, 6 months ago) by
ganis
File length: 27522 byte(s)
Diff to
previous 38709
- In TPerfStats, add 'status' field to the monitor record, an integer with the query exit status
(see TVirtualProofPlayer::EExitStatus (0 OK; 1 stopped; 2 aborted).
- Add exist status also to TStatus, so that it is available in the output list (TPerfStats takes
it from here).
- Send the output list, if non empty, also when the query is aborted, so that the TStatus object
is available also in those cases.
Revision
38180 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Feb 21 14:38:22 2011 UTC (3 years, 11 months ago) by
rdm
File length: 22836 byte(s)
Diff to
previous 38055
From Gerri:
Fix a few issues in SQL PROOF monitoring:
- In TSQLMonitoringWriter::SendParameters, drop ''' around field names in
the INSERT string; also use TString::Format(...) instead of Form(...)
where relevant;
- In TPerfStats:
- call 'proofgroup' instead of 'group' the field with the PROOF group
(interference with the 'group' keyword in SQL);
- add new field 'querytag' VARCHAR(64) with the unique query tag
- in WriteQueryLog fill also the field 'totevents'
- in PacketEvent, add switch to control whether to send te information
to the monitoring system on per packet level (may be too much for SQL).
The switch is called fMonitorPerPacket and it is globally controlled
by the rootrc variable 'Proof.MonitorPerPacket' and at
session level with the parameter PROOF_MonitorPerPacket .
Revision
38055 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 11 22:06:25 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 22084 byte(s)
Diff to
previous 36015
Introduce TVirtualPerfStats::FileUnzipEvent to be able to keep track of the cost of unzipping and use this in TTreePerfStas and TBakset ... This give a good picture of where the time in unzip or in unstreaming
Revision
30953 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 3 08:42:49 2009 UTC (5 years, 2 months ago) by
ganis
File length: 21529 byte(s)
Diff to
previous 29030
If enabled, send monitoring information from the master at each GetNextPacket
(at each call of TPerfStat::PacketEvent) to allow extrnal real-time progress
monitoring.
Revision
25918 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 22 15:00:04 2008 UTC (6 years, 3 months ago) by
ganis
File length: 19899 byte(s)
Diff to
previous 22635
First version of PROOFLite, a 2-tier realization of PROOF intended for multi-core
machines. In PROOFLite the client start directly the workers; no daemon is required.
Currently this works on Unix only. Some issues with the internal connection still need
to be sorted out on Windows.
To start a session just use
TProof::Open("") or TProof::Open("lite")
From there on everything should be as in normal PROOF. To start a standard PROOF
session (i.e. via daemons) on the localhost use TProof::Open("localhost").
(The default meaning of "" in TProof::Open() is controlled via the rc value
'Proof.LocalDefault').
To run stressProof with PROOFLite pas the argument 'lite':
./stressProof lite
By default, PROOFLite sets the number of workers equal to the number of cores.
The number of workers is controlled by the rc variable 'ProofLite.Workers'; it can
also be set on the fly with the option "workers=<n>", e.g. TProof::Open("lite?workers=2").
This patch also introduces the rc variable 'Proof.Sandbox' to control the local location
of the sandboxes. The default has been changed from "~/proof" to "~/.proof" to avoid
interferences with possible users' working areas. The sandbox contains the directory
'packages' (with the clients packages; used by standard PROOF and PROOF Lite),
and some directories used only by PROOFLite: 'cache' (binary cache) and 'datasets' with
the datasets. The location of these directories is controlled by the rc variables
'Proof.PackageDir', 'Proof.CacheDir' and 'Proof.DataSetDir'.
The working directory for PROOFLite is in the form workdir=<sandbox>/path-to-current-dir where
~/path/to/current/dir is the directory from where the session in launched. The directory
<workdir> contains the session working dirs (in the form 'session-<host>-<time>-<pid>'),
with the symlink 'last-lite-session' pointing to the last started session, and the 'queries'
directory, with the results of queries started from the current directory.
Revision
20758 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Nov 11 21:29:49 2007 UTC (7 years, 2 months ago) by
ganis
Original Path:
trunk/proofplayer/src/TPerfStats.cxx
File length: 18814 byte(s)
Diff to
previous 19826
Several small fixes and few additions imported from 'branches/dev/proof':
- XrdProofdProtocol:
+ Fix a problem in CleanuProofServ sometimes preventing proper cleanup
+ In ReadBuffer, do not trigger an error on empty files
+ Add missing protection against an empty string while setting the ROOT version
- XrdProofdManager:
+ Define the list of unique nodes to avoid broadcasting twice to the same node
- XrdProofWorker: improve matching technique
- TProof: add setter and getter for Int_t parameter
- TProofServ:
+ do not flush the log file in SendAsynMessage: this recent change
made the logs "disappear" form the ProgressDialog
+ Activate control on the max number of query result kept
- TPacketizerUnit: fix a problem with filling the histof for PerfStats
- TProofPlayer: call Begin() also during selecter re-initialization
Revision
18956 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 6 10:03:48 2007 UTC (7 years, 7 months ago) by
rdm
Original Path:
trunk/proofplayer/src/TPerfStats.cxx
File length: 15258 byte(s)
Diff to
previous 18336
write PROOF query log info in a SQL DB. This info can be used for quota
and user priority calculation. The data is stored in a table "proofquerylog"
with the definition:
CREATE TABLE proofquerylog (
id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
user VARCHAR(128) NOT NULL,
group VARCHAR(128),
begin DATETIME,
end DATETIME,
walltime INT,
cputime FLOAT,
bytesread BIGINT,
events BIGINT
)
This information will only be stored if in the xpd.cf file the option
xpd.querylogdb mysql://lxb6041.cern.ch/proof alice alicaf
is specified.
Revision
16775 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 15 17:45:55 2006 UTC (8 years, 2 months ago) by
rdm
Original Path:
trunk/proofplayer/src/TPerfStats.cxx
File length: 13136 byte(s)
Diff to
previous 15294
From Gerri and Bertrand:
Fixes and new features developed in preparation for the Intel quad-core
launch demo:
- Add instantaneous event and MB processing rates in the Progress signal
and make necessary modification to exploit it in the Progress Dialog.
- Add possibility to change the Progress and Feedback periods
- A number of small improvements and bug fixes
Revision
14745 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 19 08:22:26 2006 UTC (8 years, 9 months ago) by
rdm
Original Path:
trunk/proofplayer/src/TPerfStats.cxx
File length: 12089 byte(s)
Diff to
previous 12722
Change the TError.h macros:
Assert -> R__ASSERT
Check -> R__CHECK
Change the TCollection.h macro:
ForEach -> R__FOR_EACH
This to avoid potential problems due too trivial macro names.
The old macros will be removed in the next release. Currently
they will print out warning messages with the advice to move
to the new macro names.
Revision
12722 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Sep 16 08:48:39 2005 UTC (9 years, 4 months ago) by
rdm
Original Path:
trunk/proofplayer/src/TPerfStats.cxx
File length: 12086 byte(s)
Diff to
previous 11294
From Gerri:
Big patch supporting query queueing on the PROOF master, interogation and
interaction with the queue. Retrieval, finalization and archival of query
results and multiple PROOF sessions. Some other fixes:
- TMacro.h, .cxx
add method Checksum to get the MD5 of the current context
(needed for optimized selector reinitialization)
- Changes needed to create and fill the list of data sets (TROOT,
TDSet, TChain, TChainProof)
- A new method TSelector::IsStandardDraw() to check if a selector
is for standard draw actions
- TEventIter: fix for the missing last update (see above)
- Few protections in TPacketizers destructors
- Fix for code conventions in TPerfStats (the other will follow)
- Occasional missing last update of the progress bar (systematically
reproduced with SetParallel(1)): all this part is quite involved
and I have found a solution which does not harm but perhaps is not
the most elegant.
- SetParallel(0): the progress bar was not working at all in such a
case, since no packetizer is involved. Still it is useful to see
that things are not stuck, so I have added some progress messages
there too. The problem is that I have not found a way to get the total
number of entries from a data set without doing all the machinery
done in the packetizer.
Revision
11047 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Feb 7 18:02:37 2005 UTC (9 years, 11 months ago) by
rdm
Original Path:
trunk/proofplayer/src/TPerfStats.cxx
File length: 11868 byte(s)
Diff to
previous 10608
From Maarten and Kristjan Gulbrandsen:
Here is the patch with the restructured PROOF code. It parameterizes
PROOF startup, currently implementing "Standard", Condor, MLM and PEAC.
It can easily be extended with other "TProof" classes, e.g. AliEn Proof.
Or what ever next Grid implementation or scheduler comes along.
The MLM, Multi Level Master, code uses a basic packetizer, based on a
MSD (massstorage domain) tag. In MLM mode we have multi-tier client
server setup, like:
client -> super master -> sub master 0 -> slave[0..N]
...
-> sub master M -> slave[0..N]
It also contains a large number of smaller fixes. Credit for most of
the work goes to Kris!
We have tested this on multipe platforms (RH7.3, RH9, SL3) and in
multiple configurations. Should be stable.
Revision
10608 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 24 07:41:32 2004 UTC (10 years, 2 months ago) by
brun
Original Path:
trunk/proofplayer/src/TPerfStats.cxx
File length: 11112 byte(s)
Diff to
previous 9182
From Maarten Ballintijn and Kristj�n Gulbrandsen
a patch and files that complete the benchmark package.
I strongly recommend using this for demos rather then the H1 example.
The README file should walk you through a complete demo, which should
not take more then maybe ten minutes or so. After you have some
data files generated, you can show significant parts of the sytem
in a minute or two.
I added a feedback request api to TProof, this simplifies things
a lot from a usability perspective. I also modified TDrawFeedback
to use gProof by default, again simplifying things.
Kris wrote the support scripts and fixed the benchmark
scripts to read the full Events.
Please try it out and have fun!
It is as simple as this (I use log level 2 to slow things
down a bit :-))
gROOT->Proof()
.L make_tdset.C
TDSet *d= make_tdset("/data1/tmp",2)
TDrawFeedback fb
gProof->AddFeedback("PROOF_ProcTimeHist")
gProof->AddFeedback("PROOF_LatencyHist")
gProof->AddFeedback("PROOF_EventsHist")
gProof->UploadPackage("event.par")
gProof->EnablePackage("event")
gEnv->SetValue("Proof.StatsHist",1);
gProof->SetLogLevel(2)
d->Process("EventTree_Proc.C")
Revision
9182 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Sun Jun 13 16:26:36 2004 UTC (10 years, 7 months ago) by
rdm
Original Path:
trunk/proofplayer/src/TPerfStats.cxx
File length: 10396 byte(s)
From Maarten and Kristjan Gulbrandsen:
PROOF and file I/O performance monitoring framework. The TVirtualPerfStats
abstract class allows adding monitoring events in base classes (TFile and
TNetfile). The performance monitoring info allows detailed study of PROOF
performance (which files accessed by which slaves, latency, read/write times,
packet size, etc.). The result is send back to the client in the form of a
memory based TTree.
Remove TProofStats.h and TProofStats.cxx, .d, .o from proof/inc and src.
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
