Log of /trunk/proof/proof/src/TProofServ.cxx
Parent Directory
Revision
48992 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Mar 28 15:26:26 2013 UTC (21 months, 3 weeks ago) by
rdm
File length: 263137 byte(s)
Diff to
previous 48226
From Lifeng Sun:
The attached patchset fixes a bunch of typo in the source:
0001-succes-success.patch
0002-preceed-preced.patch
0003-informations-information.patch
0004-childs-children.patch
0005-avaliable-available.patch
0006-writeable-writable.patch
0007-comand-command.patch
0008-unkown-unknown.patch
0009-wierd-weird.patch
0010-wheter-whether.patch
0011-unecessary-unnecessary.patch
0012-splitted-split.patch
0013-registerd-registered.patch
0014-recieve-receive.patch
0015-processsing-processing.patch
0016-ouput-output.patch
0017-mutiple-multiple.patch
0018-lenght-length.patch
0019-interupted-interrupted.patch
0020-independant-independent.patch
0021-inconsistant-inconsistent.patch
0022-expresion-expression.patch
0023-explicitely-explicitly.patch
0024-enviroment-environment.patch
0025-deafult-default.patch
0026-continous-continuous.patch
0027-completly-completely.patch
0028-commited-committed.patch
0029-choosen-chosen.patch
0030-backgroud-background.patch
0031-auxilliary-auxiliary.patch
0032-authentification-authentication.patch
0033-appropiate-appropriate.patch
0034-an-other-another.patch
0035-environement-environment.patch
0036-targetting-targeting.patch
0037-suppported-supported.patch
0038-paramater-parameter.patch
Revision
48078 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Dec 17 16:48:31 2012 UTC (2 years, 1 month ago) by
rdm
File length: 262856 byte(s)
Diff to
previous 48063
From Dario:
Fixed build error of TDataSetManagerAliEn on Win32, and some warnings
concerning overloading/overriding virtual functions. Also silenced a
PROOF warning by default when no dataset stager repository is defined.
Revision
48060 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Dec 15 18:53:16 2012 UTC (2 years, 1 month ago) by
ganis
File length: 262833 byte(s)
Diff to
previous 47573
From Dario: patch introducing an improved dataset management model where the PROOF (ROOT)
dataset manager is a light frontend to the experiment file catalogs; TDataSetManagerFile
is still used as local cache of the experiment information or to store the work-in-progress
status of the dataset manager daemon. This model is expected to solve the scalability issues
observed currently at AAFs.
The patch includes the new class TDataSetManagerAliEn with the first concrete implementation
of experiment catallog interface. The equivalent for ATLAS will follow.
Revision
47528 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 21 12:47:27 2012 UTC (2 years, 2 months ago) by
ganis
File length: 258390 byte(s)
Diff to
previous 47451
In TProof::EnablePackage, add possibility to choose the level of version compatibility check
to decide whether to re-build the package or not. Three options are possible: 'off', no check
with possible failure at TSystem::Load, 'on' check the ROOT version, 'svn' check the ROOT version
and the SVN revision (default is 'on'). The choice can be done via a special option to the
option string('chkv=off|on|svn' or 'checkversion=off|on|svn') or via the ROOTrc variable
'Proof.Package.CheckVersion' .
Revision
47451 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 19 15:30:43 2012 UTC (2 years, 2 months ago) by
ganis
File length: 257902 byte(s)
Diff to
previous 46878
- In TProofServ::ResolveKeywords, add new keywords <rver> for the ROOT version in integer for, e.g. 53403,
and <build> for {version, architecture, compiler} string in the form '53403_linuxx8664gcc_gcc46'.
- Make sure that TProofServ::ResolveKeywords is always called for global package directories; allows to
optimize package building when working with many versions.
Revision
46878 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 29 16:53:40 2012 UTC (2 years, 2 months ago) by
rdm
File length: 256602 byte(s)
Diff to
previous 45740
Fix __crashreport_info__ declaration for APPLE that would previously cause
a multiple defined symbol error when static linking.
Revision
45740 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 29 16:17:09 2012 UTC (2 years, 4 months ago) by
ganis
File length: 256583 byte(s)
Diff to
previous 45718
- Add support for data dirs on external servers, fully honouring URL options; this allows to
have usable paths registered in the final TFileCollection objects. One of the addressed use-case
is when local directories need to be accessed via the local xrootd server to preserve the namespace
enforced by xrootd; in such a case something like 'root://locahost//datadir?remote=1' needs to be
specified; the option 'remote=1' ensures that the final registered URL is still in 'root://' form
with 'localhost' replaced by the FQDN of the machine.
- Move the 'tests' argument to the stressProof function in the second position, to make it
more user friendly to run a specific test in the ROOT shell; CLI behevior is unchanged.
- In ProofSimple, add function to plot the histograms also in the case a dataset is created with the
worker files; the file are opened and the histograms merged on the fly.
- Fix potential segv at xproofd startup due to a double fclose in XrdProofdManager::ValidateLocalDataSetSrc
- Remove some redundant printing in TProof when handling output files.
- Make sure that when running TSelVerifyDataSet the save-to-file functionality is switched-off .
Revision
45718 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Aug 28 10:06:59 2012 UTC (2 years, 4 months ago) by
ganis
File length: 256497 byte(s)
Diff to
previous 45656
- Extend parallel dataset verification to PROOF-Lite
- Remove some duplication between TProofServ and TProofLite
Should solve problem observed with stressProof in CDash.
Revision
45367 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Aug 3 09:22:00 2012 UTC (2 years, 5 months ago) by
ganis
File length: 256157 byte(s)
Diff to
previous 45348
In TProofOutputFile, add a status bit to indicate that the file is the result of the automatic
save-to-file output handling. This allows a better interplay between client-side and server-side
settings.
In TProof::HandleOutputOptions, fix issue with local object saving.
Revision
45348 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Aug 2 12:23:54 2012 UTC (2 years, 5 months ago) by
ganis
File length: 255999 byte(s)
Diff to
previous 45283
Consolidation of the automatic output saving functionality
- Fix an issue with saving output from memory on the master
- Remove some duplications and fix several small bugs
Revision
45283 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jul 30 10:42:38 2012 UTC (2 years, 5 months ago) by
ganis
File length: 254158 byte(s)
Diff to
previous 45029
Patch automatizing the usage of the file-based technology to handle outputs.
After this patch PROOF users can define as an option to Process an output file for their results
and choose if the merging goes via file or in memory. User can also decide to save theit results
in a set of files identified by a name (dataset).
Administrators can force merging-via-file or can make it conditional to the resident memory exceeding
a given threshold.
The new options are documented at
http://root.cern.ch/drupal/content/handling-outputs
Since TProofOutputFile is now used by TProof, the class has been moved from proof/proofplayer
(libProofPlayer) to proof/proof (libProof).
The selector tutorials/proof/ProofSimple.h,.C has been modified to allo testing the new options,
and a new test has been added to stressProof.
Revision
44507 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 4 12:30:41 2012 UTC (2 years, 7 months ago) by
axel
File length: 253540 byte(s)
Diff to
previous 44065
Remove
using namespace std;
from Riostream.h, which has huge consequences for all of ROOT.
Riostream.h is now a simple wrapper for fstream, iostream, iomanip for backward compatibility; Riosfwd.h simply wraps iosfwd.
Because of templates and their inline functions, Riostream.h needed to be included in headers, too (e.g. TParameter.h), which violated the assumption that Riostream.h is not exposing its using namespace std to headers.
ROOT now requires R__ANSISTREAM, R__SSTREAM, which does not change the set of supported compilers.
Without "using namespace std", several identifiers are now prefixed by std::; e.g. roofit/* source files now have a using namespace std to keep their coding style.
TFile::MakeProject() now generates "using namespace std" to convert the CINT-style class names into C++ ones.
Revision
44010 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 29 23:07:48 2012 UTC (2 years, 8 months ago) by
ganis
File length: 252783 byte(s)
Diff to
previous 43567
Several optimizations in the low level PROOF event loop (TProofPlayer::Process):
- Per-event calls to TEventIter::GetNextEvent are replaced by per-packet calls
to a new function TEventIter::GetNextPacket and a for loop over the returned
range;
- Progress information is filled in on per-packet pace just before asking for
a new packet, instead of being filled for each event;
- In TProofProgressStatus, the relevant setters in were inlined, and the (expensive)
calls to SetLastUpdate() reduced to the minimum required;
- Status of objects is checked using bits instead of functions calls;
- Check whether to dump memory info is done before calling CheckMemUsage.
These optimizations allow to reduce dramatically the overhead introduced by the
operations PROOF needs to perform during the event loop. A measurement of the
overhead can be obtained from a very light computational task, for example,
generating one random number and filling one histogram; executing this task
within a PROOF-Lite session with 1 worker now takes only 1.8 times the time
required by a straight loop in the parent ROOT session; the same number before
was about 13.
The patch also introduces a couple of small changes enabling the use of
TProofPlayerLocal to process selectors in the local session using the PROOF
iteration mechanism; this is very useful for optimizations studies.
Finally, a couple of missing protections were added in TProofPlayer.
Revision
43567 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Mar 30 08:59:16 2012 UTC (2 years, 9 months ago) by
ganis
File length: 252280 byte(s)
Diff to
previous 43108
In TDataSetManger::ScanFile :
- Open the file twice (the first time as 'raw') only if it is an archive (to get
the total size); this was tuned on the ALICE case, where with an xrootd back-end
the double opening was not critical; for other backends the penalty may be significant,
so it should be done only when strictly needed.
- Do not hardcode 'TIMEOUT=5' option by default when opening files; this appears to
create some problem on some back-ends, so it needs to be configurable.
In TProofServ
- In RegistersDataSets, remove the temporary dataset tags from the output list after
processing: they are not needed and generate warnings about duplicated names.
- In ProcessNext, add missing protection in the case of query abort.
Revision
43107 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 24 12:45:22 2012 UTC (2 years, 10 months ago) by
ganis
File length: 252031 byte(s)
Diff to
previous 43074
From B. Butler:
Add the possibility to force submerging at node level, i.e. one submerger per physical machine.
In this way the network traffic can be minimized, for example when merging large output files.
The new feature is enabled by setting the Int_t parameter 'PROOF_MergersByHost' (or the directive
'Proof.MergersByHost') to a non-null value.
Revision
43074 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Feb 21 17:26:56 2012 UTC (2 years, 11 months ago) by
ganis
File length: 251504 byte(s)
Diff to
previous 43069
- Properly handle the error status code in TProof::SendFile.
- Make sure that errors from TProofServ::ReceiveFile are properly transmitted back.
Revision
43044 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 17 14:34:52 2012 UTC (2 years, 11 months ago) by
ganis
File length: 250853 byte(s)
Diff to
previous 42882
From Taesoo Kim and me.
Add support for selector-by-object processing in PROOF.
The selector object, created and configured locally by the user, is added to the input
list and recuperated from there on the worker machines for processing. Any input list
setting in the selector itself is not streamed but temporarly moved to then standard
input list, so that user can use the selector input list as container of processing
information if they find convenient to do so.
Process(...) methods with the file name argument replaced by 'TSelector *' have
introduced where relevant (TProof, TProofPlayer and their derivatives, TDSet).
Two tests for the new functionality have been added to test/stressProof.cxx (tests 25
and 26).
Revision
42882 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Feb 6 11:10:21 2012 UTC (2 years, 11 months ago) by
ganis
File length: 249086 byte(s)
Diff to
previous 42761
From Sangsu Ryu and me.
Add support for paralell dataset verification. This is implemented via a dedicated
TSelector (TSelVerifyDataSet) which is run over the list of files in the dataset via
TPacketizerFile. The file order is preserved using the recently introduced index in
TFileInfo.
Parallel verification is the now the default. For test and comparison purposes it is
still possible to verify datasets serially by specifying 'S' in the option field of
VerifyDataSet and RegisterDataSet .
To avoid/reduce code duplication some of the static functions in TDataSetManager have
refactorized. The overall behavior should be unchanged.
Revision
42761 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jan 23 12:55:12 2012 UTC (3 years ago) by
ganis
File length: 245758 byte(s)
Diff to
previous 42495
- Fix problem with TProof::Load reported by Gordon Watts in the PROOF forum.
The additional files were not copied in the master sandbox but left in the
cache. A workaround for backward compatibility has also been implemented.
Revision
42458 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Dec 10 15:56:49 2011 UTC (3 years, 1 month ago) by
rdm
File length: 245232 byte(s)
Diff to
previous 42415
From Gerri:
Add support for a command line test run of 'proofserv'. This is useful
to test that the environment is setup correctly.
Revision
42415 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Dec 6 16:08:59 2011 UTC (3 years, 1 month ago) by
ganis
File length: 244785 byte(s)
Diff to
previous 42381
In TProofServ::SendStatistics:
For masters, do not overwrite the realtime with the sum of real times
on the workers, as correctly done for the CPU time. Solves a timing
problem observed in stressProof when using TProof::GetRealTime().
Revision
42381 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 2 16:39:58 2011 UTC (3 years, 1 month ago) by
ganis
File length: 244825 byte(s)
Diff to
previous 41891
Make sure the relevant stopwatch are reset before each query; solve an a problem in the performance tree with the latency time of the first packet
Revision
41788 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Nov 4 17:55:59 2011 UTC (3 years, 2 months ago) by
ganis
File length: 244344 byte(s)
Diff to
previous 41635
The file with input data must stay open during the whole processing,
otherwise retrieval of certain objects will fail (and crash).
Revision
41399 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Oct 14 21:50:52 2011 UTC (3 years, 3 months ago) by
ganis
File length: 240963 byte(s)
Diff to
previous 40923
Remove hardcoded additions of dirname(COMPILER) and of '/bin:/usr/bin:/usr/local/bin'
in front of PATH. These uncontrolled additions could hide specific settings in PATH
and be the source of weird problems appearing in PROOF only.
By default only $ROOTSYS/bin in front of the original PATH. Additions of dirname(COMPILER)
and/or of '/bin:/usr/bin:/usr/local/bin' are controlled by the rootrc ProofServ.BinPaths .
Revision
40912 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Sep 16 17:34:12 2011 UTC (3 years, 4 months ago) by
ganis
File length: 240042 byte(s)
Diff to
previous 40660
Add possibility to skip the checks for the data directories during session startup,
as they may significantly slowdown the startup process is the medium is busy.
In such a case, admins are responsible to create the directories in advance; the
session releated part fo the path is created by the session once up.
Revision
40017 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 27 16:23:01 2011 UTC (3 years, 6 months ago) by
ganis
File length: 238041 byte(s)
Diff to
previous 39878
- In TPerfStats, add 'status' field to the monitor record, an integer with the query exit status
(see TVirtualProofPlayer::EExitStatus (0 OK; 1 stopped; 2 aborted).
- Add exist status also to TStatus, so that it is available in the output list (TPerfStats takes
it from here).
- Send the output list, if non empty, also when the query is aborted, so that the TStatus object
is available also in those cases.
Revision
39696 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 13 21:35:34 2011 UTC (3 years, 7 months ago) by
pcanal
File length: 238087 byte(s)
Diff to
previous 39171
From David Dagenhart:
Implement an option to allow ROOT to use the LZMA compression
algorithm to compress data instead of the ZLIB compression algorithm.
LZMA compression typically results in smaller files, but takes more
CPU time to compress data. To use the new feature, the external XZ
package must be installed when ROOT is configured and built:
Download 5.0.3 from here http://tukaani.org/xz/
and make sure to configure with fPIC:
./configure CFLAGS='-fPIC'
Then the client C++ code must call routines to explicitly request LZMA
compression.
ZLIB compression is still the default.
Setting the Compression Level and Algorithm:
There are three equivalent ways to set the compression level and
algorithm. For example, to set the compression to the LZMA algorithm
and compression level 5.
1. TFile f(filename, option, title);
f.SetCompressionSettings(ROOT::CompressionSettings(ROOT::kLZMA, 5));
OR
2. TFile f(filename, option, title, ROOT::CompressionSettings(ROOT::kLZMA, 5));
OR
3. TFile f(filename, option, title);
f.SetCompressionAlgorithm(ROOT::kLZMA);
f.SetCompressionLevel(5);
These methods work for TFile, TBranch, TMessage, TSocket, and TBufferXML.
The compression algorithm and level settings only affect compression of
data after they have been set. TFile passes its settings to its branches
only at the time the branches are created. This can be overidden by
explicitly setting the level and algorithm for the branch. These classes
also have the following methods to access the algorithm and level for
compression.
Int_t GetCompressionAlgorithm() const;
Int_t GetCompressionLevel() const;
Int_t GetCompressionSettings() const;
If the compression level is set to 0, then no compression will be
done. All of the currently supported algorithms allow the level to be
set to any value from 1 to 9. The higher the level, the larger the
compression factors will be (smaller compressed data size). The
tradeoff is that for higher levels more CPU time is used for
compression and possibly more memory. The ZLIB algorithm takes less
CPU time during compression than the LZMA algorithm, but the LZMA
algorithm usually delivers higher compression factors.
The header file core/zip/inc/Compression.h declares the function
"CompressionSettings" and the enumeration for the algorithms.
Currently the following selections can be made for the algorithm:
kZLIB (1), kLZMA (2), kOldCompressionAlgo (3), and kUseGlobalSetting
(0). The last option refers to an older interface used to control the
algorithm that is maintained for backward compatibility. The following
function is defined in core/zip/inc/Bits.h and it set the global
variable.
R__SetZipMode(int algorithm)
If the algorithm is set to kUseGlobalSetting (0), the global variable
controls the algorithm for compression operations. This is the
default and the default value for the global variable is kZLIB.
Note that the LZMA algorithm will only be available if the LZMA
libraries from the XZ package were available when the ROOT executable
being used was configured and built. If you are building ROOT and want
LZMA then you must do something similar to the following if XZ is not
already installed on your system.
Download XZ version 5.0.3 from http://tukaani.org/xz/
unwind the tarball
cd xz-5.0.3
./configure CFLAGS='-fPIC'
make
ROOT was tested with version 5.0.3 of XZ. It might work with earlier
versions and will probably work with later versions of XZ.
Then either use "make install" to put the library and headers into
/usr/local/* so the scripts building ROOT can find them or use
arguments to the ROOT configure script to point to the proper
directories containing the XZ library and headers. These are the
library and the included header (lzma.h includes other headers in the
lzma subdirectory).
src/liblzma/.libs/liblzma.a
src/liblzma/api/lzma.h
WARNING: Data compressed with the LZMA algorithm cannot be read by
ROOT executables that were not built with LZMA support.
Revision
39074 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 3 09:30:02 2011 UTC (3 years, 8 months ago) by
ganis
File length: 237706 byte(s)
Diff to
previous 38925
From S. Ryu:
- Fix issue affecting the result of subsequent worker activation/deactivation requests when worker
ordinal numbers were not in increasing order
From me:
- Add support for block requests for worker activation/deactivation (much more efficient, because
FindUniqueSlaves is only called once); the list of workers to have their statis changed is comma-
separated.
- Make DeactivateWorker and ActivateWorker to return != 0 when something went wrong, with -2 indicating
that at least one of the worker ordinal numbers in the list is invalid (e.g. '0.134' for a cluster
with 20 workers only).
Revision
38810 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Apr 12 16:22:59 2011 UTC (3 years, 9 months ago) by
ganis
File length: 237181 byte(s)
Diff to
previous 38760
Patch to correctly honour selector abort status settings in PROOF. Currently only
the TSelector::kAbortProcess was handled by stopping processing. In particular
TSelector::kAbortFile was ignored; this recently created some problems in ALICE
with corrupted files, with repeated attempts to read events eventually leading to
bad_alloc exceptions.
This patch also fixes other related issues, in particular with the reporting of the
non-processed {files, events} in the final 'MissingFiles' list. This list should
now account much more precisely of the number of events which could not be processed.
It also fixes a problem with the final update of the progress information affecting
occasionally cases with skipped events.
Revision
38570 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Mar 23 12:05:17 2011 UTC (3 years, 10 months ago) by
ganis
File length: 236464 byte(s)
Diff to
previous 37942
Add support for log file truncation (see Savannah #74730).
Truncation is disabled by default. Enabling is controlled by the rootrc variable
ProofServ.LogFileMaxSize {<bytes>|<kilobytes>K|<megabytes>M|<gigabytes>G}
indicating the max number of bytes. The number can be followed by a {K,M,G} to indicate
KBytes, MBytes or GBytes, respectively. So, for example, the following setting in the
xproofd configuration file sets a limit to 100MBytes:
xpd.putrc ProofServ.LogFileMaxSize 100M
Revision
37942 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 2 14:32:01 2011 UTC (3 years, 11 months ago) by
ganis
File length: 233451 byte(s)
Diff to
previous 37894
- Make sure that the machine names are consistent when working on a local machine
('localhost' or 'localhost.localdomain' are mapped to gSystem->HostName()); solves
possible matching problems in the packetizer.
- In TProof::Load, add the possibility to define a list of files to be sent because
needed by the main macro file. The list is comma-separated and the first file is
considered as the main one. For example
proof->Load("<macropath>/mymacro.C+,<thispath>/thisheader.h,<thatpath>/thatheader.h")
will make sure that the files 'thisheader.h' and 'thatheader.h', needed by 'mymacro.C'
are available in the sandbox on the worker machines. Note that 'thisheader.h' and
'thatheader.h' will be available remotely in the sandbox, as 'mymacro.C'; so they should
be included directly by 'mymacro.C', e.g. '#include "thisheader.h"' .
- In TProofServ, fill the "grand total" message with more blanks, so that no remnants
of the previous message are visible on the screen.
Revision
37894 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jan 27 14:15:03 2011 UTC (3 years, 11 months ago) by
ganis
File length: 233430 byte(s)
Diff to
previous 37493
Add support for option ':lite:' to TProof::GetDataSets . This allows to fill the map
with only the summary information about the datasets (the header of TFileCollections),
significantly increasing the speed and the memory footprint when the number of datasets
is very large. Should solve issue #77303 .
Revision
37396 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 8 13:12:00 2010 UTC (4 years, 1 month ago) by
rdm
File length: 233206 byte(s)
Diff to
previous 36933
From Gerri:
- Fix a problem with the registration of missing files in the 'MissingFiles'
list. The files that could not be open during processing were not properly
registered in the list (only those found missing during validation or
giving problmes durign reading were correctly added).
- Add method TProof::ShowMissingFiles() to facilitate the display of the
list of missing files.
- Add method TProof::GetMissingFiles() to get a TFileCollection (dataset)
with the missing files for further processing.
Revision
36592 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Nov 11 10:43:17 2010 UTC (4 years, 2 months ago) by
ganis
File length: 232620 byte(s)
Diff to
previous 36577
- Terminate a proofserv after it has been stopped for large memory footprint and it has
sent the results back
- Notify information about which object was being merged if a memory exception happens
during merging on the master
- Allow to set the HWM and stop-process fractions via the environment
Revision
36553 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 9 09:55:24 2010 UTC (4 years, 2 months ago) by
ganis
File length: 230674 byte(s)
Diff to
previous 36541
- Save exception info also in the log file (it was just sent to the client)
- Fix an old issue with DeactivateWorker("*") (the session is was terminated because
no worker was active; this call coudl not be used as intermediate step to select a
small number of workers).
Revision
36537 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 8 16:16:41 2010 UTC (4 years, 2 months ago) by
ganis
File length: 230960 byte(s)
Diff to
previous 36526
Patch to improve the diagnostic in case of exceptions. Information about the event and file being
processed at the moment the exception was raised is sent to the client, e.g.
0.5: caught exception triggered by signal '1' while processing dset:'EventTree',
file:'http://root.cern.ch/files/data/event_3.root', event:1 - check logs for possible stacktrace
The patch also fixes a problem with submergers observed when a worker was stopped because above
the memory limits: this worker was established as merger but could not do the work, for obvious
reasons, freezing the session.
Revision
36526 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 8 07:39:11 2010 UTC (4 years, 2 months ago) by
ganis
File length: 228788 byte(s)
Diff to
previous 36512
Add possibility for the to disable the use of submergers when set by default by the
administrator. This is now achieved by setting the parameter PROOF_UseMergers to -1.
Revision
36512 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Nov 5 12:05:11 2010 UTC (4 years, 2 months ago) by
ganis
File length: 228731 byte(s)
Diff to
previous 36475
In TProofServ, TXProofServ:
Move the creation and start of the idle timeout from the end of SetupCommon to
the end of CreateServer, so that the timeout is not active during worker setup.
Fix a problem reported by ALICE.
Revision
36475 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 2 13:23:08 2010 UTC (4 years, 2 months ago) by
rdm
File length: 228729 byte(s)
Diff to
previous 36191
From Gerri:
add the possibility to control the resident and virtual memory of a proofserv
using 'ulimit', which has less limitations and more flexibility than setrlimit.
Revision
34557 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jul 22 15:12:01 2010 UTC (4 years, 6 months ago) by
rdm
File length: 227559 byte(s)
Diff to
previous 34427
fix cases in ROOT code where we would truncate the TTime (to avoid the
new error messages in TTime operator long on 32-bit platforms).
Revision
34427 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jul 15 11:04:05 2010 UTC (4 years, 6 months ago) by
ganis
File length: 227550 byte(s)
Diff to
previous 34388
- Fix problem with transmission of EnablePackage arguments to workers
- Add test in stressProof for package argument setting (two new test packages
added under tutorials/proof)
Should fix problem reported in Savannah #70041 .
Revision
34388 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jul 13 09:20:42 2010 UTC (4 years, 6 months ago) by
ganis
File length: 227298 byte(s)
Diff to
previous 34380
The newly introduced variable to control the use of submergers should be called Proof.UseMergers consistently with the related parameter
Revision
34380 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jul 12 14:12:13 2010 UTC (4 years, 6 months ago) by
ganis
File length: 227190 byte(s)
Diff to
previous 34379
Add support for the ROOTrc variable 'Proof.SubMergers' to trigger the use of submergers for
merging. This has the same meaning of the parameter 'PROOF_UseMergers'. It is provided to
allow AAF administrators to force the use of submergers to reduce the memory footprint.
The setting can be always superseeded by the user setting of the 'PROOF_UseMergers' in the
input list (or of the local Proof.SubMergers setting).
Revision
34379 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jul 12 13:57:12 2010 UTC (4 years, 6 months ago) by
ganis
File length: 226876 byte(s)
Diff to
previous 34311
Package manager:
- Lock the package directory during the execution of the modified SETUP to avoid clashes between
multiple workers on the same machine (see bug #69835); a more refined solution will be included
in the package manager.
- Correctly register the status code of building and loading packages on the workers, and transmit
it to the client (see bug #69836).
Revision
34282 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jul 1 15:37:22 2010 UTC (4 years, 6 months ago) by
ganis
File length: 226652 byte(s)
Diff to
previous 34254
Package manager: fix a problem with enabling multiple packages having the same signature for
SETUP: the first loaded SETUP was used for all the packages. This was introduced by patch r33749.
The macro is now renamed on the fly to <package>_SETUP so to avoid the clash.
Revision
34101 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 24 12:12:59 2010 UTC (4 years, 7 months ago) by
ganis
File length: 225643 byte(s)
Diff to
previous 33824
Following a request from ALICE, implement a timer to terminate idle sessions. The timeout
value is controlled by the variable ProofServ.IdleTimeout (value in seconds). For example,
to set a timeout of 20 mins, enter the following in the xproofd config file:
### Setup an idle timeout
xpd.putrc ProofServ.IdleTimeout 1200
The timer is disabled by default.
Revision
33824 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 10 14:43:40 2010 UTC (4 years, 7 months ago) by
ganis
File length: 222750 byte(s)
Diff to
previous 33749
- Add new directive 'xpd.datadir' to better control the user data directories and their
permission settings. The syntax is 'xpd.datadir <datadir> <opts>' where <datadir> is
the root path for the datadirs (the final datadir will be <datadir>/<group>/<user>) and
<opts> can contain a combination of the following letters: 'g', 'a', 'M', 'W' ["gW"];
the lower case control the write permissions: 'g' group, 'a' everybody; the upper case
control where the data dirs shoudl be created: 'W' on the workers, 'M' on the master,
"WM" everywhere.
- Better handle the removal of empty data directories in TProofServ and derivatives
- In TProofOutputFile:
- extend support for URL options
- correctly take the URL options into account
This should solve a problem observed in the new ALICE CAF.
Revision
33749 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jun 4 13:56:55 2010 UTC (4 years, 7 months ago) by
ganis
File length: 222207 byte(s)
Diff to
previous 33714
Patch implementing a few changes/fixes in the package management
- Add support for arguments in the SETUP function: it is possible now to pass a string
Int_t SETUP(const char *opt)
or a list of objects
Int_t SETUP(TList *optls);
Two interface methods have been added to support this:
Int_t TProof::EnablePackage(const char *package, const char *opt, Bool_t notOnClient);
Int_t TProof::EnablePackage(const char *package, TList *optls, Bool_t notOnClient);
- Make sure the return code from BUILD.sh is properly checked and execution flow stoped on
failure; this required the introduction of a check on the return code of TSystem::ClosePipe().
- Allow building packages in the global package directories if 'write' permissions are available;
this facilitates management of global packages from an authorized account.
Revision
33418 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri May 7 09:44:51 2010 UTC (4 years, 8 months ago) by
ganis
File length: 216163 byte(s)
Diff to
previous 33369
- Make sure that information used internally in the option field on the master
(about the aclic option) is not sent to workers while issuing the Process message.
Fixes issue reported on the Forum.
Revision
33364 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 4 08:04:17 2010 UTC (4 years, 8 months ago) by
ganis
File length: 215780 byte(s)
Diff to
previous 33357
Fine tune monitoring via syslog/local5:
- add additional level to increase the granularity; available levels
0 no syslog logging
1 syslog: beginning of session, main actions (exec, process, dataset
handling, cache/package handling, query handling), end of
session
2 syslog all input actions (i.e. also internal messages)
3 syslog also the application logs
- add logging on session begin (at the end of SetupCommon)
- in the end-of-session log, log also the total real and CPU times spent executing commands
Revision
33357 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 4 06:56:18 2010 UTC (4 years, 8 months ago) by
ganis
File length: 215460 byte(s)
Diff to
previous 33280
In proof/proof:
- Add more flexibility to TDataSetManager::ScanDataSet, the internal low-level method
used in dataset verification operations; there are now three option switches to control
- the set of files to process
- all files, the ones marked as staged, the ones marked as non staged
- which pre-action to do on these files:
- check the 'staged' status, open and/or touch the files marked as staged
- which action to do on the files
- no-action, full processing, locate only, stage request
Remarks:
- checking the stage status on 'all files' with 'no-action; can be used to fix the
'is-staged' bit on the dataset
- locate-only and 'stage-request' are done via the TFileStager interface and are fast
non-blocking actions
- Add public method TDataSetManager::ScanDataSet(const char *uri, const char *opts) to control
via a 'const char *' option the new functionality
- Separate out the extraction of the meta-information in the new method TDataSetManager::FillMetaData
which can be called recursively on the file sub-directories; this automatically adds support
for sub-directories in datasets, which was missing and now required by Atlas.
- Adapt TProofServ and TDataSetManagerFile to the new functionality in TDataSetManager .
In proof/pq2:
- Add support for the new options in dataset verification
- Fix issue with regexp parsing in dataset registration
In etc/proof/utils/pq2/pq2-verify:
- Add support for the new options in dataset verification
Revision
33280 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 28 14:45:01 2010 UTC (4 years, 8 months ago) by
ganis
File length: 215485 byte(s)
Diff to
previous 33209
Restructuring of the syslog logging functionality.
The activation is still controlled by the ProofServ.LogToSysLog switch, but now two
levels of logging are provided:
2 log everything to syslog (local5)
1 log only a record for each action performed by the client; by default the
format is
user message-type realtime cputime
When relevant, some information is added, like file name, selector name,
package name, bytes read, etc ...
This may be useful to get a quick monitoring of the cluster activities.
The format of the message to syslog has also been sligthly modified and made more compact;
example
Apr 28 15:50:53 pcphsft64 proofm-0[15086]: ganis 5 0.006 0.010 .!pwd
Apr 28 16:14:52 pcphsft64 proofm-0[18814]: ganis 1012 0.000 0.000 h1analysis.C
Apr 28 16:14:52 pcphsft64 proofm-0[18814]: ganis 1013 0.031 0.000
Apr 28 16:14:52 pcphsft64 proofm-0[18814]: ganis 1012 0.000 0.000 h1analysis.h
Apr 28 16:14:52 pcphsft64 proofm-0[18814]: ganis 1013 0.029 0.010
Apr 28 16:15:02 pcphsft64 proofm-0[18814]: ganis 1015 7.097 0.090 0 283813 22554834 5.920
for the action records, and
Apr 28 16:37:44 pcphsft64 proofm-0[20907]: ganis: Info:<TXProofServ::Terminate>: starting session termination operations ...
for an Info message.
Revision
33209 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Apr 26 10:27:40 2010 UTC (4 years, 8 months ago) by
ganis
File length: 211615 byte(s)
Diff to
previous 33086
- TDataSetManager, TDataSetManagerFile
- Add options to get a list of all dataset registered names from ScanDataSets
(option kList; the result is a TMap of {TObjString, TObjString} with the second
TObjString empty.
- Support wild cards in ScanDataSet .
- TProof, TProofServ
- Add interface to TDataSetManager::ShowCache/ClearCache (TProof::ShowDataSetCache,
TProof::ClearDataSetCache)
- If a non-default dataset directory was specified and could not be validated, do not
initialize the default dataset, as this could be misleading. An warning is issued
on the client shell.
Revision
33086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 18 18:15:56 2010 UTC (4 years, 9 months ago) by
ganis
File length: 211024 byte(s)
Diff to
previous 33054
- Some changes towards a parallel version of VerifyDataSet:
- Make the lower level of ScanDataSet a static in TDataSetManager and separate out
the part extracting the file meta-information into a dedicated method ScanFile
- In TDSetElement, rename the list of associated files to associated objects, so that
it can be more generally used (it was not yet used for associated files and it
was anyhow supposed to be a list of TObjString's).
- Move the parsing of the recently introduced 'DataSet.SrvMaps' env in TDataSetManager
(instead of TProof). This is more logical and it makes easier to apply the enable
the server mappings during staging requests via VerifyDataSet.
- Add a missing protection in TProof::AssertDataSet potentially causing crashes in the
case a dataset is not found.
Revision
33054 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 15 16:47:24 2010 UTC (4 years, 9 months ago) by
pcanal
File length: 210614 byte(s)
Diff to
previous 33009
From Axel:
Move CINT specific initialization (i.e. preventing CINT from #include-ing files it cannot parse) from TRint into TCint. Consistently tell the interpreter to #include <string>, not <_string>. Set the include path in TCint / TCling so they can load interpreter specific headers (like RtypesCint.h). Whether or not to remove the interpreter includes in TProofServLite::SetupOnFork() will
be studied by Gerri, for now they should not hurt.
Revision
33009 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 14 13:44:01 2010 UTC (4 years, 9 months ago) by
ganis
File length: 210971 byte(s)
Diff to
previous 32814
In TProofServ::Reset, check that the directory make sense. Fixes an isue with
a fake error message in PROOF-Lite when issuing TProof::SetParallel().
Revision
32712 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Mar 23 10:35:15 2010 UTC (4 years, 10 months ago) by
ganis
File length: 210805 byte(s)
Diff to
previous 32276
- Add methof TProof::GetStatistics to allow the client to retrieve the correct values of
fBytesRead, fRealTime and fCpuTime at any moment; this will be used to setup a sort of
ROOTmarks in stressProof .
- Fix an issue with TProof::AskStatistics (fBytesRead, fRealTime and fCpuTime were
not correctly filled on the client; the values on the master, displayed by TProof::Print
were correct).
- Add internal method GetLastLog to facilitate the parsing of the text output
of any command (this is used to extend support for new functionality where possible).
Revision
32237 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Feb 6 13:01:45 2010 UTC (4 years, 11 months ago) by
ganis
File length: 210709 byte(s)
Diff to
previous 32204
Fix a few issues found by running stressProof with PROOF-Lite:
- Order query results using TQueryResult::fStartTime instead of TQueryResult::fEndTime
(which is undefined at the beginning)
- Fix the hostname for submergers in PROOF-Lite
- Fix behaviour when submergers cannot open the required sockets (output was not
always sent ot the master)
- Fix several typos / wrong method name / wrong protocol name
- Update info about test numbers in stressProof
Revision
32204 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 3 19:17:40 2010 UTC (4 years, 11 months ago) by
ganis
File length: 208834 byte(s)
Diff to
previous 32003
Add support for processing many datasets in one go in TProof::Process(const char *dataset, ...).
Two options are provided:
- 'grand dataset': the datasets are added up and considered as a single dataset;
syntax: "dataset1|dataset2|..."
- 'keep separated': the datasets are processed one after the other; the user is
notified in the selector of the change of dataset so she/he
has the opportunity to separate the results. A new packetizer,
TPacketizerMulti, has been developed for this case: it basically
contains a list of standard packetizers (one for each dataset) and
loops over them.
Syntax: "dataset1,dataset2,..." or dataset1 dataset2 ..."
In both cases, entry-list can be applied using the syntax "dataset<<entrylist", e.g.
"dataset1<<el1|dataset2<<el2|".
See http://root.cern.ch/drupal/content/working-data-sets#currentelem for more details.
A test for the new functionality has been added to test/stressProof.cxx .
Revision
32003 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jan 13 18:00:12 2010 UTC (5 years ago) by
ganis
File length: 208256 byte(s)
Diff to
previous 31946
- Avoid a possible double deletion in TProofServ::HandleSubMerger
- Fix a small memory leak in TEventIterUnit and TEventIterObj (the
processed 'packet' was not deleted before asking the next one)
- Improve comments in the TProofPlayer
Revision
31946 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Dec 22 08:26:59 2009 UTC (5 years, 1 month ago) by
ganis
File length: 207996 byte(s)
Diff to
previous 31744
Fixes for two memory leaks appearing when running several queries in the same PROOF session:
- in TProofServ::HandleProcess, delete the TProofPlayer instance at the end of the query
- in the TEventIterTree destructor, make sure that the tree cache is also deleted
Revision
31713 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 9 10:00:31 2009 UTC (5 years, 1 month ago) by
ganis
File length: 207603 byte(s)
Diff to
previous 31683
Fix issues found by Coverity:
- #8630, #6228: possible use of 'f' after delete in CopyFromCache, HandleProcess
- #7378, #7377: use snprintf instead of sprintf in Setup, RedirectOutput
- #6066: small memory leak in HandleCache
- #4619, #4617, #854: negative return check missing in CatMotd, SendLogFile, FlushLogFile
- #427, #426, #410, #409, #167: consistency pointer check in TReaperTimer::Notify,
HandleSocketInput, MakeQueryResult, ProcessNext, ErrorHandler
- #139: unreachable code
- #49 : missing check of return code in HandleSocketInput
Revision
31683 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Dec 8 16:12:08 2009 UTC (5 years, 1 month ago) by
ganis
File length: 207273 byte(s)
Diff to
previous 31598
TProofPlayerRemote
- In SendSelector, add missing option kCpBin when sending the selector source; the binary files were
never retrieved, even if present and valid
TProofPlayerSlave
- In Process, fix a problem with cache directory locking while building the selector; the net effect
was that each worker process was re-buidling its own selector binary.
- Fix warning from the nightlies (variable possibly clobbered by `longjmp' or `vfork')
TProofServ
- Fix the order in which the log file is sent in asynchronous processing; the wrong order was screwing
up an immediate synchronous query submission after an asynchronous run; this case occured, for example,
in 'stressProof'
Revision
31598 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Dec 7 15:21:47 2009 UTC (5 years, 1 month ago) by
rdm
File length: 205509 byte(s)
Diff to
previous 31548
From Gerri:
Fix a few pending issues in PROOF-Lite:
- Source the env file before running 'proofserv'; this allows to run valgrind
or other wrapper scripts (e.g. experiment-specific environment initializers)
before proofserv, as in normal PROOF
- Fix a problem with Load()
- Add the possibility for the administrator to control the number of workers
for PROOF-Lite or eventually to forbid the use of it (request on the forum).
The rootrc variable ProofLite.MaxWorkers is read out of /etc/system.rootrc
and cannot be overwritten by users.
Improve handling of Ctrl-C:
- Fix a problem in TMonitor affecting the effectiveness of interrupts.
- Do not send the results from workers to masters if the query has been
aborted.
Miscellanea:
- Make sure that the size of the cache sent back to the packetizer by the
workers is meaningful in case the defaults are used (i.e. not -1).
- Make the current processed element available to selectors via the
input list; this allows to write a more simple and effective packetizers
for tasks where the entry is a file (e.g., dataset generation in the
benchmark suite).
Revision
31511 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Dec 3 05:50:46 2009 UTC (5 years, 1 month ago) by
brun
File length: 205187 byte(s)
Diff to
previous 31505
From Gerri:
- fix warnings in XrdProofdProofServMgr about shadowed variables
- a valgrind warning in TProofServ
- fix for MacOsX in XrdSysPlatform.hh
Revision
31505 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 2 13:20:06 2009 UTC (5 years, 1 month ago) by
brun
File length: 205204 byte(s)
Diff to
previous 31296
From Katerina Opocenska:
Patch implementing the new submerger functionality to speed up the merging phase.
At the end of the query, a set of workers are promoted submergers and assigned a
sub-set of workers to merge. Once each sub-merger has merged its sub-set of workers,
it sends its result to the master, which merges the partial results into the final
set of results.
The determination of the sub-mergers is always done dynamically, based on the recent
performance of workers. An optimal (i.e. giving the highest speed-up) number can be
calculated analytically under simpleassumptions.
Merging via submergers is by default disabled. To enable it, with the optimal number
of sub-mergers, one should set the integer parameter 'PROOF_UseMergers' to 0, i.e.
proof->SetParameter("PROOF_UseMergers", 0)
To force S sub-mergers (regardless of the optimal number) do
proof->SetParameter("PROOF_UseMergers", S)
Revision
31253 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 17 20:41:30 2009 UTC (5 years, 2 months ago) by
ganis
File length: 193346 byte(s)
Diff to
previous 31012
Make sure there is only one PROOF_Ordinal object in the output list
on the workers (in multi-master setups the object added by the submaster
was still in the list).
Revision
31012 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Nov 6 16:08:19 2009 UTC (5 years, 2 months ago) by
ganis
File length: 193233 byte(s)
Diff to
previous 30995
Save the status of a 'proofserv' session into a new file in the 'activesessions' area.
The full path of the new file is
<admin_path>/.xproofd.<port>/activesessions/<user>.<group>.<pid>.status
The status indicates whether the session is idle, running or queued.
The status is updated every 'checkfq' secs (see xpd.proofservmgr; default 30 s).
The status is dumped by the reader thread of TXProofServ and therefore its r/w access
is protected.
This change is needed for PoD optimizations.
Revision
30995 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Nov 5 17:41:03 2009 UTC (5 years, 2 months ago) by
ganis
File length: 190449 byte(s)
Diff to
previous 30939
A couple of quick additions mostly for the new bench suite under development:
- Add the possibility to give indications about the number of workers at startup.
E.g.
1. To start max 5 workers
TProof::Open("<master>","workers=5")
2. To start max 2 workers per physical machine
TProof::Open("<master>","workers=2x")
This is useful in general when running tests (equivalent but quicker then full startup
followed by TProof::SetParallel(n) or TProof::DeactivateWorker(...)).
- Add support for the worker SysInfo_t in TSlaveInfo (obtained via TProof::GetListOfSlaveInfos())
Revision
30859 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Oct 24 14:53:07 2009 UTC (5 years, 3 months ago) by
ganis
File length: 190091 byte(s)
Diff to
previous 30446
Patch for improved performance monitoring. The 'Rate Plot' button in the progress
dialog is renamed 'Performance plot' and shows up to 4 plots with the event/sec,
the average read chunck size, the number of active workers and the number of active
PROOF sessions on the cluster, all as a function of processing time.
The read chunck size plot allows to monitor the usage of the cache.
The istantaneous processing rate (event/sec) is now better estimate: a few issues
with the normalizing times have ben solved, removing the artificial structures that
were observed.
The possibility to set a max packet time length is introduced (default 30 s); this
can be changes with the parameter PROOF_MaxPacketTime.
The size of the cache is also taken into account to optimize the use of the cache.
The parameter PROOF_UseParallelUnzip has been introduced to toggle the use of the
parallel unzip (default off for now).
A page describing the new performance plots is under preparation at
http://root.cern.ch/drupal/content/progress-dialog
Revision
30421 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Sep 24 17:20:09 2009 UTC (5 years, 4 months ago) by
ganis
File length: 189036 byte(s)
Diff to
previous 30174
Some improvements in the dataset manager:
- Introduce the possibility to 'trust' the information provided by the user in the TFileCollection
object sent over upon registration (by default the status bits and the meta information are reset).
- Give the admin has the possibility to configure the dataset manager in such a way that the request
for trusting the information is ignored. A new bit TDataSetManager::kTrustInfo controls that and the
configuration option controlling it is called "Ti" (for 'Trust Info'; equivalent of "Av" for 'Allow Verification').
The current default is that the bit is set.
- Remove the unused bit TDataSetManager::kAllowStaging (staging is controlled externally, not by the
dataset manager)
- Fix a problem with browsing the existing datasets duplicating the same information in some cases.
Revision
30174 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Sep 15 14:24:56 2009 UTC (5 years, 4 months ago) by
ganis
File length: 189036 byte(s)
Diff to
previous 29568
Patch to support automatic creation of a dataset out of files created on the
worker nodes by worker processes. The implementation is an extension of the
functionality of the class TProofOutputFile used for merging via file.
TProofOutputFile:
- Add options to steer the creation of a dataset; the extemded meaning of the
constructor arguments and anew constructor are described in
http://root.cern.ch/drupal/content/handling-large-outputs-root-files
The dataset (i.e. TFileCollection) assembing happens on the master via the
Merge method. Information about the node location of the file is happended
to the relevant TFileInfo as a special "file:" url with option "node=<host-fqdn>:<port>"
This information is needed to facilitate the cleaning of the files once they
are not needed anymore.
The dataset can be just created and returned in the output list, or registered
and/or verified.
TProofPlayerRemote::MergeOutputFiles:
- Correctly transmit the dataset object into the output list
TProofServ:
- Create the new subdirectory 'data' either in the sandbox or followinf ProofServ.DataDir .
- Register and verify the created dataset if required
TProof:
- Implement two new methods:
- ShowData(), to show the content of the assigned 'data' directory
- ClearData(UInt_t what, const char *dsname), to clear all or a subset of the files
in the assigned 'data' directory
Additional changes:
- TProofPlayer:
- Correctly handle the feedback canvas
- Set the drawing attributes
- TProofServ:
- Replace all 'Form("...")' with TString::Format("...") .
- Use the new tag kCollect in debug statements where relevant
- TProof:
- Use the new tag kCollect in debug statements where relevant
Revision
29568 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jul 24 07:28:34 2009 UTC (5 years, 6 months ago) by
ganis
File length: 181010 byte(s)
Diff to
previous 29171
Fix a problem in TProofServ::CopyFromCache affecting the case where the sandbox
dir has a '.' and the macro name has no '.', e.g. compiled selectors in PROOF-Lite.
Revision
29171 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 23 14:13:06 2009 UTC (5 years, 7 months ago) by
brun
File length: 180837 byte(s)
Diff to
previous 29133
From Gerri:
- Fix a few problems found running in valgrind:
- In XrdROOTMgr::Validate, add missing protection against inproper use of a
temporary file when in non debug mode
- Fix a problem with inproper deletion of XrdProofWorker objects when working
with a dynamically varying 'proof.conf'
- In TProof::HandleInputMessage, fix possible double delete after kPROOF_STOPPROCESS
- Fix a couple of issues on reconnection to a running session:
- make sure that the dialog buttons are in the correct state
- make sure that logs are redirected
- Fix an issue with the displaying speed of very large log files (bufferize instead of
doing line-by-line)
- In TProof::Draw, change the type of a notification message from 'error' to 'debug'
- Fix coding conventions in TDataSetManagerFile.cxx
Revision
29133 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 22 12:28:50 2009 UTC (5 years, 7 months ago) by
brun
File length: 180467 byte(s)
Diff to
previous 29034
From Gerri:
Rename TProofDataSetManager and TProofDataSetManagerFile to TDataSetManager and TDataSetManagerFile.
These classes are managers of TFileCollection which do not have anything strictly dependent on PROOF.
So far they were only used internally by PROOF, but they are going to be exposed to more public for generic
file collection handling. It is the right time to give them a more proper name.
Revision
29034 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 16 18:15:44 2009 UTC (5 years, 7 months ago) by
ganis
File length: 180537 byte(s)
Diff to
previous 28856
Record the logs from the ROOT version validation tests (proofserv forked in test mode).
In case of failure - or if the debug flag is on - the log files are kept under
<xproof_adminpath>/rootsysvalidation/root.<tag>.log
(the <tag> has all the '/' replaced by '-').
This should facilitate understanding the problems when in case of validation failures.
Revision
28856 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 9 11:49:43 2009 UTC (5 years, 7 months ago) by
ganis
File length: 180506 byte(s)
Diff to
previous 28593
A few optimizations in transferring and merging of the results:
- Output objects are added to the same TMessage until a HWM is reached
(default 1MB; controlled by 'ProofServ.MsgSizeHWM'); this limits the
number of transfers in the case of large numbers of small objects.
- Reasonably small histograms (GetSize() < MsgSizeHWM) are merged in
one-go at the end instead of one-by-one to exploit the better performance
of TH1::Merge on the full list of histos.
- Add possibility to compress the messages; this is controlled by
ProofServ.CompressMessage <compression_level>
The default is still 'no compression' but this will allow to study the
impact of compression.
In addition, a sort of 'progress' counter for merging is now shown on the
client:
root [n] p->Process(...)
...
Mst-0: merging output objects ... / (4 workers still sending)
This asserts socket activity and fixes the timeout problems during long
merging phases reported on the forum.
Revision
28590 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 13 07:17:36 2009 UTC (5 years, 8 months ago) by
ganis
File length: 175513 byte(s)
Diff to
previous 28577
- Fix a potential source of deadlocks due to the usage of vnsprintf
- Fixes for permission related failures during initialization of dataset managers:
- Add the possibility to define the dataset information sources via the
directive 'xpd.datasetsrc'. In this way the permissions should be
set correctly and the related problems disappear.
- In TProofDataSetManagerFile, switch in browse mode (instead of failing
construction) if the write permissions are not present.
Revision
28395 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 30 09:27:53 2009 UTC (5 years, 8 months ago) by
ganis
File length: 175484 byte(s)
Diff to
previous 28370
Implement two new functions:
1. TProof::SetDataSetTreeName(<dataset>,<treename>)
Sets/changes the default tree name in the dataset (TFileCollection)
2. TProof::ExistsDataSet(<dataset>)
Returns kTRUE if 'dataset' exists.
Fix a problem with attaching to an existing session in TProof::Open .
Revision
28370 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Apr 28 06:43:41 2009 UTC (5 years, 8 months ago) by
ganis
File length: 175018 byte(s)
Diff to
previous 27933
Add the possibility to switch to asynchronous mode while running synchronously.
A new button "Run in background" has been added to the dialog box. The behaviour
of Ctrl-C has also been modified: the user is prompted for a choice among continuing
asynchronously, stopping (terminating) or aborting the query.
Revision
27270 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jan 28 07:44:26 2009 UTC (5 years, 11 months ago) by
ganis
File length: 172562 byte(s)
Diff to
previous 27231
Add support for query queuing also for static sessions. To limit the number of
running sessions to, for example, 3 one can just set
xpd.schedparam default mxrun:3 queue:fifo
The patch also fixes a problem with the fIdle flag in TProofServ (not always
reset after processing) and introduces new methods XrdProofSched::DumpQueues(),
showing the queue content, and XrdProofdProofServ::RemoveQuery to move all the
code to remove a query inside XrdProofdProofServ.
Revision
27231 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Jan 25 09:42:21 2009 UTC (5 years, 11 months ago) by
ganis
File length: 171882 byte(s)
Diff to
previous 27198
Patch adding support for session queuing in the scheduler (from Jan and me).
This allows to control the number of sessions allowed to process queries concurrently.
The feature is enabled by a new parameter 'queue:fifo' in the 'xpd.schedparam'
directive.
In case of static worker assigment (default, random, round-robin) the max number of
running sessions can be limited by another new parameter 'mxrun'; for example
'xpd.schedparam default mxrun:3 queue:fifo' will run concurrently only 3 sessions.
Additional requests are queued and run as soon as one of the running sessions goes idle.
The current policy is FIFO, so that there is a rotation among queued sessions.
In the case of load-based worker assignement, the max number of running queries is
determined dynamically.
This option requires dynamic, per-job scheduling.
Revision
27198 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jan 21 09:29:10 2009 UTC (6 years ago) by
ganis
File length: 171222 byte(s)
Diff to
previous 27042
TProofDraw
- Fix a problem preventing drawing into an existing histogram, i.e. chain->Draw("var>>myhist"),
to work properly (chain->Draw("var>>hnew(nbin,xmin,xmax) was working fine).
- Add missing initialization (fTree in the basic constructor)
TProofPlayer
- Remove the relevant map in the list fFeedbackLists if the final object is available in the
output list; this prevents a duplication of the final feedback object
- Improve debug messages in StoreFeedback
TProofServ, TProof
- Fix a couple of memory leaks due to missing destruction of TProofProgressStatus objects
Revision
27034 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 19 15:30:06 2008 UTC (6 years, 1 month ago) by
ganis
File length: 171041 byte(s)
Diff to
previous 26993
From Jan: prepare for query-enqueing support:
- add new protocol message to resume an enqueued session
- add framework to process enqueued queries after a resume: the processing part on masters
has been moved to a new method ProcessNext which is called also after a resume request.
- remove duplication in the definition of message types for SendCoordinator
(same codes defined in XProofProtocol.h and TXSocket.h)
Revision
26993 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 17 16:01:04 2008 UTC (6 years, 1 month ago) by
rdm
File length: 169797 byte(s)
Diff to
previous 26951
From Gerri:
- Fix a problem with filtering the SvcMsg in TProofMgrLite::GetSessionLogs
- Fix a weird backward-compatibility problem in TProof::SendFile (found
and needed by ALICE)
- Update the release notes
Revision
26951 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Dec 16 12:04:38 2008 UTC (6 years, 1 month ago) by
ganis
File length: 168935 byte(s)
Diff to
previous 26875
- Fix an issue with uploading packages in rapid sequence
- Fix an issue with TProof::Load() in forced compilation mode ('++')
- Fix an issue with grep in log retrieval
- Fix an issue with ownership of the xrootd log file
- Make sure the XrdProofdProofServMgr::fCurrentSessions is always up-to-date
Revision
26875 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 12 14:21:34 2008 UTC (6 years, 1 month ago) by
ganis
File length: 168179 byte(s)
Diff to
previous 26737
Patch to:
- Add the possibility to set upper limits on the virtual memory (changes in TProofPlayer,
TProofServ, TXProofServ and pmain (separate patch)).
If enebled, the session gets firts a warning when it reaches 80% of the limit, and then
processing is stopped whenit exceeds 95% of the limit, sending back the results.
Also, the memory footprint is notified when the session is terminated.
- Make sure the the active valuse in XrdProofWorker are always correct; this was not the
case for dynamic startup as the notification at the end of the query was not done.
This information is crucial or the scheduler.
The way the information is stored in XrdProofdProofServ had to be modified
and a new internal message type (kReleaseWorker) added.
(Changes in several proofd classes, TProof, TXsocket and TXProofServ)
- Fix problem with setting a static upper limit of the sessions that can be started,
and enable this functionality for the dynamic mode (changes in XrdProofSched)
- Remove a deleted worker from all the lists in TProof::MarkBad to avoid later attempts of use.
- Better control the use of the internal pipe for socket readiness notification in TXSocket.
Revision
26737 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Dec 8 18:00:09 2008 UTC (6 years, 1 month ago) by
ganis
File length: 167848 byte(s)
Diff to
previous 26690
TProof, TProofServ
- Make sure that SendFile has finished before sending new messages; this solve some issues
observed sometimes in fast submission of multiple queries
TProofserv
- Make sure that for all actions not allowed during processes a message meaningful to the
requester is sent back
TProof
- Protect all calls to fPlayer to avoid problems during asynchronous actions
Revision
26678 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 5 14:53:23 2008 UTC (6 years, 1 month ago) by
ganis
File length: 168059 byte(s)
Diff to
previous 26624
Patch to:
- Add the possibility to set upper limits on the virtual memory (changes in TProofPlayer,
TProofServ, TXProofServ and pmain (separate patch)).
If enebled, the session gets firts a warning when it reaches 80% of the limit, and then
processing is stopped whenit exceeds 95% of the limit, sending back the results.
Also, the memory footprint is notified when the session is terminated.
- Make sure the the active valuse in XrdProofWorker are always correct; this was not the
case for dynamic startup as the notification at the end of the query was not done.
This information is crucial or the scheduler.
The way the information is stored in XrdProofdProofServ had to be modified
and a new internal message type (kReleaseWorker) added.
(Changes in several proofd classes, TProof, TXsocket and TXProofServ)
- Fix problem with setting a static upper limit of the sessions that can be started,
and enable this functionality for the dynamic mode (changes in XrdProofSched)
- Remove a deleted worker from all the lists in TProof::MarkBad to avoid later attempts of use.
- Better control the use of the internal pipe for socket readiness notification in TXSocket.
Revision
26624 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 3 12:22:00 2008 UTC (6 years, 1 month ago) by
ganis
File length: 166984 byte(s)
Diff to
previous 26513
TProof:
- Fix a problem with Collect in asynchronous mode causing timeouts in some cases
- Optimize the sending of files when loading macros via TProof::Load(..)
- Where possible, replace printouts via fprintf(...) with Printf(...);
TProofServ:
- Fix a problem with SetParallel(0) breaking Exec("<cmd>", kTRUE)
- Fix a problem with retrieving the source files from cache when a full compilation
was required ("++") and a binary already existed.
Revision
26447 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 25 11:11:39 2008 UTC (6 years, 1 month ago) by
ganis
File length: 165829 byte(s)
Diff to
previous 26381
- Fixes on XrdProofd for Ubuntu 8.10 (from F. Furano)
- Fixes in async mode for the dynamic scheduling (from Jan)
- In Xrdproofd:
- Use per-user connections for Reset and Log retrieval
- Fully reset the client session vector upon Reset
- Add several missing protections
- Remove some debug statements in XrdSecProtocolgsi
Revision
26381 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Nov 22 17:15:24 2008 UTC (6 years, 2 months ago) by
ganis
File length: 165664 byte(s)
Diff to
previous 26268
- Fixes for the asynchronous mode:
- Fully localize the general handling of the input messages into two new methods
TProof::HandleInputMessage(TMessage *) and TProofServ::HandleSocketInput(TMessage *);
these methods are callable form any place that needs to intercept some messages and
do something specific on them (examples are: TProofServ::GetNextPacket, TXProofServ::Get,
TPacketizerAdaptive::ValidateFiles, ...); this allows to remove several duplications
and to make sure that no message is lost or wrongly dispatched.
- Simplify the recursive infrastructure for TProofServ::HandleSocketInput; in particular,
TProofServ::HandleSocketInputDuringProcess is removed as it is a special case of the
HandleSocketInput, saving duplications.
- Always use a kPROOF_CHECKFILE message in replies to check file operations; failures were
signaled with kPROOF_FATAL which may have some undesired side-effects, depending on the
timing
- Add support for one level of recursivity in TProof::Collect .
- Fix a problem with TProof::Finalize when called with default arguments (on the last query)
- In TProof::SendFile, send to unique workers only in the "cache" option is specified
- Remove the call to Finalize for DrawSelect queries, as it is done via the feedback
mechanisms
- Additions/fixes in test/stressProof.cxx:
- New test for the asynchronous mode;
- Fine-tuning of the progress display in batch mode;
Revision
26268 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 18 16:27:09 2008 UTC (6 years, 2 months ago) by
ganis
File length: 165487 byte(s)
Diff to
previous 26258
- Fix a few problems with the input data machinery (thanks to C. Cheshkov)
- Use the SingeKey option to write out containers
- Send the input data after the call to Begin() to catch up also the objects
possibly added in there
- Fully enable the input data machinery also for PROOF-Lite; this required to
move some calls from TProofServ to TProof to avoid duplications.
- Additions to test/stressProof.C
- Add a test for the input data machinery
- Add some progress info for the tests involving processing
Revision
25969 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 27 12:08:33 2008 UTC (6 years, 2 months ago) by
ganis
File length: 169469 byte(s)
Diff to
previous 25926
- Make the static ErrorHandler public so that it can be used in other places, e.g. TProofPlayerLite
- Add the possibility to control to which I/O stream the ErrorHandler logs should go (default stderr)
Revision
25918 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 22 15:00:04 2008 UTC (6 years, 3 months ago) by
ganis
File length: 168620 byte(s)
Diff to
previous 25896
First version of PROOFLite, a 2-tier realization of PROOF intended for multi-core
machines. In PROOFLite the client start directly the workers; no daemon is required.
Currently this works on Unix only. Some issues with the internal connection still need
to be sorted out on Windows.
To start a session just use
TProof::Open("") or TProof::Open("lite")
From there on everything should be as in normal PROOF. To start a standard PROOF
session (i.e. via daemons) on the localhost use TProof::Open("localhost").
(The default meaning of "" in TProof::Open() is controlled via the rc value
'Proof.LocalDefault').
To run stressProof with PROOFLite pas the argument 'lite':
./stressProof lite
By default, PROOFLite sets the number of workers equal to the number of cores.
The number of workers is controlled by the rc variable 'ProofLite.Workers'; it can
also be set on the fly with the option "workers=<n>", e.g. TProof::Open("lite?workers=2").
This patch also introduces the rc variable 'Proof.Sandbox' to control the local location
of the sandboxes. The default has been changed from "~/proof" to "~/.proof" to avoid
interferences with possible users' working areas. The sandbox contains the directory
'packages' (with the clients packages; used by standard PROOF and PROOF Lite),
and some directories used only by PROOFLite: 'cache' (binary cache) and 'datasets' with
the datasets. The location of these directories is controlled by the rc variables
'Proof.PackageDir', 'Proof.CacheDir' and 'Proof.DataSetDir'.
The working directory for PROOFLite is in the form workdir=<sandbox>/path-to-current-dir where
~/path/to/current/dir is the directory from where the session in launched. The directory
<workdir> contains the session working dirs (in the form 'session-<host>-<time>-<pid>'),
with the symlink 'last-lite-session' pointing to the last started session, and the 'queries'
directory, with the results of queries started from the current directory.
Revision
25896 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 20 17:01:31 2008 UTC (6 years, 3 months ago) by
ganis
File length: 165530 byte(s)
Diff to
previous 25868
From Jan:
- Using consistently the recently introduced TProofProgressStatus in the kPROOF_GETPACKET
messages sent to TPacketizerUnit, TPacketizerAdaptive and TPacketizer; the message contains
the status of progress since the start of processing on a given node.
- Introduce TVirtualPacketizer::TVirtualSlaveStat as a base class of all the TSlaveStat
packetizer specific auxilliary classes.
- Full implemenation of GetProgressStatus() and AddProcessed(TProofProgressStatus *st) members
for TPacketizerUnit::TSlaveStat.
This patch should fix some consistency problems experienced after the patch introducing
TProofProgressStatus .
Revision
25827 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 15 14:02:59 2008 UTC (6 years, 3 months ago) by
ganis
File length: 170196 byte(s)
Diff to
previous 25729
From Jan:
- Added the possibility to handle removed workers and partly processed packets. When a worker is stopped
while processing a packet it finishes, the current event and the rest of the packet is reassigned to another
worker. This is done via two interfaces:
- TVirtualPacketizer::AddProcessed(TSlave *sl, TProofProgressStatus *st, TList **)
- TVirtualPacketizer::ReassignPacket.
- New class TProofProgressStatus used to keep the query progress stauts in all the TProofPlayer objects and in
TPacketizerAdaptive::TSlaveStat. This class is also used to structure the relevant information send in
kPROOF_GETPACKET and kPROOF_STOPPROCESS messages.
- The class TPacketizerProgressive is removed completely.
- The PROOF protocol version is increased to 19: this is to handle the changes in the kPROOF_STOPPROCESS and
kPROOF_GETPACKET messages in Master - worker communication.
Revision
25720 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Oct 7 10:13:37 2008 UTC (6 years, 3 months ago) by
ganis
File length: 169033 byte(s)
Diff to
previous 25704
Move the code for query-result management into a new class TQueryResultManager,
so that it can be used in other places (e.g. prooflite).
Revision
25704 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Oct 5 21:44:36 2008 UTC (6 years, 3 months ago) by
ganis
File length: 186638 byte(s)
Diff to
previous 25434
TProof:
- Add support for input data objects. These are objects which are available via the input list
but - being potentially large - are distributed to the cluster in optimized mode, i.e. via a
ROOT file copied only to unique nodes. These objects are *not* automatically saved into the
TQueryResult object.
Three new public methods have been added to TProof:
- AddInputData(TObject *, Bool_t = kFALSE) flags an input object as 'data input object'
- ClearInputData(TObject * = 0) clears the list of input data objects
- ClearInputData(const char *name) removes object 'name' from the list of input
data objects
- SetInputDataFile(const char *filepath) specifies the file from where to take the
input data objects; object in the input data
list, if any, are also send over.
New internal methods:
- BroadcastFile(const char *file, Int_t opt, const char *rfile, TList *wrks);
- BroadcastFile(const char *file, Int_t opt, const char *rfile = 0, ESlaves list = kAllUnique);
- void SendInputDataFile();
- Fixes:
- in Process(TFileCollection *, ...), remove the TFileCollection object from the input list
after processing, to avoid destruction problems in case of multiple query submission.
TProofServ:
- Add support for input data objects.
New methods:
- Int_t GetInputData(TList *input);
- Int_t SaveInputData(TQueryResult *qr);
- Int_t SendInputData(TQueryResult *qr);
- Optimize the number of input list instantiations
- Improve readibility of HandleProcess by separating out the code to prepare the dataset (get it
from input list, from dataset manager, etc ...) into a new method AssertDataSet(TDSet *, TList *).
TProofQueryResult:
- add missing protection
Revision
25434 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Sep 17 15:23:36 2008 UTC (6 years, 4 months ago) by
ganis
File length: 180957 byte(s)
Diff to
previous 25273
- Fix problem with the determination and transmission of the name of the object to be processed.
The problem appeared when processing files containing >1 trees in changing order.
- Fix problem with TProof::Load (Savannah bugs #41719; forum topic #7138)
- Fix problem with the logs from TProof::Load not being transmitted to the client
Revision
25273 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 27 08:56:06 2008 UTC (6 years, 4 months ago) by
rdm
File length: 179677 byte(s)
Diff to
previous 25107
From Jan:
- A new optional version of PROOF with dynamic worker startup.
It can be enabled by the admin or a user with 'Proof.DynamicStartup'.
A session starts only on the master. When a query processing starts
at the master TXProofServ::GetWorkers() is called. It receives a
list of machines from the scheduler and the workers on the
machines are started. The environment is copied from the master
to the workers. It includes: the include and dynamic library paths,
the set of enabled packages as well as the macros loaded by the user.
- A new method TProof::AddWorkers(TList *workers) was added. It adds
the workers just before the query.
- A packet resubmitting mechanism. When a worker dies all the packets
that it processed are resubmitted.
- In TPacketizerAdaptive: fixing initialization of fgMaxSlaveCnt. By
default it was initialized twice.
Revision
25107 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Aug 10 08:06:32 2008 UTC (6 years, 5 months ago) by
ganis
File length: 179048 byte(s)
Diff to
previous 24834
Use Printf to notify abortion for missing socket activity.
The dedicated Fatal(...) implementation in TProofServ tries to use the
input socket and may lead to long waiting timeouts.
Revision
24834 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jul 15 14:39:18 2008 UTC (6 years, 6 months ago) by
ganis
File length: 179033 byte(s)
Diff to
previous 24780
Fix a problem triggering full re-build of a package upon change of a single file;
the version info file was wrongly reset; this should happen only after a re-build.
Revision
24766 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jul 10 12:35:24 2008 UTC (6 years, 6 months ago) by
ganis
File length: 179440 byte(s)
Diff to
previous 24739
TProofServ:
Terminate the master session if, as the result of the query, all workers are gone.
The session is anyhow unusable and in this way we avoid attaching again to it.
TProof:
Disable the buttons on the progress dialog after a kPROOF_FATAL
Revision
24738 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jul 9 11:33:34 2008 UTC (6 years, 6 months ago) by
ganis
File length: 179101 byte(s)
Diff to
previous 24737
When shutting down because of no-activity on the input socket, use TSystem::Abort()
instead of TProofServ::HandleTermination(). The session is anyhow lost and normal
termination may cause hanging of the process.
Revision
24719 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jul 9 07:07:25 2008 UTC (6 years, 6 months ago) by
ganis
File length: 178911 byte(s)
Diff to
previous 24701
Patch refactorizing the XrdProofd plugin.
The class XrdProofdProtocol is now in charge only of the operations strctly related
to the XProofd protocol. All auxilliary services have been moved to dedicated service
classes controlled by XrdProofdManager. In particular:
- XrdProofdClientMgr handles the clients (represented by instances of XrdProofdClient)
including login, authentication and access control
- XrdProofdProofServMgr handles the PROOF sessions (represented by instances of
XrdProofdProofServ) including creation, attachment, detachment, destruction and
environment setting
- XrdProofdNetMgr handles connections between instances of XProofd running on different
nodes
- XrdProofdPriorityMgr handles the session priorities
A special effort has been done to get rid of all possible internal dead-lock situations.
Internal actions on clients and sessions are now all asynchronous, governed by internal
pipes.
The new plugin also offers new functionality, among which:
- a XProofd admin area, located under <xrd.admin>/.xproof.<port>, keeps information about
active and terminated sessions, and the minimal state for active clients. This is used
to reguraly check the client and session activity, to cleanup orphalin sessions and
to shutdown inactive client connections.
In particular this allows a more solid implementation of Reset, which now exists in two
flavours: 'soft', TProof::Reset(<masterurl>), (default) asks the sessions to terminate gently;
'hard', TProof::Reset(<masterurl>,kTRUE) schedules all sessions for forced termination.
- support for automatic attempts for reconnection in the case the daemon restarts.
This allows to reconfigure the plugin by xrootd restart w/o affecting the running
sessions.
- support the definition of workers via config file directive, getting de facto rid of
the proof.conf file.
- domain+level control of printout message; the format has been improved: in particular
all information messages contain now the tag 'xpd-I' and all error messages the tag
'xpd-E', so that they can easily be grepped out from the log file.
The Wiki 'XrdProofdDirectives Directives' page has been updated with the new directives.
Revision
24589 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jun 27 14:17:08 2008 UTC (6 years, 6 months ago) by
ganis
File length: 177208 byte(s)
Diff to
previous 24574
- Add support for an external definition of the session tag: this will be needed
by external PROOF session launchers, when available
- Check also the env ROOTRCFILE for 'rootrc' configuration.
Revision
24574 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 26 17:06:03 2008 UTC (6 years, 6 months ago) by
ganis
File length: 176906 byte(s)
Diff to
previous 24570
Implement selective sending of logs from workers to master to avoid duplicating
too many text lines on the master log. Logs are now sent only after Exec, Print
requests and in case an error (level >= kError) occured. Of course, the full logs
can always be retrieved via TProofMgr::GetSessionLogs .
Also, remove TXProofServ::SendLogFile, which was reduced to be a duplication of
TProofServ::SendLogFile .
Revision
24570 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 26 13:45:13 2008 UTC (6 years, 6 months ago) by
ganis
File length: 176105 byte(s)
Diff to
previous 24520
- Add the possibility to cleanly terminate an active worker at any moment.
This is needed by the dynamic worker configuration.
Revision
23986 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri May 23 09:10:18 2008 UTC (6 years, 8 months ago) by
ganis
File length: 175236 byte(s)
Diff to
previous 23755
Import fixes / new functionality from branches/dev/proof:
- proof/proofplayer/src/TPacketizerAdaptive.cxx
Implement the classic strategy of the TPacketizer in TPacketizerAdaptive.
The strategy can be changed from adaptive (default) to TPacketizer with:
"PROOF_PacketizerStrategy" parameter to PROOF
- proof/proofplayer/src/TProofPlayer.cxx
Fixed error messages for 'MissingFiles' and 'FailedPackets' lists.
Improve fault detection by creating a list of failed packets upon a mismatch
in the expected and actual number of processed events; the list is added to
the output list.
- proof/proofplayer/src/TVirtualPacketizer.cxx
Make sure that something has been processed before setting kIsDone.
Improve fault detection by creating a list of failed packets upon a mismatch
in the expected and actual number of processed events; the list is added to
the output list.
- proof/proofplayer/inc/TVirtualPacketizer.h
Improve fault detection by creating a list of failed packets upon a mismatch
in the expected and actual number of processed events; the list is added to
the output list.
- proof/proofplayer/inc/TPacketizerAdaptive.h
Implement the classic strategy of the TPacketizer in TPacketizerAdaptive
- proof/proof/src/TProofServ.cxx
Fixes:
+ option string: "stageOnly" --> "stagedOnly".
+ add parenthesis to avoid a warning after the previous patch.
+ remove the objects added to the missingFiles in TDSet::Add from the 'dataset'
before deleting it
+ fixed an error HandleCheckFile ('kPROOF_WorkDir' instead of 'kPROOF_PackDir').
+ make fCacheDir and fPackageDir controllable via directive
+ in TProofServ::ErrorHandler: do not create the related additional buffer if
not logging to syslog
Added functionality:
- add possibility to flag an "Info" message as service message using
the prefix "|Svc" in the location field; e.g.
Info("SetupCommon|Svc", "Test of SvcMsg");
will produce something like
09:28:24 6892 Mst-0 | SvcMsg in <TXProofServ::SetupCommon>: Test of SvcMsg
This is needed to be able in the future to filter-out messages needed
by some services (e.g. the forthcoming memory checker) which should not be
displayed by default.
- proof/proofd/src/XrdProofSched.cxx
Improve the calculation of the number of workers to assign by using fMinForQuery
as a minimum.
Fix the length of method separators.
Fix signed/unsigned warning.
Revision
23248 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 16 09:09:35 2008 UTC (6 years, 9 months ago) by
ganis
File length: 174423 byte(s)
Diff to
previous 23240
- Fix a few issues with BrowseDataSets.
- Rename ParseDataSetUri to ParseUri
- Add a static method CreateUri to format a dataset Uri from group, user, dataset name
and object name
- Automatically fill group="default" and user="<real_user>" when either of the two
are not passed
Revision
23180 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 13 07:36:20 2008 UTC (6 years, 9 months ago) by
ganis
File length: 174430 byte(s)
Diff to
previous 23108
- Make fCacheDir and fPackageDir controllable via directive
- In TProofServ::ErrorHandler Do not create the related additional
buffer if not logging to syslog
Revision
23075 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 9 09:10:25 2008 UTC (6 years, 9 months ago) by
ganis
File length: 174317 byte(s)
Diff to
previous 22635
From Jan-Fiete Grosse-Oetringhaus (ALICE) and me.
Patch introducing the improved dataset manager developed by ALICE.
The client side interface is basically unchanged, the biggest change is the renaming
of TProof::CreateDataSet to TProof::RegisterDataSet.
On the server side the relevant code has ben extracted from TProofServ and hidden
behind an inteface class TProofDataSetManager describing the functionality needed by
TProofServ, and an implementation class TProofDataSetManagerFile handling dataset
meta data saved into ROOT files. TProofDataSetManagerFile is instantiated via the
plugin manager and by default it handles the <sand_box>/datasets area.
This separation is needed to allow loading implementation based on different backends;
for example, ATLAS foresees to use a MySQL-based backend.
A TWiki page with information and examples in on preparation.
Revision
22419 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Mar 3 00:25:01 2008 UTC (6 years, 10 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 173064 byte(s)
Diff to
previous 21666
From Andrew Savchenko:
ROOT can not be compiled with gcc-4.3.
Some ROOT source files doesn't contain required #include directives,
for example, they use strlen(), but #include <string.h> is missed or
malloc() is used and #include <stdlib.h> is missed.
Earlier versions of gcc allowed some headers to be included implicitly,
but issued a warning (-Wimplicit-function-declaration). Newer one,
gcc-4.3 denies such silly behaviour: all required headers must be explicitly
included.
Attached patch fixes this. Also it fixes another issue, which disallows
ROOT to compile under gcc-4.3: C functions don't belong to namespace std,
so expressions like std::memcpy() are no longer valid and plain memcpy()
should be used instead.
Revision
21250 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Dec 6 23:55:35 2007 UTC (7 years, 1 month ago) by
ganis
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 172067 byte(s)
Diff to
previous 21236
- Fix problem crashing the application at reconfig (bug #28778)
- Fix a bug preventing standard Scalla/Xrootd if-else-fi config constructs
to work
- Fix a bug in XrdProofdManager::GetProofConn in checking existing valid
connections
- Fix a bug in TProofServ::ApplyMaxQueries preventing the correct removal
of empty query directories
- Fix coding convention violations in TPacketizerAdaptive (from Jan)
Revision
20862 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Nov 18 14:51:42 2007 UTC (7 years, 2 months ago) by
ganis
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 172119 byte(s)
Diff to
previous 20758
Synchronize with braches/dev/proof r20835
Summary:
TProofServ:
- read session.rootrc with level kEnvChange to be able to change existing
settings
- Avoid deleting a query result twice in some special cases
- Fix a problem with the initialization of fKeptQueries
rootrc.in:
- restore default settings the asynchronous reading
TFileCacheRead:
- add missing protection in ReadBuffer
TEventIter:
- Enable the usage of TTreeCache
XrdProofdProtocol:
- Additional check on the ownership of the unix socket
- Improve notification during Reset
- Reduce default timeout on admin requests and make it configurable
XrdProofConn:
- Use the configurable maxtry everywhere where relevant
TPacketizerAdaptive:
- Store info on all the processed packets in per-worker lists
TProofMgr, TXProofMgr, TXSocket, XProofProtocol, XrdProofdProtocol:
- Add possibility for the admin to broadcast a message to the connected users
getProof:
- Add missing protection
Revision
20758 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Nov 11 21:29:49 2007 UTC (7 years, 2 months ago) by
ganis
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 172177 byte(s)
Diff to
previous 20685
Several small fixes and few additions imported from 'branches/dev/proof':
- XrdProofdProtocol:
+ Fix a problem in CleanuProofServ sometimes preventing proper cleanup
+ In ReadBuffer, do not trigger an error on empty files
+ Add missing protection against an empty string while setting the ROOT version
- XrdProofdManager:
+ Define the list of unique nodes to avoid broadcasting twice to the same node
- XrdProofWorker: improve matching technique
- TProof: add setter and getter for Int_t parameter
- TProofServ:
+ do not flush the log file in SendAsynMessage: this recent change
made the logs "disappear" form the ProgressDialog
+ Activate control on the max number of query result kept
- TPacketizerUnit: fix a problem with filling the histof for PerfStats
- TProofPlayer: call Begin() also during selecter re-initialization
Revision
20682 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 6 15:51:59 2007 UTC (7 years, 2 months ago) by
ganis
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 166175 byte(s)
Diff to
previous 20597
Import branches/dev/proof r20654
Summary:
+ Improvements
- Add support for SSH SOCKS4 tunnelling; the local port for the tunnel can be specified
in the master URL, e.g. TProof::Open("master/?tunnel:8000")
- Add the possibility to plot the estimated instantaneous rate
- Add "PROOF_ForceLocal" parameter to the !TPacketizerAdaptive; if set to 1, all the data
are processed locally.
- Add support for remote grep functionality while retrieving logs (needed by the forthcoming
memory monitor)
+ Bug fixes
- Several small fixes to revive the multi-master mode.
- XrdProofdProtocol:
- add missing lock to the client instance in SendMsg to avoid screwing up requests from
workers on the same machine
- lock the mutex of the requester when setting prorities
- add notification during Reset
- fix problem with the detection of the 'allow' directive
- fix problem with the parsing of the return value from XrdProofServProxy::TerminateProofServ()
- always use the effective user to retrieve info fom another server
(XrdProofdManager::GetProofConn is now used)
- fix possible dead-locks from debug notifications done after hard-killing a session
- re-enable the garbage collector thread of the connection manager in XrdProofConn to
fix a problem with closing physical connections;
- fix a problem with !CleanupProofServ in the case of a non-privileged daemon running
in multi-user mode
- introduce a timeout when waiting for the startup of a 'proofserv'.
- XrdProofConn: init mutex in the ctor; lock in SendRecv
- XrdProofSched: add support for using the priorities defined in the group manager to define
the number of workers for sessions
- TProof:
- Broadcast priorities to unique nodes only
- timeout after 5 mins the initial Collect to avoid clients getting stuck at this stage
- add support for generic timeout in Collect (disabled by default)
- fix a problem with SendFile.
- TXProofServ: add a call to !TProof::InterruptCurrentMonitor() in Terminate() to stop
infinite loops in Collect
- TXSocket:
- Implement a flag to interrupt a TXSocket while waiting for messages
- Split the session creation timeout in 4 attempts: the total timeout is the same but
it may circumvent occasional forking problems.
- XROOTD:
- fix a potential (possibly accademic) memory leak in the client
- fix a access permission problem with Kerberos ticket forwarding
- fix bug preventing 'locate' to work properly
- re-enable optimized 'locate'
Revision
20411 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Oct 19 12:37:58 2007 UTC (7 years, 3 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 163464 byte(s)
Diff to
previous 20307
remove conditional around include of RConfigure.h:
#ifdef R__HAVE_CONFIG
#include "RConfigure.h"
#endif
Having this conditional caused RConfigure.h to be missing from the .d
files and hence these files would not be recompiled after re-configuring
using --prefix.
Revision
20307 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 11 10:58:50 2007 UTC (7 years, 3 months ago) by
ganis
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 163498 byte(s)
Diff to
previous 19826
Import branches/dev/proof r20306
Summary (see branch logs for more details):
- Set of changes related to CPU quota control including
- broadcast of centrally determined priorities
- mechanism to renice processes in a quantitative way
- New packetizer for non-tree based analysis and related API in TProof (from L. Tran-Thanh)
- Support for merging output objects saved in files on the workers (from L. Tran-Thanh and me)
- Improve version binary compatibility checks using also the SVN revision number
(when available) to define the running version.
- Extend the version binary compatibility checks also to the cached selector binaries.
- Extend TDSet::Lookup so that in case of missing files, it can remove them from the
dataset (option removeMissing must be set).
- Move the data set lookup to the TProofPlayerRemote::Process.
- Handle properly the case of incomple datasets: if the file is not found in the lookup
don't try to validate it; add it, instead, to a 'missingFiles' list returned in the
output list (fixing bug #28800 in Savannah).
Revision
19145 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jun 22 17:16:35 2007 UTC (7 years, 7 months ago) by
ganis
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 158061 byte(s)
Diff to
previous 19099
Add support for global package directories. The colon-separated list of directories
to be searched for is defined by the directive
Proof.GlobalPackageDirs globaldir1[:globaldir2[:globaldir3[:...]]]
which can be included in the xrootd config file via the xpd.putrc directive
xpd.putrc Proof.GlobalPackageDirs globaldir1[:globaldir2[:globaldir3[:...]]]
The directories must be readable by the user (a check is made in TProofServ::Setup).
Packages are searched first in the user local cache and then in the global
directories (in the given order, so that the first occurence is used).
The content of the global package directories is listed by TProof::ShowPackages().
The functionality is available also locally on the client. The Proof.GlobalPackageDirs
has to be specified in the users $HOME/.rootrc .
Global packages can be directly enabled via TProof::EnablePackage(<package_name>).
Revision
19099 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 21 08:47:42 2007 UTC (7 years, 7 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 154532 byte(s)
Diff to
previous 19093
added new data member fGroupPriority. This priority is obtained by
GetPriority() which makes an SQL query in the proofpriority table, which
has the format:
CREATE TABLE proofpriority (
id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
group VARCHAR(32) NOT NULL,
priority INT
)
This prority is then forwarded to the xrootd on the workers.
Revision
19008 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 12 13:51:41 2007 UTC (7 years, 7 months ago) by
ganis
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 151606 byte(s)
Diff to
previous 18983
New files:
XrdProofGroup.h, .cxx
- Class describing a group and its properties
XrdProofdClient.h, .cxx
- Class describing a Xpd client (extracted from XrdProofdProtocol.h, .cxx)
XrdROOT.h, .cxx
- Class describing a ROOT version (extracted from XrdProofdProtocol.h, .cxx)
XrdProofdAux.h, .cxx
- Utility tools (extracted from XrdProofdProtocol.h, .cxx)
XrdProofdPlatform.h
- Platform-dependent system includes and defines common to all implementation
files (extracted from XrdProofdProtocol.cxx)
Patched files:
XrdProofdProtocol.h, .cxx
- Adapt to file re-organization
- Rationalization fo the way the relevant info is passed to 'proofserv':
a minimal set of environment variables is kept, the remaining parameters
are passed via a rootrc-like file created in the session working dir
- New directive "xpd.groupfile" to read a file defining groups and their
properties.
- New directive "xpd.datasetdir" to define the root path under which the
user data set dirs are created
- New directive "xpd.putrc" to input rootrc-like directives to be passed
straight to proofserv
TProofServ.cxx, TXProofServ.h, .cxx
- Read the session rootrc in the constructor
- Take relevant parameters from the session rootrc file
TProof.h
- Move protocol number up to 14 to be able to handle backward compatibility
issues due to changes in the info transfer mode between xpd and proofserv
pmain.cxx
- Take loglevel from argc #5
- Use logfile path generated by XrdProofdProtocol.cxx
Revision
18888 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri May 25 13:53:59 2007 UTC (7 years, 8 months ago) by
ganis
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 152112 byte(s)
Diff to
previous 18860
From Jan:
Add support for processing datasets "by name", i.e. by just sending the name
of a dataset known by the master.
TProof.h, .cxx
- New method
Long64_t TProof::Process(const char *dsetname, const char *selector,
Option_t *option = "", Long64_t nentries = -1,
Long64_t firstentry = 0, TEventList *evl = 0);
- Fix typos in comments
TProofServ.cxx
- In HandleProcess, retrieve the relevant dataset info "by name" from local
dirs in the case only the data set name is specified.
- In HandleRetrieve, remove a debug printout left from a previous patch.
Revision
18860 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 23 09:10:19 2007 UTC (7 years, 8 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 151182 byte(s)
Diff to
previous 18835
From Gerri:
- Add method SendAsynMessage(const char *, Bool_t lfeed) to send asynchronous
messages to the client; this allows the repetition of the same code
in many places.
- Add new member fPrefix which is initialized in the ctor to be, for example,
"master-0" and used everywhere, included the Error handler: this allows
to avoid recreating the same prefix in many places in the code.
- Add method FlushLogFile() allowing to hide from the client messages which may
important for debugging, but not so relevant for normal operations.
- Set/Unset the kWriteV3 bit in TDSet when relevant (i.e. when the client
protocol is 12 or less).
Revision
18349 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Mar 19 14:43:26 2007 UTC (7 years, 10 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 148961 byte(s)
Diff to
previous 18327
From Gerri:
- TProof::Load(const char *macro, Bool_t notOnClient) where 'macro' is the
name of a macro or the implementation file of a class (in which case the
corresponding header should have .h or .hh extension). If notOnClient
is kTRUE (default is kFALSE) the class/macro is loaded only
on master and workers.
- Re-usage of valid binaries from macros or selectors (they are cached under
<SandBox>/cache), this allows to save a few seconds while running a query
with an unchanged and compiled selector.
- Exports ROOTINCLUDEPATH with the current content of gSystem->GetIncludePath()
just before executing BUILD.sh so that the path can be used for build
purposes (bug #18079)
Revision
17562 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jan 29 15:11:11 2007 UTC (7 years, 11 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 143663 byte(s)
Diff to
previous 17431
Frrom Gerri:
There is a new classes TProofServLogHandler allowing to redirect
the output to the client in real-time. It has two main constructors,
one taking an external command (like BUILD.sh) and opening a pipe,
the other taking a pointer to a open file. TProofServ uses the class
via the guard class TProofServLogHandlerGuard to better localize the
usage. This redirection is not always activated. For the time being it
is active for Process and Cache actions,and it can be deactivated via
TProof::SetRealTimeLog(kFALSE)
(for BUILD.sh - executed externally - is always ON).
In particular, with the patch applied, the outputs from BUILD.shi
will scroll down on the screen, and clicking on "Show logs" in the
dialog box during process shows the messages and allows to see what
the session is doing: this is quite nice, because getting
an empty window from "Show Logs" is frustrating, in my opinion.
So, all this fixes the bug/request #16845.
The patch also fixes bug #22555 (hanging while enabling a package
without BUILD.sh) and several other small bugs (a couple of unitialized
variables) or typos.
It also cleanups the API a bit (we had SetQueryType and SetQueryMode
doing exactly the same thing ...).
Revision
17149 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Dec 11 13:24:49 2006 UTC (8 years, 1 month ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 135991 byte(s)
Diff to
previous 17037
Two new features:
- TProof::EnablePackage() has now a bool argument notOnClient which when
true does not build and enable the package on the client. The default
is false.
- Package building now keeps track of which PROOF version has been used
to build a package. In case the version has changed the system will
first call "PROOF-INF/BUILD.sh clean" and then "PROOF-INF/BUILD.sh".
The user has to provide in BUILD.sh the code to handle the "clean"
argument to call e.g. "make clean" or whatever is needed to clean
the package and force a rebuild.
Revision
16955 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 27 14:14:24 2006 UTC (8 years, 1 month ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 135250 byte(s)
Diff to
previous 16839
From Gerri:
This is the patch to move TDSet and TChainProof in 'proof' and remove
any dependence on PROOF in 'tree'. I have renamed TChainProof as TProofChain,
for consistency. TProofChain is now deriving from TChain and overrides only
the required methods. It is loaded via PluginManager by TChain in SetProof.
TQueryResult stores now all the input-related objects in fInputList. There
is a new method
TObject *TQueryResult::GetInputObject(const char *classname)
to retrieve the first instance of a class from the input list, so for the
TDSet:
TQueryResult *qr;
TDSet *dset = (TDSet *) qr->GetInputObject("TDSet")
The changes in TSessionViewer are needed to adapt to this new way of
accessing TDSets.
Revision
16839 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 20 15:56:36 2006 UTC (8 years, 2 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 135007 byte(s)
Diff to
previous 16796
From Maarten + Gerri:
Patch to allow run time configuration of proofserv environment. The client
can pass a set of shell variables to PROOF. These variables can be used
to startup different versions of the proofserv, or they can instruct the
system to run proofserv under valgrind or tcheck, etc. Main changes:
- default wrapper script for proofserv
- modifies the build system for the wrapper script
- adds a list of environment variables to TVirtualProof, which are
passed to the wrapper by proofd/xrootd.
- adds code to TProofServ to pickup these variables and pass them
along to the slaves/submasters
Revision
16796 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Nov 16 17:18:32 2006 UTC (8 years, 2 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 134594 byte(s)
Diff to
previous 16775
Changes to get rid of the config.h and HAVE_CONFIG define. These were way
too trivial (just do a locate config.h to see how many there are):
- replace occurances of config.h by RConfigure.h
- add #warning in config.h that is should not be used
- change HAVE_CONFIG to R__HAVE_CONFIG
Revision
16775 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 15 17:45:55 2006 UTC (8 years, 2 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 134587 byte(s)
Diff to
previous 16704
From Gerri and Bertrand:
Fixes and new features developed in preparation for the Intel quad-core
launch demo:
- Add instantaneous event and MB processing rates in the Progress signal
and make necessary modification to exploit it in the Progress Dialog.
- Add possibility to change the Progress and Feedback periods
- A number of small improvements and bug fixes
Revision
16704 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 7 16:46:37 2006 UTC (8 years, 2 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 134469 byte(s)
Diff to
previous 16579
- Fix hang in case one tried to enable a non existing packages.
- On master always try to upload package to workers just before
building the packages. That ensures that new workers will also
properly get the packages before building them.
Revision
16557 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 19 12:38:07 2006 UTC (8 years, 3 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 134087 byte(s)
Diff to
previous 16546
From Gerri:
Major set of robustness fixes, improving error reporting and reducing the
startup failures especially observed on AMD64. Summary of changes:
- XrdProofdProtocol.cxx, .h
- Remove zombie treatment: done by XrdScheduler::Reaper()
- Check the modification date of the PROOF config file so that changes
can be picked up on the fly
- Fix problem with the definition of the image
- Improved tracing (see etc/proof/xpd.cf.sample)
- Several bug fixes
- XrdProofConn
- fix problem with marshalling in case of retries.
- TProofServ, TXProofServ
- Add return codes to CreateServer() and Setup() for proper termination
in case of errors
- TProof
- Make sure that TXProofMgr is the last one in the list of sockets for proper
cleaning
Revision
16217 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Sep 7 09:27:25 2006 UTC (8 years, 4 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 134148 byte(s)
Diff to
previous 16025
From Gerri:
TPacketizer.cxx:
- Fix problem with object used after deletion (in print statement)
TProofServ.cxx:
- Send "start building" notification message before starting the build
(shows that something is happening ...)
TXNetFile.cxx, TFileMerger.cxx
- Make creation of missing path parts optional (Andreas' suggestion):
this can be set via the env "XNet.Mkpath" or on the fly adding "mkpath=1"
to the options ("mkpath=0" switches it 'off' if XNet.Mkpath sets if to 'on').
Revision
15956 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Aug 8 21:27:14 2006 UTC (8 years, 5 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 133247 byte(s)
Diff to
previous 15944
From Gerri:
the attached small patch adds the TNamed in question. After the patch, this
TNamed *nm = (TNamed *) fInput->FindObject("PROOF_QueryTag");
if (nm) {
Printf("Found query tag: %s", nm->GetTitle());
} else {
Printf("Query tag not found");
}
should return the query tag in SlaveBegin or Terminate.
The ID is created on the master so it cannot be available in Begin.
Revision
15857 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jul 26 14:28:59 2006 UTC (8 years, 6 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 132601 byte(s)
Diff to
previous 15704
From Gerri:
TVirtualProof.h
- Add abstract methods methods to communicate the data set status
(signal + sender)
pmain.cxx
- Add call to Exit(0). T(X)ProofServ::Terminate() just interrupt the event
loop (insteadof exiting) to avoid problems when the terminate is run in
a separate thread (like in xproofd).
proof/inc/TProof.h
- Add methods methods to communicate the data set status (signal + sender)
- Increase protocol number to 11 (new merging technology)
proof/inc/TProofPlayer.h, .cxx
- Implement new merging technology for the output list to reduce memory
consumption: add methods to directly merge received objects in the
output list
- Polish cleaning of the various temporary lists and objects
TProofServ.h, .cxx
- Implement new merging technology for the output list to reduce memory
consumption: objects are sent one-by-one
- Do not scan for previous query results at Setup (do it when the client
requests for it first); thsi allows to speed up start-up when a large number
of queries is present and also allos to avoid warning due dictionaries
not yet loaded.
- Add method for a global cleanup of old query results.
- Keep in memory only meta information about processed queries (i.e. not the
output list) to avoid large memory consumption in case of large
output lists
- In Terminate, change the call to Exit(0) to ExitLoop() ; Exit(0) is
called in pmain.cxx .
TProof.cxx
- Implement new merging technology for the output list to reduce memory
consumption: objects are received one-by-one and directly merged in the
output list
- Add protection in parallel startup against invelid slaves.
- Remove check on directory existence in UploadDataSet.
- Add methods to handle notifications from data set verification
- Fix inconsistency with notification in activation/deactivation
of non-existing workers
TXProofServ.cxx
- In Terminate, change the call to Exit(0) to ExitLoop() ; Exit(0) is
called in pmain.cxx.
TQueryResult.cxx
- Make a owned copy of the input list (instead of just storing the pointer) to
allow proper cleaning.
Revision
15656 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jul 3 09:33:50 2006 UTC (8 years, 6 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 125653 byte(s)
Diff to
previous 15649
From Jan:
This patch adds following issues:
- public dataset management
- separation of directories for different users
- adding TProofServ::HandleDataSets method
- spliting communication in UploadDataSet in two parts (checking name
and saving)
- I also added these methods:
virtual Int_t UploadDataSet(const char *dataset,
TList *files,
const char *dest = 0,
Int_t opt = kAskUser,
TList *skippedFiles = 0) = 0;
virtual Int_t CreateDataSet(const char *dataset,
TList *files,
Int_t opt = kAskUser) = 0;
- and added arguments to
TList *GetDataSets(const char *dir = 0);
void ShowDataSets(const char *dir = 0);
Revision
15649 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Jul 1 12:05:49 2006 UTC (8 years, 6 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 123627 byte(s)
Diff to
previous 15524
From Maarten:
- Rename TPacketizer2 to TPacketizer
- Introduce a new development version named TPacketizerDev
- Implement packetizer selection via the PROOF input list.
The packetizer is selected by adding a object like:
TNamed("PROOF_Packetizer","TMyPacketizer")
Revision
15507 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 21 16:18:27 2006 UTC (8 years, 7 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 123270 byte(s)
Diff to
previous 15348
From Gerri:
The patch implements the following new functionality:
- Shutdown of idle sessions when quitting ROOT; if not idle the session
is shutdown as soon as processing terminates; it is possible to delay
the automatic shutdown by a tunable amount of time via a new directive
in the xrootd config file 'xpd.shutdown' (see xpd.cf.example).
- New static method TProof::Reset(const char *url) to hard-cleanup an
user area (implemented for Xproofd only); this implied quite some
additions in XrdProofdProtocol.
- Possibility to temporary remove / re-insert a worker from the active list
TProof::DeactivateWorker(const char *ord)
TProof::ActivateWorker(const char *ord)
with 'ord' the ordinal numer, e.g. "0.3" .
- A default url for the disk pool
const char *TVirtualProof::GetDataPoolUrl()
It is set by default to a string send back by the master, but it can
be changed via
void TVirtualProof::SetDataPoolUrl(const char *url)
Jan should then adapt his code to use this as default.
- Posibility to timeout Collect; for now this is only used at forced
termination.
The patch includes many small changes and bug fixes. One of the most important
is in TXNetSystem (a bug preventing correct treatment of multiple request
to the same redirector).
Revision
15338 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 5 22:51:14 2006 UTC (8 years, 7 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 120980 byte(s)
Diff to
previous 15320
From Gerri:
base/inc/TVirtualProof.h, proof/inc/TProof.h, proof/src/TProof.cxx
proof/src/TProofProgressDialog.cxx:
- fix a problem with the progress dialog when detaching a session:
a new signal CloseProgressDialog() is introduced, as well as the
possibility to empty the list of submitted queries at once via
TProof::Remove().
proofx/inc/TXHandler.h, .cxx:
- add generic argument to HandleInput() and HandleError().
proofx/inc/TXProofMgr.h, .cxx:
- move to new signature for HandleInput(), HandleError().
proof/inc/TSlave.h:
- add method to flush underlying socket.
proofx/inc/TXSlave.h, .cxx:
- move to new signature for HandleInput(), HandleError().
- implement new method to flush underlying socket.
proofx/inc/TXSocket.h, .cxx:
- Add method to flush socket.
- Handle interrupt messages in the reader thread.
- Remove interrupt semaphore (not needed any more).
- Remove timeout argument from GetInterrupt (not needed any more).
proofx/inc/TXProofServ.h, .cxx:
- move to new signature for HandleInput(), HandleError().
- implement interrupt handling in HandleInput().
- implement dedicated version of LockSession to deal with new format
of session tag (this is a bug fix).
proofd/inc/XrdProofConn.h, .cxx:
- Add mutex protection in the SendRecv method (it can be called from both
the reader and main threads).
proofd/src/XrdProofdProtocol.cxx:
- use an asynchronous message (instead of a signal) for interrupts.
proofx/inc/LinkDef.h, TXSocketHandler.h, .cxx, TXUnixSocket.cxx:
- add TXSocketHandler in the dictionary.
- cosmetics.
Revision
15320 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jun 2 23:38:20 2006 UTC (8 years, 7 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 120695 byte(s)
Diff to
previous 15315
From Gerri:
- Get rid of the env to communicate the file descriptor to TProofServ.
Also, get rid of Reads. I had to add an argument to the TProofServ
(and TXProofServ) constructor with the 'FILE *' of the open log file.
Using a global required to link with libProof(x) which we droped recently.
- Drop any (for the time being) unused AFS stuff from TProofServ (this
is postponed).
Revision
15312 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jun 2 15:14:35 2006 UTC (8 years, 7 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 122763 byte(s)
Diff to
previous 14828
From Gerri:
large PROOF patch. Main issues addressed:
- Output redirection to log file at 'proofserv' startup (instead of
after Setup())
- Log messages activated at session creation
- Session sandbox restructuring in such a way that the unique session
tag, created by the top master, uniquely identifies the session on
all nodes, for example on top master:
<User_Sandbox>/session-<top_master_session_tag>/
master-0-<top_master_session_tag>/master-0-<top_master_session_tag>.log
On worker 0.0
<User_Sandbox>/session-<top_master_session_tag>/
worker-0.0-<worker_0.0_session_tag>/worker-0.0-<worker_0.0_session_tag>.log
- Automatic creation of the dataset dir
- Improved handling of error conditions
- Reorganization of TProofServ and XrdProofdProtocol
- Several bug fixes, in particular in XrdProofdProtocol
Revision
14752 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 19 10:57:44 2006 UTC (8 years, 9 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 116855 byte(s)
Diff to
previous 14687
From Gerri:
This is the big patch.
Comments:
- Client side:
- Remove use of plugin manager to avoid unwanted locks
- Improved treatment of error conditions
- Server side:
- Several new functionalities described in the release notes
Revision
14687 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 13 10:27:14 2006 UTC (8 years, 9 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 114920 byte(s)
Diff to
previous 14508
From Jan:
Changes to enable running from GUI instead of interaction on prompt
including adding skippedFiles return argument to UploadDataSet(),
TProof::GetDataSets() and changing ShowDataSet() methods as well as
enum EUploadDataSetAnswer to TVirtualProof.
VerifyDataSet returns now the number of missing files.
Changing all TSocket::Send on client side to Broadcast.
Correcting some synchronization problems and adding comments.
Using TUrl::SetPort to avoid default port 80.
When comparing user answer with "n", "none" was treated as "n".
Now we first compare with "none".
Revision
13838 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jan 17 13:23:29 2006 UTC (9 years ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 107352 byte(s)
Diff to
previous 13616
From Gerri:
As I wrote TXProofServ before the introduction of TProofResources, I forgot
to use the new class in TXProofServ::Setup(). While fixing this I realized
that we do not really need the dedicated method
TProofResourcesStatic::GetWorkDir(), which basically duplicates
TProofResourcesStatic::ReadConfigFile(). Also, I realized that the check
on the master working dir (which requires the inclusion of TProof.h and
TProofServ.h) is actually a tautology and can be removed.
TProofResourcesStatic.h,.cxx
- Remove method GetWorkDir()
- Use 'const char *' in the constructor arguments
- Remove redundant check on master work dir (this allows to drop inclusion
of TProof.h and TProofServ.h)
TProofServ.cxx
- In Setup(), changes needed by the optmization in TProofResourcesStatic
TXProofServ.cxx
- In Setup(), use TProofResourcesStatic to parse the content of the config file
Revision
13616 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Dec 10 16:51:57 2005 UTC (9 years, 1 month ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 107345 byte(s)
Diff to
previous 13565
From Gerri:
introduction of major new PROOF features. Main features:
- PROOF session manager which allows disconnect/reconnect from running
sessions
- use xrootd/xproofd as connection daemon
Much more extensive description in release notes.
Revision
13565 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 9 01:12:17 2005 UTC (9 years, 1 month ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 106709 byte(s)
Diff to
previous 13151
From Paul Nilsson:
new TProofResources base class and TProofResourcesStatic concrete class that
gets the PROOF config info from the static proof.conf file. Future PROOF
resources classes will get resource info from schedulers like LSF, Condor and
xrootd/olbd.
Revision
12722 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Sep 16 08:48:39 2005 UTC (9 years, 4 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 107105 byte(s)
Diff to
previous 12709
From Gerri:
Big patch supporting query queueing on the PROOF master, interogation and
interaction with the queue. Retrieval, finalization and archival of query
results and multiple PROOF sessions. Some other fixes:
- TMacro.h, .cxx
add method Checksum to get the MD5 of the current context
(needed for optimized selector reinitialization)
- Changes needed to create and fill the list of data sets (TROOT,
TDSet, TChain, TChainProof)
- A new method TSelector::IsStandardDraw() to check if a selector
is for standard draw actions
- TEventIter: fix for the missing last update (see above)
- Few protections in TPacketizers destructors
- Fix for code conventions in TPerfStats (the other will follow)
- Occasional missing last update of the progress bar (systematically
reproduced with SetParallel(1)): all this part is quite involved
and I have found a solution which does not harm but perhaps is not
the most elegant.
- SetParallel(0): the progress bar was not working at all in such a
case, since no packetizer is involved. Still it is useful to see
that things are not stuck, so I have added some progress messages
there too. The problem is that I have not found a way to get the total
number of entries from a data set without doing all the machinery
done in the packetizer.
Revision
12561 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Aug 30 10:25:29 2005 UTC (9 years, 4 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 79788 byte(s)
Diff to
previous 12477
From Gerri:
This is the patch to add support for asynchronous queries.
The principal modifications are in TProof and TProofPlayer.
In TProof, I have separated the parsing of the received messages
from the selection of the socket (new method CollectInputFrom()):
the new method is called directly by TProofInputHandler::Notify()
in the case of asynchronous mode.
Also in TProof I have added a method to redirect the logs to a
temporary file, to avoid having them on the main window while doing
something else. The method is TProof::RedirectLog(), it uses freopen,
fdopen and dup2, and works well back and forth.
The content of the temp file can be displayed into a log window
(new GUI class TProofProgressLog), via a new button in the
progress dialog ("Show Logs").
In TProofPlayer I have created a new method Finalize() with the
final part of Process(): this is necessary to be able to finalize
the queries processed asychronously.
There are several many other small changes and fixes: a couple
of non-backward comaptibilities introduced with the previous
patch, many small issues in the GUI (the stop and abort buttons
were never disabled ...).
Also, in view of multi-sessions, it does not make much sense to
have one progress dialog per query: one per session is enough;
I have added a method ResetProgressDialog() to reset it when a
new query is started.
Finally, as anticipation of the next patch, this patch includes
support for a list of queries created by the master and described
by the new class TProofQuery.
The list is displayed by TProof::ShowQueries().
Revision
12477 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Aug 15 15:57:18 2005 UTC (9 years, 5 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 74440 byte(s)
Diff to
previous 12361
From Gerri + some mods by me:
- This patch gets rid of TFTP, slightly modifying SendFile and CheckFile.
- The factorization of the technology dependent parts of TProof, TSlave
and TProofServ. In particular, I have moved the technicalities of Ping
and Interrupt from TProof to two new virtual methods in TSlave, which
will be overwritten in TXSlave.
In TSlave I have introduced the virtual method Init which will be
overwritten by TXSlave and is able to act on an open connection for
optimal fallback a la TXNetFile.
Slaves are created by TProof via the static method TSlave::Create
which uses the plugin manager to detect the presence of TXSlave.
In TProofServ I have made the technology-dependent methods
virtual (Setup, HandleSocketInput, HandleUrgentData, HandleSigPipe,
HandleSocketInputDuringProcess).
Revision
12361 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jul 21 17:41:33 2005 UTC (9 years, 6 months ago) by
brun
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 74406 byte(s)
Diff to
previous 12337
From Marek Biskup:
- removed circular dependency of TDSetElement on TDSet.
- moved friend TDSets into TDSetElements instead of storing them in a TDSet.
- the data from friend TDSets is now copied when adding a friend to a TDSet.
- support for an index in TDSets. There are two cases: 1) there is one Tree
with an index in the friend TDSet; 2) there is 1-1 mapping of files in the
main TDSet and the friend TDSet. Then each index is local.
Revision
12337 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jul 18 16:20:53 2005 UTC (9 years, 6 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 75145 byte(s)
Diff to
previous 12301
From Gerri:
Big patch restructuring the authentication code so libCore does not depend
on libssl etc. anymore. Isolate the current authentication code behind a
plugin library to be loaded on demand.
Setup of a framework able to manage both sets of authentication modules
(the current one and the one based on xrdsec, coming soon).
Cleanup all direct reference to authentication in TSlave and TProofServ,
allowing for significant simplification for the forthcoming changes
in PROOF for XPD.
New module: auth
New files: base/inc/TVirtualAuth.h
auth/Module.Mk
auth/inc/LinkDefRoot.h
auth/inc/TRootAuth.h auth/inc/TRootSecContext.h
auth/src/TRootAuth.cxx auth/src/TRootSecContext.cxx
Moved files:
net/inc -> auth/inc : TAuthenticate.h THostAuth.h DaemonUtils.h AuthConst.h
net/src -> auth/src : TAuthenticate.cxx THostAuth.cxx DaemonUtils.cxx
Revision
12123 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 22 20:18:12 2005 UTC (9 years, 7 months ago) by
brun
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 75248 byte(s)
Diff to
previous 12013
From Constantin Loizides
This patch implements:
- decentralized, automatic mutex initialization,
see R__LOCKGUARD2 in TVirtualMutex
- PROOF parallel startup fixes
- more thread protection (in base, cont, meta, rest to be done)
- cleanups
From Eddy Offermann:
TString::Atoi and Atof are made const.
Revision
11281 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Mar 10 17:57:05 2005 UTC (9 years, 10 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 71156 byte(s)
Diff to
previous 11264
From Marek Biskup:
major new PROOF developments:
- access to PROOF directly via the TChain interface (creating a TDSet
in the background).
- access to PROOF based trees via the TTreeViewer
- fully implemented the TChain/TDSet::Draw() interface.
- PROOF sessions can now be browsed via the TBrowser
Revision
11145 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 18 14:27:33 2005 UTC (9 years, 11 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 67625 byte(s)
Diff to
previous 11069
From Kris Gulbrand:
Added for both TProofServ and TSlave error-checking (for the startup phase)
in case a socket communication fails. Then TProofServ shuts down and TSlave
appropriately reacts. However, since old TProofServ's don't have the new
code, they exit because of a segv since it is early though, the net effect
will only be a printout in the proofd log and the TProofServ still shuts down.
Revision
11047 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Feb 7 18:02:37 2005 UTC (9 years, 11 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 66539 byte(s)
Diff to
previous 10872
From Maarten and Kristjan Gulbrandsen:
Here is the patch with the restructured PROOF code. It parameterizes
PROOF startup, currently implementing "Standard", Condor, MLM and PEAC.
It can easily be extended with other "TProof" classes, e.g. AliEn Proof.
Or what ever next Grid implementation or scheduler comes along.
The MLM, Multi Level Master, code uses a basic packetizer, based on a
MSD (massstorage domain) tag. In MLM mode we have multi-tier client
server setup, like:
client -> super master -> sub master 0 -> slave[0..N]
...
-> sub master M -> slave[0..N]
It also contains a large number of smaller fixes. Credit for most of
the work goes to Kris!
We have tested this on multipe platforms (RH7.3, RH9, SL3) and in
multiple configurations. Should be stable.
Revision
10028 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Sep 13 22:49:10 2004 UTC (10 years, 4 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 53726 byte(s)
Diff to
previous 9689
From Gerri:
Reason: fix security problem with authentication tab and key
files; adopt new naming scheme for auth tab and public
key files, with unique association to the user starting
the daemon.
o proof/src/TProofServ.cxx
- get root of key files from new env ROOTKEYFILE defined
in proofd
o proofd/src/proofd.cxx
- communicate root of key files using new env ROOTKEYFILE
- drop option '-E' to force exclusive tokens (not needed
anymore)
o rootd/src/rootd.cxx
- drop option '-E' to force exclusive tokens (not needed
anymore)
o rpdutils/inc/rpdp.h
- add new method RpdGetKeyRoot() returning the root of public
key files for use in proofd.
- drop mask related to option '-E'
o rpdutils/src/rpdutils.cxx
- add uid of the running process in auth tab and public key
file names (<TmpDir>/rpdauthtab.<uid>,<TmpDir>/rpk.<uid>_<offset>).
- insure <TmpDir>/rpdauthtab.<uid> permission be 0600 .
- eliminate redundant use of 'access()'
- improve check of ownership of the public key file when
checking the token
- drop code related to option '-E'
- few aesthaetic changes (variable naming).
Revision
9689 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Aug 7 18:19:12 2004 UTC (10 years, 5 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 53869 byte(s)
Diff to
previous 9535
From Philippe:
This patch avoids the interpretation of Rtypes.h, TError.h and
TGenericClassInfo.h (since those headers do not contains any class
with a dictionary, they are not marked as 'compiled' header).
In particular this avoid the messages:
Warning: Re-initialization ignored const kMaxUShort
FILE:/home/pcanal/root_working/code/root/include/Rtypes.h LINE:106
etc..
Revision
9322 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jun 25 17:27:09 2004 UTC (10 years, 7 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 53813 byte(s)
Diff to
previous 9316
From Maarten:
- new class TParameter
- new class TDrawFeedback
- fixes to the feedback of performance histograms
- selection of the packetizer
- cleanups
Example commands:
gROOT->Proof()
... package setup ...
TDSet *d = make_tdset(2)
gEnv->SetValue("Proof.StatsTrace",1)
gEnv->SetValue("Proof.StatsHist",1)
TList *fb = new TList
fb->SetName("FeedbackList")
fb->Add(new TObjString("EventsHist"))
fb->Add(new TObjString("CpuTimeHist"))
TDrawFeedback *dfb = new TDrawFeedback((TProof*)gProof)
gProof->AddInput(fb)
d->Process("EventTree_Proc.C","")
Revision
9316 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jun 25 16:49:09 2004 UTC (10 years, 7 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 53216 byte(s)
Diff to
previous 9182
From Gerri:
- "Added support for Blowfish encryption for passwords and
tokens based on OpenSSL, in addition to the existing RSA
encoding. The type of key with highest priority is defined via the
"RSA.KeyType" variable in .rootrc (default is Blowfish).
The server side decides what to use according to its
own availability and the client request."
Revision
9182 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Jun 13 16:26:36 2004 UTC (10 years, 7 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 53102 byte(s)
Diff to
previous 8962
From Maarten and Kristjan Gulbrandsen:
PROOF and file I/O performance monitoring framework. The TVirtualPerfStats
abstract class allows adding monitoring events in base classes (TFile and
TNetfile). The performance monitoring info allows detailed study of PROOF
performance (which files accessed by which slaves, latency, read/write times,
packet size, etc.). The result is send back to the client in the form of a
memory based TTree.
Remove TProofStats.h and TProofStats.cxx, .d, .o from proof/inc and src.
Revision
8561 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 11 18:18:01 2004 UTC (10 years, 9 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 52934 byte(s)
Diff to
previous 8420
From Maarten and me:
in proofd make a dup of the socket away from the stdin, stdout and stderr
standard descriptors to avoid that printfs go to the socket (before this
was done in proofserv itself but this was too late in case there were
prints in the global object ctors.
Also provide TProofServ::GetSessionDir() method.
Also set PROOF_SANDBOX env variable (to SessionDir), this env var can
be used in the package setup scripts.
Revision
8420 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Mar 17 17:52:24 2004 UTC (10 years, 10 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 53131 byte(s)
Diff to
previous 8362
From Gerri:
Improvements:
o Added support for creation of authentication tokens
exclusive to parent daemon and its childs; by default
tokens have inclusive nature, i.e. can be checked by another
server on the same machine with having access to the key files
(option -E to rootd/proofd)
o Added support for non-standard file name for daemon access rules
(option -D <rootdaemonrc> to rootd/proofd)
o Added support for non-standard file name for authentication directives
(PROOF only: option -A [<rootauthrc>] to proofd; see header of
proofd/src/proofd.cxx)
o Improved security in ssh authentications
Bug fixes:
o Fix a few inconsistencies in treating error conditions
o Fix bug with default init strings for ProxyDuration and ProxyKeyBits
o Fix problem with checking for valid globus proxies
o fixed message desynchronization problem in case of non valid offset
o Modified error call in case of failure of Krb5 initialization:
execution continues (call Err instead of ErrFatal)
o Fix (again) problem with cygwingcc definition in rpdutils.cxx
o Fix problem with fgUser not being correctly saved in case
of successful authentication in TAuthenticate.cxx
o Fix a problem with duplication in socket list (TPSocket)
o Set fContext=0 after deletion TPwdCtx (in TSecContext::Deactivate).
o Fix link problem with the globus patched function in rootd/Module.mk
and proofd/Module.mk
o Added function ProofdTerm (equivalent of RootdTerm) to terminate
correctly in case of interrupting signal SIGTERM or SIGINT (proofd.cxx)
o Fix several incorrect comments.
Revision
8239 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Feb 19 00:11:19 2004 UTC (10 years, 11 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 52902 byte(s)
Diff to
previous 7909
Big authentication patch by Gerri:
General purpose:
Add support for 'authenticated' sockets on client side
and prepare for support of server authenticated sockets.
Main changes:
o TSocket, TPsocket:
Added support of 'authenticated' sockets: authentication
is delegated to TSocket by calling a static method
CreateAuthSocket() returning a TSocket*.
TSocket is now the only class calling TAuthenticate.
o TFTP, TNetFile, TSlave:
modified to cope with new socket authentication scheme.
o TAuthDetails replaced by TSecContext containing:
- local authentication info
- expiration time
- pointer to a security context for the method (if any)
- information for cleanup (via TSecContextCleanup)
o A list of valid TSecContext has been added to TROOT,
accessible via gROOT->GetListOfSecContexts().
o THostAuth rewritten using arrays to avoid multiple use
of 'new'; functionality extended by the addition of
several new methods.
o TAuthenticate:
- Loading of information from .rootauthrc has been optimized;
this allowed to simplify method signatures and drop several
methods. Also collection and transmission of authentication
information from TSlave to TProofServ gets simplified by the
new scheme.
- Added support for server dependent directives in .rootauthrc
- Added check for changes in .rootauthrc each time a new
TAuthenticate is instantiated, so that changes in the file
are picked-up interactively.
- Checking for an existing TSecContext (previously TAuthDetails)
has been improved
- Improved diagnostics for SshAuth
- Enlarged support for authentication method indication in protocol:
UsrPwd: rootup/proofup
SRP: roots/proofs (already supported)
Krb5: rootk/proofk (already supported)
Globus: rootg/proofg
SSH: rootsh/proofsh
UigGid: rootug/proofug
o GlobusAuth.cxx, Krb5Auth.cxx, SRPAuth.cxx
- Adapted to new class TSecContext
- Added support for improved search for reusable TSecContext
- Drop globals used to store established security contexts
(saved in TSecContext) and related methods
o Automatic creation of <RootDir>/etc/system.rootauthrc from
compilation flags; this has priority on old envs
'Rootd.Authentication' and 'Proofd.Authentication' and is
superseded by $HOME/.rootauthrc
o Automatic creation of <RootDir>/etc/system.authdaemonrc from
compilation flags; this is superseded by $HOME/.rootauthrc
o rpdutils:
- several almost identical methods moved in from rootd/proofd
- several new methods to simplify manipulation of the
authentication tab file
- moved (and improved) check for running of sshd in RpdSshAuth,
performed only in case of failure (to avoid spurious messages
printed by sshd in /var/log/messages)
o proofd.cxx, rootd.cxx
- increased protocol version
- all initialization stuff (included authentication and
login) done via RpdInitSession
- postpone opening of parallels sockets after authentication
(to solve problem of Kerberos/Globus authentication with
parallel sockets)(rootd)
- Fixed problem with kerberos and globus authentication in rootd
with parallel sockets.
Revision
7880 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Jan 10 10:52:31 2004 UTC (11 years ago) by
brun
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 71711 byte(s)
Diff to
previous 7875
From Philippe:
This mega patch introduces a new implementation of the STL I/O
which is backward and forward compatible. In addition this is more
exactly a new implementation or an extension of the container I/O.
We are introducing a new abstract interface:
"TVirtualContainerProxy", which can be implemented to present a
proxy to any collection which the I/O (and TTreeFormula) can use
then transparently.
The TVirtualContainerProxy interface allows to the I/O system to
request some information and to execute some essential function of
the container:
what kind of object/data does it contain
does it contain pointers
how to insert data into the container
how to retrieve an array of pointer to the elements inside
how to create a collection object
how to clear the collection
how to resize the collection
how to streamer the collection (if needed)
how to calculate the sizeof the collection
how to calculate the number of elements of the collection.
Using those primitives, the I/O and TTreeFormula should be able to
access any collection. The I/O should also be able to handle the
splitting of collections that can be split (i.e. contains a single
type of object/data).
The current compromise selected between code duplication,
performance of the I/O of TClonesArray and vector of pointers and
the performance of the I/O of other containers, was to have on
function handle all collection as if they were a C-style array of
pointers to data. This implies for example that the I/O of vector
of object current passes via the construction of temporary array
of pointer. The cost of this construction is usually ~Qjust~R the
cost of calculating the addresses of the elements and assigning it
to an array element.
Registering a collection proxy will be similar to
static int dummy = GenerateInitInstance((CollectType*)0x0)-
>AdoptCollectionProxy(new CollectTypeProxy));
Follows a few details on the modifications made to some of the
files and classes.
Bytes.h:
Work around a problem in the MSVC++ 6.0 optimizer. This should
not affect other compilers.
String:
Included the std::string dictionary into G__Base2.cxx, this
insures its presence at all time.
Added a new file string.cxx holding the streamer for
std::string.
RConfig.h
Added proper ansi stream configuration for AIX, KCC
Added template specialization defect for MSVC
TBrowser
Start adding the ability to browser non TObject classes
TBuffer
To handle the reading and writing array of objects, added:
Int_t CheckByteCount(UInt_t startpos, UInt_t bcnt, const
TClass *clss, const char* classname);
Int_t CheckByteCount(UInt_t startpos, UInt_t bcnt, const
char *classname);
void ReadFastArray(void *start , TClass *cl, Int_t n=1,
TMemberStreamer *s=0);
void ReadFastArray(void **startp, TClass *cl, Int_t n=1,
Bool_t isPreAlloc=kFALSE, TMemberStreamer *s=0);
void WriteFastArray(void *start, TClass *cl, Int_t n=1,
TMemberStreamer *s=0);
Int_t WriteFastArray(void **startp, TClass *cl, Int_t n=1,
Bool_t isPreAlloc=kFALSE, TMemberStreamer *s=0);
TROOT
Enhancement to make the access to gROOT not dependent for the
library loading order. In particular we added:
ROOT::GetROOT()
which should be used instead of gROOT.
Improve support for STL. In particular, now STL containers do
have a corresponding TClass object
TRealData
Replace SetStreamer by AdoptStreamer that allow to use not only
a streamer function but also streamer object (allowing streamer
with a state for Foreign classes)
TString:
Improve streamer performance
TSystem:
More consistency of the return value of TSystem::CompileMacro
build/unix/makecintdlls.sh
Stop making the string.dll
config:
enhance sgicc makefiles
cont:
fix TBits::operator=
TClassTable now warns for duplicate only for non stl classes
TClonesArray fix a typo
gpad:
Add a new class TInspectObject to enable inspect non TObject
classes
TRootBrowser : enable inspect non TObject classes
TFormula/TTreeFormula
To enhance performance switch from using one array fOper which
contained the operation code and its arguments to using 2 arrays
fActions and fActionParams to store respectively the operation and
its parameters.
A new function Convert is used to convert old version from the
fOper to fActions. This allows cleaner coding and offer
optimization opportunities.
TTreePlayer
Start adding support in MakeClass for STL containers.
TRint/TProofServ
Insure the loading of the string support
Event.cxx
make sure to avoid memory over-write
stress.cxx
Add new parameters
stress <nevent> <style> <subbench> <portion>
if <subbench> is 1 or higher, print the benchmark results after
each test. This allows understand
which test is affect by a performance change.
portion is a binary field to indicate which test to run. This
allows to focus on a particular test.
TVirtualCollectionProxy
Abstract interface used to access any type of containers from
the I/O system and TTreeFormula. See TVectorProxy and
TEmulatedVectorProxy for examples.
TEmulatedVectorProxy
Implementation of a std::vector proxy to be able to read a
std::vector without any libraries.
TVectorProxy
Implementation of TVirtualCollectionProxy for a std::vector for
which we have the library.
TStreamerInfo.cxx
Split in 3 files: TStreamerInfo.cxx
TStreamerInfoReadBuffer.cxx TStreamerInfoWriteBuffer.cxx
All the ReadBuffer, ReadBufferClones and the new ReadBufferSTL
(similarly for WriteBuffer) have been factorized into one
function and 2 short wrappers. The new main function expect an
array of pointer to the objects (this array is most often of size
one).
TClonesArray objects pass GetObjectRef(0) to the common
ReadBuffer
vector<bla*> v pass &(v[0])
vector<bla> needs to create an intermediary array to hold the
address
This mechanism is sometimes not optimal (vector<blah>) but
allows extremely flexibly and extension. Basically, to add
support for a new container type using the StreamerInfo mechanism
(in particular allows splitting), one 'just' need to implement a
TVirtualCollectionProxy, which, in particular, will return an
array of address to the object it contains. Even std::map can be
handled with this mechanism, since, for I/O purposes, it can be
consider as a container of pairs.
Add a few optimization, including more caching via a new array
of a new struct (fComp).
Fixed a problem (re)introduced while implementing the Foreign
class CheckSum. Doing:
class Event;
f = new TFile("Event.root");
resulted in errors.
TCint
Add proper support for TClass of STL containers. Fix a memory
leak.
Add support for load TClass via a typedef.
Fix a problem with multiple inheritance
TClass
Fixed a problem (re)introduced while implementing the Foreign
class CheckSum. Doing:
class Event;
f = new TFile("Event.root");
resulted in errors.
Add a
TClass/TGenericClassInfo/TDataMember
Add support for a new interface (TVirtualCollectionProxy)
useable for STL containers or any user defined containers.
Add support for streamer with are objects (as opposed to free
standing function or methods). This allows the user a greater
flexibility in writing a streamer.
Add a few optimizations
Add CanSplit to answer the question for a whole Class (for
example some collection can not be splitted).
TClassStreamer
New class to wrap a class wide streamer
ClassStreamerFunc_t
typedef for a class wide streamer function
TMemberStreamer
New class to wrap a specific class member streamer
MemberStreamerFunc_t
typedef for a specific class member streamer function
RootStreamer
Macro to specify a free standing streamer object or function
for a class
For example:
RootStreamer(string,std_string_streamer);
TStreamerElement:
A couple of optimization/simplification.
Add support for the new STL I/O
Extend the useful
TBranchElement:
add a connection to the proper TVirtualCollectionProxy
add support for STL containers (non-split and split mode)
TTree
Make the function TTree::GetMakeClass NON-virtual for better
efficiency
Add support for STL containers
TBasket
Left (in comment) a yet unproved improvement proposed by
Victor. The preliminary tests were inconclusive performance wise
and it had (seemingly) problem with backward and forward
compatibility.
TBranch
Performance improvements
metautils
This is a new package for C++ files shared between rootcint and
meta.
It contains TClassEdit a class to help determine some property
of a class given its class name (stl, template, etc).
utils
Introduced a new file RStl.cxx to start separating rootcint in
modules.
Modified rootcint to support the new STL I/O methods.
In particular a new class RStl is in charge of doing the
generating of code for STL containers.
Revision
7810 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Dec 30 13:16:51 2003 UTC (11 years ago) by
brun
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 71637 byte(s)
Diff to
previous 7682
Mega patch to add support for large files (bigger than 2 GBytes)
----------------------------------------------------------------
==========>NEW VERSION 4.00/00 <===========
Because this new version has a considerable number of changes,
and new important additions by Philippe are scheduled for the coming days
we are starting a new major version 4.
-Support for large files
-Automatic schema evolution for foreign classes
-New data type Double32_t
Large files are currently tested only under Linux with gcc3.2.
Support for other systems will be gradually added in the coming days.
By default under Linux, files are created with the option LARGEFILE.
Note that when creating a Tree, the default maximum size for a Tree
is set to 1.9 GBytes. You can change this default value by calling
TTree::SetMaxTreeSize(Long64_t maxsize)
The implementation of this new feature has required changes in many places.
The files produced by this new version can still be read by old ROOT versions
if their size if smaller than 2 GBytes. And obviously, the new version
can digest old ROOT files.
WARNING: note the following backward incompatibility:
-TSystem::GetPathInfo has a new signature: (also TUnixSystem, TWinNTSystem, etc)
old: GetPathInfo(const char *path, Long_t *id, Long_t *size, Long_t *flags, Long_t *modtime);
new: GetPathInfo(const char *path, Long_t *id, Long64_t *size, Long_t *flags, Long_t *modtime);
-TFile::SysSeek and TFile::SysStat have a new signature (also TNetFile, TWebFile, etc)
old: Int_t SysSeek(Int_t fd, Long_t offset, Int_t whence);
Int_t SysStat(Int_t fd, Long_t *id, Long_t *size, Long_t *flags, Long_t *modtime);
new: Long64_t SysSeek(Int_t fd, Long64_t offset, Int_t whence);
Int_t SysStat(Int_t fd, Long_t *id, Long64_t *size, Long_t *flags, Long_t *modtime);
-TTree::SetMaxTreeSize has a new signature:
old: void TTree::SetMaxTreeSize(Int_t maxsize)
new: void TTree::SetMaxTreeSize(Long64_t maxsize)
All references to Seek_t have been replaced by Long64_t.
Revision
7656 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 26 10:33:08 2003 UTC (11 years, 2 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 71565 byte(s)
Diff to
previous 7556
From Gerri:
- remove globals in Krb5Auth, SRPAuth and GlobusAuth
- error printing in case of failure in TAuthenticate
(print a list of failed messages only if unsuccessful
in the end)
- fix a problem with globus credential transmission when
running as non-root
- a couple of protections in TSlave related to SRP
- a few fixes to avoid some spurious error messages
printed by proofd
Revision
7534 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Nov 7 03:29:42 2003 UTC (11 years, 2 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 71453 byte(s)
Diff to
previous 7530
From Gerri:
Index: README/README.AUTH
o Update explaining tutorial use
Index: base/src/TFile.cxx
o ::Open(): call TNetFile also if same host but different user
(to access files protected by password)
Index: globusauth/src/GlobusAuth.cxx
o Aestethic changes (shortening of a few lines)
Index: krb5auth/src/Krb5Auth.cxx
o Simplify two debug printouts
o Change debug print to stderr in Info(...; add gDebug control.
Index: net/inc/THostAuth.h
o Modify default ctor
o Add method HasMethod to test method availability
Index: net/src/TAuthenticate.cxx
o make sure that if fgUser (and nothing else) is set, this is
the default user name
o Make sure that requiring roots/rootk uses correctly details
found for those methods in .rootauthrc
o Fix return of GetPromptUser()
o Fix problem with wild cards
o Add check for wildcard host names in method GetHostAuth
o Remove debug printout in method ReadAuthRc
Index: net/src/THostAuth.cxx
o Modify default ctor
o Add method HaveMethod to test method availability
o Aestethic changes (shortening of a few lines)
Index: net/src/TNetFile.cxx
o Remove precompiler unused #if's in method ConnectServer
Index: proof/inc/TProofServ.h
o Remove method ReadProofAuth, add method RecvHostAuth
(To receive directly the authentication info in ProofServ
instead of reading it from temporary files
/usr/tmp/proofserv.<id>)
Index: proof/src/TProofServ.cxx
o All mods needed to remove method ReadProofAuth and add
method RecvHostAuth (see above).
Modified methods: Setup, CollectAuthInfo .
o Fix bug in CollectAuthInfo switching ReUse off by default
for method UsrPwd
o Removing automatic choice of UidGid for local slaves
o Setting order of default UidGid authentication to last
o Aestethic changes in CheckAuth
Index: proof/src/TSlave.cxx
o transmit passwd for UsrPwd and SRP authentication also to slaves
o All mods needed to send auth info to ProofServ instead of
proofd (see above)
Index: proofd/src/proofd.cxx
o Fix compiling problem with namespaces for Kerberos
o Remove redundant 'access' calls
o Authenticate: do not recive auth info (done in ProofServ)
o ProofExec: send TmpDir to proofserv (for key location)
o Define gSystemdaemonc as $ROOTETCDIR/system.rootdaemonrc
Index: rootd/src/rootd.cxx
o Define gSystemdaemonc as $ROOTETCDIR/system.rootdaemonrc
Index: rpdutils/inc/rpdp.h
o Rename gAuthAllow as gSystemDaemonRc
o Rename kDaemonAccess as kDaemonRc
Index: rpdutils/src/rpdutils.cxx
o Fix compiling problem with namespaces for Kerberos
o Define kDaemonRc as .rootdaemonrc (it was daemon.access)
o Add support for $HOME/.rootdaemonrc
o Add support for daemon dependent access rules
o Aestethic changes in CheckAuth (drop commented lines)
Index: srputils/src/SRPAuth.cxx
o set static flags only when authentication is successful
Index: tutorials/TestAuth.C
o Test using TFTP instead of TNetFile (faster and runs also
with local host fqdn)
o Add test for kerberos
Revision
7456 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 22 18:48:36 2003 UTC (11 years, 3 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 73410 byte(s)
Diff to
previous 7409
From Gerri:
Patched files:
configure
o improved detection of patch details for globus
README/README.GLOBUS
o updated
etc/hostcert.conf
o add example for non-root runs
globusauth/src/GlobusAuth.cxx
o remove redundant 'access' calls
net/src/TAuthenticate.cxx
o fix return value in method GetPromptUser
proofd/src/proofd.cxx
o remove redundant 'access' calls
rpdutils/inc/rpdp.h
o add new includes file depending on Globus GTK version
o add prototype for new function GblsToolCheckProxy
rpdutils/src/globus.cxx
o fix a few typos in debug messages
o add new function GblsToolCheckProxy for checking user
proxies when running as non-root
o remove redundant 'access' calls
rpdutils/src/rpdutils.cxx
o remove redundant 'access' calls
Revision
7409 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Oct 7 21:09:55 2003 UTC (11 years, 3 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 73334 byte(s)
Diff to
previous 7406
From Gerri:
Index: README/README.AUTH
o Updated
Index: globusauth/src/GlobusAuth.cxx
o Drop use of environment variables (DEFAULTUSER,AUTHREUSE,PROMPTUSER)
Index: krb5auth/src/Krb5Auth.cxx
o Drop use of environment variables (DEFAULTUSER,AUTHREUSE,PROMPTUSER)
Index: net/inc/TAuthenticate.h
Index: net/src/TAuthenticate.cxx
o Drop use of environment variables (DEFAULTUSER,AUTHREUSE,PROMPTUSER)
o Fix problem with defaults
o Fix problem with method ReadAuthRc when reading an empty .rootauthrc
o Fix problem with method ReadAuthRc when updating information for an
existing THostAuth
Index: proof/inc/TProof.h
o Propagation of new fSRPPwd flag
Index: proof/inc/TProofServ.h
o Propagation of new fSRPPwd flag
Index: proof/src/TProofServ.cxx
o Propagation of new fSRPPwd flag
Index: proof/src/TSlave.cxx
o Propagation of new fSRPPwd flag
Index: rpdutils/src/rpdutils.cxx
o Fix small problem with uninitialized variables to avoid
compiler complains (from Ioannis Papadopoulos).
Index: srputils/src/SRPAuth.cxx
o Drop use of environment variables (DEFAULTUSER,AUTHREUSE,PROMPTUSER)
Revision
7406 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Oct 7 14:03:03 2003 UTC (11 years, 3 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 73293 byte(s)
Diff to
previous 7373
From Gerri:
clib/inc/rsalib.h, clib/src/rsalib.c
o Random seed initialization moved from rsalib.c to in rootd.cxx,
proofd.cxx and TAuthenticate.
o Introduced counter to avoid infinite loops during key generation
etc/rpdauth.allow
o Introduced default with all authentication methods allowed in
rpdauth.allow
globusauth/src/GlobusAuth.cxx
o Secure RSA public transmission to servers (receiving a server
key)
o Added test on Argc() before use of the Argv() arguments
o Use isatty to check if interactive session
krb5auth/src/Krb5Auth.cxx
o Secure RSA public transmission to servers (receiving a server
key)
o Added test on Argc() before use of the Argv() arguments
o Use isatty to check if interactive session
net/inc/TAuthenticate.h, net/src/TAuthenticate.cxx
o Secure RSA public transmission to servers (receiving a server
key)
o Added possibility for a "default" THostAuth instantiation valid
for all hosts if anything more specific is not found.
Used by TProofServ/TAuthenticate for host without specific
authentication directives in proof.conf .
o Random seed initialization moved from rsalib.c to in rootd.cxx,
proofd.cxx and TAuthenticate.
o Introduced counter to avoid infinite loops during key generation
o Fixed potential problem with SSH master-to-slave authentication;
force failure if, for some reason, the slave is not accepting
the standard key files (depending on the user ssh config
directives, this could cause interactive prompt for password
on the master, therefore hanging the system).
Specific directives to avoid prompting are passed via command
line with the '-o' ssh option.
o Fixed problem with forming the message used to notify an SSH
failure
o Use mkstemp instead of tmpnam
o Use isatty to check if interactive session
o Avoid creating duplicates of TAuthDetails instantiation for
offset = -1
o Fixed a few inconsistency of debug levels error messages after
failed authentications
o Fixed a few inconsistency between authentication method names
and ROOT environment variable names
o Fixed problem with null passwd sending in ClearAuth
o Fixed problem with method attempting order during negotiation
o Add possibility to avoid check on localhost for testing purposes
(in method GetAuthMeth)
net/src/THostAuth.cxx
o Added possibility for a "default" THostAuth instantiation valid
for all hosts if anything more specific is not found.
Used by TProofServ/TAuthenticate for host without specific
authentication directives in proof.conf .
proof/inc/TProof.h, proof/src/TProof.cxx
o Moved collection of Authentication information for slaves from
TProof to TProofServ
proof/inc/TProofServ.h, proof/src/TProofServ.cxx
o Improved transmission of RSA public key between TSlave and
TProofServ
o Moved collection of Authentication information for slaves
from TProof to TProofServ
o Added test on Argc() before use of the Argv() arguments
proof/src/TSlave.cxx
o Improved transmission of RSA public key between TSlave and
TProofServ.
Added possibility to securely forward the SRP password to
the master (ROOT environment variable: Proof.SendSRPPwd,
default 0 - off).
o Removed checks on Proofd.Authentication in TSlave; security
stuff is setup in TProofServ and used in TAuthenticate
proofd/src/proofd.cxx
o Random seed initialization moved from rsalib.c to in rootd.cxx,
proofd.cxx and TAuthenticate.
rootd/src/rootd.cxx
o Random seed initialization moved from rsalib.c to in rootd.cxx,
proofd.cxx and TAuthenticate.
rpdutils/inc/rpdp.h, rpdutils/src/rpdutils.cxx
o Secure RSA public transmission from clients (sending a server
key)
o Fixed ambiguity with method index resolving from name due to
the fact that 'usrpwd' contains the string 'srp'
o Add file existence check before trying open
o Add a few debug messages
o Fixed problem with domain checking in daemon.access
('cern.ch uidgid' or '.cern.ch uidgid' were not accepted)
o Fixed problem with null passwd receiveing in RpdUser
rpdutils/src/ssh.cxx
o Fixed function name in debug message in ssh.cxx
o Use mkstemp instead of mktemp
srputils/src/SRPAuth.cxx
o Secure RSA public transmission to servers (receiving a server
key)
Revision
7275 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Sep 12 17:36:35 2003 UTC (11 years, 4 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 57786 byte(s)
Diff to
previous 7262
Few additional patches from Gerri:
- TSlave.cxx, TProofServ.cxx
send over the RSA key only when needed to decode the passwd
(ie UsrPwd and SRP)
- TAuthenticate.cxx
allow short host name in .netrc family files
- TProof.cxx
force using UidGid when the slave is on the same node as the master
- SRPAuth.cxx
Drop a left-over debug message
Revision
6243 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Mar 4 17:29:29 2003 UTC (11 years, 10 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 46904 byte(s)
Diff to
previous 5717
implementation of package manager. Package manager allows the distribution,
installation and setting up of user code in the PROOF sandbox environment.
The package manager distributes so called PAR files, which are standard
tarred and gziped files, but with extension .par. A file "mylibs.par" is
required to untar into a directory "mylibs". A PAR file contains in addition
to the user's code and libraries one special directory PROOF-INF that contains
two scripts: BUILD.sh and SETUP.C. BUILD.sh is a shell script that is
used to build the user's libraries on the PROOF cluster (which might be
of a different architecture/compiler then the user's machine). SETUP.C is
a ROOT macro that is used to load the user's libraries, built by BUILD.sh,
and to do any other kind of setup needed to run with the user libraries.
These scripts are executed in the package directory (not the PROOF-INF
directory). An example of a BUILD.sh is:
#! /bin/sh
make libEvent.so
An example of SETUP.C is:
void SETUP()
{
gSystem->Load("libEvent");
}
The package manager API is:
gProof->UploadPackage()
gProof->EnablePackage()
gProof->ClearPackage()
Revision
5717 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Dec 2 18:50:12 2002 UTC (12 years, 1 month ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 46105 byte(s)
Diff to
previous 5615
mega patch to remove almost all compiler warnings on MacOS X where the
compiler is by default in pedantic mode (LHCb also like to use this option).
The following issues have been fixed:
- removal of unused arguments
- comparison between signed and unsigned integers
- not calling of base class copy ctor in copy ctor's
To be done, the TGeo classes where we get still many hundred warnings of
the above nature. List forwarded to Andrei.
Revision
5385 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 3 18:06:46 2002 UTC (12 years, 3 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 45595 byte(s)
Diff to
previous 5336
create more explicit session directory names for the master and slaves.
Work around an issue with deletion of session directories when they are on nfs.
Pass valid passwd from master to slaves. This passwd can be used to authenticate
to rootd to open files.
More explicit info messages with slave and hostname mentioned.
Revision
4955 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jul 17 12:29:38 2002 UTC (12 years, 6 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 40744 byte(s)
Diff to
previous 4709
PROOF mods from Maarten. New TVirtualPacketizer that is the base class for
different packetizers. You want different packetizers for different
processing strategies (depending on files being available on disk or if
they need to be sequentially retrieved from mass storage, etc.).
Added global PROOF debug facility.
Revision
4210 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Mar 20 18:54:57 2002 UTC (12 years, 10 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 32419 byte(s)
Diff to
previous 4193
add cases for Kerberos5 authentication. In particular the default authentication
method for rootd and proofd can now be set in system.rootrc. The default is
clear text passwd, but SRP and Krb5 can be specified. The default is used when
"root:" or "proof:" appears in the url. The default can be overridden by
specifying explicitely "roots:" or "rootk:", idem for proof.
Revision
4190 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Mar 16 18:36:52 2002 UTC (12 years, 10 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 32099 byte(s)
Diff to
previous 4182
ProofServErrorHandler() now adds to the Info, Warning and Error messages
the master or slave id. Remove private Info version since now in TObject.
Fix handling of kMESS_CINT messages, correctly forward using Exec() instead
of SendCommand() which did not relay the scripts (like in ".x aap.C") to
the slaves.
Revision
4182 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Mar 15 17:23:40 2002 UTC (12 years, 10 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 32964 byte(s)
Diff to
previous 4167
implemented TProof::CheckFile() which is called by SendFile() to make sure
files are not send when not needed. They are only send when the md5 is
changed. It also makes a check on the remote nodes with unique file system
to see if the file is already there (by sending the md5 over in a message
of type kPROOF_CHECKFILE).
Revision
3699 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jan 18 14:24:10 2002 UTC (13 years ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 32006 byte(s)
Diff to
previous 3657
relocate some proof file to remove circular dependencies between the
treeplayer and proof shared libraries. Also rename TPlayer to TProofPlayer
et al. Also remove Begin() from TSelector, use Begin(0) instead. This to
be backward compatible with existing TSelector scripts that don't have
Begin() and therefor give warnings about Begin() being hidden in the base
class.
Revision
2940 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Oct 2 09:09:54 2001 UTC (13 years, 3 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 30591 byte(s)
Diff to
previous 1474
call TApplication base class with numOptions<0 to avoid calling
TApplication::GetOptions() and instead call our own GetOptions().
The problem was that TApplication::GetOptions() was "eating" the
PROOF config directory argument.
Revision
1205 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 13 15:13:57 2000 UTC (14 years, 1 month ago) by
brun
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 30155 byte(s)
Diff to
previous 1203
W A R N I N G !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================
A very long list of changes in this pre-release of version 3.00.
We have modified the signature of many functions (in particular TObject)
to introduce more constness in the system.
You must change your code if your class derives from TObject and uses
one of the modified functions such as ls, Print, Compare, Hash, etc.
The modified functions in TObject have the following signature:
virtual TObject *Clone() const;
virtual Int_t Compare(const TObject *obj) const;
virtual void Delete(Option_t *option=""); // *MENU*
virtual void DrawClass() const; // *MENU*
virtual void DrawClone(Option_t *option="") const; // *MENU*
virtual void Dump() const; // *MENU*
virtual TObject *FindObject(const TObject *obj) const;
virtual char *GetObjectInfo(Int_t px, Int_t py) const;
virtual ULong_t Hash() const;
virtual void Inspect() const; // *MENU*
virtual Bool_t IsEqual(const TObject *obj) const;
virtual void ls(Option_t *option="") const;
virtual void Print(Option_t *option="") const;
A similar operation has been done with classes such as TH1, TVirtualPad,
TTree, etc.
Revision
106 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Jun 11 12:25:48 2000 UTC (14 years, 7 months ago) by
rdm
Original Path:
trunk/proof/src/TProofServ.cxx
File length: 21104 byte(s)
Diff to
previous 3
now use 1093 as port number for proofd service. This port number has been
assigned by IANA for proofd usage. In the old scheme we used two ports
for PROOF: one for the master server (proofserv) and another for the client
servers (proofslave). This has now been changed to a single port (proofd)
and the server type is communicated by the client to the server.
Update the /etc/services file from:
proofserv 349/tcp
proofslave 359/tcp
rootd 432/tcp
to
proofd 1093/tcp
rootd 1094/tcp
And /etc/inetd.conf from:
proofserv stream tcp nowait root /usr/proof/bin/proofd proofserv /usr/proof
proofslave stream tcp nowait root /usr/proof/bin/proofd proofslave /usr/proof
to
proofd stream tcp nowait root /usr/proof/bin/proofd /usr/proof
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
