[root] / trunk / proof / proofx / src / TXProofServ.cxx Repository:
ViewVC logotype

Log of /trunk/proof/proofx/src/TXProofServ.cxx

Parent Directory Parent Directory


Links to HEAD: (view) (download) (as text) (annotate)
Sticky Revision:

Revision 47991 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Dec 12 18:40:25 2012 UTC (2 years, 1 month ago) by rdm
File length: 38443 byte(s)
Diff to previous 47729
Rename TCINTMutex to TClingMutex.

Revision 47729 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Nov 30 15:43:22 2012 UTC (2 years, 1 month ago) by ganis
File length: 38442 byte(s)
Diff to previous 45283
Fix a backward incompatibility introduced by patch 45283

Revision 45283 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Jul 30 10:42:38 2012 UTC (2 years, 5 months ago) by ganis
File length: 37862 byte(s)
Diff to previous 44683

   Patch automatizing the usage of the file-based technology to handle outputs.
   After this patch PROOF users can define as an option to Process an output file for their results
   and choose if the merging goes via file or in memory. User can also decide to save theit results
   in a set of files identified by a name (dataset).
   Administrators can force merging-via-file or can make it conditional to the resident memory exceeding
   a given threshold.

   The new options are documented at
                                         http://root.cern.ch/drupal/content/handling-outputs

   Since TProofOutputFile is now used by TProof, the class has been moved from proof/proofplayer
   (libProofPlayer) to proof/proof (libProof).
   The selector tutorials/proof/ProofSimple.h,.C has been modified to allo testing the new options,
   and a new test has been added to stressProof.

Revision 44683 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Jun 14 09:27:40 2012 UTC (2 years, 7 months ago) by ganis
File length: 37680 byte(s)
Diff to previous 44655
   First version of the final patch enabling multi-workers on single node using
   PROOF-Lite. This is supposed to improved scalability for large number of workers
   (>300-400) and is currently available only for non-data driven tasks.
   The new setup configuration is enabled by the directive 'xpd.remoteplite 1' in
   the xproofd configuration file.

Revision 44655 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Jun 12 07:43:53 2012 UTC (2 years, 7 months ago) by ganis
File length: 36915 byte(s)
Diff to previous 44521
Remove the obsolete fInflateFactor

Revision 44521 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Jun 4 15:09:53 2012 UTC (2 years, 7 months ago) by ganis
File length: 37056 byte(s)
Diff to previous 44507
Additional fixes fro Coverity issues

Revision 44507 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Jun 4 12:30:41 2012 UTC (2 years, 7 months ago) by axel
File length: 37053 byte(s)
Diff to previous 44491
Remove
  using namespace std;
from Riostream.h, which has huge consequences for all of ROOT.
Riostream.h is now a simple wrapper for fstream, iostream, iomanip for backward compatibility; Riosfwd.h simply wraps iosfwd.

Because of templates and their inline functions, Riostream.h needed to be included in headers, too (e.g. TParameter.h), which violated the assumption that Riostream.h is not exposing its using namespace std to headers.
ROOT now requires R__ANSISTREAM, R__SSTREAM, which does not change the set of supported compilers.

Without "using namespace std", several identifiers are now prefixed by std::; e.g. roofit/* source files now have a using namespace std to keep their coding style.
TFile::MakeProject() now generates "using namespace std" to convert the CINT-style class names into C++ ones.

Revision 44491 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Jun 4 08:33:29 2012 UTC (2 years, 7 months ago) by ganis
File length: 37043 byte(s)
Diff to previous 39696
Fix for SECURE_CODING coverity issues

Revision 39696 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Jun 13 21:35:34 2011 UTC (3 years, 7 months ago) by pcanal
File length: 37031 byte(s)
Diff to previous 39115
From David Dagenhart:

Implement an option to allow ROOT to use the LZMA compression
algorithm to compress data instead of the ZLIB compression algorithm.
LZMA compression typically results in smaller files, but takes more
CPU time to compress data. To use the new feature, the external XZ
package must be installed when ROOT is configured and built:

Download 5.0.3 from here http://tukaani.org/xz/
and make sure to configure with fPIC:
   ./configure CFLAGS='-fPIC'

Then the client C++ code must call routines to explicitly request LZMA
compression.

ZLIB compression is still the default.

Setting the Compression Level and Algorithm:

There are three equivalent ways to set the compression level and 
algorithm. For example, to set the compression to the LZMA algorithm
and compression level 5.

1. TFile f(filename, option, title);
   f.SetCompressionSettings(ROOT::CompressionSettings(ROOT::kLZMA, 5));

OR

2. TFile f(filename, option, title, ROOT::CompressionSettings(ROOT::kLZMA, 5));

OR

3. TFile f(filename, option, title);
   f.SetCompressionAlgorithm(ROOT::kLZMA);
   f.SetCompressionLevel(5);

These methods work for TFile, TBranch, TMessage, TSocket, and TBufferXML. 
The compression algorithm and level settings only affect compression of
data after they have been set. TFile passes its settings to its branches 
only at the time the branches are created. This can be overidden by 
explicitly setting the level and algorithm for the branch. These classes 
also have the following methods to access the algorithm and level for 
compression.

   Int_t GetCompressionAlgorithm() const;
   Int_t GetCompressionLevel() const;
   Int_t GetCompressionSettings() const;

If the compression level is set to 0, then no compression will be
done. All of the currently supported algorithms allow the level to be
set to any value from 1 to 9. The higher the level, the larger the
compression factors will be (smaller compressed data size). The
tradeoff is that for higher levels more CPU time is used for
compression and possibly more memory. The ZLIB algorithm takes less
CPU time during compression than the LZMA algorithm, but the LZMA
algorithm usually delivers higher compression factors.

The header file core/zip/inc/Compression.h declares the function
"CompressionSettings" and the enumeration for the algorithms.
Currently the following selections can be made for the algorithm:
kZLIB (1), kLZMA (2), kOldCompressionAlgo (3), and kUseGlobalSetting
(0). The last option refers to an older interface used to control the
algorithm that is maintained for backward compatibility. The following
function is defined in core/zip/inc/Bits.h and it set the global
variable.

   R__SetZipMode(int algorithm)

If the algorithm is set to kUseGlobalSetting (0), the global variable
controls the algorithm for compression operations.  This is the
default and the default value for the global variable is kZLIB.

Note that the LZMA algorithm will only be available if the LZMA
libraries from the XZ package were available when the ROOT executable
being used was configured and built. If you are building ROOT and want
LZMA then you must do something similar to the following if XZ is not
already installed on your system.

  Download XZ version 5.0.3 from http://tukaani.org/xz/
  unwind the tarball
  cd xz-5.0.3
  ./configure CFLAGS='-fPIC'
  make

ROOT was tested with version 5.0.3 of XZ.  It might work with earlier
versions and will probably work with later versions of XZ.

Then either use "make install" to put the library and headers into
/usr/local/* so the scripts building ROOT can find them or use
arguments to the ROOT configure script to point to the proper
directories containing the XZ library and headers. These are the
library and the included header (lzma.h includes other headers in the
lzma subdirectory).

  src/liblzma/.libs/liblzma.a
  src/liblzma/api/lzma.h

WARNING: Data compressed with the LZMA algorithm cannot be read by
ROOT executables that were not built with LZMA support.

Revision 39115 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu May 5 15:54:41 2011 UTC (3 years, 8 months ago) by ganis
File length: 37028 byte(s)
Diff to previous 38922
  - Fix bug in TXSocket::Create whose effect was to make the three additional attempts
    following the first to send a wrong and meaningless creation info (ordinal number, etc).
    The bug was introduced while fixing the log path trasmission problem.
  - Improve the way information about the logs files is saved when failure happens
    during creation. The log paths for these failing now should be now correctly saved
    and accessible via TProofLog.
  - Improve TSlave::Print when the connection is invalid (print all the valid information,
    available, not only the header one)

Revision 38922 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Apr 19 13:15:05 2011 UTC (3 years, 9 months ago) by ganis
File length: 37003 byte(s)
Diff to previous 36592
  Adapt to new XrdClient constructors allowing to setup a connection using an open socket.
  Also extend the protocol handshake and Send/Recv calls to use a generic physical connection.

Revision 36592 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Nov 11 10:43:17 2010 UTC (4 years, 2 months ago) by ganis
File length: 36449 byte(s)
Diff to previous 36512
   - Terminate a proofserv after it has been stopped for large memory footprint and it has
     sent the results back
   - Notify information about which object was being merged if a memory exception happens
     during merging on the master
   - Allow to set the HWM and stop-process fractions via the environment

Revision 36512 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Nov 5 12:05:11 2010 UTC (4 years, 2 months ago) by ganis
File length: 36609 byte(s)
Diff to previous 34101
  In TProofServ, TXProofServ:
  Move the creation and start of the idle timeout from the end of SetupCommon to
  the end of CreateServer, so that the timeout is not active during worker setup.
  Fix a problem reported by ALICE.

Revision 34101 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Jun 24 12:12:59 2010 UTC (4 years, 7 months ago) by ganis
File length: 36048 byte(s)
Diff to previous 33824
  Following a request from ALICE, implement a timer to terminate idle sessions. The timeout
  value is controlled by the variable ProofServ.IdleTimeout (value in seconds). For example,
  to set a timeout of 20 mins, enter the following in the xproofd config file:

       ### Setup an idle timeout
       xpd.putrc ProofServ.IdleTimeout 1200

  The timer is disabled by default.

Revision 33824 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Jun 10 14:43:40 2010 UTC (4 years, 7 months ago) by ganis
File length: 36127 byte(s)
Diff to previous 33693
 - Add new directive 'xpd.datadir' to better control the user data directories and their
   permission settings. The syntax is 'xpd.datadir <datadir> <opts>' where <datadir> is
   the root path for the datadirs (the final datadir will be <datadir>/<group>/<user>) and
   <opts> can contain a combination of the following letters: 'g', 'a', 'M', 'W' ["gW"];
   the lower case control the write permissions: 'g' group, 'a' everybody; the upper case
   control where the data dirs shoudl be created: 'W' on the workers, 'M' on the master,
   "WM" everywhere.
 - Better handle the removal of empty data directories in TProofServ and derivatives
 - In TProofOutputFile:
    - extend support for URL options
    - correctly take the URL options into account

   This should solve a problem observed in the new ALICE CAF.

Revision 33693 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Jun 1 11:16:30 2010 UTC (4 years, 7 months ago) by ganis
File length: 36579 byte(s)
Diff to previous 33369
Remove data directory in Terminate if empty: avoid having many empty dirs around

Revision 33369 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue May 4 09:21:42 2010 UTC (4 years, 8 months ago) by ganis
File length: 35880 byte(s)
Diff to previous 33364
Log to syslog also the PROOF group to which the suer belongs

Revision 33364 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue May 4 08:04:17 2010 UTC (4 years, 8 months ago) by ganis
File length: 35932 byte(s)
Diff to previous 33280
  Fine tune monitoring via syslog/local5:
   - add additional level to increase the granularity; available levels
      0               no syslog logging
      1               syslog: beginning of session, main actions (exec, process, dataset
                              handling, cache/package handling, query handling), end of
                              session
      2               syslog all input actions (i.e. also internal messages)
      3               syslog also the application logs
   - add logging on session begin (at the end of SetupCommon)
   - in the end-of-session log, log also the total real and CPU times spent executing commands

Revision 33280 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Apr 28 14:45:01 2010 UTC (4 years, 8 months ago) by ganis
File length: 35878 byte(s)
Diff to previous 33054
   Restructuring of the syslog logging functionality.
   The activation is still controlled by the ProofServ.LogToSysLog switch, but now two
   levels of logging are provided:

      2     log everything to syslog (local5)
      1     log only a record for each action performed by the client; by default the
            format is
                           user  message-type realtime cputime

            When relevant, some information is added, like file name, selector name, 
            package name, bytes read, etc ...
            This may be useful to get a quick monitoring of the cluster activities.

   The format of the message to syslog has also been sligthly modified and made more compact;
   example

   Apr 28 15:50:53 pcphsft64 proofm-0[15086]: ganis 5 0.006 0.010 .!pwd
   Apr 28 16:14:52 pcphsft64 proofm-0[18814]: ganis 1012 0.000 0.000 h1analysis.C
   Apr 28 16:14:52 pcphsft64 proofm-0[18814]: ganis 1013 0.031 0.000
   Apr 28 16:14:52 pcphsft64 proofm-0[18814]: ganis 1012 0.000 0.000 h1analysis.h
   Apr 28 16:14:52 pcphsft64 proofm-0[18814]: ganis 1013 0.029 0.010
   Apr 28 16:15:02 pcphsft64 proofm-0[18814]: ganis 1015 7.097 0.090 0 283813 22554834 5.920

   for the action records, and

   Apr 28 16:37:44 pcphsft64 proofm-0[20907]: ganis: Info:<TXProofServ::Terminate>: starting session termination operations ...

   for an Info message.

Revision 33054 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Apr 15 16:47:24 2010 UTC (4 years, 9 months ago) by pcanal
File length: 35708 byte(s)
Diff to previous 31028
From Axel:

Move CINT specific initialization (i.e. preventing CINT from #include-ing files it cannot parse) from TRint into TCint. Consistently tell the interpreter to #include <string>, not <_string>. Set the include path in TCint / TCling so they can load interpreter specific headers (like RtypesCint.h). Whether or not to remove the interpreter includes in TProofServLite::SetupOnFork() will
be studied by Gerri, for now they should not hurt.

Revision 31028 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Nov 9 07:37:35 2009 UTC (5 years, 2 months ago) by ganis
File length: 36065 byte(s)
Diff to previous 31012
Optimize verbosity levels in TXProofServ::HandleUrgentData

Revision 31012 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Nov 6 16:08:19 2009 UTC (5 years, 2 months ago) by ganis
File length: 36003 byte(s)
Diff to previous 30995
   Save the status of a 'proofserv' session into a new file in the 'activesessions' area.
   The full path of the new file is

          <admin_path>/.xproofd.<port>/activesessions/<user>.<group>.<pid>.status

   The status indicates whether the session is idle, running or queued.
   The status is updated every 'checkfq' secs (see xpd.proofservmgr; default 30 s).
   The status is dumped by the reader thread of TXProofServ and therefore its r/w access
   is protected.

   This change is needed for PoD optimizations.

Revision 30995 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Nov 5 17:41:03 2009 UTC (5 years, 2 months ago) by ganis
File length: 35565 byte(s)
Diff to previous 30859
   A couple of quick additions mostly for the new bench suite under development:

   - Add the possibility to give indications about the number of workers at startup.
     E.g.

       1. To start max 5 workers

             TProof::Open("<master>","workers=5")

       2. To start max 2 workers per physical machine

             TProof::Open("<master>","workers=2x")

     This is useful in general when running tests (equivalent but quicker then full startup
     followed by TProof::SetParallel(n) or TProof::DeactivateWorker(...)).

   - Add support for the worker SysInfo_t in TSlaveInfo (obtained via TProof::GetListOfSlaveInfos())

Revision 30859 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sat Oct 24 14:53:07 2009 UTC (5 years, 3 months ago) by ganis
File length: 33819 byte(s)
Diff to previous 29059
  Patch for improved performance monitoring. The 'Rate Plot' button in the progress
  dialog is renamed 'Performance plot' and shows up to 4 plots with the event/sec,
  the average read chunck size, the number of active workers and the number of active
  PROOF sessions on the cluster, all as a function of processing time.

  The read chunck size plot allows to monitor the usage of the cache.

  The istantaneous processing rate (event/sec) is now better estimate: a few issues
  with the normalizing times have ben solved, removing the artificial structures that
  were observed.

  The possibility to set a max packet time length is introduced (default 30 s); this
  can be changes with the parameter PROOF_MaxPacketTime.
  The size of the cache is also taken into account to optimize the use of the cache.

  The parameter PROOF_UseParallelUnzip has been introduced to toggle the use of the 
  parallel unzip (default off for now).

  A page describing the new performance plots is under preparation at 
           http://root.cern.ch/drupal/content/progress-dialog

Revision 29059 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jun 17 15:43:11 2009 UTC (5 years, 7 months ago) by ganis
File length: 33467 byte(s)
Diff to previous 28856
   Automatize running of PROOF sessions in valgrind.
   The second argument of TProof::Open is used to trigger the relevant settings.

   To valgrind the master session start PROOF with

      TProof::Open("<master>","valgrind")

   or

      TProof::Open("<master>","valgrind=master")

   To valgrind two workers sessions use

      TProof::Open("<master>","valgrind=workers")

   To valgrind <n> workers sessions use

      TProof::Open("<master>","valgrind=workers#<n>")


   To valgrind master and 2 workers (for <n> workers add '#<n>')

      TProof::Open("<master>","valgrind=master+workers")


   The valgrind logs are available with the tag '<ordinal>-valgrind' in the log dialog
   or form TProofMgr::GetSessionLogs() .

   To add options to valgrind execute

      TProof::AddEnvVar("PROOF_WRAPPERCMD", "valgrind_opts:<options>")

   before starting the session.

Revision 28856 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Jun 9 11:49:43 2009 UTC (5 years, 7 months ago) by ganis
File length: 32892 byte(s)
Diff to previous 27932
   A few optimizations in transferring and merging of the results:
   - Output objects are added to the same TMessage until a HWM is reached
     (default 1MB; controlled by 'ProofServ.MsgSizeHWM'); this limits the
     number of transfers in the case of large numbers of small objects.
   - Reasonably small histograms (GetSize() < MsgSizeHWM) are merged in
     one-go at the end instead of one-by-one to exploit the better performance
     of TH1::Merge on the full list of histos.
   - Add possibility to compress the messages; this is controlled by
           ProofServ.CompressMessage  <compression_level>
     The default is still 'no compression' but this will allow to study the
     impact of compression.

   In addition, a sort of 'progress' counter for merging is now shown on the
   client:

      root [n] p->Process(...)
      ...
      Mst-0: merging output objects ... / (4 workers still sending)

   This asserts socket activity and fixes the timeout problems during long
   merging phases reported on the forum.

Revision 27932 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Mar 25 15:10:07 2009 UTC (5 years, 10 months ago) by ganis
File length: 32809 byte(s)
Diff to previous 27270
Warn old clients if schema evolution is not effective

Revision 27270 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jan 28 07:44:26 2009 UTC (5 years, 11 months ago) by ganis
File length: 32333 byte(s)
Diff to previous 27231

    Add support for query queuing also for static sessions. To limit the number of
    running sessions to, for example, 3 one can just set

    xpd.schedparam default mxrun:3 queue:fifo

    The patch also fixes a problem with the fIdle flag in TProofServ (not always
    reset after processing) and introduces new methods XrdProofSched::DumpQueues(),
    showing the queue content, and XrdProofdProofServ::RemoveQuery to move all the
    code to remove a query inside XrdProofdProofServ.

Revision 27231 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sun Jan 25 09:42:21 2009 UTC (5 years, 11 months ago) by ganis
File length: 32329 byte(s)
Diff to previous 27042
  Patch adding support for session queuing in the scheduler (from Jan and me).
  This allows to control the number of sessions allowed to process queries concurrently.
  The feature is enabled by a new parameter 'queue:fifo' in the 'xpd.schedparam'
  directive.

  In case of static worker assigment (default, random, round-robin) the max number of
  running sessions can be limited by another new parameter 'mxrun'; for example
  'xpd.schedparam default mxrun:3 queue:fifo' will run concurrently only 3 sessions.
  Additional requests are queued and run as soon as one of the running sessions goes idle.
  The current policy is FIFO, so that there is a rotation among queued sessions.

  In the case of load-based worker assignement, the max number of running queries is
  determined dynamically.

  This option requires dynamic, per-job scheduling.

Revision 27042 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sat Dec 20 11:09:08 2008 UTC (6 years, 1 month ago) by ganis
File length: 31991 byte(s)
Diff to previous 27034
Fix warnings from nightlies and coding violations

Revision 27034 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Dec 19 15:30:06 2008 UTC (6 years, 1 month ago) by ganis
File length: 31970 byte(s)
Diff to previous 26993
   From Jan: prepare for query-enqueing support:
   - add new protocol message to resume an enqueued session
   - add framework to process enqueued queries after a resume: the processing part on masters
     has been moved to a new method ProcessNext which is called also after a resume request.
   - remove duplication in the definition of message types for SendCoordinator
     (same codes defined in XProofProtocol.h and TXSocket.h)

Revision 26993 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Dec 17 16:01:04 2008 UTC (6 years, 1 month ago) by rdm
File length: 31468 byte(s)
Diff to previous 26951
From Gerri:
- Fix a problem with filtering the SvcMsg in TProofMgrLite::GetSessionLogs
- Fix a weird backward-compatibility problem in TProof::SendFile (found
  and needed by ALICE)
- Update the release notes

Revision 26951 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Dec 16 12:04:38 2008 UTC (6 years, 1 month ago) by ganis
File length: 31766 byte(s)
Diff to previous 26728
   - Fix an issue with uploading packages in rapid sequence
   - Fix an issue with TProof::Load() in forced compilation mode ('++')
   - Fix an issue with grep in log retrieval
   - Fix an issue with ownership of the xrootd log file
   - Make sure the XrdProofdProofServMgr::fCurrentSessions is always up-to-date

Revision 26728 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Dec 8 13:30:31 2008 UTC (6 years, 1 month ago) by ganis
File length: 31468 byte(s)
Diff to previous 26678
Add a debug statement for ping actions

Revision 26678 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Dec 5 14:53:23 2008 UTC (6 years, 1 month ago) by ganis
File length: 31336 byte(s)
Diff to previous 26499

   Patch to:
   - Add the possibility to set upper limits on the virtual memory (changes in TProofPlayer,
     TProofServ, TXProofServ and pmain (separate patch)).
     If enebled, the session gets firts a warning when it reaches 80% of the limit, and then
     processing is stopped whenit exceeds 95% of the limit, sending back the results.
     Also, the memory footprint is notified when the session is terminated.
   - Make sure the the active valuse in XrdProofWorker are always correct; this was not the
     case for dynamic startup as the notification at the end of the query was not done.
     This information is crucial or the scheduler.
     The way the information is stored in XrdProofdProofServ had to be modified
     and a new internal message type (kReleaseWorker) added.
     (Changes in several proofd classes, TProof, TXsocket and TXProofServ)
   - Fix problem with setting a static upper limit of the sessions that can be started,
     and enable this functionality for the dynamic mode (changes in XrdProofSched)
   - Remove a deleted worker from all the lists in TProof::MarkBad to avoid later attempts of use.
   - Better control the use of the internal pipe for socket readiness notification in TXSocket.

Revision 26499 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Nov 27 15:00:13 2008 UTC (6 years, 1 month ago) by ganis
File length: 30393 byte(s)
Diff to previous 25720
Modify the notification message upon reception of a SIGTERM

Revision 25720 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Oct 7 10:13:37 2008 UTC (6 years, 3 months ago) by ganis
File length: 30411 byte(s)
Diff to previous 25535
   Move the code for query-result management into a new class TQueryResultManager,
   so that it can be used in other places (e.g. prooflite).

Revision 25535 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Sep 25 13:02:38 2008 UTC (6 years, 3 months ago) by ganis
File length: 30144 byte(s)
Diff to previous 24784
   Flush the log file in case of successfully connection recovery to avoid displaying
   temporary errors.

Revision 24784 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Jul 11 17:35:07 2008 UTC (6 years, 6 months ago) by ganis
File length: 30119 byte(s)
Diff to previous 24739
   Fix a problem causing the whole query to stop even in the case a worker
   was terminated gently with SIGTERM.

Revision 24739 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jul 9 12:21:45 2008 UTC (6 years, 6 months ago) by ganis
File length: 30066 byte(s)
Diff to previous 24737
Improved naming to avoid confusing the developers

Revision 24737 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jul 9 11:21:30 2008 UTC (6 years, 6 months ago) by ganis
File length: 30078 byte(s)
Diff to previous 24719
Fine-tune the way logging to syslog and sending of logs to the master are controlled

Revision 24719 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jul 9 07:07:25 2008 UTC (6 years, 6 months ago) by ganis
File length: 30102 byte(s)
Diff to previous 24589

    Patch refactorizing the XrdProofd plugin.
    The class XrdProofdProtocol is now in charge only of the operations strctly related
    to the XProofd protocol. All auxilliary services have been moved to dedicated service
    classes controlled by XrdProofdManager. In particular:
    - XrdProofdClientMgr handles the clients (represented by instances of XrdProofdClient)
      including login, authentication and access control
    - XrdProofdProofServMgr handles the PROOF sessions (represented by instances of
      XrdProofdProofServ) including creation, attachment, detachment, destruction and
      environment setting
    - XrdProofdNetMgr handles connections between instances of XProofd running on different
      nodes
    - XrdProofdPriorityMgr handles the session priorities

    A special effort has been done to get rid of all possible internal dead-lock situations.
    Internal actions on clients and sessions are now all asynchronous, governed by internal
    pipes.

    The new plugin also offers new functionality, among which:
    - a XProofd admin area, located under <xrd.admin>/.xproof.<port>, keeps information about
      active and terminated sessions, and the minimal state for active clients. This is used
      to reguraly check the client and session activity, to cleanup orphalin sessions and
      to shutdown inactive client connections.
      In particular this allows a more solid implementation of Reset, which now exists in two
      flavours: 'soft', TProof::Reset(<masterurl>), (default) asks the sessions to terminate gently;
      'hard', TProof::Reset(<masterurl>,kTRUE) schedules all sessions for forced termination.
    - support for automatic attempts for reconnection in the case the daemon restarts.
      This allows to reconfigure the plugin by xrootd restart w/o affecting the running
      sessions.
    - support the definition of workers via config file directive, getting de facto rid of
      the proof.conf file.
    - domain+level control of printout message; the format has been improved: in particular
      all information messages contain now the tag 'xpd-I' and all error messages the tag
      'xpd-E', so that they can easily be grepped out from the log file.

    The Wiki 'XrdProofdDirectives Directives' page has been updated with the new directives.

Revision 24589 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Jun 27 14:17:08 2008 UTC (6 years, 6 months ago) by ganis
File length: 31505 byte(s)
Diff to previous 24574

  - Add support for an external definition of the session tag: this will be needed
    by external PROOF session launchers, when available
  - Check also the env ROOTRCFILE for 'rootrc' configuration.

Revision 24574 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Jun 26 17:06:03 2008 UTC (6 years, 6 months ago) by ganis
File length: 31345 byte(s)
Diff to previous 23181
  Implement selective sending of logs from workers to master to avoid duplicating
  too many text lines on the master log. Logs are now sent only after Exec, Print
  requests and in case an error (level >= kError) occured. Of course, the full logs
  can always be retrieved via TProofMgr::GetSessionLogs .

  Also, remove TXProofServ::SendLogFile, which was reduced to be a duplication of
  TProofServ::SendLogFile .

Revision 23181 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sun Apr 13 07:39:52 2008 UTC (6 years, 9 months ago) by ganis
File length: 33235 byte(s)
Diff to previous 23069
   Restore sending of full workers logs to master to make Exec working properly in all
   cases. Optimized logging will need a different strategy.

Revision 23069 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Apr 9 07:12:24 2008 UTC (6 years, 9 months ago) by ganis
File length: 33311 byte(s)
Diff to previous 22635
On workers: do not send logs to the master; attempt to stop processing when terminating

Revision 22635 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Mar 13 10:50:20 2008 UTC (6 years, 10 months ago) by rdm
File length: 33076 byte(s)
Diff to previous 20882
move all PROOF related libraries under the new proof directory.

Revision 20882 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Nov 19 11:31:26 2007 UTC (7 years, 2 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 33076 byte(s)
Diff to previous 20682
Set property svn:eol-style LF on all source and Makefiles. This should avoid
problems with Win32 line endings ending up in the repository. All MS tools
support LF eols fine.

Revision 20682 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Nov 6 15:51:59 2007 UTC (7 years, 2 months ago) by ganis
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 33076 byte(s)
Diff to previous 20411
  Import branches/dev/proof r20654

  Summary:

  + Improvements
    - Add support for SSH SOCKS4 tunnelling; the local port for the tunnel can be specified
      in the master URL, e.g. TProof::Open("master/?tunnel:8000")
    - Add the possibility to plot the estimated instantaneous rate
    - Add "PROOF_ForceLocal" parameter to the !TPacketizerAdaptive; if set to 1, all the data
      are processed locally.
    - Add support for remote grep functionality while retrieving logs (needed by the forthcoming
      memory monitor)
  + Bug fixes
    - Several small fixes to revive the multi-master mode.
    - XrdProofdProtocol:
      - add missing lock to the client instance in SendMsg to avoid screwing up requests from
        workers on the same machine
         - lock the mutex of the requester when setting prorities
         - add notification during Reset
         - fix problem with the detection of the 'allow' directive
         - fix problem with the parsing of the return value from XrdProofServProxy::TerminateProofServ()
         - always use the effective user to retrieve info fom another server
           (XrdProofdManager::GetProofConn is now used)
         - fix possible dead-locks from debug notifications done after hard-killing a session
         - re-enable the garbage collector thread of the connection manager in XrdProofConn to
           fix a problem with closing physical connections;
         - fix a problem with !CleanupProofServ in the case of a non-privileged daemon running
           in multi-user mode
         - introduce a timeout when waiting for the startup of a 'proofserv'.
    - XrdProofConn: init mutex in the ctor; lock in SendRecv
    - XrdProofSched: add support for using the priorities defined in the group manager to define
      the number of workers for sessions
      - TProof:
         - Broadcast priorities to unique nodes only
         - timeout after 5 mins the initial Collect to avoid clients getting stuck at this stage
         - add support for generic timeout in Collect (disabled by default)
         - fix a problem with SendFile.
      - TXProofServ: add a call to !TProof::InterruptCurrentMonitor() in Terminate() to stop 
        infinite loops in Collect
      - TXSocket:
         - Implement a flag to interrupt a TXSocket while waiting for messages
         - Split the session creation timeout in 4 attempts: the total timeout is the same but
           it may circumvent occasional forking problems.
      - XROOTD: 
         - fix a potential (possibly accademic) memory leak in the client
         - fix a access permission problem with Kerberos ticket forwarding
         - fix bug preventing 'locate' to work properly
         - re-enable optimized 'locate'

Revision 20411 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Oct 19 12:37:58 2007 UTC (7 years, 3 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 39119 byte(s)
Diff to previous 20307
remove conditional around include of RConfigure.h:

#ifdef R__HAVE_CONFIG
#include "RConfigure.h"
#endif 

Having this conditional caused RConfigure.h to be missing from the .d
files and hence these files would not be recompiled after re-configuring
using --prefix.

Revision 20307 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Oct 11 10:58:50 2007 UTC (7 years, 3 months ago) by ganis
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 39149 byte(s)
Diff to previous 19826
  Import branches/dev/proof r20306

  Summary (see branch logs for more details):

  - Set of changes related to CPU quota control including
    - broadcast of centrally determined priorities
    - mechanism to renice processes in a quantitative way
  - New packetizer for non-tree based analysis and related API in TProof (from L. Tran-Thanh)
  - Support for merging output objects saved in files on the workers (from L. Tran-Thanh and me)
  - Improve version binary compatibility checks using also the SVN revision number
   (when available) to define the running version.
  - Extend the version binary compatibility checks also to the cached selector binaries.
  - Extend TDSet::Lookup so that in case of missing files, it can remove them from the
    dataset (option removeMissing must be set).
  - Move the data set lookup to the TProofPlayerRemote::Process.
  - Handle properly the case of incomple datasets: if the file is not found in the lookup
    don't try to validate it; add it, instead, to a 'missingFiles' list returned in the
    output list (fixing bug #28800 in Savannah).

Revision 19826 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Sep 19 19:56:11 2007 UTC (7 years, 4 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 39142 byte(s)
Diff to previous 19825
imported svn:keywords Id property

Revision 19825 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Sep 19 19:49:10 2007 UTC (7 years, 4 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 39197 byte(s)
Diff to previous 19582
remove :$ from tag line

Revision 19582 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Aug 9 14:06:25 2007 UTC (7 years, 5 months ago) by ganis
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 39207 byte(s)
Diff to previous 19248
Fix a problem with the transmission of the inflate factor.

Revision 19248 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Jul 3 16:01:33 2007 UTC (7 years, 6 months ago) by ganis
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 39048 byte(s)
Diff to previous 19145
Add functionality to broadcast group priorities.

Revision 19145 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Jun 22 17:16:35 2007 UTC (7 years, 7 months ago) by ganis
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38373 byte(s)
Diff to previous 19099
Add support for global package directories. The colon-separated list of directories
to be searched for is defined by the directive

Proof.GlobalPackageDirs  globaldir1[:globaldir2[:globaldir3[:...]]]

which can be included in the xrootd config file via the xpd.putrc directive

xpd.putrc Proof.GlobalPackageDirs  globaldir1[:globaldir2[:globaldir3[:...]]]

The directories must be readable by the user (a check is made in TProofServ::Setup).
Packages are searched first in the user local cache and then in the global
directories (in the given order, so that the first occurence is used).

The content of the global package directories is listed by TProof::ShowPackages().

The functionality is available also locally on the client. The Proof.GlobalPackageDirs
has to be specified in the users $HOME/.rootrc .

Global packages can be directly enabled via TProof::EnablePackage(<package_name>).

Revision 19099 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Jun 21 08:47:42 2007 UTC (7 years, 7 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37337 byte(s)
Diff to previous 19093
added new data member fGroupPriority. This priority is obtained by
GetPriority() which makes an SQL query in the proofpriority table, which
has the format:
CREATE TABLE proofpriority (
   id            INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
   group         VARCHAR(32) NOT NULL,
   priority      INT
)
This prority is then forwarded to the xrootd on the workers.

Revision 19093 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Jun 21 07:12:37 2007 UTC (7 years, 7 months ago) by ganis
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37253 byte(s)
Diff to previous 19008
  Add member to store the inflate factor and methods to handle it.
  Apply inflate factor if defined.

Revision 19008 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Jun 12 13:51:41 2007 UTC (7 years, 7 months ago) by ganis
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 36932 byte(s)
Diff to previous 18862
  New files:

  XrdProofGroup.h, .cxx
    - Class describing a group and its properties
  XrdProofdClient.h, .cxx
    - Class describing a Xpd client (extracted from XrdProofdProtocol.h, .cxx)
  XrdROOT.h, .cxx
    - Class describing a ROOT version (extracted from XrdProofdProtocol.h, .cxx)
  XrdProofdAux.h, .cxx
    - Utility tools (extracted from XrdProofdProtocol.h, .cxx)
  XrdProofdPlatform.h
    - Platform-dependent system includes and defines common to all implementation
      files (extracted from XrdProofdProtocol.cxx)

  Patched files:

  XrdProofdProtocol.h, .cxx
    - Adapt to file re-organization
    - Rationalization fo the way the relevant info is passed to 'proofserv':
      a minimal set of environment variables is kept, the remaining parameters
      are passed via a rootrc-like file created in the session working dir
    - New directive "xpd.groupfile" to read a file defining groups and their
      properties.
    - New directive "xpd.datasetdir" to define the root path under which the
      user data set dirs are created
    - New directive "xpd.putrc" to input rootrc-like directives to be passed
      straight to proofserv

  TProofServ.cxx, TXProofServ.h, .cxx
    - Read the session rootrc in the constructor
    - Take relevant parameters from the session rootrc file

  TProof.h
    - Move protocol number up to 14 to be able to handle backward compatibility
      issues due to changes in the info transfer mode between xpd and proofserv

  pmain.cxx
    - Take loglevel from argc #5
    - Use logfile path generated by XrdProofdProtocol.cxx

Revision 18862 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed May 23 09:11:26 2007 UTC (7 years, 8 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38733 byte(s)
Diff to previous 18835
From Gerri:
- Use FlushLogFile() where relevant.

Revision 18835 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon May 21 00:46:19 2007 UTC (7 years, 8 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38790 byte(s)
Diff to previous 18353
From Gerri:
- Send back version/arch/compiler info at the end of Setup().

Revision 18353 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Mar 19 15:14:10 2007 UTC (7 years, 10 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38398 byte(s)
Diff to previous 18327
From Gerri:
- Support for multiple ROOT versions on a PROOF cluster.
  TProofMgr::ShowROOTVersions() giving the following output:

root [0] TProof::Mgr("")->ShowROOTVersions()
----------------------------------------------------------

Available versions (tag ROOT-vers remote-path PROOF-version):

 * cvs 5.15/03 /home/ganis/local/root/head/head 12
   dev 5.15/03 /home/ganis/local/root/head/root 12

----------------------------------------------------------

- TProofMgr::SetROOTVersion(const char *version_tag)

    to change the default for the user, eg

root [1] TProof::Mgr("")->SetROOTVersion("dev")
root [2] TProof::Mgr("")->ShowROOTVersions()
----------------------------------------------------------

Available versions (tag ROOT-vers remote-path PROOF-version):

   cvs 5.15/03 /home/ganis/local/root/head/head 12
 * dev 5.15/03 /home/ganis/local/root/head/root 12

----------------------------------------------------------

Available version are defined by the administrator via the
"xpd.rootsys" directive:

xpd.rootsys /home/ganis/local/root/head/head cvs
xpd.rootsys /home/ganis/local/root/head/root dev

in the xrootd config file.

- Improve error reporting in TXSocket
- Fix a problem with resetting a label in TProofProgressDialog

Revision 18327 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Mar 19 01:36:56 2007 UTC (7 years, 10 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38398 byte(s)
Diff to previous 18320
more libProof restructuring. Now libProof is also independent of libHist.
Last remaining step is to remove libTree dependency. On startup proofserv.exe
now loads 9 libs, was 16.

Revision 18320 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sat Mar 17 18:04:02 2007 UTC (7 years, 10 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38530 byte(s)
Diff to previous 18107
move TProofDraw to proofplayer. Also move the TProofServ GetTreeHeader code
to proofplayer.

Revision 18107 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Feb 28 16:50:55 2007 UTC (7 years, 10 months ago) by brun
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38529 byte(s)
Diff to previous 17695
From Gerri:
Implement a cleaner receipe to avoid deadlock situations when terminating

Revision 17695 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Feb 5 14:04:04 2007 UTC (7 years, 11 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38612 byte(s)
Diff to previous 17689
From Gerri:
comment.

Revision 17689 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Feb 5 10:40:30 2007 UTC (7 years, 11 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38620 byte(s)
Diff to previous 17562
From Gerri:
introduce single shot timer for proofserv termination. This timer makes
sure that there is still at least on item the will make DispatchOneEvent
return as all sockets are already closed.

Revision 17562 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Jan 29 15:11:11 2007 UTC (7 years, 11 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38581 byte(s)
Diff to previous 17553
Frrom Gerri:
There is a new classes TProofServLogHandler allowing to redirect
the output to the client in real-time. It has two main constructors,
one taking an external command (like BUILD.sh) and opening a pipe,
the other taking a pointer to a open file. TProofServ uses the class
via the guard class TProofServLogHandlerGuard to better localize the
usage. This redirection is not always activated. For the time being it
is active for Process and Cache actions,and it can be deactivated via

TProof::SetRealTimeLog(kFALSE)

(for BUILD.sh - executed externally - is always ON).
In particular, with the patch applied, the outputs from BUILD.shi
will scroll down on the screen, and clicking on "Show logs" in the
dialog box during process shows the messages and allows to see what
the session is doing: this is quite nice, because getting
an empty window from "Show Logs" is frustrating, in my opinion.

So, all this fixes the bug/request #16845.

The patch also fixes bug #22555 (hanging while enabling a package
without BUILD.sh) and several other small bugs (a couple of unitialized
variables) or typos.
It also cleanups the API a bit (we had  SetQueryType and SetQueryMode
doing exactly the same thing ...).

Revision 17553 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Jan 29 10:06:51 2007 UTC (7 years, 11 months ago) by brun
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38356 byte(s)
Diff to previous 17392
Add explicit inclusion of "TClass.h" in view of a future removal of TClass.h
from TQObject.h.

Revision 17392 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Jan 18 17:09:50 2007 UTC (8 years ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38337 byte(s)
Diff to previous 17384
From Gerri:
- suppress redefinition of $HOME (already done by the launching daemon).

Revision 17384 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jan 17 21:24:26 2007 UTC (8 years ago) by brun
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38500 byte(s)
Diff to previous 17035
Following the recent changes in TMacro, TObjString must be included.

Revision 17035 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sun Dec 3 23:34:04 2006 UTC (8 years, 1 month ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38476 byte(s)
Diff to previous 16969
From Gerri:
0. Introduction of the static:
      TProof::Mgr(const char *url)
   to get the manager to the cluster; this gives access to all the manager
   functionality, included Reset(). TProof::Reset() is removed.

1. Two new classes: TProofLog and TProofLogElem.
   TProofLog contains a list of TProofLogElem, one per node. TProofLogElem
   basically contains a TMacro plus methods to play  with it.

2. The TProofLog object can be retrieved via the manager:

      TProofLog *pl = TProof::Mgr("cluster")->GetSessionLogs()

   by default the last session is retrieved: the signature for GetSessionLogs is

      TProofLog *TProofMgr::GetSessionLogs(Int_t ridx, const char *stag)

   the first argument is a reversed relative index (-1 gets the next to last
   session), the second is the session unique tag; if the second is
   specified the first is ignored.

3. TProofMgr has a new method ReadBuffer to quickly read a buffer from
   a file via XrdProofd: this allows to avoid the second xrootd on the
   redirector and in general is useful when a quick reading is needed.
   Also, the buffer is always retrieved via the master, so we should not
   have firewall problems in accessing directly the xrootd of the workers.

4. In the sandbox I have kept the .sessions file with the active sessions
   and I have added .terminated in each session working dir which have
   been terminated. On the master, each session working dir has the file
   .workers with the relevant information about the workers.

5. The max number of old session workdirs is set to 10 by default
   (configurable in xpd.cf): the sandbox is trimmed when closing a session.

6. Some unrelated changes/fixes contained in the patch:
   - in XrdProofdProtocol:
      - use of /proc (linux, solaris) or sysctl (mac) to verify/search
        for running of a process. For other platforms I have left the
        parsing of the output of 'ps'.
      - I have introduced a 'cron'  thread: it runs every minute and for
        the moment it just cleans a vector; this is the place where to do
        client independent actions, like checking if the config files have
        changed, for example.
      - I have added the possibility to control which user can access
        the cluster; requested by Jan Fiete.

7. The patch contains also several fixes here and there I have found by
   testing.

Revision 16969 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Nov 28 12:10:52 2006 UTC (8 years, 1 month ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38373 byte(s)
Diff to previous 16839
From Gerri:
This patch eliminates TVirtualProof, TVirtualProofMgr and TVirtualProofDesc.
The first is integrated in TProof, the second in TProofMgr and the third
becomes TProofDesc defined in TProofMgr.h.

Revision 16839 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Nov 20 15:56:36 2006 UTC (8 years, 2 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38415 byte(s)
Diff to previous 16826
From Maarten + Gerri:
Patch to allow run time configuration of proofserv environment. The client
can pass a set of shell variables to PROOF. These variables can be used
to startup different versions of the proofserv, or they can instruct the
system to run proofserv under valgrind or tcheck, etc. Main changes:
- default wrapper script for proofserv
- modifies the build system for the wrapper script
- adds a list of environment variables to TVirtualProof, which are
  passed to the wrapper by proofd/xrootd.
- adds code to TProofServ to pickup these variables and pass them
  along to the slaves/submasters

Revision 16826 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Nov 20 10:31:48 2006 UTC (8 years, 2 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37972 byte(s)
Diff to previous 16796
From Gerri:
- undo a small optimization that resulted in the system not starting up anymore.
- small fix for issue found by tcheck.

Revision 16796 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Nov 16 17:18:32 2006 UTC (8 years, 2 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37704 byte(s)
Diff to previous 16775
Changes to get rid of the config.h and HAVE_CONFIG define. These were way
too trivial (just do a locate config.h to see how many there are):
- replace occurances of config.h by RConfigure.h
- add #warning in config.h that is should not be used
- change HAVE_CONFIG to R__HAVE_CONFIG

Revision 16775 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Nov 15 17:45:55 2006 UTC (8 years, 2 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37697 byte(s)
Diff to previous 16664
From Gerri and Bertrand:
Fixes and new features developed in preparation for the Intel quad-core
launch demo:
- Add instantaneous event and MB processing rates in the Progress signal
  and make necessary modification to exploit it in the Progress Dialog.
- Add possibility to change the Progress and Feedback periods
- A number of small improvements and bug fixes

Revision 16664 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Nov 1 16:01:58 2006 UTC (8 years, 2 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37757 byte(s)
Diff to previous 16560
From Gerri:
setup HandleError as soon as the socket is open so that failures
at Setup are treated correctly; should reduce the number of hanging proofservs.

Revision 16560 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Oct 19 13:02:38 2006 UTC (8 years, 3 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37757 byte(s)
Diff to previous 16557
add protection against fProof being 0.

Revision 16557 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Oct 19 12:38:07 2006 UTC (8 years, 3 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37736 byte(s)
Diff to previous 16417
From Gerri:
Major set of robustness fixes, improving error reporting and reducing the
startup failures especially observed on AMD64. Summary of changes:
- XrdProofdProtocol.cxx, .h
   - Remove zombie treatment: done by XrdScheduler::Reaper()
   - Check the modification date of the PROOF config file so that changes
     can be picked up on the fly
   - Fix problem with the definition of the image
   - Improved tracing (see etc/proof/xpd.cf.sample)
   - Several bug fixes

- XrdProofConn
   - fix problem with marshalling in case of retries.

- TProofServ, TXProofServ
   - Add return codes to CreateServer() and Setup() for proper termination
     in case of errors

- TProof
   - Make sure that TXProofMgr is the last one in the list of sockets for proper
     cleaning

Revision 16417 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Oct 3 14:04:57 2006 UTC (8 years, 3 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37771 byte(s)
Diff to previous 16412
fix compilation error due to missng netinet/in.h include.

Revision 16412 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Oct 3 13:28:29 2006 UTC (8 years, 3 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 37748 byte(s)
Diff to previous 15930
Include file cleanup, many includes were not needed.

Revision 15930 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sat Aug 5 20:04:47 2006 UTC (8 years, 5 months ago) by brun
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 38999 byte(s)
Diff to previous 15857
From Gerri:
I have realized that the new tarball requires the mods to the XrdProofd plugin
that I wanted to keep separated. This explains the problem on Mac (and also
 everywhere the plug-in is build).
It also contains the fix of the problem mention by Fons with selectors from a library.

Revision 15857 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jul 26 14:28:59 2006 UTC (8 years, 6 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 36064 byte(s)
Diff to previous 15524
From Gerri:
TVirtualProof.h
  - Add abstract methods methods to communicate the data set status
    (signal + sender)

pmain.cxx
  - Add call to Exit(0). T(X)ProofServ::Terminate() just interrupt the event
    loop (insteadof exiting) to avoid problems when the terminate is run in
    a separate thread (like in xproofd).

proof/inc/TProof.h
  - Add methods methods to communicate the data set status (signal + sender)
  - Increase protocol number to 11 (new merging technology)

proof/inc/TProofPlayer.h, .cxx
  - Implement new merging technology for the output list to reduce memory
    consumption: add methods to directly merge received objects in the
    output list
  - Polish cleaning of the various temporary lists and  objects

TProofServ.h, .cxx
  - Implement new merging technology for the output list to reduce memory
    consumption: objects are sent one-by-one
  - Do not scan for previous query results at Setup (do it when the client
    requests for it first); thsi allows to speed up start-up when a large number
    of queries is present and also allos to avoid warning due dictionaries
    not yet loaded.
  - Add method for a global cleanup of old query results.
  - Keep in memory only meta information about processed queries (i.e. not the
    output list) to avoid large memory consumption in case of large
    output lists
  - In Terminate, change the call to Exit(0) to ExitLoop() ; Exit(0) is
    called in pmain.cxx .

TProof.cxx
  - Implement new merging technology for the output list to reduce memory
    consumption: objects are received one-by-one and directly merged in the
    output list
  - Add protection in parallel startup against invelid slaves.
  - Remove check on directory existence in UploadDataSet.
  - Add methods to handle notifications from data set verification
  - Fix inconsistency with notification in activation/deactivation
    of non-existing workers

TXProofServ.cxx
  - In Terminate, change the call to Exit(0) to ExitLoop() ; Exit(0) is
    called in pmain.cxx.

TQueryResult.cxx
  - Make a owned copy of the input list (instead of just storing the pointer) to
    allow proper cleaning.

Revision 15524 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Jun 23 13:26:56 2006 UTC (8 years, 7 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 35876 byte(s)
Diff to previous 15507
From Maarten:
- Simplify and improve proofserv logging.

Revision 15507 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jun 21 16:18:27 2006 UTC (8 years, 7 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 36044 byte(s)
Diff to previous 15338
From Gerri:
The patch implements the following new functionality:

  - Shutdown of idle sessions when quitting ROOT; if not idle the session
    is shutdown as soon as processing terminates; it is possible to delay
    the automatic shutdown by a tunable amount of time via a new directive
    in the xrootd config file 'xpd.shutdown' (see xpd.cf.example).

  - New static method TProof::Reset(const char *url) to hard-cleanup an
    user area (implemented for Xproofd only); this implied quite some
    additions in XrdProofdProtocol.

  - Possibility to temporary remove / re-insert a worker from the active list
       TProof::DeactivateWorker(const char *ord)
       TProof::ActivateWorker(const char *ord)
    with 'ord' the ordinal numer, e.g. "0.3" .

  - A default url for the disk pool
       const char *TVirtualProof::GetDataPoolUrl()
    It is set by default to a string send back by the master, but it can
    be changed via
       void  TVirtualProof::SetDataPoolUrl(const char *url)

    Jan should then adapt his code to use this as default.

  - Posibility to timeout Collect; for now this is only used at forced
    termination.

The patch includes many small changes and bug fixes. One of the most important
is in TXNetSystem (a bug preventing correct treatment of multiple request
to the same redirector).

Revision 15338 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Jun 5 22:51:14 2006 UTC (8 years, 7 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 32939 byte(s)
Diff to previous 15320
From Gerri:
base/inc/TVirtualProof.h, proof/inc/TProof.h, proof/src/TProof.cxx
proof/src/TProofProgressDialog.cxx:
 - fix a problem with the progress dialog when detaching a session:
   a new signal CloseProgressDialog() is introduced, as well as the
   possibility to empty the list of submitted queries at once via
   TProof::Remove().

proofx/inc/TXHandler.h, .cxx:
 - add generic argument to HandleInput() and HandleError().

proofx/inc/TXProofMgr.h, .cxx:
 - move to new signature for HandleInput(), HandleError().

proof/inc/TSlave.h:
 - add method to flush underlying socket.

proofx/inc/TXSlave.h, .cxx:
 - move to new signature for HandleInput(), HandleError().
 - implement new method to flush underlying socket.

proofx/inc/TXSocket.h, .cxx:
 - Add method to flush socket.
 - Handle interrupt messages in the reader thread.
 - Remove interrupt semaphore (not needed any more).
 - Remove timeout argument from GetInterrupt (not needed any more).

proofx/inc/TXProofServ.h, .cxx:
 - move to new signature for HandleInput(), HandleError().
 - implement interrupt handling in HandleInput().
 - implement dedicated version of LockSession to deal with new format
   of session tag (this is a bug fix).

proofd/inc/XrdProofConn.h, .cxx:
 - Add mutex protection in the SendRecv method (it can be called from both
   the reader and main threads).

proofd/src/XrdProofdProtocol.cxx:
 - use an asynchronous message (instead of a signal) for interrupts.

proofx/inc/LinkDef.h, TXSocketHandler.h, .cxx, TXUnixSocket.cxx:
 - add TXSocketHandler in the dictionary.
 - cosmetics.

Revision 15320 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Jun 2 23:38:20 2006 UTC (8 years, 7 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 30433 byte(s)
Diff to previous 15312
From Gerri:
- Get rid of the env to communicate the file descriptor to TProofServ.
  Also, get rid of Reads. I had to add an argument to the TProofServ
  (and TXProofServ) constructor with the 'FILE *' of the open log file.
  Using a global required to link with libProof(x) which we droped recently.
- Drop any (for the time being) unused AFS stuff from TProofServ (this
  is postponed).

Revision 15312 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Jun 2 15:14:35 2006 UTC (8 years, 7 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 30170 byte(s)
Diff to previous 14752
From Gerri:
large PROOF patch. Main issues addressed:

- Output redirection to log file at 'proofserv' startup (instead of
  after Setup())
- Log messages activated at session creation
- Session sandbox restructuring in such a way that the unique session
  tag, created by the top master, uniquely identifies the session on
  all nodes, for example on top master:

   <User_Sandbox>/session-<top_master_session_tag>/
    master-0-<top_master_session_tag>/master-0-<top_master_session_tag>.log

  On worker 0.0

   <User_Sandbox>/session-<top_master_session_tag>/
    worker-0.0-<worker_0.0_session_tag>/worker-0.0-<worker_0.0_session_tag>.log

- Automatic creation of the dataset dir
- Improved handling of error conditions
- Reorganization of TProofServ and XrdProofdProtocol
- Several bug fixes, in particular in XrdProofdProtocol

Revision 14752 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Apr 19 10:57:44 2006 UTC (8 years, 9 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 29961 byte(s)
Diff to previous 14110
From Gerri:
This is the big patch.

Comments:

- Client side:
  - Remove use of plugin manager to avoid unwanted locks
  - Improved treatment of error conditions

- Server side:
  - Several new functionalities described in the release notes

Revision 14110 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sun Feb 26 16:09:57 2006 UTC (8 years, 10 months ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 26031 byte(s)
Diff to previous 13838
Fix missing cvs tag lines.

Revision 13838 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Jan 17 13:23:29 2006 UTC (9 years ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 26032 byte(s)
Diff to previous 13662
From Gerri:
As I wrote TXProofServ before the introduction of TProofResources, I forgot
to use the new class in TXProofServ::Setup(). While fixing this I realized
that we do not really need the dedicated method
TProofResourcesStatic::GetWorkDir(), which basically duplicates
TProofResourcesStatic::ReadConfigFile(). Also, I realized that the check
on the master working dir (which requires the inclusion of TProof.h and
TProofServ.h) is actually a tautology and can be removed.

TProofResourcesStatic.h,.cxx
 - Remove method GetWorkDir()
 - Use 'const char *' in the constructor arguments
 - Remove redundant check on master work dir (this allows to drop inclusion
   of TProof.h and TProofServ.h)

TProofServ.cxx
 - In Setup(), changes needed by the optmization in TProofResourcesStatic

TXProofServ.cxx
 - In Setup(), use TProofResourcesStatic to parse the content of the config file

Revision 13662 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Dec 12 16:42:14 2005 UTC (9 years, 1 month ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 27520 byte(s)
Diff to previous 13648
From Gerri:
- add missing cvs ident lines
- fix an issue with usage of the TProofResources class.

Revision 13648 - (view) (download) (as text) (annotate) - [select for diffs]
Added Mon Dec 12 12:54:27 2005 UTC (9 years, 1 month ago) by rdm
Original Path: trunk/proofx/src/TXProofServ.cxx
File length: 27479 byte(s)
From Gerri:
- last part of the major PROOF upgrade to support disconnect/reconnect
  features using the libXrdProofd.so xrootd plugin.
More on how to configure and run with xrootd/xproofd coming very soon.

This form allows you to request diffs between any two revisions of this file. For each of the two "sides" of the diff, enter a numeric revision.

  Diffs between and
  Type of Diff should be a

Sort log by:

Subversion Admin
ViewVC Help
Powered by ViewVC 1.0.9