[root] Log of /trunk/proof/proofx/inc/TXProofMgr.h

Revision 42858 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Feb 3 11:34:05 2012 UTC (2 years, 11 months ago) by ganis
File length: 4460 byte(s)
Diff to previous 38580

   This patch adds in TProofMgr two static functions to upload files.
   These functionare called UploadFiles and differ in the first argument, which is used
   to pass the files to be uploaded. These can be given as a list (of TFileInfo or TObjString),
   a directory or specified in a text file.
   The functions are documented at
              http://root.cern.ch/drupal/content/uploading-data-files-proof-cluster

   The patch also adds TProofMgr::GetMssUrl() and TProofMgr::SetMssUrl(...) to manage the mass
   storage URL attached to a given cluster. By the default the information is retrieved by the
   server daemon, but can be modified by the user. The functions TProof::GetDataPoolUrl() and
   TProof::SetDataPoolUrl(...) now redirect to TProofMgr::GetMssUrl() and TProofMgr::SetMssUrl(...).

   The obsolete functions TProof::UploadDataset have now been removed.

   This patch addresses some issues raised on Savannah.

Revision 38580 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Mar 23 15:24:54 2011 UTC (3 years, 10 months ago) by ganis
File length: 4417 byte(s)
Diff to previous 30171

  Patch fixing a problem with log path transmission when the node dies early or not even starts.
  The log path was empty and wrong was filled in when retrieving the log buffers, disorienting
  debugging. This was experienced frequently by ALICE.

  The log file is now trasmitted as part of the responde to the session 'Create' request, so it
  is always registered by the master in the relevant field in TSlave. Also, the last errors, if
  any, typically logged in the xproofd log file, are now copied to the relevant worker or master
  log file so that they are also accessible to client.

  The patch also factorizes some calls in XrdProofdProofServMgr in preparation to the improved
  session lauch strategy.

Revision 28746 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu May 28 13:03:29 2009 UTC (5 years, 7 months ago) by ganis
File length: 3155 byte(s)
Diff to previous 26951

   Cleanup of the TProofMgr fucntions DetachSession and ShutdownSession, and better
   handling of the internal list registration, to fix potential segvs when reopening
   a PROOF session inside the same ROOT session.
   Now, for both normal PROOF and PROOF-Lite, it should be possible to close a
   session handled by the TProof object 'p' with any of the following:

       root[] delete p
   or
       root[] TProof::Mgr("<server>")->ShutdownSession(p)
   or
       root[] TProof::Mgr("<server>")->ShutdownSession(id_p)

       (where id_p is the local session id given by TProof::Mgr("<server>")->QuerySessions())

   and re-open a session on the same server right after.

   Other fixes / changes:
    - New auxilliary methods:
      - TProofMgr::DiscardSession(TProof *) to remove a session from the internal
        list of a manager;
      - TProofDesc *TProofMgr::GetProofDesc(TProof *) to quickly get the TProofDesc
        container associated with a session by object
    - Make sure that the default port is always 1093 (in a couple of places it was 1094!)
    - Improve some comments

Revision 26951 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Dec 16 12:04:38 2008 UTC (6 years, 1 month ago) by ganis
File length: 3098 byte(s)
Diff to previous 26823

   - Fix an issue with uploading packages in rapid sequence
   - Fix an issue with TProof::Load() in forced compilation mode ('++')
   - Fix an issue with grep in log retrieval
   - Fix an issue with ownership of the xrootd log file
   - Make sure the XrdProofdProofServMgr::fCurrentSessions is always up-to-date

Revision 26823 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Thu Dec 11 09:26:32 2008 UTC (6 years, 1 month ago) by ganis
File length: 3094 byte(s)
Diff to previous 24719

   - In logs. filter-out by default the service messages used for memory monitoring;
     a new check box has been added to the log dialog box to force display of those
     messages; this reduces considerably the number of 'noise' messages and also the
     default soze of logs.

Revision 24719 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jul 9 07:07:25 2008 UTC (6 years, 6 months ago) by ganis
File length: 3082 byte(s)
Diff to previous 24588

Patch refactorizing the XrdProofd plugin.
The class XrdProofdProtocol is now in charge only of the operations strctly related
to the XProofd protocol. All auxilliary services have been moved to dedicated service
classes controlled by XrdProofdManager. In particular:
- XrdProofdClientMgr handles the clients (represented by instances of XrdProofdClient)
including login, authentication and access control
- XrdProofdProofServMgr handles the PROOF sessions (represented by instances of
XrdProofdProofServ) including creation, attachment, detachment, destruction and
environment setting
- XrdProofdNetMgr handles connections between instances of XProofd running on different
nodes
- XrdProofdPriorityMgr handles the session priorities

A special effort has been done to get rid of all possible internal dead-lock situations.
Internal actions on clients and sessions are now all asynchronous, governed by internal
pipes.

The new plugin also offers new functionality, among which:
- a XProofd admin area, located under <xrd.admin>/.xproof.<port>, keeps information about
active and terminated sessions, and the minimal state for active clients. This is used
to reguraly check the client and session activity, to cleanup orphalin sessions and
to shutdown inactive client connections.
In particular this allows a more solid implementation of Reset, which now exists in two
flavours: 'soft', TProof::Reset(<masterurl>), (default) asks the sessions to terminate gently;
'hard', TProof::Reset(<masterurl>,kTRUE) schedules all sessions for forced termination.
- support for automatic attempts for reconnection in the case the daemon restarts.
This allows to reconfigure the plugin by xrootd restart w/o affecting the running
sessions.
- support the definition of workers via config file directive, getting de facto rid of
the proof.conf file.
- domain+level control of printout message; the format has been improved: in particular
all information messages contain now the tag 'xpd-I' and all error messages the tag
'xpd-E', so that they can easily be grepped out from the log file.

The Wiki 'XrdProofdDirectives Directives' page has been updated with the new directives.

Revision 20862 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sun Nov 18 14:51:42 2007 UTC (7 years, 2 months ago) by ganis
Original Path: trunk/proofx/inc/TXProofMgr.h
File length: 2856 byte(s)
Diff to previous 20682

   Synchronize with braches/dev/proof r20835

   Summary:

   TProofServ:
     - read session.rootrc with level kEnvChange to be able to change existing
       settings
     - Avoid deleting a query result twice in some special cases
     - Fix a problem with the initialization of fKeptQueries
   rootrc.in:
     - restore default settings the asynchronous reading
   TFileCacheRead:
     - add missing protection in ReadBuffer
   TEventIter:
     - Enable the usage of TTreeCache
   XrdProofdProtocol:
     - Additional check on the ownership of the unix socket
     - Improve notification during Reset
     - Reduce default timeout on admin requests and make it configurable
   XrdProofConn:
     - Use the configurable maxtry everywhere where relevant
   TPacketizerAdaptive:
     - Store info on all the processed packets in per-worker lists

   TProofMgr, TXProofMgr, TXSocket, XProofProtocol, XrdProofdProtocol:
     - Add possibility for the admin to broadcast a message to the connected users

   getProof:
     - Add missing protection

Revision 20682 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Nov 6 15:51:59 2007 UTC (7 years, 2 months ago) by ganis
Original Path: trunk/proofx/inc/TXProofMgr.h
File length: 2787 byte(s)
Diff to previous 19826

  Import branches/dev/proof r20654

  Summary:

  + Improvements
    - Add support for SSH SOCKS4 tunnelling; the local port for the tunnel can be specified
      in the master URL, e.g. TProof::Open("master/?tunnel:8000")
    - Add the possibility to plot the estimated instantaneous rate
    - Add "PROOF_ForceLocal" parameter to the !TPacketizerAdaptive; if set to 1, all the data
      are processed locally.
    - Add support for remote grep functionality while retrieving logs (needed by the forthcoming
      memory monitor)
  + Bug fixes
    - Several small fixes to revive the multi-master mode.
    - XrdProofdProtocol:
      - add missing lock to the client instance in SendMsg to avoid screwing up requests from
        workers on the same machine
         - lock the mutex of the requester when setting prorities
         - add notification during Reset
         - fix problem with the detection of the 'allow' directive
         - fix problem with the parsing of the return value from XrdProofServProxy::TerminateProofServ()
         - always use the effective user to retrieve info fom another server
           (XrdProofdManager::GetProofConn is now used)
         - fix possible dead-locks from debug notifications done after hard-killing a session
         - re-enable the garbage collector thread of the connection manager in XrdProofConn to
           fix a problem with closing physical connections;
         - fix a problem with !CleanupProofServ in the case of a non-privileged daemon running
           in multi-user mode
         - introduce a timeout when waiting for the startup of a 'proofserv'.
    - XrdProofConn: init mutex in the ctor; lock in SendRecv
    - XrdProofSched: add support for using the priorities defined in the group manager to define
      the number of workers for sessions
      - TProof:
         - Broadcast priorities to unique nodes only
         - timeout after 5 mins the initial Collect to avoid clients getting stuck at this stage
         - add support for generic timeout in Collect (disabled by default)
         - fix a problem with SendFile.
      - TXProofServ: add a call to !TProof::InterruptCurrentMonitor() in Terminate() to stop 
        infinite loops in Collect
      - TXSocket:
         - Implement a flag to interrupt a TXSocket while waiting for messages
         - Split the session creation timeout in 4 attempts: the total timeout is the same but
           it may circumvent occasional forking problems.
      - XROOTD: 
         - fix a potential (possibly accademic) memory leak in the client
         - fix a access permission problem with Kerberos ticket forwarding
         - fix bug preventing 'locate' to work properly
         - re-enable optimized 'locate'

Revision 18353 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Mar 19 15:14:10 2007 UTC (7 years, 10 months ago) by rdm
Original Path: trunk/proofx/inc/TXProofMgr.h
File length: 2725 byte(s)
Diff to previous 17035

From Gerri:
- Support for multiple ROOT versions on a PROOF cluster.
  TProofMgr::ShowROOTVersions() giving the following output:

root [0] TProof::Mgr("")->ShowROOTVersions()
----------------------------------------------------------

Available versions (tag ROOT-vers remote-path PROOF-version):

 * cvs 5.15/03 /home/ganis/local/root/head/head 12
   dev 5.15/03 /home/ganis/local/root/head/root 12

----------------------------------------------------------

- TProofMgr::SetROOTVersion(const char *version_tag)

    to change the default for the user, eg

root [1] TProof::Mgr("")->SetROOTVersion("dev")
root [2] TProof::Mgr("")->ShowROOTVersions()
----------------------------------------------------------

Available versions (tag ROOT-vers remote-path PROOF-version):

   cvs 5.15/03 /home/ganis/local/root/head/head 12
 * dev 5.15/03 /home/ganis/local/root/head/root 12

----------------------------------------------------------

Available version are defined by the administrator via the
"xpd.rootsys" directive:

xpd.rootsys /home/ganis/local/root/head/head cvs
xpd.rootsys /home/ganis/local/root/head/root dev

in the xrootd config file.

- Improve error reporting in TXSocket
- Fix a problem with resetting a label in TProofProgressDialog

Revision 17035 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Sun Dec 3 23:34:04 2006 UTC (8 years, 1 month ago) by rdm
Original Path: trunk/proofx/inc/TXProofMgr.h
File length: 2642 byte(s)
Diff to previous 16969

From Gerri:
0. Introduction of the static:
      TProof::Mgr(const char *url)
   to get the manager to the cluster; this gives access to all the manager
   functionality, included Reset(). TProof::Reset() is removed.

1. Two new classes: TProofLog and TProofLogElem.
   TProofLog contains a list of TProofLogElem, one per node. TProofLogElem
   basically contains a TMacro plus methods to play  with it.

2. The TProofLog object can be retrieved via the manager:

      TProofLog *pl = TProof::Mgr("cluster")->GetSessionLogs()

   by default the last session is retrieved: the signature for GetSessionLogs is

      TProofLog *TProofMgr::GetSessionLogs(Int_t ridx, const char *stag)

   the first argument is a reversed relative index (-1 gets the next to last
   session), the second is the session unique tag; if the second is
   specified the first is ignored.

3. TProofMgr has a new method ReadBuffer to quickly read a buffer from
   a file via XrdProofd: this allows to avoid the second xrootd on the
   redirector and in general is useful when a quick reading is needed.
   Also, the buffer is always retrieved via the master, so we should not
   have firewall problems in accessing directly the xrootd of the workers.

4. In the sandbox I have kept the .sessions file with the active sessions
   and I have added .terminated in each session working dir which have
   been terminated. On the master, each session working dir has the file
   .workers with the relevant information about the workers.

5. The max number of old session workdirs is set to 10 by default
   (configurable in xpd.cf): the sandbox is trimmed when closing a session.

6. Some unrelated changes/fixes contained in the patch:
   - in XrdProofdProtocol:
      - use of /proc (linux, solaris) or sysctl (mac) to verify/search
        for running of a process. For other platforms I have left the
        parsing of the output of 'ps'.
      - I have introduced a 'cron'  thread: it runs every minute and for
        the moment it just cleans a vector; this is the place where to do
        client independent actions, like checking if the config files have
        changed, for example.
      - I have added the possibility to control which user can access
        the cluster; requested by Jan Fiete.

7. The patch contains also several fixes here and there I have found by
   testing.

Revision 16969 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Tue Nov 28 12:10:52 2006 UTC (8 years, 1 month ago) by rdm
Original Path: trunk/proofx/inc/TXProofMgr.h
File length: 2499 byte(s)
Diff to previous 15507

From Gerri:
This patch eliminates TVirtualProof, TVirtualProofMgr and TVirtualProofDesc.
The first is integrated in TProof, the second in TProofMgr and the third
becomes TProofDesc defined in TProofMgr.h.

Revision 15507 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Jun 21 16:18:27 2006 UTC (8 years, 7 months ago) by rdm
Original Path: trunk/proofx/inc/TXProofMgr.h
File length: 2527 byte(s)
Diff to previous 15338

From Gerri:
The patch implements the following new functionality:

  - Shutdown of idle sessions when quitting ROOT; if not idle the session
    is shutdown as soon as processing terminates; it is possible to delay
    the automatic shutdown by a tunable amount of time via a new directive
    in the xrootd config file 'xpd.shutdown' (see xpd.cf.example).

  - New static method TProof::Reset(const char *url) to hard-cleanup an
    user area (implemented for Xproofd only); this implied quite some
    additions in XrdProofdProtocol.

  - Possibility to temporary remove / re-insert a worker from the active list
       TProof::DeactivateWorker(const char *ord)
       TProof::ActivateWorker(const char *ord)
    with 'ord' the ordinal numer, e.g. "0.3" .

  - A default url for the disk pool
       const char *TVirtualProof::GetDataPoolUrl()
    It is set by default to a string send back by the master, but it can
    be changed via
       void  TVirtualProof::SetDataPoolUrl(const char *url)

    Jan should then adapt his code to use this as default.

  - Posibility to timeout Collect; for now this is only used at forced
    termination.

The patch includes many small changes and bug fixes. One of the most important
is in TXNetSystem (a bug preventing correct treatment of multiple request
to the same redirector).

Revision 15338 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Jun 5 22:51:14 2006 UTC (8 years, 7 months ago) by rdm
Original Path: trunk/proofx/inc/TXProofMgr.h
File length: 2484 byte(s)
Diff to previous 15312

From Gerri:
base/inc/TVirtualProof.h, proof/inc/TProof.h, proof/src/TProof.cxx
proof/src/TProofProgressDialog.cxx:
 - fix a problem with the progress dialog when detaching a session:
   a new signal CloseProgressDialog() is introduced, as well as the
   possibility to empty the list of submitted queries at once via
   TProof::Remove().

proofx/inc/TXHandler.h, .cxx:
 - add generic argument to HandleInput() and HandleError().

proofx/inc/TXProofMgr.h, .cxx:
 - move to new signature for HandleInput(), HandleError().

proof/inc/TSlave.h:
 - add method to flush underlying socket.

proofx/inc/TXSlave.h, .cxx:
 - move to new signature for HandleInput(), HandleError().
 - implement new method to flush underlying socket.

proofx/inc/TXSocket.h, .cxx:
 - Add method to flush socket.
 - Handle interrupt messages in the reader thread.
 - Remove interrupt semaphore (not needed any more).
 - Remove timeout argument from GetInterrupt (not needed any more).

proofx/inc/TXProofServ.h, .cxx:
 - move to new signature for HandleInput(), HandleError().
 - implement interrupt handling in HandleInput().
 - implement dedicated version of LockSession to deal with new format
   of session tag (this is a bug fix).

proofd/inc/XrdProofConn.h, .cxx:
 - Add mutex protection in the SendRecv method (it can be called from both
   the reader and main threads).

proofd/src/XrdProofdProtocol.cxx:
 - use an asynchronous message (instead of a signal) for interrupts.

proofx/inc/LinkDef.h, TXSocketHandler.h, .cxx, TXUnixSocket.cxx:
 - add TXSocketHandler in the dictionary.
 - cosmetics.

Revision 15312 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Fri Jun 2 15:14:35 2006 UTC (8 years, 7 months ago) by rdm
Original Path: trunk/proofx/inc/TXProofMgr.h
File length: 2466 byte(s)
Diff to previous 14752

From Gerri:
large PROOF patch. Main issues addressed:

- Output redirection to log file at 'proofserv' startup (instead of
  after Setup())
- Log messages activated at session creation
- Session sandbox restructuring in such a way that the unique session
  tag, created by the top master, uniquely identifies the session on
  all nodes, for example on top master:

   <User_Sandbox>/session-<top_master_session_tag>/
    master-0-<top_master_session_tag>/master-0-<top_master_session_tag>.log

  On worker 0.0

   <User_Sandbox>/session-<top_master_session_tag>/
    worker-0.0-<worker_0.0_session_tag>/worker-0.0-<worker_0.0_session_tag>.log

- Automatic creation of the dataset dir
- Improved handling of error conditions
- Reorganization of TProofServ and XrdProofdProtocol
- Several bug fixes, in particular in XrdProofdProtocol

Revision 14752 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Wed Apr 19 10:57:44 2006 UTC (8 years, 9 months ago) by rdm
Original Path: trunk/proofx/inc/TXProofMgr.h
File length: 2292 byte(s)
Diff to previous 14110

From Gerri:
This is the big patch.

Comments:

- Client side:
  - Remove use of plugin manager to avoid unwanted locks
  - Improved treatment of error conditions

- Server side:
  - Several new functionalities described in the release notes

This form allows you to request diffs between any two revisions of this file. For each of the two "sides" of the diff, enter a numeric revision.

Links to HEAD:	(view) (download) (as text) (annotate)
Sticky Revision: