You are here

Enabling query monitoring

  1. Syslog-based monitoring
    1. Configuration
      1. Defining the log file
      2. ProofServ.LogToSysLog
    2. Information posted
  2. The ProofServ.Monitoring directive 
    1. Configuration
      1. MonALISA
      2. SQL
      3. The sendopts switch
    2. Information posted
      1. MonALISA
        1. Query Summary Record
        2. Dataset Record
        3. Files Record
      2. SQL
        1. Query Summary Record
        2. Dataset Record
        3. Files Record
        4. Configuring the bulk INSERT
    3. Additional monitoring (MonALISA only)

1. Syslog-based monitoring

Since ROOT version 5.27/04 (SVN revision #33369; the functionality is also available in the special branch 5.26/00-proof) it is possible to configure PROOF in such a way that some information about the activity on the cluster is logged via the syslog machinery. A tool to parse and visualize this information is under development. 

1.1 Configuration

Enabling monitoring via syslog requires two steps: i) instructing the syslog system to redirect the 'local5' logs to a log file dedicated to PROOF monitoring; ii) instructing PROOF to send information to syslog.

1.1.1 Defining the log file

The syslog facility used by PROOF is 'local5' . In order to connect this facility to a local file, first choose a location for the file, e.g. /var/log/xpdmon.log (in the following we will use this path in the examples), and create the empty file. We have then to instruct the syslog daemon that logs to local5 should go to tye chosen file; to achieve this we have to edit the syslog config file, usually /etc/syslog.conf to add the following line

#  /etc/syslog.conf     Configuration file for syslogd.
...
#
# PROOF logging
local5.debug                     /var/log/xpdmon.log

We have also to tell syslog not to send local5 logs to the default file, usually /var/log/messages. For that we need to find in /etc/syslog.conf the line similar to this

*.info;mail.none;news.none;authpriv.none  /var/log/messages
and change it to look as this
*.info;local5,mail.none;news.none;authpriv.none  /var/log/messages
To make the changes affective we need to restart the syslog daemon
/etc/init.d/syslog restart
If a full restart is not desired, forcing a re-initialization should be sufficient: this can be done by sending a SIGHUP signal to the daemon
kill -SIGHUP ‘cat /var/run/syslogd.pid‘

1.1.2 The ProofServ.LogToSysLog directive

By default, the proofserv application does not log anything to syslog. To enable sending the information one needs to set a ProofServ.LogToSysLog directive . This is usually done via the xpd.putrc directive of the XrdProofd daemon.

### Log to syslog
xpd.putrc ProofServ.LogToSysLog 
The directive defines a 2 character string: the first character is a letter which can take the following values:
a log all
m log only master
w log only workers

The second character is a number indicating the log level:

0 log disabled
1 log start-of-session, main actions (exec, process, dataset handling, cache/package handling, query handling) end-of-session
2 As 1 but log also all the remaining activity on the input socket (this includes handling of some auxilliary messages)
3 Log all proofserv logs (i.e. those usually going to the log file) in addition to those obtained with 2 

The typical setting to monitor the cluster usage is 'm1', i.e. log level 1 on the master.

Example of output obtained from running the "eventproc" tutorial example:

May  4 10:43:55 pcphsft64 proofm-0[14889]: ganis:proofteam 0 0.000 0.000
May  4 10:43:57 pcphsft64 proofm-0[14889]: ganis:proofteam 1012 0.346 0.000 +event.par
May  4 10:43:58 pcphsft64 proofm-0[14889]: ganis:proofteam 1018 0.949 0.020 6 event 0
May  4 10:43:59 pcphsft64 proofm-0[14889]: ganis:proofteam 1018 0.069 0.000 7 event 0
May  4 10:43:59 pcphsft64 proofm-0[14889]: ganis:proofteam 1018 0.001 0.000 8 0
May  4 10:43:59 pcphsft64 proofm-0[14889]: ganis:proofteam 1012 0.055 0.020 ProofEventProc.C
May  4 10:43:59 pcphsft64 proofm-0[14889]: ganis:proofteam 1012 0.009 0.000 ProofEventProc.h
May  4 10:44:58 pcphsft64 proofm-0[14889]: ganis:proofteam 1015 59.365 0.670 0 1000000 596961398 83.530
May  4 10:45:04 pcphsft64 proofm-0[14889]: ganis:proofteam -1 60.819 0.720

1.2 Information posted

We describe in this section the format of the information obtained in the case of level 1. The record hs the following format

date nodename proofid[pid]: user:group what realtime cputime auxinfo
where
  • date is the date in the form 'day month time', e.g. 'May  4 11:37:54'
  • nodename is the name of the node where the information is recorded
  • proofid is a tag indicating the role of the node, e.g. proofm-0 for a top master, or proofw-0.16 for a worker (ordinal 0.16)
  • pid is the process ID of the proofserv logging the information
  • user is the user name of the session producing the info
  • group is the PROOF group of the user producing the info
  • what is the message type of the requested action
  • realtime is the real time spent executing the action
  • cputime is the CPU time spent executing the action
  • auxinfo is some auxilliary information which may be present depending on what (see below).

A record with what = 0 indicates the beginning of the session (realtime and cputime are 0.000 in this case). A record with what = -1 indocates the end of the session; realtime and cputime are, in this case, the total real and CPU time spent in execution during the whole session, respectively.

Auxilliary information is present in the following cases:

The monitoring information is sent by the master at the end of the query. The quantities posted are shown in the table:

what (name) what (id) Format Description
kMESS_CINT 5 cmd Command being executed
kPROOF_PROCESS 1015 status entries bytes cputime

status: query status

entries: number of entries processed (or cycles performed)

bytes: number of bytes read if processing data

cputime: total CPU time used by the workers

kPROOF_RETRIEVE

kPROOF_REMOVE

1032

1034

queryref Query reference ID
kPROOF_ARCHIVE 1033 queryref path Query reference ID and archiving path
kPROOF_CHECKFILE 1012 filename Name of the file being checked
kPROOF_CACHE 1018 type aux Cache handling sub-action type; the auxilliary info is described in the table below
kPROOF_DATASETS 1042 type aux Dataset handling sub-action type; the auxilliary info is described in the table below

 

 

 

2. The ProofServ.Monitoring directive

It is possible to configure PROOF in such a way that some information about the queries processed on the cluster are sent to a MonALISA monitoring system or to a SQL database. Starting from 5.29/02 (and 5.28/00c) it is possible to send the information to multiple collectors entering multiple 'ProofServ.Monitoring' directives, separated by a ',' or a '|' or a '\'; for example

ProofServ.Monitoring SQL mysql://localhost:3306  proof.proofquerylog,
+ProofServ.Monitoring: MonaLisa 192.168.2.22 ::::

 

 

To enable posting the monitoring information one has to give instruction about which monitoring system has to be used. This is done via the rootrc directive 'ProofServ.Monitoring'. This directive takes a name and up to 9 'const char *' additional configuration arguments for the relevant TProofMonSender plug-in. The TProofMonSender plug-ins available currently are TProofMonSenderML (for a MonALISA backend) and TProofMonSenderSQL (for SQL backends).

Note that these variables can be defined in the XROOTD configuration file via the 'xpd.putrc' directive .

Note that, for ROOT version up to 5.28/00f and 5.30/01, the directives were used to intialize directly the TMonaLisaWriter and TSQLMonitoringWriter plug-ins.

 

2.1 Configuration

2.1.1 MonALISA

The directive to load the MonALISA writer has the form:

 ProofServ.Monitoring MonaLisa 

The parameter server specifies the server to whom to send the monitoring UDP packets. If not specified, the hostname is specified in the environment variable APMON_CONFIG, with the default port.

The parameter tag is equivalent to the MonaLisa farm name; for the case of process monitoring it can be a process identifier e.g. a PROOF session ID.

The parameter id is equivalent to the MonaLisa node name; for the case of process monitoring it can be just an identifier to classify the type of jobs e.g. PROOF_PROCESSING. If id is not specified, TMonaLisaWriter tries to set it from environment variables in this order: PROOF_JOB_ID, GRID_JOB_ID, LCG_JOB_ID, ALIEN_MASTERJOB_ID, ALIEN_PROC_ID.

The parameter subid can be used to have finer identifier granularity.

The parameter option is currently only used as local/global switch: if set as "global" the global gMonitoringWriter is set to the instantiated instance of TMonaLisaWriter, so that a unique monitoring writer is used in the session. In PROOF, use "global" to send fine grained processing information (see below).

The sendopts switch is described below.

2.1.2 SQL

The directive to load the SQL writer is:

 ProofServ.Monitoring SQL  [. [.] [.]] [sendopts:]

The parameter DBMS is the URL used to identify the DB server, in the form

://[:][/]

The parameter user is the user-name to be used to connect to the DBMS.

The parameter passwd is the password associated with the user-name above mentioned.

The fourth parameter indicates the database and the table where to insert the monitoring information. Specifying the tables databases and names is not mandatory. The default for the database is 'proof' and for the tables 'proofquerylog', 'proofquerydsets' and 'proofqueryfiles'. If the  summarytable and its database dbs  are specified and the other tables are not, then the database dbs is used for all the tables, with default table names.

The sendopts switch is described below.

2.1.3 The sendopts switch

The sendopts string allows to configure the records to be sent and their version (see below); the string contains ':' separated information for each record in the form [{+,-}]RecordTag[RecordVersion]; the '-'('+') prefix disables(enables) the sending of a record (if absent '+' is assumed). Recognized values for the RecordTag are given in the table:

RecordTag Description
S Query summary
D Information about processed datasets
F Information about processed files

For example, if sendopts is set to 'S1:D0:-F', the process will post the query summary record, version 1, and the dataset record, version 0, while detailed information about the processed files will not be sent.

The optional RecordVersion is an integer number specifying the version of the record. For dataset and files information records, the version constrols the amount of information in the record as described below in the relevant sections. For the query summary record, it is a way to provide the possibility to post information as in previous ROOT versions. The following table gives the correspondence:

Summary Record Version

ROOT Versions
0
1 5.28/00c -> 5.28/00f, 5.30/00, 5.30/01
2 > 5.28/00f, > 5.30/01

 

2.2 Information posted

The monitoring information is sent by the master at the end of the query. In this section we give a detailed secription of the records sent under the different configuration options.

2.2.1 MonALISA

2.2.1.1 Query Summary Record

The quantities posted in the query summary record are shown in the table:

Description Name Type  ver 0  ver 1  ver 2
User name user XDR_STRING o o o
Group name group XDR_STRING o    
Group name proofgroup XDR_STRING   o o
Starting processing time begin XDR_STRING o o o
End processing time end XDR_STRING o o o
Total wall time walltime XDR_REAL64 o o o
Total CPU time cputime XDR_REAL64 o o o
Number of bytes read bytesread XDR_REAL64 o o o
Number of events processed events XDR_REAL64 o o o
Number of workers assigned workers XDR_REAL64 o o o
Max virtual memory used by workers (kB) vmemmxw XDR_REAL64   o o
Max resident memory used by workers (kB) rmemmxw XDR_REAL64   o o
Max virtual memory used during merging (kB) vmemmxm XDR_REAL64   o o
Max resident memory used during merging (kB) rmemmxm XDR_REAL64   o o
Dataset name dataset XDR_STRING   o  
Number of files in the dataset numfiles XDR_REAL64   o o
Number of missing files missfiles XDR_REAL64     o
Query exit status status XDR_REAL64     o
ROOT versions rootver XDR_STRING     o

The entry is tagged with the unique tag "-", e.g. "pcphsft64-1237223578-30248-3" .

The 'vmemmxw' and 'rmemmxw' represent the maximum virtual and resident memory (in kB) used by worker during processing; the 'vmemmxm' and 'rmemmxm' are the maximum values during merging on the master.

For version 1, the user and proof-group names are left in the dataset string only if different from the 'user' and 'proofgroup' of the query; multiple dataset are entered as a comma- separated list; for non-dataset based queries, the name of the class used to define the set of files is entered, e.g. TChain, TDSet or TFileCollection. The string '+++none+++' is used for non-data drive analysis.

The 'status' field contains the exit status of the query (0 = OK; 1 = stopped, 2 = aborted).

2.2.1.2 Dataset Record

The quantities posted in the dataset record are shown in the table:

Description Name Type  ver 0  ver 1
Dataset name dsn XDR_STRING o o
Query tag querytag XDR_STRING o o
Starting processing time querybegin XDR_STRING   o
Number of files in the dataset numfiles XDR_REAL64  o o
Number of missing files missfiles XDR_REAL64 o  o

The entry is tagged with "dataset_", e.g. "dataset_d056756f0" . The 'querytag' gives the link with the query summary record to lookup other quantities, e.g. the user requiring the dataset. The number of files is the number of files registered to this dataset; the number of missing files is the number of the required files not available or that could not be opened at the moment the query was run.

2.2.1.3 Files Record

The quantities posted in the files record are shown in the table:

Description Name Type  ver 0  ver 1
File basename lfn XDR_STRING o o
Full path (including Url) path XDR_STRING o o
Starting processing time querybegin XDR_STRING   o
Availability status status XDR_REAL64  o o

 

The entry is tagged with "file_", e.g. "file_e256736a1" . The 'querytag' gives the link with the query summary record to lookup other quantities, e.g. the user requiring the file. The 'status' tells if the file was successfully analysed (value 1) or not (value 0) by the query.

 

2.2.2 SQL

2.2.2.1 Query Summary Record

The quantities posted in the query summary record are shown in the table:

Description Name Type  ver 0  ver 1  ver 2
User name user VARCHAR(32) o o  
User name proofuser VARCHAR(32)     o
Group name group VARCHAR(32) o    
Group name proofgroup VARCHAR(32)   o o
Starting processing time begin DATETIME o o  
Starting processing time querybegin DATETIME     o
End processing time end DATETIME o o  
End processing time queryend DATETIME     o
Total wall time walltime INT o o o
Total CPU time cputime FLOAT o o o
Number of bytes read bytesread BIGINT o o o
Number of events processed events BIGINT o o o
Number of workers assigned workers INT o o o
Query unique tag query VARCHAR(64)   o o
Max virtual memory used by workers (kB) vmemmxw BIGINT   o o
Max resident memory used by workers (kB) rmemmxw BIGINT   o o
Max virtual memory used during merging (kB) vmemmxm BIGINT   o o
Max resident memory used during merging (kB) rmemmxm BIGINT   o o
Dataset name dataset VARCHAR(512)   o  
# of files requested numfiles INT   o o
# of missing files missfiles INT     o
Query exit status status INT     o
ROOT version rootver VARCHAR(32)     o

 

The 'vmemmxw' and 'rmemmxw' represent the maximum virtual and resident memory (in kB) used by worker during processing; the 'vmemmxm' and 'rmemmxm' are the maximum values during merging on the master.

For version 1, the user and proof-group names are left in the dataset string only if different from the 'user' and 'proofgroup' of the query. Multiple dataset are entered as a comma- separated list. The dataset string is in any case truncated at 512 max characters. For non-dataset based queries, the name of the class used to define the set of files is entered, e.g. TChain, TDSet or TFileCollection. The string '+++none+++' is used for non-data drive analysis.

The 'status' field contains the exit status of the query (0 = OK; 1 = stopped, 2 = aborted).

The table can created in the following way (any name can be used for the table, provided that the correct value is used in the relevant ProofServ.Monitoring directive) :

Version 2:

CREATE TABLE proofquerylog (
  id int(11) NOT NULL auto_increment,
  proofuser varchar(32) NOT NULL,
  proofgroup varchar(32) default NULL,
  querybegin datetime default NULL,
  queryend datetime default NULL,
  walltime int(11) default NULL,
  cputime float default NULL,
  bytesread bigint(20) default NULL,
  events bigint(20) default NULL,
  totevents bigint(20) default NULL,
  workers int(11) default NULL,
  querytag varchar(64) NOT NULL,
  vmemmxw bigint(20) default NULL,
  rmemmxw bigint(20) default NULL,
  vmemmxm bigint(20) default NULL,
  rmemmxm bigint(20) default NULL,
  numfiles int(11) default NULL,
  missfiles int(11) default NULL,
  status int(11) default NULL,
  rootver varchar(32) NOT NULL,
  PRIMARY KEY  (id)
)

Version 1:

CREATE TABLE proofquerylog (
  id int(11) NOT NULL auto_increment,
  proofuser varchar(32) NOT NULL,
  proofgroup varchar(32) default NULL,
  querybegin datetime default NULL,
  queryend datetime default NULL,
  walltime int(11) default NULL,
  cputime float default NULL,
  bytesread bigint(20) default NULL,
  events bigint(20) default NULL,
  totevents bigint(20) default NULL,
  workers int(11) default NULL,
  querytag varchar(64) NOT NULL,
  vmemmxw bigint(20) default NULL,
  rmemmxw bigint(20) default NULL,
  vmemmxm bigint(20) default NULL,
  rmemmxm bigint(20) default NULL,
  numfiles int(11) default NULL,
  dataset varchar(512) NOT NULL,
  PRIMARY KEY  (id)
)

Version 0:

CREATE TABLE proofquerylog (
  id int(11) NOT NULL auto_increment,
  proofuser varchar(32) NOT NULL,
  proofgroup varchar(32) default NULL,
  querybegin datetime default NULL,
  queryend datetime default NULL,
  walltime int(11) default NULL,
  cputime float default NULL,
  bytesread bigint(20) default NULL,
  events bigint(20) default NULL,
  totevents bigint(20) default NULL,
  workers int(11) default NULL,
  PRIMARY KEY  (id)
)

 

2.2.2.2 Dataset Record

The quantities posted in the dataset record are shown in the table:

Description Name Type  ver 0  ver 1
Dataset name dsn VARCHAR(512) o o
Query unique tag querytag VARCHAR(64)  o o
Starting processing time querybegin DATETIME    o
# of files requested numfiles INT  o o
# of missing files missfiles INT  o  o

 

The 'querytag' gives the link with the query summary record to lookup other quantities, e.g. the user requiring the dataset. The number of files is the number of files registered to this dataset; the number of missing files is the number of the required files not available or that could not be opened at the moment the query was run.

 

The table can created in the following way (any name can be used for the table, provided that the correct value is used in the relevant ProofServ.Monitoring directive) :

Version 1:

CREATE TABLE proofquerydsets (
   id int(11) NOT NULL auto_increment,
   dsn varchar(512) NOT NULL,
   querytag varchar(64) NOT NULL,
   querybegin datetime default NULL,
   numfiles int(11) default NULL,
   missfiles int(11) default NULL,
   PRIMARY KEY (id),
   KEY ix_querytag (querytag)
)

Version 0:

CREATE TABLE proofquerydsets (
   id int(11) NOT NULL auto_increment,
   dsn varchar(512) NOT NULL,
   querytag varchar(64) NOT NULL,
   numfiles int(11) default NULL,
   missfiles int(11) default NULL,
   PRIMARY KEY (id),
   KEY ix_querytag (querytag)
)

 

2.2.2.3 Files Record

The quantities posted in the files record are shown in the table:

Description Name Type  ver 0  ver 1
File basename lfn VARCHAR(255) o o
Full path (including Url) path VARCHAR(2048) o o
Query unique tag querytag VARCHAR(64) o o
Starting processing time querybegin DATETIME   o
File availability status status enum('Ok','Failed') o o

 

The 'querytag' gives the link with the query summary record to lookup other quantities, e.g. the user requiring the file. The 'status' tells if the file was successfully analysed (value 1) or not (value 0) by the query.

 

The table can created in the following way (any name can be used for the table, provided that the correct value is used in the relevant ProofServ.Monitoring directive):

Version 1:

CREATE TABLE proofqueryfiles (
   id int(11) NOT NULL auto_increment,
   lfn varchar(255) NOT NULL,
   path varchar(2048) NOT NULL,
   querytag varchar(64) NOT NULL,
   querybegin datetime default NULL,
   status enum('Ok','Failed') NOT NULL default 'Ok',
   PRIMARY KEY (id),
   KEY ix_querytag (querytag)
)

Version 0:

CREATE TABLE proofquerydsets (
   id int(11) NOT NULL auto_increment,
   lfn varchar(255) NOT NULL,
   path varchar(2048) NOT NULL,
   querytag varchar(64) NOT NULL,
   status enum('Ok','Failed') NOT NULL default 'Ok',
   PRIMARY KEY (id),
   KEY ix_querytag (querytag)
)

 

2.2.2.4 Configuring the bulk INSERT

The dataset and files records are posted with a bulk INSERT. The maximal size for such an operation is controlled by the max_allowed_packet configuration parameter of 'mysqld', which should default to 16 MB . The TSQLMonitoring plug-in honours this by sending off a bulk chunk as soon as it reaches 90% of the maximal value, set by default to 16 MB. The maximal value can be changed with the directive SQLMonitoringWriter.MaxBulkSize which understands bytes, kilobytes (suffix 'k' or 'K'), megabytes (suffix 'm' or 'M') and gigabytes (suffix 'g' or 'G'). For example, the following

xpd.putrc SQLMonitoringWriter.MaxBulkSize 1048576
xpd.putrc SQLMonitoringWriter.MaxBulkSize 1024k 
xpd.putrc SQLMonitoringWriter.MaxBulkSize 1M 
are equivalent ways to set the maximal value to 1 MB.

 

3 Additional monitoring (MonALISA only)

If working with MonALISA, it is possible to send processing progress information to the monitoring server. The following sequence is sent on per worker base:

    1) Just before requesting the first packet, an entry with:

Name Type Description Value
status XDR_STRING Status flag STARTED
hostname XDR_STRING Worker host name  
subid XDR_STRING MonaLisa instance subid  

    2) After each call to TSelector::Process, an entry with:

Name Type Description Value
subid XDR_STRING MonaLisa instance subid  
events XDR_REAL64 Number of events processed  
processedbytes XDR_REAL64 Number of bytes read  
realtime XDR_REAL64 Real (wall) time from the start  
cputime XDR_REAL64 CPU time used  
totmem XDR_REAL64 Total memory used  
rssmem XDR_REAL64 Resident memory used  
shdmem XDR_REAL64 Shared memory used 0
events_str XDR_STRING String version of 'events'  
processedbytes_str XDR_STRING String version of 'processedbytes'  
realtime_str XDR_STRING String version of 'realtime'  
cputime_str XDR_STRING String version of 'cputime'  
totmem_str XDR_STRING String version of 'totmem'  
rssmem_str XDR_STRING String version of 'rssmem'  
shdmem_str XDR_STRING String version of 'shdmem' "0"
hostname XDR_STRING Worker host name  

    3) Just after the last call to TSelector::Process, an entry with:

Name Type Description Value
status XDR_STRING Status flag DONE
hostname XDR_STRING Worker host name  
subid XDR_STRING MonaLisa instance subid  

To enable the sending of this information the option parameter must be set to "global" in the MonALISA plug-in configuration directive.