Log of /trunk/proof/proof/src/TDSet.cxx
Parent Directory
Revision
43044 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 17 14:34:52 2012 UTC (2 years, 11 months ago) by
ganis
File length: 64334 byte(s)
Diff to
previous 40699
From Taesoo Kim and me.
Add support for selector-by-object processing in PROOF.
The selector object, created and configured locally by the user, is added to the input
list and recuperated from there on the worker machines for processing. Any input list
setting in the selector itself is not streamed but temporarly moved to then standard
input list, so that user can use the selector input list as container of processing
information if they find convenient to do so.
Process(...) methods with the file name argument replaced by 'TSelector *' have
introduced where relevant (TProof, TProofPlayer and their derivatives, TDSet).
Two tests for the new functionality have been added to test/stressProof.cxx (tests 25
and 26).
Revision
40699 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Aug 25 10:36:29 2011 UTC (3 years, 5 months ago) by
rdm
File length: 63507 byte(s)
Diff to
previous 38810
From Gerri:
Patch adding support for backend-dependent record formatting of PROOF
monitoring. This is achieved by introducing a new layer, described by
the abstract interface TProofMonSender, with the record format defined
in the backend implemenattion (currently TProofMonSenderML, for MonaLisa,
and TProofMonSenderSQL, for SQL backends).
Currently three types of records are sent: 'summary' (derived from what
was currently posted), 'dataset', with entries per dataset processed
in the query, and 'files', with entries per file processed in the query.
In SQL terms, each of this records corresponds to a different table.
Sending of any of the three records can be toggled independently.
Record format have been versionned, so that previous versions of the
'summary' record can be still sent chosing the corresponding version.
The latest version of 'summary' has been augmented with information
about the number of missing files and the ROOT version.
The headers of the relevant methods in TProofMonSenderSQL and
TProofMonSenderML contain the details about the records.
The Web doc is up-to-date.
The patch also add to TSQMonitoring::SendParameters the possibility to
do a bulk insert. This feature is used by TProofMonSenderSQL when
sending dataset or files information.
Revision
38810 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Apr 12 16:22:59 2011 UTC (3 years, 9 months ago) by
ganis
File length: 63449 byte(s)
Diff to
previous 38709
Patch to correctly honour selector abort status settings in PROOF. Currently only
the TSelector::kAbortProcess was handled by stopping processing. In particular
TSelector::kAbortFile was ignored; this recently created some problems in ALICE
with corrupted files, with repeated attempts to read events eventually leading to
bad_alloc exceptions.
This patch also fixes other related issues, in particular with the reporting of the
non-processed {files, events} in the final 'MissingFiles' list. This list should
now account much more precisely of the number of events which could not be processed.
It also fixes a problem with the final update of the progress information affecting
occasionally cases with skipped events.
Revision
34416 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jul 14 15:39:20 2010 UTC (4 years, 6 months ago) by
ganis
File length: 62962 byte(s)
Diff to
previous 34254
Fix problem with packet re-assignment in case of a worker death. Some packets
were processed twice or more times.
A new method MergeElement(TDSetElement *elem) has been added to TDSetElement in
order to simplify merging of contiguous or overlapping packets and avoid artificial
fragmentation of the re-assigned parts.
Revision
33640 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri May 28 13:55:26 2010 UTC (4 years, 7 months ago) by
ganis
File length: 60937 byte(s)
Diff to
previous 33086
Fix a few issues affecting the usage of tree friends in PROOF (see reports on the forum)
- In TDSet:
- Add friend alias as an option to the URL ("friend_alias=<alias>|") instead of an
object string in a TPair (non streamable!)
- In TVirtualPacketizer::CreateNewPacket
- Adapt to the change in TDSet
- In TEventIter:
- Adapt to the change in TDSet
- Make sure that the file with the friend is closed only when is not used anymore.
The patch also fixes an issue with the locality check for files in TDSet.
Revision
33086 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Apr 18 18:15:56 2010 UTC (4 years, 9 months ago) by
ganis
File length: 60347 byte(s)
Diff to
previous 33045
- Some changes towards a parallel version of VerifyDataSet:
- Make the lower level of ScanDataSet a static in TDataSetManager and separate out
the part extracting the file meta-information into a dedicated method ScanFile
- In TDSetElement, rename the list of associated files to associated objects, so that
it can be more generally used (it was not yet used for associated files and it
was anyhow supposed to be a list of TObjString's).
- Move the parsing of the recently introduced 'DataSet.SrvMaps' env in TDataSetManager
(instead of TProof). This is more logical and it makes easier to apply the enable
the server mappings during staging requests via VerifyDataSet.
- Add a missing protection in TProof::AssertDataSet potentially causing crashes in the
case a dataset is not found.
Revision
33045 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 15 14:57:32 2010 UTC (4 years, 9 months ago) by
ganis
File length: 60512 byte(s)
Diff to
previous 32930
Add the possibility to remap the server for the files in a dataset. This allows,
for example, to reuse the dataset information for the same files stored in a different
cluster.
The mapping can be defined globally by the cluster administrator using the ROOT env
variable
DataSet.SrvMaps: srv1|map1 srv2|map2 ...
where srv1 are the {protocol,host,port} of the server to be matched, and map1 are the
{protocol,host,port} to be used for those files mapping srv1, and so on.
The matching server string can contain the wildcard '*' in the host field or it can
be empty, in which case the replacement is done for all the dataset files.
If the '|' character is missing the whole string is assigned to the map part; i.e.
'map1' alone is equivalent to '*|map1' or '|map1' .
For example, the following can be used to relocate the files via 'redir.dom.ain'
DataSet.SrvMaps: root://dserv-*.dom.ain/|root://redir.dom.ain/
The user can change the default mapping(s) on the fly with the parameter PROOF_DataSetSrvMaps:
the above is equivalent to setting
p->SetParameter("PROOF_DataSetSrvMaps", "root://dserv-*.dom.ain/|root://redir.dom.ain/")
before starting a query.
Revision
32930 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Apr 9 16:18:32 2010 UTC (4 years, 9 months ago) by
pcanal
File length: 58469 byte(s)
Diff to
previous 32204
Prefer the use of InheritsFrom(TClass*) over InheritsFrom(const char*) as long as
it does not increase the library dependencies.
Revision
32204 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 3 19:17:40 2010 UTC (4 years, 11 months ago) by
ganis
File length: 58462 byte(s)
Diff to
previous 31718
Add support for processing many datasets in one go in TProof::Process(const char *dataset, ...).
Two options are provided:
- 'grand dataset': the datasets are added up and considered as a single dataset;
syntax: "dataset1|dataset2|..."
- 'keep separated': the datasets are processed one after the other; the user is
notified in the selector of the change of dataset so she/he
has the opportunity to separate the results. A new packetizer,
TPacketizerMulti, has been developed for this case: it basically
contains a list of standard packetizers (one for each dataset) and
loops over them.
Syntax: "dataset1,dataset2,..." or dataset1 dataset2 ..."
In both cases, entry-list can be applied using the syntax "dataset<<entrylist", e.g.
"dataset1<<el1|dataset2<<el2|".
See http://root.cern.ch/drupal/content/working-data-sets#currentelem for more details.
A test for the new functionality has been added to test/stressProof.cxx .
Revision
30720 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Oct 14 11:34:58 2009 UTC (5 years, 3 months ago) by
ganis
File length: 56296 byte(s)
Diff to
previous 30628
Enable mass storage domain settings when working with TChain's in multi-master mode.
The Mass Storage Domain must be specified as option in the URL
chain.AddFile("root:// .....?msd=CERN")
and the string must match the value specified in defining the submaster node.
Revision
26791 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 10 10:47:51 2008 UTC (6 years, 1 month ago) by
ganis
File length: 56519 byte(s)
Diff to
previous 26400
Fix a problem with the lookup status counter due to a missing notification message.
This happened in the case the number of files was not an exact multiple of the
notification frequency.
Revision
26400 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 24 01:49:58 2008 UTC (6 years, 2 months ago) by
rdm
File length: 56403 byte(s)
Diff to
previous 25434
On PROOF-Lite support local (i.e. non-absolute) file names in chains,
by making them relative to the client working directory. This allows
chain definitions for plain ROOT to be used directly on PROOF-Lite.
Revision
25434 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Sep 17 15:23:36 2008 UTC (6 years, 4 months ago) by
ganis
File length: 56100 byte(s)
Diff to
previous 23906
- Fix problem with the determination and transmission of the name of the object to be processed.
The problem appeared when processing files containing >1 trees in changing order.
- Fix problem with TProof::Load (Savannah bugs #41719; forum topic #7138)
- Fix problem with the logs from TProof::Load not being transmitted to the client
Revision
23075 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 9 09:10:25 2008 UTC (6 years, 9 months ago) by
ganis
File length: 55118 byte(s)
Diff to
previous 22635
From Jan-Fiete Grosse-Oetringhaus (ALICE) and me.
Patch introducing the improved dataset manager developed by ALICE.
The client side interface is basically unchanged, the biggest change is the renaming
of TProof::CreateDataSet to TProof::RegisterDataSet.
On the server side the relevant code has ben extracted from TProofServ and hidden
behind an inteface class TProofDataSetManager describing the functionality needed by
TProofServ, and an implementation class TProofDataSetManagerFile handling dataset
meta data saved into ROOT files. TProofDataSetManagerFile is instantiated via the
plugin manager and by default it handles the <sand_box>/datasets area.
This separation is needed to allow loading implementation based on different backends;
for example, ATLAS foresees to use a MySQL-based backend.
A TWiki page with information and examples in on preparation.
Revision
21267 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 7 11:40:47 2007 UTC (7 years, 1 month ago) by
ganis
Original Path:
trunk/proof/src/TDSet.cxx
File length: 51384 byte(s)
Diff to
previous 21236
Fix bugs preventing in some cases proper validation in multi-level master mode
- Make sure that options and anchors are not removed after TDSetElement::GetEntries()
- Use only the file name (and not the full Url) in matching elements in
TDSetElement::Validate(TDSetElement *)
Revision
20682 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 6 15:51:59 2007 UTC (7 years, 2 months ago) by
ganis
Original Path:
trunk/proof/src/TDSet.cxx
File length: 51054 byte(s)
Diff to
previous 20307
Import branches/dev/proof r20654
Summary:
+ Improvements
- Add support for SSH SOCKS4 tunnelling; the local port for the tunnel can be specified
in the master URL, e.g. TProof::Open("master/?tunnel:8000")
- Add the possibility to plot the estimated instantaneous rate
- Add "PROOF_ForceLocal" parameter to the !TPacketizerAdaptive; if set to 1, all the data
are processed locally.
- Add support for remote grep functionality while retrieving logs (needed by the forthcoming
memory monitor)
+ Bug fixes
- Several small fixes to revive the multi-master mode.
- XrdProofdProtocol:
- add missing lock to the client instance in SendMsg to avoid screwing up requests from
workers on the same machine
- lock the mutex of the requester when setting prorities
- add notification during Reset
- fix problem with the detection of the 'allow' directive
- fix problem with the parsing of the return value from XrdProofServProxy::TerminateProofServ()
- always use the effective user to retrieve info fom another server
(XrdProofdManager::GetProofConn is now used)
- fix possible dead-locks from debug notifications done after hard-killing a session
- re-enable the garbage collector thread of the connection manager in XrdProofConn to
fix a problem with closing physical connections;
- fix a problem with !CleanupProofServ in the case of a non-privileged daemon running
in multi-user mode
- introduce a timeout when waiting for the startup of a 'proofserv'.
- XrdProofConn: init mutex in the ctor; lock in SendRecv
- XrdProofSched: add support for using the priorities defined in the group manager to define
the number of workers for sessions
- TProof:
- Broadcast priorities to unique nodes only
- timeout after 5 mins the initial Collect to avoid clients getting stuck at this stage
- add support for generic timeout in Collect (disabled by default)
- fix a problem with SendFile.
- TXProofServ: add a call to !TProof::InterruptCurrentMonitor() in Terminate() to stop
infinite loops in Collect
- TXSocket:
- Implement a flag to interrupt a TXSocket while waiting for messages
- Split the session creation timeout in 4 attempts: the total timeout is the same but
it may circumvent occasional forking problems.
- XROOTD:
- fix a potential (possibly accademic) memory leak in the client
- fix a access permission problem with Kerberos ticket forwarding
- fix bug preventing 'locate' to work properly
- re-enable optimized 'locate'
Revision
20307 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 11 10:58:50 2007 UTC (7 years, 3 months ago) by
ganis
Original Path:
trunk/proof/src/TDSet.cxx
File length: 50779 byte(s)
Diff to
previous 20098
Import branches/dev/proof r20306
Summary (see branch logs for more details):
- Set of changes related to CPU quota control including
- broadcast of centrally determined priorities
- mechanism to renice processes in a quantitative way
- New packetizer for non-tree based analysis and related API in TProof (from L. Tran-Thanh)
- Support for merging output objects saved in files on the workers (from L. Tran-Thanh and me)
- Improve version binary compatibility checks using also the SVN revision number
(when available) to define the running version.
- Extend the version binary compatibility checks also to the cached selector binaries.
- Extend TDSet::Lookup so that in case of missing files, it can remove them from the
dataset (option removeMissing must be set).
- Move the data set lookup to the TProofPlayerRemote::Process.
- Handle properly the case of incomple datasets: if the file is not found in the lookup
don't try to validate it; add it, instead, to a 'missingFiles' list returned in the
output list (fixing bug #28800 in Savannah).
Revision
18874 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu May 24 07:19:39 2007 UTC (7 years, 8 months ago) by
brun
Original Path:
trunk/proof/src/TDSet.cxx
File length: 43543 byte(s)
Diff to
previous 18859
From Ilka:
fixes following errors:
proof/src/TDSet.cxx: In member function `virtual void
TDSet::Streamer(TBuffer&)':
proof/src/TDSet.cxx:1350: warning: suggest parentheses around assignment used as truth value
proof/src/TDSet.cxx:1373: warning: suggest parentheses around assignment used as truth value
Revision
18648 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu May 3 15:27:40 2007 UTC (7 years, 8 months ago) by
rdm
Original Path:
trunk/proof/src/TDSet.cxx
File length: 38369 byte(s)
Diff to
previous 18527
From Gerri:
This is the last patch to synchronize with the CAF:
TDSet.h, .cxx
- Use TCollection instead of TList
- Support thre object types in TDSet::Add(TCollection *): TFileInfo, TUrl
and TObjString .
- Conserve the looked-up status of TChainElements when instantiating
from a TChain.
TChainElement.h, .cxx
- Use BIT(15) to flag the looked-up status
TChain.h, .cxx
- Use TCollection instead of TList
- Support thre object types in TDSet::Add(TCollection *): TFileInfo, TUrl
and TObjString .
- Save / Test the looked-up status of elements to avoid multiple lookup;
an option is available in TChain::Lookup(Bool_t force = kFALSE) to
force a new lookup.
Revision
18527 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Apr 19 09:33:40 2007 UTC (7 years, 9 months ago) by
rdm
Original Path:
trunk/proof/src/TDSet.cxx
File length: 37685 byte(s)
Diff to
previous 18511
From Gerri:
get rid of not really needed usage of queue, map, pair and list.
- TDSet.h, .cxx
Use TList, TPair instead of std::list<std::pair<... >>
- TPacketizer.cxx, TAdaptivePacketizer.cxx, TEventIter.cxx,
TPacketizerDev.cxx, TPacketizerProgressive.cxx
Adapt to the previous change in TDSet
Revision
18449 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Mar 30 16:44:34 2007 UTC (7 years, 9 months ago) by
rdm
Original Path:
trunk/proof/src/TDSet.cxx
File length: 37946 byte(s)
Diff to
previous 18194
From Gerri:
Anna reported to me the following problem:
TChain::Lookup (and TDSet::Lookup) makes now use of TFileStager::Locate
to locate the files. But for local URLs TFileStager is not defined, so
the lookup fails.
This patch adds default functionality (based on TSystem::AccessPathName)
to the base TFileStager, so that it works trasparently for local files.
Revision
18194 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Mar 8 12:09:09 2007 UTC (7 years, 10 months ago) by
rdm
Original Path:
trunk/proof/src/TDSet.cxx
File length: 37852 byte(s)
Diff to
previous 17888
From Gerri:
This patch introduces TFileStager::Locate and uses it to improve lookup
performances in TDSetElement::Lookup and TChain::Lookup.
There is a further optimization to be done in XrdClientAdmin under discussion
with Fabrizio. But that should have no (or very little) impact on this
implementation.
Revision
16955 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Mon Nov 27 14:14:24 2006 UTC (8 years, 1 month ago) by
rdm
Original Path:
trunk/proof/src/TDSet.cxx
File length: 37685 byte(s)
From Gerri:
This is the patch to move TDSet and TChainProof in 'proof' and remove
any dependence on PROOF in 'tree'. I have renamed TChainProof as TProofChain,
for consistency. TProofChain is now deriving from TChain and overrides only
the required methods. It is loaded via PluginManager by TChain in SetProof.
TQueryResult stores now all the input-related objects in fInputList. There
is a new method
TObject *TQueryResult::GetInputObject(const char *classname)
to retrieve the first instance of a class from the input list, so for the
TDSet:
TQueryResult *qr;
TDSet *dset = (TDSet *) qr->GetInputObject("TDSet")
The changes in TSessionViewer are needed to adapt to this new way of
accessing TDSets.
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
