Log of /trunk/proof/proof/src/TProofOutputFile.cxx
Parent Directory
Revision
45740 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 29 16:17:09 2012 UTC (2 years, 4 months ago) by
ganis
File length: 20110 byte(s)
Diff to
previous 45367
- Add support for data dirs on external servers, fully honouring URL options; this allows to
have usable paths registered in the final TFileCollection objects. One of the addressed use-case
is when local directories need to be accessed via the local xrootd server to preserve the namespace
enforced by xrootd; in such a case something like 'root://locahost//datadir?remote=1' needs to be
specified; the option 'remote=1' ensures that the final registered URL is still in 'root://' form
with 'localhost' replaced by the FQDN of the machine.
- Move the 'tests' argument to the stressProof function in the second position, to make it
more user friendly to run a specific test in the ROOT shell; CLI behevior is unchanged.
- In ProofSimple, add function to plot the histograms also in the case a dataset is created with the
worker files; the file are opened and the histograms merged on the fly.
- Fix potential segv at xproofd startup due to a double fclose in XrdProofdManager::ValidateLocalDataSetSrc
- Remove some redundant printing in TProof when handling output files.
- Make sure that when running TSelVerifyDataSet the save-to-file functionality is switched-off .
Revision
45367 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Aug 3 09:22:00 2012 UTC (2 years, 5 months ago) by
ganis
File length: 20063 byte(s)
Diff to
previous 45348
In TProofOutputFile, add a status bit to indicate that the file is the result of the automatic
save-to-file output handling. This allows a better interplay between client-side and server-side
settings.
In TProof::HandleOutputOptions, fix issue with local object saving.
Revision
45348 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Aug 2 12:23:54 2012 UTC (2 years, 5 months ago) by
ganis
File length: 20021 byte(s)
Diff to
previous 45283
Consolidation of the automatic output saving functionality
- Fix an issue with saving output from memory on the master
- Remove some duplications and fix several small bugs
Revision
45283 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jul 30 10:42:38 2012 UTC (2 years, 5 months ago) by
ganis
File length: 20520 byte(s)
Copied from:
trunk/proof/proofplayer/src/TProofOutputFile.cxx revision 45216
Diff to
previous 44065
Patch automatizing the usage of the file-based technology to handle outputs.
After this patch PROOF users can define as an option to Process an output file for their results
and choose if the merging goes via file or in memory. User can also decide to save theit results
in a set of files identified by a name (dataset).
Administrators can force merging-via-file or can make it conditional to the resident memory exceeding
a given threshold.
The new options are documented at
http://root.cern.ch/drupal/content/handling-outputs
Since TProofOutputFile is now used by TProof, the class has been moved from proof/proofplayer
(libProofPlayer) to proof/proof (libProof).
The selector tutorials/proof/ProofSimple.h,.C has been modified to allo testing the new options,
and a new test has been added to stressProof.
Revision
33824 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 10 14:43:40 2010 UTC (4 years, 7 months ago) by
ganis
Original Path:
trunk/proof/proofplayer/src/TProofOutputFile.cxx
File length: 17202 byte(s)
Diff to
previous 30648
- Add new directive 'xpd.datadir' to better control the user data directories and their
permission settings. The syntax is 'xpd.datadir <datadir> <opts>' where <datadir> is
the root path for the datadirs (the final datadir will be <datadir>/<group>/<user>) and
<opts> can contain a combination of the following letters: 'g', 'a', 'M', 'W' ["gW"];
the lower case control the write permissions: 'g' group, 'a' everybody; the upper case
control where the data dirs shoudl be created: 'W' on the workers, 'M' on the master,
"WM" everywhere.
- Better handle the removal of empty data directories in TProofServ and derivatives
- In TProofOutputFile:
- extend support for URL options
- correctly take the URL options into account
This should solve a problem observed in the new ALICE CAF.
Revision
30648 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Oct 9 12:51:35 2009 UTC (5 years, 3 months ago) by
ganis
Original Path:
trunk/proof/proofplayer/src/TProofOutputFile.cxx
File length: 17017 byte(s)
Diff to
previous 30174
Fix a few issues reported by ALICE:
- Do not drop anchors and options from the file path, if any
- In dataset mode, also add an URL with the redirector in TFileInfos when the
redirector URL is available (rootrc variable ProofServ.PoolUrl). This is needed
by the logic of the dataset stager daemon.
The elements of the datasets created this way will therefore have three URLs:
1. The URL pointing to the data server on the machine
2. The same URL but with the redirector protocol, user, host and port
3. The special 'file:' URL with the local path on the machine used for cleaning
Revision
30174 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Sep 15 14:24:56 2009 UTC (5 years, 4 months ago) by
ganis
Original Path:
trunk/proof/proofplayer/src/TProofOutputFile.cxx
File length: 15652 byte(s)
Diff to
previous 29875
Patch to support automatic creation of a dataset out of files created on the
worker nodes by worker processes. The implementation is an extension of the
functionality of the class TProofOutputFile used for merging via file.
TProofOutputFile:
- Add options to steer the creation of a dataset; the extemded meaning of the
constructor arguments and anew constructor are described in
http://root.cern.ch/drupal/content/handling-large-outputs-root-files
The dataset (i.e. TFileCollection) assembing happens on the master via the
Merge method. Information about the node location of the file is happended
to the relevant TFileInfo as a special "file:" url with option "node=<host-fqdn>:<port>"
This information is needed to facilitate the cleaning of the files once they
are not needed anymore.
The dataset can be just created and returned in the output list, or registered
and/or verified.
TProofPlayerRemote::MergeOutputFiles:
- Correctly transmit the dataset object into the output list
TProofServ:
- Create the new subdirectory 'data' either in the sandbox or followinf ProofServ.DataDir .
- Register and verify the created dataset if required
TProof:
- Implement two new methods:
- ShowData(), to show the content of the assigned 'data' directory
- ClearData(UInt_t what, const char *dsname), to clear all or a subset of the files
in the assigned 'data' directory
Additional changes:
- TProofPlayer:
- Correctly handle the feedback canvas
- Set the drawing attributes
- TProofServ:
- Replace all 'Form("...")' with TString::Format("...") .
- Use the new tag kCollect in debug statements where relevant
- TProof:
- Use the new tag kCollect in debug statements where relevant
Revision
29491 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jul 16 15:12:51 2009 UTC (5 years, 6 months ago) by
ganis
Original Path:
trunk/proof/proofplayer/src/TProofOutputFile.cxx
File length: 9857 byte(s)
Diff to
previous 28393
Fix several issues in TProofOutputFile:
- Remove the 'mode' parameter as the 'SEQUENTIAL' mode was not really an option
for the cases when this merging technique is needed
- Fix a problem with the determination of the fDir member; this problem affects
mostly PROOF-Lite and it appeared after the change #25949 .
- Make sure the the local server information is correctly included in AdoptFile()
- Fix a serious issue with the 'CENTRAL' mode whose net effect was to delete the
outputfile just after having open it
In TProofPlayer and XrdProofdProofServMgr
- Cope with the changes in TProofOutputFile
Revision
25949 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Oct 24 18:03:49 2008 UTC (6 years, 3 months ago) by
ganis
Original Path:
trunk/proof/proofplayer/src/TProofOutputFile.cxx
File length: 15633 byte(s)
Diff to
previous 25590
Add possibility to adopt a file already open. Instead of opening the unique file via
TFile *TProofOutputFile::OpenFile(const char *mode) one can open do
TFile *f = TFile::Open("<file>", "<mode>")
TProofOutputFile *outputfile = new TProofOutputFile("analysis");
outputfile->AdoptFile(f);
fOutput->Add(outputfile);
This is useful in cases one wants to merge files open in other places.
Revision
20682 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 6 15:51:59 2007 UTC (7 years, 2 months ago) by
ganis
Original Path:
trunk/proofplayer/src/TProofFile.cxx
File length: 13279 byte(s)
Diff to
previous 20307
Import branches/dev/proof r20654
Summary:
+ Improvements
- Add support for SSH SOCKS4 tunnelling; the local port for the tunnel can be specified
in the master URL, e.g. TProof::Open("master/?tunnel:8000")
- Add the possibility to plot the estimated instantaneous rate
- Add "PROOF_ForceLocal" parameter to the !TPacketizerAdaptive; if set to 1, all the data
are processed locally.
- Add support for remote grep functionality while retrieving logs (needed by the forthcoming
memory monitor)
+ Bug fixes
- Several small fixes to revive the multi-master mode.
- XrdProofdProtocol:
- add missing lock to the client instance in SendMsg to avoid screwing up requests from
workers on the same machine
- lock the mutex of the requester when setting prorities
- add notification during Reset
- fix problem with the detection of the 'allow' directive
- fix problem with the parsing of the return value from XrdProofServProxy::TerminateProofServ()
- always use the effective user to retrieve info fom another server
(XrdProofdManager::GetProofConn is now used)
- fix possible dead-locks from debug notifications done after hard-killing a session
- re-enable the garbage collector thread of the connection manager in XrdProofConn to
fix a problem with closing physical connections;
- fix a problem with !CleanupProofServ in the case of a non-privileged daemon running
in multi-user mode
- introduce a timeout when waiting for the startup of a 'proofserv'.
- XrdProofConn: init mutex in the ctor; lock in SendRecv
- XrdProofSched: add support for using the priorities defined in the group manager to define
the number of workers for sessions
- TProof:
- Broadcast priorities to unique nodes only
- timeout after 5 mins the initial Collect to avoid clients getting stuck at this stage
- add support for generic timeout in Collect (disabled by default)
- fix a problem with SendFile.
- TXProofServ: add a call to !TProof::InterruptCurrentMonitor() in Terminate() to stop
infinite loops in Collect
- TXSocket:
- Implement a flag to interrupt a TXSocket while waiting for messages
- Split the session creation timeout in 4 attempts: the total timeout is the same but
it may circumvent occasional forking problems.
- XROOTD:
- fix a potential (possibly accademic) memory leak in the client
- fix a access permission problem with Kerberos ticket forwarding
- fix bug preventing 'locate' to work properly
- re-enable optimized 'locate'
Revision
20307 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 11 10:58:50 2007 UTC (7 years, 3 months ago) by
ganis
Original Path:
trunk/proofplayer/src/TProofFile.cxx
File length: 13284 byte(s)
Copied from:
branches/dev/proof/proofplayer/src/TProofFile.cxx revision 20306
Diff to
previous 20296
Import branches/dev/proof r20306
Summary (see branch logs for more details):
- Set of changes related to CPU quota control including
- broadcast of centrally determined priorities
- mechanism to renice processes in a quantitative way
- New packetizer for non-tree based analysis and related API in TProof (from L. Tran-Thanh)
- Support for merging output objects saved in files on the workers (from L. Tran-Thanh and me)
- Improve version binary compatibility checks using also the SVN revision number
(when available) to define the running version.
- Extend the version binary compatibility checks also to the cached selector binaries.
- Extend TDSet::Lookup so that in case of missing files, it can remove them from the
dataset (option removeMissing must be set).
- Move the data set lookup to the TProofPlayerRemote::Process.
- Handle properly the case of incomple datasets: if the file is not found in the lookup
don't try to validate it; add it, instead, to a 'missingFiles' list returned in the
output list (fixing bug #28800 in Savannah).
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
