Log of /trunk/io/io/inc/TFileMerger.h
Parent Directory
Revision
48338 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jan 18 15:54:08 2013 UTC (2 years ago) by
pcanal
File length: 6872 byte(s)
Diff to
previous 41444
From Andrei:
Added possibility to merge only a list of objects/folders from the input files, specified by name via: TFileMerger::AddObjectNames(). One can merge only specified objects via TFileMerger::PartialMerge(default_flags | kOnlyListed), or skip them from merging via TFileMerger::PartialMerge(default_flags | kSkipListed). Example macro provided in tutorials/io/mergeSelective.C
This has also been added to the v5-34-00 patch branch.
Revision
41444 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Oct 18 09:36:59 2011 UTC (3 years, 3 months ago) by
pcanal
File length: 6366 byte(s)
Diff to
previous 41408
Update hadd and TFileMerger so that they prefix all their information message
with their names (when running hadd, the TFileMerger message are prefixed by hadd):
$ hadd -v 0 -f output.root input1.root input2.root
$ hadd -v 1 -f output.root input1.root input2.root
hadd merged 2 input files in output.root.
$ hadd -v 2 -f output.root input1.root input2.root
hadd target file: output.root
hadd Source file 1: input1.root
hadd Source file 2: input2.root
hadd Target path: output.root:/
Revision
41408 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Oct 16 21:28:45 2011 UTC (3 years, 3 months ago) by
pcanal
File length: 6142 byte(s)
Diff to
previous 40569
Introduce the class TParallelMergingFile part of the net package. This class connect ot a parallel merge server
and upload its content every time Write is called on the file object. After the upload the object of classes
with a ResetAfterMerge function are reset.
A TParallelMergingFile is created whether a ?pmerge option is passed to TFile::Open as part of the file name.
For example:
TFile::Open("mergedClient.root?pmerge","RECREATE"); // For now contact localhost:1095
TFile::Open("mergedClient.root?pmerge=localhost:1095","RECREATE");
TFile::Open("rootd://root.cern.ch/files/output.root?pmerger=pcanal:password@locahost:1095","NEW")
tutorials/net/treeClient.C and fastMergeServer.C: update to follow the change in interfaces
Introduce the tutorials parallelMergerClient.C and the temporary tutorials parallelMergerServer.C
to demonstrate the parallel merging (with parallelMergerServer.C being the prototype of the upcoming
parallel merger server executable).
TKey:
Introduce a new constructor to copy a TKey from one file to another without unzipping nor unstreaming its content.
TFileMerger:
New method AddFile(TFile*) that appends the TFile to the list of file to be merged but does not delete it after
the merge operation.
Replace the method IncrementalMerge(Bool_t) with the method PartialMerge(Int_t) the argument defines the type
of merge as define by the bit values in EPartialMergeType:
// kRegular : normal merge, overwritting the output file
// kIncremental : merge the input file with the content of the output file (if already exising) (default)
// kAll : merge all type of objects (default)
// kResetable : merge only the objects with a MergeAfterReset member function.
// kNonResetable : merge only the objects without a MergeAfterReset member function.
Remove RecursiveMerge from the public interface.
rootcint.cxx:
Distinguish objects of classes that can explicitly be reset after a Merge operation and those that should not.
Only the object of classes with a method ResetAfterMerge will be Reset by TMemFile::ResetAfterMerge (previous
both object of classes with a ResetAfterMerge and those with a simple Reset method where reset).
Revision
40569 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Aug 11 19:37:20 2011 UTC (3 years, 5 months ago) by
pcanal
File length: 5222 byte(s)
Diff to
previous 39824
Prevent TFileMerger (and hadd) from trying to open too many files.
Add a new member function TFileMerger::SetMaxOpenedFiles and
new command line option to hadd ( -n requested_max ) to allow
the user to reduce the number of files opened even further.
Revision
39611 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 8 19:35:17 2011 UTC (3 years, 7 months ago) by
pcanal
File length: 4809 byte(s)
Diff to
previous 39370
Introduce TMemFile and update TFileMerger to support incremental merges.
Add new tutorials ( net/treeClient.C + net/fastMergeServer.C )
demonstrating how a TMemFile can be used to do parallel merge
from many client. ( TMemFile still needs to be better integrated
with TMessage and TSocket).
The new TMemFile class support the TFile interface but only store
the information in memory. This version is limited to 32MB.
TMessage mess;
...
mess->ReadFastArray(scratch,length);
transient = new TMemFile("hsimple.memroot",scratch,length);
will copy the content of 'scratch' into the in-memory buffer
created by/for the TMemFile.
TMemFile *file = new TMemFile("hsimple.memroot","RECREATE");
Will create an empty in-memory of (currently fixed) size 32MB.
file->ResetAfterMerge(0);
Will reset the objects in the TDirectory list of objects
so that they are ready for more data accumulations (i.e.
returns the data to 0 but keep the customizations).
Introduce the new function TFileMerger::IncrementalMerge will
will Merge the list of file _with_ the content of the output
file (if any). This allows make several successive Merge
into the same TFile object.
Introduce non-static version of TFile::Cp allows the copy of
an existing TFile object.
Introduce new explicit interface for providing reseting
capability after a merge. If a class has a method with
the name and signature:
void ResetAfterMerge(TFileMergeInfo*);
it will be used by a TMemFile to reset its objects after
a merge operation has been done.
If this method does not exist, the TClass will use
a method with the name and signature:
void Reset(Optiont_t *);
TClass now provides a quick access to these merging
function via TClass::GetResetAfterMerge. The wrapper function
is automatically created by rootcint and can be installed
via TClass::SetResetAfterMerge. The wrapper function should have
the signature/type ROOT::ResetAfterMergeFunc_t:
void (*)(void *thisobj, TFileMergeInfo*);
ResetAfterMerge functions were added to the following classes:
TDirectoryFile, TMemFile, TTree, TChain, TBranch, TBranhcElement,
TBranchClones, TBranchObject and TBranchRef.
Revision
39365 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue May 24 19:53:19 2011 UTC (3 years, 8 months ago) by
pcanal
Original Path:
trunk/proof/proofplayer/inc/TFileMerger.h
File length: 4559 byte(s)
Diff to
previous 39329
Remove hard dependencies of TFileMerger on TH1 and TTree.
(Soft dependencies still exist to be able to disable the
merging of TTrees and to be able to disable the AutoAdd
behavior of TH1).
Introduce new explicit interface for providing merging
capability. If a class has a method with the name and
signature:
Long64_t Merge(TCollection *input, TFileMergeInfo*);
it will be used by a TFileMerger to merge one or more
other objects into the current object. Merge should
return a negative value if the merging failed.
If this method does not exist, the TFileMerger will use
a method with the name and signature:
Long64_t Merge(TColletion *input);
TClass now provides a quick access to these merging
function via TClass::GetMerge. The wrapper function
is automatically created by rootcint and can be installed
via TClass::SetMerge. The wrapper function should have
the signature/type ROOT::MergeFunc_t:
Long64_t (*)(void *thisobj, TCollection *input, TFileMergeInfo*);
Added the new Merge function to TTree and THStack.
Also add the new Merge function to TQCommand as the
existing TQCommand::Merge does _not_ have the right
semantic (in part because TQCommand is a collection).
Fix the return value of TEfficiency::Merge
In TFileMerger, add a PrintLevel to allow hadd to request
more output than regular TFileMerger.
The object TFileMergeInfo can be used inside the Merge
function to pass information between runs of the Merge
(see below). In particular it contains:
TDirectory *fOutputDirectory; // Target directory where the merged object will be written.
Bool_t fIsFirst; // True if this is the first call to Merge for this series of object.
TString fOptions; // Additional text based option being passed down to customize the merge.
TObject *fUserData; // Place holder to pass extra information. This object will be deleted at the end of each series of objects.
The default in TFileMerger is to call Merge for every object
in the series (i.e the collection has exactly one element) in
order to save memory (by not having all the object in memory
at the same time).
However for histograms, the default is to first load all the
objects and then merge them in one go ; this is customizable
when creating the TFileMerger object.
Revision
30052 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Sep 7 12:39:28 2009 UTC (5 years, 4 months ago) by
ganis
Original Path:
trunk/proof/proofplayer/inc/TFileMerger.h
File length: 3849 byte(s)
Diff to
previous 29999
A few improvements on the way to make TFileMerger and hadd totally equivalent
(the goal being to use TFileMerger inside 'hadd'):
- import from hadd an optimization of key hashing
- import from hadd a better way to invoke Merge for generic objects
- add option to merge histograms in one go, instead of one-by-one as for generic
objects (this option is not yet supported by hadd).
The improvement for merging 10000 H1F with 100 bins is about a factor of 2.
Revision
28860 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 9 13:57:24 2009 UTC (5 years, 7 months ago) by
ganis
Original Path:
trunk/proof/proofplayer/inc/TFileMerger.h
File length: 3764 byte(s)
Diff to
previous 27117
- Create directly the output file at the final destination do not make
a local copy in the temp directory first (if needed, one can always
set the temporary destination to temp followed by a TFile::Cp to the
final destination); this allows to avoid problems with small temp
partitions (see Forum).
- Remove usage of Form(...) and other cosmetic changes.
Revision
11900 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Fri May 27 13:47:37 2005 UTC (9 years, 7 months ago) by
rdm
Original Path:
trunk/proofplayer/inc/TFileMerger.h
File length: 3180 byte(s)
From Andreas Peters + me:
moved the file copy and merge API from TGrid to a stand-alone class
TFileMerger. This class allows for easy copying of two files using the
TFile plugins (i.e. it can copy from castor to dcap, or from xrootd to
chirp, etc.). It file merge functionality is taken from hadd. It is part
of libProof since that library is already linked with libHist and libTree.
To use it do, e.g.:
TFileMerger m
m->Cp("srcUrl", "destUrl")
or
m->AddFile("url1")
m->AddFile("url2")
m->Merge()
the AddFile() and Merge() use the Cp() to copy the file locally before
making the merge, and if the output file is remote the merged file
will be copied back to the remote destination.
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
