Log of /trunk/tree/tree/src/TBasket.cxx
Parent Directory
Revision
44681 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 13 19:31:49 2012 UTC (2 years, 7 months ago) by
pcanal
File length: 34863 byte(s)
Diff to
previous 43859
From Peter van Gemmeren and Jack Weinstein:
Index: tree/tree/src/TBasket.cxx
* Disable the default TTreeCache before calls to TFile::ReadBuffer() to avoid needless calls to default TFileCacheRead ReadBuffer.
Index: tree/tree/inc/TTreeCache.h
Index: tree/tree/src/TTreeCache.cxx
* Define and implement new Enable(), Disable() and IsEnabled() functions
Revision
43148 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Feb 27 21:09:01 2012 UTC (2 years, 10 months ago) by
pcanal
File length: 34164 byte(s)
Diff to
previous 42754
From Peter:
Index: io/io/inc/TFileCacheRead.h
Index: io/io/src/TFileCacheRead.cxx
* Enhance Print() function to report number of bytes read and read calls for current cache rather than totals on associated file.
* Distingurish counter for bytes read and read calls for learning phase.
Index: tree/tree/src/TBasket.cxx
* When calling cache ReadBuffer(), update number of read calls and bytes read.
Revision
42754 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jan 20 20:13:07 2012 UTC (3 years ago) by
pcanal
File length: 33960 byte(s)
Diff to
previous 41531
From Peter van Gemmeren:
Allow multiple read caches per file by adding a fCacheReadMap member and owner arguments to GetCacheRead() and SetCacheRead()
to TFile.
Index: io/io/inc/TFileCacheRead.h
Index: io/io/src/TFileCacheRead.cxx
* Adding new argument for constructor to specify TTree owner
* whitespace indents
Index: tree/tree/inc/TBasket.h
Index: tree/tree/src/TBasket.cxx
* Add optional tree owner argument to LoadBasketBuffers() to retrieve cache.
* In LoadBasketBuffers() and ReadBuffers() check the cache first than call TFile::ReadBuffer only if needed.
Index: tree/tree/inc/TChain.h
Index: tree/tree/src/TChain.cxx
* Override SetCacheSize() function to update TTree owner on the current cache.
* whitespace indents
Index: tree/tree/src/TTree.cxx
* Get TTreeCache for current TTree, by specifying owner in GetCacheRead() calls
* Always delete owned cache
Index: tree/tree/src/TTreeCache.cxx
* Adding new argument for constructor for TFileCacheRead to specify TTree owner
Index: tree/treeplayer/src/TTreePlayer.cxx
* Get TTreeCache for current TTree, by specifying owner in GetCacheRead() calls
Index: tree/tree/src/TTreeCloner.cxx
* Call TBasket::LoadBasketBuffers() with current TTree as owner of cache
Index: proof/proofplayer/src/TEventIter.cxx
* Get TTreeCache for current TTree, by specifying owner in GetCacheRead() and SetCacheRead() calls
Revision
39696 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 13 21:35:34 2011 UTC (3 years, 7 months ago) by
pcanal
File length: 32034 byte(s)
Diff to
previous 39644
From David Dagenhart:
Implement an option to allow ROOT to use the LZMA compression
algorithm to compress data instead of the ZLIB compression algorithm.
LZMA compression typically results in smaller files, but takes more
CPU time to compress data. To use the new feature, the external XZ
package must be installed when ROOT is configured and built:
Download 5.0.3 from here http://tukaani.org/xz/
and make sure to configure with fPIC:
./configure CFLAGS='-fPIC'
Then the client C++ code must call routines to explicitly request LZMA
compression.
ZLIB compression is still the default.
Setting the Compression Level and Algorithm:
There are three equivalent ways to set the compression level and
algorithm. For example, to set the compression to the LZMA algorithm
and compression level 5.
1. TFile f(filename, option, title);
f.SetCompressionSettings(ROOT::CompressionSettings(ROOT::kLZMA, 5));
OR
2. TFile f(filename, option, title, ROOT::CompressionSettings(ROOT::kLZMA, 5));
OR
3. TFile f(filename, option, title);
f.SetCompressionAlgorithm(ROOT::kLZMA);
f.SetCompressionLevel(5);
These methods work for TFile, TBranch, TMessage, TSocket, and TBufferXML.
The compression algorithm and level settings only affect compression of
data after they have been set. TFile passes its settings to its branches
only at the time the branches are created. This can be overidden by
explicitly setting the level and algorithm for the branch. These classes
also have the following methods to access the algorithm and level for
compression.
Int_t GetCompressionAlgorithm() const;
Int_t GetCompressionLevel() const;
Int_t GetCompressionSettings() const;
If the compression level is set to 0, then no compression will be
done. All of the currently supported algorithms allow the level to be
set to any value from 1 to 9. The higher the level, the larger the
compression factors will be (smaller compressed data size). The
tradeoff is that for higher levels more CPU time is used for
compression and possibly more memory. The ZLIB algorithm takes less
CPU time during compression than the LZMA algorithm, but the LZMA
algorithm usually delivers higher compression factors.
The header file core/zip/inc/Compression.h declares the function
"CompressionSettings" and the enumeration for the algorithms.
Currently the following selections can be made for the algorithm:
kZLIB (1), kLZMA (2), kOldCompressionAlgo (3), and kUseGlobalSetting
(0). The last option refers to an older interface used to control the
algorithm that is maintained for backward compatibility. The following
function is defined in core/zip/inc/Bits.h and it set the global
variable.
R__SetZipMode(int algorithm)
If the algorithm is set to kUseGlobalSetting (0), the global variable
controls the algorithm for compression operations. This is the
default and the default value for the global variable is kZLIB.
Note that the LZMA algorithm will only be available if the LZMA
libraries from the XZ package were available when the ROOT executable
being used was configured and built. If you are building ROOT and want
LZMA then you must do something similar to the following if XZ is not
already installed on your system.
Download XZ version 5.0.3 from http://tukaani.org/xz/
unwind the tarball
cd xz-5.0.3
./configure CFLAGS='-fPIC'
make
ROOT was tested with version 5.0.3 of XZ. It might work with earlier
versions and will probably work with later versions of XZ.
Then either use "make install" to put the library and headers into
/usr/local/* so the scripts building ROOT can find them or use
arguments to the ROOT configure script to point to the proper
directories containing the XZ library and headers. These are the
library and the included header (lzma.h includes other headers in the
lzma subdirectory).
src/liblzma/.libs/liblzma.a
src/liblzma/api/lzma.h
WARNING: Data compressed with the LZMA algorithm cannot be read by
ROOT executables that were not built with LZMA support.
Revision
39642 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 9 17:18:42 2011 UTC (3 years, 7 months ago) by
pcanal
File length: 31961 byte(s)
Diff to
previous 39551
From Brian and Philippe:
Reduce the memory used by a TTree in half.
Refactor the code reading and writing the TBasket data.
A single transient buffer holding the compressed data is
now managed by TTree (and could be made thread local)
rather than having one per TBranch.
Revision
39551 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jun 3 18:37:05 2011 UTC (3 years, 7 months ago) by
pcanal
File length: 29664 byte(s)
Diff to
previous 38808
Add parameter 'copy' to TBuffer::Expand indicating whether the caller
want the existing data to be copy forward (the old behavior and the
new default is to copy the data forward).
Use this feature is TBasket::Reset to avoid a useless memcpy.
Revision
38801 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Apr 11 20:10:24 2011 UTC (3 years, 9 months ago) by
pcanal
File length: 29252 byte(s)
Diff to
previous 38462
Reduce by 40% the time taken GetEntry for a branch created using a leaflist (exclusive of the decompression time).
Cache the current basket and its limit.
Reorder the set of test to reduce the number of conditional jump in the most common cases
and use:
#define unlikely(expr) __builtin_expect(!!(expr), 0)
#define likely(expr) __builtin_expect(!!(expr), 1)
to tell the compiler/processor which branch is more likely.
Record whether the buffer's map of classname and object is used to avoid resetting unnecessarily at read time.
Revision
38080 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 16 00:38:11 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 29161 byte(s)
Diff to
previous 38071
Prevent the unlimited growth of the TBasket's buffer even if the basket is reused.
When the Basket is 'Reset' (this happens when it is written and will be reused),
if the TBuffer size is greater than
- twice the data in the current basket
and - twice the average data in each basket (of this branch)
and - twice the requeste basket size (TBranch::GetBasketSize).
the size of the buffer is reduced to the max of
'the data in the current basket' and 'the average' and the requested
buffer size and aligned to next highest multiple of 512.
Revision
38071 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Feb 15 06:58:59 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 28279 byte(s)
Diff to
previous 38055
Make that even when the TBasket is 'reused' the allocated length of the fEntryOffset array is decreased (to handle the case of the 'lone' large entry)
Revision
38055 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 11 22:06:25 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 28284 byte(s)
Diff to
previous 37985
Introduce TVirtualPerfStats::FileUnzipEvent to be able to keep track of the cost of unzipping and use this in TTreePerfStas and TBakset ... This give a good picture of where the time in unzip or in unstreaming
Revision
37985 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Feb 4 21:37:55 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 27970 byte(s)
Diff to
previous 37947
From Constantin Loizides:
Explicitly check the validity of the zipped buffer before calling
R__unzip, this allow for better error recovery.
Introduce R__unzip_header which does the check and calculate the length values
Revision
37947 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 2 16:23:35 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 27573 byte(s)
Diff to
previous 37578
In TBasket::AdjustSize (called by TTree::OptimizeBaskets), reallocate the correct
buffer (hence avoiding a large memory leak).
In TBasket::ReadBasketBuffers avoid continuing to read (the offset table) after
an early failure.
Revision
37405 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Dec 8 16:25:42 2010 UTC (4 years, 1 month ago) by
pcanal
File length: 27345 byte(s)
Diff to
previous 35236
Fix memory leak affecting (significantly) the fast cloning (this is needed since the update to start 'reusing' the TBasket/TBuffer)
Revision
35231 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Sep 10 17:59:10 2010 UTC (4 years, 4 months ago) by
pcanal
File length: 27488 byte(s)
Diff to
previous 34681
Dramatically reduce the amount of memory allocation induces by the management of the TBasket and TBuffer
for each branch. Instead of creating one TBasket object and one TBuffer object and its associated memory buffer
for each onfile basket of each branch, we now create only one TBasket and one TBuffer object for the lifetime of
each branch. The memory buffer associated with the TBuffer object is also created once and rarely reallocated;
it is reallocated only when the buffer size is reset (for example by the AutoFlush mechanism) and when the user
object do not fit in the currently allocated memory (but we do not shrink it after that. The same minization
is applied to the scratch area used to read the compressed version of a basket from the file.
In TBasket introduce new data member fCompressedBuffer and fCompressedSize to keep track of the scratch memory
area used to read compressed data from the file (Even though we already have fBufferRef and fBuffer from TKey,
they are too tied together to properly used in the case of long live TBasket).
Add a TBasket::Reset member function to return the basket to its original 'unread' or 'unfilled' state.
Add TBranch::GetFreshBasket to factor out the code deciding whether to create a new TBasket object or
to reuse an existing one.
Revision
34444 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jul 16 02:17:39 2010 UTC (4 years, 6 months ago) by
pcanal
File length: 24286 byte(s)
Diff to
previous 32705
Avoid doing seeks on the physical file reprensation when it is not absolutely necessary
(i.e. it is not need if we are going to pick the data from our cache).
Introduce a SetOffset to change fOffset (i.e. the logical position in the file) without doing a lseek.
Introduce ReadBuffer(char * buf, Long64_t pos, Int_t len) to avoid the lseek when the data is in the cache.
See <http://savannah.cern.ch/bugs/?69845>
Revision
32705 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Mar 23 01:35:18 2010 UTC (4 years, 10 months ago) by
pcanal
File length: 24305 byte(s)
Diff to
previous 31605
Avoid preventable crashes if the meta data about the basket/key has been corrupted on the disk.
The TKey/TBasket is now set to be a zombie if we find that some of the size value are obviously incorrect (less than zero).
Revision
31457 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 30 13:36:37 2009 UTC (5 years, 1 month ago) by
brun
File length: 23887 byte(s)
Diff to
previous 29599
From Fabrizio:
In TFileCacheRead add a new virtual function GetUnzipBuffer in order
to avoid InheritsFrom in TBasket::ReadBasketBuffers.
In TTreeCacheUnzip optimizations related to CPU cycles and memory.
Related fixes to TFileCacheRead. Also comment "if gDebug" statements.
In TXNetFile fixes related to the reporting of TTreePerfStats.
Revision
29599 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jul 27 15:42:23 2009 UTC (5 years, 5 months ago) by
pcanal
File length: 24092 byte(s)
Diff to
previous 29586
The default length fEntryOffset can now be set via TTree::SetDefaultEntryOffsetLen which can be optionially applied to the existing branches.
Revision
29586 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sun Jul 26 21:37:25 2009 UTC (5 years, 5 months ago) by
pcanal
File length: 24176 byte(s)
Diff to
previous 25980
Allow for fEntryOffsetLen to decrease if the number of entries in the basket is less than 1/4 of fEntryOffsetLen, allow decrease of fEntryOffsetLen even after the 10th baskets
Revision
25980 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Oct 27 18:17:39 2008 UTC (6 years, 2 months ago) by
pcanal
File length: 24151 byte(s)
Diff to
previous 25058
From Christian Klein-Boesing and me:
Synchronize the algorithm to find the file where a branch is stored and the algorithm used
to find the file for closure (in the TBranch destructor).
Also, add protection in case branches are stored in a separate file and the file can not be open
Revision
25058 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Aug 5 19:20:32 2008 UTC (6 years, 5 months ago) by
pcanal
File length: 24083 byte(s)
Diff to
previous 24454
Insure that the in-memory tree (not attached to a file)
are saved as expected by revision 24454 (i.e. each
basket saved separately) and prevent the printing
of the misleading error message:
Error in <TBasket::Create>: Cannot create key without file
Revision
24454 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Jun 21 03:13:04 2008 UTC (6 years, 7 months ago) by
pcanal
File length: 24093 byte(s)
Diff to
previous 23685
Disk and Memory Space Gain
In ROOT older than v5.20/00, the branches' last basket, also known as the write basket, was always saved in the same "key" as the TTree object and was always present in memory when reading or writing. When reading this write basket was always present in memory even if the branch was never accessed.
Starting in v5.20/00, TTree::Write closes out, compresses (when requested) and writes to disk in their own file record the write baskets of all the branches. (This is implemented via the new function TTree::FlushBaskets, TBranch::FlushBaskets, TBranch::FlushOneBaskets)
TTree::AutoSave supports a new option "FlushBaskets" which will call FlushBaskets before saving the TTree object.
Benefits
- Flushing the write baskets has several advantages:
- Reduce the file size of the TTree object (it not longer contains the last basket), improving read time of the TTree object
- Reduce memory footprint of the TTree object.
- In a TTree which "flushed" buffer, there is now usually only zero or one buffer in memory.
- Previously each branch always had at least one basket in memory and usually 2 (the write basket and one read basket).
- Now only the basket of the branches actually read are loaded in memory.
- allow for the basket to be compressed and stored separated, increasing the compression factor.
Note: Calling FlushBaskets too often (either directly of via AutoSave("FlushBaskets")) can lead to unnecessary fragmentation of the ROOT file, since it write the baskets to disk (and a new basket will be started at the next fill) whether or not the content was close to filling the basket or not.
The fast tree cloning (TTreeCloner) was enhanced to support copying in-memory TTrees (that have been save as a single key on file). This issue was preventing hadd to fast clone files containing any 'in-memory' tree.
Revision
23685 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 7 15:18:39 2008 UTC (6 years, 8 months ago) by
brun
File length: 24059 byte(s)
Diff to
previous 22902
Introducing a parallel unzipping algorithm for pre-fetched buffers. Since we already know what buffers are going to be read, we can decompress a few of them in advance in an additional thread and give the impression that the data decompression comes for free (we gain up to 30% in reading intensive jobs).
The size of this unzipping cache is 20% the size of the TTreeCache and can be modified with TTreeCache::SetUnzipBufferSize(Long64_t bufferSize). Theoretically, we only need one buffer in advance but in practice we might fall short if the unzipping cache is too small (synchronization costs).
This experimental feature is disabled by default, to activate it use the static function TTreeCache::SetParallelUnzip(TTreeCacheUnzip::EParUnzipMode option = TTreeCacheUnzip::kEnable). The possible values to pass are: TTreeCacheUnzip::kEnable to enable it,TTreeCacheUnzip::kDisable to disable it and TTreeCacheUnzip::kForce to force it. Since it will only work if you have more than one core, the TTreeCacheUnzip::kForce option is useful to run it even if you have only one cpu (to measure the overhead, for example).
Revision
18667 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri May 4 17:32:45 2007 UTC (7 years, 8 months ago) by
pcanal
Original Path:
trunk/tree/src/TBasket.cxx
File length: 21659 byte(s)
Diff to
previous 18409
The value of TBasket:;fBasket was incorrect in some cases when using
the TTreeCloner. This is solved (and thread safetly is improved) by
removing the gBranch global variable and using alternative mechanism
to set TBakset::fBasket.
Revision
17412 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Jan 20 19:29:35 2007 UTC (8 years ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 21646 byte(s)
Diff to
previous 17402
-Remove Varargs.h from TString.h and TObject.h and add this include
in the few files that need it.
-Move the following enums from TBuffer.h to TBufferFile.h
enum { kMapSize = 503 };
enum { kStreamedMemberWise = BIT(14) }; //added to version number to know if a collection has been stored member-wise
enum { kNotDecompressed = BIT(15) }; //indicates a weird buffer, used by TBasket
enum { kCannotHandleMemberWiseStreaming = BIT(17), //if set TClonesArray should not use memeber wise streaming
kTextBasedStreaming = BIT(18) }; // indicates if buffer used for XML/SQL object streaming
Revision
17402 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jan 19 16:48:00 2007 UTC (8 years ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 21609 byte(s)
Diff to
previous 15653
CVs been changed into a pure abstract interface.
The concrete implementation is in the new class TBufferFile.
All classes previously deriving from TBuffer derive now from TBufferFile, ie
TBuffer <- TBufferFile <- TMessage
<- TBufferXML
<- TBufferSQL
<- TBufferSQL2
Because there are several problems with C++ operators overloading,
The I/O operators are defined in TBuffer. These are inline functions
calling C++ virtual functions defined in TBuffer and overloaded
by TBufferFile and all other derived classes when necessary.
The previous implementation of TBuffer.h included <vector> and Bytes.h.
The two include statements have been moved to TBufferFile.h. As a result the
compilation of the ROOT system is now slightly faster and a big bonus
is that changes in TBufferFile or Bytes.h will affect only TBufferFile
and will not force the recompilation of the entire system.
This change has some side-effects. If you assumed that include <vector>
was done by TBuffer.h, you may have to specify this include directly
in your class. This was the case for a few ROOT classes.
: ----------------------------------------------------------------------
Revision
13914 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jan 24 21:32:46 2006 UTC (9 years ago) by
pcanal
Original Path:
trunk/tree/src/TBasket.cxx
File length: 20703 byte(s)
Diff to
previous 13897
From Sergei Linev:
1) In normal TBasket constructor mother directory assigned to TBranch directory.
2) In TBasket::CopyTo method file for copy is used directly without setting gFile/gDirectory
3) In TBasket::WriteBuffer mother dir assigned once again. It is required when basket with
default constructor was created.
Revision
13896 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jan 23 19:38:34 2006 UTC (9 years ago) by
pcanal
Original Path:
trunk/tree/src/TBasket.cxx
File length: 20190 byte(s)
Diff to
previous 13262
TTreeCloner::CopyMemoryBasket (line 230) had memory leak.
TBasket was created but not used and lost.
TBasket::CopyTo (line 135). Member fHeaderOnly set to kTRUE twice. I think,
second time after basket streaming should be fHeaderOnly = kFALSE.
Thanks Sergei for pointing those out.
Revision
9968 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Sep 10 09:40:56 2004 UTC (10 years, 4 months ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 19385 byte(s)
Diff to
previous 9697
In the TBasket constructor always set fVersion > 1000.
This solves a nasty problem happening when having TBasket created
long before the file reaches 2 GBytes and written long after
the file has been above 2 GBytes.
This creates a small overhead in file size (about 1 per mille
for the Event example).
With this change some cleanup can be made in TKey::Streamer
Revision
9663 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Aug 3 14:50:51 2004 UTC (10 years, 5 months ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 18558 byte(s)
Diff to
previous 7880
Implement a first version of circular buffers for memory resident Trees.
New data member
Long64_t fMaxEntries; // Maximum number of entries for the circular buffers
and corresponding function:
void TTree::SetCircular(Long64_t maxEntries)
where maxEntries is the maximum number of entries to be kept in the buffers.
When the number of entries exceeds this value, the first entries in the Tree
are deleted and the buffers used again.
The implementation minimizes memory operations by swapping basket pointers
and only shifting the data in the new first buffer.
When TTree::Fill reaches fMaxEntries, it calls the new TBranch function:
virtual void KeepCircular(Long64_t maxEntries);
In case a branch buffer needs to be recomputed, this function calls
the new function in TBasket
virtual void MoveEntries(Int_t dentries);
An example of a script using a circular buffer is shown below
void circular() {
gROOT->cd(); //make sure that the Tree is memory resident
TTree *T = new TTree("T","test circular buffers");
TRandom r;
Float_t px,py,pz;
Double_t random;
UShort_t i;
T->Branch("px",&px,"px/F");
T->Branch("py",&py,"px/F");
T->Branch("pz",&pz,"px/F");
T->Branch("random",&random,"random/D");
T->Branch("i",&i,"i/s");
T->SetCircular(20000);
for (i = 0; i < 65000; i++) {
r.Rannor(px,py);
pz = px*px + py*py;
random = r.Rndm();
T->Fill();
}
T->Print();
}
Revision
7880 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Jan 10 10:52:31 2004 UTC (11 years ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 17984 byte(s)
Diff to
previous 7810
From Philippe:
This mega patch introduces a new implementation of the STL I/O
which is backward and forward compatible. In addition this is more
exactly a new implementation or an extension of the container I/O.
We are introducing a new abstract interface:
"TVirtualContainerProxy", which can be implemented to present a
proxy to any collection which the I/O (and TTreeFormula) can use
then transparently.
The TVirtualContainerProxy interface allows to the I/O system to
request some information and to execute some essential function of
the container:
what kind of object/data does it contain
does it contain pointers
how to insert data into the container
how to retrieve an array of pointer to the elements inside
how to create a collection object
how to clear the collection
how to resize the collection
how to streamer the collection (if needed)
how to calculate the sizeof the collection
how to calculate the number of elements of the collection.
Using those primitives, the I/O and TTreeFormula should be able to
access any collection. The I/O should also be able to handle the
splitting of collections that can be split (i.e. contains a single
type of object/data).
The current compromise selected between code duplication,
performance of the I/O of TClonesArray and vector of pointers and
the performance of the I/O of other containers, was to have on
function handle all collection as if they were a C-style array of
pointers to data. This implies for example that the I/O of vector
of object current passes via the construction of temporary array
of pointer. The cost of this construction is usually ~Qjust~R the
cost of calculating the addresses of the elements and assigning it
to an array element.
Registering a collection proxy will be similar to
static int dummy = GenerateInitInstance((CollectType*)0x0)-
>AdoptCollectionProxy(new CollectTypeProxy));
Follows a few details on the modifications made to some of the
files and classes.
Bytes.h:
Work around a problem in the MSVC++ 6.0 optimizer. This should
not affect other compilers.
String:
Included the std::string dictionary into G__Base2.cxx, this
insures its presence at all time.
Added a new file string.cxx holding the streamer for
std::string.
RConfig.h
Added proper ansi stream configuration for AIX, KCC
Added template specialization defect for MSVC
TBrowser
Start adding the ability to browser non TObject classes
TBuffer
To handle the reading and writing array of objects, added:
Int_t CheckByteCount(UInt_t startpos, UInt_t bcnt, const
TClass *clss, const char* classname);
Int_t CheckByteCount(UInt_t startpos, UInt_t bcnt, const
char *classname);
void ReadFastArray(void *start , TClass *cl, Int_t n=1,
TMemberStreamer *s=0);
void ReadFastArray(void **startp, TClass *cl, Int_t n=1,
Bool_t isPreAlloc=kFALSE, TMemberStreamer *s=0);
void WriteFastArray(void *start, TClass *cl, Int_t n=1,
TMemberStreamer *s=0);
Int_t WriteFastArray(void **startp, TClass *cl, Int_t n=1,
Bool_t isPreAlloc=kFALSE, TMemberStreamer *s=0);
TROOT
Enhancement to make the access to gROOT not dependent for the
library loading order. In particular we added:
ROOT::GetROOT()
which should be used instead of gROOT.
Improve support for STL. In particular, now STL containers do
have a corresponding TClass object
TRealData
Replace SetStreamer by AdoptStreamer that allow to use not only
a streamer function but also streamer object (allowing streamer
with a state for Foreign classes)
TString:
Improve streamer performance
TSystem:
More consistency of the return value of TSystem::CompileMacro
build/unix/makecintdlls.sh
Stop making the string.dll
config:
enhance sgicc makefiles
cont:
fix TBits::operator=
TClassTable now warns for duplicate only for non stl classes
TClonesArray fix a typo
gpad:
Add a new class TInspectObject to enable inspect non TObject
classes
TRootBrowser : enable inspect non TObject classes
TFormula/TTreeFormula
To enhance performance switch from using one array fOper which
contained the operation code and its arguments to using 2 arrays
fActions and fActionParams to store respectively the operation and
its parameters.
A new function Convert is used to convert old version from the
fOper to fActions. This allows cleaner coding and offer
optimization opportunities.
TTreePlayer
Start adding support in MakeClass for STL containers.
TRint/TProofServ
Insure the loading of the string support
Event.cxx
make sure to avoid memory over-write
stress.cxx
Add new parameters
stress <nevent> <style> <subbench> <portion>
if <subbench> is 1 or higher, print the benchmark results after
each test. This allows understand
which test is affect by a performance change.
portion is a binary field to indicate which test to run. This
allows to focus on a particular test.
TVirtualCollectionProxy
Abstract interface used to access any type of containers from
the I/O system and TTreeFormula. See TVectorProxy and
TEmulatedVectorProxy for examples.
TEmulatedVectorProxy
Implementation of a std::vector proxy to be able to read a
std::vector without any libraries.
TVectorProxy
Implementation of TVirtualCollectionProxy for a std::vector for
which we have the library.
TStreamerInfo.cxx
Split in 3 files: TStreamerInfo.cxx
TStreamerInfoReadBuffer.cxx TStreamerInfoWriteBuffer.cxx
All the ReadBuffer, ReadBufferClones and the new ReadBufferSTL
(similarly for WriteBuffer) have been factorized into one
function and 2 short wrappers. The new main function expect an
array of pointer to the objects (this array is most often of size
one).
TClonesArray objects pass GetObjectRef(0) to the common
ReadBuffer
vector<bla*> v pass &(v[0])
vector<bla> needs to create an intermediary array to hold the
address
This mechanism is sometimes not optimal (vector<blah>) but
allows extremely flexibly and extension. Basically, to add
support for a new container type using the StreamerInfo mechanism
(in particular allows splitting), one 'just' need to implement a
TVirtualCollectionProxy, which, in particular, will return an
array of address to the object it contains. Even std::map can be
handled with this mechanism, since, for I/O purposes, it can be
consider as a container of pairs.
Add a few optimization, including more caching via a new array
of a new struct (fComp).
Fixed a problem (re)introduced while implementing the Foreign
class CheckSum. Doing:
class Event;
f = new TFile("Event.root");
resulted in errors.
TCint
Add proper support for TClass of STL containers. Fix a memory
leak.
Add support for load TClass via a typedef.
Fix a problem with multiple inheritance
TClass
Fixed a problem (re)introduced while implementing the Foreign
class CheckSum. Doing:
class Event;
f = new TFile("Event.root");
resulted in errors.
Add a
TClass/TGenericClassInfo/TDataMember
Add support for a new interface (TVirtualCollectionProxy)
useable for STL containers or any user defined containers.
Add support for streamer with are objects (as opposed to free
standing function or methods). This allows the user a greater
flexibility in writing a streamer.
Add a few optimizations
Add CanSplit to answer the question for a whole Class (for
example some collection can not be splitted).
TClassStreamer
New class to wrap a class wide streamer
ClassStreamerFunc_t
typedef for a class wide streamer function
TMemberStreamer
New class to wrap a specific class member streamer
MemberStreamerFunc_t
typedef for a specific class member streamer function
RootStreamer
Macro to specify a free standing streamer object or function
for a class
For example:
RootStreamer(string,std_string_streamer);
TStreamerElement:
A couple of optimization/simplification.
Add support for the new STL I/O
Extend the useful
TBranchElement:
add a connection to the proper TVirtualCollectionProxy
add support for STL containers (non-split and split mode)
TTree
Make the function TTree::GetMakeClass NON-virtual for better
efficiency
Add support for STL containers
TBasket
Left (in comment) a yet unproved improvement proposed by
Victor. The preliminary tests were inconclusive performance wise
and it had (seemingly) problem with backward and forward
compatibility.
TBranch
Performance improvements
metautils
This is a new package for C++ files shared between rootcint and
meta.
It contains TClassEdit a class to help determine some property
of a class given its class name (stl, template, etc).
utils
Introduced a new file RStl.cxx to start separating rootcint in
modules.
Modified rootcint to support the new STL I/O methods.
In particular a new class RStl is in charge of doing the
generating of code for STL containers.
Revision
7810 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Dec 30 13:16:51 2003 UTC (11 years ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 16826 byte(s)
Diff to
previous 7703
Mega patch to add support for large files (bigger than 2 GBytes)
----------------------------------------------------------------
==========>NEW VERSION 4.00/00 <===========
Because this new version has a considerable number of changes,
and new important additions by Philippe are scheduled for the coming days
we are starting a new major version 4.
-Support for large files
-Automatic schema evolution for foreign classes
-New data type Double32_t
Large files are currently tested only under Linux with gcc3.2.
Support for other systems will be gradually added in the coming days.
By default under Linux, files are created with the option LARGEFILE.
Note that when creating a Tree, the default maximum size for a Tree
is set to 1.9 GBytes. You can change this default value by calling
TTree::SetMaxTreeSize(Long64_t maxsize)
The implementation of this new feature has required changes in many places.
The files produced by this new version can still be read by old ROOT versions
if their size if smaller than 2 GBytes. And obviously, the new version
can digest old ROOT files.
WARNING: note the following backward incompatibility:
-TSystem::GetPathInfo has a new signature: (also TUnixSystem, TWinNTSystem, etc)
old: GetPathInfo(const char *path, Long_t *id, Long_t *size, Long_t *flags, Long_t *modtime);
new: GetPathInfo(const char *path, Long_t *id, Long64_t *size, Long_t *flags, Long_t *modtime);
-TFile::SysSeek and TFile::SysStat have a new signature (also TNetFile, TWebFile, etc)
old: Int_t SysSeek(Int_t fd, Long_t offset, Int_t whence);
Int_t SysStat(Int_t fd, Long_t *id, Long_t *size, Long_t *flags, Long_t *modtime);
new: Long64_t SysSeek(Int_t fd, Long64_t offset, Int_t whence);
Int_t SysStat(Int_t fd, Long_t *id, Long64_t *size, Long_t *flags, Long_t *modtime);
-TTree::SetMaxTreeSize has a new signature:
old: void TTree::SetMaxTreeSize(Int_t maxsize)
new: void TTree::SetMaxTreeSize(Long64_t maxsize)
All references to Seek_t have been replaced by Long64_t.
Revision
7703 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Dec 8 16:04:36 2003 UTC (11 years, 1 month ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 16822 byte(s)
Diff to
previous 7628
In the TBasket constructor, preset the array fEntryOffset.
The non-initialized members of fEntryOffset were not used, but
were given a problem to the ZIP algorithm and the cause of the
many errors reported by valgrind.
Revision
7368 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Sep 27 10:49:55 2003 UTC (11 years, 3 months ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 16707 byte(s)
Diff to
previous 7287
In TBasket::ReadBasketBuffers, add support for reading buffers that could not be
compressed with very old versions of Root.
We already added support for version 3.02, 3.03 and 3.04.
This patch can also process files written with older versions.
Revision
6512 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 30 16:29:31 2003 UTC (11 years, 8 months ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 16428 byte(s)
Diff to
previous 6186
From Philippe:
This patch adds the capability to transfer buffer directly from one tree to another
without creating the contained objects or even without uncompressing the buffer
(in the case where there is only one object in the buffer).
This code has been in usage in CDF for at least 1 1/2 year and the code has not been
changed since at least 12 months.
Revision
6186 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 26 10:11:51 2003 UTC (11 years, 10 months ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 14983 byte(s)
Diff to
previous 5819
Implement all the changes required to process files > 2 GBytes.
Activating this option will imply changing Seek_t to long long.
We hope that Masa will implement the longlong type by default
in the not too distant future in CINT.
The file format remains compatible for files smaller than the
threshold set in TFile::kStartBigFile = 2 GigaBytes.
The changes are the following:
in TFile
========
Remove the enum kBegin, kUnits (not used anymore).
Add new enum kStartBigFile
When creating a new file (in TFile::Init) fBEGIN is now set to 100
bytes instead of 64. TFile::WriteHeader is called assuming that
the file will be smaller than kStartBigFile.
When closing the file, if the file is bigger than kStartBigFile,
a new format is used in WriteHeader using 64 bits to store all
values of type Seek_t and the file version (fVersion) is set
to fVersion+1000000.
All functions like TFile::Map, TFile::Recover have been adapted
to follow the new convention.
in TDirectory
=============
TDirectory::Sizeof has been modified to return 12 bytes in addition
for fSeekDir, fSeekParent and fSeekKeys when writing a directory
in a file bigger than kStartBigFile.
The TDirectory constructor is setting a temporary bit to inform
the function Sizeof that it has to return 12 addtional bytes.
TDirectory::FillBuffer writes fSeekDir, fSeekParent and fSeekKeys
with 64 bits if necessary. The version number of the directory
is set to TDirectory class version + 1000 in this case.
Corresponding changes in TDirectory::ReadBuffer and TDirectory::Streamer.
in TFree
========
TFree::Sizeof has been moved to the implementation file.
Like for TDirectory, the functions FillBuffer and ReadBuffer have been
modified to write fFirst and fLast in 64 bits if necessary.
When writing in 64 bits mode, the TFree version is incremented by 1000.
in TKey and TBasket
===================
Same changesas in TDirectory in teh functions FillBuffer, ReadBuffer
and Streamer. The TKey version number is incremented by 1000
when writing in 64 bits mode.
Revision
5807 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Dec 13 19:17:47 2002 UTC (12 years, 1 month ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 14511 byte(s)
Diff to
previous 4618
In TBasket::WriteBuffer, if after compression the size of the compressed buffer
is identical to the size of the uncompressed buffer, write the buffer not compressed.
This situation may happen in some rare cases depending on the data in the buffer.
Revision
2079 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Apr 27 06:35:39 2001 UTC (13 years, 9 months ago) by
brun
Original Path:
trunk/tree/src/TBasket.cxx
File length: 13034 byte(s)
Diff to
previous 1583
Fix in TBasket::Streamer to take into account a complex case when
one reads an entry in the first basket, an entry in the last basket
then Fill again the Tree. (Thanks Anton Fokin for reporting this problem).
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
