Log of /trunk/tree/tree/inc/TTreeCache.h
Parent Directory
Revision
45620 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 15 16:52:04 2012 UTC (2 years, 5 months ago) by
pcanal
File length: 4937 byte(s)
Diff to
previous 45619
From Jack (from ANL):
Added TTreeCache::LearnPrefill, which does a large cache pre-fill the first time an entry is read from a branch of a tree. I let TTreeCache::FillBuffer do all the prefetching, so it supports the cases where the first cluster and/or the first basket of each branch don't fit in the cache.
Revision
45619 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 15 16:23:03 2012 UTC (2 years, 5 months ago) by
pcanal
File length: 4898 byte(s)
Diff to
previous 44681
From Jack (from ANL):
Add accessor to TTreeCache::GetEntryMin and GetEntryMax. Previously, they could be set with TTreeCache::SetEntryRange, but not read.
Revision
44681 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 13 19:31:49 2012 UTC (2 years, 7 months ago) by
pcanal
File length: 4772 byte(s)
Diff to
previous 44679
From Peter van Gemmeren and Jack Weinstein:
Index: tree/tree/src/TBasket.cxx
* Disable the default TTreeCache before calls to TFile::ReadBuffer() to avoid needless calls to default TFileCacheRead ReadBuffer.
Index: tree/tree/inc/TTreeCache.h
Index: tree/tree/src/TTreeCache.cxx
* Define and implement new Enable(), Disable() and IsEnabled() functions
Revision
44679 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 13 18:23:51 2012 UTC (2 years, 7 months ago) by
pcanal
File length: 4536 byte(s)
Diff to
previous 42856
Restore support for the use case (existing in CMS code for example):
TFile *input = new TFile(filename);
TTree *tree; input.GetObject(treename,tree);
TTreeCache *cache = new TTreeCache(tree,cachesize);
input.SetCacheRead(cache);
..... read the tree ...
input.SetCacheRead(0);
delete cache;
delete input; // This was leading to a double delete due to TFile::fReadCacheMap still holding a reference to the cache
Revision
42856 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Feb 2 21:09:30 2012 UTC (2 years, 11 months ago) by
pcanal
File length: 4491 byte(s)
Diff to
previous 41101
In TTreeCache remove the (mostly unused) location notion of owner (fOwner) as the ownership is now shared explicitly between a TTree and its TFile. In TTreeCache::UpdateBranches, also migrate the ownership from one file to the other
Revision
39275 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu May 19 18:17:37 2011 UTC (3 years, 8 months ago) by
pcanal
File length: 4472 byte(s)
Diff to
previous 34935
From Elvin Alin Sindrilaru:
The prefetching mechanism uses two new classes (TFilePrefetch.h and
TFPBlock.h) to prefetch in advance a block of entries. There is a second
thread which takes care of actually transferring the blocks and making
them available to the main requesting thread. Therefore, the time spent
by the main thread waiting for the data before processing considerably
decreases. Besides the prefetching mechanisms there is also a local
caching option which can be enabled by the user. Both capabilities are
disabled by default and must be explicitly enabled by the user.
In order to enable the prefetching the user must define the environment
variable "TFile.AsyncPrefetching" as follows:
gEnv->SetValue("TFile.AsyncPrefetching", 1).
Only when the prefetching is enabled can the user set the local cache
directory in which the file transferred can be saved. For subsequent
reads of the same file the system will use the local copy of the file
from cache. To set up a local cache directory, a client can use the
following commands:
TString cachedir="file:/tmp/xcache/";
// or using xrootd on port 2000
// TString cachedir="root://localhost:2000//tmp/xrdcache1/";
gEnv->SetValue("Cache.Directory", cachedir.Data());
The "TFilePrefetch" class is responsible with actually reading and storing
the requests received from the main thread. It also creates the working
thread which will transfer all the information. Apart from managing the
block requests, it also deals with caching the blocks on the local machine
and retrieving them when necessary.
The "TFPBlock" class represents the encapsulation of a block request. It
contains the chunks to be prefetched and also serves as a container for
the information read.
In order to accommodate the new prefetching mechanism the following files
have suffered considerable modifications: TFileCacheRead.cxx/.h,
TTreeCache.cxx/.h and to a less extent: TXNetFile.cxx, TFile.h.
Basically in TFileCacheRead we've added the logic for dealing with the
second buffer that is prefetched. In TTreeCache during prefeching the
method FillBuffer is called after each read so that once the main thread
starts reading from the last available buffer, the second thread starts
prefetching the next block.
Revision
34935 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Aug 23 09:13:37 2010 UTC (4 years, 5 months ago) by
brun
File length: 3793 byte(s)
Diff to
previous 31457
Implement TTreeCache::Print that shows information like:
// ******TreeCache statistics for file: cms2.root ******
// Number of branches in the cache ...: 1093
// Cache Efficiency ..................: 0.997372
// Cache Efficiency Rel...............: 1.000000
// Learn entries......................: 100
// Reading............................: 72761843 bytes in 7 transactions
// Readahead..........................: 256000 bytes with overhead = 0 bytes
// Average transaction................: 10394.549000 Kbytes
// Number of blocks in current cache..: 210, total size: 6280352
This function can be called directly from TTreePrint
T->PrintCacheStats();
Revision
31457 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 30 13:36:37 2009 UTC (5 years, 1 month ago) by
brun
File length: 3724 byte(s)
Diff to
previous 31077
From Fabrizio:
In TFileCacheRead add a new virtual function GetUnzipBuffer in order
to avoid InheritsFrom in TBasket::ReadBasketBuffers.
In TTreeCacheUnzip optimizations related to CPU cycles and memory.
Related fixes to TFileCacheRead. Also comment "if gDebug" statements.
In TXNetFile fixes related to the reporting of TTreePerfStats.
Revision
31077 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 11 08:51:57 2009 UTC (5 years, 2 months ago) by
brun
File length: 3839 byte(s)
Diff to
previous 30962
Implement the cache functions AddBranch and SetSkipZip in TFileCacheRead too.
This speeds up TBranch::GetBasket where it is not necessary to test if the file cache
is a TTreeCache.
Revision
30962 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 3 15:21:40 2009 UTC (5 years, 2 months ago) by
pcanal
File length: 3979 byte(s)
Diff to
previous 30208
Clarify and correct when FillBuffer is actually executed.
Document when (and how) the learning phase is started and stopped.
Fix SetEntryRange when the Cache learning was manual.
With the previous implementation requesting the entry from a branch
that was not marked to be cached when the entry (number) was less
than fEntryNext resulted in the re-reading of any baskets between
this entry number and fEntryNext. (This would happen also if
the branch was being cached but for some reason some of its baskets
did not fit in the cache).
Revision
30208 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Sep 16 17:30:17 2009 UTC (5 years, 4 months ago) by
brun
File length: 3901 byte(s)
Diff to
previous 23685
Add a direct interface from TTree to TTreeCache with the functions:
virtual void AddBranch2Cache(const char *bname, Bool_t subbranches = kFALSE);
virtual void AddBranch2Cache(TBranch *branch, Bool_t subbranches = kFALSE);
virtual void SetCacheEntryRange(Long64_t first, Long64_t last);
virtual void SetCacheLearnEntries(Int_t n=10);
virtual void StopCacheLearningPhase();
When adding all branches to the cache the learning phase is automatically
stopped.
Change the learning phase from 100 to 10 entries.
It still remains to change TTree::Process such that the cache entry range
is automatically set.
Revision
23685 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 7 15:18:39 2008 UTC (6 years, 8 months ago) by
brun
File length: 3902 byte(s)
Diff to
previous 22902
Introducing a parallel unzipping algorithm for pre-fetched buffers. Since we already know what buffers are going to be read, we can decompress a few of them in advance in an additional thread and give the impression that the data decompression comes for free (we gain up to 30% in reading intensive jobs).
The size of this unzipping cache is 20% the size of the TTreeCache and can be modified with TTreeCache::SetUnzipBufferSize(Long64_t bufferSize). Theoretically, we only need one buffer in advance but in practice we might fall short if the unzipping cache is too small (synchronization costs).
This experimental feature is disabled by default, to activate it use the static function TTreeCache::SetParallelUnzip(TTreeCacheUnzip::EParUnzipMode option = TTreeCacheUnzip::kEnable). The possible values to pass are: TTreeCacheUnzip::kEnable to enable it,TTreeCacheUnzip::kDisable to disable it and TTreeCacheUnzip::kForce to force it. Since it will only work if you have more than one core, the TTreeCacheUnzip::kForce option is useful to run it even if you have only one cpu (to measure the overhead, for example).
Revision
20794 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 14 01:41:56 2007 UTC (7 years, 2 months ago) by
brun
Original Path:
trunk/tree/inc/TTreeCache.h
File length: 3609 byte(s)
Diff to
previous 20379
From Gerri:
Following the implementation in TChain, I have succeded in enabling the TTree cache in Proof.
However, for that I need a small addition in TTreeCache to be able to change also the 'fOwner'
member, because on the workers we have to delete the previous tree, so the owner tree creating
the cache is gone after the first file.
In the attached patch I propose to add a second argument to TTreeCache::UpdateBranches, a boolean
indicating whether to set also the ownership or not. Default is not, so we do not break any existing
usage of the call.
With this, and other changes in TEventIter that I am finalizing, I get for h1analysis with http files
from my place and my single core machine
TChain 52s
Proof 1 worker 57-62s
i.e. a 10-20% overhead in Proof, which is what we expect without the local optimizations.
Revision
20280 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Oct 9 18:47:29 2007 UTC (7 years, 3 months ago) by
pcanal
Original Path:
trunk/tree/inc/TTreeCache.h
File length: 3546 byte(s)
Diff to
previous 19826
From Leo:
Add the ability to manually enter the list of used branches.
root [0] f = TFile::Open("root://localhost://tmp/Event.root");
root [1] TTree *T = (TTree*)f->Get("T");
root [2] T->SetCacheSize(10000000);
root [4] TFile::GetFileReadCalls()
(Int_t)(7)
root [5] f->GetBytesRead()
(const Long64_t)638632
root [3] TTreeCache *tpf = (TTreeCache*) f->GetCacheRead();
root [6] tpf->AddBranch("fTracks", kTRUE);
root [7] tpf->StopLearningPhase();
root [8] T->Draw("fTracks.fPx", "fTracks.fPy");
<TCanvas::MakeDefCanvas>: created default TCanvas with name c1
root [9] TFile::GetFileReadCalls()
(Int_t)(42)
root [10] f->GetBytesRead()
(const Long64_t)359898352
Revision
15626 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 29 22:15:37 2006 UTC (8 years, 6 months ago) by
rdm
Original Path:
trunk/tree/inc/TTreeCache.h
File length: 2964 byte(s)
Diff to
previous 15572
Several fixes in the new cache handling:
- TFile
- GetBytesWritten() includes also the number of bytes still in the
write cache
- New method FlushWriteCache()
- Call FlushWriteCache() also when closing files in raw mode
- Call FlushWriteCache() also in Flush()
- TFileCacheWrite:
- new method GetBytesInCache() returning bytes still in cache
used by TFile::GetBytesWritten()
- WriteBuffer() returns int iso bool, -1 write failure, 0 in case
recusively called and 1 in case copied in cache
- TFileCacheRead:
- ReadBuffer() returns int iso bool, -1 read failure, 0 in case not
in cache and 1 in case read from cache.
- TTreeCache:
- use int return type for ReadBuffer()
- TNetFile and TXNetFile:
- call FlushWriteCache() in their Flush().
Revision
15572 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Tue Jun 27 14:36:28 2006 UTC (8 years, 6 months ago) by
brun
Original Path:
trunk/tree/inc/TTreeCache.h
File length: 2976 byte(s)
-Rename class TFilePrefetch -> TFileCacheRead
-Rename class TTreeFilePrefetch -> TTreeCache
-Add new class TFileCacheWrite that implements automatic caching
when writing network files. This class replaces the previous class TCache.
-old TCache class deleted.
-TFileCacheRead can read buffers from the wite cache in TFileCacheWrite.
-As a result TFile::UseCache is obsolete. The function is kept for
backward compatibility.
-Remove references to TCache from TXMLFile, TSQLFile, TwebFile, TNetFile
-remove pointer TFile::fCache
-Add pointers TFile::fCacheRead and TFile::fCacheWrite
-Remove members fMaxCacheSize and fPageSize from TChain
NOTE that it is recommended to
rm -f base/src/*.d base/src/*.o
rm -f net/src/*.d net/src/*.o
rm -f tree/src/*.d tree/src/*.o
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
