Log of /trunk/tree/tree/src/TTreeCache.cxx
Parent Directory
Revision
48992 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Mar 28 15:26:26 2013 UTC (21 months, 3 weeks ago) by
rdm
File length: 47862 byte(s)
Diff to
previous 48306
From Lifeng Sun:
The attached patchset fixes a bunch of typo in the source:
0001-succes-success.patch
0002-preceed-preced.patch
0003-informations-information.patch
0004-childs-children.patch
0005-avaliable-available.patch
0006-writeable-writable.patch
0007-comand-command.patch
0008-unkown-unknown.patch
0009-wierd-weird.patch
0010-wheter-whether.patch
0011-unecessary-unnecessary.patch
0012-splitted-split.patch
0013-registerd-registered.patch
0014-recieve-receive.patch
0015-processsing-processing.patch
0016-ouput-output.patch
0017-mutiple-multiple.patch
0018-lenght-length.patch
0019-interupted-interrupted.patch
0020-independant-independent.patch
0021-inconsistant-inconsistent.patch
0022-expresion-expression.patch
0023-explicitely-explicitly.patch
0024-enviroment-environment.patch
0025-deafult-default.patch
0026-continous-continuous.patch
0027-completly-completely.patch
0028-commited-committed.patch
0029-choosen-chosen.patch
0030-backgroud-background.patch
0031-auxilliary-auxiliary.patch
0032-authentification-authentication.patch
0033-appropiate-appropriate.patch
0034-an-other-another.patch
0035-environement-environment.patch
0036-targetting-targeting.patch
0037-suppported-supported.patch
0038-paramater-parameter.patch
Revision
48306 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jan 16 16:01:47 2013 UTC (2 years ago) by
pcanal
File length: 47861 byte(s)
Diff to
previous 48146
From Elvin:
There was a race condition between the destructor of the TTree object and the functionality to kill the asynchronous thread doing the prefetching.
In more detail: In the the TTree::~TTree line 789 the TFileCacheRead object of the current file is set to 0. All the logic to kill the async thread is done in the destructor of TFilePrefetch which in turn is called from the destructor of TFileCacheRead. In the same file two lines below the destructor of TFileCacheRead is called. And initially TFilePrefetch held a pointer to the file object in TFileCacheRead which now is 0. Therefore, during the destruction of the TFilePrefetch object we don't have any longer a valid TFile pointer. So, we can not wait for the ongoing requests to be satisfied. This was the reason of the crash. To fix this, I removed the killing of the async thread form the destructor of the TFilePrefetch method and I've put it in a separate method called WaitFinishPrefetch. In this way, we avoid the potential scenario of trying to wait for some request for a file while not having a valid pointer to that file.
Revision
45620 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 15 16:52:04 2012 UTC (2 years, 5 months ago) by
pcanal
File length: 46523 byte(s)
Diff to
previous 45494
From Jack (from ANL):
Added TTreeCache::LearnPrefill, which does a large cache pre-fill the first time an entry is read from a branch of a tree. I let TTreeCache::FillBuffer do all the prefetching, so it supports the cases where the first cluster and/or the first basket of each branch don't fit in the cache.
Revision
45494 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Aug 8 19:03:13 2012 UTC (2 years, 5 months ago) by
pcanal
File length: 45053 byte(s)
Diff to
previous 45487
From Elvin:
I received a test program from an user in which the async prefetching was blocking. After doing some debugging I managed to fix the problem. The issue was that when prefetching and reaching the end of the entries, the program should stop trying to prefetch further if the current entry is equal to fEntryMax.
This was not happening in the test program because there I am reading only some entries stopping well before the end of the total number of entries. The fix for this was the addition of a condition to FillBuffer around line 640.
Revision
44750 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 26 20:11:39 2012 UTC (2 years, 6 months ago) by
pcanal
File length: 44858 byte(s)
Diff to
previous 44681
From Peter, Jack and Elvin:
Indeed one can drop the unnecessary clearing of cache after looping through the collection of baskets. This affects in no way the prefetching. It was left it there for "historical" reasons, when trying to figure out how all the mechanism was working. The FillBuffer code is now pretty entangled and initially it was thought the prefetcher was using this to get the direction of the reads but no, this can be correctly established even without the clearing of the cache.
Revision
44681 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 13 19:31:49 2012 UTC (2 years, 7 months ago) by
pcanal
File length: 44965 byte(s)
Diff to
previous 44679
From Peter van Gemmeren and Jack Weinstein:
Index: tree/tree/src/TBasket.cxx
* Disable the default TTreeCache before calls to TFile::ReadBuffer() to avoid needless calls to default TFileCacheRead ReadBuffer.
Index: tree/tree/inc/TTreeCache.h
Index: tree/tree/src/TTreeCache.cxx
* Define and implement new Enable(), Disable() and IsEnabled() functions
Revision
44679 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 13 18:23:51 2012 UTC (2 years, 7 months ago) by
pcanal
File length: 44843 byte(s)
Diff to
previous 44671
Restore support for the use case (existing in CMS code for example):
TFile *input = new TFile(filename);
TTree *tree; input.GetObject(treename,tree);
TTreeCache *cache = new TTreeCache(tree,cachesize);
input.SetCacheRead(cache);
..... read the tree ...
input.SetCacheRead(0);
delete cache;
delete input; // This was leading to a double delete due to TFile::fReadCacheMap still holding a reference to the cache
Revision
43934 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Apr 24 21:55:59 2012 UTC (2 years, 9 months ago) by
pcanal
File length: 44237 byte(s)
Diff to
previous 43518
In case where the TTree has a dramatic dynamic range with a lots of very small entries
at the beginning and very large entries at the end, the size in bytes of the cluster for the
later entries will be very large (because of the cluster size in entries is large!).
TTreeCache::FillBuffer was always attempting to load complete clusters not matter the
size (even with the size was larger than 2GB!).
This patch resolves the issue by limiting the amount of memory used to
a) The requested size if more than one cluster fits in the cache
b) Twice the requested size if at least one basket per branch fits in the cache
c) Four time the requested size in the case where the cache can not even hold one
basket per branch.
The filling will restart at the next cluster boundary in the case a) and will
restart at the maximum of entry number read in the cache in the case b) and c).
Baskets that are below this boundary and did not fit in the cache will be read
individually.
In TTree::Streamer properly estimate the necessary cache size (this cache size
is used by TTree::Draw).
Revision
43185 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 29 16:53:23 2012 UTC (2 years, 10 months ago) by
pcanal
File length: 39963 byte(s)
Diff to
previous 43162
In TFile: TMap::FindObject and TMap::GetValue are not interchangeable! The former returns a TPair.
Also add missing check for null pointer in TTreeCache.
Revision
43162 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Feb 28 17:58:47 2012 UTC (2 years, 10 months ago) by
pcanal
File length: 39931 byte(s)
Diff to
previous 42856
From Peter:
Index: io/io/src/TFile.cxx
* Have TFile SetCacheRead() disassociate itself from cache, when cache is set to 0.
Index: tree/tree/src/TTreeCache.cxx
* Let TFile handle association between file and cache.
Index: tree/tree/src/TChain.cxx
* Let TFile handle association between file and cache.
Revision
42856 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Feb 2 21:09:30 2012 UTC (2 years, 11 months ago) by
pcanal
File length: 40113 byte(s)
Diff to
previous 42755
In TTreeCache remove the (mostly unused) location notion of owner (fOwner) as the ownership is now shared explicitly between a TTree and its TFile. In TTreeCache::UpdateBranches, also migrate the ownership from one file to the other
Revision
42755 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jan 20 20:51:49 2012 UTC (3 years ago) by
pcanal
File length: 40247 byte(s)
Diff to
previous 42754
Update r42751 to support the legacy mechanism used directly by user
code to support multiple cache per TFile, e.g.:
TTree *tree1; input.GetObject("tree1",tree1);
tree1->SetCacheSize(300*1024);
TFileCacheRead *cache1 = input.GetCacheRead();
input.SetCacheRead(0);
TTree *tree2; input.GetObject("tree2",tree2);
tree2->SetCacheSize(400*2048);
TFileCacheRead *cache2 = input.GetCacheRead();
input.SetCacheRead(0);
and then
input.SetCacheRead(cache1);
tree1->GetEntry(entry1);
input.SetCacheRead(cache2);
tree2->GetEntry(entry2);
and then
delete cache1;
Note that this is 'just' for the sake of backward compatilility and
the code should be updated to the simpler:
TTree *tree1; input.GetObject("tree1",tree1);
tree1->SetCacheSize(300*1024);
TTree *tree2; input.GetObject("tree2",tree2);
tree2->SetCacheSize(400*2048);
tree1->GetEntry(entry1);
tree2->GetEntry(entry2);
Revision
42754 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jan 20 20:13:07 2012 UTC (3 years ago) by
pcanal
File length: 40093 byte(s)
Diff to
previous 41698
From Peter van Gemmeren:
Allow multiple read caches per file by adding a fCacheReadMap member and owner arguments to GetCacheRead() and SetCacheRead()
to TFile.
Index: io/io/inc/TFileCacheRead.h
Index: io/io/src/TFileCacheRead.cxx
* Adding new argument for constructor to specify TTree owner
* whitespace indents
Index: tree/tree/inc/TBasket.h
Index: tree/tree/src/TBasket.cxx
* Add optional tree owner argument to LoadBasketBuffers() to retrieve cache.
* In LoadBasketBuffers() and ReadBuffers() check the cache first than call TFile::ReadBuffer only if needed.
Index: tree/tree/inc/TChain.h
Index: tree/tree/src/TChain.cxx
* Override SetCacheSize() function to update TTree owner on the current cache.
* whitespace indents
Index: tree/tree/src/TTree.cxx
* Get TTreeCache for current TTree, by specifying owner in GetCacheRead() calls
* Always delete owned cache
Index: tree/tree/src/TTreeCache.cxx
* Adding new argument for constructor for TFileCacheRead to specify TTree owner
Index: tree/treeplayer/src/TTreePlayer.cxx
* Get TTreeCache for current TTree, by specifying owner in GetCacheRead() calls
Index: tree/tree/src/TTreeCloner.cxx
* Call TBasket::LoadBasketBuffers() with current TTree as owner of cache
Index: proof/proofplayer/src/TEventIter.cxx
* Get TTreeCache for current TTree, by specifying owner in GetCacheRead() and SetCacheRead() calls
Revision
41698 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 1 21:22:54 2011 UTC (3 years, 2 months ago) by
pcanal
File length: 40088 byte(s)
Diff to
previous 41345
From Elvin:
Last week Martin Vala from ALICE came to me with a problem that he had
while using the asynchronous prefetching. There were basically two
main problems:
1. Trying to read a root file from an archive. Here the problem was
that when reading from an archive there is an offset of the file which
was not taken into consideration when the file was saved in the cache.
And this lead to a miss when reading the file from cache. I fixed it,
but I had to expose the value of fArchiveOffset from TFile.
2. The second problem was when reading using a TChain. There were some
synchronization issues concerned to the asynchronous thread that
actually does the reading. All this was happening because in the case
of TChain there is only one file cache which is re-utilized as we move
from one file to another. This was a pretty tricky issue.
I attached a patch made against the current trunk which fixes both
this problems. I gave the patch first to Martin to test it, and he was
satisfied with it. There is a small delay when the TChain moves from
one file to another because I have to wait for the async thread to
finish it's worked but over all Martin said that the performance is
way better than before. When I initially did the asyn pre-fetching I
had no idea about these two use cases, so that's why they popped up
now.
Revision
41345 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Oct 11 22:20:46 2011 UTC (3 years, 3 months ago) by
pcanal
File length: 39980 byte(s)
Diff to
previous 41101
From Peter Van Gemmeren:
Add an option value ("cachedbranches") to the Print() function of TTreeCache to be able to print the list of cached branches.
Revision
39673 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jun 10 16:11:43 2011 UTC (3 years, 7 months ago) by
pcanal
File length: 35488 byte(s)
Diff to
previous 39458
From Elvin
- completely remove the recycle list and I recycle blocks directly from the read list ( the oldest block in the list is recycled first)
- improve the prefetching strategy so that if the user reads sparsely (only one entry from a block) then the prefetching thread won't prefetch the following block as it will never be used. But it will prefect the block corresponding to the new entry requested.
- so now for example if one wants to read only entries 0, 1000, 2000 and 3000 the program will only prefetch 4 blocks (in comparison to 32 as it did before)
- this also leads to smaller run times when reading sparsely
- by removing the recycle list, during any type of execution (sequential, sparse) I only use two TFPBlock thus considerably reducing the memory footprint. (you can see how blocks are created and recycled by putting two prints in TFilePrefetch::CreateObject), and valgrind --tool=massif shows a maximum size of 60 MB allocated for TFPBlock.
Revision
39458 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri May 27 15:07:18 2011 UTC (3 years, 7 months ago) by
pcanal
File length: 35383 byte(s)
Diff to
previous 39344
From Elvin:
Disable the normal reading mode as a fall-back method of reading and use only the prefetching mechanism.
For the case when a request is not satisfied from the first try, we now will continue to prefetch until
the request is within the blocks read.
The problem seemed to appear only in the TWebFile plug-in as it was using the same connection for sending
requests regardless of the thread. From what I understood in xrd things are different and this problem
didn't appear while reading using the TXNetFile plug-in.
Also change the type of the prefetching thread from a detached one to a joined one as there were some
issues with synchronization if the main thread finished reading before the worker thread finished prefetching blocks.
Revision
39275 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu May 19 18:17:37 2011 UTC (3 years, 8 months ago) by
pcanal
File length: 35401 byte(s)
Diff to
previous 39255
From Elvin Alin Sindrilaru:
The prefetching mechanism uses two new classes (TFilePrefetch.h and
TFPBlock.h) to prefetch in advance a block of entries. There is a second
thread which takes care of actually transferring the blocks and making
them available to the main requesting thread. Therefore, the time spent
by the main thread waiting for the data before processing considerably
decreases. Besides the prefetching mechanisms there is also a local
caching option which can be enabled by the user. Both capabilities are
disabled by default and must be explicitly enabled by the user.
In order to enable the prefetching the user must define the environment
variable "TFile.AsyncPrefetching" as follows:
gEnv->SetValue("TFile.AsyncPrefetching", 1).
Only when the prefetching is enabled can the user set the local cache
directory in which the file transferred can be saved. For subsequent
reads of the same file the system will use the local copy of the file
from cache. To set up a local cache directory, a client can use the
following commands:
TString cachedir="file:/tmp/xcache/";
// or using xrootd on port 2000
// TString cachedir="root://localhost:2000//tmp/xrdcache1/";
gEnv->SetValue("Cache.Directory", cachedir.Data());
The "TFilePrefetch" class is responsible with actually reading and storing
the requests received from the main thread. It also creates the working
thread which will transfer all the information. Apart from managing the
block requests, it also deals with caching the blocks on the local machine
and retrieving them when necessary.
The "TFPBlock" class represents the encapsulation of a block request. It
contains the chunks to be prefetched and also serves as a container for
the information read.
In order to accommodate the new prefetching mechanism the following files
have suffered considerable modifications: TFileCacheRead.cxx/.h,
TTreeCache.cxx/.h and to a less extent: TXNetFile.cxx, TFile.h.
Basically in TFileCacheRead we've added the logic for dealing with the
second buffer that is prefetched. In TTreeCache during prefeching the
method FillBuffer is called after each read so that once the main thread
starts reading from the last available buffer, the second thread starts
prefetching the next block.
Revision
39059 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Sat Apr 30 20:28:06 2011 UTC (3 years, 8 months ago) by
pcanal
File length: 29268 byte(s)
Diff to
previous 37955
Introduce support for TTree with variable cluster size (i.e. value of fAutoFlush).
The information is kept in the new data members: fNClusterRange, fMaxClusterRange, fClusterRangeEnd, fClusterSize.
(See TTree::SetAutoFlush for more details).
Iterating through the cluster should be done via the new class TTree::TClusterIterator (i.e. this replaces += fAutoFlush):
TTree::TClusterIterator clusterIter = tree->GetClusterIterator(which_entry_to_start_from);
Long64_t clusterStart;
while( (clusterStart = clusterIter()) < tree->GetEntries()) {
printf("The cluster starts at %lld and ends at %lld\n",clusterStart,clusterIter.GetNextEntry()-1);
}
See TTreeCache::FillBuffer for a concrete usage example.
Revision
37955 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 2 19:00:41 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 29834 byte(s)
Diff to
previous 37950
From Brian Bockelman:
Add more clusters to the TTreeCache buffer until fBufferMinSize is hit to avoid severely underfilled buffer when
a low number of branches is selected/used.
Revision
37950 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Feb 2 17:06:20 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 29208 byte(s)
Diff to
previous 37919
From Brian Bockelman:
Fix for the case of old files where fEntryNext was set to entries + (some value computed via integer division).
Where the value computed may indeed be zero.
Hence, fEntryNext == entries and the TTC does nothing
This was a case where there was one 12MB event in the TTree (the CMSSW "Run" tree)
Revision
37917 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jan 31 18:12:34 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 29220 byte(s)
Diff to
previous 37916
When resetting the learning also reset fEntryCurrent to properly catch the 'learning range'.
When adding all branches explicitly (AddBranch("*)), request the reading of the cluster, whether
we are in learning phase or not.
When seting fEntryCurrent to match the start of a cluster, also make sure it does not go below fEntryMin.
(With this, the TTreeCache is really only used when the entry requested in within the user provided range).
Revision
37916 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jan 31 17:08:07 2011 UTC (3 years, 11 months ago) by
pcanal
File length: 28981 byte(s)
Diff to
previous 37906
From Yuri Fisyak:
Fix the fact that sometime with rather small TTree with large numbers of entries
one could get division by 0 because averageEntrySize is 0.
Revision
34935 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Aug 23 09:13:37 2010 UTC (4 years, 5 months ago) by
brun
File length: 28827 byte(s)
Diff to
previous 34768
Implement TTreeCache::Print that shows information like:
// ******TreeCache statistics for file: cms2.root ******
// Number of branches in the cache ...: 1093
// Cache Efficiency ..................: 0.997372
// Cache Efficiency Rel...............: 1.000000
// Learn entries......................: 100
// Reading............................: 72761843 bytes in 7 transactions
// Readahead..........................: 256000 bytes with overhead = 0 bytes
// Average transaction................: 10394.549000 Kbytes
// Number of blocks in current cache..: 210, total size: 6280352
This function can be called directly from TTreePrint
T->PrintCacheStats();
Revision
31035 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 9 10:01:05 2009 UTC (5 years, 2 months ago) by
brun
File length: 27817 byte(s)
Diff to
previous 30962
Optimize the logic for the AutoFlush mechanism such that the TreeCache
will read always up to the point where FlushBaskets has been called.
This minimizes the number of cases where one has to seek backward when reading.
The new interface for TTree::SetAutoFlush is:
void TTree::SetAutoFlush(Long64_t autof)
// This function may be called at the start of a program to change
// the default value for fAutoFlush.
//
// CASE 1 : autof > 0
// ------------------
// autof is the number of consecutive entries after which TTree::Fill will
// flush all branch buffers to disk.
//
// CASE 2 : autof < 0
// ------------------
// When filling the Tree the branch buffers will be flushed to disk when
// more than autof bytes have been written to the file. At the first FlushBaskets
// TTree::Fill will replace fAutoFlush by the current value of fEntries.
//
// Calling this function with autof<0 is interesting when it is hard to estimate
// the size of one entry. This value is also independent of the Tree.
//
// When calling SetAutoFlush with no arguments, the
// default value is -30000000, ie that the first AutoFlush will be done when
// 30 MBytes of data are written to the file.
//
// CASE 3 : autof = 0
// ------------------
// The AutoFlush mechanism is disabled.
//
// Flushing the buffers at regular intervals optimize the location of
// consecutive entries on the disk.
Revision
30962 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Nov 3 15:21:40 2009 UTC (5 years, 2 months ago) by
pcanal
File length: 27348 byte(s)
Diff to
previous 30219
Clarify and correct when FillBuffer is actually executed.
Document when (and how) the learning phase is started and stopped.
Fix SetEntryRange when the Cache learning was manual.
With the previous implementation requesting the entry from a branch
that was not marked to be cached when the entry (number) was less
than fEntryNext resulted in the re-reading of any baskets between
this entry number and fEntryNext. (This would happen also if
the branch was being cached but for some reason some of its baskets
did not fit in the cache).
Revision
30219 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Sep 17 12:29:31 2009 UTC (5 years, 4 months ago) by
brun
File length: 25835 byte(s)
Diff to
previous 30208
Add new interface functions for the TreeCache:
virtual void AddBranchToCache(const char *bname, Bool_t subbranches = kFALSE);
virtual void AddBranchToCache(TBranch *branch, Bool_t subbranches = kFALSE);
virtual void PrintCacheStats(Option_t* option = "") const;
virtual void SetParallelUnzip(Bool_t opt=kTRUE);
Revision
30208 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Sep 16 17:30:17 2009 UTC (5 years, 4 months ago) by
brun
File length: 25831 byte(s)
Diff to
previous 30197
Add a direct interface from TTree to TTreeCache with the functions:
virtual void AddBranch2Cache(const char *bname, Bool_t subbranches = kFALSE);
virtual void AddBranch2Cache(TBranch *branch, Bool_t subbranches = kFALSE);
virtual void SetCacheEntryRange(Long64_t first, Long64_t last);
virtual void SetCacheLearnEntries(Int_t n=10);
virtual void StopCacheLearningPhase();
When adding all branches to the cache the learning phase is automatically
stopped.
Change the learning phase from 100 to 10 entries.
It still remains to change TTree::Process such that the cache entry range
is automatically set.
Revision
29125 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Jun 22 09:03:21 2009 UTC (5 years, 7 months ago) by
brun
File length: 18018 byte(s)
Diff to
previous 28797
From Gerri:
In SetEntryRange: re-initialize the cache parameters only if in learning phase; fixes a problem
making the cache de facto ineffective in PROOF
Revision
28787 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 3 00:54:10 2009 UTC (5 years, 7 months ago) by
pcanal
File length: 17848 byte(s)
Diff to
previous 25698
Make sure that the TTreeCache is not attempting to cache (wrongly) the content of branches that are in an auxiliary files.
Make sure that FillBuffer does it work when the learning phase is over even if the entry number is 'low'
Revision
23685 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 7 15:18:39 2008 UTC (6 years, 8 months ago) by
brun
File length: 17522 byte(s)
Diff to
previous 23554
Introducing a parallel unzipping algorithm for pre-fetched buffers. Since we already know what buffers are going to be read, we can decompress a few of them in advance in an additional thread and give the impression that the data decompression comes for free (we gain up to 30% in reading intensive jobs).
The size of this unzipping cache is 20% the size of the TTreeCache and can be modified with TTreeCache::SetUnzipBufferSize(Long64_t bufferSize). Theoretically, we only need one buffer in advance but in practice we might fall short if the unzipping cache is too small (synchronization costs).
This experimental feature is disabled by default, to activate it use the static function TTreeCache::SetParallelUnzip(TTreeCacheUnzip::EParUnzipMode option = TTreeCacheUnzip::kEnable). The possible values to pass are: TTreeCacheUnzip::kEnable to enable it,TTreeCacheUnzip::kDisable to disable it and TTreeCacheUnzip::kForce to force it. Since it will only work if you have more than one core, the TTreeCacheUnzip::kForce option is useful to run it even if you have only one cpu (to measure the overhead, for example).
Revision
20794 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 14 01:41:56 2007 UTC (7 years, 2 months ago) by
brun
Original Path:
trunk/tree/src/TTreeCache.cxx
File length: 16888 byte(s)
Diff to
previous 20378
From Gerri:
Following the implementation in TChain, I have succeded in enabling the TTree cache in Proof.
However, for that I need a small addition in TTreeCache to be able to change also the 'fOwner'
member, because on the workers we have to delete the previous tree, so the owner tree creating
the cache is gone after the first file.
In the attached patch I propose to add a second argument to TTreeCache::UpdateBranches, a boolean
indicating whether to set also the ownership or not. Default is not, so we do not break any existing
usage of the call.
With this, and other changes in TEventIter that I am finalizing, I get for h1analysis with http files
from my place and my single core machine
TChain 52s
Proof 1 worker 57-62s
i.e. a 10-20% overhead in Proof, which is what we expect without the local optimizations.
Revision
20280 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Oct 9 18:47:29 2007 UTC (7 years, 3 months ago) by
pcanal
Original Path:
trunk/tree/src/TTreeCache.cxx
File length: 16703 byte(s)
Diff to
previous 19826
From Leo:
Add the ability to manually enter the list of used branches.
root [0] f = TFile::Open("root://localhost://tmp/Event.root");
root [1] TTree *T = (TTree*)f->Get("T");
root [2] T->SetCacheSize(10000000);
root [4] TFile::GetFileReadCalls()
(Int_t)(7)
root [5] f->GetBytesRead()
(const Long64_t)638632
root [3] TTreeCache *tpf = (TTreeCache*) f->GetCacheRead();
root [6] tpf->AddBranch("fTracks", kTRUE);
root [7] tpf->StopLearningPhase();
root [8] T->Draw("fTracks.fPx", "fTracks.fPy");
<TCanvas::MakeDefCanvas>: created default TCanvas with name c1
root [9] TFile::GetFileReadCalls()
(Int_t)(42)
root [10] f->GetBytesRead()
(const Long64_t)359898352
Revision
15971 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Aug 10 10:21:25 2006 UTC (8 years, 5 months ago) by
brun
Original Path:
trunk/tree/src/TTreeCache.cxx
File length: 10974 byte(s)
Diff to
previous 15682
In TTreeCache::FillBuffer add a protection to prevent the cache to grow
for ever in case the estimation of fEntryNext is not accurate enough.
This case may happen when a TEventlist has been defined and the distribution
of entries in the list is far from being uniform.
Revision
15626 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 29 22:15:37 2006 UTC (8 years, 6 months ago) by
rdm
Original Path:
trunk/tree/src/TTreeCache.cxx
File length: 10619 byte(s)
Diff to
previous 15572
Several fixes in the new cache handling:
- TFile
- GetBytesWritten() includes also the number of bytes still in the
write cache
- New method FlushWriteCache()
- Call FlushWriteCache() also when closing files in raw mode
- Call FlushWriteCache() also in Flush()
- TFileCacheWrite:
- new method GetBytesInCache() returning bytes still in cache
used by TFile::GetBytesWritten()
- WriteBuffer() returns int iso bool, -1 write failure, 0 in case
recusively called and 1 in case copied in cache
- TFileCacheRead:
- ReadBuffer() returns int iso bool, -1 read failure, 0 in case not
in cache and 1 in case read from cache.
- TTreeCache:
- use int return type for ReadBuffer()
- TNetFile and TXNetFile:
- call FlushWriteCache() in their Flush().
Revision
15572 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Tue Jun 27 14:36:28 2006 UTC (8 years, 6 months ago) by
brun
Original Path:
trunk/tree/src/TTreeCache.cxx
File length: 10703 byte(s)
-Rename class TFilePrefetch -> TFileCacheRead
-Rename class TTreeFilePrefetch -> TTreeCache
-Add new class TFileCacheWrite that implements automatic caching
when writing network files. This class replaces the previous class TCache.
-old TCache class deleted.
-TFileCacheRead can read buffers from the wite cache in TFileCacheWrite.
-As a result TFile::UseCache is obsolete. The function is kept for
backward compatibility.
-Remove references to TCache from TXMLFile, TSQLFile, TwebFile, TNetFile
-remove pointer TFile::fCache
-Add pointers TFile::fCacheRead and TFile::fCacheWrite
-Remove members fMaxCacheSize and fPageSize from TChain
NOTE that it is recommended to
rm -f base/src/*.d base/src/*.o
rm -f net/src/*.d net/src/*.o
rm -f tree/src/*.d tree/src/*.o
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
