Log of /trunk/io/io/inc/TFileCacheRead.h
Parent Directory
Revision
48306 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jan 16 16:01:47 2013 UTC (2 years ago) by
pcanal
File length: 7289 byte(s)
Diff to
previous 45485
From Elvin:
There was a race condition between the destructor of the TTree object and the functionality to kill the asynchronous thread doing the prefetching.
In more detail: In the the TTree::~TTree line 789 the TFileCacheRead object of the current file is set to 0. All the logic to kill the async thread is done in the destructor of TFilePrefetch which in turn is called from the destructor of TFileCacheRead. In the same file two lines below the destructor of TFileCacheRead is called. And initially TFilePrefetch held a pointer to the file object in TFileCacheRead which now is 0. Therefore, during the destruction of the TFilePrefetch object we don't have any longer a valid TFile pointer. So, we can not wait for the ongoing requests to be satisfied. This was the reason of the crash. To fix this, I removed the killing of the async thread form the destructor of the TFilePrefetch method and I've put it in a separate method called WaitFinishPrefetch. In this way, we avoid the potential scenario of trying to wait for some request for a file while not having a valid pointer to that file.
Revision
44841 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jul 3 10:29:51 2012 UTC (2 years, 6 months ago) by
pcanal
File length: 6951 byte(s)
Diff to
previous 44785
Add an explicit 'Close' for the read cache so that we can insure that all the (concurrent) outstanding connection/use of the TFile are closed before closing the file
Revision
44785 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jun 27 20:38:24 2012 UTC (2 years, 6 months ago) by
pcanal
File length: 6900 byte(s)
Diff to
previous 44756
Add a non virtual SetEnablePrefetchingImpl to be called form the constructor and from the
virtual SetEnablePrefetching.
Make sure that even in the usual case (fEnablePrefetching == false), fAsyncReading is set.
Add documentation in the new functions.
Revision
44756 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Jun 26 21:14:16 2012 UTC (2 years, 6 months ago) by
pcanal
File length: 6770 byte(s)
Diff to
previous 43149
From Brian:
When hooking prefetching up into the CMSSW configuration system, I found the attached patch necessary.
Without it, if you change the prefetch settings after the cache is created, I/O operations cause a segfault.
Revision
43149 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Feb 27 21:09:07 2012 UTC (2 years, 10 months ago) by
pcanal
File length: 6810 byte(s)
Diff to
previous 42778
From Peter:
Index: io/io/inc/TFileCacheRead.h
Index: io/io/src/TFileCacheRead.cxx
* Enhance Print() function to report number of bytes read and read calls for current cache rather than totals on associated file.
* Distingurish counter for bytes read and read calls for learning phase.
Index: tree/tree/src/TBasket.cxx
* When calling cache ReadBuffer(), update number of read calls and bytes read.
Revision
42754 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jan 20 20:13:07 2012 UTC (3 years ago) by
pcanal
File length: 5526 byte(s)
Diff to
previous 39466
From Peter van Gemmeren:
Allow multiple read caches per file by adding a fCacheReadMap member and owner arguments to GetCacheRead() and SetCacheRead()
to TFile.
Index: io/io/inc/TFileCacheRead.h
Index: io/io/src/TFileCacheRead.cxx
* Adding new argument for constructor to specify TTree owner
* whitespace indents
Index: tree/tree/inc/TBasket.h
Index: tree/tree/src/TBasket.cxx
* Add optional tree owner argument to LoadBasketBuffers() to retrieve cache.
* In LoadBasketBuffers() and ReadBuffers() check the cache first than call TFile::ReadBuffer only if needed.
Index: tree/tree/inc/TChain.h
Index: tree/tree/src/TChain.cxx
* Override SetCacheSize() function to update TTree owner on the current cache.
* whitespace indents
Index: tree/tree/src/TTree.cxx
* Get TTreeCache for current TTree, by specifying owner in GetCacheRead() calls
* Always delete owned cache
Index: tree/tree/src/TTreeCache.cxx
* Adding new argument for constructor for TFileCacheRead to specify TTree owner
Index: tree/treeplayer/src/TTreePlayer.cxx
* Get TTreeCache for current TTree, by specifying owner in GetCacheRead() calls
Index: tree/tree/src/TTreeCloner.cxx
* Call TBasket::LoadBasketBuffers() with current TTree as owner of cache
Index: proof/proofplayer/src/TEventIter.cxx
* Get TTreeCache for current TTree, by specifying owner in GetCacheRead() and SetCacheRead() calls
Revision
39275 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu May 19 18:17:37 2011 UTC (3 years, 8 months ago) by
pcanal
File length: 5432 byte(s)
Diff to
previous 31503
From Elvin Alin Sindrilaru:
The prefetching mechanism uses two new classes (TFilePrefetch.h and
TFPBlock.h) to prefetch in advance a block of entries. There is a second
thread which takes care of actually transferring the blocks and making
them available to the main requesting thread. Therefore, the time spent
by the main thread waiting for the data before processing considerably
decreases. Besides the prefetching mechanisms there is also a local
caching option which can be enabled by the user. Both capabilities are
disabled by default and must be explicitly enabled by the user.
In order to enable the prefetching the user must define the environment
variable "TFile.AsyncPrefetching" as follows:
gEnv->SetValue("TFile.AsyncPrefetching", 1).
Only when the prefetching is enabled can the user set the local cache
directory in which the file transferred can be saved. For subsequent
reads of the same file the system will use the local copy of the file
from cache. To set up a local cache directory, a client can use the
following commands:
TString cachedir="file:/tmp/xcache/";
// or using xrootd on port 2000
// TString cachedir="root://localhost:2000//tmp/xrdcache1/";
gEnv->SetValue("Cache.Directory", cachedir.Data());
The "TFilePrefetch" class is responsible with actually reading and storing
the requests received from the main thread. It also creates the working
thread which will transfer all the information. Apart from managing the
block requests, it also deals with caching the blocks on the local machine
and retrieving them when necessary.
The "TFPBlock" class represents the encapsulation of a block request. It
contains the chunks to be prefetched and also serves as a container for
the information read.
In order to accommodate the new prefetching mechanism the following files
have suffered considerable modifications: TFileCacheRead.cxx/.h,
TTreeCache.cxx/.h and to a less extent: TXNetFile.cxx, TFile.h.
Basically in TFileCacheRead we've added the logic for dealing with the
second buffer that is prefetched. In TTreeCache during prefeching the
method FillBuffer is called after each read so that once the main thread
starts reading from the last available buffer, the second thread starts
prefetching the next block.
Revision
31457 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Nov 30 13:36:37 2009 UTC (5 years, 1 month ago) by
brun
File length: 4024 byte(s)
Diff to
previous 31077
From Fabrizio:
In TFileCacheRead add a new virtual function GetUnzipBuffer in order
to avoid InheritsFrom in TBasket::ReadBasketBuffers.
In TTreeCacheUnzip optimizations related to CPU cycles and memory.
Related fixes to TFileCacheRead. Also comment "if gDebug" statements.
In TXNetFile fixes related to the reporting of TTreePerfStats.
Revision
31077 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Nov 11 08:51:57 2009 UTC (5 years, 2 months ago) by
brun
File length: 3903 byte(s)
Diff to
previous 29877
Implement the cache functions AddBranch and SetSkipZip in TFileCacheRead too.
This speeds up TBranch::GetBasket where it is not necessary to test if the file cache
is a TTreeCache.
Revision
26028 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 30 20:09:44 2008 UTC (6 years, 2 months ago) by
brun
File length: 3565 byte(s)
Diff to
previous 23685
From Lorenzo:
this patch fixes some not-consistent usage of TMath::Sort. If it is not
applied, the compilation will fail when I will change
Tmath::Sort<Element,Index> (Long64 n, Element x, Index w)
in
Tmath::Sort<Element,Index> (Index n, Element x, Index w)
to avoid some possible conversion errors from Long64 to Index
Revision
23685 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed May 7 15:18:39 2008 UTC (6 years, 8 months ago) by
brun
File length: 3565 byte(s)
Diff to
previous 23122
Introducing a parallel unzipping algorithm for pre-fetched buffers. Since we already know what buffers are going to be read, we can decompress a few of them in advance in an additional thread and give the impression that the data decompression comes for free (we gain up to 30% in reading intensive jobs).
The size of this unzipping cache is 20% the size of the TTreeCache and can be modified with TTreeCache::SetUnzipBufferSize(Long64_t bufferSize). Theoretically, we only need one buffer in advance but in practice we might fall short if the unzipping cache is too small (synchronization costs).
This experimental feature is disabled by default, to activate it use the static function TTreeCache::SetParallelUnzip(TTreeCacheUnzip::EParUnzipMode option = TTreeCacheUnzip::kEnable). The possible values to pass are: TTreeCacheUnzip::kEnable to enable it,TTreeCacheUnzip::kDisable to disable it and TTreeCacheUnzip::kForce to force it. Since it will only work if you have more than one core, the TTreeCacheUnzip::kForce option is useful to run it even if you have only one cpu (to measure the overhead, for example).
Revision
20488 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Oct 25 13:22:56 2007 UTC (7 years, 3 months ago) by
brun
Original Path:
trunk/io/inc/TFileCacheRead.h
File length: 3406 byte(s)
Diff to
previous 19826
From Fabrizio & Gerri:
Patch enabling asynchronous reading via XROOTD; a new method TFile::ReadBufferAsync
is introduced which gets appropriately overritten in TXNetFile .
The new feature can be enabled / disabled via the env 'TFile.AsyncReading' in .rootrc,
the default being ON.
The patch also cleans up the netx section in the system.rootrc, by removing
very old unused variables and commenting and setting appropriate default
values for the others (no additions, except for the comments).
Revision
15626 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Jun 29 22:15:37 2006 UTC (8 years, 6 months ago) by
rdm
Original Path:
trunk/io/inc/TFileCacheRead.h
File length: 3154 byte(s)
Diff to
previous 15572
Several fixes in the new cache handling:
- TFile
- GetBytesWritten() includes also the number of bytes still in the
write cache
- New method FlushWriteCache()
- Call FlushWriteCache() also when closing files in raw mode
- Call FlushWriteCache() also in Flush()
- TFileCacheWrite:
- new method GetBytesInCache() returning bytes still in cache
used by TFile::GetBytesWritten()
- WriteBuffer() returns int iso bool, -1 write failure, 0 in case
recusively called and 1 in case copied in cache
- TFileCacheRead:
- ReadBuffer() returns int iso bool, -1 read failure, 0 in case not
in cache and 1 in case read from cache.
- TTreeCache:
- use int return type for ReadBuffer()
- TNetFile and TXNetFile:
- call FlushWriteCache() in their Flush().
Revision
15572 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Added
Tue Jun 27 14:36:28 2006 UTC (8 years, 6 months ago) by
brun
Original Path:
trunk/io/inc/TFileCacheRead.h
File length: 3154 byte(s)
-Rename class TFilePrefetch -> TFileCacheRead
-Rename class TTreeFilePrefetch -> TTreeCache
-Add new class TFileCacheWrite that implements automatic caching
when writing network files. This class replaces the previous class TCache.
-old TCache class deleted.
-TFileCacheRead can read buffers from the wite cache in TFileCacheWrite.
-As a result TFile::UseCache is obsolete. The function is kept for
backward compatibility.
-Remove references to TCache from TXMLFile, TSQLFile, TwebFile, TNetFile
-remove pointer TFile::fCache
-Add pointers TFile::fCacheRead and TFile::fCacheWrite
-Remove members fMaxCacheSize and fPageSize from TChain
NOTE that it is recommended to
rm -f base/src/*.d base/src/*.o
rm -f net/src/*.d net/src/*.o
rm -f tree/src/*.d tree/src/*.o
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
