Specialization of TTreeCache for parallel Unzipping.

Fabrizio Furano (CERN) Aug 2009 Core TTree-related code borrowed from the previous version by Leandro Franco and Rene Brun

Parallel Unzipping

TTreeCache has been specialised in order to let additional threads free to unzip in advance its content. In this implementation we support up to 10 threads, but right now it makes more sense to limit their number to 1-2

The application reading data is carefully synchronized, in order to:

if the block it wants is not unzipped, it self-unzips it without waiting
if the block is being unzipped in parallel, it waits only for that unzip to finish
if the block has already been unzipped, it takes it

This is supposed to cancel a part of the unzipping latency, at the expenses of cpu time.

The default parameters are the same of the prev version, i.e. 20% of the TTreeCache cache size. To change it use TTreeCache::SetUnzipBufferSize(Long64_t bufferSize) where bufferSize must be passed in bytes.

Definition at line 41 of file TTreeCacheUnzip.h.

Classes
struct	UnzipState

Public Types
enum	EParUnzipMode { kEnable, kDisable, kForce }

enum	EUnzipState { kUntouched, kProgress, kFinished }

Public Types inherited from TTreeCache
enum	EPrefillType { kNoPrefill, kAllBranches }

Public Types inherited from TObject
enum	{ kIsOnHeap = 0x01000000, kNotDeleted = 0x02000000, kZombie = 0x04000000, kInconsistent = 0x08000000, kBitMask = 0x00ffffff }

enum	{ kSingleKey = BIT(0), kOverwrite = BIT(1), kWriteDelete = BIT(2) }

enum	EDeprecatedStatusBits { kObjInCanvas = BIT(3) }

enum	EStatusBits { kCanDelete = BIT(0), kMustCleanup = BIT(3), kIsReferenced = BIT(4), kHasUUID = BIT(5), kCannotPick = BIT(6), kNoContextMenu = BIT(8), kInvalidObject = BIT(13) }

Public Member Functions
	TTreeCacheUnzip ()

	TTreeCacheUnzip (TTree *tree, Int_t buffersize=0)
	Constructor. More...

virtual	~TTreeCacheUnzip ()
	Destructor. (in general called by the TFile destructor) More...

virtual Int_t	AddBranch (TBranch *b, Bool_t subbranches=kFALSE)
	Add a branch to the list of branches to be stored in the cache this function is called by TBranch::GetBasket Returns: More...

virtual Int_t	AddBranch (const char *branch, Bool_t subbranches=kFALSE)
	Add a branch to the list of branches to be stored in the cache this function is called by TBranch::GetBasket Returns: More...

Int_t	CreateTasks ()
	We create a TTaskGroup and asynchronously maps each group of baskets(> 100 kB in total) to a task. More...

Bool_t	FillBuffer ()
	Fill the cache buffer with the branches in the cache. More...

Int_t	GetNFound ()

Int_t	GetNMissed ()

Int_t	GetNUnzip ()

Int_t	GetRecordHeader (char *buf, Int_t maxbytes, Int_t &nbytes, Int_t &objlen, Int_t &keylen)
	Read the logical record header from the buffer buf. More...

virtual Int_t	GetUnzipBuffer (char *buf, Long64_t pos, Int_t len, Bool_t free)
	We try to read a buffer that has already been unzipped Returns -1 in case of read failure, 0 in case it's not in the cache and n>0 in case read from cache (number of bytes copied). More...

Int_t	GetUnzipGroupSize ()

void	Print (Option_t *option="") const
	Print cache statistics. More...

virtual Int_t	ReadBufferExt (char *buf, Long64_t pos, Int_t len, Int_t &loc)

virtual void	ResetCache ()
	This will delete the list of buffers that are in the unzipping cache and will reset certain values in the cache. More...

virtual Int_t	SetBufferSize (Int_t buffersize)
	Change the underlying buffer size of the cache. More...

void	SetEntryRange (Long64_t emin, Long64_t emax)
	Set the minimum and maximum entry number to be processed this information helps to optimize the number of baskets to read when prefetching the branch buffers. More...

void	SetUnzipBufferSize (Long64_t bufferSize)
	Sets the size for the unzipping cache... More...

void	SetUnzipGroupSize (Int_t groupSize)

virtual void	StopLearningPhase ()
	It's the same as TTreeCache::StopLearningPhase but we guarantee that we start the unzipping just after getting the buffers. More...

Int_t	UnzipBuffer (char *dest, char src)
	Unzips a ROOT specific buffer... More...

Int_t	UnzipCache (Int_t index)
	This inflates a basket in the cache. More...

void	UpdateBranches (TTree *tree)
	update pointer to current Tree and recompute pointers to the branches in the cache More...

Public Member Functions inherited from TTreeCache
	TTreeCache ()
	Default Constructor. More...

	TTreeCache (TTree *tree, Int_t buffersize=0)
	Constructor. More...

virtual	~TTreeCache ()
	Destructor. (in general called by the TFile destructor) More...

virtual void	Disable ()

virtual Int_t	DropBranch (TBranch *b, Bool_t subbranches=kFALSE)
	Remove a branch to the list of branches to be stored in the cache this function is called by TBranch::GetBasket. More...

virtual Int_t	DropBranch (const char *branch, Bool_t subbranches=kFALSE)
	Remove a branch to the list of branches to be stored in the cache this is to be used by user (thats why we pass the name of the branch). More...

virtual void	Enable ()

const TObjArray *	GetCachedBranches () const

EPrefillType	GetConfiguredPrefillType () const
	Return the desired prefill type from the environment or resource variable. More...

Double_t	GetEfficiency () const
	Give the total efficiency of the primary cache... More...

Double_t	GetEfficiencyRel () const
	This will indicate a sort of relative efficiency... More...

virtual Int_t	GetEntryMax () const

virtual Int_t	GetEntryMin () const

virtual EPrefillType	GetLearnPrefill () const

Double_t	GetMissEfficiency () const
	The total efficiency of the 'miss cache' - defined as the ratio of blocks found in the cache versus the number of blocks prefetched. More...

Double_t	GetMissEfficiencyRel () const
	Relative efficiency of the 'miss cache' - ratio of the reads found in cache to the number of reads so far. More...

Bool_t	GetOptimizeMisses () const

TTree *	GetTree () const

Bool_t	IsAutoCreated () const

virtual Bool_t	IsEnabled () const

virtual Bool_t	IsLearning () const

virtual void	LearnPrefill ()
	Perform an initial prefetch, attempting to read as much of the learning phase baskets for all branches at once. More...

virtual Int_t	ReadBuffer (char *buf, Long64_t pos, Int_t len)
	Read buffer at position pos if the request is in the list of prefetched blocks read from fBuffer. More...

virtual Int_t	ReadBufferNormal (char *buf, Long64_t pos, Int_t len)
	Old method ReadBuffer before the addition of the prefetch mechanism. More...

virtual Int_t	ReadBufferPrefetch (char *buf, Long64_t pos, Int_t len)
	Used to read a chunk from a block previously fetched. More...

void	ResetMissCache ()
	Reset all the miss cache training. More...

void	SetAutoCreated (Bool_t val)

virtual void	SetFile (TFile *file, TFile::ECacheAction action=TFile::kDisconnect)
	Overload to make sure that the object specific. More...

virtual void	SetLearnPrefill (EPrefillType type=kNoPrefill)
	Set whether the learning period is started with a prefilling of the cache and which type of prefilling is used. More...

void	SetOptimizeMisses (Bool_t opt)
	Start of methods for the miss cache. More...

void	StartLearningPhase ()
	The name should be enough to explain the method. More...

Public Member Functions inherited from TFileCacheRead
	TFileCacheRead ()
	Default Constructor. More...

	TFileCacheRead (TFile file, Int_t buffersize, TObject tree=0)
	Creates a TFileCacheRead data structure. More...

virtual	~TFileCacheRead ()
	Destructor. More...

virtual void	AddNoCacheBytesRead (Long64_t len)

virtual void	AddNoCacheReadCalls (Int_t reads)

virtual void	Close (Option_t *option="")
	Close out any threads or asynchronous fetches used by the underlying implementation. More...

virtual Int_t	GetBufferSize () const

virtual Long64_t	GetBytesRead () const

virtual Long64_t	GetBytesReadExtra () const

TFile *	GetFile () const

virtual Long64_t	GetNoCacheBytesRead () const

virtual Int_t	GetNoCacheReadCalls () const

Int_t	GetNseek () const

Int_t	GetNtot () const

Long64_t	GetPrefetchedBlocks () const

virtual TFilePrefetch *	GetPrefetchObj ()

virtual Int_t	GetReadCalls () const

virtual Bool_t	IsAsyncReading () const

virtual Bool_t	IsEnablePrefetching () const

virtual void	Prefetch (Long64_t pos, Int_t len)
	Add block of length len at position pos in the list of blocks to be prefetched. More...

virtual Int_t	ReadBufferExtNormal (char *buf, Long64_t pos, Int_t len, Int_t &loc)
	Base function for ReadBuffer. More...

virtual Int_t	ReadBufferExtPrefetch (char *buf, Long64_t pos, Int_t len, Int_t &loc)
	prefetch the first block More...

virtual void	SecondPrefetch (Long64_t, Int_t)

virtual void	SecondSort ()
	Sort buffers to be prefetched in increasing order of positions. More...

virtual void	SetEnablePrefetching (Bool_t setPrefetching=kFALSE)
	Set the prefetching mode of this file. More...

virtual void	SetSkipZip (Bool_t=kTRUE)

virtual void	Sort ()
	Sort buffers to be prefetched in increasing order of positions. More...

virtual void	WaitFinishPrefetch ()

Public Member Functions inherited from TObject
	TObject ()
	TObject constructor. More...

	TObject (const TObject &object)
	TObject copy ctor. More...

virtual	~TObject ()
	TObject destructor. More...

void	AbstractMethod (const char *method) const
	Use this method to implement an "abstract" method that you don't want to leave purely abstract. More...

virtual void	AppendPad (Option_t *option="")
	Append graphics object to current pad. More...

virtual void	Browse (TBrowser *b)
	Browse object. May be overridden for another default action. More...

ULong_t	CheckedHash ()
	Checked and record whether for this class has a consistent Hash/RecursiveRemove setup (*) and then return the regular Hash value for this object. More...

virtual const char *	ClassName () const
	Returns name of class to which the object belongs. More...

virtual void	Clear (Option_t *="")

virtual TObject *	Clone (const char *newname="") const
	Make a clone of an object using the Streamer facility. More...

virtual Int_t	Compare (const TObject *obj) const
	Compare abstract method. More...

virtual void	Copy (TObject &object) const
	Copy this to obj. More...

virtual void	Delete (Option_t *option="")
	Delete this object. More...

virtual Int_t	DistancetoPrimitive (Int_t px, Int_t py)
	Computes distance from point (px,py) to the object. More...

virtual void	Draw (Option_t *option="")
	Default Draw method for all objects. More...

virtual void	DrawClass () const
	Draw class inheritance tree of the class to which this object belongs. More...

virtual TObject *	DrawClone (Option_t *option="") const
	Draw a clone of this object in the current selected pad for instance with: `gROOT->SetSelectedPad(gPad)`. More...

virtual void	Dump () const
	Dump contents of object on stdout. More...

virtual void	Error (const char method, const char msgfmt,...) const
	Issue error message. More...

virtual void	Execute (const char method, const char params, Int_t *error=0)
	Execute method on this object with the given parameter string, e.g. More...

virtual void	Execute (TMethod method, TObjArray params, Int_t *error=0)
	Execute method on this object with parameters stored in the TObjArray. More...

virtual void	ExecuteEvent (Int_t event, Int_t px, Int_t py)
	Execute action corresponding to an event at (px,py). More...

virtual void	Fatal (const char method, const char msgfmt,...) const
	Issue fatal error message. More...

virtual TObject *	FindObject (const char *name) const
	Must be redefined in derived classes. More...

virtual TObject *	FindObject (const TObject *obj) const
	Must be redefined in derived classes. More...

virtual Option_t *	GetDrawOption () const
	Get option used by the graphics system to draw this object. More...

virtual const char *	GetIconName () const
	Returns mime type name of object. More...

virtual const char *	GetName () const
	Returns name of object. More...

virtual char *	GetObjectInfo (Int_t px, Int_t py) const
	Returns string containing info about the object at position (px,py). More...

virtual Option_t *	GetOption () const

virtual const char *	GetTitle () const
	Returns title of object. More...

virtual UInt_t	GetUniqueID () const
	Return the unique object id. More...

virtual Bool_t	HandleTimer (TTimer *timer)
	Execute action in response of a timer timing out. More...

virtual ULong_t	Hash () const
	Return hash value for this object. More...

Bool_t	HasInconsistentHash () const
	Return true is the type of this object is known to have an inconsistent setup for Hash and RecursiveRemove (i.e. More...

virtual void	Info (const char method, const char msgfmt,...) const
	Issue info message. More...

virtual Bool_t	InheritsFrom (const char *classname) const
	Returns kTRUE if object inherits from class "classname". More...

virtual Bool_t	InheritsFrom (const TClass *cl) const
	Returns kTRUE if object inherits from TClass cl. More...

virtual void	Inspect () const
	Dump contents of this object in a graphics canvas. More...

void	InvertBit (UInt_t f)

virtual Bool_t	IsEqual (const TObject *obj) const
	Default equal comparison (objects are equal if they have the same address in memory). More...

virtual Bool_t	IsFolder () const
	Returns kTRUE in case object contains browsable objects (like containers or lists of other objects). More...

R__ALWAYS_INLINE Bool_t	IsOnHeap () const

virtual Bool_t	IsSortable () const

R__ALWAYS_INLINE Bool_t	IsZombie () const

virtual void	ls (Option_t *option="") const
	The ls function lists the contents of a class on stdout. More...

void	MayNotUse (const char *method) const
	Use this method to signal that a method (defined in a base class) may not be called in a derived class (in principle against good design since a child class should not provide less functionality than its parent, however, sometimes it is necessary). More...

virtual Bool_t	Notify ()
	This method must be overridden to handle object notification. More...

void	Obsolete (const char method, const char asOfVers, const char *removedFromVers) const
	Use this method to declare a method obsolete. More...

void	operator delete (void *ptr)
	Operator delete. More...

void	operator delete[] (void *ptr)
	Operator delete []. More...

void *	operator new (size_t sz)

void *	operator new (size_t sz, void *vp)

void *	operator new[] (size_t sz)

void *	operator new[] (size_t sz, void *vp)

TObject &	operator= (const TObject &rhs)
	TObject assignment operator. More...

virtual void	Paint (Option_t *option="")
	This method must be overridden if a class wants to paint itself. More...

virtual void	Pop ()
	Pop on object drawn in a pad to the top of the display list. More...

virtual Int_t	Read (const char *name)
	Read contents of object with specified name from the current directory. More...

virtual void	RecursiveRemove (TObject *obj)
	Recursively remove this object from a list. More...

void	ResetBit (UInt_t f)

virtual void	SaveAs (const char filename="", Option_t option="") const
	Save this object in the file specified by filename. More...

virtual void	SavePrimitive (std::ostream &out, Option_t *option="")
	Save a primitive as a C++ statement(s) on output stream "out". More...

void	SetBit (UInt_t f, Bool_t set)
	Set or unset the user status bits as specified in f. More...

void	SetBit (UInt_t f)

virtual void	SetDrawOption (Option_t *option="")
	Set drawing option for object. More...

virtual void	SetUniqueID (UInt_t uid)
	Set the unique object id. More...

virtual void	SysError (const char method, const char msgfmt,...) const
	Issue system error message. More...

R__ALWAYS_INLINE Bool_t	TestBit (UInt_t f) const

Int_t	TestBits (UInt_t f) const

virtual void	UseCurrentStyle ()
	Set current style settings in this object This function is called when either TCanvas::UseCurrentStyle or TROOT::ForceStyle have been invoked. More...

virtual void	Warning (const char method, const char msgfmt,...) const
	Issue warning message. More...

virtual Int_t	Write (const char *name=0, Int_t option=0, Int_t bufsize=0)
	Write this object to the current directory. More...

virtual Int_t	Write (const char *name=0, Int_t option=0, Int_t bufsize=0) const
	Write this object to the current directory. More...

Static Public Member Functions
static EParUnzipMode	GetParallelUnzip ()
	Static function that returns the parallel option (to indicate an additional thread) More...

static Bool_t	IsParallelUnzip ()
	Static function that tells wether the multithreading unzipping is activated. More...

static Int_t	SetParallelUnzip (TTreeCacheUnzip::EParUnzipMode option=TTreeCacheUnzip::kEnable)
	Static function that (de)activates multithreading unzipping. More...

static void	SetUnzipRelBufferSize (Float_t relbufferSize)
	static function: Sets the unzip relatibe buffer size More...

Static Public Member Functions inherited from TTreeCache
static Int_t	GetLearnEntries ()
	Static function returning the number of entries used to train the cache see SetLearnEntries. More...

static void	SetLearnEntries (Int_t n=10)
	Static function to set the number of entries to be used in learning mode The default value for n is 10. More...

Static Public Member Functions inherited from TObject
static Long_t	GetDtorOnly ()
	Return destructor only flag. More...

static Bool_t	GetObjectStat ()
	Get status of object stat flag. More...

static void	SetDtorOnly (void *obj)
	Set destructor only flag. More...

static void	SetObjectStat (Bool_t stat)
	Turn on/off tracking of objects in the TObjectTable. More...

Protected Types
typedef struct UnzipState	UnzipState_t

Protected Attributes
Bool_t	fAsyncReading

Int_t	fCycle

Bool_t	fEmpty

TMutex *	fIOMutex

Int_t	fNFound
	! number of blocks that were found in the cache More...

Int_t	fNMissed
	! number of blocks that were not found in the cache and were unzipped More...

Int_t	fNseekMax
	! fNseek can change so we need to know its max size More...

Int_t	fNStalls
	! number of hits which caused a stall More...

Int_t	fNUnzip
	! number of blocks that were unzipped More...

Bool_t	fParallel
	Indicate if we want to activate the parallelism (for this instance) More...

Long64_t	fUnzipBufferSize
	! Max Size for the ready unzipped blocks (default is 2*fBufferSize) More...

Int_t	fUnzipGroupSize
	! Min accumulated size of a group of baskets ready to be unzipped by a IMT task More...

UnzipState_t	fUnzipState

std::unique_ptr< ROOT::Experimental::TTaskGroup >	fUnzipTaskGroup

Protected Attributes inherited from TTreeCache
Bool_t	fAutoCreated {kFALSE}
	! true if cache was automatically created More...

TObjArray *	fBranches {nullptr}
	! List of branches to be stored in the cache More...

TList *	fBrNames {nullptr}
	! list of branch names in the cache More...

Long64_t	fCurrentClusterStart {-1}
	! Start of the cluster(s) where the current content was picked out More...

Bool_t	fEnabled {kTRUE}
	! cache enabled for cached reading More...

Long64_t	fEntryCurrent {-1}
	! current lowest entry number in the cache More...

Long64_t	fEntryMax {1}
	! last entry in the cache More...

Long64_t	fEntryMin {0}
	! first entry in the cache More...

Long64_t	fEntryNext {-1}
	! next entry number where cache must be filled More...

Int_t	fFillTimes {0}
	! how many times we can fill the current buffer More...

Bool_t	fFirstBuffer {kTRUE}
	! true if first buffer is used for prefetching More...

Long64_t	fFirstEntry {-1}
	! save the value of the first entry More...

Long64_t	fFirstMiss {-1}
	! set to the event # of the first miss. More...

Bool_t	fFirstTime {kTRUE}
	! save the fact that we processes the first entry More...

Bool_t	fIsLearning {kTRUE}
	! true if cache is in learning mode More...

Bool_t	fIsManual {kFALSE}
	! true if cache is StopLearningPhase was used More...

Long64_t	fLastMiss {-1}
	! set to the event # of the last miss. More...

std::unique_ptr< MissCache >	fMissCache
	! Cache contents for misses More...

Int_t	fNbranches {0}
	! Number of branches in the cache More...

Long64_t	fNextClusterStart {-1}
	! End+1 of the cluster(s) where the current content was picked out More...

Int_t	fNMissReadMiss {0}
	Number of blocks read and not found in either cache. More...

Int_t	fNMissReadOk {0}
	Number of blocks read, not found in the primary cache, and found in the secondary cache. More...

Int_t	fNMissReadPref {0}
	Number of blocks read into the secondary ("miss") cache. More...

Int_t	fNReadMiss {0}
	Number of blocks read and not found in the cache. More...

Int_t	fNReadOk {0}
	Number of blocks read and found in the cache. More...

Int_t	fNReadPref {0}
	Number of blocks that were prefetched. More...

Bool_t	fOneTime {kFALSE}
	! used in the learning phase More...

Bool_t	fOptimizeMisses {kFALSE}
	! true if we should optimize cache misses. More...

EPrefillType	fPrefillType
	Whether a pre-filling is enabled (and if applicable which type) More...

Bool_t	fReadDirectionSet {kFALSE}
	! read direction established More...

Bool_t	fReverseRead {kFALSE}
	! reading in reverse mode More...

TTree *	fTree {nullptr}
	! pointer to the current Tree More...

Protected Attributes inherited from TFileCacheRead
Bool_t	fAsyncReading

Bool_t	fBIsSorted

Bool_t	fBIsTransferred

Int_t *	fBLen
	[fBNb] More...

Int_t	fBNb

Int_t	fBNseek

Int_t	fBNtot

Long64_t *	fBPos
	[fBNb] More...

Long64_t *	fBSeek
	[fBNseek] More...

Int_t *	fBSeekIndex
	[fBNseek] More...

Int_t *	fBSeekLen
	[fBNseek] More...

Int_t *	fBSeekPos
	[fBNseek] More...

Int_t	fBSeekSize

Long64_t *	fBSeekSort
	[fBNseek] More...

Int_t *	fBSeekSortLen
	[fBNseek] More...

char *	fBuffer
	[fBufferSize] buffer of contiguous prefetched blocks More...

Int_t	fBufferLen
	Current buffer length (<= fBufferSize) More...

Int_t	fBufferSize
	Allocated size of fBuffer (at a given time) More...

Int_t	fBufferSizeMin
	Original size of fBuffer. More...

Long64_t	fBytesRead
	Number of bytes read for this cache. More...

Long64_t	fBytesReadExtra
	Number of extra bytes (overhead) read by the readahead buffer. More...

Bool_t	fEnablePrefetching
	reading by prefetching asynchronously More...

TFile *	fFile
	Pointer to file. More...

Bool_t	fIsSorted
	True if fSeek array is sorted. More...

Bool_t	fIsTransferred
	True when fBuffer contains something valid. More...

Int_t *	fLen
	[fNb] Length of long buffers More...

Int_t	fNb
	Number of long buffers. More...

Long64_t	fNoCacheBytesRead
	Number of bytes read by basket to fill cached tree. More...

Int_t	fNoCacheReadCalls
	Number of read calls by basket to fill cached tree. More...

Int_t	fNseek
	Number of blocks to be prefetched. More...

Int_t	fNtot
	Total size of prefetched blocks. More...

Long64_t *	fPos
	[fNb] start of long buffers More...

TFilePrefetch *	fPrefetch
	! Object that does the asynchronous reading in another thread More...

Long64_t	fPrefetchedBlocks
	Number of blocks prefetched. More...

Int_t	fReadCalls
	Number of read calls for this cache. More...

Long64_t *	fSeek
	[fNseek] Position on file of buffers to be prefetched More...

Int_t *	fSeekIndex
	[fNseek] sorted index table of fSeek More...

Int_t *	fSeekLen
	[fNseek] Length of buffers to be prefetched More...

Int_t *	fSeekPos
	[fNseek] Position of sorted blocks in fBuffer More...

Int_t	fSeekSize
	Allocated size of fSeek. More...

Long64_t *	fSeekSort
	[fNseek] Position on file of buffers to be prefetched (sorted) More...

Int_t *	fSeekSortLen
	[fNseek] Length of buffers to be prefetched (sorted) More...

Static Protected Attributes
static TTreeCacheUnzip::EParUnzipMode	fgParallel = TTreeCacheUnzip::kDisable
	Indicate if we want to activate the parallelism. More...

static Double_t	fgRelBuffSize = .5
	This is the percentage of the TTreeCacheUnzip that will be used. More...

Static Protected Attributes inherited from TTreeCache
static Int_t	fgLearnEntries = 100
	number of entries used for learning mode More...

Private Member Functions
	TTreeCacheUnzip (const TTreeCacheUnzip &)

void	Init ()
	Initialization procedure common to all the constructors. More...

TTreeCacheUnzip &	operator= (const TTreeCacheUnzip &)

Private Attributes
char *	fCompBuffer

Int_t	fCompBufferSize

Additional Inherited Members
Protected Member Functions inherited from TFileCacheRead
void	SetEnablePrefetchingImpl (Bool_t setPrefetching=kFALSE)
	TFileCacheRead implementation of SetEnablePrefetching. More...

Protected Member Functions inherited from TObject
virtual void	DoError (int level, const char location, const char fmt, va_list va) const
	Interface to ErrorHandler (protected). More...

void	MakeZombie ()

#include <TTreeCacheUnzip.h>

Inheritance diagram for TTreeCacheUnzip:

[legend]

Member Typedef Documentation

◆ UnzipState_t

typedef struct UnzipState TTreeCacheUnzip::UnzipState_t

protected

Definition at line 83 of file TTreeCacheUnzip.h.

Member Enumeration Documentation

◆ EParUnzipMode

enum TTreeCacheUnzip::EParUnzipMode

Enumerator
kEnable
kDisable
kForce

Definition at line 46 of file TTreeCacheUnzip.h.

◆ EUnzipState

enum TTreeCacheUnzip::EUnzipState

Enumerator
kUntouched
kProgress
kFinished

Definition at line 49 of file TTreeCacheUnzip.h.

Constructor & Destructor Documentation

◆ TTreeCacheUnzip() [1/3]

TTreeCacheUnzip::TTreeCacheUnzip ( const TTreeCacheUnzip & )

private

◆ TTreeCacheUnzip() [2/3]

TTreeCacheUnzip::TTreeCacheUnzip ( )

Definition at line 178 of file TTreeCacheUnzip.cxx.

◆ TTreeCacheUnzip() [3/3]

TTreeCacheUnzip::TTreeCacheUnzip	(	TTree *	tree,
		Int_t	buffersize = `0`
	)

Constructor.

Definition at line 197 of file TTreeCacheUnzip.cxx.

◆ ~TTreeCacheUnzip()

TTreeCacheUnzip::~TTreeCacheUnzip ( )

virtual

Destructor. (in general called by the TFile destructor)

Definition at line 254 of file TTreeCacheUnzip.cxx.

Member Function Documentation

◆ AddBranch() [1/2]

Int_t TTreeCacheUnzip::AddBranch	(	TBranch *	b,
		Bool_t	subbranches = `kFALSE`
	)

virtual

Add a branch to the list of branches to be stored in the cache this function is called by TBranch::GetBasket Returns:

0 branch added or already included
-1 on error

Reimplemented from TTreeCache.

Definition at line 268 of file TTreeCacheUnzip.cxx.

◆ AddBranch() [2/2]

Int_t TTreeCacheUnzip::AddBranch	(	const char *	branch,
		Bool_t	subbranches = `kFALSE`
	)

virtual

Add a branch to the list of branches to be stored in the cache this function is called by TBranch::GetBasket Returns:

0 branch added or already included
-1 on error

Reimplemented from TTreeCache.

Definition at line 280 of file TTreeCacheUnzip.cxx.

◆ CreateTasks()

Int_t TTreeCacheUnzip::CreateTasks ( )

We create a TTaskGroup and asynchronously maps each group of baskets(> 100 kB in total) to a task.

In TTaskGroup, we use TThreadExecutor to do the actually work of unzipping a group of basket. The purpose of creating TTaskGroup is to avoid competing with main thread.

Definition at line 631 of file TTreeCacheUnzip.cxx.

◆ FillBuffer()

Bool_t TTreeCacheUnzip::FillBuffer ( )

virtual

Fill the cache buffer with the branches in the cache.

Reimplemented from TTreeCache.

Definition at line 287 of file TTreeCacheUnzip.cxx.

◆ GetNFound()

Int_t TTreeCacheUnzip::GetNFound ( )

inline

Definition at line 160 of file TTreeCacheUnzip.h.

◆ GetNMissed()

Int_t TTreeCacheUnzip::GetNMissed ( )

inline

Definition at line 159 of file TTreeCacheUnzip.h.

◆ GetNUnzip()

Int_t TTreeCacheUnzip::GetNUnzip ( )

inline

Definition at line 158 of file TTreeCacheUnzip.h.

◆ GetParallelUnzip()

TTreeCacheUnzip::EParUnzipMode TTreeCacheUnzip::GetParallelUnzip ( )

static

Static function that returns the parallel option (to indicate an additional thread)

Definition at line 427 of file TTreeCacheUnzip.cxx.

◆ GetRecordHeader()

Int_t TTreeCacheUnzip::GetRecordHeader	(	char *	buf,
		Int_t	maxbytes,
		Int_t &	nbytes,
		Int_t &	objlen,
		Int_t &	keylen
	)

Read the logical record header from the buffer buf.

That must be the pointer tho the header part not the object by itself and must contain data of at least maxbytes Returns nread;

In output arguments:

nbytes : number of bytes in record if negative, this is a deleted record if 0, cannot read record, wrong value of argument first
objlen : uncompressed object size
keylen : length of logical record header

Note that the arguments objlen and keylen are returned only if maxbytes >=16 Note: This was adapted from TFile... so some things dont apply

Definition at line 489 of file TTreeCacheUnzip.cxx.

◆ GetUnzipBuffer()

Int_t TTreeCacheUnzip::GetUnzipBuffer	(	char **	buf,
		Long64_t	pos,
		Int_t	len,
		Bool_t *	free
	)

virtual

We try to read a buffer that has already been unzipped Returns -1 in case of read failure, 0 in case it's not in the cache and n>0 in case read from cache (number of bytes copied).

pos and len are the original values as were passed to ReadBuffer but instead we will return the inflated buffer. Note!! : If *buf == 0 we will allocate the buffer and it will be the responsability of the caller to free it... it is useful for example to pass it to the creator of TBuffer

Reimplemented from TFileCacheRead.

Definition at line 686 of file TTreeCacheUnzip.cxx.

◆ GetUnzipGroupSize()

Int_t TTreeCacheUnzip::GetUnzipGroupSize ( )

inline

Definition at line 148 of file TTreeCacheUnzip.h.

◆ Init()

void TTreeCacheUnzip::Init ( void )

private

Initialization procedure common to all the constructors.

Definition at line 215 of file TTreeCacheUnzip.cxx.

◆ IsParallelUnzip()

Bool_t TTreeCacheUnzip::IsParallelUnzip ( )

static

Static function that tells wether the multithreading unzipping is activated.

Definition at line 435 of file TTreeCacheUnzip.cxx.

◆ operator=()

TTreeCacheUnzip& TTreeCacheUnzip::operator= ( const TTreeCacheUnzip & )

private

◆ Print()

void TTreeCacheUnzip::Print ( Option_t * option = "" ) const

virtual

Print cache statistics.

Like:

******TreeCache statistics for file: cms2.root ******
Number of branches in the cache ...: 1093
Cache Efficiency ..................: 0.997372
Cache Efficiency Rel...............: 1.000000
Learn entries......................: 100
Reading............................: 72761843 bytes in 7 transactions
Readahead..........................: 256000 bytes with overhead = 0 bytes
Average transaction................: 10394.549000 Kbytes
Number of blocks in current cache..: 210, total size: 6280352

if option = "a" the list of blocks in the cache is printed see also class TTreePerfStats.
if option contains 'cachedbranches', the list of branches being cached is printed.

Reimplemented from TTreeCache.

Definition at line 963 of file TTreeCacheUnzip.cxx.

◆ ReadBufferExt()

Int_t TTreeCacheUnzip::ReadBufferExt	(	char *	buf,
		Long64_t	pos,
		Int_t	len,
		Int_t &	loc
	)

virtual

Reimplemented from TFileCacheRead.

Definition at line 977 of file TTreeCacheUnzip.cxx.

◆ ResetCache()

void TTreeCacheUnzip::ResetCache ( )

virtual

This will delete the list of buffers that are in the unzipping cache and will reset certain values in the cache.

This name is ambiguos because the method doesn't reset the whole cache, only the part related to the unzipping Note: This method is completely different from TTreeCache::ResetCache(), in that method we were cleaning the prefetching buffer while here we delete the information about the unzipped buffers

Reimplemented from TTreeCache.

Definition at line 521 of file TTreeCacheUnzip.cxx.

◆ SetBufferSize()

Int_t TTreeCacheUnzip::SetBufferSize ( Int_t buffersize )

virtual

Change the underlying buffer size of the cache.

Returns:

0 if the buffer content is still available
1 if some or all of the buffer content has been made unavailable
-1 on error

Reimplemented from TTreeCache.

Definition at line 379 of file TTreeCacheUnzip.cxx.

◆ SetEntryRange()

void TTreeCacheUnzip::SetEntryRange	(	Long64_t	emin,
		Long64_t	emax
	)

virtual

Set the minimum and maximum entry number to be processed this information helps to optimize the number of baskets to read when prefetching the branch buffers.

Reimplemented from TTreeCache.

Definition at line 395 of file TTreeCacheUnzip.cxx.

◆ SetParallelUnzip()

Int_t TTreeCacheUnzip::SetParallelUnzip ( TTreeCacheUnzip::EParUnzipMode option = TTreeCacheUnzip::kEnable )

static

Static function that (de)activates multithreading unzipping.

The possible options are:

kEnable Enable it, which causes an automatic detection and launches the additional thread if the number of cores in the machine is greater than one
kDisable Disable will not activate the additional thread.
kForce Force will start the additional thread even if there is only one core. the default will be taken as kEnable.

Returns 0 if there was an error, 1 otherwise.

Definition at line 456 of file TTreeCacheUnzip.cxx.

◆ SetUnzipBufferSize()

void TTreeCacheUnzip::SetUnzipBufferSize ( Long64_t bufferSize )

Sets the size for the unzipping cache...

by default it should be two times the size of the prefetching cache

Definition at line 855 of file TTreeCacheUnzip.cxx.

◆ SetUnzipGroupSize()

void TTreeCacheUnzip::SetUnzipGroupSize ( Int_t groupSize )

inline

Definition at line 152 of file TTreeCacheUnzip.h.

◆ SetUnzipRelBufferSize()

void TTreeCacheUnzip::SetUnzipRelBufferSize ( Float_t relbufferSize )

static

static function: Sets the unzip relatibe buffer size

Definition at line 846 of file TTreeCacheUnzip.cxx.

◆ StopLearningPhase()

void TTreeCacheUnzip::StopLearningPhase ( )

virtual

It's the same as TTreeCache::StopLearningPhase but we guarantee that we start the unzipping just after getting the buffers.

Reimplemented from TTreeCache.

Definition at line 404 of file TTreeCacheUnzip.cxx.

◆ UnzipBuffer()

Int_t TTreeCacheUnzip::UnzipBuffer	(	char **	dest,
		char *	src
	)

Unzips a ROOT specific buffer...

by reading the header at the beginning. returns the size of the inflated buffer or -1 if error Note!! : If *dest == 0 we will allocate the buffer and it will be the responsability of the caller to free it... it is useful for example to pass it to the creator of TBuffer src is the original buffer with the record (header+compressed data) *dest is the inflated buffer (including the header)

Definition at line 869 of file TTreeCacheUnzip.cxx.

◆ UnzipCache()

Int_t TTreeCacheUnzip::UnzipCache ( Int_t index )

This inflates a basket in the cache.

. passing the data to a new buffer that will only wait there to be read... This function is responsible to update corresponding elements in fUnzipStatus, fUnzipChunks and fUnzipLen. Since we use atomic variables in fUnzipStatus to exclusively unzip the basket, we must update fUnzipStatus after fUnzipChunks and fUnzipLen and make sure fUnzipChunks and fUnzipLen are ready before main thread fetch the data.

Definition at line 546 of file TTreeCacheUnzip.cxx.

◆ UpdateBranches()

void TTreeCacheUnzip::UpdateBranches ( TTree * tree )

virtual

update pointer to current Tree and recompute pointers to the branches in the cache

Reimplemented from TTreeCache.

Definition at line 412 of file TTreeCacheUnzip.cxx.

Member Data Documentation

◆ fAsyncReading

Bool_t TTreeCacheUnzip::fAsyncReading

protected

Definition at line 87 of file TTreeCacheUnzip.h.

◆ fCompBuffer

char* TTreeCacheUnzip::fCompBuffer

private

Definition at line 118 of file TTreeCacheUnzip.h.

◆ fCompBufferSize

Int_t TTreeCacheUnzip::fCompBufferSize

private

Definition at line 119 of file TTreeCacheUnzip.h.

◆ fCycle

Int_t TTreeCacheUnzip::fCycle

protected

Definition at line 89 of file TTreeCacheUnzip.h.

◆ fEmpty

Bool_t TTreeCacheUnzip::fEmpty

protected

Definition at line 88 of file TTreeCacheUnzip.h.

◆ fgParallel

TTreeCacheUnzip::EParUnzipMode TTreeCacheUnzip::fgParallel = TTreeCacheUnzip::kDisable

staticprotected

Indicate if we want to activate the parallelism.

Definition at line 94 of file TTreeCacheUnzip.h.

◆ fgRelBuffSize

Double_t TTreeCacheUnzip::fgRelBuffSize = .5

staticprotected

This is the percentage of the TTreeCacheUnzip that will be used.

Definition at line 106 of file TTreeCacheUnzip.h.

◆ fIOMutex

TMutex* TTreeCacheUnzip::fIOMutex

protected

Definition at line 92 of file TTreeCacheUnzip.h.

◆ fNFound

Int_t TTreeCacheUnzip::fNFound

protected

! number of blocks that were found in the cache

Definition at line 109 of file TTreeCacheUnzip.h.

◆ fNMissed

Int_t TTreeCacheUnzip::fNMissed

protected

! number of blocks that were not found in the cache and were unzipped

Definition at line 110 of file TTreeCacheUnzip.h.

◆ fNseekMax

Int_t TTreeCacheUnzip::fNseekMax

protected

! fNseek can change so we need to know its max size

Definition at line 102 of file TTreeCacheUnzip.h.

◆ fNStalls

Int_t TTreeCacheUnzip::fNStalls

protected

! number of hits which caused a stall

Definition at line 111 of file TTreeCacheUnzip.h.

◆ fNUnzip

Int_t TTreeCacheUnzip::fNUnzip

protected

! number of blocks that were unzipped

Definition at line 112 of file TTreeCacheUnzip.h.

◆ fParallel

Bool_t TTreeCacheUnzip::fParallel

protected

Indicate if we want to activate the parallelism (for this instance)

Definition at line 90 of file TTreeCacheUnzip.h.

◆ fUnzipBufferSize

Long64_t TTreeCacheUnzip::fUnzipBufferSize

protected

! Max Size for the ready unzipped blocks (default is 2*fBufferSize)

Definition at line 104 of file TTreeCacheUnzip.h.

◆ fUnzipGroupSize

Int_t TTreeCacheUnzip::fUnzipGroupSize

protected

! Min accumulated size of a group of baskets ready to be unzipped by a IMT task

Definition at line 103 of file TTreeCacheUnzip.h.

◆ fUnzipState

UnzipState_t TTreeCacheUnzip::fUnzipState

protected

Definition at line 84 of file TTreeCacheUnzip.h.

◆ fUnzipTaskGroup

std::unique_ptr<ROOT::Experimental::TTaskGroup> TTreeCacheUnzip::fUnzipTaskGroup

protected

Definition at line 98 of file TTreeCacheUnzip.h.

Libraries for TTreeCacheUnzip:

[legend]

Parallel Unzipping

Classes

Public Types

Public Member Functions

Static Public Member Functions

Protected Types

Protected Attributes

Static Protected Attributes

Private Member Functions

Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ UnzipState_t

Member Enumeration Documentation

◆ EParUnzipMode

◆ EUnzipState

Constructor & Destructor Documentation

◆ TTreeCacheUnzip() [1/3]

◆ TTreeCacheUnzip() [2/3]

◆ TTreeCacheUnzip() [3/3]

◆ ~TTreeCacheUnzip()

Member Function Documentation

◆ AddBranch() [1/2]

◆ AddBranch() [2/2]

◆ CreateTasks()

◆ FillBuffer()

◆ GetNFound()

◆ GetNMissed()

◆ GetNUnzip()

◆ GetParallelUnzip()

◆ GetRecordHeader()

◆ GetUnzipBuffer()

◆ GetUnzipGroupSize()

◆ Init()

◆ IsParallelUnzip()

◆ operator=()

◆ Print()

◆ ReadBufferExt()

◆ ResetCache()

◆ SetBufferSize()

◆ SetEntryRange()

◆ SetParallelUnzip()

◆ SetUnzipBufferSize()

◆ SetUnzipGroupSize()

◆ SetUnzipRelBufferSize()

◆ StopLearningPhase()

◆ UnzipBuffer()

◆ UnzipCache()

◆ UpdateBranches()

Member Data Documentation

◆ fAsyncReading

◆ fCompBuffer

◆ fCompBufferSize

◆ fCycle

◆ fEmpty

◆ fgParallel

◆ fgRelBuffSize

◆ fIOMutex

◆ fNFound

◆ fNMissed

◆ fNseekMax

◆ fNStalls

◆ fNUnzip

◆ fParallel

◆ fUnzipBufferSize

◆ fUnzipGroupSize

◆ fUnzipState

◆ fUnzipTaskGroup