Re: [ROOT] Writing large TTree causes lots of page faults...

From: Rene Brun (Rene.Brun@cern.ch)
Date: Mon Jun 30 2003 - 18:23:13 MEST


Hi Ed,

You can allocate large buffers when you create your Tree branches (see
argument buffersize).

TTree::Print is trying to compute a precise number for the size of the buffers
in memory once they are dumped to the file (when compression).
The algorithm is not designed for large Trees in memory. In this case,
the current algorithm should be bypassed and it should report only the memory
occupation. I will add this protection.

Rene Brun

Ed Oltman wrote:
> 
> Hello,
> 
>  I have a program that writes a "large" TTree - 7 branches: 6 hold a float,
> the 7th a short,  with about 20E6 entries.  This TTree holds data that I
> spin through multiple time to obtain calibration factors.  The current
> directory is NOT writable - e.g. gFile->IsWritable() returns 0.  This forces
> the TTree to remain memory resident saving me the I/O overhead on subsequent
> spins through the data.  I have 1 GB of ram - more than enough to hold the
> entire TTree.
> 
> My question is: Is it possible to pre-allocate the memory required for this
> TTree?  What happens is that my program page faults about 800 times/second,
> each time increasing the memory used by my program by about 3 MB -
> eventually gets about 540 MB - its very slow!  subsequent spins through the
> tree, however are much quicker then what it would take if TTree is on disk.
> 
> I am using the Win32 version of root version 3.05/3 on Win2K.  I compiled my
> program using VC++ 6.0
> 
> One other thing: TTree::Print() causes:
> 
> Fatal in <TStorage::ReAllocChar>: storage exhausted
> aborting
> Warning in <TWinNTSystem::StackTrace>: this method must be overridden!
> 
> abnormal program termination
> 
> I can see that my program tries to allocate too much memory - the peak is
> about 850MB - why should TTree::Print() allocate memory?
> 
> Thanks,
>  Ed Oltman
> 
>   --------------------------------------------------------------------------------
>                   Name: winmail.dat
>    winmail.dat    Type: application/ms-tnef
>               Encoding: base64



This archive was generated by hypermail 2b29 : Thu Jan 01 2004 - 17:50:12 MET