Hi Ed, Your numbers confirm that Windows is very bad for I/O. I already noticed this problem, see: http://root.cern.ch/root/roottalk/roottalk03/1534.html and related mails. In this mail, I was comparing the results running the stress test suite under VC++, CYGWIN/gcc3.2 and Linux on the same machine. Windows/VC++ shows a very bad RealTime/CpuTime ratio. I tried tuning several parameters of the file system without success. Advice from a Windows guru would be appreciated. Rene Brun On Mon, 30 Jun 2003, Ed Oltman wrote: > Rene, > I have included two .gif files - screen shots of the windows task manager. > Default Buffer.gif was made with defult buffersize in my call to > TTree::Branch() > while Big Buffer.gif was made with buffersize=10000000. > > My raw tree is read and processed a first time between A and B > At B I re-read this same file and write my "memory resident" TTree > At C and D, I begin analysis passes through the memory resident TTree. > > Both runs (buffersize=32000 and 10000000) result in roughly the same number > of > page faults and wall clock time to write the memory resedent tree. You can > see > > > Any other hints on how I might avoid all of the page faults in making a > large > memory resident trees are welcome/appreciated! > > Thanks.. > Ed Oltman > > > > > -----Original Message----- > > From: brun@pcbrun.cern.ch [mailto:brun@pcbrun.cern.ch]On Behalf Of Rene > > Brun > > Sent: Monday, June 30, 2003 11:23 AM > > To: Ed Oltman > > Cc: Roottalk@Pcroot. Cern. Ch > > Subject: Re: [ROOT] Writing large TTree causes lots of page faults... > > > > > > Hi Ed, > > > > You can allocate large buffers when you create your Tree branches (see > > argument buffersize). > > > > TTree::Print is trying to compute a precise number for the size > > of the buffers > > in memory once they are dumped to the file (when compression). > > The algorithm is not designed for large Trees in memory. In this case, > > the current algorithm should be bypassed and it should report > > only the memory > > occupation. I will add this protection. > > > > Rene Brun > > > > Ed Oltman wrote: > > > > > > Hello, > > > > > > I have a program that writes a "large" TTree - 7 branches: 6 > > hold a float, > > > the 7th a short, with about 20E6 entries. This TTree holds data that I > > > spin through multiple time to obtain calibration factors. The current > > > directory is NOT writable - e.g. gFile->IsWritable() returns 0. > > This forces > > > the TTree to remain memory resident saving me the I/O overhead > > on subsequent > > > spins through the data. I have 1 GB of ram - more than enough > > to hold the > > > entire TTree. > > > > > > My question is: Is it possible to pre-allocate the memory > > required for this > > > TTree? What happens is that my program page faults about 800 > > times/second, > > > each time increasing the memory used by my program by about 3 MB - > > > eventually gets about 540 MB - its very slow! subsequent spins > > through the > > > tree, however are much quicker then what it would take if TTree > > is on disk. > > > > > > I am using the Win32 version of root version 3.05/3 on Win2K. > > I compiled my > > > program using VC++ 6.0 > > > > > > One other thing: TTree::Print() causes: > > > > > > Fatal in <TStorage::ReAllocChar>: storage exhausted > > > aborting > > > Warning in <TWinNTSystem::StackTrace>: this method must be overridden! > > > > > > abnormal program termination > > > > > > I can see that my program tries to allocate too much memory - > > the peak is > > > about 850MB - why should TTree::Print() allocate memory? > > > > > > Thanks, > > > Ed Oltman > > > > > > > > ------------------------------------------------------------------ > > -------------- > > > Name: winmail.dat > > > winmail.dat Type: application/ms-tnef > > > Encoding: base64 > > > > > > >
This archive was generated by hypermail 2b29 : Thu Jan 01 2004 - 17:50:12 MET