Re: root crashes when creating large histogram/histograms with certan number of channels

Hi Axel,

I have more details about the memory problem/leak. Andrey Blazhev and Norbert Braun from Cologne, who are part of our group working on the analysis looked into the allocated memory from large TH2D histogram on a 64bit machine with 24GB of RAM. They reported:

20k * 30k -> 900 MB allocated
30k * 30k -> Gives the known exception
33k * 33k -> Works; almost no memory allocated (NOTE: size in memory is 2**15 * 2**15 * sizeof(double) > 4 GiB)
50k * 30k -> similar to the 20k * 30k case which means that the Integer value is wrapped around.

In the analysis we had problems with crashes then many large histograms were created in memory. As a solution we reduced the number of large histograms until we had no bad_alloc error. Nevertheless we observed "fake" peaks in some of our histograms. They were high peaks with width of one channel (which can not be created from our detectors).

I implement THnSparse for all 2D histograms in our analysis. Now the output root file which contains all histograms is 4 times larger compare to the output file which was generated when TH2I were used, supporting the suggestion that we had a memory leak when creating many TH2I histograms before.

The question is how can we ensure that we still do not have a memory leak? (That all peaks in the histograms are real).

I can track the memory allocated for histograms looking at how many channels we allocated in TH1I and with GetSparseFractionMem for the THnSparse histograms. Can this be a solution?

On you question: we first project the full range ProjectionX('..',-1,-1)
Then for each peak in the spectrum we make 3 ProjectionX:

ProjectionX('..",p_low,p_high) selecting a peak

and two
ProjectionX('..',b1_low,b1_high) and ProjectionX('..',b2_low,b2_high) selecting the background around the peak.

Then we subtract ProjectionX('..",p_low,p_high) - w1*ProjectionX('..',b1_low,b1_high)-w2*ProjectionX('..',b2_low,b2_high)

from the peaks we see in the obtained spectra we decide were to gate next and repeat the above procedure. As there are many possible coincidences we need look at, the time it takes to generate the final spectra is of importance.

One more detail, the TH2I matrices are build symmetric(or if the matrix is not build symmetric we have to produce ProjectionY too).

Considering this is a TTree or THnSparse histogram better?

Cheers,
Plamen

On Fri, 2009-11-06 at 16:15 +0100, Axel Naumann wrote:
> Hi Plamen,
>
> thanks, now we have a clearer idea how to help :-) Follow-up question:
> do you care about the full y range when scanning for a coincidence with
> x, or only for the n closest bins?
>
> I can suggest two options: you can use a THnSparse, as suggested by
> David. That one only needs memory for *filled* bins. Or you can use a
> TTree, which will allow you to keep only a relevant part of the data in
> memory. We can discuss its layout if you want; you might want to create
> branches for e.g. all x bins.
>
> Let us know!
>
> Cheers, Axel.
>
> Plamen Boutachkov wrote on 11/06/2009 03:50 PM:
> > Hi Axel,
> >
> > We are analyzing an experiment in which we study the gamma decay of
> > unstable nuclei. For each nucleus we record the gamma rays emitted in a
> > coincidence with a hyper pure germanium detector array.
> >
> > We fill the TH2I(5001,0.,5000.,5001,0.,5000.) histograms with all
> > possible pairs (E_i, E_j) of observed gamma rays energies in an event of
> > interest. Latter on we make multiple gates on one of the axis of the
> > matrix and look for the gamma rays in coincidence.
> >
> > There are a couple of these TH2I histograms per nucleus (created with
> > different conditions) and about 10 nuclei which we would like to study.
> > It takes about 1h to sort through all the data in order to produce the
> > TH2I histograms.
> >
> > We need the high number of channels to conserve the resolution of the
> > gamma detectors. On other hand there are many channels with few or zero
> > counts in the TH2I histograms.
> >
> > Cheers,
> > Plamen
> >
> >
> >
> > On Fri, 2009-11-06 at 15:03 +0100, Axel Naumann wrote:
> >> Hi Plamen,
> >>
> >> reaching these limits usually means that your approach is wrong. What
> >> are you trying to do?
> >>
> >> Cheers, Axel.
> >>
> >> Plamen Boutachkov wrote on 11/06/2009 02:57 PM:
> >>> Hi David,
> >>>
> >>> I do not know enough about the root internal memory management. I was
> >>> hoping that the swap of the machines will help.
> >>>
> >>> But then if the limit is the physical RAM of the machine why I can
> >>> create
> >>> TH2D *t = new TH2D("test","t",70000,0.,1.,70000,0.,1.)
> >>> 70000*70000* 8 B = 36GB
> >>>
> >>> On the same machine 32 bit 4GB RAM I tried:
> >>> TH2I *t = new TH2I("test","t",100000,0.,1.,100000,0.,1.)
> >>> a 37GB histogram, it worked.
> >>>
> >>> Is there a difference how one and many histograms are handled in
> >>> memory?
> >>> Should one track the total memory spend for histograms and make sure it
> >>> is bellow a given number?
> >>>
> >>> Thank you for the suggestion of THnSparse.
> >>>
> >>> Plamen
> >>>
> >>> On Fri, 2009-11-06 at 14:10 +0100, David Gonzalez Maline wrote:
> >>>> Hi,
> >>>>
> >>>> Just some simple maths would help.
> >>>>
> >>>> 5001 xbins * 5001 ybins * 4bytes each bin * 40 histograms ~= 3.72 GB of
> >>>> memory...
> >>>>
> >>>> Obviously in the second machine you don't get such an error until you
> >>>> change the number of bins to a much bigger number. Consider using
> >>>> THnSparse instead and/or reducing the number of bins.
> >>>>
> >>>> David
> >>>>
> >>>>
> >>>> Plamen Boutachkov wrote:
> >>>>> Hello,
> >>>>>
> >>>>> I need to create a large number of TH2I, this leads to a crash. Here
> >>>>> is an example script which reproduces the problem:
> >>>>>
> >>>>> {
> >>>>> Int_t nHistograms = 40 ;
> >>>>> TH2I *h[nHistograms] ;
> >>>>>
> >>>>> for(Int_t nIndex = 0; nIndex < nHistograms ; ++nIndex ) {
> >>>>> TString sHistogramNname = Form("h%d",nIndex);
> >>>>> cout<<"Histogram Number:"<< nIndex << endl;
> >>>>>
> >>>>> h[nIndex] = new TH2I(sHistogramNname.Data(),sHistogramNname.Data()
> >>>>> ,5001,0.,5000.
> >>>>> ,5001,0.,5000.);
> >>>>> }
> >>>>> }
> >>>>>
> >>>>> executing the script gives:
> >>>>>
> >>>>> Histogram Number:0
> >>>>> ...
> >>>>> Histogram Number:29
> >>>>> Histogram Number:30
> >>>>> Error: Symbol G__exception is not defined in current scope
> >>>>> scripts/test.C:11:
> >>>>> Error: type G__exception not defined FILE:/tmp/test.C LINE:11
> >>>>> *** Interpreter error recovered ***
> >>>>>
> >>>>> If I include the above loop in a stand alone program the error is:
> >>>>> Histogram Number:0
> >>>>> ...
> >>>>> Histogram Number:30
> >>>>> terminate called after throwing an instance of 'std::bad_alloc'
> >>>>> what(): std::bad_alloc
> >>>>> Aborted
> >>>>>
> >>>>>
> >>>>> I can produce a similar error by executing:
> >>>>>
> >>>>> TH2D *t = new TH2D("test","t",30000,0.,1.,30000,0.,1.)
> >>>>> Error: Symbol G__exception is not defined in current scope (tmpfile):1:
> >>>>> Error: type G__exception not defined FILE:(tmpfile) LINE:1
> >>>>>
> >>>>> or
> >>>>>
> >>>>> root [0] TH2D *t = new TH2D("test","t",40000,0.,1.,40000,0.,1.)
> >>>>> Error: Symbol G__exception is not defined in current scope (tmpfile):1:
> >>>>> Error: type G__exception not defined FILE:(tmpfile) LINE:1
> >>>>>
> >>>>> On other hand creating a larger histograms:
> >>>>>
> >>>>> root [0] TH2D *t = new TH2D("test","t",50000,0.,1.,50000,0.,1.)
> >>>>> or
> >>>>> root [0] TH2D *t = new TH2D("test","t",60000,0.,1.,60000,0.,1.)
> >>>>> or
> >>>>> root [0] TH2D *t = new TH2D("test","t",70000,0.,1.,70000,0.,1.)
> >>>>> works.
> >>>>>
> >>>>> I am running ROOT 5.24/00b on a 32bit machine with 3GB of memory.
> >>>>>
> >>>>> I tried the same on a 64bit machine, 24GB of memory, root Version
> >>>>> 5.18/00b
> >>>>> executing
> >>>>>
> >>>>> root [0] TH2D *t = new TH2D("test","t",30000,0.,1.,30000,0.,1.)
> >>>>>
> >>>>> does not give an error, but
> >>>>>
> >>>>> root [0] TH2D *t = new TH2D("test","t",40000,0.,1.,40000,0.,1.)
> >>>>> Error: Symbol G__exception is not defined in current scope (tmpfile):1:
> >>>>> Error: type G__exception not defined FILE:(tmpfile) LINE:1
> >>>>>
> >>>>> Regards,
> >>>>> Plamen
> >>>>>
> >>>>> --
> >>>>> -----------------------------
> >>>>> Dr. Plamen Boutachkov
> >>>>> Kernstruktur, GSI Darmstadt
> >>>>> Planckstr. 1
> >>>>> D-64291 Darmstadt
> >>>>> Germany
> >>>>>
> >>>>> Tel: +49 6159-71-2436
> >>>>> Fax: +49-6159-71-2898
> >>>>> -----------------------------
> >>>>>
> >>>>>
> >>>>>
> >>>
> >
> >

-- 
-----------------------------
Dr. Plamen Boutachkov
KP II, GSI Darmstadt
Planckstr. 1
D-64291 Darmstadt
Germany

Tel: +49-6159-71 2436
Fax: +49-6159-71 2809
-----------------------------