Re: root crashes when creating large histogram/histograms with certan number of channels

Hi Axel,

We are analyzing an experiment in which we study the gamma decay of unstable nuclei. For each nucleus we record the gamma rays emitted in a coincidence with a hyper pure germanium detector array.

We fill the TH2I(5001,0.,5000.,5001,0.,5000.) histograms with all possible pairs (E_i, E_j) of observed gamma rays energies in an event of interest. Latter on we make multiple gates on one of the axis of the matrix and look for the gamma rays in coincidence.

There are a couple of these TH2I histograms per nucleus (created with different conditions) and about 10 nuclei which we would like to study. It takes about 1h to sort through all the data in order to produce the TH2I histograms.

We need the high number of channels to conserve the resolution of the gamma detectors. On other hand there are many channels with few or zero counts in the TH2I histograms.

Cheers,
Plamen

On Fri, 2009-11-06 at 15:03 +0100, Axel Naumann wrote:
> Hi Plamen,
>
> reaching these limits usually means that your approach is wrong. What
> are you trying to do?
>
> Cheers, Axel.
>
> Plamen Boutachkov wrote on 11/06/2009 02:57 PM:
> > Hi David,
> >
> > I do not know enough about the root internal memory management. I was
> > hoping that the swap of the machines will help.
> >
> > But then if the limit is the physical RAM of the machine why I can
> > create
> > TH2D *t = new TH2D("test","t",70000,0.,1.,70000,0.,1.)
> > 70000*70000* 8 B = 36GB
> >
> > On the same machine 32 bit 4GB RAM I tried:
> > TH2I *t = new TH2I("test","t",100000,0.,1.,100000,0.,1.)
> > a 37GB histogram, it worked.
> >
> > Is there a difference how one and many histograms are handled in
> > memory?
> > Should one track the total memory spend for histograms and make sure it
> > is bellow a given number?
> >
> > Thank you for the suggestion of THnSparse.
> >
> > Plamen
> >
> > On Fri, 2009-11-06 at 14:10 +0100, David Gonzalez Maline wrote:
> >> Hi,
> >>
> >> Just some simple maths would help.
> >>
> >> 5001 xbins * 5001 ybins * 4bytes each bin * 40 histograms ~= 3.72 GB of
> >> memory...
> >>
> >> Obviously in the second machine you don't get such an error until you
> >> change the number of bins to a much bigger number. Consider using
> >> THnSparse instead and/or reducing the number of bins.
> >>
> >> David
> >>
> >>
> >> Plamen Boutachkov wrote:
> >>> Hello,
> >>>
> >>> I need to create a large number of TH2I, this leads to a crash. Here
> >>> is an example script which reproduces the problem:
> >>>
> >>> {
> >>> Int_t nHistograms = 40 ;
> >>> TH2I *h[nHistograms] ;
> >>>
> >>> for(Int_t nIndex = 0; nIndex < nHistograms ; ++nIndex ) {
> >>> TString sHistogramNname = Form("h%d",nIndex);
> >>> cout<<"Histogram Number:"<< nIndex << endl;
> >>>
> >>> h[nIndex] = new TH2I(sHistogramNname.Data(),sHistogramNname.Data()
> >>> ,5001,0.,5000.
> >>> ,5001,0.,5000.);
> >>> }
> >>> }
> >>>
> >>> executing the script gives:
> >>>
> >>> Histogram Number:0
> >>> ...
> >>> Histogram Number:29
> >>> Histogram Number:30
> >>> Error: Symbol G__exception is not defined in current scope
> >>> scripts/test.C:11:
> >>> Error: type G__exception not defined FILE:/tmp/test.C LINE:11
> >>> *** Interpreter error recovered ***
> >>>
> >>> If I include the above loop in a stand alone program the error is:
> >>> Histogram Number:0
> >>> ...
> >>> Histogram Number:30
> >>> terminate called after throwing an instance of 'std::bad_alloc'
> >>> what(): std::bad_alloc
> >>> Aborted
> >>>
> >>>
> >>> I can produce a similar error by executing:
> >>>
> >>> TH2D *t = new TH2D("test","t",30000,0.,1.,30000,0.,1.)
> >>> Error: Symbol G__exception is not defined in current scope (tmpfile):1:
> >>> Error: type G__exception not defined FILE:(tmpfile) LINE:1
> >>>
> >>> or
> >>>
> >>> root [0] TH2D *t = new TH2D("test","t",40000,0.,1.,40000,0.,1.)
> >>> Error: Symbol G__exception is not defined in current scope (tmpfile):1:
> >>> Error: type G__exception not defined FILE:(tmpfile) LINE:1
> >>>
> >>> On other hand creating a larger histograms:
> >>>
> >>> root [0] TH2D *t = new TH2D("test","t",50000,0.,1.,50000,0.,1.)
> >>> or
> >>> root [0] TH2D *t = new TH2D("test","t",60000,0.,1.,60000,0.,1.)
> >>> or
> >>> root [0] TH2D *t = new TH2D("test","t",70000,0.,1.,70000,0.,1.)
> >>> works.
> >>>
> >>> I am running ROOT 5.24/00b on a 32bit machine with 3GB of memory.
> >>>
> >>> I tried the same on a 64bit machine, 24GB of memory, root Version
> >>> 5.18/00b
> >>> executing
> >>>
> >>> root [0] TH2D *t = new TH2D("test","t",30000,0.,1.,30000,0.,1.)
> >>>
> >>> does not give an error, but
> >>>
> >>> root [0] TH2D *t = new TH2D("test","t",40000,0.,1.,40000,0.,1.)
> >>> Error: Symbol G__exception is not defined in current scope (tmpfile):1:
> >>> Error: type G__exception not defined FILE:(tmpfile) LINE:1
> >>>
> >>> Regards,
> >>> Plamen
> >>>
> >>> --
> >>> -----------------------------
> >>> Dr. Plamen Boutachkov
> >>> Kernstruktur, GSI Darmstadt
> >>> Planckstr. 1
> >>> D-64291 Darmstadt
> >>> Germany
> >>>
> >>> Tel: +49 6159-71-2436
> >>> Fax: +49-6159-71-2898
> >>> -----------------------------
> >>>
> >>>
> >>>
> >
> >

-- 
-----------------------------
Dr. Plamen Boutachkov
Kernstruktur, GSI Darmstadt
Planckstr. 1
D-64291 Darmstadt
Germany

Tel: +49 6159-71-2436
Fax: +49-6159-71-2898
-----------------------------