Hi Plamen,
thanks, now we have a clearer idea how to help :-) Follow-up question:
do you care about the full y range when scanning for a coincidence with
x, or only for the n closest bins?
I can suggest two options: you can use a THnSparse, as suggested by
David. That one only needs memory for *filled* bins. Or you can use a
TTree, which will allow you to keep only a relevant part of the data in
memory. We can discuss its layout if you want; you might want to create
branches for e.g. all x bins.
Let us know!
Cheers, Axel.
Plamen Boutachkov wrote on 11/06/2009 03:50 PM:
> Hi Axel,
>
> We are analyzing an experiment in which we study the gamma decay of
> unstable nuclei. For each nucleus we record the gamma rays emitted in a
> coincidence with a hyper pure germanium detector array.
>
> We fill the TH2I(5001,0.,5000.,5001,0.,5000.) histograms with all
> possible pairs (E_i, E_j) of observed gamma rays energies in an event of
> interest. Latter on we make multiple gates on one of the axis of the
> matrix and look for the gamma rays in coincidence.
>
> There are a couple of these TH2I histograms per nucleus (created with
> different conditions) and about 10 nuclei which we would like to study.
> It takes about 1h to sort through all the data in order to produce the
> TH2I histograms.
>
> We need the high number of channels to conserve the resolution of the
> gamma detectors. On other hand there are many channels with few or zero
> counts in the TH2I histograms.
>
> Cheers,
> Plamen
>
>
>
> On Fri, 2009-11-06 at 15:03 +0100, Axel Naumann wrote:
>> Hi Plamen,
>>
>> reaching these limits usually means that your approach is wrong. What
>> are you trying to do?
>>
>> Cheers, Axel.
>>
>> Plamen Boutachkov wrote on 11/06/2009 02:57 PM:
>>> Hi David,
>>>
>>> I do not know enough about the root internal memory management. I was
>>> hoping that the swap of the machines will help.
>>>
>>> But then if the limit is the physical RAM of the machine why I can
>>> create
>>> TH2D *t = new TH2D("test","t",70000,0.,1.,70000,0.,1.)
>>> 70000*70000* 8 B = 36GB
>>>
>>> On the same machine 32 bit 4GB RAM I tried:
>>> TH2I *t = new TH2I("test","t",100000,0.,1.,100000,0.,1.)
>>> a 37GB histogram, it worked.
>>>
>>> Is there a difference how one and many histograms are handled in
>>> memory?
>>> Should one track the total memory spend for histograms and make sure it
>>> is bellow a given number?
>>>
>>> Thank you for the suggestion of THnSparse.
>>>
>>> Plamen
>>>
>>> On Fri, 2009-11-06 at 14:10 +0100, David Gonzalez Maline wrote:
>>>> Hi,
>>>>
>>>> Just some simple maths would help.
>>>>
>>>> 5001 xbins * 5001 ybins * 4bytes each bin * 40 histograms ~= 3.72 GB of
>>>> memory...
>>>>
>>>> Obviously in the second machine you don't get such an error until you
>>>> change the number of bins to a much bigger number. Consider using
>>>> THnSparse instead and/or reducing the number of bins.
>>>>
>>>> David
>>>>
>>>>
>>>> Plamen Boutachkov wrote:
>>>>> Hello,
>>>>>
>>>>> I need to create a large number of TH2I, this leads to a crash. Here
>>>>> is an example script which reproduces the problem:
>>>>>
>>>>> {
>>>>> Int_t nHistograms = 40 ;
>>>>> TH2I *h[nHistograms] ;
>>>>>
>>>>> for(Int_t nIndex = 0; nIndex < nHistograms ; ++nIndex ) {
>>>>> TString sHistogramNname = Form("h%d",nIndex);
>>>>> cout<<"Histogram Number:"<< nIndex << endl;
>>>>>
>>>>> h[nIndex] = new TH2I(sHistogramNname.Data(),sHistogramNname.Data()
>>>>> ,5001,0.,5000.
>>>>> ,5001,0.,5000.);
>>>>> }
>>>>> }
>>>>>
>>>>> executing the script gives:
>>>>>
>>>>> Histogram Number:0
>>>>> ...
>>>>> Histogram Number:29
>>>>> Histogram Number:30
>>>>> Error: Symbol G__exception is not defined in current scope
>>>>> scripts/test.C:11:
>>>>> Error: type G__exception not defined FILE:/tmp/test.C LINE:11
>>>>> *** Interpreter error recovered ***
>>>>>
>>>>> If I include the above loop in a stand alone program the error is:
>>>>> Histogram Number:0
>>>>> ...
>>>>> Histogram Number:30
>>>>> terminate called after throwing an instance of 'std::bad_alloc'
>>>>> what(): std::bad_alloc
>>>>> Aborted
>>>>>
>>>>>
>>>>> I can produce a similar error by executing:
>>>>>
>>>>> TH2D *t = new TH2D("test","t",30000,0.,1.,30000,0.,1.)
>>>>> Error: Symbol G__exception is not defined in current scope (tmpfile):1:
>>>>> Error: type G__exception not defined FILE:(tmpfile) LINE:1
>>>>>
>>>>> or
>>>>>
>>>>> root [0] TH2D *t = new TH2D("test","t",40000,0.,1.,40000,0.,1.)
>>>>> Error: Symbol G__exception is not defined in current scope (tmpfile):1:
>>>>> Error: type G__exception not defined FILE:(tmpfile) LINE:1
>>>>>
>>>>> On other hand creating a larger histograms:
>>>>>
>>>>> root [0] TH2D *t = new TH2D("test","t",50000,0.,1.,50000,0.,1.)
>>>>> or
>>>>> root [0] TH2D *t = new TH2D("test","t",60000,0.,1.,60000,0.,1.)
>>>>> or
>>>>> root [0] TH2D *t = new TH2D("test","t",70000,0.,1.,70000,0.,1.)
>>>>> works.
>>>>>
>>>>> I am running ROOT 5.24/00b on a 32bit machine with 3GB of memory.
>>>>>
>>>>> I tried the same on a 64bit machine, 24GB of memory, root Version
>>>>> 5.18/00b
>>>>> executing
>>>>>
>>>>> root [0] TH2D *t = new TH2D("test","t",30000,0.,1.,30000,0.,1.)
>>>>>
>>>>> does not give an error, but
>>>>>
>>>>> root [0] TH2D *t = new TH2D("test","t",40000,0.,1.,40000,0.,1.)
>>>>> Error: Symbol G__exception is not defined in current scope (tmpfile):1:
>>>>> Error: type G__exception not defined FILE:(tmpfile) LINE:1
>>>>>
>>>>> Regards,
>>>>> Plamen
>>>>>
>>>>> --
>>>>> -----------------------------
>>>>> Dr. Plamen Boutachkov
>>>>> Kernstruktur, GSI Darmstadt
>>>>> Planckstr. 1
>>>>> D-64291 Darmstadt
>>>>> Germany
>>>>>
>>>>> Tel: +49 6159-71-2436
>>>>> Fax: +49-6159-71-2898
>>>>> -----------------------------
>>>>>
>>>>>
>>>>>
>>>
>
>
Received on Fri Nov 06 2009 - 16:14:25 CET