RE: TEventList::Enter(), 2 issues

From: LEE, KERRY T. (JSC-SF) (UHCL) <kerry.t.lee1_at_jsc.nasa.gov>
Date: Mon, 6 Feb 2006 01:49:42 -0600


 Hi Rene,

I am only interested in distinguishing two variables 1)ion type 2) energy range. I have a total of about 10 variables for each event in my tree to be sampled.

For example I have energies basically from about 10-10000 MeV/n and I'd like to divide this up into 1MeV energy bins for the first few hundred MeV and then lump the rest into an upper high energy bin. I also do not care if I sample the same event more than once.

I tried dividing each bin into a separate tree, but creating >14000 trees caused a memory error. I've given thought to dividing each ion into a separate tree, but then I must have a TBits object of ~300 bit length to distinguish which energy it lies in.

An additional complication, is that I want to be able to rebin my 1MeV bins into large bins if neccessary. When using the TEventList I simply created a 2D array of TEventList[NumIons][NumBins] then used the nice feature of TEventList::Add(TEventList*) to create any size energy bins I wished. Then I sampled on these larger energy bins creating a vector<long> sample list, so that I could sample the same event more than once (since TEventList does not allow multiple entries of the same event number).

In this case I cannot think of a good way to use TBits. It seems to be more efficient to store the entry number for each energy and ion.

If you can think of a more efficient way, that would be great. For now I'm going to continue trying to store the entry lists in some way, be it vectors or in separate Tree branches.

Thanks for your help,

Kerry

-----Original Message-----
From: Rene Brun
To: LEE, KERRY T. (JSC-SF) (UHCL)
Cc: 'Rene Brun '; '''Philippe Canal ' ' '; '''owner-roottalk_at_pcroot.cern.ch ' ' '; '''roottalk_at_pcroot.cern.ch ' ' '
Sent: 2/5/2006 1:51 AM
Subject: RE: [ROOT] TEventList::Enter(), 2 issues

Hi Kerry,

It is difficult to come with a valuable advice without knowing all your conditions:

   -how many variables per event
   -sampling method; for example, do you accept the same event     multiple times?
Did you consider having as many Trees as you have Ions? in this case you can have one TBits object per Tree, both being much smaller.

Rene

On Fri, 3
Feb 2006, LEE, KERRY T. (JSC-SF) (UHCL) wrote:

> Hi Rene,
>
> This suggestion sounds great, but I'm not sure how to implement it.
> Currently I have a TEventList for each ion and each energy bin (>14000
> lists). I could create two TBits objects, one to represent Ions and
the
> other energy bin, and then set the appropriate bit in each TBits
object for
> each entry in my TChain, but this would require 47 bits for the ions
and
> about 300bits for the energy bins, or about 44bytes/entry as opposed
to the
> 8bytes/entry for the TEventList method.
>
> This must not be what you had in mind, or maybe I did not make it
clear that
> I had a large number of TEventList objects.
>
> Thanks
> Kerry
>
> -----Original Message-----
> From: Rene Brun
> To: LEE, KERRY T. (JSC-SF) (UHCL)
> Cc: 'Rene Brun '; ''Philippe Canal ' ';
''owner-roottalk_at_pcroot.cern.ch ' ';
> ''roottalk_at_pcroot.cern.ch ' '
> Sent: 2/3/2006 1:40 PM
> Subject: RE: [ROOT] TEventList::Enter(), 2 issues
>
> Hi Kerry,
>
> If you want to make minimal changes to your program, I suggest
> to use a TBits object instead of a TEventList. You will gain a factor
64
> in memory and the speed should be equivalent (if not better).
> Fill the TBits object somewhow in the same way that you fill
> the TEventList. Then, make the event loop yourself, finding
> the next bit set via "UInt_t FirstSetBit(UInt_t startBit=0)"
>
> Rene Brun
>
> On
> Fri, 3
> Feb 2006, LEE, KERRY T. (JSC-SF) (UHCL) wrote:
>
>> Rene,
>>
>> I am attempting a rather complex proceedure to avoid running my monte
> carlo
>> many times. I will try to describe my method and reason as briefly as
>> possible.
>>
>> I create particles of all types of ions from 1H-58Fe, all with a
> separate
>> energy spectrum and run these through my MC and store the calculated
> energy
>> deposition in regions of interest. I then store all the results in
> TTrees
>> and then gather them into a single TChain. I now want to sample all
> these
>> ions according to some abuandance scheme, then also sample certain
> energy
>> ranges in each spectra with different weights. Then the results of
> this
>> random sampling go into my Detector simulation algorithm to produce
> data of
>> the same form as the raw data so I can do my analysis on it. I do
not
> know
>> the exact expected spectra so I cannot do my whole simulation in a
> fixed
>> manner, so I start with a "good" guess and iterate over the above
> process
>> until I get good agreement between data and simulation.
>>
>> I need many statistics and this iteration process would take a very
> long
>> time if I had to rerun the MC for each iteration. That is why I
> choose to
>> do one big run and sample it. I make a TEventList for each ion and
> each
>> energy range and that way my sampling can be very fast and the result
> is
>> simply a list of event numbers to run through my detector simulation
> where I
>> only need to supply this list and the original TChain. The
TEventList
> that
>> holds the 1H ions is 77% of all the statistics, so it is huge
compared
> to
>> all the others and this is where the problem lies.
>>
>> I will try storing the lists in a TTree as Philippe has suggested.
My
> main
>> objective is to keep the execution time as short as possible and the
>> TEventList did a very good job with fewer statistics, so I was trying
> to
>> maintain this scheme with more statistics. If you have any further
>> suggestions I would be greatful.
>>
>> Thanks
>> Kerry
>>
>> -----Original Message-----
>> From: Rene Brun
>> To: LEE, KERRY T. (JSC-SF) (UHCL)
>> Cc: 'Philippe Canal '; 'owner-roottalk_at_pcroot.cern.ch ';
>> 'roottalk_at_pcroot.cern.ch '
>> Sent: 2/3/2006 11:30 AM
>> Subject: RE: [ROOT] TEventList::Enter(), 2 issues
>>
>> Kerry,
>>
>> What is the point of having a TEventlist of 200 million entries?
>>
>> Rene
>>
>> On Fri, 3
>> Feb 2006, LEE, KERRY T. (JSC-SF) (UHCL) wrote:
>>
>>> Thanks Philippe,
>>>
>>> Resizing works. I was trying to avoid writing to a new tree to
>> increase
>>> speed. Now the problem is that when I try to write the very large
>>> TEventList to a TFile via TFile::Write(), I get the same pointer
>> error. Is
>>> there a way to keep TFile from trying to allocate so much memory? I
>> tried
>>> using TObject::kOverwrite, but that did not help. Also, did you see
>> the
>>> second issue with TEventList::Enter() that I mentioned in my first
>> mail?
>>>
>>> Thanks
>>> Kerry
>>>
>>> -----Original Message-----
>>> From: owner-roottalk_at_pcroot.cern.ch
>>> To: LEE, KERRY T. (JSC-SF) (UHCL); roottalk_at_pcroot.cern.ch
>>> Sent: 2/2/2006 12:25 PM
>>> Subject: RE: [ROOT] TEventList::Enter(), 2 issues
>>>
>>> Hi,
>>>
>>> The entries in the TEventList are stored as Long64_t. When reaching
>>> 105000000, TEventList will allocates to holf twice than many entry
>>> and requires more than 1.600Gb of memory.
>>>
>>> There are some technics will could use to mitigate the issue (aka
>> allow
>>> for sligtly bigger event list) but given the your pattern of use,
>>> TEventList is no longer appropriate.
>>>
>>> Instead I recommend that you store the data in another TTree.
>>>
>>> If your limit is really 200 million, you can also make it work by
>>> pre-setting the size (this avoid the need to copy and to hold in
>>> memory 2 gigantic buffer):
>>>
>>> root [0] TEventList *E = new TEventList();
>>> root [2] E->Resize(200000000);
>>> root [1] {
>>> end with '}', '@':abort > for(int i=0;i<105000000;i++){
>>> end with '}', '@':abort > E->Enter(i);
>>> end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;
>>> end with '}', '@':abort > }
>>> end with '}', '@':abort > }
>>>
>>> Cheers,
>>> Philippe.
>>>
>>> -----Original Message-----
>>> From: owner-roottalk_at_pcroot.cern.ch
>>> [mailto:owner-roottalk_at_pcroot.cern.ch]
>>> On Behalf Of LEE, KERRY T. (JSC-SF) (UHCL)
>>> Sent: Thursday, February 02, 2006 10:38 AM
>>> To: 'roottalk_at_pcroot.cern.ch'
>>> Subject: [ROOT] TEventList::Enter(), 2 issues
>>>
>>> Dear ROOT team,
>>>
>>> I am using ROOT v5.08 on linux CentOS 4 compiled with gcc 3.4.4.
>>>
>>> First issue:
>>>
>>> I would like to create a very large TEventList with about 200
million
>>> entries. When it reaches just under 105 millon entries I get an
>> error.
>>> I
>>> have plenty of physical memory (2GB + 2GB swap) to perform this
task.
>> A
>>> session below shows the problem. On the other hand, I have no
problem
>>> creating an array of type Long64_t with 200 million entries.
>>>
>>> *******************************************
>>> * *
>>> * W E L C O M E to R O O T *
>>> * *
>>> * Version 5.08/00 13 December 2005 *
>>> * *
>>> * You are welcome to visit our Web site *
>>> * http://root.cern.ch *
>>> * *
>>> *******************************************
>>>
>>> FreeType Engine v2.1.9 used to render TrueType fonts.
>>> Compiled on 20 December 2005 for linux with thread support.
>>>
>>> CINT/ROOT C/C++ Interpreter version 5.16.5, November 30 2005
>>> Type ? for help. Commands must be C++ statements.
>>> Enclose multiple statements between { }.
>>> root [0] TEventList *E = new TEventList();
>>> root [1] {
>>> end with '}', '@':abort > for(int i=0;i<105000000;i++){
>>> end with '}', '@':abort > E->Enter(i);
>>> end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;
>>> end with '}', '@':abort > }
>>> end with '}', '@':abort > }
>>> 104857599Syntax Error: * (tmpfile):3:
>>> Error: Illegal pointer operation (tovalue) (tmpfile):3:
>>> *** Interpreter error recovered ***
>>>
>>>
>>> Second issue:
>>>
>>>
>>> I would like to add random integers to a TEventList, but it seems
> that
>>> only higher values than the previous one entered into the TEventList
>> is
>>> allowed.
>>>
>>> root [11] TEventList *lll = new TEventList()
>>> root [12] lll->Enter(1)
>>> root [13] lll->GetEntry(0)
>>> (const Long64_t)1
>>> root [14] lll->Enter(5)
>>> root [16] lll->GetEntry(1)
>>> (const Long64_t)5
>>> root [17] lll->Enter(3)
>>> root [18] lll->GetEntry(2)
>>> (const Long64_t)5
>>>
>>> As you can see adding the number 3 to the list after adding 5
> produces
>>> an entry of 5. Is this expected, or is this a bug? If for some
>> reason
>>> this is not allowed an error message would be very useful. I can
> only
>>> assume, since there exists a TEventList::Sort() function this is not
>> the
>>> desired behavior.
>>>
>>> The purpose for doing this is to randomly sample a ROOT tree without
>>> having to know anything about it except the total number of entries.
>> I
>>> would like to save this TEventList to a file, then read it and the
>> TTree
>>> back into an interacive ROOT session.
>>>
>>> Thanks
>>> Kerry
>>>
>>
>
Received on Mon Feb 06 2006 - 08:54:03 MET

This archive was generated by hypermail 2.2.0 : Mon Jan 01 2007 - 16:31:57 MET