RE: TEventList::Enter(), 2 issues

Hi Kerry,

It is difficult to come with a valuable advice without knowing all your conditions:

   -how many variables per event
   -sampling method; for example, do you accept the same event     multiple times?
Did you consider having as many Trees as you have Ions? in this case you can have one TBits object per Tree, both being much smaller.

Rene

On Fri, 3
Feb 2006, LEE, KERRY T. (JSC-SF) (UHCL) wrote:

> Hi Rene,
>
> This suggestion sounds great, but I'm not sure how to implement it.
> Currently I have a TEventList for each ion and each energy bin (>14000
> lists). I could create two TBits objects, one to represent Ions and the
> other energy bin, and then set the appropriate bit in each TBits object for
> each entry in my TChain, but this would require 47 bits for the ions and
> about 300bits for the energy bins, or about 44bytes/entry as opposed to the
> 8bytes/entry for the TEventList method.
>
> This must not be what you had in mind, or maybe I did not make it clear that
> I had a large number of TEventList objects.
>
> Thanks
> Kerry
>
> -----Original Message-----
> From: Rene Brun
> To: LEE, KERRY T. (JSC-SF) (UHCL)
> Cc: 'Rene Brun '; ''Philippe Canal ' '; ''owner-roottalk_at_pcroot.cern.ch ' ';
> ''roottalk_at_pcroot.cern.ch ' '
> Sent: 2/3/2006 1:40 PM
> Subject: RE: [ROOT] TEventList::Enter(), 2 issues
>
> Hi Kerry,
>
> If you want to make minimal changes to your program, I suggest
> to use a TBits object instead of a TEventList. You will gain a factor 64
> in memory and the speed should be equivalent (if not better).
> Fill the TBits object somewhow in the same way that you fill
> the TEventList. Then, make the event loop yourself, finding
> the next bit set via "UInt_t FirstSetBit(UInt_t startBit=0)"
>
> Rene Brun
>
> On
> Fri, 3
> Feb 2006, LEE, KERRY T. (JSC-SF) (UHCL) wrote:
>
>> Rene,
>>
>> I am attempting a rather complex proceedure to avoid running my monte
> carlo
>> many times. I will try to describe my method and reason as briefly as
>> possible.
>>
>> I create particles of all types of ions from 1H-58Fe, all with a
> separate
>> energy spectrum and run these through my MC and store the calculated
> energy
>> deposition in regions of interest. I then store all the results in
> TTrees
>> and then gather them into a single TChain. I now want to sample all
> these
>> ions according to some abuandance scheme, then also sample certain
> energy
>> ranges in each spectra with different weights. Then the results of
> this
>> random sampling go into my Detector simulation algorithm to produce
> data of
>> the same form as the raw data so I can do my analysis on it. I do not
> know
>> the exact expected spectra so I cannot do my whole simulation in a
> fixed
>> manner, so I start with a "good" guess and iterate over the above
> process
>> until I get good agreement between data and simulation.
>>
>> I need many statistics and this iteration process would take a very
> long
>> time if I had to rerun the MC for each iteration. That is why I
> choose to
>> do one big run and sample it. I make a TEventList for each ion and
> each
>> energy range and that way my sampling can be very fast and the result
> is
>> simply a list of event numbers to run through my detector simulation
> where I
>> only need to supply this list and the original TChain. The TEventList
> that
>> holds the 1H ions is 77% of all the statistics, so it is huge compared
> to
>> all the others and this is where the problem lies.
>>
>> I will try storing the lists in a TTree as Philippe has suggested. My
> main
>> objective is to keep the execution time as short as possible and the
>> TEventList did a very good job with fewer statistics, so I was trying
> to
>> maintain this scheme with more statistics. If you have any further
>> suggestions I would be greatful.
>>
>> Thanks
>> Kerry
>>
>> -----Original Message-----
>> From: Rene Brun
>> To: LEE, KERRY T. (JSC-SF) (UHCL)
>> Cc: 'Philippe Canal '; 'owner-roottalk_at_pcroot.cern.ch ';
>> 'roottalk_at_pcroot.cern.ch '
>> Sent: 2/3/2006 11:30 AM
>> Subject: RE: [ROOT] TEventList::Enter(), 2 issues
>>
>> Kerry,
>>
>> What is the point of having a TEventlist of 200 million entries?
>>
>> Rene
>>
>> On Fri, 3
>> Feb 2006, LEE, KERRY T. (JSC-SF) (UHCL) wrote:
>>
>>> Thanks Philippe,
>>>
>>> Resizing works. I was trying to avoid writing to a new tree to
>> increase
>>> speed. Now the problem is that when I try to write the very large
>>> TEventList to a TFile via TFile::Write(), I get the same pointer
>> error. Is
>>> there a way to keep TFile from trying to allocate so much memory? I
>> tried
>>> using TObject::kOverwrite, but that did not help. Also, did you see
>> the
>>> second issue with TEventList::Enter() that I mentioned in my first
>> mail?
>>>
>>> Thanks
>>> Kerry
>>>
>>> -----Original Message-----
>>> From: owner-roottalk_at_pcroot.cern.ch
>>> To: LEE, KERRY T. (JSC-SF) (UHCL); roottalk_at_pcroot.cern.ch
>>> Sent: 2/2/2006 12:25 PM
>>> Subject: RE: [ROOT] TEventList::Enter(), 2 issues
>>>
>>> Hi,
>>>
>>> The entries in the TEventList are stored as Long64_t. When reaching
>>> 105000000, TEventList will allocates to holf twice than many entry
>>> and requires more than 1.600Gb of memory.
>>>
>>> There are some technics will could use to mitigate the issue (aka
>> allow
>>> for sligtly bigger event list) but given the your pattern of use,
>>> TEventList is no longer appropriate.
>>>
>>> Instead I recommend that you store the data in another TTree.
>>>
>>> If your limit is really 200 million, you can also make it work by
>>> pre-setting the size (this avoid the need to copy and to hold in
>>> memory 2 gigantic buffer):
>>>
>>> root [0] TEventList *E = new TEventList();
>>> root [2] E->Resize(200000000);
>>> root [1] {
>>> end with '}', '@':abort > for(int i=0;i<105000000;i++){
>>> end with '}', '@':abort > E->Enter(i);
>>> end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;
>>> end with '}', '@':abort > }
>>> end with '}', '@':abort > }
>>>
>>> Cheers,
>>> Philippe.
>>>
>>> -----Original Message-----
>>> From: owner-roottalk_at_pcroot.cern.ch
>>> [mailto:owner-roottalk_at_pcroot.cern.ch]
>>> On Behalf Of LEE, KERRY T. (JSC-SF) (UHCL)
>>> Sent: Thursday, February 02, 2006 10:38 AM
>>> To: 'roottalk_at_pcroot.cern.ch'
>>> Subject: [ROOT] TEventList::Enter(), 2 issues
>>>
>>> Dear ROOT team,
>>>
>>> I am using ROOT v5.08 on linux CentOS 4 compiled with gcc 3.4.4.
>>>
>>> First issue:
>>>
>>> I would like to create a very large TEventList with about 200 million
>>> entries. When it reaches just under 105 millon entries I get an
>> error.
>>> I
>>> have plenty of physical memory (2GB + 2GB swap) to perform this task.
>> A
>>> session below shows the problem. On the other hand, I have no problem
>>> creating an array of type Long64_t with 200 million entries.
>>>
>>> *******************************************
>>> * *
>>> * W E L C O M E to R O O T *
>>> * *
>>> * Version 5.08/00 13 December 2005 *
>>> * *
>>> * You are welcome to visit our Web site *
>>> * http://root.cern.ch *
>>> * *
>>> *******************************************
>>>
>>> FreeType Engine v2.1.9 used to render TrueType fonts.
>>> Compiled on 20 December 2005 for linux with thread support.
>>>
>>> CINT/ROOT C/C++ Interpreter version 5.16.5, November 30 2005
>>> Type ? for help. Commands must be C++ statements.
>>> Enclose multiple statements between { }.
>>> root [0] TEventList *E = new TEventList();
>>> root [1] {
>>> end with '}', '@':abort > for(int i=0;i<105000000;i++){
>>> end with '}', '@':abort > E->Enter(i);
>>> end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;
>>> end with '}', '@':abort > }
>>> end with '}', '@':abort > }
>>> 104857599Syntax Error: * (tmpfile):3:
>>> Error: Illegal pointer operation (tovalue) (tmpfile):3:
>>> *** Interpreter error recovered ***
>>>
>>>
>>> Second issue:
>>>
>>>
>>> I would like to add random integers to a TEventList, but it seems
> that
>>> only higher values than the previous one entered into the TEventList
>> is
>>> allowed.
>>>
>>> root [11] TEventList *lll = new TEventList()
>>> root [12] lll->Enter(1)
>>> root [13] lll->GetEntry(0)
>>> (const Long64_t)1
>>> root [14] lll->Enter(5)
>>> root [16] lll->GetEntry(1)
>>> (const Long64_t)5
>>> root [17] lll->Enter(3)
>>> root [18] lll->GetEntry(2)
>>> (const Long64_t)5
>>>
>>> As you can see adding the number 3 to the list after adding 5
> produces
>>> an entry of 5. Is this expected, or is this a bug? If for some
>> reason
>>> this is not allowed an error message would be very useful. I can
> only
>>> assume, since there exists a TEventList::Sort() function this is not
>> the
>>> desired behavior.
>>>
>>> The purpose for doing this is to randomly sample a ROOT tree without
>>> having to know anything about it except the total number of entries.
>> I
>>> would like to save this TEventList to a file, then read it and the
>> TTree
>>> back into an interacive ROOT session.
>>>
>>> Thanks
>>> Kerry
>>>
>>
>
Received on Sun Feb 05 2006 - 08:51:23 MET