RE: TEventList::Enter(), 2 issues

From: LEE, KERRY T. (JSC-SF) (UHCL) <kerry.t.lee1_at_jsc.nasa.gov>
Date: Fri, 3 Feb 2006 15:01:57 -0600


Hi Rene,

This suggestion sounds great, but I'm not sure how to implement it. Currently I have a TEventList for each ion and each energy bin (>14000 lists). I could create two TBits objects, one to represent Ions and the other energy bin, and then set the appropriate bit in each TBits object for each entry in my TChain, but this would require 47 bits for the ions and about 300bits for the energy bins, or about 44bytes/entry as opposed to the 8bytes/entry for the TEventList method.

This must not be what you had in mind, or maybe I did not make it clear that I had a large number of TEventList objects.

Thanks
Kerry

-----Original Message-----
From: Rene Brun
To: LEE, KERRY T. (JSC-SF) (UHCL)
Cc: 'Rene Brun '; ''Philippe Canal ' '; ''owner-roottalk_at_pcroot.cern.ch ' '; ''roottalk_at_pcroot.cern.ch ' '
Sent: 2/3/2006 1:40 PM
Subject: RE: [ROOT] TEventList::Enter(), 2 issues

Hi Kerry,

If you want to make minimal changes to your program, I suggest to use a TBits object instead of a TEventList. You will gain a factor 64 in memory and the speed should be equivalent (if not better). Fill the TBits object somewhow in the same way that you fill the TEventList. Then, make the event loop yourself, finding the next bit set via "UInt_t FirstSetBit(UInt_t startBit=0)"

Rene Brun

On
Fri, 3
Feb 2006, LEE, KERRY T. (JSC-SF) (UHCL) wrote:

> Rene,
>
> I am attempting a rather complex proceedure to avoid running my monte
carlo
> many times. I will try to describe my method and reason as briefly as
> possible.
>
> I create particles of all types of ions from 1H-58Fe, all with a
separate
> energy spectrum and run these through my MC and store the calculated
energy
> deposition in regions of interest. I then store all the results in
TTrees
> and then gather them into a single TChain. I now want to sample all
these
> ions according to some abuandance scheme, then also sample certain
energy
> ranges in each spectra with different weights. Then the results of
this
> random sampling go into my Detector simulation algorithm to produce
data of
> the same form as the raw data so I can do my analysis on it. I do not
know
> the exact expected spectra so I cannot do my whole simulation in a
fixed
> manner, so I start with a "good" guess and iterate over the above
process
> until I get good agreement between data and simulation.
>
> I need many statistics and this iteration process would take a very
long
> time if I had to rerun the MC for each iteration. That is why I
choose to
> do one big run and sample it. I make a TEventList for each ion and
each
> energy range and that way my sampling can be very fast and the result
is
> simply a list of event numbers to run through my detector simulation
where I
> only need to supply this list and the original TChain. The TEventList
that
> holds the 1H ions is 77% of all the statistics, so it is huge compared
to
> all the others and this is where the problem lies.
>
> I will try storing the lists in a TTree as Philippe has suggested. My
main
> objective is to keep the execution time as short as possible and the
> TEventList did a very good job with fewer statistics, so I was trying
to
> maintain this scheme with more statistics. If you have any further
> suggestions I would be greatful.
>
> Thanks
> Kerry
>
> -----Original Message-----
> From: Rene Brun
> To: LEE, KERRY T. (JSC-SF) (UHCL)
> Cc: 'Philippe Canal '; 'owner-roottalk_at_pcroot.cern.ch ';
> 'roottalk_at_pcroot.cern.ch '
> Sent: 2/3/2006 11:30 AM
> Subject: RE: [ROOT] TEventList::Enter(), 2 issues
>
> Kerry,
>
> What is the point of having a TEventlist of 200 million entries?
>
> Rene
>
> On Fri, 3
> Feb 2006, LEE, KERRY T. (JSC-SF) (UHCL) wrote:
>
>> Thanks Philippe,
>>
>> Resizing works. I was trying to avoid writing to a new tree to
> increase
>> speed. Now the problem is that when I try to write the very large
>> TEventList to a TFile via TFile::Write(), I get the same pointer
> error. Is
>> there a way to keep TFile from trying to allocate so much memory? I
> tried
>> using TObject::kOverwrite, but that did not help. Also, did you see
> the
>> second issue with TEventList::Enter() that I mentioned in my first
> mail?
>>
>> Thanks
>> Kerry
>>
>> -----Original Message-----
>> From: owner-roottalk_at_pcroot.cern.ch
>> To: LEE, KERRY T. (JSC-SF) (UHCL); roottalk_at_pcroot.cern.ch
>> Sent: 2/2/2006 12:25 PM
>> Subject: RE: [ROOT] TEventList::Enter(), 2 issues
>>
>> Hi,
>>
>> The entries in the TEventList are stored as Long64_t. When reaching
>> 105000000, TEventList will allocates to holf twice than many entry
>> and requires more than 1.600Gb of memory.
>>
>> There are some technics will could use to mitigate the issue (aka
> allow
>> for sligtly bigger event list) but given the your pattern of use,
>> TEventList is no longer appropriate.
>>
>> Instead I recommend that you store the data in another TTree.
>>
>> If your limit is really 200 million, you can also make it work by
>> pre-setting the size (this avoid the need to copy and to hold in
>> memory 2 gigantic buffer):
>>
>> root [0] TEventList *E = new TEventList();
>> root [2] E->Resize(200000000);
>> root [1] {
>> end with '}', '@':abort > for(int i=0;i<105000000;i++){
>> end with '}', '@':abort > E->Enter(i);
>> end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;
>> end with '}', '@':abort > }
>> end with '}', '@':abort > }
>>
>> Cheers,
>> Philippe.
>>
>> -----Original Message-----
>> From: owner-roottalk_at_pcroot.cern.ch
>> [mailto:owner-roottalk_at_pcroot.cern.ch]
>> On Behalf Of LEE, KERRY T. (JSC-SF) (UHCL)
>> Sent: Thursday, February 02, 2006 10:38 AM
>> To: 'roottalk_at_pcroot.cern.ch'
>> Subject: [ROOT] TEventList::Enter(), 2 issues
>>
>> Dear ROOT team,
>>
>> I am using ROOT v5.08 on linux CentOS 4 compiled with gcc 3.4.4.
>>
>> First issue:
>>
>> I would like to create a very large TEventList with about 200 million
>> entries. When it reaches just under 105 millon entries I get an
> error.
>> I
>> have plenty of physical memory (2GB + 2GB swap) to perform this task.
> A
>> session below shows the problem. On the other hand, I have no problem
>> creating an array of type Long64_t with 200 million entries.
>>
>> *******************************************
>> * *
>> * W E L C O M E to R O O T *
>> * *
>> * Version 5.08/00 13 December 2005 *
>> * *
>> * You are welcome to visit our Web site *
>> * http://root.cern.ch *
>> * *
>> *******************************************
>>
>> FreeType Engine v2.1.9 used to render TrueType fonts.
>> Compiled on 20 December 2005 for linux with thread support.
>>
>> CINT/ROOT C/C++ Interpreter version 5.16.5, November 30 2005
>> Type ? for help. Commands must be C++ statements.
>> Enclose multiple statements between { }.
>> root [0] TEventList *E = new TEventList();
>> root [1] {
>> end with '}', '@':abort > for(int i=0;i<105000000;i++){
>> end with '}', '@':abort > E->Enter(i);
>> end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;
>> end with '}', '@':abort > }
>> end with '}', '@':abort > }
>> 104857599Syntax Error: * (tmpfile):3:
>> Error: Illegal pointer operation (tovalue) (tmpfile):3:
>> *** Interpreter error recovered ***
>>
>>
>> Second issue:
>>
>>
>> I would like to add random integers to a TEventList, but it seems
that
>> only higher values than the previous one entered into the TEventList
> is
>> allowed.
>>
>> root [11] TEventList *lll = new TEventList()
>> root [12] lll->Enter(1)
>> root [13] lll->GetEntry(0)
>> (const Long64_t)1
>> root [14] lll->Enter(5)
>> root [16] lll->GetEntry(1)
>> (const Long64_t)5
>> root [17] lll->Enter(3)
>> root [18] lll->GetEntry(2)
>> (const Long64_t)5
>>
>> As you can see adding the number 3 to the list after adding 5
produces
>> an entry of 5. Is this expected, or is this a bug? If for some
> reason
>> this is not allowed an error message would be very useful. I can
only
>> assume, since there exists a TEventList::Sort() function this is not
> the
>> desired behavior.
>>
>> The purpose for doing this is to randomly sample a ROOT tree without
>> having to know anything about it except the total number of entries.
> I
>> would like to save this TEventList to a file, then read it and the
> TTree
>> back into an interacive ROOT session.
>>
>> Thanks
>> Kerry
>>
>
Received on Fri Feb 03 2006 - 22:06:15 MET

This archive was generated by hypermail 2.2.0 : Mon Jan 01 2007 - 16:31:57 MET