RE: TEventList::Enter(), 2 issues

Rene,

I am attempting a rather complex proceedure to avoid running my monte carlo many times. I will try to describe my method and reason as briefly as possible.

I create particles of all types of ions from 1H-58Fe, all with a separate energy spectrum and run these through my MC and store the calculated energy deposition in regions of interest. I then store all the results in TTrees and then gather them into a single TChain. I now want to sample all these ions according to some abuandance scheme, then also sample certain energy ranges in each spectra with different weights. Then the results of this random sampling go into my Detector simulation algorithm to produce data of the same form as the raw data so I can do my analysis on it. I do not know the exact expected spectra so I cannot do my whole simulation in a fixed manner, so I start with a "good" guess and iterate over the above process until I get good agreement between data and simulation.

I need many statistics and this iteration process would take a very long time if I had to rerun the MC for each iteration. That is why I choose to do one big run and sample it. I make a TEventList for each ion and each energy range and that way my sampling can be very fast and the result is simply a list of event numbers to run through my detector simulation where I only need to supply this list and the original TChain. The TEventList that holds the 1H ions is 77% of all the statistics, so it is huge compared to all the others and this is where the problem lies.

I will try storing the lists in a TTree as Philippe has suggested. My main objective is to keep the execution time as short as possible and the TEventList did a very good job with fewer statistics, so I was trying to maintain this scheme with more statistics. If you have any further suggestions I would be greatful.

Thanks
Kerry

-----Original Message-----
From: Rene Brun
To: LEE, KERRY T. (JSC-SF) (UHCL)
Cc: 'Philippe Canal '; 'owner-roottalk_at_pcroot.cern.ch '; 'roottalk_at_pcroot.cern.ch '
Sent: 2/3/2006 11:30 AM
Subject: RE: [ROOT] TEventList::Enter(), 2 issues

Kerry,

What is the point of having a TEventlist of 200 million entries?

Rene

On Fri, 3
Feb 2006, LEE, KERRY T. (JSC-SF) (UHCL) wrote:

> Thanks Philippe,
>
> Resizing works. I was trying to avoid writing to a new tree to
increase
> speed. Now the problem is that when I try to write the very large
> TEventList to a TFile via TFile::Write(), I get the same pointer
error. Is
> there a way to keep TFile from trying to allocate so much memory? I
tried
> using TObject::kOverwrite, but that did not help. Also, did you see
the
> second issue with TEventList::Enter() that I mentioned in my first
mail?
>
> Thanks
> Kerry
>
> -----Original Message-----
> From: owner-roottalk_at_pcroot.cern.ch
> To: LEE, KERRY T. (JSC-SF) (UHCL); roottalk_at_pcroot.cern.ch
> Sent: 2/2/2006 12:25 PM
> Subject: RE: [ROOT] TEventList::Enter(), 2 issues
>
> Hi,
>
> The entries in the TEventList are stored as Long64_t. When reaching
> 105000000, TEventList will allocates to holf twice than many entry
> and requires more than 1.600Gb of memory.
>
> There are some technics will could use to mitigate the issue (aka
allow
> for sligtly bigger event list) but given the your pattern of use,
> TEventList is no longer appropriate.
>
> Instead I recommend that you store the data in another TTree.
>
> If your limit is really 200 million, you can also make it work by
> pre-setting the size (this avoid the need to copy and to hold in
> memory 2 gigantic buffer):
>
> root [0] TEventList *E = new TEventList();
> root [2] E->Resize(200000000);
> root [1] {
> end with '}', '@':abort > for(int i=0;i<105000000;i++){
> end with '}', '@':abort > E->Enter(i);
> end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;
> end with '}', '@':abort > }
> end with '}', '@':abort > }
>
> Cheers,
> Philippe.
>
> -----Original Message-----
> From: owner-roottalk_at_pcroot.cern.ch
> [mailto:owner-roottalk_at_pcroot.cern.ch]
> On Behalf Of LEE, KERRY T. (JSC-SF) (UHCL)
> Sent: Thursday, February 02, 2006 10:38 AM
> To: 'roottalk_at_pcroot.cern.ch'
> Subject: [ROOT] TEventList::Enter(), 2 issues
>
> Dear ROOT team,
>
> I am using ROOT v5.08 on linux CentOS 4 compiled with gcc 3.4.4.
>
> First issue:
>
> I would like to create a very large TEventList with about 200 million
> entries. When it reaches just under 105 millon entries I get an
error.
> I
> have plenty of physical memory (2GB + 2GB swap) to perform this task.
A
> session below shows the problem. On the other hand, I have no problem
> creating an array of type Long64_t with 200 million entries.
>
> *******************************************
> * *
> * W E L C O M E to R O O T *
> * *
> * Version 5.08/00 13 December 2005 *
> * *
> * You are welcome to visit our Web site *
> * http://root.cern.ch *
> * *
> *******************************************
>
> FreeType Engine v2.1.9 used to render TrueType fonts.
> Compiled on 20 December 2005 for linux with thread support.
>
> CINT/ROOT C/C++ Interpreter version 5.16.5, November 30 2005
> Type ? for help. Commands must be C++ statements.
> Enclose multiple statements between { }.
> root [0] TEventList *E = new TEventList();
> root [1] {
> end with '}', '@':abort > for(int i=0;i<105000000;i++){
> end with '}', '@':abort > E->Enter(i);
> end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;
> end with '}', '@':abort > }
> end with '}', '@':abort > }
> 104857599Syntax Error: * (tmpfile):3:
> Error: Illegal pointer operation (tovalue) (tmpfile):3:
> *** Interpreter error recovered ***
>
>
> Second issue:
>
>
> I would like to add random integers to a TEventList, but it seems that
> only higher values than the previous one entered into the TEventList
is
> allowed.
>
> root [11] TEventList *lll = new TEventList()
> root [12] lll->Enter(1)
> root [13] lll->GetEntry(0)
> (const Long64_t)1
> root [14] lll->Enter(5)
> root [16] lll->GetEntry(1)
> (const Long64_t)5
> root [17] lll->Enter(3)
> root [18] lll->GetEntry(2)
> (const Long64_t)5
>
> As you can see adding the number 3 to the list after adding 5 produces
> an entry of 5. Is this expected, or is this a bug? If for some
reason
> this is not allowed an error message would be very useful. I can only
> assume, since there exists a TEventList::Sort() function this is not
the
> desired behavior.
>
> The purpose for doing this is to randomly sample a ROOT tree without
> having to know anything about it except the total number of entries.
I
> would like to save this TEventList to a file, then read it and the
TTree
> back into an interacive ROOT session.
>
> Thanks
> Kerry
>
Received on Fri Feb 03 2006 - 19:28:59 MET