RE: TEventList::Enter(), 2 issues

From: Philippe Canal <pcanal_at_fnal.gov>
Date: Fri, 03 Feb 2006 11:16:33 -0600

Hi Kerry,

There is no way to avoid the allocation of more memory by the TFile. This is because it has to create a buffer where to store the properly layed-out (and byte-swapped) version of the in-memory object.

Again you could still store the TEventList as a serie of entrie in a new TTree to avoid allocating to much memory.

I am looking at your other issue.

Cheers,
Philippe.

-----Original Message-----
From: LEE, KERRY T. (JSC-SF) (UHCL) [mailto:kerry.t.lee1_at_jsc.nasa.gov] Sent: Friday, February 03, 2006 10:58 AM To: 'Philippe Canal '; 'owner-roottalk_at_pcroot.cern.ch '; 'roottalk_at_pcroot.cern.ch '
Subject: RE: [ROOT] TEventList::Enter(), 2 issues

 Thanks Philippe,

Resizing works. I was trying to avoid writing to a new tree to increase speed. Now the problem is that when I try to write the very large TEventList to a TFile via TFile::Write(), I get the same pointer error. Is there a way to keep TFile from trying to allocate so much memory? I tried using TObject::kOverwrite, but that did not help. Also, did you see the second issue with TEventList::Enter() that I mentioned in my first mail?

Thanks
Kerry

-----Original Message-----
From: owner-roottalk_at_pcroot.cern.ch
To: LEE, KERRY T. (JSC-SF) (UHCL); roottalk_at_pcroot.cern.ch Sent: 2/2/2006 12:25 PM
Subject: RE: [ROOT] TEventList::Enter(), 2 issues

Hi,

The entries in the TEventList are stored as Long64_t. When reaching 105000000, TEventList will allocates to holf twice than many entry and requires more than 1.600Gb of memory.

There are some technics will could use to mitigate the issue (aka allow for sligtly bigger event list) but given the your pattern of use, TEventList is no longer appropriate.

Instead I recommend that you store the data in another TTree.

If your limit is really 200 million, you can also make it work by pre-setting the size (this avoid the need to copy and to hold in memory 2 gigantic buffer):    

root [0] TEventList *E = new TEventList();
root [2] E->Resize(200000000);
root [1] {
end with '}', '@':abort > for(int i=0;i<105000000;i++){
end with '}', '@':abort > E->Enter(i);
end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;

end with '}', '@':abort > }
end with '}', '@':abort > }

Cheers,
Philippe.

-----Original Message-----
From: owner-roottalk_at_pcroot.cern.ch
[mailto:owner-roottalk_at_pcroot.cern.ch]
On Behalf Of LEE, KERRY T. (JSC-SF) (UHCL) Sent: Thursday, February 02, 2006 10:38 AM To: 'roottalk_at_pcroot.cern.ch'
Subject: [ROOT] TEventList::Enter(), 2 issues

Dear ROOT team,

I am using ROOT v5.08 on linux CentOS 4 compiled with gcc 3.4.4.

First issue:

I would like to create a very large TEventList with about 200 million entries. When it reaches just under 105 millon entries I get an error. I
have plenty of physical memory (2GB + 2GB swap) to perform this task. A session below shows the problem. On the other hand, I have no problem creating an array of type Long64_t with 200 million entries.

---

• *
• W E L C O M E to R O O T *
• *
• Version 5.08/00 13 December 2005 *
• *
• You are welcome to visit our Web site *
• http://root.cern.ch *
• *

  ---

FreeType Engine v2.1.9 used to render TrueType fonts. Compiled on 20 December 2005 for linux with thread support.

CINT/ROOT C/C++ Interpreter version 5.16.5, November 30 2005 Type ? for help. Commands must be C++ statements. Enclose multiple statements between { }. root [0] TEventList *E = new TEventList(); root [1] {

end with '}', '@':abort > for(int i=0;i<105000000;i++){
end with '}', '@':abort > E->Enter(i);
end with '}', '@':abort > if(i>104500000)cout<<"\r"<<i<<flush;
end with '}', '@':abort > }
end with '}', '@':abort > }

104857599Syntax Error: * (tmpfile):3:
Error: Illegal pointer operation (tovalue) (tmpfile):3: *** Interpreter error recovered ***

Second issue:

I would like to add random integers to a TEventList, but it seems that only higher values than the previous one entered into the TEventList is allowed.

root [11] TEventList *lll = new TEventList()
root [12] lll->Enter(1)
root [13] lll->GetEntry(0)

(const Long64_t)1
root [14] lll->Enter(5)
root [16] lll->GetEntry(1)
(const Long64_t)5
root [17] lll->Enter(3)
root [18] lll->GetEntry(2)
(const Long64_t)5

As you can see adding the number 3 to the list after adding 5 produces an entry of 5. Is this expected, or is this a bug? If for some reason this is not allowed an error message would be very useful. I can only assume, since there exists a TEventList::Sort() function this is not the desired behavior.

The purpose for doing this is to randomly sample a ROOT tree without having to know anything about it except the total number of entries. I would like to save this TEventList to a file, then read it and the TTree back into an interacive ROOT session.

Thanks
Kerry Received on Fri Feb 03 2006 - 18:23:40 MET