Re: TTree memory usage

> Do I understand right that this means that in v5.28, the memory use
> of the process is going to be roughly the uncompressed size of the
> largest basket read so far?

More or so, it is actually between one time and 2 times the uncompressed size of the largest basket due to the scratch buffer (eliminated in v5.30).

 > each basket in my input files contains a handful of events (is this a possible configuration?)

Yes and yes it is possible to configure it. To reduce the memory use call tree->SetAutoFlush( - compressed_size); which will flush each basket approximatively when the compressed size reach the (inverse) of the value. In turn this will be use to allocate enough total basket size to hold in memory all the events needed to create (in average) 'compressed_size' worth of compressed data.

This configuration is only useful when done at write time. The default value is 30Mb and thus the default memory use is 30Mb multiplied by the compression ratio of your file (times 2 in v5.28). [that does not include the data objects themselves).

Cheers,
Philippe.

On 8/12/11 2:18 PM, Philip Rodrigues wrote:
> Hi Philippe,
>
> Thanks for the explanations - that makes sense. Do I understand right that
> this means that in v5.28, the memory use of the process is going to be roughly
> the uncompressed size of the largest basket read so far? That would explain
> the memory use pattern in my plot.
>
>>> I tried fiddling around with TTree:SetMaxVirtualSize
>> Increasing this value would allow the TTree to hold to more than one basket
>> at a time hence increase memory consumption and possibly improving random
>> access within the TTree. Decreasing its value would not reduce the actual
>> usage below one basket per (used) branch.
> Ah, I see - that does explain what I saw. If I'm understanding the contents of
> TBranch::GetBasketEntry() correctly, each basket in my input files contains a
> handful of events (is this a possible configuration? My understanding of how
> branches, baskets and entries relate is a little fuzzy), so the only way to
> reduce the peak memory usage would be to reduce the number of entries per
> basket. Is it possible to do this?
>
> (I guess a better solution to all of these problems is to reduce the size of
> the events [especially the few very large ones], which is something we're
> working on, but now I'm curious about how things work, hence the questions).
>
> Thanks,
> Phil
Received on Fri Aug 12 2011 - 23:09:44 CEST