RE: hadd: "too many open files"

From: Vladimir Savinov [mailto:vps3_at_pitt.edu] Sent: Saturday, August 06, 2011 5:22 PM
To: Fine, Valeri
Cc: Noel Dawe; roottalk_at_cern.ch; Sacha.Vaniachine_at_cern.ch; nevski_at_mail.cern.ch
Subject: Re: [ROOT] hadd: "too many open files"  

Valeri,
does this script also merge TTrees?

[Fine, Valeri]

Yes, it does.

And, also, do you know why hadd needs to keep all files open until the end of merging?

[Fine, Valeri] No, I had no time evaluate that. It was done many years
ago. ( I hope one still can find that discussion on RootForum) At the time we (STAR) were looking for the reliable way to merge the large amount of the ROOT files containing histograms and "ttrees" quickly, with no crash.  

In fact this script is a part of the some "complex" system to allow to produce the ROOT files concurrently (using some distributed facility with the common file system, NFS, AFS whatever) and merge them as soon as any single file is closed and ready to be merged. This approach was adopted to eliminate the complex bookkeeping of the intermediate temporary shorter files.

thanks,
v

On 8/6/2011 4:13 PM, Fine, Valeri wrote:

There is no need to open all files to merge it.

For example STAR experiment uses the MergeHistogramFile.C attached.

( it can be found on afs

/afs/rhic.bnl.gov/star/packages/DEV/StRoot/macros/analysis/MergeHistogra mFile.C )

The script opens the input files one by one. Been compiled with ACliC it works much faster than the regular 'hadd"

.

Best regards , Valeri Fine

Brookhaven National Laboratory

Upton, NY 11973

Phone +1-631-344-7806  

From: owner-roottalk_at_root.cern.ch [mailto:owner-roottalk_at_root.cern.ch] On Behalf Of Noel Dawe
Sent: Saturday, August 06, 2011 6:19 AM
To: roottalk_at_cern.ch
Subject: [ROOT] hadd: "too many open files"  

I don't know why hadd needs to open all the files at the same time but probably a better way to write this tool would be to never open more than two files at once: copy the first file to the destination and keep it open, then pop off the next file, open it, merge it into the first, close it, then pop off the next file and open it, etc...

Noel Received on Mon Aug 08 2011 - 01:24:35 CEST