RE: hadd: "too many open files"

From: Fine, Valeri <fine_at_bnl.gov>
Date: Mon, 8 Aug 2011 00:21:57 -0400

> The current scheme comes

Well, there are really the different "use cases". The STAR macro targeted the "use case" to merge about 1000's files produced by 1000's computing nodes into one bigger file. The large file was the mandatory requirement of using the mass storage system like HPSS effectively. There was no "deep hierarchy". The files were "local". The time to merge was O(n).

On the other hand, it was no feasible to open 1000's files by the merging application simultaneously (using the STAR computers) . In addition in "our" case the files were created in parallel, merged and discarded right away. In fact, the merging application had no idea and did not care how many files it is to merge.  

Best regards , Valeri Fine

---

Brookhaven National Laboratory
Upton, NY 11973
Phone +1-631-344-7806

> -----Original Message-----
> From: owner-roottalk_at_root.cern.ch [mailto:owner-roottalk_at_root.cern.ch]
On Behalf Of
> Philippe Canal
> Sent: Sunday, August 07, 2011 7:52 PM
> To: Noel Dawe
> Cc: roottalk_at_cern.ch
> Subject: Re: [ROOT] hadd: "too many open files"
>
> Hi Noel,
>
> The current scheme comes from 2 observation, one being that opening a
file is comparitively
> slow especially if the file is not local.
> The 2nd is that it is more efficient time wise to get one object to be
merged and then merge
> into this object the equivalent objects from all the remaining files
and then to move on to the
> next object/directory. This is particular helpful with deep directory
hierarchy are its reduced
> the number of traversal that are needed.
>
> Cheers,
> Philippe.
>
> On 8/6/11 5:19 AM, Noel Dawe wrote:
> > I don't know why hadd needs to open all the files at the same time
but
> > probably a better way to write this tool would be to never open more
> > than two files at once: copy the first file to the destination and
keep it open, then pop off
> the next file, open it, merge it into the first, close it, then pop
off the next file and open it,
> etc...
> >
> > Noel
Received on Mon Aug 08 2011 - 06:22:08 CEST