Re: hadd: "too many open files"

Hi Noel,

The current scheme comes from 2 observation, one being that opening a file is comparitively slow especially if the file is not local. The 2nd is that it is more efficient time wise to get one object to be merged and then merge into this object the equivalent objects from all the remaining files and then to move on to the next object/directory. This is particular helpful with deep directory hierarchy are its reduced the number of traversal that are needed.

Cheers,
Philippe.

On 8/6/11 5:19 AM, Noel Dawe wrote:
> I don't know why hadd needs to open all the files at the same time but probably a better way to write this tool would be to never
> open more than two files at once: copy the first file to the destination and keep it open, then pop off the next file, open it,
> merge it into the first, close it, then pop off the next file and open it, etc...
>
> Noel
Received on Mon Aug 08 2011 - 01:51:50 CEST