Hi,
I am trying to combine lots of root files together by ‘hadd’ in root. However, I got this error:
(1) each file is small, but the total size is like 1 GB
(2) the computer memory is 2 GB
(3) I checked each file, they are fine (created without error)
It seems that it is due to memory consumption. I really need to combine all those files to one file. Any alternative approach, please?
Thank you,
Zhiyi.
Linux: 2.4
ROOT: v4.04
It is likely that your Tree is a memory resident Tree, you should fix this problem first, please.
Moving to 5.24 you could connect each Tree file and do
TFile f(file1.root,"update");
TTree *T = (TTree*)f.Get("name of your tree");
T->FlushBaskets();
T->Write();
and repeat this opetartion for each file, then you can use hadd.
Rene
Thanks for your message.
You mean this operation is only available for “5.24”? How about other versions like v5.22?
[quote=“brun”]It is likely that your Tree is a memory resident Tree, you should fix this problem first, please.
Moving to 5.24 you could connect each Tree file and do
TFile f(file1.root,"update");
TTree *T = (TTree*)f.Get("name of your tree");
T->FlushBaskets();
T->Write();
and repeat this opetartion for each file, then you can use hadd.
Rene[/quote]
Hi,
FlushBaskets was introduced in v5.20 and since then is called by default in TTree::Write.
Cheers,
Philippe
Dear ROOT Wizards,
I am having the same problem when trying to merge even just two ROOT files (but the output file is large):
hadd md3pdm2x-v5.1-sa4t_data_lq_periodI.root md3pdm2x-v5.1-sa4t_data_lq_periodI1.root md3pdm2x-v5.1-sa4t_data_lq_periodI2.root
Target file: md3pdm2x-v5.1-sa4t_data_lq_periodI.root
Source file 1: md3pdm2x-v5.1-sa4t_data_lq_periodI1.root
Source file 2: md3pdm2x-v5.1-sa4t_data_lq_periodI2.root
Target path: md3pdm2x-v5.1-sa4t_data_lq_periodI.root:/
Found subdirectory MakerD3PDm2x
Target path: md3pdm2x-v5.1-sa4t_data_lq_periodI.root:/MakerD3PDm2x
md3pdm2x-v5.1-sa4t_data_lq_periodI1.root tree:MakerD3PDm2x/anaTree entries=1234567890
md3pdm2x-v5.1-sa4t_data_lq_periodI2.root tree:MakerD3PDm2x/anaTree entries=1234567890
terminate called after throwing an instance of 'std::bad_alloc’
what(): std::bad_alloc
Abort (core dumped)
The sizes of input files:
3403712 md3pdm2x-v5.1-sa4t_data_lq_periodI1.root
2289104 md3pdm2x-v5.1-sa4t_data_lq_periodI2.root
SLC5 / 64bit 24GB of real memory
root_v5.27.06.Linux-slc5-gcc4.3
Applying this fix:
TFile f(file,“update”);
TTree T = (TTree)f.Get(“MakerD3PDm2x/anaTree”);
T->FlushBaskets();
T->Write();
f.Close();
to each of the two input files didn’t help,
could you please help?
thanks!
Vladimir
p.s. using root_v5.27.06.Linux-slc5_amd64-gcc4.3 solved my immediate problem
Hi,
If you tell me where to find those 2 files, I will attempt to reproduce the problem.
Cheers,
Philippe.
Philippe,
these files are on a local computer in my office -
I would need to copy them to elsewhere,
and I will do so, but it’s not clear to me there is
enough room on afs at CERN for that.
There is one more piece of info:
when I was (successfully) merging my huge TTrees using hadd
(64bit ROOT) I noticed that the trees were definitely being
merged in memory: my machine has 24 real GB, and, when the
total size of files to be merged approached approx. 20-22 GB,
the machine started swappping like crazy and, after an hour of
not even being able to log in, I was able to recover by Ctrl/C
ROOT session that was doing the merging. My interpretation is
that ROOT was trying to merge my TTrees in RAM, though I tried
the prescription suggested by Rene earlier in this thread -
as soon as the files are at CERN, I will let you know -
thanks,
v
[quote=“pcanal”]Hi,
If you tell me where to find those 2 files, I will attempt to reproduce the problem.
Cheers,
Philippe.[/quote]
Hi,
[quote]My interpretation is that ROOT was trying to merge my TTrees in RAM,
[/quote]This should not really happen …
Can you send me the result of tree->Print(); and myfile->Map(); for one of the two files …
Thanks,
Philippe.
[quote=“pcanal”]Hi,
[quote]My interpretation is that ROOT was trying to merge my TTrees in RAM,
[/quote]This should not really happen …
Can you send me the result of tree->Print(); and myfile->Map(); for one of the two files …
Thanks,
Philippe.[/quote]
Philippe, the files are huge, but I can let you log in to this computer, if you could help that way - please let me know.
now, the result of Print and Map:
everything is in proxy.phyast.pitt.edu/~savinov/tmp/ROOT/ ,
more specifically:
proxy.phyast.pitt.edu/~savinov/t … _print.log
this is an ENORMOUS file, I had to ctrl/c!:
proxy.phyast.pitt.edu/~savinov/t … le_map.log
btw, now, this is how I “fixed” them before trying to hadd:
void fix_all_files() {
fix_one_file("./md3pdm2x-v5.1-sa4t_data_lq_periodI1.root");
fix_one_file("./md3pdm2x-v5.1-sa4t_data_lq_periodI2.root");
}
void fix_one_file ( Char_t* file, Char_t* source = “MakerD3PDm2x/anaTree”) {
cout << "Will try to fix TTree " << source << " from file " << file << endl;
TFile f(file,“update”);
TTree T = (TTree)f.Get(source);
T->FlushBaskets();
T->Write();
f.Close();
return;
}
thanks,
v
Hi,
[quote]btw, now, this is how I “fixed” them before trying to hadd:[/quote]Is the problem happening before or after you did the fix? Are the log before or after you did the fix?
Philippe.
[quote=“pcanal”]Hi,
[quote]btw, now, this is how I “fixed” them before trying to hadd:[/quote]Is the problem happening before or after you did the fix? Are the log before or after you did the fix?
Philippe.[/quote]
the problem happens regardless of me “fixing” or not “fixing” the files.
(I am “fixing” copies, not the originals) -
the results
of
map and print are shown after “fixing” the files -
btw, I expected this “fixing” to take a lot of time, but “fixing” each file takes just a few seconds…
thanks!
v
[quote]Philippe, the files are huge, but I can let you log in to this computer, if you could help that way - please let me know[/quote]So this seems like the only way to make progress. My email is pcanal@fnal.gov.
Cheers,
Philippe.
Hi Vladimir,
Thank you very much for reporting this problem (a significant memory in the TTreeClone in the development release v5.27/06) which is now fixed by revision 37405 of the trunk.
Cheers,
Philippe.
Philippe,
thank you very much for fixing this bug!
Just for fun, earlier today I used the new hadd
to merge two sets of files and successfully did
so for output file size exceeding 30GB!
thanks!
v