TProofOutputFile on LSF

ROOT
1

Dear experts!

UPD: after yet another restart of the server the problem solved itself :slight_smile: I get the files to the local directory. (For somw reason I have to merge them manually with hadd, but that’s not very hard :slight_smile:

I’m trying to run PoD 3.16 on lxplus using LSF. After first try, I realized, that my output is too big and I need to store it locally on workers, so I tried to implement the tip from http://pod.gsi.de/doc/nightly/Tips.html about TProofOutputFile usage.

When running on PoD the files “BCut0.0.root”, “BCut.0.1.root”, … are created in the specified directory but are empty. I’ve checked my implementation with Proof-Lite and TProof::AddEnvVar("LOCALDATASERVER", "file://"); and it produces files with histograms in them and merges them.

I added the line “xrd.rootd allow” to xrd configuration and I see it in .PoD/log/PoDServer/xpd.log

Extracts from my Selector:

void SlaveBegin(){
        proof_file = new TProofOutputFile("BCut<ord>.root", "M");
        proof_file->SetOutputFileName("/afs/cern.ch/user/d/dmironov/my_afs/selector/BCut<ord>.root");

        fOutput->Add(proof_file);

        if (!(local_file = proof_file->OpenFile("RECREATE"))) {
	   Warning("SlaveBegin", "problems opening file: %s/%s",
	   proof_file->GetDir(), proof_file->GetFileName());
	}

}

void SlaveTerminate()
{
        local_file->cd();

        TIter out_iter(histos);
        while(TObject* out = out_iter()){
                out->Write();
        }
        proof_file->Print();
        local_file->Close();
}

In one of workers logs I have: <TProofOutputFile::Print>: dir: rootd://dmironov@b62fe9cf02.cern.ch:21001//pool/lsf/dmironov/621989453.3/PoDWorker_6qrrBnoVoW/proof/dmironov/session-lxplus0033-1423747505-790/worker-0.0-b62fe9cf02-1423747506-21684/

When I manually try TFile *rb = TFile::Open("rootd://dmironov@b62fe9cf02.cern.ch:21001//pool/lsf/dmironov/621989453.3/PoDWorker_6qrrBnoVoW/proof/dmironov/session-lxplus0033-1423747505-790/worker-0.0-b62fe9cf02-1423747506-21684/BCut0.0.root")

I get

SysError in <TUnixSystem::UnixTcpConnect>: connect (b62fe9cf02.cern.ch:21001) (Connection refused)
Error in <TFTP::TFTP>: can't open connection to rootd on host b62fe9cf02.cern.ch at port 21001
SysError in <TUnixSystem::UnixTcpConnect>: connect (b62fe9cf02.cern.ch:21001) (Connection refused)
Error in <TNetFile::TNetFile>: can't open connection to rootd on host b62fe9cf02.cern.ch at port 21001
Error in <TNetFile::Create>: authentication attempt unsuccessful
Error in <TNetFile::Create>: failing on file rootd://dmironov@b62fe9cf02.cern.ch:21001//pool/lsf/dmironov/621989453.3/PoDWorker_6qrrBnoVoW/proof/dmironov/session-lxplus0033-1423747505-790/worker-0.0-b62fe9cf02-1423747506-21684/BCut0.0.root

I attach a bit cleaned version of xpd.log

Do you have any ideas, how can I get my output to master?
xpd.log.txt (41.6 KB)

2

Hi,

Sorry for the late reply.
Did you find in the master log any of those failed connection attempts?
Where did you out exactly ‘xpd.rootd allow’ ?
My suspicion is that this is not taken into account, for some reason.

G. Ganis

3

Dear community!

Let me post a workaround here, since I have received an email with this question.

I ended up avoiding usage of TProofOutputFile completely. Instead, I’ve used a usual TFile and let PROOF to deal with that itself. I add histos to fOutput.

I have a special function to get them:

and so I fill them like

This is the Terminate function, where all the things are written:

[code]void BEventsSelector::Terminate()
{
cout << endl;

local_file = new TFile(output_file_name->Data(), "RECREATE"); 

	local_file->cd();

	TIter out_iter(fOutput);
	while(TObject* out = out_iter()){
	    if (TString(out->GetName()).BeginsWith("PROOF")) continue;
		if (out->IsA() != TH1F::Class()) continue;
		TH1F * h = dynamic_cast<TH1F*>(out);
		h->Write();
		//h->SetDirectory(0);
	}

local_file->Close();

}[/code]

Hope that helps.
Cheers,
Dima

4

Dear dmironov,

Alternatively you can do:

proof->Process(dataset, "BEventsSelector.C+", "of=myoutput.root")

See root.cern.ch/handling-outputs#outputfile .

G Ganis