Compiled vs interpreted code

Hi experts,
I am trying to use PROOF in compiled code, and everything compiles and runs. All the methods of my TSelector are called: Begin, SlaveBegin, Process, SlaveTerminate and Terminate are all good. And the output tree and file are correctly made and written.

But I get lower performance than single-threaded.

Here is my main function:

#include "MySel.h"
using namespace std;

int main (int argc, char *argv[]) 
{
   const TString inFileName  = argv[1];
   TFile  *inFile = TFile::Open(inFileName);
   TChain *inTree = (TChain*)inFile->Get("usertree");
   TString workingDirectory = TString::Format("%s/", gSystem->WorkingDirectory());
   TString selectorLibrary = workingDirectory + "MySel.so";

   TProof *proof  = TProof::Open("");//"workers=8");
   //proof->SetParallel(0);
   //gProofDebugMask = TProofDebug::kAll;
   //gProofDebugLevel = 5;

   proof->Load(selectorLibrary,kTRUE);
   inTree->SetProof(proof);
   Long64_t nentries=100000; Option_t *option="";
   MySel *s = new MySel();
   
   //All three of these approaches give single-threaded performance
   inTree->Process(s,option,nentries,0);
   //inTree->Process("MySel.C+",option,nentries,0);
   //inTree->Process("MySel",option,nentries,0);
   
   delete s;
   delete inTree;
   inFile->Close();
   return 0;
}

I think that the problem must be in my main program since the methods are all fine. But for completeness I attach the needed files to do exactly what I do.

I’ve tried the three approaches above. But none give a speed-up.

Here is the output at execution time

 +++ Starting PROOF-Lite with 20 workers +++
Opening connections to workers: OK (20 workers)                 
Setting up worker servers: OK (20 workers)                 
PROOF set to parallel mode (20 workers)
11:38:03 21720 Wrk-0.0 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21722 Wrk-0.1 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21736 Wrk-0.8 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21726 Wrk-0.3 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21728 Wrk-0.4 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21724 Wrk-0.2 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21738 Wrk-0.9 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21734 Wrk-0.7 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21740 Wrk-0.10 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21730 Wrk-0.5 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:03 21752 Wrk-0.15 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:04 21748 Wrk-0.14 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:04 21744 Wrk-0.12 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:04 21754 Wrk-0.16 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:04 21760 Wrk-0.19 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:04 21758 Wrk-0.18 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:04 21746 Wrk-0.13 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:04 21742 Wrk-0.11 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:04 21732 Wrk-0.6 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
11:38:04 21756 Wrk-0.17 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
***class instance created*** 
-->Client beginning...
-->Slave beginning...
-->Slave finished.
--->fChain initialized.
-->Terminating... 
--->successfully made the bush.
--->it has # of entries = 99833
~~~class instance destroyed~~~ 

real	0m16.414s
user	0m9.918s
sys	0m4.462s

Is there something basic that I am missing about how to ensure parallel operation with PROOF?

Many thanks!
headers.h (1.05 KB)
main.cpp (769 Bytes)
makefile.txt (1.73 KB)
MySel.C (923 Bytes)
MySel.h (1.64 KB)
LinkDef.h (155 Bytes)

Dear mgv4ce,

PROOF can help in providing more CPU but the bottleneck can be somewhere else, especially if the files do not reside on a fast media, like SSD. Also, with respect to serial processing, there is the additional merging step in the end which can be heavy for large output objects. So the optimal number of workers may not be the maximum available.

All this said, I would start by checking what happens with 1 worker (this gives an idea of the overhead of PROOF in your setup) and 2 and 4 workers which gives hints about the scale up.

Can you also provide the following information:

  • size of the files and type of device where they reside
  • amount of memory available on the machine

G Ganis

Dear Ganis,
The file size is 95MB. It is located on a very fast lustre-based file system, with 64GB of RAM. The disk is a hard disk however, not an SSD. I believe the entire file is cached in RAM anyway after I have run the program a few times.

The information you ask for is below:

For 1 worker:

-bash-4.1$ time ./selector result90000.root
 +++ Starting PROOF-Lite with 1 workers +++
Opening connections to workers: OK (1 workers)                 
Setting up worker servers: OK (1 workers)                 
PROOF set to parallel mode (1 worker)
10:53:32 173776 Wrk-0.0 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
***class instance created*** 
-->Client beginning...
-->Slave beginning...
-->Slave finished.
--->fChain initialized.
-->Terminating... 
--->successfully made the bush.
--->it has # of entries = 99833
~~~class instance destroyed~~~ 

real	0m7.466s
user	0m6.572s
sys	0m0.451s

For 2 workers:

-bash-4.1$ time ./selector result90000.root
+++ Starting PROOF-Lite with 2 workers +++
Opening connections to workers: OK (2 workers)                 
Setting up worker servers: OK (2 workers)                 
PROOF set to parallel mode (2 workers)
10:52:03 173051 Wrk-0.0 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
10:52:03 173053 Wrk-0.1 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
***class instance created*** 
-->Client beginning...
-->Slave beginning...
-->Slave finished.
--->fChain initialized.
-->Terminating... 
--->successfully made the bush.
--->it has # of entries = 99833
~~~class instance destroyed~~~ 

real	0m7.230s
user	0m6.377s
sys	0m0.419s

For 4 workers:

-bash-4.1$ time ./selector result90000.root
 +++ Starting PROOF-Lite with 4 workers +++
Opening connections to workers: OK (4 workers)                 
Setting up worker servers: OK (4 workers)                 
PROOF set to parallel mode (4 workers)
10:55:32 174821 Wrk-0.0 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
10:55:32 174828 Wrk-0.3 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
10:55:32 174825 Wrk-0.2 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
10:55:32 174823 Wrk-0.1 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
***class instance created*** 
-->Client beginning...
-->Slave beginning...
-->Slave finished.
--->fChain initialized.
-->Terminating... 
--->successfully made the bush.
--->it has # of entries = 99833
~~~class instance destroyed~~~ 

real	0m7.201s
user	0m6.213s
sys	0m0.424s

I also use “top” command to see my usage during execution, and this is what I get:

PID        USER      PR  NI  VIRT  RES  SHR S    %CPU %MEM    TIME+  COMMAND
171639 mgv4ce    20   0  413m 103m  20m R 41.0  0.2   0:01.25 selector                                                                                                            
171644 mgv4ce    20   0  141m  28m  15m S  5.2    0.0   0:00.16 proofserv.exe                                                                                                       
171642 mgv4ce    20   0  141m  26m  15m S  4.9    0.0   0:00.15 proofserv.exe                                                                                                       
171646 mgv4ce    20   0  141m  26m  15m S  4.9    0.0   0:00.15 proofserv.exe                                                                                                       
171648 mgv4ce    20   0  141m  27m  15m S  4.9    0.0   0:00.15 proofserv.exe    

I can also say that, subjectively, most of the time comes after the initialization of the fChain, so during the processing of the events. There are 100K events in the input and output tree.

Many thanks!

Dear mgv4ce,

PROOF is not used in your case.
I guess it is because you force the tree to be a chain (I did not notice before). Try this:

int main (int argc, char *argv[]) 
{
   const TString inFileName  = argv[1];

//   TFile  *inFile = TFile::Open(inFileName);
//   TChain *inTree = (TChain*)inFile->Get("usertree");
   TChain *inTree = new TChain("usertree");
   inTree->Add(inFileName);
   TString workingDirectory = TString::Format("%s/", gSystem->WorkingDirectory());
   TString selectorLibrary = workingDirectory + "MySel.so";

   TProof *proof  = TProof::Open("workers=2");//"workers=8");
   //proof->SetParallel(0);
   gProofDebugMask = TProofDebug::kAll;
   gProofDebugLevel = 5;
   proof->Load(selectorLibrary,kTRUE);
   inTree->SetProof(proof);
   Long64_t nentries=100000; Option_t *option="";
   MySel *s = new MySel();
   inTree->Process(s,option,nentries,0);
   delete s;
   delete inTree;
  // inFile->Close();
   return 0;

PROOF takes care of opening the file.

G Ganis

Dear Ganis,
Thanks! This works very well if I use the “MySel” argument instead of the actual TSelector. That is

   TChain *inTree = new TChain("usertree");
   inTree->Add(inFileName);
   
   TString workingDirectory = TString::Format("%s/", gSystem->WorkingDirectory());
   TString selectorLibrary = workingDirectory + "MySel.so";

   TProof *proof  = TProof::Open("workers=8");
   proof->Load(selectorLibrary,kTRUE);
   inTree->SetProof(proof);
   MySel *s = new MySel();
   Long64_t nentries=100000; Option_t *option="";
   inTree->Process("MySel",option,nentries,0);  //this works
   //inTree->Process(s,option,nentries,0);        //this doesn't

The former option also gives one complaint:

Warning in <TOutputListSelectorDataMap::SetDataMembers()>: potential memory leak: replacing data member `outTree' != 0. Please initialize outTree to 0 in constructor MySel::MySel()

Implementing this is trivial. But it made me realize that the Process call invokes the constructor, which was not clear.

One question: does the Process call always invoke the constructor?
Because I would generally prefer to pass the TSelector as argument, and this is only possible if it has already been constructed.

If I create a MySel object and pass it to Process, I get a segfault that is no doubt due to the redundant calls to the constructor:

 +++ Starting PROOF-Lite with 8 workers +++
Opening connections to workers: OK (8 workers)                 
Setting up worker servers: OK (8 workers)                 
PROOF set to parallel mode (8 workers)
12:36:12 220663 Wrk-0.0 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
12:36:12 220671 Wrk-0.4 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
12:36:12 220665 Wrk-0.1 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
12:36:12 220675 Wrk-0.6 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
12:36:12 220667 Wrk-0.2 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
12:36:12 220673 Wrk-0.5 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
12:36:12 220669 Wrk-0.3 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
12:36:12 220677 Wrk-0.7 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
***class instance created*** 
 
Info in <TProofLite::SetQueryRunning>: starting query: 1
Info in <TProofQueryResult::SetRunning>: nwrks: 8
-->Client beginning...
Looking up for exact location of files: OK (1 files)                 
Looking up for exact location of files: OK (1 files)                 
Info in <TPacketizer::TPacketizer>: Initial number of workers: 8
Validating files: OK (1 files)                 

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00002b500c5c865e in waitpid () from /lib64/libc.so.6
#1  0x00002b500c55a609 in do_system () from /lib64/libc.so.6
#2  0x00002b5006a86338 in TUnixSystem::StackTrace (this=0x17434d0) at /scratch/bkw1a/root_v5.34.28/core/unix/src/TUnixSystem.cxx:2419
#3  0x00002b5006a851b3 in TUnixSystem::DispatchSignals (this=0x17434d0, sig=kSigSegmentationViolation) at /scratch/bkw1a/root_v5.34.28/core/unix/src/TUnixSystem.cxx:1294
#4  <signal handler called>
#5  0x00002b500669c0cb in TBuffer& operator<< <TTree>(TBuffer&, TTree const*) () from /nv/blue/mgv4ce/testing/MySel.so
#6  0x00002b500669b1d0 in MySel::Streamer(TBuffer&) () from /nv/blue/mgv4ce/testing/MySel.so
#7  0x00002b5007dc398e in Streamer (this=0x7ffdce848040, actualObjectStart=0x1f7cf60, actualClass=0x1fd36f0) at /scratch/bkw1a/root_v5.34.28/build/include/TClass.h:451
#8  TBufferFile::WriteObjectClass (this=0x7ffdce848040, actualObjectStart=0x1f7cf60, actualClass=0x1fd36f0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2605
#9  0x00002b5007dc0564 in TBufferFile::WriteObjectAny (this=0x7ffdce848040, obj=<value optimized out>, ptrClass=0x1a74530) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2659
#10 0x00002b5006a2a990 in operator<< (this=<value optimized out>, b=...) at /scratch/bkw1a/root_v5.34.28/build/include/TBuffer.h:401
#11 TList::Streamer (this=<value optimized out>, b=...) at /scratch/bkw1a/root_v5.34.28/core/cont/src/TList.cxx:1070
#12 0x00002b5007dc398e in Streamer (this=0x7ffdce848040, actualObjectStart=0x1b1bb80, actualClass=0x1f7d2e0) at /scratch/bkw1a/root_v5.34.28/build/include/TClass.h:451
#13 TBufferFile::WriteObjectClass (this=0x7ffdce848040, actualObjectStart=0x1b1bb80, actualClass=0x1f7d2e0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2605
#14 0x00002b5007dc051f in TBufferFile::WriteObjectAny (this=0x7ffdce848040, obj=0x1b1bb80, ptrClass=0x1f7d2e0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2662
#15 0x00002b500b189789 in operator<< <TList> (buf=..., obj=0x1b1bb80) at /scratch/bkw1a/root_v5.34.28/build/include/TBuffer.h:388
#16 0x00002b500b4d530c in TProofPlayerLite::Process (this=0x1b1b9e0, dset=<value optimized out>, selector_file=<value optimized out>, option=0x0, nentries=0, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proofplayer/src/TProofPlayerLite.cxx:266
#17 0x00002b500b4d4278 in TProofPlayerLite::Process (this=0x1b1b9e0, dset=0x1f2daf0, selector=<value optimized out>, option=0x7ffdce848469 "", nentries=100000, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proofplayer/src/TProofPlayerLite.cxx:97
#18 0x00002b500b149946 in TProofLite::Process (this=0x1a6dfa0, dset=0x1f2daf0, selector=0x0, option=<value optimized out>, nentries=100000, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proof/src/TProofLite.cxx:1313
#19 0x00002b500b18eec4 in TProof::Process (this=0x1a6dfa0, dset=<value optimized out>, selector=<value optimized out>, option=<value optimized out>, nentries=<value optimized out>, first=<value optimized out>) at /scratch/bkw1a/root_v5.34.28/proof/proof/src/TProof.cxx:5687
#20 0x0000000000402311 in main ()
===========================================================


The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00002b500669c0cb in TBuffer& operator<< <TTree>(TBuffer&, TTree const*) () from /nv/blue/mgv4ce/testing/MySel.so
#6  0x00002b500669b1d0 in MySel::Streamer(TBuffer&) () from /nv/blue/mgv4ce/testing/MySel.so
#7  0x00002b5007dc398e in Streamer (this=0x7ffdce848040, actualObjectStart=0x1f7cf60, actualClass=0x1fd36f0) at /scratch/bkw1a/root_v5.34.28/build/include/TClass.h:451
#8  TBufferFile::WriteObjectClass (this=0x7ffdce848040, actualObjectStart=0x1f7cf60, actualClass=0x1fd36f0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2605
#9  0x00002b5007dc0564 in TBufferFile::WriteObjectAny (this=0x7ffdce848040, obj=<value optimized out>, ptrClass=0x1a74530) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2659
#10 0x00002b5006a2a990 in operator<< (this=<value optimized out>, b=...) at /scratch/bkw1a/root_v5.34.28/build/include/TBuffer.h:401
#11 TList::Streamer (this=<value optimized out>, b=...) at /scratch/bkw1a/root_v5.34.28/core/cont/src/TList.cxx:1070
#12 0x00002b5007dc398e in Streamer (this=0x7ffdce848040, actualObjectStart=0x1b1bb80, actualClass=0x1f7d2e0) at /scratch/bkw1a/root_v5.34.28/build/include/TClass.h:451
#13 TBufferFile::WriteObjectClass (this=0x7ffdce848040, actualObjectStart=0x1b1bb80, actualClass=0x1f7d2e0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2605
#14 0x00002b5007dc051f in TBufferFile::WriteObjectAny (this=0x7ffdce848040, obj=0x1b1bb80, ptrClass=0x1f7d2e0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2662
#15 0x00002b500b189789 in operator<< <TList> (buf=..., obj=0x1b1bb80) at /scratch/bkw1a/root_v5.34.28/build/include/TBuffer.h:388
#16 0x00002b500b4d530c in TProofPlayerLite::Process (this=0x1b1b9e0, dset=<value optimized out>, selector_file=<value optimized out>, option=0x0, nentries=0, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proofplayer/src/TProofPlayerLite.cxx:266
#17 0x00002b500b4d4278 in TProofPlayerLite::Process (this=0x1b1b9e0, dset=0x1f2daf0, selector=<value optimized out>, option=0x7ffdce848469 "", nentries=100000, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proofplayer/src/TProofPlayerLite.cxx:97
#18 0x00002b500b149946 in TProofLite::Process (this=0x1a6dfa0, dset=0x1f2daf0, selector=0x0, option=<value optimized out>, nentries=100000, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proof/src/TProofLite.cxx:1313
#19 0x00002b500b18eec4 in TProof::Process (this=0x1a6dfa0, dset=<value optimized out>, selector=<value optimized out>, option=<value optimized out>, nentries=<value optimized out>, first=<value optimized out>) at /scratch/bkw1a/root_v5.34.28/proof/proof/src/TProof.cxx:5687
#20 0x0000000000402311 in main ()
===========================================================


~~~class instance destroyed~~~ 

If there is no way to pass the TSelector, I am still happy! I got a nice speed-up!

Many Thanks!

It seems I was a bit hasty in my jubilation. I do indeed need some means of controlling the constructor more explicitly, or else I cannot pass any parameters to the TSelector.

Am I correct that the segfault I see when trying to pass the TSelector as an argument is due to repeated constructors being called?

Thanks again!

If it useful, I found one more clue. When I pass the TSelector as the argument, I get a different error depending on whether I initialize the outTree in the constructor.

When I do NOT initialize the outTree to zero in the constructor, and pass the TSelector as argument I get the segfault above.

When I DO initialize the outTree to zero in the constructor, and pass the TSelector as the argument I get:

-bash-4.1$ time ./selector /home/mgv4ce/testing/result90000.root
/nv/blue/mgv4ce/testing
 +++ Starting PROOF-Lite with 8 workers +++
Opening connections to workers: OK (8 workers)                 
Setting up worker servers: OK (8 workers)                 
PROOF set to parallel mode (8 workers)
15:27:32 131963 Wrk-0.0 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
15:27:32 131969 Wrk-0.3 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
15:27:32 131967 Wrk-0.2 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
15:27:32 131975 Wrk-0.6 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
15:27:32 131977 Wrk-0.7 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
15:27:32 131965 Wrk-0.1 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
15:27:32 131971 Wrk-0.4 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
15:27:32 131973 Wrk-0.5 | Info in <TProofServLite::HandleCache>: loading macro MySel.so ...
***class instance created*** 
 
Info in <TProofLite::SetQueryRunning>: starting query: 1
Info in <TProofQueryResult::SetRunning>: nwrks: 8
-->Client beginning...
Looking up for exact location of files: OK (1 files)                 
Looking up for exact location of files: OK (1 files)                 
Info in <TPacketizer::TPacketizer>: Initial number of workers: 8
Validating files: OK (1 files)                 

 *** Break *** bus error



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00002adff54ae65e in waitpid () from /lib64/libc.so.6
#1  0x00002adff5440609 in do_system () from /lib64/libc.so.6
#2  0x00002adfef96c338 in TUnixSystem::StackTrace (this=0x1c174d0) at /scratch/bkw1a/root_v5.34.28/core/unix/src/TUnixSystem.cxx:2419
#3  0x00002adfef96b1b3 in TUnixSystem::DispatchSignals (this=0x1c174d0, sig=kSigBus) at /scratch/bkw1a/root_v5.34.28/core/unix/src/TUnixSystem.cxx:1294
#4  <signal handler called>
#5  0x00002adfef58146c in MySel::Streamer(TBuffer&) () from /nv/blue/mgv4ce/testing/MySel.so
#6  0x00002adff0ca998e in Streamer (this=0x7ffe949c03e0, actualObjectStart=0x2450ef0, actualClass=0x24a76c0) at /scratch/bkw1a/root_v5.34.28/build/include/TClass.h:451
#7  TBufferFile::WriteObjectClass (this=0x7ffe949c03e0, actualObjectStart=0x2450ef0, actualClass=0x24a76c0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2605
#8  0x00002adff0ca6564 in TBufferFile::WriteObjectAny (this=0x7ffe949c03e0, obj=<value optimized out>, ptrClass=0x1f48550) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2659
#9  0x00002adfef910990 in operator<< (this=<value optimized out>, b=...) at /scratch/bkw1a/root_v5.34.28/build/include/TBuffer.h:401
#10 TList::Streamer (this=<value optimized out>, b=...) at /scratch/bkw1a/root_v5.34.28/core/cont/src/TList.cxx:1070
#11 0x00002adff0ca998e in Streamer (this=0x7ffe949c03e0, actualObjectStart=0x1fefba0, actualClass=0x24512b0) at /scratch/bkw1a/root_v5.34.28/build/include/TClass.h:451
#12 TBufferFile::WriteObjectClass (this=0x7ffe949c03e0, actualObjectStart=0x1fefba0, actualClass=0x24512b0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2605
#13 0x00002adff0ca651f in TBufferFile::WriteObjectAny (this=0x7ffe949c03e0, obj=0x1fefba0, ptrClass=0x24512b0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2662
#14 0x00002adff406f789 in operator<< <TList> (buf=..., obj=0x1fefba0) at /scratch/bkw1a/root_v5.34.28/build/include/TBuffer.h:388
#15 0x00002adff43bb30c in TProofPlayerLite::Process (this=0x1fefa00, dset=<value optimized out>, selector_file=<value optimized out>, option=0x0, nentries=0, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proofplayer/src/TProofPlayerLite.cxx:266
#16 0x00002adff43ba278 in TProofPlayerLite::Process (this=0x1fefa00, dset=0x2401a80, selector=<value optimized out>, option=0x7ffe949c0809 "", nentries=100000, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proofplayer/src/TProofPlayerLite.cxx:97
#17 0x00002adff402f946 in TProofLite::Process (this=0x1f41fc0, dset=0x2401a80, selector=0x0, option=<value optimized out>, nentries=100000, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proof/src/TProofLite.cxx:1313
#18 0x00002adff4074ec4 in TProof::Process (this=0x1f41fc0, dset=<value optimized out>, selector=<value optimized out>, option=<value optimized out>, nentries=<value optimized out>, first=<value optimized out>) at /scratch/bkw1a/root_v5.34.28/proof/proof/src/TProof.cxx:5687
#19 0x0000000000402805 in main ()
===========================================================


The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00002adfef58146c in MySel::Streamer(TBuffer&) () from /nv/blue/mgv4ce/testing/MySel.so
#6  0x00002adff0ca998e in Streamer (this=0x7ffe949c03e0, actualObjectStart=0x2450ef0, actualClass=0x24a76c0) at /scratch/bkw1a/root_v5.34.28/build/include/TClass.h:451
#7  TBufferFile::WriteObjectClass (this=0x7ffe949c03e0, actualObjectStart=0x2450ef0, actualClass=0x24a76c0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2605
#8  0x00002adff0ca6564 in TBufferFile::WriteObjectAny (this=0x7ffe949c03e0, obj=<value optimized out>, ptrClass=0x1f48550) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2659
#9  0x00002adfef910990 in operator<< (this=<value optimized out>, b=...) at /scratch/bkw1a/root_v5.34.28/build/include/TBuffer.h:401
#10 TList::Streamer (this=<value optimized out>, b=...) at /scratch/bkw1a/root_v5.34.28/core/cont/src/TList.cxx:1070
#11 0x00002adff0ca998e in Streamer (this=0x7ffe949c03e0, actualObjectStart=0x1fefba0, actualClass=0x24512b0) at /scratch/bkw1a/root_v5.34.28/build/include/TClass.h:451
#12 TBufferFile::WriteObjectClass (this=0x7ffe949c03e0, actualObjectStart=0x1fefba0, actualClass=0x24512b0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2605
#13 0x00002adff0ca651f in TBufferFile::WriteObjectAny (this=0x7ffe949c03e0, obj=0x1fefba0, ptrClass=0x24512b0) at /scratch/bkw1a/root_v5.34.28/io/io/src/TBufferFile.cxx:2662
#14 0x00002adff406f789 in operator<< <TList> (buf=..., obj=0x1fefba0) at /scratch/bkw1a/root_v5.34.28/build/include/TBuffer.h:388
#15 0x00002adff43bb30c in TProofPlayerLite::Process (this=0x1fefa00, dset=<value optimized out>, selector_file=<value optimized out>, option=0x0, nentries=0, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proofplayer/src/TProofPlayerLite.cxx:266
#16 0x00002adff43ba278 in TProofPlayerLite::Process (this=0x1fefa00, dset=0x2401a80, selector=<value optimized out>, option=0x7ffe949c0809 "", nentries=100000, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proofplayer/src/TProofPlayerLite.cxx:97
#17 0x00002adff402f946 in TProofLite::Process (this=0x1f41fc0, dset=0x2401a80, selector=0x0, option=<value optimized out>, nentries=100000, first=0) at /scratch/bkw1a/root_v5.34.28/proof/proof/src/TProofLite.cxx:1313
#18 0x00002adff4074ec4 in TProof::Process (this=0x1f41fc0, dset=<value optimized out>, selector=<value optimized out>, option=<value optimized out>, nentries=<value optimized out>, first=<value optimized out>) at /scratch/bkw1a/root_v5.34.28/proof/proof/src/TProof.cxx:5687
#19 0x0000000000402805 in main ()
===========================================================


~~~class instance destroyed~~~