Re: parallel build does not work

From: Tom Roberts <tjrob_at_fnal.gov>
Date: Thu, 20 Aug 2009 12:37:46 -0500


Fons:

I was puzzled by the difference between your results and mine. I started having some successes, but failures and successes would come and go. Since my last email I have built Root more than 50 times, exploring the problem. I found in one case that adding an empty directory to the front of PATH made failures much more likely -- this points to some sort of timing problem between threads.

Here is a simpler and more direct demonstration that there is something in the makefiles that is not parallel-safe; I do not know what the problem is but strongly suspect it is related to constructing rootcint_tmp.cxx (most but not all failures mention it; it is possible there are multiple problems).

I started by creating a new account "testuser", and checked out the svn source once. I then ran this loop -- be sure you have lots of disk space:

   # Root svn source put into ~g4bl/testRoot/rootSvnSource    for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20    do

	cp -Rp ~g4bl/testRoot/rootSvnSource root-$i
	cd root-$i
	pwd
	./configure >build.log 2>&1
	if make -j 4 >>build.log 2>&1
	then
		echo SUCCESS! >>build.log 2>&1
		echo SUCCESS!
	else
		echo FAILURE >>build.log 2>&1
		echo FAILURE!
	fi
	cd ..

   done

I stopped it after 13 iterations, with 11 successes and 2 failures. I cannot explain that any way other than that the makefiles are not truly parallel-safe.

These two failures happen to look quite similar; I have had failures that look completely different.

Tom Roberts

Here is the environment:

tjrob:testRoot testuser$ env
MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11/man TERM=xterm-color
SHELL=/bin/bash
OLDPWD=/Users/testuser/testRoot/root-10
USER=testuser
PATH=/bin:/usr/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin PWD=/Users/testuser/testRoot
SHLVL=1
HOME=/Users/testuser
_=/usr/bin/env
tjrob:testRoot testuser$ g++ -v
Using built-in specs.
Target: i686-apple-darwin9
Configured with: /var/tmp/gcc/gcc-5493~1/src/configure

--disable-checking -enable-werror --prefix=/usr --mandir=/share/man 
--enable-languages=c,objc,c++,obj-c++ 
--program-transform-name=/^[cg][^.-]*$/s/$/-4.0/ 
--with-gxx-include-dir=/include/c++/4.0.0 --with-slibdir=/usr/lib 
--build=i686-apple-darwin9 --with-arch=apple --with-tune=generic 
--host=i686-apple-darwin9 --target=i686-apple-darwin9
Thread model: posix
gcc version 4.0.1 (Apple Inc. build 5493) tjrob:testRoot testuser$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for powerpc-apple-darwin9.0 tjrob:testRoot testuser$ type make
make is hashed (/usr/bin/make)
tjrob:testRoot testuser$ file /usr/bin/make
/usr/bin/make: Mach-O universal binary with 2 architectures
/usr/bin/make (for architecture i386):	Mach-O executable i386
/usr/bin/make (for architecture ppc7400):	Mach-O executable ppc
tjrob:testRoot testuser$ uname -a
Darwin tjrob.dhcp.fnal.gov 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386

Here are the tails of the failing build.log-s:

g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/Xrdcp.o Xrdcp.cc

Compiling XrdcpXtremeRead.cc
g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/XrdcpXtremeRead.o XrdcpXtremeRead.cc
Creating executable ../../bin/xrdcp
g++ -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__ -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2
../../obj/Xrdcp.o ../../obj/XrdcpXtremeRead.o ../../lib/libXrdClient.a
../../lib/libXrdCrypto.a ../../lib/libXrdSut.a ../../lib/libXrdNet.a
../../lib/libXrdOuc.a ../../lib/libXrdSys.a -ldl -lc -lz -o ../../bin/xrdcp
Compiling XrdCommandLine.cc
g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/XrdCommandLine.o XrdCommandLine.cc
Creating executable ../../bin/xrd
g++ -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__ -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2
../../obj/XrdCommandLine.o ../../lib/libXrdClient.a
../../lib/libXrdCrypto.a ../../lib/libXrdSut.a ../../lib/libXrdNet.a
../../lib/libXrdOuc.a ../../lib/libXrdSys.a -ldl -lc -o ../../bin/xrd
if [ "" != "" ]; then \

		echo Creating executable ../../bin/xrdadmin; \
		sed -e "s|#!/usr/bin/perl|#!|" < xrdadmin > ../../bin/xrdadmin; \
		chmod a+x ../../bin/xrdadmin; \
	fi;

Compiling XrdClientPrep.cc
g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/XrdClientPrep.o XrdClientPrep.cc
Creating executable ../../bin/xprep
g++ -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__ -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2
../../obj/XrdClientPrep.o ../../lib/libXrdClient.a
../../lib/libXrdCrypto.a ../../lib/libXrdSut.a ../../lib/libXrdNet.a
../../lib/libXrdOuc.a ../../lib/libXrdSys.a -ldl -lc -o ../../bin/xprep
Compiling XrdStageTool.cc
g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/XrdStageTool.o XrdStageTool.cc
Creating executable ../../bin/xrdstagetool g++ -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__ -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2
../../obj/XrdStageTool.o ../../lib/libXrdClient.a
../../lib/libXrdCrypto.a ../../lib/libXrdSut.a ../../lib/libXrdNet.a
../../lib/libXrdOuc.a ../../lib/libXrdSys.a -ldl -lc -o
../../bin/xrdstagetool

cp *.pm ../../lib
Make perlmodule done.
Make XrdClient done.
Make all done
Copying net/xrootd/src/xrootd/lib/libXrdBwm.so ...
Copying net/xrootd/src/xrootd/lib/libXrdClient.so ...
Copying net/xrootd/src/xrootd/lib/libXrdCrypto.so ...
Copying net/xrootd/src/xrootd/lib/libXrdCryptossl.so ...
Copying net/xrootd/src/xrootd/lib/libXrdOfs.so ...
Copying net/xrootd/src/xrootd/lib/libXrdRootd.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSec.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecgsi.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecgsiGMAPLDAP.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecpwd.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecsss.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecunix.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSut.so ...
Copying net/xrootd/src/xrootd/bin/xrootd executables ...
Copying net/xrootd/src/xrootd/bin/olbd executables ...
Copying net/xrootd/src/xrootd/bin/xrdcp executables ...
Copying net/xrootd/src/xrootd/bin/xrd executables ...
Copying net/xrootd/src/xrootd/bin/xrdpwdadmin executables ...
Copying net/xrootd/src/xrootd/bin/cmsd executables ...
Copying net/xrootd/src/xrootd/bin/xrdstagetool executables ...
Copying net/xrootd/src/xrootd/bin/xprep executables ...
Copying net/xrootd/src/xrootd/bin/xrdgsiproxy executables ...
rm core/utils/src/RStl_tmp.cxx core/utils/src/rootcint_tmp.cxx FAILURE g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/Xrdcp.o Xrdcp.cc

Compiling XrdcpXtremeRead.cc
g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/XrdcpXtremeRead.o XrdcpXtremeRead.cc
Creating executable ../../bin/xrdcp
g++ -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__ -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2
../../obj/Xrdcp.o ../../obj/XrdcpXtremeRead.o ../../lib/libXrdClient.a
../../lib/libXrdCrypto.a ../../lib/libXrdSut.a ../../lib/libXrdNet.a
../../lib/libXrdOuc.a ../../lib/libXrdSys.a -ldl -lc -lz -o ../../bin/xrdcp
Compiling XrdCommandLine.cc
g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/XrdCommandLine.o XrdCommandLine.cc
Creating executable ../../bin/xrd
g++ -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__ -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2
../../obj/XrdCommandLine.o ../../lib/libXrdClient.a
../../lib/libXrdCrypto.a ../../lib/libXrdSut.a ../../lib/libXrdNet.a
../../lib/libXrdOuc.a ../../lib/libXrdSys.a -ldl -lc -o ../../bin/xrd
if [ "" != "" ]; then \
		echo Creating executable ../../bin/xrdadmin; \
		sed -e "s|#!/usr/bin/perl|#!|" < xrdadmin > ../../bin/xrdadmin; \
		chmod a+x ../../bin/xrdadmin; \
	fi;

Compiling XrdClientPrep.cc
g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/XrdClientPrep.o XrdClientPrep.cc
Creating executable ../../bin/xprep
g++ -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__ -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2
../../obj/XrdClientPrep.o ../../lib/libXrdClient.a
../../lib/libXrdCrypto.a ../../lib/libXrdSut.a ../../lib/libXrdNet.a
../../lib/libXrdOuc.a ../../lib/libXrdSys.a -ldl -lc -o ../../bin/xprep
Compiling XrdStageTool.cc
g++ -g -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__   -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2 -DXrdDEBUG=0 -DHAVE_STRLCPY -DHAVE_LIBZ -I. -I.. -o
../../obj/XrdStageTool.o XrdStageTool.cc
Creating executable ../../bin/xrdstagetool g++ -m64 -D_REENTRANT -D_GNU_SOURCE -Wall -D__macos__ -Wno-deprecated -undefined dynamic_lookup -multiply_defined suppress -O2
../../obj/XrdStageTool.o ../../lib/libXrdClient.a
../../lib/libXrdCrypto.a ../../lib/libXrdSut.a ../../lib/libXrdNet.a
../../lib/libXrdOuc.a ../../lib/libXrdSys.a -ldl -lc -o
../../bin/xrdstagetool

cp *.pm ../../lib
Make perlmodule done.
Make XrdClient done.
Make all done
Copying net/xrootd/src/xrootd/lib/libXrdBwm.so ...
Copying net/xrootd/src/xrootd/lib/libXrdClient.so ...
Copying net/xrootd/src/xrootd/lib/libXrdCrypto.so ...
Copying net/xrootd/src/xrootd/lib/libXrdCryptossl.so ...
Copying net/xrootd/src/xrootd/lib/libXrdOfs.so ...
Copying net/xrootd/src/xrootd/lib/libXrdRootd.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSec.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecgsi.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecgsiGMAPLDAP.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecpwd.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecsss.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSecunix.so ...
Copying net/xrootd/src/xrootd/lib/libXrdSut.so ...
Copying net/xrootd/src/xrootd/bin/xrootd executables ...
Copying net/xrootd/src/xrootd/bin/olbd executables ...
Copying net/xrootd/src/xrootd/bin/xrdcp executables ...
Copying net/xrootd/src/xrootd/bin/xrd executables ...
Copying net/xrootd/src/xrootd/bin/xrdpwdadmin executables ...
Copying net/xrootd/src/xrootd/bin/cmsd executables ...
Copying net/xrootd/src/xrootd/bin/xrdstagetool executables ...
Copying net/xrootd/src/xrootd/bin/xprep executables ...
Copying net/xrootd/src/xrootd/bin/xrdgsiproxy executables ...
rm core/utils/src/RStl_tmp.cxx core/utils/src/rootcint_tmp.cxx FAILURE Received on Thu Aug 20 2009 - 19:37:52 CEST

This archive was generated by hypermail 2.2.0 : Fri Aug 21 2009 - 11:50:02 CEST