Package manager for a modular ROOT

From Tim Head:
At the ROOT User’s workshop we discussed making ROOT more modular and/or providing a way to have ROOT packages which provide additional functionality. Some thoughts below.
There seem to be two options:

  1. use the operating system’s package manager
  2. use a cross platform package manager
    (1) has the drawback that you need to create a recipe for every package on every OS, however the packages would integrate well with the OS.
    (2) only needs one recipe to be written, but cross platform packaging is hard (especially for compiled things).
    One attempt to do (2) is conda.pydata.org/docs/ which has taken over packaging in the python world. conda was created for python programs but can package and distribute any software. It is particularly useful for packages that require a compiler/are not written python.
    It is available for linux, MacOS and Windows and there are already packages for root as a whole (not sure who maintains them).

From Oliver Freyermuth:
Dear ROOTers,
let me add my 2 cents from my limited experience with packaging:
(2) seems a nice approach at first, since it seems to reduce replicated workload. But one should consider it basically throws your software out of any distribution’s packaging, users will not find you in distribution repositories anymore.
A famous example is texlive: Few people know there is tlmgr, the texlive package manager. Distributions can not use it, since it can not be integrated with distributions packaging, so they still package all parts of texlive themselves. The added tlmgr-layer makes the job even harder for distribution maintainers (consider Ubuntu, where TeXlive packaging is almost always lagging more than half a year behind). The end result for the user is that the well-working package manager tlmgr is not used at all, but they just use their apt-get / zypper / yum / emerge as usual.
Using non-distribution integrated solutions also make prelinking next to impossible and splitting of debug symbols overly complex (since distribution package managers do that already, you’d have to replicate). Note you also have to deal well with distributions hardcoding e.g. linker flags in their toolchain (famous Ubuntu does that) and handle updates of system libraries (which may break linkage easily - an integrated package manager notices that). You’d also have to provide a way to package everything which depends directly or indirectly on ROOT.
This is why I went with packaging all our software dependencies for the most widespread distributions in our collaboration (ROOT luckily was already there for OpenSUSE and Gentoo and MacPorts and is well supported and nicely splitted into modules there). For me, it was the by far easier way.
So for (1), there are existing solutions for many distributions, but of course it takes some manpower to maintain or improve them.
Cheers,
Oliver

I think I would distinguish (a) packaging ROOT itself, from (b) packaging ROOT ‘modules’. The later requires more than just installing (or re-building) the necessary module artifacts (libraries, headers, executables, config files) to the local installation of ROOT, but also we need to tell ROOT that this module is actually installed and can be used from the interpreter or applications (plugin manager). I guess some registration process.

For (a), I think I would use (1) the operating system’s package manager, unless the HSF community comes with something very general that works everywhere. There is an ongoing discussion at groups.google.com/forum/#!forum … ckaging-wg that you are most welcome to contribute.
We have already a number of people that package ROOT for a number of OS specific packaging systems.

For (b), I think I would use the (2) approach, since I think is essential that is platform independent. We can certainly have a look at tools like conda. The main problem I see is that some modules will require external packages (for example the ROOT module ‘MathMore’ requires the package GSL) and these should be found in the system or installed when the module gets ‘auto-installed’. It is clear that some smooth connection should be between (1) and (2) otherwise a lot of work will be duplicated.

Hi all,

As part of the XENON experiment, we’re working on making ROOT binaries available for installation via conda. This should make it far less painfull to use PyROOT as well, with support for python 2/3 and ROOT 5/6.

You can find the instructions on using the ROOT binaries here:
github.com/remenska/conda-recipes

In principle, if you already have a conda installation, this should be it:

$ conda config --add channels https://conda.anaconda.org/NLeSC
$ conda create --name=testenv root=6 python=3
$ source activate testenv
$ root -b -q
$ python
Type "help", "copyright", "credits" or "license" for more information.
>>> import ROOT
>>> f = ROOT.TFile.Open("test.root","recreate")

While our focus is on Linux, MacOS stable builds should also come in the future. Package dependencies should be automatically resolved. This should save headaches with setting environment variables, compiling/linking issues with ROOT.

I’m curious if people find it useful.

“Packaging” is unfortunately a very context sensitive term - so I guess the absolute first question is "describe what is meant by a ‘ROOT package’". Pere’s description is fairly clear

[quote=“mato”]I think I would distinguish (a) packaging ROOT itself, from (b) packaging ROOT ‘modules’. The later requires more than just installing (or re-building) the necessary module artifacts (libraries, headers, executables, config files) to the local installation of ROOT, but also we need to tell ROOT that this module is actually installed and can be used from the interpreter or applications (plugin manager). I guess some registration process.
[/quote]

My reading of that is “ROOT modules” sound like Python/Ruby/Node/languageofchoice extensions - so “packaging” in this context is the equivalent of Pip/Gem/Npm/whatever, just for C++. I think that’s a closer use case than conda and the scope of the issue addressed by HSF (which is software stacks rather than extensions to a single application).

Rather than start with the package manager, it might be better to review how ROOT looks up/loads “module artifacts”. This is partially documented in the User Guide (gSystem calls?), but having a clear example case of a simple extension and the steps needed to create the artifacts could help to identify where a package manager might help, and equally where it may not be needed.