Logo ROOT  
Reference Guide
 
Loading...
Searching...
No Matches
ROOT::Experimental::RNTupleDS Class Referencefinal

The RDataSource implementation for RNTuple.

It lets RDataFrame read RNTuple data.

An RDataFrame that reads RNTuple data can be constructed using FromRNTuple().

For each column containing an array or a collection, a corresponding column #colname is available to access colname.size() without reading and deserializing the collection values.

Definition at line 43 of file RNTupleDS.hxx.

Classes

struct  REntryRangeDS
 The PrepareNextRanges() method populates the fNextRanges list with REntryRangeDS records. More...
 
struct  RFieldInfo
 Holds useful information about fields added to the RNTupleDS. More...
 

Public Member Functions

 RNTupleDS (ROOT::Experimental::RNTuple *ntuple)
 
 RNTupleDS (std::string_view ntupleName, const std::vector< std::string > &fileNames)
 
 RNTupleDS (std::string_view ntupleName, std::string_view fileName)
 
 ~RNTupleDS ()
 
void Finalize () final
 Convenience method called after concluding an event-loop.
 
void FinalizeSlot (unsigned int slot) final
 Convenience method called at the end of the data processing associated to a slot.
 
const std::vector< std::string > & GetColumnNames () const final
 Returns a reference to the collection of the dataset's column names.
 
std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBaseGetColumnReaders (unsigned int, std::string_view, const std::type_info &) final
 If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.
 
std::vector< std::pair< ULong64_t, ULong64_t > > GetEntryRanges () final
 Return ranges of entries to distribute to tasks.
 
std::string GetLabel () final
 Return a string representation of the datasource type.
 
std::string GetTypeName (std::string_view colName) const final
 Type of a column as a string, e.g.
 
bool HasColumn (std::string_view colName) const final
 Checks if the dataset has a certain column.
 
void Initialize () final
 Convenience method called before starting an event-loop.
 
void InitSlot (unsigned int slot, ULong64_t firstEntry) final
 Convenience method called at the start of the data processing associated to a slot.
 
bool SetEntry (unsigned int slot, ULong64_t entry) final
 Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.
 
void SetNSlots (unsigned int nSlots) final
 Inform RDataSource of the number of processing slots (i.e.
 
- Public Member Functions inherited from ROOT::RDF::RDataSource
virtual ~RDataSource ()=default
 
template<typename T >
std::vector< T ** > GetColumnReaders (std::string_view columnName)
 Called at most once per column by RDF.
 

Protected Member Functions

Record_t GetColumnReadersImpl (std::string_view name, const std::type_info &) final
 type-erased vector of pointers to pointers to column values - one per slot
 
- Protected Member Functions inherited from ROOT::RDF::RDataSource
virtual std::string AsString ()
 

Private Member Functions

 RNTupleDS (std::unique_ptr< ROOT::Experimental::Internal::RPageSource > pageSource)
 
void AddField (const RNTupleDescriptor &desc, std::string_view colName, DescriptorId_t fieldId, std::vector< RFieldInfo > fieldInfos)
 Provides the RDF column "colName" given the field identified by fieldID.
 
void PrepareNextRanges ()
 Populates fNextRanges with the next set of entry ranges.
 

Private Attributes

std::vector< std::vector< Internal::RNTupleColumnReader * > > fActiveColumnReaders
 List of column readers returned by GetColumnReaders() organized by slot.
 
std::vector< std::string > fColumnNames
 
std::vector< std::string > fColumnTypes
 
std::vector< REntryRangeDSfCurrentRanges
 Basis for the ranges returned by the last GetEntryRanges() call.
 
std::unordered_map< ROOT::Experimental::DescriptorId_t, std::string > fFieldId2QualifiedName
 Connects the IDs of active proto fields and their subfields to their fully qualified name (a.b.c.d).
 
std::vector< std::string > fFileNames
 
std::unordered_map< ULong64_t, std::size_t > fFirstEntry2RangeIdx
 Maps the first entries from the ranges of the last GetEntryRanges() call to their corresponding index in the fCurrentRanges vectors.
 
std::size_t fNextFileIndex = 0
 Index into fFileNames to the next file to process.
 
std::vector< REntryRangeDSfNextRanges
 Basis for the ranges populated by the PrepareNextRanges() call.
 
unsigned int fNSlots = 0
 
std::string fNTupleName
 The data source may be constructed with an ntuple name and a list of files.
 
std::unique_ptr< RNTupleDescriptorfPrincipalDescriptor
 A clone of the first pages source's descriptor.
 
std::unique_ptr< Internal::RPageSourcefPrincipalSource
 The first source is used to extract the schema and build the prototype fields.
 
std::vector< std::unique_ptr< ROOT::Experimental::RFieldBase > > fProtoFields
 We prepare a prototype field for every column.
 
ULong64_t fSeenEntries = 0
 The number of entries so far returned by GetEntryRanges()
 

Friends

class Internal::RNTupleColumnReader
 

Additional Inherited Members

- Protected Types inherited from ROOT::RDF::RDataSource
using Record_t = std::vector< void * >
 

#include <ROOT/RNTupleDS.hxx>

Inheritance diagram for ROOT::Experimental::RNTupleDS:
[legend]

Constructor & Destructor Documentation

◆ RNTupleDS() [1/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( std::unique_ptr< ROOT::Experimental::Internal::RPageSource pageSource)
explicitprivate

Definition at line 376 of file RNTupleDS.cxx.

◆ RNTupleDS() [2/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( std::string_view  ntupleName,
std::string_view  fileName 
)

Definition at line 385 of file RNTupleDS.cxx.

◆ RNTupleDS() [3/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( ROOT::Experimental::RNTuple ntuple)

Definition at line 390 of file RNTupleDS.cxx.

◆ RNTupleDS() [4/4]

ROOT::Experimental::RNTupleDS::RNTupleDS ( std::string_view  ntupleName,
const std::vector< std::string > &  fileNames 
)

Definition at line 395 of file RNTupleDS.cxx.

◆ ~RNTupleDS()

ROOT::Experimental::RNTupleDS::~RNTupleDS ( )
default

Member Function Documentation

◆ AddField()

void ROOT::Experimental::RNTupleDS::AddField ( const RNTupleDescriptor desc,
std::string_view  colName,
DescriptorId_t  fieldId,
std::vector< RFieldInfo fieldInfos 
)
private

Provides the RDF column "colName" given the field identified by fieldID.

For records and collections, AddField recurses into the sub fields. The fieldInfos argument is a list of objects holding info about the fields of the outer collection(s) (w.r.t. fieldId). For instance, if fieldId refers to an std::vector<Jet>, with struct Jet { float pt; float eta; }; AddField will recurse into Jet.pt and Jet.eta and provide the two inner fields as std::vector<float> each.

Definition at line 232 of file RNTupleDS.cxx.

◆ Finalize()

void ROOT::Experimental::RNTupleDS::Finalize ( )
finalvirtual

Convenience method called after concluding an event-loop.

See Initialize for more details.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 634 of file RNTupleDS.cxx.

◆ FinalizeSlot()

void ROOT::Experimental::RNTupleDS::FinalizeSlot ( unsigned int  )
finalvirtual

Convenience method called at the end of the data processing associated to a slot.

Parameters
[in]slotThe data processing slot wihch needs to be finalized This method might be called multiple times per thread per event-loop.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 600 of file RNTupleDS.cxx.

◆ GetColumnNames()

const std::vector< std::string > & ROOT::Experimental::RNTupleDS::GetColumnNames ( ) const
inlinefinalvirtual

Returns a reference to the collection of the dataset's column names.

Implements ROOT::RDF::RDataSource.

Definition at line 127 of file RNTupleDS.hxx.

◆ GetColumnReaders()

std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > ROOT::Experimental::RNTupleDS::GetColumnReaders ( unsigned int  ,
std::string_view  ,
const std::type_info &   
)
finalvirtual

If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.

Parameters
[in]slotThe data processing slot that needs to be considered
[in]nameThe name of the column for which a column reader needs to be returned
[in]tidA type_info At least one of the two must return a non-empty/non-null value.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 409 of file RNTupleDS.cxx.

◆ GetColumnReadersImpl()

RDF::RDataSource::Record_t ROOT::Experimental::RNTupleDS::GetColumnReadersImpl ( std::string_view  name,
const std::type_info &   
)
finalprotectedvirtual

type-erased vector of pointers to pointers to column values - one per slot

Implements ROOT::RDF::RDataSource.

Definition at line 402 of file RNTupleDS.cxx.

◆ GetEntryRanges()

std::vector< std::pair< ULong64_t, ULong64_t > > ROOT::Experimental::RNTupleDS::GetEntryRanges ( )
finalvirtual

Return ranges of entries to distribute to tasks.

They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.

Implements ROOT::RDF::RDataSource.

Definition at line 533 of file RNTupleDS.cxx.

◆ GetLabel()

std::string ROOT::Experimental::RNTupleDS::GetLabel ( )
inlinefinalvirtual

Return a string representation of the datasource type.

The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 131 of file RNTupleDS.hxx.

◆ GetTypeName()

std::string ROOT::Experimental::RNTupleDS::GetTypeName ( std::string_view  colName) const
finalvirtual

Type of a column as a string, e.g.

GetTypeName("x") == "double". Required for jitting e.g. df.Filter("x>0").

Parameters
[in]colNameThe name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 610 of file RNTupleDS.cxx.

◆ HasColumn()

bool ROOT::Experimental::RNTupleDS::HasColumn ( std::string_view  colName) const
finalvirtual

Checks if the dataset has a certain column.

Parameters
[in]colNameThe name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 616 of file RNTupleDS.cxx.

◆ Initialize()

void ROOT::Experimental::RNTupleDS::Initialize ( )
finalvirtual

Convenience method called before starting an event-loop.

This method might be called multiple times over the lifetime of a RDataSource, since users can run multiple event-loops with the same RDataFrame. Ideally, Initialize should set the state of the RDataSource so that multiple identical event-loops will produce identical results.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 621 of file RNTupleDS.cxx.

◆ InitSlot()

void ROOT::Experimental::RNTupleDS::InitSlot ( unsigned int  ,
ULong64_t   
)
finalvirtual

Convenience method called at the start of the data processing associated to a slot.

Parameters
[in]slotThe data processing slot wihch needs to be initialized
[in]firstEntryThe first entry of the range that the task will process. This method might be called multiple times per thread per event-loop.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 589 of file RNTupleDS.cxx.

◆ PrepareNextRanges()

void ROOT::Experimental::RNTupleDS::PrepareNextRanges ( )
private

Populates fNextRanges with the next set of entry ranges.

Opens files from the chain as necessary and aligns ranges with cluster boundaries for scheduling the tail of files. Upon return, the fNextRanges list is ordered. It has usually fNSlots elements; fewer if there is not enough work to give at least one cluster to every slot.

Definition at line 435 of file RNTupleDS.cxx.

◆ SetEntry()

bool ROOT::Experimental::RNTupleDS::SetEntry ( unsigned int  slot,
ULong64_t  entry 
)
finalvirtual

Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.

Parameters
[in]slotThe data processing slot that needs to be considered
[in]entryThe entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise.

Implements ROOT::RDF::RDataSource.

Definition at line 429 of file RNTupleDS.cxx.

◆ SetNSlots()

void ROOT::Experimental::RNTupleDS::SetNSlots ( unsigned int  nSlots)
finalvirtual

Inform RDataSource of the number of processing slots (i.e.

worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.

Implements ROOT::RDF::RDataSource.

Definition at line 643 of file RNTupleDS.cxx.

Friends And Related Symbol Documentation

◆ Internal::RNTupleColumnReader

friend class Internal::RNTupleColumnReader
friend

Definition at line 44 of file RNTupleDS.hxx.

Member Data Documentation

◆ fActiveColumnReaders

std::vector<std::vector<Internal::RNTupleColumnReader *> > ROOT::Experimental::RNTupleDS::fActiveColumnReaders
private

List of column readers returned by GetColumnReaders() organized by slot.

Used to reconnect readers to new page sources when the files in the chain change.

Definition at line 81 of file RNTupleDS.hxx.

◆ fColumnNames

std::vector<std::string> ROOT::Experimental::RNTupleDS::fColumnNames
private

Definition at line 77 of file RNTupleDS.hxx.

◆ fColumnTypes

std::vector<std::string> ROOT::Experimental::RNTupleDS::fColumnTypes
private

Definition at line 78 of file RNTupleDS.hxx.

◆ fCurrentRanges

std::vector<REntryRangeDS> ROOT::Experimental::RNTupleDS::fCurrentRanges
private

Basis for the ranges returned by the last GetEntryRanges() call.

Definition at line 85 of file RNTupleDS.hxx.

◆ fFieldId2QualifiedName

std::unordered_map<ROOT::Experimental::DescriptorId_t, std::string> ROOT::Experimental::RNTupleDS::fFieldId2QualifiedName
private

Connects the IDs of active proto fields and their subfields to their fully qualified name (a.b.c.d).

This enables the column reader to rewire the field IDs when the file changes (chain), using the fully qualified name as a search key in the descriptor of the other page sources.

Definition at line 76 of file RNTupleDS.hxx.

◆ fFileNames

std::vector<std::string> ROOT::Experimental::RNTupleDS::fFileNames
private

Definition at line 65 of file RNTupleDS.hxx.

◆ fFirstEntry2RangeIdx

std::unordered_map<ULong64_t, std::size_t> ROOT::Experimental::RNTupleDS::fFirstEntry2RangeIdx
private

Maps the first entries from the ranges of the last GetEntryRanges() call to their corresponding index in the fCurrentRanges vectors.

This is necessary because the returned ranges get distributed arbitrarily onto slots. In the InitSlot method, the column readers use this map to find the correct range to connect to.

Definition at line 90 of file RNTupleDS.hxx.

◆ fNextFileIndex

std::size_t ROOT::Experimental::RNTupleDS::fNextFileIndex = 0
private

Index into fFileNames to the next file to process.

Definition at line 66 of file RNTupleDS.hxx.

◆ fNextRanges

std::vector<REntryRangeDS> ROOT::Experimental::RNTupleDS::fNextRanges
private

Basis for the ranges populated by the PrepareNextRanges() call.

Definition at line 86 of file RNTupleDS.hxx.

◆ fNSlots

unsigned int ROOT::Experimental::RNTupleDS::fNSlots = 0
private

Definition at line 83 of file RNTupleDS.hxx.

◆ fNTupleName

std::string ROOT::Experimental::RNTupleDS::fNTupleName
private

The data source may be constructed with an ntuple name and a list of files.

Definition at line 64 of file RNTupleDS.hxx.

◆ fPrincipalDescriptor

std::unique_ptr<RNTupleDescriptor> ROOT::Experimental::RNTupleDS::fPrincipalDescriptor
private

A clone of the first pages source's descriptor.

Definition at line 61 of file RNTupleDS.hxx.

◆ fPrincipalSource

std::unique_ptr<Internal::RPageSource> ROOT::Experimental::RNTupleDS::fPrincipalSource
private

The first source is used to extract the schema and build the prototype fields.

The page source is used to extract a clone of the descriptor to fPrincipalDescriptor. Afterwards it is moved into the first REntryRangeDS.

Definition at line 59 of file RNTupleDS.hxx.

◆ fProtoFields

std::vector<std::unique_ptr<ROOT::Experimental::RFieldBase> > ROOT::Experimental::RNTupleDS::fProtoFields
private

We prepare a prototype field for every column.

If a column reader is actually requested in GetColumnReaders(), we move a clone of the field into a new column reader for RDataFrame. Only the clone connects to the backing page store and acquires I/O resources. The field IDs are set in the context of the first source and used as keys in fFieldId2QualifiedName.

Definition at line 72 of file RNTupleDS.hxx.

◆ fSeenEntries

ULong64_t ROOT::Experimental::RNTupleDS::fSeenEntries = 0
private

The number of entries so far returned by GetEntryRanges()

Definition at line 84 of file RNTupleDS.hxx.

Libraries for ROOT::Experimental::RNTupleDS:

The documentation for this class was generated from the following files: