RDataSource defines an API that RDataFrame can use to read arbitrary data formats.
A concrete RDataSource implementation (i.e. a class that inherits from RDataSource and implements all of its pure methods) provides an adaptor that RDataFrame can leverage to read any kind of tabular data formats. RDataFrame calls into RDataSource to retrieve information about the data, retrieve (thread-local) readers or "cursors" for selected columns and to advance the readers to the desired data entry.
The sequence of calls that RDataFrame (or any other client of a RDataSource) performs is the following:
RDataSource implementations must support running multiple event-loops consecutively (although sequentially) on the same dataset.
Advanced users that plan to implement a custom RDataSource can check out existing implementations, e.g. RCsvDS or RNTupleDS. See the inheritance diagram below for the full list of existing concrete implementations.
Definition at line 142 of file RDataSource.hxx.
Public Member Functions | |
RDataSource ()=default | |
RDataSource (const RDataSource &)=delete | |
RDataSource (RDataSource &&)=delete | |
virtual | ~RDataSource ()=default |
virtual void | Finalize () |
Convenience method called after concluding an event-loop. | |
virtual void | FinalizeSlot (unsigned int) |
Convenience method called at the end of the data processing associated to a slot. | |
virtual const std::vector< std::string > & | GetColumnNames () const =0 |
Returns a reference to the collection of the dataset's column names. | |
template<typename T > | |
std::vector< T ** > | GetColumnReaders (std::string_view columnName) |
Called at most once per column by RDF. | |
virtual std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > | GetColumnReaders (unsigned int, std::string_view, const std::type_info &) |
If the other GetColumnReaders overload returns an empty vector, this overload will be called instead. | |
virtual std::vector< std::pair< ULong64_t, ULong64_t > > | GetEntryRanges ()=0 |
Return ranges of entries to distribute to tasks. | |
virtual std::string | GetLabel () |
Return a string representation of the datasource type. | |
virtual std::size_t | GetNFiles () const |
Returns the number of files from which the dataset is constructed. | |
virtual std::string | GetTypeName (std::string_view colName) const =0 |
Type of a column as a string, e.g. | |
virtual bool | HasColumn (std::string_view colName) const =0 |
Checks if the dataset has a certain column. | |
virtual void | Initialize () |
Convenience method called before starting an event-loop. | |
virtual void | InitSlot (unsigned int, ULong64_t) |
Convenience method called at the start of the data processing associated to a slot. | |
RDataSource & | operator= (const RDataSource &)=delete |
RDataSource & | operator= (RDataSource &&)=delete |
virtual bool | SetEntry (unsigned int slot, ULong64_t entry)=0 |
Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot. | |
virtual void | SetGlobalEntryRange (std::pair< ULong64_t, ULong64_t > entryRange) |
Restrict processing to a [begin, end) range of entries. | |
virtual void | SetNSlots (unsigned int nSlots) |
Inform RDataSource of the number of processing slots (i.e. | |
Protected Types | |
using | Record_t = std::vector<void *> |
Protected Member Functions | |
virtual std::string | AsString () |
virtual std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBase > | CreateColumnReader (unsigned int slot, std::string_view col, const std::type_info &tid, TTreeReader *) |
Creates a column reader for the requested column. | |
virtual ROOT::RDF::RSampleInfo | CreateSampleInfo (unsigned int, const std::unordered_map< std::string, ROOT::RDF::Experimental::RSample * > &) const |
virtual std::string | DescribeDataset () |
virtual const std::vector< std::string > & | GetColumnNamesNoDuplicates () const |
virtual Record_t | GetColumnReadersImpl (std::string_view name, const std::type_info &)=0 |
type-erased vector of pointers to pointers to column values - one per slot | |
virtual const std::vector< std::string > & | GetTopLevelFieldNames () const |
virtual std::string | GetTypeNameWithOpts (std::string_view colName, bool) const |
virtual void | InitializeWithOpts (const std::set< std::string > &) |
virtual void | ProcessMT (ROOT::Detail::RDF::RLoopManager &) |
virtual void | RunFinalChecks (bool) const |
Protected Attributes | |
std::optional< std::pair< ULong64_t, ULong64_t > > | fGlobalEntryRange {} |
unsigned int | fNSlots {} |
#include <ROOT/RDataSource.hxx>
|
protected |
Definition at line 145 of file RDataSource.hxx.
|
default |
|
delete |
|
delete |
|
virtualdefault |
|
inlineprotectedvirtual |
Reimplemented in ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::Internal::RDF::RRootDS, ROOT::RDF::RTrivialDS, and ROOT::Internal::RDF::RTTreeDS.
Definition at line 148 of file RDataSource.hxx.
|
inlineprotectedvirtual |
Creates a column reader for the requested column.
In the general case, this is just a redirect to the right GetColumnReaders overload. The signature notably also has a TTreeReader * parameter. This is currently necessary to still allow the TTree-based MT scheduling via TTreeProcessorMT. We use the TTreeProcessorMT::Process method to launch the same kernel across all threads. In each thread task, TTreeProcessorMT creates a thread-local instance of a TTreeReader which is going to read the range of events assigned to that task. That TTreeReader instance is what is passed to this method whenever a column reader needs to be created in a thread task. In the future this method might be removed by either allowing to request a handle to the thread-local TTreeReader instance programmatically from the TTreeProcessorMT, or refactoring the TTreeProcessorMT scheduling into RTTreeDS altogether.
Reimplemented in ROOT::Internal::RDF::RTTreeDS.
Definition at line 198 of file RDataSource.hxx.
|
protectedvirtual |
Reimplemented in ROOT::RDF::RNTupleDS, and ROOT::Internal::RDF::RTTreeDS.
Definition at line 11 of file RDataSource.cxx.
|
inlineprotectedvirtual |
Reimplemented in ROOT::Internal::RDF::RTTreeDS.
Definition at line 168 of file RDataSource.hxx.
|
inlinevirtual |
Convenience method called after concluding an event-loop.
See Initialize for more details.
Reimplemented in ROOT::RDF::RCsvDS, ROOT::RDF::RNTupleDS, and ROOT::Internal::RDF::RTTreeDS.
Definition at line 320 of file RDataSource.hxx.
Convenience method called at the end of the data processing associated to a slot.
[in] | slot | The data processing slot wihch needs to be finalized This method might be called multiple times per thread per event-loop. |
Reimplemented in ROOT::RDF::RNTupleDS, and ROOT::Internal::RDF::RRootDS.
Definition at line 314 of file RDataSource.hxx.
|
pure virtual |
Returns a reference to the collection of the dataset's column names.
Implemented in ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RArrowDS, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RNTupleDS, ROOT::Internal::RDF::RRootDS, ROOT::RDF::RSqliteDS, ROOT::RDF::RTrivialDS, and ROOT::Internal::RDF::RTTreeDS.
|
inlineprotectedvirtual |
Reimplemented in ROOT::Internal::RDF::RTTreeDS.
Definition at line 162 of file RDataSource.hxx.
|
inline |
Called at most once per column by RDF.
Return vector of pointers to pointers to column values - one per slot.
T | The type of the data stored in the column |
[in] | columnName | The name of the column |
These pointers are veritable cursors: it's a responsibility of the RDataSource implementation that they point to the "right" memory region.
Definition at line 251 of file RDataSource.hxx.
|
inlinevirtual |
If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.
[in] | slot | The data processing slot that needs to be considered |
[in] | name | The name of the column for which a column reader needs to be returned |
[in] | tid | A type_info At least one of the two must return a non-empty/non-null value. |
Reimplemented in ROOT::RDF::RNTupleDS, and ROOT::Internal::RDF::RTTreeDS.
Definition at line 266 of file RDataSource.hxx.
|
protectedpure virtual |
type-erased vector of pointers to pointers to column values - one per slot
Implemented in ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RNTupleDS, ROOT::RDF::RSqliteDS, ROOT::RDF::RTrivialDS, ROOT::RDF::RArrowDS, ROOT::RDF::RCsvDS, ROOT::Internal::RDF::RRootDS, and ROOT::Internal::RDF::RTTreeDS.
|
pure virtual |
Return ranges of entries to distribute to tasks.
They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.
Implemented in ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RArrowDS, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RNTupleDS, ROOT::Internal::RDF::RRootDS, ROOT::RDF::RSqliteDS, ROOT::RDF::RTrivialDS, and ROOT::Internal::RDF::RTTreeDS.
|
inlinevirtual |
Return a string representation of the datasource type.
The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.
Reimplemented in ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RArrowDS, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RNTupleDS, ROOT::Internal::RDF::RRootDS, ROOT::RDF::RSqliteDS, ROOT::RDF::RTrivialDS, and ROOT::Internal::RDF::RTTreeDS.
Definition at line 326 of file RDataSource.hxx.
|
inlinevirtual |
Returns the number of files from which the dataset is constructed.
Reimplemented in ROOT::RDF::RCsvDS, ROOT::RDF::RNTupleDS, ROOT::Internal::RDF::RRootDS, and ROOT::Internal::RDF::RTTreeDS.
Definition at line 225 of file RDataSource.hxx.
|
inlineprotectedvirtual |
Reimplemented in ROOT::Internal::RDF::RTTreeDS.
Definition at line 158 of file RDataSource.hxx.
|
pure virtual |
Type of a column as a string, e.g.
GetTypeName("x") == "double"
. Required for jitting e.g. df.Filter("x>0")
.
[in] | colName | The name of the column |
Implemented in ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RArrowDS, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RNTupleDS, ROOT::Internal::RDF::RRootDS, ROOT::RDF::RSqliteDS, ROOT::Internal::RDF::RTTreeDS, and ROOT::RDF::RTrivialDS.
|
inlineprotectedvirtual |
Reimplemented in ROOT::Internal::RDF::RTTreeDS.
Definition at line 155 of file RDataSource.hxx.
Checks if the dataset has a certain column.
[in] | colName | The name of the column |
Implemented in ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RArrowDS, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RNTupleDS, ROOT::Internal::RDF::RRootDS, ROOT::RDF::RSqliteDS, ROOT::RDF::RTrivialDS, and ROOT::Internal::RDF::RTTreeDS.
|
inlinevirtual |
Convenience method called before starting an event-loop.
This method might be called multiple times over the lifetime of a RDataSource, since users can run multiple event-loops with the same RDataFrame. Ideally, Initialize
should set the state of the RDataSource so that multiple identical event-loops will produce identical results.
Reimplemented in ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RArrowDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RNTupleDS, ROOT::Internal::RDF::RRootDS, ROOT::RDF::RSqliteDS, ROOT::RDF::RTrivialDS, and ROOT::Internal::RDF::RTTreeDS.
Definition at line 299 of file RDataSource.hxx.
|
inlineprotectedvirtual |
Reimplemented in ROOT::Internal::RDF::RTTreeDS.
Definition at line 165 of file RDataSource.hxx.
Convenience method called at the start of the data processing associated to a slot.
[in] | slot | The data processing slot wihch needs to be initialized |
[in] | firstEntry | The first entry of the range that the task will process. This method might be called multiple times per thread per event-loop. |
Reimplemented in ROOT::RDF::RArrowDS, ROOT::RDF::RNTupleDS, and ROOT::Internal::RDF::RRootDS.
Definition at line 307 of file RDataSource.hxx.
|
delete |
|
delete |
|
protectedvirtual |
Reimplemented in ROOT::Internal::RDF::RTTreeDS.
Definition at line 19 of file RDataSource.cxx.
|
inlineprotectedvirtual |
Reimplemented in ROOT::Internal::RDF::RTTreeDS.
Definition at line 177 of file RDataSource.hxx.
Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.
[in] | slot | The data processing slot that needs to be considered |
[in] | entry | The entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise. |
Implemented in ROOT::Internal::RDF::RVecDS< ColumnTypes >, ROOT::RDF::RArrowDS, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::Internal::RDF::RRootDS, ROOT::RDF::RSqliteDS, ROOT::RDF::RTrivialDS, ROOT::Internal::RDF::RTTreeDS, and ROOT::RDF::RNTupleDS.
|
inlinevirtual |
Restrict processing to a [begin, end) range of entries.
entryRange | The range of entries to process. |
Definition at line 330 of file RDataSource.hxx.
Inform RDataSource of the number of processing slots (i.e.
worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.
Reimplemented in ROOT::RDF::RArrowDS, ROOT::RDF::RCsvDS, ROOT::RDF::RLazyDS< ColumnTypes >, ROOT::RDF::RNTupleDS, ROOT::Internal::RDF::RRootDS, ROOT::RDF::RSqliteDS, ROOT::RDF::RTrivialDS, and ROOT::Internal::RDF::RVecDS< ColumnTypes >.
Definition at line 217 of file RDataSource.hxx.
|
friend |
|
friend |
|
friend |
|
friend |
|
friend |
|
friend |
|
friend |
|
friend |
|
friend |
|
protected |
Definition at line 152 of file RDataSource.hxx.
|
protected |
Definition at line 150 of file RDataSource.hxx.