RDataFrame data source class to interface with Apache Arrow.

The RArrowDS implements a proxy RDataSource to be able to use Apache Arrow tables with RDataFrame.

A RDataFrame that adapts an arrow::Table class can be constructed using the factory method ROOT::RDF::MakeArrowDataFrame, which accepts one parameter:

An arrow::Table smart pointer.

The types of the columns are derived from the types in the associated arrow::Schema.

Definition at line 23 of file RArrowDS.hxx.

Public Member Functions
	RArrowDS (std::shared_ptr< arrow::Table > table, std::vector< std::string > const &columns)
	Constructor to create an Arrow RDataSource for RDataFrame. More...

	~RArrowDS ()
	Destructor. More...

const std::vector< std::string > &	GetColumnNames () const override
	Returns a reference to the collection of the dataset's column names. More...

std::vector< std::pair< ULong64_t, ULong64_t > >	GetEntryRanges () override
	Return ranges of entries to distribute to tasks. More...

std::string	GetTypeName (std::string_view colName) const override
	Type of a column as a string, e.g. More...

bool	HasColumn (std::string_view colName) const override
	Checks if the dataset has a certain column. More...

void	Initialise () override
	Convenience method called before starting an event-loop. More...

void	InitSlot (unsigned int slot, ULong64_t firstEntry) override
	Convenience method called at the start of the data processing associated to a slot. More...

bool	SetEntry (unsigned int slot, ULong64_t entry) override
	Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot. More...

void	SetNSlots (unsigned int nSlots) override
	Inform RDataSource of the number of processing slots (i.e. More...

Public Member Functions inherited from ROOT::RDF::RDataSource
virtual	~RDataSource ()=default

virtual void	Finalise ()
	Convenience method called after concluding an event-loop. More...

virtual void	FinaliseSlot (unsigned int)
	Convenience method called at the end of the data processing associated to a slot. More...

template<typename T >
std::vector< T ** >	GetColumnReaders (std::string_view columnName)
	Called at most once per column by RDF. More...

Private Member Functions
std::vector< void * >	GetColumnReadersImpl (std::string_view name, const std::type_info &type) override
	This needs to return a pointer to the pointer each value getter will point to. More...

Private Attributes
std::vector< std::string >	fColumnNames

std::vector< std::pair< ULong64_t, ULong64_t > >	fEntryRanges

std::vector< std::pair< size_t, size_t > >	fGetterIndex

size_t	fNSlots = 0U

std::shared_ptr< arrow::Table >	fTable

std::vector< std::unique_ptr< ROOT::Internal::RDF::TValueGetter > >	fValueGetters

Additional Inherited Members
Protected Types inherited from ROOT::RDF::RDataSource
using	Record_t = std::vector< void * >

#include <ROOT/RArrowDS.hxx>

Inheritance diagram for ROOT::RDF::RArrowDS:

[legend]

Constructor & Destructor Documentation

◆ RArrowDS()

ROOT::RDF::RArrowDS::RArrowDS	(	std::shared_ptr< arrow::Table >	inTable,
		std::vector< std::string > const &	inColumns
	)

Constructor to create an Arrow RDataSource for RDataFrame.

Parameters

[in]	table	the arrow Table to observe.
[in]	columns	the name of the columns to use In case columns is empty, we use all the columns found in the table

Definition at line 289 of file RArrowDS.cxx.

◆ ~RArrowDS()

ROOT::RDF::RArrowDS::~RArrowDS ( )

Destructor.

Definition at line 364 of file RArrowDS.cxx.

Member Function Documentation

◆ GetColumnNames()

const std::vector< std::string > & ROOT::RDF::RArrowDS::GetColumnNames ( ) const

overridevirtual

Returns a reference to the collection of the dataset's column names.

Implements ROOT::RDF::RDataSource.

Definition at line 368 of file RArrowDS.cxx.

◆ GetColumnReadersImpl()

std::vector< void * > ROOT::RDF::RArrowDS::GetColumnReadersImpl	(	std::string_view	colName,
		const std::type_info &	type
	)

overrideprivatevirtual

This needs to return a pointer to the pointer each value getter will point to.

Implements ROOT::RDF::RDataSource.

Definition at line 472 of file RArrowDS.cxx.

◆ GetEntryRanges()

std::vector< std::pair< ULong64_t, ULong64_t > > ROOT::RDF::RArrowDS::GetEntryRanges ( )

overridevirtual

Return ranges of entries to distribute to tasks.

They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries.

Implements ROOT::RDF::RDataSource.

Definition at line 373 of file RArrowDS.cxx.

◆ GetTypeName()

std::string ROOT::RDF::RArrowDS::GetTypeName ( std::string_view ) const

overridevirtual

Type of a column as a string, e.g.

GetTypeName("x") == "double". Required for jitting e.g. df.Filter("x>0").

Parameters

[in] columnName The name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 379 of file RArrowDS.cxx.

◆ HasColumn()

bool ROOT::RDF::RArrowDS::HasColumn ( std::string_view ) const

overridevirtual

Checks if the dataset has a certain column.

Parameters

[in] columnName The name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 397 of file RArrowDS.cxx.

◆ Initialise()

void ROOT::RDF::RArrowDS::Initialise ( )

overridevirtual

Convenience method called before starting an event-loop.

This method might be called multiple times over the lifetime of a RDataSource, since users can run multiple event-loops with the same RDataFrame. Ideally, Initialise should set the state of the RDataSource so that multiple identical event-loops will produce identical results.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 492 of file RArrowDS.cxx.

◆ InitSlot()

void ROOT::RDF::RArrowDS::InitSlot	(	unsigned int	,
		ULong64_t
	)

overridevirtual

Convenience method called at the start of the data processing associated to a slot.

Parameters

[in]	slot	The data processing slot wihch needs to be initialised
[in]	firstEntry	The first entry of the range that the task will process. This method might be called multiple times per thread per event-loop.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 416 of file RArrowDS.cxx.

◆ SetEntry()

bool ROOT::RDF::RArrowDS::SetEntry	(	unsigned int	slot,
		ULong64_t	entry
	)

overridevirtual

Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.

Parameters

[in]	slot	The data processing slot that needs to be considered
[in]	entry	The entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise.

Implements ROOT::RDF::RDataSource.

Definition at line 406 of file RArrowDS.cxx.

◆ SetNSlots()

void ROOT::RDF::RArrowDS::SetNSlots ( unsigned int nSlots )

overridevirtual

Inform RDataSource of the number of processing slots (i.e.

worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.

Implements ROOT::RDF::RDataSource.

Definition at line 425 of file RArrowDS.cxx.

Member Data Documentation

◆ fColumnNames

std::vector<std::string> ROOT::RDF::RArrowDS::fColumnNames

private

Definition at line 27 of file RArrowDS.hxx.

◆ fEntryRanges

std::vector<std::pair<ULong64_t, ULong64_t> > ROOT::RDF::RArrowDS::fEntryRanges

private

Definition at line 26 of file RArrowDS.hxx.

◆ fGetterIndex

std::vector<std::pair<size_t, size_t> > ROOT::RDF::RArrowDS::fGetterIndex

private

Definition at line 30 of file RArrowDS.hxx.

◆ fNSlots

size_t ROOT::RDF::RArrowDS::fNSlots = 0U

private

Definition at line 28 of file RArrowDS.hxx.

◆ fTable

std::shared_ptr<arrow::Table> ROOT::RDF::RArrowDS::fTable

private

Definition at line 25 of file RArrowDS.hxx.

◆ fValueGetters

std::vector<std::unique_ptr<ROOT::Internal::RDF::TValueGetter> > ROOT::RDF::RArrowDS::fValueGetters

private

Definition at line 31 of file RArrowDS.hxx.

Libraries for ROOT::RDF::RArrowDS:

[legend]

Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Constructor & Destructor Documentation

◆ RArrowDS()

◆ ~RArrowDS()

Member Function Documentation

◆ GetColumnNames()

◆ GetColumnReadersImpl()

◆ GetEntryRanges()

◆ GetTypeName()

◆ HasColumn()

◆ Initialise()

◆ InitSlot()

◆ SetEntry()

◆ SetNSlots()

Member Data Documentation

◆ fColumnNames

◆ fEntryRanges

◆ fGetterIndex

◆ fNSlots

◆ fTable

◆ fValueGetters