RDataFrame data source class for reading CSV files.

The RCsvDS class implements a CSV file reader for RDataFrame.

A RDataFrame that reads from a CSV file can be constructed using the factory method ROOT::RDF::MakeCsvDataFrame, which accepts three parameters:

Path to the CSV file.
Boolean that specifies whether the first row of the CSV file contains headers or not (optional, default true). If false, header names will be automatically generated as Col0, Col1, ..., ColN.
Delimiter (optional, default ',').

The types of the columns in the CSV file are automatically inferred. The supported types are:

Integer: stored as a 64-bit long long int.
Floating point number: stored with double precision.
Boolean: matches the literals true and false.
String: stored as an std::string, matches anything that does not fall into any of the previous types.

These are some formatting rules expected by the RCsvDS implementation:

All records must have the same number of fields, in the same order.
Any field may be quoted.
"1997","Ford","E350"
Fields with embedded delimiters (e.g. comma) must be quoted.
1997,Ford,E350,"Super, luxurious truck"
Fields with double-quote characters must be quoted, and each of the embedded double-quote characters must be represented by a pair of double-quote characters.
1997,Ford,E350,"Super, ""luxurious"" truck"
Fields with embedded line breaks are not supported, even when quoted.
1997,Ford,E350,"Go get one now

they are going fast"
Spaces are considered part of a field and are not ignored.
1997, Ford , E350

not same as

1997,Ford,E350

but same as

1997, "Ford" , E350

Rcpp::as
TString as(SEXP s)
Definition: RExports.h:71
If a header row is provided, it must contain column names for each of the fields.
Year,Make,Model

1997,Ford,E350

2000,Mercury,Cougar

The current implementation of RCsvDS reads the entire CSV file content into memory before RDataFrame starts processing it. Therefore, before creating a CSV RDataFrame, it is important to check both how much memory is available and the size of the CSV file.

Definition at line 28 of file RCsvDS.hxx.

Public Member Functions
	RCsvDS (std::string_view fileName, bool readHeaders=true, char delimiter=',', Long64_t linesChunkSize=-1LL)
	Constructor to create a CSV RDataSource for RDataFrame. More...

	~RCsvDS ()
	Destructor. More...

void	Finalise ()
	Convenience method called after concluding an event-loop. More...

void	FreeRecords ()

const std::vector< std::string > &	GetColumnNames () const
	Returns a reference to the collection of the dataset's column names. More...

std::vector< std::pair< ULong64_t, ULong64_t > >	GetEntryRanges ()
	Return ranges of entries to distribute to tasks. More...

std::string	GetLabel ()
	Return a string representation of the datasource type. More...

std::string	GetTypeName (std::string_view colName) const
	Type of a column as a string, e.g. More...

bool	HasColumn (std::string_view colName) const
	Checks if the dataset has a certain column. More...

bool	SetEntry (unsigned int slot, ULong64_t entry)
	Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot. More...

void	SetNSlots (unsigned int nSlots)
	Inform RDataSource of the number of processing slots (i.e. More...

Public Member Functions inherited from ROOT::RDF::RDataSource
virtual	~RDataSource ()=default

virtual void	Finalise ()
	Convenience method called after concluding an event-loop. More...

virtual void	FinaliseSlot (unsigned int)
	Convenience method called at the end of the data processing associated to a slot. More...

virtual const std::vector< std::string > &	GetColumnNames () const =0
	Returns a reference to the collection of the dataset's column names. More...

template<typename T >
std::vector< T ** >	GetColumnReaders (std::string_view columnName)
	Called at most once per column by RDF. More...

virtual std::vector< std::pair< ULong64_t, ULong64_t > >	GetEntryRanges ()=0
	Return ranges of entries to distribute to tasks. More...

virtual std::string	GetLabel ()
	Return a string representation of the datasource type. More...

virtual std::string	GetTypeName (std::string_view) const =0
	Type of a column as a string, e.g. More...

virtual bool	HasColumn (std::string_view) const =0
	Checks if the dataset has a certain column. More...

virtual void	Initialise ()
	Convenience method called before starting an event-loop. More...

virtual void	InitSlot (unsigned int, ULong64_t)
	Convenience method called at the start of the data processing associated to a slot. More...

virtual bool	SetEntry (unsigned int slot, ULong64_t entry)=0
	Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot. More...

virtual void	SetNSlots (unsigned int nSlots)=0
	Inform RDataSource of the number of processing slots (i.e. More...

Protected Member Functions
std::string	AsString ()

virtual std::string	AsString ()

virtual Record_t	GetColumnReadersImpl (std::string_view name, const std::type_info &)=0
	type-erased vector of pointers to pointers to column values - one per slot More...

Private Types
using	ColType_t = char

Private Member Functions
void	FillHeaders (const std::string &)

void	FillRecord (const std::string &, Record_t &)

void	GenerateHeaders (size_t)

std::vector< void * >	GetColumnReadersImpl (std::string_view, const std::type_info &)
	type-erased vector of pointers to pointers to column values - one per slot More...

ColType_t	GetType (std::string_view colName) const

void	InferColTypes (std::vector< std::string > &)

void	InferType (const std::string &, unsigned int)

std::vector< std::string >	ParseColumns (const std::string &)

size_t	ParseValue (const std::string &, std::vector< std::string > &, size_t)

Private Attributes
std::vector< std::deque< bool > >	fBoolEvtValues

std::vector< std::vector< void * > >	fColAddresses

std::map< std::string, ColType_t >	fColTypes

std::list< ColType_t >	fColTypesList

std::streampos	fDataPos = 0

const char	fDelimiter

std::vector< std::vector< double > >	fDoubleEvtValues

ULong64_t	fEntryRangesRequested = 0ULL

std::vector< std::string >	fHeaders

const Long64_t	fLinesChunkSize

std::vector< std::vector< Long64_t > >	fLong64EvtValues

unsigned int	fNSlots = 0U

ULong64_t	fProcessedLines = 0ULL

bool	fReadHeaders = false

std::vector< Record_t >	fRecords

std::ifstream	fStream

std::vector< std::vector< std::string > >	fStringEvtValues

Static Private Attributes
static TRegexp	doubleRegex1

static TRegexp	doubleRegex2

static TRegexp	doubleRegex3

static TRegexp	falseRegex

static const std::map< ColType_t, std::string >	fgColTypeMap

static TRegexp	intRegex

static TRegexp	trueRegex

Additional Inherited Members
Protected Types inherited from ROOT::RDF::RDataSource
using	Record_t = std::vector< void * >

#include <ROOT/RCsvDS.hxx>

Inheritance diagram for ROOT::RDF::RCsvDS:

[legend]

Member Typedef Documentation

◆ ColType_t

using ROOT::RDF::RCsvDS::ColType_t = char

private

Definition at line 32 of file RCsvDS.hxx.

Constructor & Destructor Documentation

◆ RCsvDS()

ROOT::RDF::RCsvDS::RCsvDS	(	std::string_view	fileName,
		bool	readHeaders = `true`,
		char	delimiter = `','`,
		Long64_t	linesChunkSize = `-1LL`
	)

Constructor to create a CSV RDataSource for RDataFrame.

Parameters

[in]	fileName	Path of the CSV file.
[in]	readHeaders	`true` if the CSV file contains headers as first row, `false` otherwise (default `true`).
[in]	delimiter	Delimiter character (default ',').

Definition at line 260 of file RCsvDS.cxx.

◆ ~RCsvDS()

ROOT::RDF::RCsvDS::~RCsvDS ( )

Destructor.

Definition at line 335 of file RCsvDS.cxx.

Member Function Documentation

◆ AsString()

std::string ROOT::RDF::RCsvDS::AsString ( )

protectedvirtual

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 89 of file RCsvDS.cxx.

◆ FillHeaders()

void ROOT::RDF::RCsvDS::FillHeaders ( const std::string & line )

private

Definition at line 105 of file RCsvDS.cxx.

◆ FillRecord()

void ROOT::RDF::RCsvDS::FillRecord	(	const std::string &	line,
		Record_t &	record
	)

private

Definition at line 113 of file RCsvDS.cxx.

◆ Finalise()

void ROOT::RDF::RCsvDS::Finalise ( )

virtual

Convenience method called after concluding an event-loop.

See Initialise for more details.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 340 of file RCsvDS.cxx.

◆ FreeRecords()

void ROOT::RDF::RCsvDS::FreeRecords ( )

Definition at line 304 of file RCsvDS.cxx.

◆ GenerateHeaders()

void ROOT::RDF::RCsvDS::GenerateHeaders ( size_t size )

private

Definition at line 148 of file RCsvDS.cxx.

◆ GetColumnNames()

const std::vector< std::string > & ROOT::RDF::RCsvDS::GetColumnNames ( ) const

virtual

Returns a reference to the collection of the dataset's column names.

Implements ROOT::RDF::RDataSource.

Definition at line 349 of file RCsvDS.cxx.

◆ GetColumnReadersImpl()

std::vector< void * > ROOT::RDF::RCsvDS::GetColumnReadersImpl	(	std::string_view	name,
		const std::type_info &
	)

privatevirtual

type-erased vector of pointers to pointers to column values - one per slot

Implements ROOT::RDF::RDataSource.

Definition at line 155 of file RCsvDS.cxx.

◆ GetEntryRanges()

std::vector< std::pair< ULong64_t, ULong64_t > > ROOT::RDF::RCsvDS::GetEntryRanges ( )

virtual

Return ranges of entries to distribute to tasks.

They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.

Implements ROOT::RDF::RDataSource.

Definition at line 354 of file RCsvDS.cxx.

◆ GetLabel()

std::string ROOT::RDF::RCsvDS::GetLabel ( )

virtual

Return a string representation of the datasource type.

The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 470 of file RCsvDS.cxx.

◆ GetType()

RCsvDS::ColType_t ROOT::RDF::RCsvDS::GetType ( std::string_view colName ) const

private

Definition at line 401 of file RCsvDS.cxx.

◆ GetTypeName()

std::string ROOT::RDF::RCsvDS::GetTypeName ( std::string_view ) const

virtual

Type of a column as a string, e.g.

GetTypeName("x") == "double". Required for jitting e.g. df.Filter("x>0").

Parameters

[in] columnName The name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 412 of file RCsvDS.cxx.

◆ HasColumn()

bool ROOT::RDF::RCsvDS::HasColumn ( std::string_view ) const

virtual

Checks if the dataset has a certain column.

Parameters

[in] columnName The name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 417 of file RCsvDS.cxx.

◆ InferColTypes()

void ROOT::RDF::RCsvDS::InferColTypes ( std::vector< std::string > & columns )

private

Definition at line 187 of file RCsvDS.cxx.

◆ InferType()

void ROOT::RDF::RCsvDS::InferType	(	const std::string &	col,
		unsigned int	idxCol
	)

private

Definition at line 196 of file RCsvDS.cxx.

◆ ParseColumns()

std::vector< std::string > ROOT::RDF::RCsvDS::ParseColumns ( const std::string & line )

private

Definition at line 218 of file RCsvDS.cxx.

◆ ParseValue()

size_t ROOT::RDF::RCsvDS::ParseValue	(	const std::string &	line,
		std::vector< std::string > &	columns,
		size_t	i
	)

private

Definition at line 229 of file RCsvDS.cxx.

◆ SetEntry()

bool ROOT::RDF::RCsvDS::SetEntry	(	unsigned int	slot,
		ULong64_t	entry
	)

virtual

Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.

Parameters

[in]	slot	The data processing slot that needs to be considered
[in]	entry	The entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise.

Implements ROOT::RDF::RDataSource.

Definition at line 422 of file RCsvDS.cxx.

◆ SetNSlots()

void ROOT::RDF::RCsvDS::SetNSlots ( unsigned int nSlots )

virtual

Inform RDataSource of the number of processing slots (i.e.

worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.

Implements ROOT::RDF::RDataSource.

Definition at line 453 of file RCsvDS.cxx.

Member Data Documentation

◆ doubleRegex1

TRegexp ROOT::RDF::RCsvDS::doubleRegex1

staticprivate

Definition at line 55 of file RCsvDS.hxx.

◆ doubleRegex2

TRegexp ROOT::RDF::RCsvDS::doubleRegex2

staticprivate

Definition at line 55 of file RCsvDS.hxx.

◆ doubleRegex3

TRegexp ROOT::RDF::RCsvDS::doubleRegex3

staticprivate

Definition at line 55 of file RCsvDS.hxx.

◆ falseRegex

TRegexp ROOT::RDF::RCsvDS::falseRegex

staticprivate

Definition at line 55 of file RCsvDS.hxx.

◆ fBoolEvtValues

std::vector<std::deque<bool> > ROOT::RDF::RCsvDS::fBoolEvtValues

private

Definition at line 53 of file RCsvDS.hxx.

◆ fColAddresses

std::vector<std::vector<void *> > ROOT::RDF::RCsvDS::fColAddresses

private

Definition at line 46 of file RCsvDS.hxx.

◆ fColTypes

std::map<std::string, ColType_t> ROOT::RDF::RCsvDS::fColTypes

private

Definition at line 44 of file RCsvDS.hxx.

◆ fColTypesList

std::list<ColType_t> ROOT::RDF::RCsvDS::fColTypesList

private

Definition at line 45 of file RCsvDS.hxx.

◆ fDataPos

std::streampos ROOT::RDF::RCsvDS::fDataPos = 0

private

Definition at line 35 of file RCsvDS.hxx.

◆ fDelimiter

const char ROOT::RDF::RCsvDS::fDelimiter

private

Definition at line 39 of file RCsvDS.hxx.

◆ fDoubleEvtValues

std::vector<std::vector<double> > ROOT::RDF::RCsvDS::fDoubleEvtValues

private

Definition at line 48 of file RCsvDS.hxx.

◆ fEntryRangesRequested

ULong64_t ROOT::RDF::RCsvDS::fEntryRangesRequested = 0ULL

private

Definition at line 41 of file RCsvDS.hxx.

◆ fgColTypeMap

const std::map< RCsvDS::ColType_t, std::string > ROOT::RDF::RCsvDS::fgColTypeMap

staticprivate

Definition at line 33 of file RCsvDS.hxx.

◆ fHeaders

std::vector<std::string> ROOT::RDF::RCsvDS::fHeaders

private

Definition at line 43 of file RCsvDS.hxx.

◆ fLinesChunkSize

const Long64_t ROOT::RDF::RCsvDS::fLinesChunkSize

private

Definition at line 40 of file RCsvDS.hxx.

◆ fLong64EvtValues

std::vector<std::vector<Long64_t> > ROOT::RDF::RCsvDS::fLong64EvtValues

private

Definition at line 49 of file RCsvDS.hxx.

◆ fNSlots

unsigned int ROOT::RDF::RCsvDS::fNSlots = 0U

private

Definition at line 37 of file RCsvDS.hxx.

◆ fProcessedLines

ULong64_t ROOT::RDF::RCsvDS::fProcessedLines = 0ULL

private

Definition at line 42 of file RCsvDS.hxx.

◆ fReadHeaders

bool ROOT::RDF::RCsvDS::fReadHeaders = false

private

Definition at line 36 of file RCsvDS.hxx.

◆ fRecords

std::vector<Record_t> ROOT::RDF::RCsvDS::fRecords

private

Definition at line 47 of file RCsvDS.hxx.

◆ fStream

std::ifstream ROOT::RDF::RCsvDS::fStream

private

Definition at line 38 of file RCsvDS.hxx.

◆ fStringEvtValues

std::vector<std::vector<std::string> > ROOT::RDF::RCsvDS::fStringEvtValues

private

Definition at line 50 of file RCsvDS.hxx.

◆ intRegex

TRegexp ROOT::RDF::RCsvDS::intRegex

staticprivate

Definition at line 55 of file RCsvDS.hxx.

◆ trueRegex

TRegexp ROOT::RDF::RCsvDS::trueRegex

staticprivate

Definition at line 55 of file RCsvDS.hxx.

Libraries for ROOT::RDF::RCsvDS:

[legend]

Public Member Functions

Protected Member Functions

Private Types

Private Member Functions

Private Attributes

Static Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ ColType_t

Constructor & Destructor Documentation

◆ RCsvDS()

◆ ~RCsvDS()

Member Function Documentation

◆ AsString()

◆ FillHeaders()

◆ FillRecord()

◆ Finalise()

◆ FreeRecords()

◆ GenerateHeaders()

◆ GetColumnNames()

◆ GetColumnReadersImpl()

◆ GetEntryRanges()

◆ GetLabel()

◆ GetType()

◆ GetTypeName()

◆ HasColumn()

◆ InferColTypes()

◆ InferType()

◆ ParseColumns()

◆ ParseValue()

◆ SetEntry()

◆ SetNSlots()

Member Data Documentation

◆ doubleRegex1

◆ doubleRegex2

◆ doubleRegex3

◆ falseRegex

◆ fBoolEvtValues

◆ fColAddresses

◆ fColTypes

◆ fColTypesList

◆ fDataPos

◆ fDelimiter

◆ fDoubleEvtValues

◆ fEntryRangesRequested

◆ fgColTypeMap

◆ fHeaders

◆ fLinesChunkSize

◆ fLong64EvtValues

◆ fNSlots

◆ fProcessedLines

◆ fReadHeaders

◆ fRecords

◆ fStream

◆ fStringEvtValues

◆ intRegex

◆ trueRegex