Logo ROOT  
Reference Guide
 
Loading...
Searching...
No Matches
ROOT::RDF::RCsvDS Class Referencefinal

RDataFrame data source class for reading CSV files.

The RCsvDS class implements a CSV file reader for RDataFrame.

A RDataFrame that reads from a CSV file can be constructed using the factory method ROOT::RDF::FromCSV, which accepts five parameters:

  1. Path to the CSV file.
  2. Boolean that specifies whether the first row of the CSV file contains headers or not (optional, default true). If false, header names will be automatically generated as Col0, Col1, ..., ColN.
  3. Delimiter (optional, default ',').
  4. Chunk size (optional, default is -1 to read all) - number of lines to read at a time
  5. Column Types (optional, default is an empty map). A map with column names as keys and their type (expressed as a single character, see below) as values.

The type of columns that do not appear in the map is inferred from the data. The supported types are:

  • Integer: stored as a 64-bit long long int; can be specified in the column types map with 'L'.
  • Floating point number: stored with double precision; specified with 'D'.
  • Boolean: matches the literals true and false; specified with 'O'.
  • String: stored as an std::string, matches anything that does not fall into any of the previous types; specified with 'T'.

These are some formatting rules expected by the RCsvDS implementation:

  • All records must have the same number of fields, in the same order.
  • Any field may be quoted.
    "1997","Ford","E350"
  • Fields with embedded delimiters (e.g. comma) must be quoted.
    1997,Ford,E350,"Super, luxurious truck"
  • Fields with double-quote characters must be quoted, and each of the embedded double-quote characters must be represented by a pair of double-quote characters.
    1997,Ford,E350,"Super, ""luxurious"" truck"
  • Fields with embedded line breaks are not supported, even when quoted.
    1997,Ford,E350,"Go get one now
    they are going fast"
  • Spaces are considered part of a field and are not ignored.
    1997, Ford , E350
    not same as
    1997,Ford,E350
    but same as
    1997, "Ford" , E350
  • If a header row is provided, it must contain column names for each of the fields.
    Year,Make,Model
    1997,Ford,E350
    2000,Mercury,Cougar

The current implementation of RCsvDS reads the entire CSV file content into memory before RDataFrame starts processing it. Therefore, before creating a CSV RDataFrame, it is important to check both how much memory is available and the size of the CSV file.

RCsvDS can handle empty cells and also allows the usage of the special keywords "NaN" and "nan" to indicate nan values. If the column is of type double, these cells are stored internally as nan. Empty cells and explicit nan-s inside columns of type Long64_t/bool are stored as zeros.

Definition at line 35 of file RCsvDS.hxx.

Public Member Functions

 RCsvDS (std::string_view fileName, bool readHeaders=true, char delimiter=',', Long64_t linesChunkSize=-1LL, std::unordered_map< std::string, char > &&colTypes={})
 Constructor to create a CSV RDataSource for RDataFrame.
 
 ~RCsvDS ()
 Destructor.
 
void Finalize () final
 Convenience method called after concluding an event-loop.
 
const std::vector< std::string > & GetColumnNames () const final
 Returns a reference to the collection of the dataset's column names.
 
std::vector< std::pair< ULong64_t, ULong64_t > > GetEntryRanges () final
 Return ranges of entries to distribute to tasks.
 
std::string GetLabel () final
 Return a string representation of the datasource type.
 
std::size_t GetNFiles () const final
 Returns the number of files from which the dataset is constructed.
 
std::string GetTypeName (std::string_view colName) const final
 Type of a column as a string, e.g.
 
bool HasColumn (std::string_view colName) const final
 Checks if the dataset has a certain column.
 
bool SetEntry (unsigned int slot, ULong64_t entry) final
 Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.
 
void SetNSlots (unsigned int nSlots) final
 Inform RDataSource of the number of processing slots (i.e.
 
- Public Member Functions inherited from ROOT::RDF::RDataSource
virtual ~RDataSource ()=default
 
virtual void FinalizeSlot (unsigned int)
 Convenience method called at the end of the data processing associated to a slot.
 
template<typename T >
std::vector< T ** > GetColumnReaders (std::string_view columnName)
 Called at most once per column by RDF.
 
virtual std::unique_ptr< ROOT::Detail::RDF::RColumnReaderBaseGetColumnReaders (unsigned int, std::string_view, const std::type_info &)
 If the other GetColumnReaders overload returns an empty vector, this overload will be called instead.
 
virtual void Initialize ()
 Convenience method called before starting an event-loop.
 
virtual void InitSlot (unsigned int, ULong64_t)
 Convenience method called at the start of the data processing associated to a slot.
 

Protected Member Functions

std::string AsString () final
 

Private Types

using ColType_t = char
 

Private Member Functions

void FillHeaders (const std::string &)
 
void FillRecord (const std::string &, Record_t &)
 
void FreeRecords ()
 
void GenerateHeaders (size_t)
 
std::vector< void * > GetColumnReadersImpl (std::string_view, const std::type_info &) final
 type-erased vector of pointers to pointers to column values - one per slot
 
ColType_t GetType (std::string_view colName) const
 
void InferColTypes (std::vector< std::string > &)
 
void InferType (const std::string &, unsigned int)
 
std::vector< std::string > ParseColumns (const std::string &)
 
size_t ParseValue (const std::string &, std::vector< std::string > &, size_t)
 
void ValidateColTypes (std::vector< std::string > &) const
 

Private Attributes

std::vector< std::deque< bool > > fBoolEvtValues
 
std::vector< std::vector< void * > > fColAddresses
 
std::set< std::string > fColContainingEmpty
 
std::unordered_map< std::string, ColType_tfColTypes
 
std::list< ColType_tfColTypesList
 
std::unique_ptr< ROOT::Internal::RRawFilefCsvFile
 
std::uint64_t fDataPos = 0
 
const char fDelimiter
 
std::vector< std::vector< double > > fDoubleEvtValues
 
ULong64_t fEntryRangesRequested = 0ULL
 
std::vector< std::string > fHeaders
 
const Long64_t fLinesChunkSize
 
std::vector< std::vector< Long64_t > > fLong64EvtValues
 
unsigned int fNSlots = 0U
 
ULong64_t fProcessedLines = 0ULL
 
bool fReadHeaders = false
 
std::vector< Record_tfRecords
 
std::vector< std::vector< std::string > > fStringEvtValues
 

Static Private Attributes

static const std::unordered_map< ColType_t, std::string > fgColTypeMap
 
static const TRegexp fgDoubleRegex1
 
static const TRegexp fgDoubleRegex2
 
static const TRegexp fgDoubleRegex3
 
static const TRegexp fgFalseRegex
 
static const TRegexp fgIntRegex
 
static const TRegexp fgTrueRegex
 

Additional Inherited Members

- Protected Types inherited from ROOT::RDF::RDataSource
using Record_t = std::vector< void * >
 

#include <ROOT/RCsvDS.hxx>

Inheritance diagram for ROOT::RDF::RCsvDS:
[legend]

Member Typedef Documentation

◆ ColType_t

using ROOT::RDF::RCsvDS::ColType_t = char
private

Definition at line 39 of file RCsvDS.hxx.

Constructor & Destructor Documentation

◆ RCsvDS()

ROOT::RDF::RCsvDS::RCsvDS ( std::string_view  fileName,
bool  readHeaders = true,
char  delimiter = ',',
Long64_t  linesChunkSize = -1LL,
std::unordered_map< std::string, char > &&  colTypes = {} 
)

Constructor to create a CSV RDataSource for RDataFrame.

Parameters
[in]fileNamePath or URL of the CSV file.
[in]readHeaderstrue if the CSV file contains headers as first row, false otherwise (default true).
[in]delimiterDelimiter character (default ',').
[in]linesChunkSizebunch of lines to read, use -1 to read all
[in]colTypesAllows users to manually specify column types. Accepts an unordered map with keys being column names, values being type specifiers ('O' for boolean, 'D' for double, 'L' for Long64_t, 'T' for std::string)

Definition at line 331 of file RCsvDS.cxx.

◆ ~RCsvDS()

ROOT::RDF::RCsvDS::~RCsvDS ( )

Destructor.

Definition at line 408 of file RCsvDS.cxx.

Member Function Documentation

◆ AsString()

std::string ROOT::RDF::RCsvDS::AsString ( )
finalprotectedvirtual

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 95 of file RCsvDS.cxx.

◆ FillHeaders()

void ROOT::RDF::RCsvDS::FillHeaders ( const std::string &  line)
private

Definition at line 111 of file RCsvDS.cxx.

◆ FillRecord()

void ROOT::RDF::RCsvDS::FillRecord ( const std::string &  line,
Record_t record 
)
private

Definition at line 120 of file RCsvDS.cxx.

◆ Finalize()

void ROOT::RDF::RCsvDS::Finalize ( )
finalvirtual

Convenience method called after concluding an event-loop.

See Initialize for more details.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 413 of file RCsvDS.cxx.

◆ FreeRecords()

void ROOT::RDF::RCsvDS::FreeRecords ( )
private

Definition at line 377 of file RCsvDS.cxx.

◆ GenerateHeaders()

void ROOT::RDF::RCsvDS::GenerateHeaders ( size_t  size)
private

Definition at line 163 of file RCsvDS.cxx.

◆ GetColumnNames()

const std::vector< std::string > & ROOT::RDF::RCsvDS::GetColumnNames ( ) const
finalvirtual

Returns a reference to the collection of the dataset's column names.

Implements ROOT::RDF::RDataSource.

Definition at line 421 of file RCsvDS.cxx.

◆ GetColumnReadersImpl()

std::vector< void * > ROOT::RDF::RCsvDS::GetColumnReadersImpl ( std::string_view  name,
const std::type_info &   
)
finalprivatevirtual

type-erased vector of pointers to pointers to column values - one per slot

Implements ROOT::RDF::RDataSource.

Definition at line 171 of file RCsvDS.cxx.

◆ GetEntryRanges()

std::vector< std::pair< ULong64_t, ULong64_t > > ROOT::RDF::RCsvDS::GetEntryRanges ( )
finalvirtual

Return ranges of entries to distribute to tasks.

They are required to be contiguous intervals with no entries skipped. Supposing a dataset with nEntries, the intervals must start at 0 and end at nEntries, e.g. [0-5],[5-10] for 10 entries. This function will be invoked repeatedly by RDataFrame as it needs additional entries to process. The same entry range should not be returned more than once. Returning an empty collection of ranges signals to RDataFrame that the processing can stop.

Implements ROOT::RDF::RDataSource.

Definition at line 426 of file RCsvDS.cxx.

◆ GetLabel()

std::string ROOT::RDF::RCsvDS::GetLabel ( )
finalvirtual

Return a string representation of the datasource type.

The returned string will be used by ROOT::RDF::SaveGraph() to represent the datasource in the visualization of the computation graph. Concrete datasources can override the default implementation.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 553 of file RCsvDS.cxx.

◆ GetNFiles()

std::size_t ROOT::RDF::RCsvDS::GetNFiles ( ) const
inlinefinalvirtual

Returns the number of files from which the dataset is constructed.

Reimplemented from ROOT::RDF::RDataSource.

Definition at line 86 of file RCsvDS.hxx.

◆ GetType()

RCsvDS::ColType_t ROOT::RDF::RCsvDS::GetType ( std::string_view  colName) const
private

Definition at line 484 of file RCsvDS.cxx.

◆ GetTypeName()

std::string ROOT::RDF::RCsvDS::GetTypeName ( std::string_view  colName) const
finalvirtual

Type of a column as a string, e.g.

GetTypeName("x") == "double". Required for jitting e.g. df.Filter("x>0").

Parameters
[in]colNameThe name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 495 of file RCsvDS.cxx.

◆ HasColumn()

bool ROOT::RDF::RCsvDS::HasColumn ( std::string_view  colName) const
finalvirtual

Checks if the dataset has a certain column.

Parameters
[in]colNameThe name of the column

Implements ROOT::RDF::RDataSource.

Definition at line 500 of file RCsvDS.cxx.

◆ InferColTypes()

void ROOT::RDF::RCsvDS::InferColTypes ( std::vector< std::string > &  columns)
private

Definition at line 222 of file RCsvDS.cxx.

◆ InferType()

void ROOT::RDF::RCsvDS::InferType ( const std::string &  col,
unsigned int  idxCol 
)
private

Definition at line 255 of file RCsvDS.cxx.

◆ ParseColumns()

std::vector< std::string > ROOT::RDF::RCsvDS::ParseColumns ( const std::string &  line)
private

Definition at line 276 of file RCsvDS.cxx.

◆ ParseValue()

size_t ROOT::RDF::RCsvDS::ParseValue ( const std::string &  line,
std::vector< std::string > &  columns,
size_t  i 
)
private

Definition at line 287 of file RCsvDS.cxx.

◆ SetEntry()

bool ROOT::RDF::RCsvDS::SetEntry ( unsigned int  slot,
ULong64_t  entry 
)
finalvirtual

Advance the "cursors" returned by GetColumnReaders to the selected entry for a particular slot.

Parameters
[in]slotThe data processing slot that needs to be considered
[in]entryThe entry which needs to be pointed to by the reader pointers Slots are adopted to accommodate parallel data processing. Different workers will loop over different ranges and will be labelled by different "slot" values. Returns true if the entry has to be processed, false otherwise.

Implements ROOT::RDF::RDataSource.

Definition at line 505 of file RCsvDS.cxx.

◆ SetNSlots()

void ROOT::RDF::RCsvDS::SetNSlots ( unsigned int  nSlots)
finalvirtual

Inform RDataSource of the number of processing slots (i.e.

worker threads) used by the associated RDataFrame. Slots numbers are used to simplify parallel execution: RDataFrame guarantees that different threads will always pass different slot values when calling methods concurrently.

Implements ROOT::RDF::RDataSource.

Definition at line 536 of file RCsvDS.cxx.

◆ ValidateColTypes()

void ROOT::RDF::RCsvDS::ValidateColTypes ( std::vector< std::string > &  columns) const
private

Definition at line 203 of file RCsvDS.cxx.

Member Data Documentation

◆ fBoolEvtValues

std::vector<std::deque<bool> > ROOT::RDF::RCsvDS::fBoolEvtValues
private

Definition at line 64 of file RCsvDS.hxx.

◆ fColAddresses

std::vector<std::vector<void *> > ROOT::RDF::RCsvDS::fColAddresses
private

Definition at line 57 of file RCsvDS.hxx.

◆ fColContainingEmpty

std::set<std::string> ROOT::RDF::RCsvDS::fColContainingEmpty
private

Definition at line 55 of file RCsvDS.hxx.

◆ fColTypes

std::unordered_map<std::string, ColType_t> ROOT::RDF::RCsvDS::fColTypes
private

Definition at line 54 of file RCsvDS.hxx.

◆ fColTypesList

std::list<ColType_t> ROOT::RDF::RCsvDS::fColTypesList
private

Definition at line 56 of file RCsvDS.hxx.

◆ fCsvFile

std::unique_ptr<ROOT::Internal::RRawFile> ROOT::RDF::RCsvDS::fCsvFile
private

Definition at line 48 of file RCsvDS.hxx.

◆ fDataPos

std::uint64_t ROOT::RDF::RCsvDS::fDataPos = 0
private

Definition at line 45 of file RCsvDS.hxx.

◆ fDelimiter

const char ROOT::RDF::RCsvDS::fDelimiter
private

Definition at line 49 of file RCsvDS.hxx.

◆ fDoubleEvtValues

std::vector<std::vector<double> > ROOT::RDF::RCsvDS::fDoubleEvtValues
private

Definition at line 59 of file RCsvDS.hxx.

◆ fEntryRangesRequested

ULong64_t ROOT::RDF::RCsvDS::fEntryRangesRequested = 0ULL
private

Definition at line 51 of file RCsvDS.hxx.

◆ fgColTypeMap

const std::unordered_map< RCsvDS::ColType_t, std::string > ROOT::RDF::RCsvDS::fgColTypeMap
staticprivate

Definition at line 40 of file RCsvDS.hxx.

◆ fgDoubleRegex1

const TRegexp ROOT::RDF::RCsvDS::fgDoubleRegex1
staticprivate

Definition at line 43 of file RCsvDS.hxx.

◆ fgDoubleRegex2

const TRegexp ROOT::RDF::RCsvDS::fgDoubleRegex2
staticprivate

Definition at line 43 of file RCsvDS.hxx.

◆ fgDoubleRegex3

const TRegexp ROOT::RDF::RCsvDS::fgDoubleRegex3
staticprivate

Definition at line 43 of file RCsvDS.hxx.

◆ fgFalseRegex

const TRegexp ROOT::RDF::RCsvDS::fgFalseRegex
staticprivate

Definition at line 43 of file RCsvDS.hxx.

◆ fgIntRegex

const TRegexp ROOT::RDF::RCsvDS::fgIntRegex
staticprivate

Definition at line 43 of file RCsvDS.hxx.

◆ fgTrueRegex

const TRegexp ROOT::RDF::RCsvDS::fgTrueRegex
staticprivate

Definition at line 43 of file RCsvDS.hxx.

◆ fHeaders

std::vector<std::string> ROOT::RDF::RCsvDS::fHeaders
private

Definition at line 53 of file RCsvDS.hxx.

◆ fLinesChunkSize

const Long64_t ROOT::RDF::RCsvDS::fLinesChunkSize
private

Definition at line 50 of file RCsvDS.hxx.

◆ fLong64EvtValues

std::vector<std::vector<Long64_t> > ROOT::RDF::RCsvDS::fLong64EvtValues
private

Definition at line 60 of file RCsvDS.hxx.

◆ fNSlots

unsigned int ROOT::RDF::RCsvDS::fNSlots = 0U
private

Definition at line 47 of file RCsvDS.hxx.

◆ fProcessedLines

ULong64_t ROOT::RDF::RCsvDS::fProcessedLines = 0ULL
private

Definition at line 52 of file RCsvDS.hxx.

◆ fReadHeaders

bool ROOT::RDF::RCsvDS::fReadHeaders = false
private

Definition at line 46 of file RCsvDS.hxx.

◆ fRecords

std::vector<Record_t> ROOT::RDF::RCsvDS::fRecords
private

Definition at line 58 of file RCsvDS.hxx.

◆ fStringEvtValues

std::vector<std::vector<std::string> > ROOT::RDF::RCsvDS::fStringEvtValues
private

Definition at line 61 of file RCsvDS.hxx.

Libraries for ROOT::RDF::RCsvDS:

The documentation for this class was generated from the following files: