Wrapper sink that coalesces cluster column page writes.

TODO(jblomer): The interplay of derived class and RPageSink is not yet optimally designed for page storage wrapper classes like this one. Header and footer serialization, e.g., are done twice. To be revised.

Definition at line 42 of file RPageSinkBuf.hxx.

Classes
class	RColumnBuf
	A buffered column. More...

struct	RCounters
	I/O performance counters that get registered in fMetrics. More...

Public Member Functions
	RPageSinkBuf (const RPageSinkBuf &)=delete

	RPageSinkBuf (RPageSinkBuf &&)=default

	RPageSinkBuf (std::unique_ptr< RPageSink > inner)

	~RPageSinkBuf () override=default

RNTupleMetrics &	GetMetrics () final
	Returns the default metrics object. Subclasses might alternatively provide their own metrics object by overriding this.

RPageSinkBuf &	operator= (const RPageSinkBuf &)=delete

RPageSinkBuf &	operator= (RPageSinkBuf &&)=default

void	ReleasePage (RPage &page) final
	Every page store needs to be able to free pages it handed out.

RPage	ReservePage (ColumnHandle_t columnHandle, std::size_t nElements) final
	Get a new, empty page for the given column that can be filled with up to nElements.

void	UpdateSchema (const RNTupleModelChangeset &changeset) final
	Incorporate incremental changes to the model into the ntuple descriptor.

Public Member Functions inherited from ROOT::Experimental::Detail::RPageSink
	RPageSink (const RPageSink &)=delete

	RPageSink (RPageSink &&)=default

	RPageSink (std::string_view ntupleName, const RNTupleWriteOptions &options)

	~RPageSink () override

ColumnHandle_t	AddColumn (DescriptorId_t fieldId, const RColumn &column) final
	Register a new column.

std::uint64_t	CommitCluster (NTupleSize_t nEntries)
	Finalize the current cluster and create a new one for the following data.

void	CommitClusterGroup ()
	Write out the page locations (page list envelope) for all the committed clusters since the last call of CommitClusterGroup (or the beginning of writing).

void	CommitDataset ()
	Finalize the current cluster and the entrire data set.

void	CommitPage (ColumnHandle_t columnHandle, const RPage &page)
	Write a page to the storage. The column must have been added before.

void	CommitSealedPage (DescriptorId_t physicalColumnId, const RPageStorage::RSealedPage &sealedPage)
	Write a preprocessed page to storage. The column must have been added before.

void	CommitSealedPageV (std::span< RPageStorage::RSealedPageGroup > ranges)
	Write a vector of preprocessed pages to storage. The corresponding columns must have been added before.

void	Create (RNTupleModel &model)
	Physically creates the storage container to hold the ntuple (e.g., a keys a TFile or an S3 bucket) To do so, Create() calls CreateImpl() after updating the descriptor.

void	DropColumn (ColumnHandle_t) final
	Unregisters a column.

RNTupleMetrics &	GetMetrics () override
	Returns the default metrics object. Subclasses might alternatively provide their own metrics object by overriding this.

EPageStorageType	GetType () final
	Whether the concrete implementation is a sink or a source.

const RNTupleWriteOptions &	GetWriteOptions () const
	Returns the sink's write options.

RPageSink &	operator= (const RPageSink &)=delete

RPageSink &	operator= (RPageSink &&)=default

virtual RPage	ReservePage (ColumnHandle_t columnHandle, std::size_t nElements)=0
	Get a new, empty page for the given column that can be filled with up to nElements.

virtual void	UpdateSchema (const RNTupleModelChangeset &changeset)
	Incorporate incremental changes to the model into the ntuple descriptor.

Public Member Functions inherited from ROOT::Experimental::Detail::RPageStorage
	RPageStorage (const RPageStorage &other)=delete

	RPageStorage (RPageStorage &&other)=default

	RPageStorage (std::string_view name)

virtual	~RPageStorage ()

virtual ColumnHandle_t	AddColumn (DescriptorId_t fieldId, const RColumn &column)=0
	Register a new column.

virtual void	DropColumn (ColumnHandle_t columnHandle)=0
	Unregisters a column.

virtual RNTupleMetrics &	GetMetrics ()=0
	Page storage implementations have their own metrics.

const std::string &	GetNTupleName () const
	Returns the NTuple name.

virtual EPageStorageType	GetType ()=0
	Whether the concrete implementation is a sink or a source.

RPageStorage &	operator= (const RPageStorage &other)=delete

RPageStorage &	operator= (RPageStorage &&other)=default

virtual void	ReleasePage (RPage &page)=0
	Every page store needs to be able to free pages it handed out.

void	SetTaskScheduler (RTaskScheduler *taskScheduler)

Protected Member Functions
RNTupleLocator	CommitClusterGroupImpl (unsigned char *serializedPageList, std::uint32_t length) final
	Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

std::uint64_t	CommitClusterImpl (NTupleSize_t nEntries) final
	Returns the number of bytes written to storage (excluding metadata)

void	CommitDatasetImpl (unsigned char *serializedFooter, std::uint32_t length) final

RNTupleLocator	CommitPageImpl (ColumnHandle_t columnHandle, const RPage &page) final

RNTupleLocator	CommitSealedPageImpl (DescriptorId_t physicalColumnId, const RSealedPage &sealedPage) final

void	CreateImpl (const RNTupleModel &model, unsigned char *serializedHeader, std::uint32_t length) final

Protected Member Functions inherited from ROOT::Experimental::Detail::RPageSink
virtual RNTupleLocator	CommitClusterGroupImpl (unsigned char *serializedPageList, std::uint32_t length)=0
	Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

virtual std::uint64_t	CommitClusterImpl (NTupleSize_t nEntries)=0
	Returns the number of bytes written to storage (excluding metadata)

virtual void	CommitDatasetImpl (unsigned char *serializedFooter, std::uint32_t length)=0

virtual RNTupleLocator	CommitPageImpl (ColumnHandle_t columnHandle, const RPage &page)=0

virtual RNTupleLocator	CommitSealedPageImpl (DescriptorId_t physicalColumnId, const RPageStorage::RSealedPage &sealedPage)=0

virtual std::vector< RNTupleLocator >	CommitSealedPageVImpl (std::span< RPageStorage::RSealedPageGroup > ranges)
	Vector commit of preprocessed pages.

virtual void	CreateImpl (const RNTupleModel &model, unsigned char *serializedHeader, std::uint32_t length)=0

void	EnableDefaultMetrics (const std::string &prefix)
	Enables the default set of metrics provided by RPageSink.

RSealedPage	SealPage (const RPage &page, const RColumnElementBase &element, int compressionSetting)
	Helper for streaming a page.

Protected Member Functions inherited from ROOT::Experimental::Detail::RPageStorage
void	WaitForAllTasks ()

Private Attributes
std::vector< RColumnBuf >	fBufferedColumns
	Vector of buffered column pages. Indexed by column id.

std::unique_ptr< RCounters >	fCounters

std::unique_ptr< RNTupleModel >	fInnerModel
	The buffered page sink maintains a copy of the RNTupleModel for the inner sink.

std::unique_ptr< RPageSink >	fInnerSink
	The inner sink, responsible for actually performing I/O.

RNTupleMetrics	fMetrics

Additional Inherited Members
Public Types inherited from ROOT::Experimental::Detail::RPageStorage
using	ColumnHandle_t = RColumnHandle
	The column handle identifies a column with the current open page storage.

using	SealedPageSequence_t = std::deque< RSealedPage >

Static Public Member Functions inherited from ROOT::Experimental::Detail::RPageSink
static std::unique_ptr< RPageSink >	Create (std::string_view ntupleName, std::string_view location, const RNTupleWriteOptions &options=RNTupleWriteOptions())
	Guess the concrete derived page source from the file name (location)

Static Protected Member Functions inherited from ROOT::Experimental::Detail::RPageSink
static RSealedPage	SealPage (const RPage &page, const RColumnElementBase &element, int compressionSetting, void *buf)
	Seal a page using the provided buffer.

Protected Attributes inherited from ROOT::Experimental::Detail::RPageSink
std::unique_ptr< RNTupleCompressor >	fCompressor
	Helper to zip pages and header/footer; includes a 16MB (kMAXZIPBUF) zip buffer.

std::unique_ptr< RCounters >	fCounters

RNTupleDescriptorBuilder	fDescriptorBuilder

RNTupleMetrics	fMetrics

std::uint64_t	fNextClusterInGroup = 0
	Remembers the starting cluster id for the next cluster group.

std::vector< RClusterDescriptor::RColumnRange >	fOpenColumnRanges
	Keeps track of the number of elements in the currently open cluster. Indexed by column id.

std::vector< RClusterDescriptor::RPageRange >	fOpenPageRanges
	Keeps track of the written pages in the currently open cluster. Indexed by column id.

std::unique_ptr< RNTupleWriteOptions >	fOptions

NTupleSize_t	fPrevClusterNEntries = 0
	Used to calculate the number of entries in the current cluster.

Protected Attributes inherited from ROOT::Experimental::Detail::RPageStorage
std::string	fNTupleName

RTaskScheduler *	fTaskScheduler = nullptr

#include <ROOT/RPageSinkBuf.hxx>

Inheritance diagram for ROOT::Experimental::Detail::RPageSinkBuf:

[legend]

Constructor & Destructor Documentation

◆ RPageSinkBuf() [1/3]

ROOT::Experimental::Detail::RPageSinkBuf::RPageSinkBuf ( std::unique_ptr< RPageSink > inner )

explicit

Definition at line 24 of file RPageSinkBuf.cxx.

◆ RPageSinkBuf() [2/3]

ROOT::Experimental::Detail::RPageSinkBuf::RPageSinkBuf ( const RPageSinkBuf & )

delete

◆ RPageSinkBuf() [3/3]

ROOT::Experimental::Detail::RPageSinkBuf::RPageSinkBuf ( RPageSinkBuf && )

default

◆ ~RPageSinkBuf()

ROOT::Experimental::Detail::RPageSinkBuf::~RPageSinkBuf ( )

overridedefault

Member Function Documentation

◆ CommitClusterGroupImpl()

ROOT::Experimental::RNTupleLocator ROOT::Experimental::Detail::RPageSinkBuf::CommitClusterGroupImpl	(	unsigned char *	serializedPageList,
		std::uint32_t	length
	)

finalprotectedvirtual

Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

Typically, the implementation takes care of compressing and writing the provided buffer.

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 174 of file RPageSinkBuf.cxx.

◆ CommitClusterImpl()

std::uint64_t ROOT::Experimental::Detail::RPageSinkBuf::CommitClusterImpl ( NTupleSize_t nEntries )

finalprotectedvirtual

Returns the number of bytes written to storage (excluding metadata)

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 126 of file RPageSinkBuf.cxx.

◆ CommitDatasetImpl()

void ROOT::Experimental::Detail::RPageSinkBuf::CommitDatasetImpl	(	unsigned char *	serializedFooter,
		std::uint32_t	length
	)

finalprotectedvirtual

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 182 of file RPageSinkBuf.cxx.

◆ CommitPageImpl()

ROOT::Experimental::RNTupleLocator ROOT::Experimental::Detail::RPageSinkBuf::CommitPageImpl	(	ColumnHandle_t	columnHandle,
		const RPage &	page
	)

finalprotectedvirtual

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 83 of file RPageSinkBuf.cxx.

◆ CommitSealedPageImpl()

ROOT::Experimental::RNTupleLocator ROOT::Experimental::Detail::RPageSinkBuf::CommitSealedPageImpl	(	DescriptorId_t	physicalColumnId,
		const RSealedPage &	sealedPage
	)

finalprotectedvirtual

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 116 of file RPageSinkBuf.cxx.

◆ CreateImpl()

void ROOT::Experimental::Detail::RPageSinkBuf::CreateImpl	(	const RNTupleModel &	model,
		unsigned char *	serializedHeader,
		std::uint32_t	length
	)

finalprotectedvirtual

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 36 of file RPageSinkBuf.cxx.

◆ GetMetrics()

RNTupleMetrics & ROOT::Experimental::Detail::RPageSinkBuf::GetMetrics ( )

inlinefinalvirtual

Returns the default metrics object. Subclasses might alternatively provide their own metrics object by overriding this.

Reimplemented from ROOT::Experimental::Detail::RPageSink.

Definition at line 149 of file RPageSinkBuf.hxx.

◆ operator=() [1/2]

RPageSinkBuf & ROOT::Experimental::Detail::RPageSinkBuf::operator= ( const RPageSinkBuf & )

delete

◆ operator=() [2/2]

RPageSinkBuf & ROOT::Experimental::Detail::RPageSinkBuf::operator= ( RPageSinkBuf && )

default

◆ ReleasePage()

void ROOT::Experimental::Detail::RPageSinkBuf::ReleasePage ( RPage & page )

finalvirtual

Every page store needs to be able to free pages it handed out.

But Sinks and sources have different means of allocating pages.

Implements ROOT::Experimental::Detail::RPageStorage.

Definition at line 194 of file RPageSinkBuf.cxx.

◆ ReservePage()

ROOT::Experimental::Detail::RPage ROOT::Experimental::Detail::RPageSinkBuf::ReservePage	(	ColumnHandle_t	columnHandle,
		std::size_t	nElements
	)

finalvirtual

Get a new, empty page for the given column that can be filled with up to nElements.

If nElements is zero, the page sink picks an appropriate size.

Implements ROOT::Experimental::Detail::RPageSink.

Definition at line 189 of file RPageSinkBuf.cxx.

◆ UpdateSchema()

void ROOT::Experimental::Detail::RPageSinkBuf::UpdateSchema ( const RNTupleModelChangeset & changeset )

finalvirtual

Incorporate incremental changes to the model into the ntuple descriptor.

This happens, e.g. if new fields were added after the initial call to RPageSink::Create(RNTupleModel &)

Reimplemented from ROOT::Experimental::Detail::RPageSink.

Definition at line 44 of file RPageSinkBuf.cxx.

Member Data Documentation

◆ fBufferedColumns

std::vector<RColumnBuf> ROOT::Experimental::Detail::RPageSinkBuf::fBufferedColumns

private

Vector of buffered column pages. Indexed by column id.

Definition at line 127 of file RPageSinkBuf.hxx.

◆ fCounters

std::unique_ptr<RCounters> ROOT::Experimental::Detail::RPageSinkBuf::fCounters

private

Definition at line 119 of file RPageSinkBuf.hxx.

◆ fInnerModel

std::unique_ptr<RNTupleModel> ROOT::Experimental::Detail::RPageSinkBuf::fInnerModel

private

The buffered page sink maintains a copy of the RNTupleModel for the inner sink.

For the unbuffered case, the RNTupleModel is instead managed by a RNTupleWriter.

Definition at line 125 of file RPageSinkBuf.hxx.

◆ fInnerSink

std::unique_ptr<RPageSink> ROOT::Experimental::Detail::RPageSinkBuf::fInnerSink

private

The inner sink, responsible for actually performing I/O.

Definition at line 122 of file RPageSinkBuf.hxx.

◆ fMetrics

RNTupleMetrics ROOT::Experimental::Detail::RPageSinkBuf::fMetrics

private

Definition at line 120 of file RPageSinkBuf.hxx.

Libraries for ROOT::Experimental::Detail::RPageSinkBuf:

[legend]

The documentation for this class was generated from the following files:

tree/ntuple/v7/inc/ROOT/RPageSinkBuf.hxx
tree/ntuple/v7/src/RPageSinkBuf.cxx

Classes

Public Member Functions

Protected Member Functions

Private Attributes

Additional Inherited Members

Constructor & Destructor Documentation

◆ RPageSinkBuf() [1/3]

◆ RPageSinkBuf() [2/3]

◆ RPageSinkBuf() [3/3]

◆ ~RPageSinkBuf()

Member Function Documentation

◆ CommitClusterGroupImpl()

◆ CommitClusterImpl()

◆ CommitDatasetImpl()

◆ CommitPageImpl()

◆ CommitSealedPageImpl()

◆ CreateImpl()

◆ GetMetrics()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ ReleasePage()

◆ ReservePage()

◆ UpdateSchema()

Member Data Documentation

◆ fBufferedColumns

◆ fCounters

◆ fInnerModel

◆ fInnerSink

◆ fMetrics