Storage provider that writes ntuple pages into S3-compatible object storage.

Currently implements Mode B (one sealed page per S3 object, kTypeObject64 locators). Mode A (multiple packed pages per object, kTypeMulti locators) will be added separately.

Warning: The S3 backend is experimental and under active development.

Definition at line 94 of file RPageStorageS3.hxx.

Classes
struct	RFromBaseUrl
	Tag to select the internal constructor that takes an already-resolved base URL. More...

Public Types
using	Callback_t = std::function<void(RPageSink &)>

using	ColumnHandle_t = RColumnHandle
	The column handle identifies a column with the current open page storage.

using	SealedPageSequence_t = std::deque<RSealedPage>

Public Member Functions
	RPageSinkS3 (std::string_view ntupleName, std::string_view uri, const ROOT::RNTupleWriteOptions &options)

	~RPageSinkS3 () override

ColumnHandle_t	AddColumn (ROOT::DescriptorId_t fieldId, ROOT::Internal::RColumn &column) final
	Register a new column.

std::unique_ptr< ROOT::Internal::RPageSink >	CloneAsHidden (std::string_view name, const ROOT::RNTupleWriteOptions &opts) const final
	Creates a new sink with the same underlying storage as this but writing to a different RNTuple named `name`.

void	CommitAttributeSet (std::string_view attrSetName, const RNTupleLink &attrAnchorInfo) final
	Adds the given anchor information (name + locator) into the main RNTuple's descriptor as an attribute set linked to it with the given name.

virtual std::uint64_t	CommitCluster (ROOT::NTupleSize_t nNewEntries)
	Finalize the current cluster and create a new one for the following data.

void	CommitClusterGroup () final
	Write out the page locations (page list envelope) for all the committed clusters since the last call of CommitClusterGroup (or the beginning of writing).

RNTupleLink	CommitDataset ()
	Run the registered callbacks and finalize the current cluster and the entrire data set.

void	CommitPage (ColumnHandle_t columnHandle, const ROOT::Internal::RPage &page) final
	Write a page to the storage. The column must have been added before.

void	CommitSealedPage (ROOT::DescriptorId_t physicalColumnId, const RPageStorage::RSealedPage &sealedPage) final
	Write a preprocessed page to storage. The column must have been added before.

void	CommitSealedPageV (std::span< RPageStorage::RSealedPageGroup > ranges) final
	Write a vector of preprocessed pages to storage. The corresponding columns must have been added before.

void	CommitStagedClusters (std::span< RStagedCluster > clusters) final
	Commit staged clusters, logically appending them to the ntuple descriptor.

void	CommitSuppressedColumn (ColumnHandle_t columnHandle) final
	Commits a suppressed column for the current cluster.

void	DropColumn (ColumnHandle_t) final
	Unregisters a column.

ROOT::DescriptorId_t	GetColumnId (ColumnHandle_t columnHandle) const

const ROOT::RNTupleDescriptor &	GetDescriptor () const final
	Return the RNTupleDescriptor being constructed.

virtual ROOT::Experimental::Detail::RNTupleMetrics &	GetMetrics ()
	Returns the default metrics object.

ROOT::NTupleSize_t	GetNEntries () const final

const std::string &	GetNTupleName () const
	Returns the NTuple name.

virtual RSinkGuard	GetSinkGuard ()

EPageStorageType	GetType () final
	Whether the concrete implementation is a sink or a source.

const ROOT::RNTupleWriteOptions &	GetWriteOptions () const
	Returns the sink's write options.

void	Init (RNTupleModel &model)
	Physically creates the storage container to hold the ntuple (e.g., a keys a TFile or an S3 bucket) Init() associates column handles to the columns referenced by the model.

std::unique_ptr< RNTupleModel >	InitFromDescriptor (const ROOT::RNTupleDescriptor &descriptor, bool copyClusters)
	Initialize sink based on an existing descriptor and fill into the descriptor builder, optionally copying over the descriptor's clusters to this sink's descriptor.

bool	IsInitialized () const

void	RegisterOnCommitDatasetCallback (const Callback_t &cb)
	The registered callback is executed at the beginning of CommitDataset();.

virtual ROOT::Internal::RPage	ReservePage (ColumnHandle_t columnHandle, std::size_t nElements)
	Get a new, empty page for the given column that can be filled with up to nElements; nElements must be larger than zero.

void	SetTaskScheduler (RTaskScheduler *taskScheduler)

RStagedCluster	StageCluster (ROOT::NTupleSize_t nNewEntries) final
	Stage the current cluster and create a new one for the following data.

void	UpdateExtraTypeInfo (const ROOT::RExtraTypeInfoDescriptor &extraTypeInfo) final
	Adds an extra type information record to schema.

void	UpdateSchema (const ROOT::Internal::RNTupleModelChangeset &changeset, ROOT::NTupleSize_t firstEntry) override
	Incorporate incremental changes to the model into the ntuple descriptor.

Static Public Member Functions
static std::unique_ptr< RPageSink >	Create (std::string_view ntupleName, std::string_view location, const ROOT::RNTupleWriteOptions &options=ROOT::RNTupleWriteOptions())
	Guess the concrete derived page source from the location.

static RSealedPage	SealPage (const RSealPageConfig &config)
	Seal a page using the provided info.

Static Public Attributes
static constexpr std::size_t	kNBytesPageChecksum = sizeof(std::uint64_t)
	The page checksum is a 64bit xxhash3.

Protected Member Functions
RNTupleLocator	CommitClusterGroupImpl (unsigned char *serializedPageList, std::uint32_t length) final
	Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

RNTupleLink	CommitDatasetImpl () final

ROOT::Internal::RNTupleLink	CommitDatasetImpl (unsigned char *serializedFooter, std::uint32_t length) final

virtual RNTupleLink	CommitDatasetImpl (unsigned char *serializedFooter, std::uint32_t length)=0

RNTupleLocator	CommitSealedPageImpl (ROOT::DescriptorId_t physicalColumnId, const RPageStorage::RSealedPage &sealedPage) final

virtual std::vector< RNTupleLocator >	CommitSealedPageVImpl (std::span< RPageStorage::RSealedPageGroup > ranges, const std::vector< bool > &mask)
	Vector commit of preprocessed pages.

void	EnableDefaultMetrics (const std::string &prefix)
	Enables the default set of metrics provided by RPageSink.

void	InitImpl (RNTupleModel &model) final
	Updates the descriptor and calls InitImpl() that handles the backend-specific details (file, DAOS, etc.)

void	InitImpl (unsigned char *serializedHeader, std::uint32_t length) final

virtual void	InitImpl (unsigned char *serializedHeader, std::uint32_t length)=0

RSealedPage	SealPage (const ROOT::Internal::RPage &page, const ROOT::Internal::RColumnElementBase &element)
	Helper for streaming a page.

std::uint64_t	StageClusterImpl () final
	Returns the number of bytes written to storage (excluding metadata)

void	WaitForAllTasks ()

Protected Attributes
std::unique_ptr< RCounters >	fCounters

ROOT::Internal::RNTupleDescriptorBuilder	fDescriptorBuilder

RFeatures	fFeatures

bool	fIsInitialized = false
	Flag if sink was initialized.

ROOT::Experimental::Detail::RNTupleMetrics	fMetrics

std::string	fNTupleName

std::unique_ptr< ROOT::RNTupleWriteOptions >	fOptions

std::unique_ptr< ROOT::Internal::RPageAllocator >	fPageAllocator
	For the time being, we will use the heap allocator for all sources and sinks. This may change in the future.

RTaskScheduler *	fTaskScheduler = nullptr

Private Member Functions
	RPageSinkS3 (std::string_view ntupleName, std::string_view baseUrl, const ROOT::RNTupleWriteOptions &options, RFromBaseUrl)
	Internal constructor used by CloneAsHidden: the public constructor derives the base URL by parsing an s3 scheme URI, whereas a clone already has a resolved base URL to write under.

std::string	MakeObjectUrl (std::uint64_t objId) const
	Resolve a numeric object ID to its full HTTP URL.

void	PutObject (const std::string &url, const unsigned char *data, std::size_t size)
	Upload raw bytes to the given S3 URL via an HTTP PUT request.

Private Attributes
RNTupleAnchorS3	fAnchor
	Anchor metadata populated during the write path and uploaded last in CommitDatasetImpl.

std::string	fBaseUrl
	HTTP base URL for this ntuple (derived from the s3 scheme URI); never has a trailing slash.

ROOT::Internal::RCurlConnection	fConnection
	One HTTP connection reused for every upload, so curl keeps it alive across objects on the same host instead of re-handshaking per object.

ROOT::Internal::RNTupleSerializer::StreamerInfoMap_t	fInfosOfStreamerFields
	Union of the streamer info records that are sent from streamer fields to the sink before committing the dataset.

std::uint64_t	fNBytesCurrentCluster {0}
	Tracks the number of bytes committed to the current cluster (reset in StageClusterImpl)

std::uint64_t	fNextClusterInGroup = 0
	Remembers the starting cluster id for the next cluster group.

std::uint64_t	fObjectId {0}
	Object ID counter; incremented for each object written.

std::vector< Callback_t >	fOnDatasetCommitCallbacks

std::vector< ROOT::RClusterDescriptor::RColumnRange >	fOpenColumnRanges
	Keeps track of the number of elements in the currently open cluster. Indexed by column id.

std::vector< ROOT::RClusterDescriptor::RPageRange >	fOpenPageRanges
	Keeps track of the written pages in the currently open cluster. Indexed by column id.

ROOT::NTupleSize_t	fPrevClusterNEntries = 0
	Used to calculate the number of entries in the current cluster.

std::vector< unsigned char >	fSealPageBuffer
	Used as destination buffer in the simple SealPage overload.

ROOT::Internal::RNTupleSerializer::RContext	fSerializationContext
	Used to map the IDs of the descriptor to the physical IDs issued during header/footer serialization.

RWritePageMemoryManager	fWritePageMemoryManager
	Used in ReservePage to maintain the page buffer budget.

#include <ROOT/RPageStorageS3.hxx>

Inheritance diagram for ROOT::Experimental::Internal::RPageSinkS3:

[legend]

Member Typedef Documentation

◆ Callback_t

using ROOT::Internal::RPageSink::Callback_t = std::function<void(RPageSink &)>

inherited

Definition at line 258 of file RPageStorage.hxx.

◆ ColumnHandle_t

using ROOT::Internal::RPageStorage::ColumnHandle_t = RColumnHandle

inherited

The column handle identifies a column with the current open page storage.

Definition at line 180 of file RPageStorage.hxx.

◆ SealedPageSequence_t

using ROOT::Internal::RPageStorage::SealedPageSequence_t = std::deque<RSealedPage>

inherited

Definition at line 130 of file RPageStorage.hxx.

Constructor & Destructor Documentation

◆ RPageSinkS3() [1/2]

ROOT::Experimental::Internal::RPageSinkS3::RPageSinkS3	(	std::string_view	ntupleName,
		std::string_view	baseUrl,
		const ROOT::RNTupleWriteOptions &	options,
		RFromBaseUrl	)

private

Internal constructor used by CloneAsHidden: the public constructor derives the base URL by parsing an s3 scheme URI, whereas a clone already has a resolved base URL to write under.

Definition at line 171 of file RPageStorageS3.cxx.

◆ RPageSinkS3() [2/2]

ROOT::Experimental::Internal::RPageSinkS3::RPageSinkS3	(	std::string_view	ntupleName,
		std::string_view	uri,
		const ROOT::RNTupleWriteOptions &	options )

Definition at line 165 of file RPageStorageS3.cxx.

◆ ~RPageSinkS3()

ROOT::Experimental::Internal::RPageSinkS3::~RPageSinkS3 ( )

overridedefault

Member Function Documentation

◆ AddColumn()

ROOT::Internal::RPageStorage::ColumnHandle_t ROOT::Internal::RPagePersistentSink::AddColumn	(	ROOT::DescriptorId_t	fieldId,
		ROOT::Internal::RColumn &	column )

finalvirtualinherited

Register a new column.

When reading, the column must exist in the ntuple on disk corresponding to the metadata. When writing, every column can only be attached once.

Implements ROOT::Internal::RPageStorage.

Definition at line 952 of file RPageStorage.cxx.

◆ CloneAsHidden()

std::unique_ptr< ROOT::Internal::RPageSink > ROOT::Experimental::Internal::RPageSinkS3::CloneAsHidden	(	std::string_view	name,
		const ROOT::RNTupleWriteOptions &	opts ) const

finalvirtual

Creates a new sink with the same underlying storage as this but writing to a different RNTuple named name.

Only one of the two sinks can safely write at the same time. The RNTuple written by this cloned sink will be stored in a hidden key (this is a convenient assumption we make now since this method is only used to create attribute RNTuples).

Implements ROOT::Internal::RPageSink.

Definition at line 300 of file RPageStorageS3.cxx.

◆ CommitAttributeSet()

void ROOT::Internal::RPagePersistentSink::CommitAttributeSet	(	std::string_view	attrSetName,
		const RNTupleLink &	attrAnchorInfo )

finalvirtualinherited

Adds the given anchor information (name + locator) into the main RNTuple's descriptor as an attribute set linked to it with the given name.

The attribute set must have already been written to storage via RNTupleAttrSetWriter::Commit(). Note that, by RNTuple specs, this is only legal to call on a non-attribute RNTuple's sink.

Implements ROOT::Internal::RPageSink.

Definition at line 1410 of file RPageStorage.cxx.

◆ CommitCluster()

virtual std::uint64_t ROOT::Internal::RPageSink::CommitCluster ( ROOT::NTupleSize_t nNewEntries )

inlinevirtualinherited

Finalize the current cluster and create a new one for the following data.

Returns the number of bytes written to storage (excluding metadata).

Reimplemented in ROOT::Internal::RPageSinkBuf.

Definition at line 379 of file RPageStorage.hxx.

◆ CommitClusterGroup()

void ROOT::Internal::RPagePersistentSink::CommitClusterGroup ( )

finalvirtualinherited

Write out the page locations (page list envelope) for all the committed clusters since the last call of CommitClusterGroup (or the beginning of writing).

Implements ROOT::Internal::RPageSink.

Definition at line 1368 of file RPageStorage.cxx.

◆ CommitClusterGroupImpl()

ROOT::RNTupleLocator ROOT::Experimental::Internal::RPageSinkS3::CommitClusterGroupImpl	(	unsigned char *	serializedPageList,
		std::uint32_t	length )

finalprotectedvirtual

Returns the locator of the page list envelope of the given buffer that contains the serialized page list.

Typically, the implementation takes care of compressing and writing the provided buffer.

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 250 of file RPageStorageS3.cxx.

◆ CommitDataset()

ROOT::Internal::RNTupleLink ROOT::Internal::RPageSink::CommitDataset ( )

inherited

Run the registered callbacks and finalize the current cluster and the entrire data set.

Definition at line 893 of file RPageStorage.cxx.

◆ CommitDatasetImpl() [1/3]

ROOT::Internal::RNTupleLink ROOT::Internal::RPagePersistentSink::CommitDatasetImpl ( )

finalprotectedvirtual

Returns: The locator and length of the written anchor.

Reimplemented from ROOT::Internal::RPagePersistentSink.

Definition at line 510 of file RPageStorage.cxx.

◆ CommitDatasetImpl() [2/3]

ROOT::Internal::RNTupleLink ROOT::Experimental::Internal::RPageSinkS3::CommitDatasetImpl	(	unsigned char *	serializedFooter,
		std::uint32_t	length )

finalprotectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 272 of file RPageStorageS3.cxx.

◆ CommitDatasetImpl() [3/3]

virtual RNTupleLink ROOT::Internal::RPagePersistentSink::CommitDatasetImpl	(	unsigned char *	serializedFooter,
		std::uint32_t	length )

protectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

◆ CommitPage()

void ROOT::Internal::RPagePersistentSink::CommitPage	(	ColumnHandle_t	columnHandle,
		const ROOT::Internal::RPage &	page )

finalvirtualinherited

Write a page to the storage. The column must have been added before.

Implements ROOT::Internal::RPageSink.

Definition at line 1184 of file RPageStorage.cxx.

◆ CommitSealedPage()

void ROOT::Internal::RPagePersistentSink::CommitSealedPage	(	ROOT::DescriptorId_t	physicalColumnId,
		const RPageStorage::RSealedPage &	sealedPage )

finalvirtualinherited

Write a preprocessed page to storage. The column must have been added before.

Implements ROOT::Internal::RPageSink.

Definition at line 1203 of file RPageStorage.cxx.

◆ CommitSealedPageImpl()

ROOT::RNTupleLocator ROOT::Experimental::Internal::RPageSinkS3::CommitSealedPageImpl	(	ROOT::DescriptorId_t	physicalColumnId,
		const RPageStorage::RSealedPage &	sealedPage )

finalprotectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 223 of file RPageStorageS3.cxx.

◆ CommitSealedPageV()

void ROOT::Internal::RPagePersistentSink::CommitSealedPageV ( std::span< RPageStorage::RSealedPageGroup > ranges )

finalvirtualinherited

Write a vector of preprocessed pages to storage. The corresponding columns must have been added before.

Implements ROOT::Internal::RPageSink.

Definition at line 1232 of file RPageStorage.cxx.

◆ CommitSealedPageVImpl()

std::vector< ROOT::RNTupleLocator > ROOT::Internal::RPagePersistentSink::CommitSealedPageVImpl	(	std::span< RPageStorage::RSealedPageGroup >	ranges,
		const std::vector< bool > &	mask )

protectedvirtualinherited

Vector commit of preprocessed pages.

The ranges array specifies a range of sealed pages to be committed for each column. The returned vector contains, in order, the RNTupleLocator for each page on each range in ranges, i.e. the first N entries refer to the N pages in ranges[0], followed by M entries that refer to the M pages in ranges[1], etc. The mask allows to skip writing out certain pages. The vector has the size of all the pages. For every false value in the mask, the corresponding locator is skipped (missing) in the output vector. The default is to call CommitSealedPageImpl for each page; derived classes may provide an optimized implementation though.

Reimplemented in ROOT::Experimental::Internal::RPageSinkDaos, and ROOT::Internal::RPageSinkFile.

Definition at line 1216 of file RPageStorage.cxx.

◆ CommitStagedClusters()

void ROOT::Internal::RPagePersistentSink::CommitStagedClusters ( std::span< RStagedCluster > clusters )

finalvirtualinherited

Commit staged clusters, logically appending them to the ntuple descriptor.

Implements ROOT::Internal::RPageSink.

Definition at line 1331 of file RPageStorage.cxx.

◆ CommitSuppressedColumn()

void ROOT::Internal::RPagePersistentSink::CommitSuppressedColumn ( ColumnHandle_t columnHandle )

finalvirtualinherited

Commits a suppressed column for the current cluster.

Can be called anytime before CommitCluster(). For any given column and cluster, there must be no calls to both CommitSuppressedColumn() and page commits.

Implements ROOT::Internal::RPageSink.

Definition at line 1179 of file RPageStorage.cxx.

◆ Create()

std::unique_ptr< ROOT::Internal::RPageSink > ROOT::Internal::RPagePersistentSink::Create	(	std::string_view	ntupleName,
		std::string_view	location,
		const ROOT::RNTupleWriteOptions &	options = ROOT::RNTupleWriteOptions() )

staticinherited

Guess the concrete derived page source from the location.

Definition at line 913 of file RPageStorage.cxx.

◆ DropColumn()

void ROOT::Internal::RPageSink::DropColumn ( ColumnHandle_t columnHandle )

inlinefinalvirtualinherited

Unregisters a column.

A page source decreases the reference counter for the corresponding active column. For a page sink, dropping columns is currently a no-op.

Implements ROOT::Internal::RPageStorage.

Definition at line 307 of file RPageStorage.hxx.

◆ EnableDefaultMetrics()

void ROOT::Internal::RPagePersistentSink::EnableDefaultMetrics ( const std::string & prefix )

protectedinherited

Enables the default set of metrics provided by RPageSink.

prefix will be used as the prefix for the counters registered in the internal RNTupleMetrics object. This set of counters can be extended by a subclass by calling fMetrics.MakeCounter<...>().

A subclass using the default set of metrics is always responsible for updating the counters appropriately, e.g. fCounters->fNPageCommited.Inc()

Definition at line 1456 of file RPageStorage.cxx.

◆ GetColumnId()

ROOT::DescriptorId_t ROOT::Internal::RPageStorage::GetColumnId ( ColumnHandle_t columnHandle ) const

inlineinherited

Definition at line 188 of file RPageStorage.hxx.

◆ GetDescriptor()

const ROOT::RNTupleDescriptor & ROOT::Internal::RPagePersistentSink::GetDescriptor ( ) const

inlinefinalvirtualinherited

Return the RNTupleDescriptor being constructed.

Implements ROOT::Internal::RPageSink.

Definition at line 535 of file RPageStorage.hxx.

◆ GetMetrics()

virtual ROOT::Experimental::Detail::RNTupleMetrics & ROOT::Internal::RPageStorage::GetMetrics ( )

inlinevirtualinherited

Returns the default metrics object.

Subclasses might alternatively provide their own metrics object by overriding this.

Definition at line 192 of file RPageStorage.hxx.

◆ GetNEntries()

ROOT::NTupleSize_t ROOT::Internal::RPagePersistentSink::GetNEntries ( ) const

inlinefinalvirtualinherited

Implements ROOT::Internal::RPageSink.

Definition at line 537 of file RPageStorage.hxx.

◆ GetNTupleName()

const std::string & ROOT::Internal::RPageStorage::GetNTupleName ( ) const

inlineinherited

Returns the NTuple name.

Definition at line 195 of file RPageStorage.hxx.

◆ GetSinkGuard()

virtual RSinkGuard ROOT::Internal::RPageSink::GetSinkGuard ( )

inlinevirtualinherited

Definition at line 433 of file RPageStorage.hxx.

◆ GetType()

EPageStorageType ROOT::Internal::RPageSink::GetType ( )

inlinefinalvirtualinherited

Whether the concrete implementation is a sink or a source.

Implements ROOT::Internal::RPageStorage.

Definition at line 303 of file RPageStorage.hxx.

◆ GetWriteOptions()

const ROOT::RNTupleWriteOptions & ROOT::Internal::RPageSink::GetWriteOptions ( ) const

inlineinherited

Returns the sink's write options.

Definition at line 305 of file RPageStorage.hxx.

◆ Init()

void ROOT::Internal::RPageSink::Init ( RNTupleModel & model )

inlineinherited

Physically creates the storage container to hold the ntuple (e.g., a keys a TFile or an S3 bucket) Init() associates column handles to the columns referenced by the model.

Definition at line 318 of file RPageStorage.hxx.

◆ InitFromDescriptor()

std::unique_ptr< ROOT::RNTupleModel > ROOT::Internal::RPagePersistentSink::InitFromDescriptor	(	const ROOT::RNTupleDescriptor &	descriptor,
		bool	copyClusters )

inherited

Initialize sink based on an existing descriptor and fill into the descriptor builder, optionally copying over the descriptor's clusters to this sink's descriptor.

Returns: The model created from the new sink's descriptor. This model should be kept alive for at least as long as the sink.

Definition at line 1108 of file RPageStorage.cxx.

◆ InitImpl() [1/3]

void ROOT::Internal::RPagePersistentSink::InitImpl ( ROOT::RNTupleModel & model )

finalprotectedvirtual

Updates the descriptor and calls InitImpl() that handles the backend-specific details (file, DAOS, etc.)

Reimplemented from ROOT::Internal::RPagePersistentSink.

Definition at line 488 of file RPageStorage.cxx.

◆ InitImpl() [2/3]

void ROOT::Experimental::Internal::RPageSinkS3::InitImpl	(	unsigned char *	serializedHeader,
		std::uint32_t	length )

finalprotectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 202 of file RPageStorageS3.cxx.

◆ InitImpl() [3/3]

virtual void ROOT::Internal::RPagePersistentSink::InitImpl	(	unsigned char *	serializedHeader,
		std::uint32_t	length )

protectedvirtual

Implements ROOT::Internal::RPagePersistentSink.

◆ IsInitialized()

bool ROOT::Internal::RPageSink::IsInitialized ( ) const

inlineinherited

Definition at line 309 of file RPageStorage.hxx.

◆ MakeObjectUrl()

std::string ROOT::Experimental::Internal::RPageSinkS3::MakeObjectUrl ( std::uint64_t objId ) const

private

Resolve a numeric object ID to its full HTTP URL.

Definition at line 186 of file RPageStorageS3.cxx.

◆ PutObject()

void ROOT::Experimental::Internal::RPageSinkS3::PutObject	(	const std::string &	url,
		const unsigned char *	data,
		std::size_t	size )

private

Upload raw bytes to the given S3 URL via an HTTP PUT request.

Definition at line 191 of file RPageStorageS3.cxx.

◆ RegisterOnCommitDatasetCallback()

void ROOT::Internal::RPageSink::RegisterOnCommitDatasetCallback ( const Callback_t & cb )

inlineinherited

The registered callback is executed at the beginning of CommitDataset();.

Definition at line 402 of file RPageStorage.hxx.

◆ ReservePage()

ROOT::Internal::RPage ROOT::Internal::RPageSink::ReservePage	(	ColumnHandle_t	columnHandle,
		std::size_t	nElements )

virtualinherited

Get a new, empty page for the given column that can be filled with up to nElements; nElements must be larger than zero.

Reimplemented in ROOT::Internal::RPageSinkBuf.

Definition at line 900 of file RPageStorage.cxx.

◆ SealPage() [1/2]

ROOT::Internal::RPageStorage::RSealedPage ROOT::Internal::RPageSink::SealPage	(	const ROOT::Internal::RPage &	page,
		const ROOT::Internal::RColumnElementBase &	element )

protectedinherited

Helper for streaming a page.

This is commonly used in derived, concrete page sinks. Note that if compressionSetting is 0 (uncompressed) and the page is mappable and not checksummed, the returned sealed page will point directly to the input page buffer. Otherwise, the sealed page references fSealPageBuffer. Thus, the buffer pointed to by the RSealedPage should never be freed.

Definition at line 876 of file RPageStorage.cxx.

◆ SealPage() [2/2]

ROOT::Internal::RPageStorage::RSealedPage ROOT::Internal::RPageSink::SealPage ( const RSealPageConfig & config )

staticinherited

Seal a page using the provided info.

Definition at line 838 of file RPageStorage.cxx.

◆ SetTaskScheduler()

void ROOT::Internal::RPageStorage::SetTaskScheduler ( RTaskScheduler * taskScheduler )

inlineinherited

Definition at line 197 of file RPageStorage.hxx.

◆ StageCluster()

ROOT::Internal::RPageSink::RStagedCluster ROOT::Internal::RPagePersistentSink::StageCluster ( ROOT::NTupleSize_t nNewEntries )

finalvirtualinherited

Stage the current cluster and create a new one for the following data.

Returns the object that must be passed to CommitStagedClusters to logically append the staged cluster to the ntuple descriptor.

Implements ROOT::Internal::RPageSink.

Definition at line 1302 of file RPageStorage.cxx.

◆ StageClusterImpl()

std::uint64_t ROOT::Experimental::Internal::RPageSinkS3::StageClusterImpl ( )

finalprotectedvirtual

Returns the number of bytes written to storage (excluding metadata)

Implements ROOT::Internal::RPagePersistentSink.

Definition at line 244 of file RPageStorageS3.cxx.

◆ UpdateExtraTypeInfo()

void ROOT::Internal::RPagePersistentSink::UpdateExtraTypeInfo ( const ROOT::RExtraTypeInfoDescriptor & extraTypeInfo )

finalvirtualinherited

Adds an extra type information record to schema.

The extra type information will be written to the extension header. The information in the record will be merged with the existing information, e.g. duplicate streamer info records will be removed. This method is called by the "on commit dataset" callback registered by specific fields (e.g., streamer field) and during merging.

Implements ROOT::Internal::RPageSink.

Definition at line 1071 of file RPageStorage.cxx.

◆ UpdateSchema()

void ROOT::Internal::RPagePersistentSink::UpdateSchema	(	const ROOT::Internal::RNTupleModelChangeset &	changeset,
		ROOT::NTupleSize_t	firstEntry )

overridevirtualinherited

Incorporate incremental changes to the model into the ntuple descriptor.

This happens, e.g. if new fields were added after the initial call to RPageSink::Init(RNTupleModel &). firstEntry specifies the global index for the first stored element in the added columns.

Implements ROOT::Internal::RPageSink.

Reimplemented in ROOT::Internal::RPageSinkFile.

Definition at line 973 of file RPageStorage.cxx.

◆ WaitForAllTasks()

void ROOT::Internal::RPageStorage::WaitForAllTasks ( )

inlineprotectedinherited

Definition at line 153 of file RPageStorage.hxx.

Member Data Documentation

◆ fAnchor

RNTupleAnchorS3 ROOT::Experimental::Internal::RPageSinkS3::fAnchor

private

Anchor metadata populated during the write path and uploaded last in CommitDatasetImpl.

Definition at line 106 of file RPageStorageS3.hxx.

◆ fBaseUrl

std::string ROOT::Experimental::Internal::RPageSinkS3::fBaseUrl

private

HTTP base URL for this ntuple (derived from the s3 scheme URI); never has a trailing slash.

Definition at line 97 of file RPageStorageS3.hxx.

◆ fConnection

ROOT::Internal::RCurlConnection ROOT::Experimental::Internal::RPageSinkS3::fConnection

private

One HTTP connection reused for every upload, so curl keeps it alive across objects on the same host instead of re-handshaking per object.

Definition at line 100 of file RPageStorageS3.hxx.

◆ fCounters

std::unique_ptr<RCounters> ROOT::Internal::RPagePersistentSink::fCounters

protectedinherited

Definition at line 483 of file RPageStorage.hxx.

◆ fDescriptorBuilder

ROOT::Internal::RNTupleDescriptorBuilder ROOT::Internal::RPagePersistentSink::fDescriptorBuilder

protectedinherited

Definition at line 471 of file RPageStorage.hxx.

◆ fFeatures

RFeatures ROOT::Internal::RPagePersistentSink::fFeatures

protectedinherited

Definition at line 470 of file RPageStorage.hxx.

◆ fInfosOfStreamerFields

ROOT::Internal::RNTupleSerializer::StreamerInfoMap_t ROOT::Internal::RPagePersistentSink::fInfosOfStreamerFields

privateinherited

Union of the streamer info records that are sent from streamer fields to the sink before committing the dataset.

Definition at line 462 of file RPageStorage.hxx.

◆ fIsInitialized

bool ROOT::Internal::RPageSink::fIsInitialized = false

protectedinherited

Flag if sink was initialized.

Definition at line 279 of file RPageStorage.hxx.

◆ fMetrics

ROOT::Experimental::Detail::RNTupleMetrics ROOT::Internal::RPageStorage::fMetrics

protectedinherited

Definition at line 146 of file RPageStorage.hxx.

◆ fNBytesCurrentCluster

std::uint64_t ROOT::Experimental::Internal::RPageSinkS3::fNBytesCurrentCluster {0}

private

Tracks the number of bytes committed to the current cluster (reset in StageClusterImpl)

Definition at line 104 of file RPageStorageS3.hxx.

◆ fNextClusterInGroup

std::uint64_t ROOT::Internal::RPagePersistentSink::fNextClusterInGroup = 0

privateinherited

Remembers the starting cluster id for the next cluster group.

Definition at line 453 of file RPageStorage.hxx.

◆ fNTupleName

std::string ROOT::Internal::RPageStorage::fNTupleName

protectedinherited

Definition at line 151 of file RPageStorage.hxx.

◆ fObjectId

std::uint64_t ROOT::Experimental::Internal::RPageSinkS3::fObjectId {0}

private

Object ID counter; incremented for each object written.

Definition at line 102 of file RPageStorageS3.hxx.

◆ fOnDatasetCommitCallbacks

std::vector<Callback_t> ROOT::Internal::RPageSink::fOnDatasetCommitCallbacks

privateinherited

Definition at line 288 of file RPageStorage.hxx.

◆ fOpenColumnRanges

std::vector<ROOT::RClusterDescriptor::RColumnRange> ROOT::Internal::RPagePersistentSink::fOpenColumnRanges

privateinherited

Keeps track of the number of elements in the currently open cluster. Indexed by column id.

Definition at line 457 of file RPageStorage.hxx.

◆ fOpenPageRanges

std::vector<ROOT::RClusterDescriptor::RPageRange> ROOT::Internal::RPagePersistentSink::fOpenPageRanges

privateinherited

Keeps track of the written pages in the currently open cluster. Indexed by column id.

Definition at line 459 of file RPageStorage.hxx.

◆ fOptions

std::unique_ptr<ROOT::RNTupleWriteOptions> ROOT::Internal::RPageSink::fOptions

protectedinherited

Definition at line 276 of file RPageStorage.hxx.

◆ fPageAllocator

std::unique_ptr<ROOT::Internal::RPageAllocator> ROOT::Internal::RPageStorage::fPageAllocator

protectedinherited

For the time being, we will use the heap allocator for all sources and sinks. This may change in the future.

Definition at line 149 of file RPageStorage.hxx.

◆ fPrevClusterNEntries

ROOT::NTupleSize_t ROOT::Internal::RPagePersistentSink::fPrevClusterNEntries = 0

privateinherited

Used to calculate the number of entries in the current cluster.

Definition at line 455 of file RPageStorage.hxx.

◆ fSealPageBuffer

std::vector<unsigned char> ROOT::Internal::RPageSink::fSealPageBuffer

privateinherited

Used as destination buffer in the simple SealPage overload.

Definition at line 289 of file RPageStorage.hxx.

◆ fSerializationContext

ROOT::Internal::RNTupleSerializer::RContext ROOT::Internal::RPagePersistentSink::fSerializationContext

privateinherited

Used to map the IDs of the descriptor to the physical IDs issued during header/footer serialization.

Definition at line 450 of file RPageStorage.hxx.

◆ fTaskScheduler

RTaskScheduler* ROOT::Internal::RPageStorage::fTaskScheduler = nullptr

protectedinherited

Definition at line 152 of file RPageStorage.hxx.

◆ fWritePageMemoryManager

RWritePageMemoryManager ROOT::Internal::RPageSink::fWritePageMemoryManager

privateinherited

Used in ReservePage to maintain the page buffer budget.

Definition at line 292 of file RPageStorage.hxx.

◆ kNBytesPageChecksum

constexpr std::size_t ROOT::Internal::RPageStorage::kNBytesPageChecksum = sizeof(std::uint64_t)

staticconstexprinherited

The page checksum is a 64bit xxhash3.

Definition at line 73 of file RPageStorage.hxx.

tree/ntuple/inc/ROOT/RPageStorageS3.hxx

tree/ntuple/src/RPageStorageS3.cxx

Classes

Public Types

Public Member Functions

Static Public Member Functions

Static Public Attributes

Protected Member Functions

Protected Attributes

Private Member Functions

Private Attributes

Member Typedef Documentation

◆ Callback_t

◆ ColumnHandle_t

◆ SealedPageSequence_t

Constructor & Destructor Documentation

◆ RPageSinkS3() [1/2]

◆ RPageSinkS3() [2/2]

◆ ~RPageSinkS3()

Member Function Documentation

◆ AddColumn()

◆ CloneAsHidden()

◆ CommitAttributeSet()

◆ CommitCluster()

◆ CommitClusterGroup()

◆ CommitClusterGroupImpl()

◆ CommitDataset()

◆ CommitDatasetImpl() [1/3]

◆ CommitDatasetImpl() [2/3]

◆ CommitDatasetImpl() [3/3]

◆ CommitPage()

◆ CommitSealedPage()

◆ CommitSealedPageImpl()

◆ CommitSealedPageV()

◆ CommitSealedPageVImpl()

◆ CommitStagedClusters()

◆ CommitSuppressedColumn()

◆ Create()

◆ DropColumn()

◆ EnableDefaultMetrics()

◆ GetColumnId()

◆ GetDescriptor()

◆ GetMetrics()

◆ GetNEntries()

◆ GetNTupleName()

◆ GetSinkGuard()

◆ GetType()

◆ GetWriteOptions()

◆ Init()

◆ InitFromDescriptor()

◆ InitImpl() [1/3]

◆ InitImpl() [2/3]

◆ InitImpl() [3/3]

◆ IsInitialized()

◆ MakeObjectUrl()

◆ PutObject()

◆ RegisterOnCommitDatasetCallback()

◆ ReservePage()

◆ SealPage() [1/2]

◆ SealPage() [2/2]

◆ SetTaskScheduler()

◆ StageCluster()

◆ StageClusterImpl()

◆ UpdateExtraTypeInfo()

◆ UpdateSchema()

◆ WaitForAllTasks()

Member Data Documentation

◆ fAnchor

◆ fBaseUrl

◆ fConnection

◆ fCounters

◆ fDescriptorBuilder

◆ fFeatures

◆ fInfosOfStreamerFields

◆ fIsInitialized

◆ fMetrics

◆ fNBytesCurrentCluster

◆ fNextClusterInGroup

◆ fNTupleName

◆ fObjectId

◆ fOnDatasetCommitCallbacks

◆ fOpenColumnRanges