Storage provider that reads ntuple pages from a DAOS container.

Definition at line 150 of file RPageStorageDaos.hxx.

Public Member Functions
	RPageSourceDaos (std::string_view ntupleName, std::string_view uri, const ROOT::RNTupleReadOptions &options)

	~RPageSourceDaos () override

std::string	GetObjectClass () const
	Return the object class used for user data OIDs in this ntuple.

std::vector< std::unique_ptr< ROOT::Internal::RCluster > >	LoadClusters (std::span< ROOT::Internal::RCluster::RKey > clusterKeys) final
	Populates all the pages of the given cluster ids and columns; it is possible that some columns do not contain any pages. The page source may load more columns than the minimal necessary set from `columns`. To indicate which columns have been loaded, `LoadClusters()must mark them with`SetColumnAvailable()`. That includes the ones from the`columns`that don't have pages; otherwise subsequent requests for the cluster would assume an incomplete cluster and trigger loading again.` LoadClusters()` is typically called from the I/O thread of a cluster pool, i.e. the method runs concurrently to other methods of the page source.

void	LoadSealedPage (ROOT::DescriptorId_t physicalColumnId, RNTupleLocalIndex localIndex, RSealedPage &sealedPage) final
	Read the packed and compressed bytes of a page into the memory buffer provided by `sealedPage`. The sealed page can be used subsequently in a call to `RPageSink::CommitSealedPage`. The `fSize` and `fNElements` member of the sealedPage parameters are always set. If `sealedPage.fBuffer` is `nullptr`, no data will be copied but the returned size information can be used by the caller to allocate a large enough buffer and call `LoadSealedPage` again.

void	LoadStreamerInfo () final
	Forces the loading of ROOT StreamerInfo from the underlying file. This currently only has an effect for TFile-backed sources.

std::unique_ptr< RPageSource >	OpenWithDifferentAnchor (const ROOT::Internal::RNTupleLink &anchorLink, const ROOT::RNTupleReadOptions &options={}) final
	Creates a new PageSource using the same underlying file as this but referring to a different RNTuple, described by `anchorLink`.

Public Member Functions inherited from ROOT::Internal::RPageSource
	RPageSource (const RPageSource &)=delete

	RPageSource (RPageSource &&)=delete

	RPageSource (std::string_view ntupleName, const ROOT::RNTupleReadOptions &fOptions)

	~RPageSource () override

ColumnHandle_t	AddColumn (ROOT::DescriptorId_t fieldId, ROOT::Internal::RColumn &column) override
	Register a new column.

void	Attach (ROOT::Internal::RNTupleSerializer::EDescriptorDeserializeMode mode=ROOT::Internal::RNTupleSerializer::EDescriptorDeserializeMode::kForReading)
	Open the physical storage container and deserialize header and footer.

std::unique_ptr< RPageSource >	Clone () const
	Open the same storage multiple time, e.g. for reading in multiple threads. If the source is already attached, the clone will be attached, too. The clone will use, however, it's own connection to the underlying storage (e.g., file descriptor, XRootD handle, etc.)

void	DropColumn (ColumnHandle_t columnHandle) override
	Unregisters a column.

REntryRange	GetEntryRange () const

ROOT::NTupleSize_t	GetNElements (ColumnHandle_t columnHandle)

ROOT::NTupleSize_t	GetNEntries ()

const std::unordered_set< ROOT::DescriptorId_t > &	GetPinnedClusters () const

const ROOT::RNTupleReadOptions &	GetReadOptions () const

const RSharedDescriptorGuard	GetSharedDescriptorGuard () const
	Takes the read lock for the descriptor. Multiple threads can take the lock concurrently. The underlying `std::shared_mutex`, however, is neither read nor write recursive: within one thread, only one lock (shared or exclusive) must be acquired at the same time. This requires special care in sections protected by `GetSharedDescriptorGuard()` and `GetExclDescriptorGuard()` especially to avoid that the locks are acquired indirectly. As a general guideline, no other method of the page source should be called (directly or indirectly) in a guarded section.

EPageStorageType	GetType () final
	Whether the concrete implementation is a sink or a source.

virtual ROOT::Internal::RPageRef	LoadPage (ColumnHandle_t columnHandle, RNTupleLocalIndex localIndex)
	Another version of `LoadPage` that allows to specify cluster-relative indexes. Returns a default-constructed RPage for suppressed columns.

virtual ROOT::Internal::RPageRef	LoadPage (ColumnHandle_t columnHandle, ROOT::NTupleSize_t globalIndex)
	Allocates and fills a page that contains the index-th element. The default implementation searches the page and calls LoadPageImpl(). Returns a default-constructed RPage for suppressed columns.

void	LoadStructure ()
	Loads header and footer without decompressing or deserializing them. This can be used to asynchronously open a file in the background. The method is idempotent and it is called as a first step in `Attach()`. Pages sources may or may not make use of splitting loading and processing metadata. Therefore, `LoadStructure()` may do nothing and defer loading the metadata to `Attach()`.

RPageSource &	operator= (const RPageSource &)=delete

RPageSource &	operator= (RPageSource &&)=delete

void	PinCluster (ROOT::DescriptorId_t clusterId)
	Instructs the cluster pool and page pool to consider the given cluster as active (should stay cached).

void	RegisterStreamerInfos ()
	Builds the streamer info records from the descriptor's extra type info section. This is necessary when connecting streamer fields so that emulated classes can be read.

void	SetEntryRange (const REntryRange &range)
	Promise to only read from the given entry range. If set, prevents the cluster pool from reading-ahead beyond the given range. The range needs to be within `[0, GetNEntries())`.

void	UnpinCluster (ROOT::DescriptorId_t clusterId)
	Allows the given cluster to be evicted from the cluster pool and page pool.

RResult< ROOT::Internal::RPage >	UnsealPage (const RSealedPage &sealedPage, const ROOT::Internal::RColumnElementBase &element)

void	UnzipCluster (ROOT::Internal::RCluster *cluster)
	Parallel decompression and unpacking of the pages in the given cluster. The unzipped pages are supposed to be preloaded in a page pool attached to the source. The method is triggered by the cluster pool's unzip thread. It is an optional optimization, the method can safely do nothing. In particular, the actual implementation will only run if a task scheduler is set. In practice, a task scheduler is set if implicit multi-threading is turned on.

Public Member Functions inherited from ROOT::Internal::RPageStorage
	RPageStorage (const RPageStorage &other)=delete

	RPageStorage (RPageStorage &&other)=default

	RPageStorage (std::string_view name)

virtual	~RPageStorage ()

ROOT::DescriptorId_t	GetColumnId (ColumnHandle_t columnHandle) const

virtual ROOT::Experimental::Detail::RNTupleMetrics &	GetMetrics ()
	Returns the default metrics object.

const std::string &	GetNTupleName () const
	Returns the NTuple name.

RPageStorage &	operator= (const RPageStorage &other)=delete

RPageStorage &	operator= (RPageStorage &&other)=default

void	SetTaskScheduler (RTaskScheduler *taskScheduler)

Protected Member Functions
ROOT::RNTupleDescriptor	AttachImpl (ROOT::Internal::RNTupleSerializer::EDescriptorDeserializeMode mode) final
	`LoadStructureImpl()` has been called before `AttachImpl()` is called

std::unique_ptr< RPageSource >	CloneImpl () const final
	The cloned page source creates a new connection to the pool/container.

void	LoadStructureImpl () final

Protected Member Functions inherited from ROOT::Internal::RPageSource
void	EnableDefaultMetrics (const std::string &prefix)
	Enables the default set of metrics provided by RPageSource. `prefix` will be used as the prefix for the counters registered in the internal RNTupleMetrics object. A subclass using the default set of metrics is responsible for updating the counters appropriately, e.g. `fCounters->fNRead.Inc()` Alternatively, a subclass might provide its own RNTupleMetrics object by overriding the `GetMetrics()` member function.

RExclDescriptorGuard	GetExclDescriptorGuard ()
	Note that the underlying lock is not recursive. See `GetSharedDescriptorGuard()` for further information.

void	PrepareLoadCluster (const ROOT::Internal::RCluster::RKey &clusterKey, ROOT::Internal::ROnDiskPageMap &pageZeroMap, std::function< void(ROOT::DescriptorId_t, ROOT::NTupleSize_t, const ROOT::RClusterDescriptor::RPageInfo &)> perPageFunc)
	Prepare a page range read for the column set in `clusterKey`. Specifically, pages referencing the `kTypePageZero` locator are filled in `pageZeroMap`; otherwise, `perPageFunc` is called for each page. This is commonly used as part of `LoadClusters()` in derived classes.

virtual void	UnzipClusterImpl (ROOT::Internal::RCluster *cluster)

Protected Member Functions inherited from ROOT::Internal::RPageStorage
void	WaitForAllTasks ()

Private Member Functions
ROOT::Internal::RPageRef	LoadPageImpl (ColumnHandle_t columnHandle, const RClusterInfo &clusterInfo, ROOT::NTupleSize_t idxInCluster) final

Private Attributes
ROOT::Internal::RCluster *	fCurrentCluster = nullptr
	The last cluster from which a page got loaded. Points into fClusterPool->fPool.

std::unique_ptr< RDaosContainer >	fDaosContainer
	A container that stores object data (header/footer, pages, etc.)

ROOT::Internal::RNTupleDescriptorBuilder	fDescriptorBuilder

ntuple_index_t	fNTupleIndex {0}

std::string	fURI
	A URI to a DAOS pool of the form 'daos://pool-label/container-label'.

Additional Inherited Members
Public Types inherited from ROOT::Internal::RPageStorage
using	ColumnHandle_t = RColumnHandle
	The column handle identifies a column with the current open page storage.

using	SealedPageSequence_t = std::deque<RSealedPage>

Static Public Member Functions inherited from ROOT::Internal::RPageSource
static std::unique_ptr< RPageSource >	Create (std::string_view ntupleName, std::string_view location, const ROOT::RNTupleReadOptions &options=ROOT::RNTupleReadOptions())
	Guess the concrete derived page source from the file name (location)

static RResult< ROOT::Internal::RPage >	UnsealPage (const RSealedPage &sealedPage, const ROOT::Internal::RColumnElementBase &element, ROOT::Internal::RPageAllocator &pageAlloc)
	Helper for unstreaming a page. This is commonly used in derived, concrete page sources. The implementation currently always makes a memory copy, even if the sealed page is uncompressed and in the final memory layout. The optimization of directly mapping pages is left to the concrete page source implementations.

Static Public Attributes inherited from ROOT::Internal::RPageStorage
static constexpr std::size_t	kNBytesPageChecksum = sizeof(std::uint64_t)
	The page checksum is a 64bit xxhash3.

Protected Attributes inherited from ROOT::Internal::RPageSource
RActivePhysicalColumns	fActivePhysicalColumns
	The active columns are implicitly defined by the model fields or views.

ROOT::Internal::RClusterPool	fClusterPool
	The cluster pool asynchronously preloads the next few clusters. Note that derived classes should call fClusterPool.StopBackgroundThread() in their destructor so that the I/O background thread does not call methods from the destructed derived class.

std::unique_ptr< RCounters >	fCounters

ROOT::RNTupleReadOptions	fOptions

ROOT::Internal::RPagePool	fPagePool
	Pages that are unzipped with IMT are staged into the page pool.

std::unordered_set< ROOT::DescriptorId_t >	fPinnedClusters
	Pinned clusters and their $2 * (cluster bunch size) - 1$ successors will not be evicted from the cluster pool. Pages of pinned clusters won't be evicted from the page pool.

Protected Attributes inherited from ROOT::Internal::RPageStorage
ROOT::Experimental::Detail::RNTupleMetrics	fMetrics

std::string	fNTupleName

std::unique_ptr< ROOT::Internal::RPageAllocator >	fPageAllocator
	For the time being, we will use the heap allocator for all sources and sinks. This may change in the future.

RTaskScheduler *	fTaskScheduler = nullptr

#include <ROOT/RPageStorageDaos.hxx>

Inheritance diagram for ROOT::Experimental::Internal::RPageSourceDaos:

[legend]

Constructor & Destructor Documentation

◆ RPageSourceDaos()

ROOT::Experimental::Internal::RPageSourceDaos::RPageSourceDaos	(	std::string_view	ntupleName,
		std::string_view	uri,
		const ROOT::RNTupleReadOptions &	options )

Definition at line 462 of file RPageStorageDaos.cxx.

◆ ~RPageSourceDaos()

ROOT::Experimental::Internal::RPageSourceDaos::~RPageSourceDaos ( )

override

Definition at line 473 of file RPageStorageDaos.cxx.

Member Function Documentation

◆ AttachImpl()

ROOT::RNTupleDescriptor ROOT::Experimental::Internal::RPageSourceDaos::AttachImpl ( ROOT::Internal::RNTupleSerializer::EDescriptorDeserializeMode mode )

finalprotectedvirtual

LoadStructureImpl() has been called before AttachImpl() is called

Implements ROOT::Internal::RPageSource.

Definition at line 479 of file RPageStorageDaos.cxx.

◆ CloneImpl()

std::unique_ptr< ROOT::Internal::RPageSource > ROOT::Experimental::Internal::RPageSourceDaos::CloneImpl ( ) const

finalprotectedvirtual

The cloned page source creates a new connection to the pool/container.

Implements ROOT::Internal::RPageSource.

Definition at line 621 of file RPageStorageDaos.cxx.

◆ GetObjectClass()

std::string ROOT::Experimental::Internal::RPageSourceDaos::GetObjectClass ( ) const

Return the object class used for user data OIDs in this ntuple.

Definition at line 514 of file RPageStorageDaos.cxx.

◆ LoadClusters()

std::vector< std::unique_ptr< RCluster > > ROOT::Experimental::Internal::RPageSourceDaos::LoadClusters ( std::span< ROOT::Internal::RCluster::RKey > clusterKeys )

finalvirtual

Populates all the pages of the given cluster ids and columns; it is possible that some columns do not contain any pages. The page source may load more columns than the minimal necessary set from columns. To indicate which columns have been loaded, LoadClusters()must mark them withSetColumnAvailable(). That includes the ones from thecolumnsthat don't have pages; otherwise subsequent requests for the cluster would assume an incomplete cluster and trigger loading again. LoadClusters()` is typically called from the I/O thread of a cluster pool, i.e. the method runs concurrently to other methods of the page source.

Implements ROOT::Internal::RPageSource.

Definition at line 628 of file RPageStorageDaos.cxx.

◆ LoadPageImpl()

ROOT::Internal::RPageRef ROOT::Experimental::Internal::RPageSourceDaos::LoadPageImpl	(	ColumnHandle_t	columnHandle,
		const RClusterInfo &	clusterInfo,
		ROOT::NTupleSize_t	idxInCluster )

finalprivatevirtual

Implements ROOT::Internal::RPageSource.

Definition at line 554 of file RPageStorageDaos.cxx.

◆ LoadSealedPage()

void ROOT::Experimental::Internal::RPageSourceDaos::LoadSealedPage	(	ROOT::DescriptorId_t	physicalColumnId,
		RNTupleLocalIndex	localIndex,
		RSealedPage &	sealedPage )

finalvirtual

Read the packed and compressed bytes of a page into the memory buffer provided by sealedPage. The sealed page can be used subsequently in a call to RPageSink::CommitSealedPage. The fSize and fNElements member of the sealedPage parameters are always set. If sealedPage.fBuffer is nullptr, no data will be copied but the returned size information can be used by the caller to allocate a large enough buffer and call LoadSealedPage again.

Implements ROOT::Internal::RPageSource.

Definition at line 519 of file RPageStorageDaos.cxx.

◆ LoadStreamerInfo()

void ROOT::Experimental::Internal::RPageSourceDaos::LoadStreamerInfo ( )

finalvirtual

Forces the loading of ROOT StreamerInfo from the underlying file. This currently only has an effect for TFile-backed sources.

Implements ROOT::Internal::RPageSource.

Definition at line 715 of file RPageStorageDaos.cxx.

◆ LoadStructureImpl()

void ROOT::Experimental::Internal::RPageSourceDaos::LoadStructureImpl ( )

inlinefinalprotectedvirtual

Implements ROOT::Internal::RPageSource.

Definition at line 167 of file RPageStorageDaos.hxx.

◆ OpenWithDifferentAnchor()

std::unique_ptr< ROOT::Internal::RPageSource > ROOT::Experimental::Internal::RPageSourceDaos::OpenWithDifferentAnchor	(	const ROOT::Internal::RNTupleLink &	anchorLink,
		const ROOT::RNTupleReadOptions &	options = {} )

finalvirtual

Creates a new PageSource using the same underlying file as this but referring to a different RNTuple, described by anchorLink.

Implements ROOT::Internal::RPageSource.

Definition at line 721 of file RPageStorageDaos.cxx.

Member Data Documentation

◆ fCurrentCluster

ROOT::Internal::RCluster* ROOT::Experimental::Internal::RPageSourceDaos::fCurrentCluster = nullptr

private

The last cluster from which a page got loaded. Points into fClusterPool->fPool.

Definition at line 155 of file RPageStorageDaos.hxx.

◆ fDaosContainer

std::unique_ptr<RDaosContainer> ROOT::Experimental::Internal::RPageSourceDaos::fDaosContainer

private

A container that stores object data (header/footer, pages, etc.)

Definition at line 157 of file RPageStorageDaos.hxx.

◆ fDescriptorBuilder

ROOT::Internal::RNTupleDescriptorBuilder ROOT::Experimental::Internal::RPageSourceDaos::fDescriptorBuilder

private

Definition at line 161 of file RPageStorageDaos.hxx.

◆ fNTupleIndex

ntuple_index_t ROOT::Experimental::Internal::RPageSourceDaos::fNTupleIndex {0}

private

Definition at line 152 of file RPageStorageDaos.hxx.

◆ fURI

std::string ROOT::Experimental::Internal::RPageSourceDaos::fURI

private

A URI to a DAOS pool of the form 'daos://pool-label/container-label'.

Definition at line 159 of file RPageStorageDaos.hxx.

tree/ntuple/inc/ROOT/RPageStorageDaos.hxx

tree/ntuple/src/RPageStorageDaos.cxx

Public Member Functions

Protected Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Constructor & Destructor Documentation

◆ RPageSourceDaos()

◆ ~RPageSourceDaos()

Member Function Documentation

◆ AttachImpl()

◆ CloneImpl()

◆ GetObjectClass()

◆ LoadClusters()

◆ LoadPageImpl()

◆ LoadSealedPage()

◆ LoadStreamerInfo()

◆ LoadStructureImpl()

◆ OpenWithDifferentAnchor()

Member Data Documentation

◆ fCurrentCluster

◆ fDaosContainer

◆ fDescriptorBuilder

◆ fNTupleIndex

◆ fURI