The logic for constructing chunks from a dataset.

This struct handles the logic for splitting a dataset into smaller subsets known as chunks, which are constructed from blocks.

A chunk is the largest portion of the dataset loaded into memory at once, and each chunk is further divided into batches for machine learning training.

The dataset is split into disjoint chunks based on a user-defined chunk size. There are two types of chunks:

Full chunks: contain exactly the number of entries specified by the chunk size.
Leftover chunk: contains any remaining entries that don't make up a full chunk.

Each chunk is constructed from blocks based on a user-defined block size. There are two types of blocks:

Full blocks: contain exactly the number of entries specified by the block size.
Leftover block: contains any remaining entries that don't make up a full block.

The blocks are defined by their start and end entries, which correspond to positions within the dataset’s total number of entries.

Definition at line 45 of file RChunkConstructor.hxx.

Public Member Functions
	RChunkConstructor (const std::size_t numEntries, const std::size_t chunkSize, const std::size_t blockSize)

void	CreateChunksIntervals ()
	Creates chunks from the dataset consisting of blocks with the begin and end entry.

void	DistributeBlockIntervals ()
	Group the blocks based on the block type (full or leftover) based on the size of the block.

void	SizeOfChunks ()
	Fills a vector with the size of every chunk from the dataset.

Public Attributes
std::vector< std::pair< Long_t, Long_t > >	BlockIntervals

std::size_t	BlockPerFullChunk

std::size_t	BlockPerLeftoverChunk

std::size_t	Chunks

std::vector< std::vector< std::pair< Long_t, Long_t > > >	ChunksIntervals

std::vector< std::size_t >	ChunksSizes

std::size_t	fBlockSize {}

std::size_t	fChunkSize {}

std::size_t	fNumEntries {}

std::vector< std::pair< Long_t, Long_t > >	FullBlockIntervalsInFullChunks

std::vector< std::pair< Long_t, Long_t > >	FullBlockIntervalsInLeftoverChunks

std::size_t	FullBlocksInFullChunks

std::size_t	FullBlocksInLeftoverChunks

std::size_t	FullBlocksPerFullChunk

std::size_t	FullBlocksPerLeftoverChunk

std::size_t	FullChunks

std::vector< std::pair< Long_t, Long_t > >	LeftoverBlockIntervalsInFullChunks

std::vector< std::pair< Long_t, Long_t > >	LeftoverBlockIntervalsInLeftoverChunks

std::size_t	LeftoverBlocksInFullChunks

std::size_t	LeftoverBlocksInLeftoverChunks

std::size_t	LeftoverBlocksPerFullChunk

std::size_t	LeftoverBlocksPerLeftoverChunk

std::size_t	LeftoverChunks

std::size_t	NumberOfBlocks

std::vector< std::size_t >	NumberOfDifferentBlocks

std::vector< std::size_t >	SizeOfBlocks

std::size_t	SizeOfFullBlockInFullChunk

std::size_t	SizeOfFullBlockInLeftoverChunk

std::size_t	SizeOfFullChunk

std::size_t	SizeOfLeftoverBlockInFullChunk

std::size_t	SizeOfLeftoverBlockInLeftoverChunk

std::size_t	SizeOfLeftoverChunk

#include <ROOT/ML/RChunkConstructor.hxx>

Constructor & Destructor Documentation

◆ RChunkConstructor()

ROOT::Experimental::Internal::ML::RChunkConstructor::RChunkConstructor	(	const std::size_t	numEntries,
		const std::size_t	chunkSize,
		const std::size_t	blockSize )

Definition at line 7 of file RChunkConstructor.cxx.

Member Function Documentation

◆ CreateChunksIntervals()

void ROOT::Experimental::Internal::ML::RChunkConstructor::CreateChunksIntervals ( )

Creates chunks from the dataset consisting of blocks with the begin and end entry.

Definition at line 80 of file RChunkConstructor.cxx.

◆ DistributeBlockIntervals()

void ROOT::Experimental::Internal::ML::RChunkConstructor::DistributeBlockIntervals ( )

Group the blocks based on the block type (full or leftover) based on the size of the block.

Definition at line 58 of file RChunkConstructor.cxx.

◆ SizeOfChunks()

void ROOT::Experimental::Internal::ML::RChunkConstructor::SizeOfChunks ( )

Fills a vector with the size of every chunk from the dataset.

Definition at line 118 of file RChunkConstructor.cxx.

Member Data Documentation

◆ BlockIntervals

std::vector<std::pair<Long_t, Long_t> > ROOT::Experimental::Internal::ML::RChunkConstructor::BlockIntervals

Definition at line 93 of file RChunkConstructor.hxx.

◆ BlockPerFullChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::BlockPerFullChunk

Definition at line 68 of file RChunkConstructor.hxx.

◆ BlockPerLeftoverChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::BlockPerLeftoverChunk

Definition at line 73 of file RChunkConstructor.hxx.

◆ Chunks

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::Chunks

Definition at line 63 of file RChunkConstructor.hxx.

◆ ChunksIntervals

std::vector<std::vector<std::pair<Long_t, Long_t> > > ROOT::Experimental::Internal::ML::RChunkConstructor::ChunksIntervals

Definition at line 101 of file RChunkConstructor.hxx.

◆ ChunksSizes

std::vector<std::size_t> ROOT::Experimental::Internal::ML::RChunkConstructor::ChunksSizes

Definition at line 103 of file RChunkConstructor.hxx.

◆ fBlockSize

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::fBlockSize {}

Definition at line 48 of file RChunkConstructor.hxx.

◆ fChunkSize

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::fChunkSize {}

Definition at line 47 of file RChunkConstructor.hxx.

◆ fNumEntries

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::fNumEntries {}

Definition at line 46 of file RChunkConstructor.hxx.

◆ FullBlockIntervalsInFullChunks

std::vector<std::pair<Long_t, Long_t> > ROOT::Experimental::Internal::ML::RChunkConstructor::FullBlockIntervalsInFullChunks

Definition at line 95 of file RChunkConstructor.hxx.

◆ FullBlockIntervalsInLeftoverChunks

std::vector<std::pair<Long_t, Long_t> > ROOT::Experimental::Internal::ML::RChunkConstructor::FullBlockIntervalsInLeftoverChunks

Definition at line 98 of file RChunkConstructor.hxx.

◆ FullBlocksInFullChunks

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::FullBlocksInFullChunks

Definition at line 76 of file RChunkConstructor.hxx.

◆ FullBlocksInLeftoverChunks

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::FullBlocksInLeftoverChunks

Definition at line 80 of file RChunkConstructor.hxx.

◆ FullBlocksPerFullChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::FullBlocksPerFullChunk

Definition at line 66 of file RChunkConstructor.hxx.

◆ FullBlocksPerLeftoverChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::FullBlocksPerLeftoverChunk

Definition at line 71 of file RChunkConstructor.hxx.

◆ FullChunks

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::FullChunks

Definition at line 61 of file RChunkConstructor.hxx.

◆ LeftoverBlockIntervalsInFullChunks

std::vector<std::pair<Long_t, Long_t> > ROOT::Experimental::Internal::ML::RChunkConstructor::LeftoverBlockIntervalsInFullChunks

Definition at line 96 of file RChunkConstructor.hxx.

◆ LeftoverBlockIntervalsInLeftoverChunks

std::vector<std::pair<Long_t, Long_t> > ROOT::Experimental::Internal::ML::RChunkConstructor::LeftoverBlockIntervalsInLeftoverChunks

Definition at line 99 of file RChunkConstructor.hxx.

◆ LeftoverBlocksInFullChunks

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::LeftoverBlocksInFullChunks

Definition at line 77 of file RChunkConstructor.hxx.

◆ LeftoverBlocksInLeftoverChunks

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::LeftoverBlocksInLeftoverChunks

Definition at line 81 of file RChunkConstructor.hxx.

◆ LeftoverBlocksPerFullChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::LeftoverBlocksPerFullChunk

Definition at line 67 of file RChunkConstructor.hxx.

◆ LeftoverBlocksPerLeftoverChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::LeftoverBlocksPerLeftoverChunk

Definition at line 72 of file RChunkConstructor.hxx.

◆ LeftoverChunks

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::LeftoverChunks

Definition at line 62 of file RChunkConstructor.hxx.

◆ NumberOfBlocks

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::NumberOfBlocks

Definition at line 90 of file RChunkConstructor.hxx.

◆ NumberOfDifferentBlocks

std::vector<std::size_t> ROOT::Experimental::Internal::ML::RChunkConstructor::NumberOfDifferentBlocks

Definition at line 87 of file RChunkConstructor.hxx.

◆ SizeOfBlocks

std::vector<std::size_t> ROOT::Experimental::Internal::ML::RChunkConstructor::SizeOfBlocks

Definition at line 84 of file RChunkConstructor.hxx.

◆ SizeOfFullBlockInFullChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::SizeOfFullBlockInFullChunk

Definition at line 55 of file RChunkConstructor.hxx.

◆ SizeOfFullBlockInLeftoverChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::SizeOfFullBlockInLeftoverChunk

Definition at line 57 of file RChunkConstructor.hxx.

◆ SizeOfFullChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::SizeOfFullChunk

Definition at line 51 of file RChunkConstructor.hxx.

◆ SizeOfLeftoverBlockInFullChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::SizeOfLeftoverBlockInFullChunk

Definition at line 56 of file RChunkConstructor.hxx.

◆ SizeOfLeftoverBlockInLeftoverChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::SizeOfLeftoverBlockInLeftoverChunk

Definition at line 58 of file RChunkConstructor.hxx.

◆ SizeOfLeftoverChunk

std::size_t ROOT::Experimental::Internal::ML::RChunkConstructor::SizeOfLeftoverChunk

Definition at line 52 of file RChunkConstructor.hxx.

Collaboration diagram for ROOT::Experimental::Internal::ML::RChunkConstructor:

[legend]

The documentation for this class was generated from the following files:

tree/ml/inc/ROOT/ML/RChunkConstructor.hxx
tree/ml/src/RChunkConstructor.cxx

Public Member Functions

Public Attributes