ROOT  6.07/01
Reference Guide
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Groups Pages
List of all members | Public Member Functions | Private Member Functions | Private Attributes | List of all members
TPacketizerAdaptive Class Reference

This packetizer is based on TPacketizer but uses different load-balancing algorithms and data structures.

Two main improvements in the load-balancing strategy:

The data structures: TFileStat, TFileNode and TSlaveStat are enriched + changed and TFileNode::Compare method is changed.

Definition at line 50 of file TPacketizerAdaptive.h.

Public Member Functions

 TPacketizerAdaptive (TDSet *dset, TList *slaves, Long64_t first, Long64_t num, TList *input, TProofProgressStatus *st)
 
virtual ~TPacketizerAdaptive ()
 Destructor. More...
 
Int_t AddProcessed (TSlave *sl, TProofProgressStatus *st, Double_t latency, TList **listOfMissingFiles=0)
 To be used by GetNextPacket but also in reaction to kPROOF_STOPPROCESS message (when the worker was asked to stop processing during a packet). More...
 
Int_t GetEstEntriesProcessed (Float_t, Long64_t &ent, Long64_t &bytes, Long64_t &calls)
 Get estimation for the number of processed entries and bytes read at time t, based on the numbers already processed and the latests worker measured speeds. More...
 
Float_t GetCurrentRate (Bool_t &all)
 Get Estimation of the current rate; just summing the current rates of the active workers. More...
 
Int_t CalculatePacketSize (TObject *slstat, Long64_t cachesz, Int_t learnent)
 The result depends on the fStrategy. More...
 
TDSetElementGetNextPacket (TSlave *sl, TMessage *r)
 Get next packet; A meaningfull difference to TPacketizer is the fact that this packetizer, for each worker, tries to predict whether the worker will finish processing it's local files before the end of the query. More...
 
void MarkBad (TSlave *s, TProofProgressStatus *status, TList **missingFiles)
 This method can be called at any time during processing as an effect of handling kPROOF_STOPPROCESS If the output list from this worker is going to be sent back to the master, the 'status' includes the number of entries processed by the slave. More...
 
Int_t GetActiveWorkers ()
 Return the number of workers still processing. More...
 
- Public Member Functions inherited from TVirtualPacketizer
virtual ~TVirtualPacketizer ()
 Destructor. More...
 
virtual Int_t AssignWork (TDSet *, Long64_t, Long64_t)
 
Bool_t IsValid () const
 
Long64_t GetEntriesProcessed () const
 
Long64_t GetTotalEntries () const
 
virtual void SetInitTime ()
 Set the initialization time. More...
 
virtual void StopProcess (Bool_t abort, Bool_t stoptimer=kFALSE)
 Stop process. More...
 
TListGetFailedPackets ()
 
void SetFailedPackets (TList *list)
 
virtual Int_t AddWorkers (TList *workers)
 Adds new workers. More...
 
Long64_t GetBytesRead () const
 
Long64_t GetReadCalls () const
 
Double_t GetCumProcTime () const
 
Float_t GetInitTime () const
 
Float_t GetProcTime () const
 
TNtupleGetProgressPerf (Bool_t steal=kFALSE)
 
TListGetConfigParams (Bool_t steal=kFALSE)
 
TProofProgressStatusGetStatus ()
 
void SetProgressStatus (TProofProgressStatus *st)
 
void SetTotalEntries (Long64_t ent)
 
TMapGetSlaveStats () const
 
- Public Member Functions inherited from TObject
 TObject ()
 
 TObject (const TObject &object)
 TObject copy ctor. More...
 
TObjectoperator= (const TObject &rhs)
 TObject assignment operator. More...
 
virtual ~TObject ()
 TObject destructor. More...
 
virtual void AppendPad (Option_t *option="")
 Append graphics object to current pad. More...
 
virtual void Browse (TBrowser *b)
 Browse object. May be overridden for another default action. More...
 
virtual const char * ClassName () const
 Returns name of class to which the object belongs. More...
 
virtual void Clear (Option_t *="")
 
virtual TObjectClone (const char *newname="") const
 Make a clone of an object using the Streamer facility. More...
 
virtual Int_t Compare (const TObject *obj) const
 Compare abstract method. More...
 
virtual void Copy (TObject &object) const
 Copy this to obj. More...
 
virtual void Delete (Option_t *option="")
 Delete this object. More...
 
virtual Int_t DistancetoPrimitive (Int_t px, Int_t py)
 Computes distance from point (px,py) to the object. More...
 
virtual void Draw (Option_t *option="")
 Default Draw method for all objects. More...
 
virtual void DrawClass () const
 Draw class inheritance tree of the class to which this object belongs. More...
 
virtual TObjectDrawClone (Option_t *option="") const
 Draw a clone of this object in the current pad. More...
 
virtual void Dump () const
 Dump contents of object on stdout. More...
 
virtual void Execute (const char *method, const char *params, Int_t *error=0)
 Execute method on this object with the given parameter string, e.g. More...
 
virtual void Execute (TMethod *method, TObjArray *params, Int_t *error=0)
 Execute method on this object with parameters stored in the TObjArray. More...
 
virtual void ExecuteEvent (Int_t event, Int_t px, Int_t py)
 Execute action corresponding to an event at (px,py). More...
 
virtual TObjectFindObject (const char *name) const
 Must be redefined in derived classes. More...
 
virtual TObjectFindObject (const TObject *obj) const
 Must be redefined in derived classes. More...
 
virtual Option_tGetDrawOption () const
 Get option used by the graphics system to draw this object. More...
 
virtual UInt_t GetUniqueID () const
 Return the unique object id. More...
 
virtual const char * GetName () const
 Returns name of object. More...
 
virtual const char * GetIconName () const
 Returns mime type name of object. More...
 
virtual Option_tGetOption () const
 
virtual char * GetObjectInfo (Int_t px, Int_t py) const
 Returns string containing info about the object at position (px,py). More...
 
virtual const char * GetTitle () const
 Returns title of object. More...
 
virtual ULong_t Hash () const
 Return hash value for this object. More...
 
virtual Bool_t InheritsFrom (const char *classname) const
 Returns kTRUE if object inherits from class "classname". More...
 
virtual Bool_t InheritsFrom (const TClass *cl) const
 Returns kTRUE if object inherits from TClass cl. More...
 
virtual void Inspect () const
 Dump contents of this object in a graphics canvas. More...
 
virtual Bool_t IsFolder () const
 Returns kTRUE in case object contains browsable objects (like containers or lists of other objects). More...
 
virtual Bool_t IsEqual (const TObject *obj) const
 Default equal comparison (objects are equal if they have the same address in memory). More...
 
virtual Bool_t IsSortable () const
 
Bool_t IsOnHeap () const
 
Bool_t IsZombie () const
 
virtual Bool_t Notify ()
 This method must be overridden to handle object notification. More...
 
virtual void ls (Option_t *option="") const
 The ls function lists the contents of a class on stdout. More...
 
virtual void Paint (Option_t *option="")
 This method must be overridden if a class wants to paint itself. More...
 
virtual void Pop ()
 Pop on object drawn in a pad to the top of the display list. More...
 
virtual void Print (Option_t *option="") const
 This method must be overridden when a class wants to print itself. More...
 
virtual Int_t Read (const char *name)
 Read contents of object with specified name from the current directory. More...
 
virtual void RecursiveRemove (TObject *obj)
 Recursively remove this object from a list. More...
 
virtual void SaveAs (const char *filename="", Option_t *option="") const
 Save this object in the file specified by filename. More...
 
virtual void SavePrimitive (std::ostream &out, Option_t *option="")
 Save a primitive as a C++ statement(s) on output stream "out". More...
 
virtual void SetDrawOption (Option_t *option="")
 Set drawing option for object. More...
 
virtual void SetUniqueID (UInt_t uid)
 Set the unique object id. More...
 
virtual void UseCurrentStyle ()
 Set current style settings in this object This function is called when either TCanvas::UseCurrentStyle or TROOT::ForceStyle have been invoked. More...
 
virtual Int_t Write (const char *name=0, Int_t option=0, Int_t bufsize=0)
 Write this object to the current directory. More...
 
virtual Int_t Write (const char *name=0, Int_t option=0, Int_t bufsize=0) const
 Write this object to the current directory. More...
 
voidoperator new (size_t sz)
 
voidoperator new[] (size_t sz)
 
voidoperator new (size_t sz, void *vp)
 
voidoperator new[] (size_t sz, void *vp)
 
void operator delete (void *ptr)
 Operator delete. More...
 
void operator delete[] (void *ptr)
 Operator delete []. More...
 
void SetBit (UInt_t f, Bool_t set)
 Set or unset the user status bits as specified in f. More...
 
void SetBit (UInt_t f)
 
void ResetBit (UInt_t f)
 
Bool_t TestBit (UInt_t f) const
 
Int_t TestBits (UInt_t f) const
 
void InvertBit (UInt_t f)
 
virtual void Info (const char *method, const char *msgfmt,...) const
 Issue info message. More...
 
virtual void Warning (const char *method, const char *msgfmt,...) const
 Issue warning message. More...
 
virtual void Error (const char *method, const char *msgfmt,...) const
 Issue error message. More...
 
virtual void SysError (const char *method, const char *msgfmt,...) const
 Issue system error message. More...
 
virtual void Fatal (const char *method, const char *msgfmt,...) const
 Issue fatal error message. More...
 
void AbstractMethod (const char *method) const
 Use this method to implement an "abstract" method that you don't want to leave purely abstract. More...
 
void MayNotUse (const char *method) const
 Use this method to signal that a method (defined in a base class) may not be called in a derived class (in principle against good design since a child class should not provide less functionality than its parent, however, sometimes it is necessary). More...
 
void Obsolete (const char *method, const char *asOfVers, const char *removedFromVers) const
 Use this method to declare a method obsolete. More...
 

Private Member Functions

 TPacketizerAdaptive ()
 
 TPacketizerAdaptive (const TPacketizerAdaptive &)
 
void InitStats ()
 (re)initialise the statistics called at the begining or after a worker dies. More...
 
void operator= (const TPacketizerAdaptive &)
 
TFileNode * NextNode ()
 Get next node which has unallocated files. More...
 
void RemoveUnAllocNode (TFileNode *)
 Remove unallocated node. More...
 
TFileNode * NextActiveNode ()
 Get next active node. More...
 
void RemoveActiveNode (TFileNode *)
 Remove node from the list of actives. More...
 
TFileStat * GetNextUnAlloc (TFileNode *node=0, const char *nodeHostName=0)
 Get next unallocated file from 'node' or other nodes: First try 'node'. More...
 
TFileStat * GetNextActive ()
 Get next active file. More...
 
void RemoveActive (TFileStat *file)
 Remove file from the list of actives. More...
 
void Reset ()
 Reset the internal data structure for packet distribution. More...
 
void ValidateFiles (TDSet *dset, TList *slaves, Long64_t maxent=-1, Bool_t byfile=kFALSE)
 Check existence of file/dir/tree an get number of entries. More...
 
Int_t ReassignPacket (TDSetElement *e, TList **listOfMissingFiles)
 The file in the listOfMissingFiles can appear several times; in order to fix that, a TDSetElement::Merge method is needed. More...
 
void SplitPerHost (TList *elements, TList **listOfMissingFiles)
 Split into per host entries The files in the listOfMissingFiles can appear several times; in order to fix that, a TDSetElement::Merge method is needed. More...
 

Private Attributes

TListfFileNodes
 
TListfUnAllocated
 
TListfActive
 
Int_t fMaxPerfIdx
 
TListfPartitions
 
TSortedListfFilesToProcess
 
Bool_t fCachePacketSync
 
Double_t fMaxEntriesRatio
 
Float_t fFractionOfRemoteFiles
 
Long64_t fNEventsOnRemLoc
 
Float_t fBaseLocalPreference
 
Bool_t fForceLocal
 
Long_t fMaxSlaveCnt
 
Int_t fPacketAsAFraction
 
Int_t fStrategy
 
Int_t fTryReassign
 

Additional Inherited Members

- Public Types inherited from TVirtualPacketizer
enum  EStatusBits { kIsInitializing = BIT(16), kIsDone = BIT(17), kIsTree = BIT(18) }
 
- Public Types inherited from TObject
enum  EStatusBits {
  kCanDelete = BIT(0), kMustCleanup = BIT(3), kObjInCanvas = BIT(3), kIsReferenced = BIT(4),
  kHasUUID = BIT(5), kCannotPick = BIT(6), kNoContextMenu = BIT(8), kInvalidObject = BIT(13)
}
 
enum  { kIsOnHeap = 0x01000000, kNotDeleted = 0x02000000, kZombie = 0x04000000, kBitMask = 0x00ffffff }
 
enum  { kSingleKey = BIT(0), kOverwrite = BIT(1), kWriteDelete = BIT(2) }
 
- Static Public Member Functions inherited from TObject
static Long_t GetDtorOnly ()
 Return destructor only flag. More...
 
static void SetDtorOnly (void *obj)
 Set destructor only flag. More...
 
static Bool_t GetObjectStat ()
 Get status of object stat flag. More...
 
static void SetObjectStat (Bool_t stat)
 Turn on/off tracking of objects in the TObjectTable. More...
 
- Protected Types inherited from TVirtualPacketizer
enum  EUseEstOpt { kEstOff = 0, kEstCurrent = 1, kEstAverage = 2 }
 
- Protected Member Functions inherited from TVirtualPacketizer
 TVirtualPacketizer (TList *input, TProofProgressStatus *st=0)
 
 TVirtualPacketizer (const TVirtualPacketizer &)
 
void operator= (const TVirtualPacketizer &)
 
TDSetElementCreateNewPacket (TDSetElement *base, Long64_t first, Long64_t num)
 Creates a new TDSetElement from from base packet starting from the first entry with num entries. More...
 
Long64_t GetEntries (Bool_t tree, TDSetElement *e)
 Get entries. More...
 
virtual Bool_t HandleTimer (TTimer *timer)
 Send progress message to client. More...
 
- Protected Member Functions inherited from TObject
void MakeZombie ()
 
virtual void DoError (int level, const char *location, const char *fmt, va_list va) const
 Interface to ErrorHandler (protected). More...
 
- Protected Attributes inherited from TVirtualPacketizer
Double_t fMinPacketTime
 
Double_t fMaxPacketTime
 
TListfConfigParams
 
TMapfSlaveStats
 
TProofProgressStatusfProgressStatus
 
TTimerfProgress
 
Long64_t fTotalEntries
 
TListfFailedPackets
 
TTime fStartTime
 
Float_t fInitTime
 
Float_t fProcTime
 
Float_t fTimeUpdt
 
TNtupleDfCircProg
 
Long_t fCircN
 
TNtuplefProgressPerf
 
Float_t fProcTimeLast
 
Int_t fActWrksLast
 
Float_t fEvtRateLast
 
Float_t fMBsReadLast
 
Float_t fEffSessLast
 
Bool_t fAWLastFill
 
Float_t fReportPeriod
 
EUseEstOpt fUseEstOpt
 
Bool_t fValid
 
Bool_t fStop
 
TString fDataSet
 
TListfInput
 

#include <TPacketizerAdaptive.h>

Inheritance diagram for TPacketizerAdaptive:
[legend]

Constructor & Destructor Documentation

TPacketizerAdaptive::TPacketizerAdaptive ( )
private
TPacketizerAdaptive::TPacketizerAdaptive ( const TPacketizerAdaptive )
private
TPacketizerAdaptive::TPacketizerAdaptive ( TDSet dset,
TList slaves,
Long64_t  first,
Long64_t  num,
TList input,
TProofProgressStatus st 
)
TPacketizerAdaptive::~TPacketizerAdaptive ( )
virtual

Destructor.

Definition at line 880 of file TPacketizerAdaptive.cxx.

Member Function Documentation

Int_t TPacketizerAdaptive::AddProcessed ( TSlave sl,
TProofProgressStatus status,
Double_t  latency,
TList **  listOfMissingFiles = 0 
)
virtual

To be used by GetNextPacket but also in reaction to kPROOF_STOPPROCESS message (when the worker was asked to stop processing during a packet).

returns the #entries intended in the last packet - #processed entries

Reimplemented from TVirtualPacketizer.

Definition at line 1569 of file TPacketizerAdaptive.cxx.

Int_t TPacketizerAdaptive::CalculatePacketSize ( TObject slstat,
Long64_t  cachesz,
Int_t  learnent 
)

The result depends on the fStrategy.

Definition at line 1479 of file TPacketizerAdaptive.cxx.

Referenced by GetNextPacket().

Int_t TPacketizerAdaptive::GetActiveWorkers ( )
virtual

Return the number of workers still processing.

Reimplemented from TVirtualPacketizer.

Definition at line 1949 of file TPacketizerAdaptive.cxx.

Float_t TPacketizerAdaptive::GetCurrentRate ( Bool_t all)
virtual

Get Estimation of the current rate; just summing the current rates of the active workers.

Reimplemented from TVirtualPacketizer.

Definition at line 1966 of file TPacketizerAdaptive.cxx.

Int_t TPacketizerAdaptive::GetEstEntriesProcessed ( Float_t  t,
Long64_t ent,
Long64_t bytes,
Long64_t calls 
)
virtual

Get estimation for the number of processed entries and bytes read at time t, based on the numbers already processed and the latests worker measured speeds.

If t <= 0 the current time is used. Only the estimation for the entries is currently implemented. This is needed to smooth the instantaneous rate plot.

Reimplemented from TVirtualPacketizer.

Definition at line 1995 of file TPacketizerAdaptive.cxx.

TPacketizerAdaptive::TFileStat * TPacketizerAdaptive::GetNextActive ( )
private

Get next active file.

Definition at line 1047 of file TPacketizerAdaptive.cxx.

TDSetElement * TPacketizerAdaptive::GetNextPacket ( TSlave sl,
TMessage r 
)
virtual

Get next packet; A meaningfull difference to TPacketizer is the fact that this packetizer, for each worker, tries to predict whether the worker will finish processing it's local files before the end of the query.

If yes, it allocates, to those workers, files from non-slave filenodes or from slaves that are overloaded. The check is done every time a new file needs to be assigned.

Reimplemented from TVirtualPacketizer.

Definition at line 1666 of file TPacketizerAdaptive.cxx.

TPacketizerAdaptive::TFileStat * TPacketizerAdaptive::GetNextUnAlloc ( TFileNode *  node = 0,
const char *  nodeHostName = 0 
)
private

Get next unallocated file from 'node' or other nodes: First try 'node'.

If there is no more files, keep trying to find an unallocated file on other nodes.

Definition at line 935 of file TPacketizerAdaptive.cxx.

void TPacketizerAdaptive::InitStats ( )
private

(re)initialise the statistics called at the begining or after a worker dies.

Definition at line 897 of file TPacketizerAdaptive.cxx.

Referenced by ClassImp(), and MarkBad().

void TPacketizerAdaptive::MarkBad ( TSlave s,
TProofProgressStatus status,
TList **  listOfMissingFiles 
)
virtual

This method can be called at any time during processing as an effect of handling kPROOF_STOPPROCESS If the output list from this worker is going to be sent back to the master, the 'status' includes the number of entries processed by the slave.

From this we calculate the remaining part of the packet. 0 indicates that the results from that worker were lost completely. Assume that the filenodes for which we have a TFileNode object are still up and running.

Reimplemented from TVirtualPacketizer.

Definition at line 2070 of file TPacketizerAdaptive.cxx.

TPacketizerAdaptive::TFileNode * TPacketizerAdaptive::NextActiveNode ( )
private

Get next active node.

Definition at line 1064 of file TPacketizerAdaptive.cxx.

TPacketizerAdaptive::TFileNode * TPacketizerAdaptive::NextNode ( )
private

Get next node which has unallocated files.

the order is determined by TFileNode::Compare

Definition at line 1018 of file TPacketizerAdaptive.cxx.

Referenced by GetNextUnAlloc().

void TPacketizerAdaptive::operator= ( const TPacketizerAdaptive )
private
Int_t TPacketizerAdaptive::ReassignPacket ( TDSetElement e,
TList **  listOfMissingFiles 
)
private

The file in the listOfMissingFiles can appear several times; in order to fix that, a TDSetElement::Merge method is needed.

Definition at line 2130 of file TPacketizerAdaptive.cxx.

Referenced by AddProcessed(), and SplitPerHost().

void TPacketizerAdaptive::RemoveActive ( TFileStat *  file)
private

Remove file from the list of actives.

Definition at line 1086 of file TPacketizerAdaptive.cxx.

void TPacketizerAdaptive::RemoveActiveNode ( TFileNode *  node)
private

Remove node from the list of actives.

Definition at line 1097 of file TPacketizerAdaptive.cxx.

void TPacketizerAdaptive::RemoveUnAllocNode ( TFileNode *  node)
private

Remove unallocated node.

Definition at line 1039 of file TPacketizerAdaptive.cxx.

void TPacketizerAdaptive::Reset ( void  )
private

Reset the internal data structure for packet distribution.

Definition at line 1105 of file TPacketizerAdaptive.cxx.

void TPacketizerAdaptive::SplitPerHost ( TList elements,
TList **  listOfMissingFiles 
)
private

Split into per host entries The files in the listOfMissingFiles can appear several times; in order to fix that, a TDSetElement::Merge method is needed.

Definition at line 2175 of file TPacketizerAdaptive.cxx.

Referenced by MarkBad().

void TPacketizerAdaptive::ValidateFiles ( TDSet dset,
TList slaves,
Long64_t  maxent = -1,
Bool_t  byfile = kFALSE 
)
private

Check existence of file/dir/tree an get number of entries.

Assumes the files have been setup.

Definition at line 1154 of file TPacketizerAdaptive.cxx.

Member Data Documentation

TList* TPacketizerAdaptive::fActive
private

Definition at line 60 of file TPacketizerAdaptive.h.

Float_t TPacketizerAdaptive::fBaseLocalPreference
private

Definition at line 72 of file TPacketizerAdaptive.h.

Referenced by ClassImp(), and GetNextPacket().

Bool_t TPacketizerAdaptive::fCachePacketSync
private

Definition at line 66 of file TPacketizerAdaptive.h.

Referenced by CalculatePacketSize(), and ClassImp().

TList* TPacketizerAdaptive::fFileNodes
private

Definition at line 55 of file TPacketizerAdaptive.h.

TSortedList* TPacketizerAdaptive::fFilesToProcess
private
Bool_t TPacketizerAdaptive::fForceLocal
private

Definition at line 74 of file TPacketizerAdaptive.h.

Referenced by ClassImp(), and GetNextPacket().

Float_t TPacketizerAdaptive::fFractionOfRemoteFiles
private

Definition at line 69 of file TPacketizerAdaptive.h.

Referenced by InitStats().

Double_t TPacketizerAdaptive::fMaxEntriesRatio
private

Definition at line 67 of file TPacketizerAdaptive.h.

Referenced by CalculatePacketSize(), and ClassImp().

Int_t TPacketizerAdaptive::fMaxPerfIdx
private

Definition at line 61 of file TPacketizerAdaptive.h.

Long_t TPacketizerAdaptive::fMaxSlaveCnt
private

Definition at line 76 of file TPacketizerAdaptive.h.

Long64_t TPacketizerAdaptive::fNEventsOnRemLoc
private

Definition at line 70 of file TPacketizerAdaptive.h.

Referenced by GetNextPacket(), and InitStats().

Int_t TPacketizerAdaptive::fPacketAsAFraction
private

Definition at line 78 of file TPacketizerAdaptive.h.

TList* TPacketizerAdaptive::fPartitions
private

Definition at line 62 of file TPacketizerAdaptive.h.

Int_t TPacketizerAdaptive::fStrategy
private

Definition at line 84 of file TPacketizerAdaptive.h.

Referenced by CalculatePacketSize(), ClassImp(), and GetNextPacket().

Int_t TPacketizerAdaptive::fTryReassign
private

Definition at line 85 of file TPacketizerAdaptive.h.

Referenced by ClassImp(), and ReassignPacket().

TList* TPacketizerAdaptive::fUnAllocated
private

Definition at line 59 of file TPacketizerAdaptive.h.

Collaboration diagram for TPacketizerAdaptive:
[legend]

The documentation for this class was generated from the following files: