Field3D
SparseFile::Reference< Data_T > Class Template Reference

#include <SparseFile.h>

Public Types

typedef std::vector
< Sparse::SparseBlock< Data_T > * > 
BlockPtrs
 

Public Member Functions

float averageLoads () const
 Returns the average number of loads per accessed block in this file, for cache statistics. More...
 
int blockSize (int blockIdx) const
 Returns the number of bytes used by the data in the block. More...
 
void decBlockRef (int blockIdx)
 Decrement reference count on a block. More...
 
bool fileIsOpen ()
 Checks if the file used by this reference is open already. More...
 
void incBlockRef (int blockIdx)
 Increment reference count on a block, indicates the block is currently in use, so prevents it from being unloaded. More...
 
void loadBlock (int blockIdx)
 Loads the block with the given index into memory. We don't pass in a reference to where the data should go since this is already know in the blocks data member. More...
 
int numLoadedBlocks () const
 Returns the total number of blocks that are currently loaded, for statistics. More...
 
void openFile ()
 Opens the file. This is done just before the first request to loadBlock. This is delayed so that the original file open has closed the file and doesn't cause any Hdf5 hiccups. More...
 
Referenceoperator= (const Reference &o)
 Assignment operator. Clears ref counts and rebuilds mutex array. More...
 
 Reference (const std::string filename, const std::string layerPath)
 Constructor. Requires the filename and layer path of the field to be known. More...
 
 Reference (const Reference &o)
 Copy constructor. Clears ref counts and rebuilds mutex array. More...
 
void resetCacheStatistics ()
 Resets counts of total block loads. More...
 
void setNumBlocks (int numBlocks)
 Sets the number of blocks used by the SparseField we're supporting. More...
 
int totalLoadedBlocks () const
 Returns the total number of blocks that were ever loaded (max 1 per block, not the number of blocks), for statistics. More...
 
int totalLoads () const
 Returns the total number of loads of the blocks of this file, for cache statistics. More...
 
void unloadBlock (int blockIdx)
 Unloads the block with the given index from memory. More...
 
 ~Reference ()
 

Public Attributes

std::vector< int > blockLoaded
 Whether each block is loaded. We don't use bools since vector<bool> is weird. More...
 
boost::mutex * blockMutex
 Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-safe updates of the ref counts. More...
 
BlockPtrs blocks
 Pointers to each block. This is so we can go in and manipulate them as we please. More...
 
std::vector< bool > blockUsed
 Flags of whether the blocks have been accessed since they were last considered for deallocation by the Second-chance/Clock caching system. More...
 
std::vector< int > fileBlockIndices
 Index in file for each block. More...
 
std::string filename
 
std::string layerPath
 
std::vector< int > loadCounts
 Per-block counts of the number of times each block has been loaded, for cache statistics. More...
 
int occupiedBlocks
 
std::vector< int > refCounts
 Per-block counts of the number of current references to the blocks. If a block's ref count is non-zero, then the block shouldn't be unloaded. More...
 
int valuesPerBlock
 

Private Attributes

hid_t m_fileHandle
 Holds the Hdf5 handle to the file. More...
 
Hdf5Util::H5ScopedGopen m_layerGroup
 Hold the group containing the data open for the duration of the Reference's existence. More...
 
boost::mutex m_mutex
 Mutex to prevent two threads from modifying conflicting data. More...
 
SparseDataReader< Data_T > * m_reader
 Pointer to the reader object. NULL at construction time. Created in openFile(). More...
 

Detailed Description

template<class Data_T>
class SparseFile::Reference< Data_T >

Handles information about and block loading for a single SparseField as referenced on disk

Definition at line 95 of file SparseFile.h.

Member Typedef Documentation

template<class Data_T>
typedef std::vector<Sparse::SparseBlock<Data_T>*> SparseFile::Reference< Data_T >::BlockPtrs

Definition at line 101 of file SparseFile.h.

Constructor & Destructor Documentation

template<class Data_T >
SparseFile::Reference< Data_T >::Reference ( const std::string  filename,
const std::string  layerPath 
)

Constructor. Requires the filename and layer path of the field to be known.

Definition at line 475 of file SparseFile.h.

477  : filename(a_filename), layerPath(a_layerPath),
479  blockMutex(NULL), m_fileHandle(-1), m_reader(NULL) {
480  /* Empty */
481 }
hid_t m_fileHandle
Holds the Hdf5 handle to the file.
Definition: SparseFile.h:187
std::string layerPath
Definition: SparseFile.h:106
SparseDataReader< Data_T > * m_reader
Pointer to the reader object. NULL at construction time. Created in openFile().
Definition: SparseFile.h:195
std::string filename
Definition: SparseFile.h:105
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:132
template<class Data_T >
SparseFile::Reference< Data_T >::~Reference ( )

Definition at line 486 of file SparseFile.h.

487 {
488  if (m_reader)
489  delete m_reader;
490 
491  if (blockMutex)
492  delete [] blockMutex;
493 }
SparseDataReader< Data_T > * m_reader
Pointer to the reader object. NULL at construction time. Created in openFile().
Definition: SparseFile.h:195
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:132
template<class Data_T >
SparseFile::Reference< Data_T >::Reference ( const Reference< Data_T > &  o)

Copy constructor. Clears ref counts and rebuilds mutex array.

Definition at line 498 of file SparseFile.h.

499 {
500  m_reader = NULL;
501  blockMutex = NULL;
502  *this = o;
503 }
SparseDataReader< Data_T > * m_reader
Pointer to the reader object. NULL at construction time. Created in openFile().
Definition: SparseFile.h:195
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:132

Member Function Documentation

template<class Data_T >
Reference< Data_T > & SparseFile::Reference< Data_T >::operator= ( const Reference< Data_T > &  o)

Assignment operator. Clears ref counts and rebuilds mutex array.

Definition at line 509 of file SparseFile.h.

References SparseFile::Reference< Data_T >::blockLoaded, SparseFile::Reference< Data_T >::blocks, SparseFile::Reference< Data_T >::blockUsed, SparseFile::Reference< Data_T >::fileBlockIndices, SparseFile::Reference< Data_T >::filename, SparseFile::Reference< Data_T >::layerPath, SparseFile::Reference< Data_T >::loadCounts, SparseFile::Reference< Data_T >::m_fileHandle, SparseFile::Reference< Data_T >::occupiedBlocks, SparseFile::Reference< Data_T >::refCounts, and SparseFile::Reference< Data_T >::valuesPerBlock.

510 {
511  // Copy public member variables (where appropriate)
512  filename = o.filename;
513  layerPath = o.layerPath;
514  valuesPerBlock = o.valuesPerBlock;
515  occupiedBlocks = o.occupiedBlocks;
516  fileBlockIndices = o.fileBlockIndices;
517  blockLoaded = o.blockLoaded;
518  blocks = o.blocks;
519  blockUsed = o.blockUsed;
520  loadCounts = o.loadCounts;
521  refCounts = o.refCounts;
522  if (blockMutex)
523  delete[] blockMutex;
524  blockMutex = new boost::mutex[blocks.size()];
525 
526  // Copy private member variables (where appropriate)
527  m_fileHandle = o.m_fileHandle;
528  // Don't copy id, let hdf5 generate a new one.
529  if (m_fileHandle >= 0) {
531  }
532 
533  if (m_reader)
534  delete m_reader;
535  m_reader = NULL;
536 
537  return *this;
538 }
hid_t m_fileHandle
Holds the Hdf5 handle to the file.
Definition: SparseFile.h:187
std::vector< int > blockLoaded
Whether each block is loaded. We don't use bools since vector is weird.
Definition: SparseFile.h:114
std::vector< bool > blockUsed
Flags of whether the blocks have been accessed since they were last considered for deallocation by th...
Definition: SparseFile.h:121
std::vector< int > fileBlockIndices
Index in file for each block.
Definition: SparseFile.h:111
std::string layerPath
Definition: SparseFile.h:106
std::vector< int > refCounts
Per-block counts of the number of current references to the blocks. If a block's ref count is non-zer...
Definition: SparseFile.h:128
SparseDataReader< Data_T > * m_reader
Pointer to the reader object. NULL at construction time. Created in openFile().
Definition: SparseFile.h:195
BlockPtrs blocks
Pointers to each block. This is so we can go in and manipulate them as we please. ...
Definition: SparseFile.h:117
Hdf5Util::H5ScopedGopen m_layerGroup
Hold the group containing the data open for the duration of the Reference's existence.
Definition: SparseFile.h:191
std::string filename
Definition: SparseFile.h:105
std::vector< int > loadCounts
Per-block counts of the number of times each block has been loaded, for cache statistics.
Definition: SparseFile.h:124
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:132
void open(hid_t parentLocation, const std::string &name)
Definition: Hdf5Util.h:192
template<class Data_T >
bool SparseFile::Reference< Data_T >::fileIsOpen ( )

Checks if the file used by this reference is open already.

Definition at line 543 of file SparseFile.h.

Referenced by SparseFileManager::activateBlock().

544 {
545  return m_fileHandle >= 0;
546 }
hid_t m_fileHandle
Holds the Hdf5 handle to the file.
Definition: SparseFile.h:187
template<class Data_T >
void SparseFile::Reference< Data_T >::setNumBlocks ( int  numBlocks)

Sets the number of blocks used by the SparseField we're supporting.

Definition at line 551 of file SparseFile.h.

552 {
553  boost::mutex::scoped_lock lock(m_mutex);
554 
555  fileBlockIndices.resize(numBlocks);
556  blockLoaded.resize(numBlocks, 0);
557  blocks.resize(numBlocks, 0);
558  blockUsed.resize(numBlocks, false);
559  loadCounts.resize(numBlocks, 0);
560  refCounts.resize(numBlocks, 0);
561  if (blockMutex)
562  delete[] blockMutex;
563  blockMutex = new boost::mutex[numBlocks];
564 }
std::vector< int > blockLoaded
Whether each block is loaded. We don't use bools since vector is weird.
Definition: SparseFile.h:114
std::vector< bool > blockUsed
Flags of whether the blocks have been accessed since they were last considered for deallocation by th...
Definition: SparseFile.h:121
std::vector< int > fileBlockIndices
Index in file for each block.
Definition: SparseFile.h:111
std::vector< int > refCounts
Per-block counts of the number of current references to the blocks. If a block's ref count is non-zer...
Definition: SparseFile.h:128
BlockPtrs blocks
Pointers to each block. This is so we can go in and manipulate them as we please. ...
Definition: SparseFile.h:117
boost::mutex m_mutex
Mutex to prevent two threads from modifying conflicting data.
Definition: SparseFile.h:198
std::vector< int > loadCounts
Per-block counts of the number of times each block has been loaded, for cache statistics.
Definition: SparseFile.h:124
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:132
template<class Data_T >
void SparseFile::Reference< Data_T >::openFile ( )

Opens the file. This is done just before the first request to loadBlock. This is delayed so that the original file open has closed the file and doesn't cause any Hdf5 hiccups.

Definition at line 569 of file SparseFile.h.

References Msg::print(), and Msg::SevWarning.

Referenced by SparseFileManager::activateBlock().

570 {
571  using namespace Exc;
572  using namespace Hdf5Util;
573 
574  boost::mutex::scoped_lock lock_A(m_mutex);
575 
576  // check that the file wasn't already opened before obtaining the lock
577  if (fileIsOpen()) {
578  return;
579  }
580 
581  m_fileHandle = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, H5P_DEFAULT);
582  if (m_fileHandle < 0)
583  throw NoSuchFileException(filename);
584 
586  if (m_layerGroup.id() < 0) {
587  Msg::print(Msg::SevWarning, "In SparseFile::Reference::openFile: "
588  "Couldn't find layer group " + layerPath +
589  " in .f3d file ");
590  throw FileIntegrityException(filename);
591  }
592 
595 }
Contains utility functions and classes for Hdf5 files.
Definition: Hdf5Util.h:76
hid_t m_fileHandle
Holds the Hdf5 handle to the file.
Definition: SparseFile.h:187
Namespace for Exception objects.
Definition: Exception.h:57
void print(Severity severity, const std::string &message)
Sends the string to the assigned output, prefixing the message with the severity. ...
Definition: Log.cpp:62
std::string layerPath
Definition: SparseFile.h:106
SparseDataReader< Data_T > * m_reader
Pointer to the reader object. NULL at construction time. Created in openFile().
Definition: SparseFile.h:195
This class gets used by SparseFieldIO and SparseFileManager to read the block data. On creation it will open the data set and not close it until the object is destroyed.
boost::mutex m_mutex
Mutex to prevent two threads from modifying conflicting data.
Definition: SparseFile.h:198
Hdf5Util::H5ScopedGopen m_layerGroup
Hold the group containing the data open for the duration of the Reference's existence.
Definition: SparseFile.h:191
std::string filename
Definition: SparseFile.h:105
bool fileIsOpen()
Checks if the file used by this reference is open already.
Definition: SparseFile.h:543
void open(hid_t parentLocation, const std::string &name)
Definition: Hdf5Util.h:192
hid_t id() const
Query the hid_t value.
Definition: Hdf5Util.h:90
template<class Data_T >
void SparseFile::Reference< Data_T >::loadBlock ( int  blockIdx)

Loads the block with the given index into memory. We don't pass in a reference to where the data should go since this is already know in the blocks data member.

Definition at line 600 of file SparseFile.h.

Referenced by SparseFileManager::activateBlock().

601 {
602  boost::mutex::scoped_lock lock(m_mutex);
603 
604  // Allocate the block
605  blocks[blockIdx]->resize(valuesPerBlock);
606  assert(blocks[blockIdx]->data.size() > 0);
607  // Read the data
608  assert(m_reader);
609  m_reader->readBlock(fileBlockIndices[blockIdx], blocks[blockIdx]->dataRef());
610  // Mark block as loaded
611  blockLoaded[blockIdx] = 1;
612 }
std::vector< int > blockLoaded
Whether each block is loaded. We don't use bools since vector is weird.
Definition: SparseFile.h:114
std::vector< int > fileBlockIndices
Index in file for each block.
Definition: SparseFile.h:111
SparseDataReader< Data_T > * m_reader
Pointer to the reader object. NULL at construction time. Created in openFile().
Definition: SparseFile.h:195
BlockPtrs blocks
Pointers to each block. This is so we can go in and manipulate them as we please. ...
Definition: SparseFile.h:117
boost::mutex m_mutex
Mutex to prevent two threads from modifying conflicting data.
Definition: SparseFile.h:198
template<class Data_T >
void SparseFile::Reference< Data_T >::unloadBlock ( int  blockIdx)

Unloads the block with the given index from memory.

Definition at line 617 of file SparseFile.h.

Referenced by SparseFileManager::deallocateBlock().

618 {
619  // Deallocate the block
620  blocks[blockIdx]->clear();
621 
622  // Mark block as unloaded
623  blockLoaded[blockIdx] = 0;
624 }
std::vector< int > blockLoaded
Whether each block is loaded. We don't use bools since vector is weird.
Definition: SparseFile.h:114
BlockPtrs blocks
Pointers to each block. This is so we can go in and manipulate them as we please. ...
Definition: SparseFile.h:117
template<class Data_T >
void SparseFile::Reference< Data_T >::incBlockRef ( int  blockIdx)

Increment reference count on a block, indicates the block is currently in use, so prevents it from being unloaded.

Definition at line 629 of file SparseFile.h.

Referenced by SparseFileManager::incBlockRef().

630 {
631  boost::mutex::scoped_lock lock(blockMutex[blockIdx]);
632  ++refCounts[blockIdx];
633 }
std::vector< int > refCounts
Per-block counts of the number of current references to the blocks. If a block's ref count is non-zer...
Definition: SparseFile.h:128
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:132
template<class Data_T >
void SparseFile::Reference< Data_T >::decBlockRef ( int  blockIdx)

Decrement reference count on a block.

Definition at line 638 of file SparseFile.h.

Referenced by SparseFileManager::decBlockRef().

639 {
640  boost::mutex::scoped_lock lock(blockMutex[blockIdx]);
641  --refCounts[blockIdx];
642 }
std::vector< int > refCounts
Per-block counts of the number of current references to the blocks. If a block's ref count is non-zer...
Definition: SparseFile.h:128
boost::mutex * blockMutex
Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-s...
Definition: SparseFile.h:132
template<class Data_T >
int SparseFile::Reference< Data_T >::blockSize ( int  blockIdx) const

Returns the number of bytes used by the data in the block.

Definition at line 647 of file SparseFile.h.

Referenced by SparseFileManager::activateBlock(), SparseFileManager::deallocateBlock(), and SparseFileManager::removeFieldFromCache().

648 {
649  return valuesPerBlock * sizeof(Data_T);
650 }
template<class Data_T >
int SparseFile::Reference< Data_T >::totalLoads ( ) const

Returns the total number of loads of the blocks of this file, for cache statistics.

Definition at line 655 of file SparseFile.h.

656 {
657  std::vector<int>::const_iterator i = loadCounts.begin();
658  std::vector<int>::const_iterator end = loadCounts.end();
659  int numLoads = 0;
660  for (; i != end; ++i)
661  numLoads += *i;
662 
663  return numLoads;
664 }
std::vector< int > loadCounts
Per-block counts of the number of times each block has been loaded, for cache statistics.
Definition: SparseFile.h:124
template<class Data_T >
int SparseFile::Reference< Data_T >::numLoadedBlocks ( ) const

Returns the total number of blocks that are currently loaded, for statistics.

Definition at line 669 of file SparseFile.h.

670 {
671  std::vector<int>::const_iterator i = blockLoaded.begin();
672  std::vector<int>::const_iterator end = blockLoaded.end();
673  int numBlocks = 0;
674  for (; i != end; ++i)
675  if (*i)
676  numBlocks++;
677 
678  return numBlocks;
679 }
std::vector< int > blockLoaded
Whether each block is loaded. We don't use bools since vector is weird.
Definition: SparseFile.h:114
template<class Data_T >
int SparseFile::Reference< Data_T >::totalLoadedBlocks ( ) const

Returns the total number of blocks that were ever loaded (max 1 per block, not the number of blocks), for statistics.

Definition at line 684 of file SparseFile.h.

685 {
686  std::vector<int>::const_iterator i = loadCounts.begin();
687  std::vector<int>::const_iterator li = blockLoaded.begin();
688  std::vector<int>::const_iterator end = loadCounts.end();
689  int numBlocks = 0;
690 
691  if (blockLoaded.size() == 0) {
692  for (; i != end; ++i)
693  if (*i)
694  numBlocks++;
695  } else {
696  assert(loadCounts.size() == blockLoaded.size());
697 
698  for (; i != end; ++i, ++li)
699  if (*i || *li)
700  numBlocks++;
701  }
702 
703  return numBlocks;
704 }
std::vector< int > blockLoaded
Whether each block is loaded. We don't use bools since vector is weird.
Definition: SparseFile.h:114
std::vector< int > loadCounts
Per-block counts of the number of times each block has been loaded, for cache statistics.
Definition: SparseFile.h:124
template<class Data_T >
float SparseFile::Reference< Data_T >::averageLoads ( ) const

Returns the average number of loads per accessed block in this file, for cache statistics.

Definition at line 709 of file SparseFile.h.

710 {
711  std::vector<int>::const_iterator i = loadCounts.begin();
712  std::vector<int>::const_iterator end = loadCounts.end();
713  int numLoads = 0, numBlocks = 0;
714  for (; i != end; ++i) {
715  if (*i) {
716  numLoads += *i;
717  numBlocks++;
718  }
719  }
720 
721  return (float)numLoads / std::max(1, numBlocks);
722 }
std::vector< int > loadCounts
Per-block counts of the number of times each block has been loaded, for cache statistics.
Definition: SparseFile.h:124
template<class Data_T >
void SparseFile::Reference< Data_T >::resetCacheStatistics ( )

Resets counts of total block loads.

Definition at line 727 of file SparseFile.h.

728 {
729  std::vector<int>::iterator li = loadCounts.begin();
730  std::vector<int>::iterator lend = loadCounts.end();
731  for (; li != lend; ++li)
732  *li = 0;
733 }
std::vector< int > loadCounts
Per-block counts of the number of times each block has been loaded, for cache statistics.
Definition: SparseFile.h:124

Member Data Documentation

template<class Data_T>
std::string SparseFile::Reference< Data_T >::filename
template<class Data_T>
std::string SparseFile::Reference< Data_T >::layerPath
template<class Data_T>
int SparseFile::Reference< Data_T >::valuesPerBlock
template<class Data_T>
int SparseFile::Reference< Data_T >::occupiedBlocks
template<class Data_T>
std::vector<int> SparseFile::Reference< Data_T >::blockLoaded

Whether each block is loaded. We don't use bools since vector<bool> is weird.

Definition at line 114 of file SparseFile.h.

Referenced by SparseFileManager::activateBlock(), SparseFile::Reference< Data_T >::operator=(), and SparseFileManager::removeFieldFromCache().

template<class Data_T>
BlockPtrs SparseFile::Reference< Data_T >::blocks

Pointers to each block. This is so we can go in and manipulate them as we please.

Definition at line 117 of file SparseFile.h.

Referenced by SparseFile::Reference< Data_T >::operator=(), SparseFileManager::removeFieldFromCache(), and SparseField< Data_T >::setupReferenceBlocks().

template<class Data_T>
std::vector<bool> SparseFile::Reference< Data_T >::blockUsed

Flags of whether the blocks have been accessed since they were last considered for deallocation by the Second-chance/Clock caching system.

Definition at line 121 of file SparseFile.h.

Referenced by SparseFileManager::activateBlock(), SparseFileManager::deallocateBlock(), SparseFile::Reference< Data_T >::operator=(), and SparseFileManager::removeFieldFromCache().

template<class Data_T>
std::vector<int> SparseFile::Reference< Data_T >::loadCounts

Per-block counts of the number of times each block has been loaded, for cache statistics.

Definition at line 124 of file SparseFile.h.

Referenced by SparseFileManager::activateBlock(), and SparseFile::Reference< Data_T >::operator=().

template<class Data_T>
std::vector<int> SparseFile::Reference< Data_T >::refCounts

Per-block counts of the number of current references to the blocks. If a block's ref count is non-zero, then the block shouldn't be unloaded.

Definition at line 128 of file SparseFile.h.

Referenced by SparseFileManager::deallocateBlock(), and SparseFile::Reference< Data_T >::operator=().

template<class Data_T>
boost::mutex* SparseFile::Reference< Data_T >::blockMutex

Allocated array of mutexes, one per block, to lock each block individually, for guaranteeing thread-safe updates of the ref counts.

Definition at line 132 of file SparseFile.h.

Referenced by SparseFileManager::activateBlock(), and SparseFileManager::deallocateBlock().

template<class Data_T>
hid_t SparseFile::Reference< Data_T >::m_fileHandle
private

Holds the Hdf5 handle to the file.

Definition at line 187 of file SparseFile.h.

Referenced by SparseFile::Reference< Data_T >::operator=().

template<class Data_T>
Hdf5Util::H5ScopedGopen SparseFile::Reference< Data_T >::m_layerGroup
private

Hold the group containing the data open for the duration of the Reference's existence.

Definition at line 191 of file SparseFile.h.

template<class Data_T>
SparseDataReader<Data_T>* SparseFile::Reference< Data_T >::m_reader
private

Pointer to the reader object. NULL at construction time. Created in openFile().

Definition at line 195 of file SparseFile.h.

template<class Data_T>
boost::mutex SparseFile::Reference< Data_T >::m_mutex
private

Mutex to prevent two threads from modifying conflicting data.

Definition at line 198 of file SparseFile.h.


The documentation for this class was generated from the following file: