Class FlashJournalRM

java.lang.Object
com.tangosol.io.journal.AbstractJournalRM
com.tangosol.io.journal.FlashJournalRM
All Implemented Interfaces:
Disposable, BinaryStoreManager, ClassLoaderAware, JournalMBean, Controllable, AutoCloseable
public class FlashJournalRM extends AbstractJournalRM
A FlashJournalRM manages disk I/O for temporary journal-based files, and acts as a shared resource for any number of journals that share a common configuration.

To use the Resource Manager, configure it using Dependency Injection by passing in the populated dependency object at instantiation. Once configured, start the Resource Manager via the AbstractJournalRM.start() method, and then obtain BinaryStore instances via the AbstractJournalRM.createBinaryStore() method. When a BinaryStore instance is no longer required, dispose of it via the Disposable interface, and similarly stop() or AbstractJournalRM.dispose() of the Resource Manager when it is no longer needed to ensure that any resources it allocated are cleaned up and released.

While the Resource Manager is running, there will be one or more files created in the specified temporary directory (or the default temporary directory if none is specified). The number of files is not related to the number of BinaryStore instances, but rather to the total amount of data that the Resource Manager is managing at any given time. Additionally, the Resource Manager uses three daemon threads to help manage the data: "Journal-Preparer", "Journal-Writer" and "Journal-Collector". These threads respectively lay out the data to be written, write the data, and perform asynchronous GC of the storage files.

The limits on the journal are as follows:

  • Values are limited by default to 64MB;
  • An individual file is limited by default to 2GB (and maximum 4GB);
  • A journal is composed of up to 512 files;
  • A journal is thus limited by default to 1TB, with a theoretical maximum of 2TB

Since:
Coherence 3.7
Author:
cp/cf 2010-06-10

  • Field Details

    • SHIFT_FILE_ID

      protected static final int SHIFT_FILE_ID
      The location of the file id within the ticket.
      See Also:

    • MASK_FILE_ID

      protected static final long MASK_FILE_ID
      The 64-bit bitmask for the file index. The 9 bits from 62-54.
      See Also:

    • SHIFT_OFFSET

      protected static final int SHIFT_OFFSET
      The location of the value-offset within the ticket.
      See Also:

    • SHIFT_OFFSET_RAW

      protected static final int SHIFT_OFFSET_RAW
      From an encoding point of view, the raw offset is shifted four bits less than necessary, because the four least significant bits of the offset are always zero (due to paragraph alignment).
      See Also:

    • MASK_OFFSET

      protected static final long MASK_OFFSET
      The 64-bit bitmask for the value offset. The 28 bits 53-26.

      Note that the least significant four bits of the offset MUST ALWAYS be zero, because they are NOT encoded into the ticket -- hence the 16-byte paragraph boundary for entries.

      See Also:

    • SHIFT_LENGTH

      protected static final int SHIFT_LENGTH
      The location of the value-length within the ticket.
      See Also:

    • MASK_LENGTH

      protected static final long MASK_LENGTH
      The 64-bit bitmask for the value length. The 26 bits 25-0.
      See Also:

    • TYPE_NAME

      protected static final String TYPE_NAME
      A simple name to be used as part of MBean ObjectNames.
      See Also:

    • EMPTY_WRITER_STATISTICS

      protected static final FlashJournalRM.WriterStatistics EMPTY_WRITER_STATISTICS
      Empty statistics returned while the writer daemon is not running.

    • WAIT_CONGESTION_ACTION

      protected static final FlashJournalRM.WaitCongestionPolicy WAIT_CONGESTION_ACTION
      The default congestion policy which blocks the client thread until awoken by the FlashJournalRM.WriterDaemon.

    • FAIL_CONGESTION_ACTION

      protected static final FlashJournalRM.FailCongestionPolicy FAIL_CONGESTION_ACTION
      A FlashJournalRM.CongestionPolicy which fails, by throwing an unchecked exception, when the file is deemed congested.

    • SIMULATE_DISK_FULL

      protected static final boolean SIMULATE_DISK_FULL
      A flag used for flash journal testing.
      Since:
      12.2.1.4.26

    • IO_EXCEPTION_MESSAGE

      public static final String IO_EXCEPTION_MESSAGE
      IOException message. Used to check for IOException in the error message.
      Since:
      12.2.1.4.26
      See Also:

    • m_congestionPolicy

      protected volatile FlashJournalRM.CongestionPolicy m_congestionPolicy
      The FlashJournalRM.CongestionPolicy to be used when in a congested state.

    • m_statsCongestion

      protected final FlashJournalRM.CongestionStatistics m_statsCongestion
      Journal-wide congestion statistics.

    • m_statsFileLifecycle

      protected volatile FlashJournalRM.FileLifecycleStatistics m_statsFileLifecycle
      Resettable file lifecycle statistics.
      Since:
      26.1.0.0

    • m_lFileCreationSequence

      protected final AtomicLong m_lFileCreationSequence
      Non-resetting sequence assigned to successfully constructed files.
      Since:
      26.1.0.0

    • m_ldtMostRecentFileCreation

      protected final AtomicLong m_ldtMostRecentFileCreation
      Creation time of the most recently created file.
      Since:
      26.1.0.0

    • m_ldtMostRecentFileDeletion

      protected final AtomicLong m_ldtMostRecentFileDeletion
      Deletion time of the most recently deleted file.
      Since:
      26.1.0.0

    • m_ldtMostRecentCollectorScanCompletion

      protected final AtomicLong m_ldtMostRecentCollectorScanCompletion
      Completion time of the most recent collector scan.
      Since:
      26.1.0.0

    • m_dirTemp

      protected File m_dirTemp
      Directory to store the temporary files.

    • m_fLoggedDirectory

      protected boolean m_fLoggedDirectory
      Set to true after the directory location has been logged.

    • m_cbBacklog

      protected AtomicInteger m_cbBacklog
      Total amount of data enqueued to be written.

    • m_mapLocators

      protected ConcurrentMap<Long,Binary> m_mapLocators
      Map of locators to Binary values that have been queued to be written. A "locator" is a ticket without an encoded length.

    • m_pool

      protected FlashJournalRM.BufferPool m_pool
      The BufferPool of Buffer objects that are used to format data to append to the journal.

    • m_daemonPreparer

      protected FlashJournalRM.PreparerDaemon m_daemonPreparer
      The "rendering" thread that renders (copies and encrypts) the queued binaries into buffers to be written.

    • m_daemonWriter

      protected FlashJournalRM.WriterDaemon m_daemonWriter
      The "writing" thread that just does appends of prepared buffers to journal files.

  • Constructor Details

  • Method Details

    • stop

      public void stop()
      Hard-stop the controllable service. Use Controllable.shutdown() for normal service termination. Calling this method for a service that has already stopped has no effect.
      Specified by:
      stop in interface Controllable
      Overrides:
      stop in class AbstractJournalRM

    • getMaxTotalRam

      public long getMaxTotalRam()
      Determine the total amount, in bytes, of RAM that will be used for the Journal.

      This property is specific to a RamJournalRM implementation.

      Returns:
      the maximum number of bytes that will be allocated for Journal storage

    • isNioRam

      public boolean isNioRam()
      Determine if the journal will use direct buffers (NIO RAM) instead of on-heap buffers (byte arrays).

      This property is specific to a RamJournalRM implementation.

      Returns:
      true iff the journal is configured to use NIO "direct buffer" RAM

    • getMaxBacklogSize

      public int getMaxBacklogSize()
      Determine the maximum allowable size, in bytes, of the backlog; when the backlog reaches this level, writes are delayed until the backlog drops below its maximum.

      This property is specific to a FlashJournalRM implementation.

      Returns:
      the maximum allowable size of the backlog

    • getBacklogSize

      public int getBacklogSize()
      Determine the total size in bytes of the serialized values that have not yet been persisted to disk. This value is also referred to as the "backlog".

      This property is specific to a FlashJournalRM implementation.

      Returns:
      the number of bytes queued to be written

    • getBacklogCount

      public int getBacklogCount()
      Determine the number of serialized values that have not yet been persisted to disk.

      This property is specific to a FlashJournalRM implementation.

      Returns:
      the number of serialized values queued to be written

    • getBufferSize

      public int getBufferSize()
      Determine the size of the buffers used to write a chunk of data at a time to an underlying journal file.

      This property is specific to a FlashJournalRM implementation.

      Returns:
      the size, in bytes, of each buffer

    • getMaxPoolSize

      public long getMaxPoolSize()
      Determine the maximum size (in bytes) of the buffers that can be held by the pool. This is not a limit of how many buffers can be allocated, since some may be in use at any time, but rather how many will be held by the pool (i.e. recycled) as they are released.
      Returns:
      the maximum size in bytes of all of the buffers that the buffer pool can hold onto

    • getOldestActiveFileCreationTime

      public long getOldestActiveFileCreationTime()
      Description copied from interface: JournalMBean
      Determine the creation time of the oldest currently active FlashJournal file. Active files are appending, congested, full, or evacuating; garbage and discarded files are excluded. This lifetime value is not changed by JournalMBean.resetStatistics().
      Returns:
      the creation time in epoch milliseconds, or zero if there is no active file, or -1 if unsupported

    • getOldestActiveFileCreationSequenceNumber

      public long getOldestActiveFileCreationSequenceNumber()
      Description copied from interface: JournalMBean
      Determine the non-resetting creation sequence number assigned to the oldest currently active FlashJournal file. Active files are appending, congested, full, or evacuating; garbage and discarded files are excluded.
      Returns:
      the creation sequence number, or zero if there is no active file, or -1 if unsupported

    • getMostRecentFileCreationTime

      public long getMostRecentFileCreationTime()
      Description copied from interface: JournalMBean
      Determine the creation time of the most recently created FlashJournal file. This lifetime value is not changed by JournalMBean.resetStatistics().
      Returns:
      the creation time in epoch milliseconds, or zero if no file has been created, or -1 if unsupported

    • getMostRecentFileCreationSequenceNumber

      public long getMostRecentFileCreationSequenceNumber()
      Description copied from interface: JournalMBean
      Determine the most recently assigned, non-resetting FlashJournal file creation sequence number.
      Returns:
      the creation sequence number, or zero if no file has been created, or -1 if unsupported

    • getTotalFileCreationCount

      public long getTotalFileCreationCount()
      Description copied from interface: JournalMBean
      Determine the number of FlashJournal files successfully created since statistics were reset.
      Returns:
      the successful file creation count, or -1 if unsupported

    • getTotalFileCreationErrors

      public long getTotalFileCreationErrors()
      Description copied from interface: JournalMBean
      Determine the number of failed FlashJournal file or channel creation attempts since statistics were reset.
      Returns:
      the failed file creation count, or -1 if unsupported

    • getMostRecentFileDeletionTime

      public long getMostRecentFileDeletionTime()
      Description copied from interface: JournalMBean
      Determine the deletion time of the most recently deleted FlashJournal managed file. This lifetime value is not changed by JournalMBean.resetStatistics().
      Returns:
      the deletion time in epoch milliseconds, or zero if no file has been deleted, or -1 if unsupported

    • getTotalFileDeletionCount

      public long getTotalFileDeletionCount()
      Description copied from interface: JournalMBean
      Determine the number of managed FlashJournal files successfully deleted since statistics were reset.
      Returns:
      the successful file deletion count, or -1 if unsupported

    • getTotalFileDeletionErrors

      public long getTotalFileDeletionErrors()
      Description copied from interface: JournalMBean
      Determine the number of failed managed FlashJournal file deletion attempts since statistics were reset, including attempts for which File.delete() returned false.
      Returns:
      the failed file deletion count, or -1 if unsupported

    • getMostRecentCollectorScanCompletionTime

      public long getMostRecentCollectorScanCompletionTime()
      Description copied from interface: JournalMBean
      Determine when the FlashJournal collector most recently completed a scan and successfully finalized its results. This lifetime value is not changed by JournalMBean.resetStatistics().
      Returns:
      the scan completion time in epoch milliseconds, or zero if no scan has completed, or -1 if unsupported

    • getPoolSize

      public int getPoolSize()
      Determine the size (in bytes) of the buffers that are currently available in the pool. This is not a measurement of how many buffers are currently in use or how many have been allocated.

      This property is specific to a FlashJournalRM implementation.

      Returns:
      the total size in bytes of all of the buffers that are currently in the buffer pool

    • getHighFileCount

      public int getHighFileCount()
      Determine the high file count for this Journal. Compulsory compaction (GC) occurs when the journal file count reaches the high file count.

      This property is specific to a FlashJournalRM implementation.

      Returns:
      the high file count for this journal

    • getTotalWriteOperationCount

      public long getTotalWriteOperationCount()
      Description copied from interface: JournalMBean
      Determine the total number of physical FileChannel.write invocations attempted by the FlashJournal writer. A scatter/gather invocation counts as one operation regardless of the number of buffers.
      Returns:
      the total number of physical write operations

    • getTotalWriteBytes

      public long getTotalWriteBytes()
      Description copied from interface: JournalMBean
      Determine the total number of bytes reported written by physical FileChannel.write invocations, including recovery writes.
      Returns:
      the total number of bytes reported written

    • getTotalWriteTime

      public long getTotalWriteTime()
      Description copied from interface: JournalMBean
      Determine the cumulative time spent in physical FileChannel.write invocations.
      Returns:
      the cumulative write time in milliseconds

    • getMaximumWriteTime

      public long getMaximumWriteTime()
      Description copied from interface: JournalMBean
      Determine the longest physical FileChannel.write invocation.
      Returns:
      the maximum write time in milliseconds

    • getTotalForceCount

      public long getTotalForceCount()
      Description copied from interface: JournalMBean
      Determine the total number of physical FileChannel.force invocations attempted by the FlashJournal writer.
      Returns:
      the total number of force operations

    • getTotalForceTime

      public long getTotalForceTime()
      Description copied from interface: JournalMBean
      Determine the cumulative time spent in physical FileChannel.force invocations.
      Returns:
      the cumulative force time in milliseconds

    • getMaximumForceTime

      public long getMaximumForceTime()
      Description copied from interface: JournalMBean
      Determine the longest physical FileChannel.force invocation.
      Returns:
      the maximum force time in milliseconds

    • getTotalWriteErrors

      public long getTotalWriteErrors()
      Description copied from interface: JournalMBean
      Determine the total number of I/O exceptions thrown by physical write or force operations.
      Returns:
      the total number of write-path errors

    • getTotalFileWriteErrors

      public long getTotalFileWriteErrors()
      Description copied from interface: JournalMBean
      Determine the number of write errors thrown by physical FileChannel.write invocations.
      Returns:
      the total number of physical write errors

    • getTotalForceErrors

      public long getTotalForceErrors()
      Description copied from interface: JournalMBean
      Determine the number of write errors thrown by physical FileChannel.force invocations.
      Returns:
      the total number of force errors

    • getTotalWriteRetries

      public long getTotalWriteRetries()
      Description copied from interface: JournalMBean
      Determine the number of recovery attempts made after a physical write or force operation failed.
      Returns:
      the total number of write retries

    • getTotalCongestionCount

      public long getTotalCongestionCount()
      Description copied from interface: JournalMBean
      Determine the number of FlashJournal congestion episodes. A congestion episode begins when the journal stops accepting new values because its write backlog exceeded the configured maximum.
      Returns:
      the total number of congestion episodes

    • getTotalCongestionTime

      public long getTotalCongestionTime()
      Description copied from interface: JournalMBean
      Determine the cumulative wall-clock time during which the FlashJournal was unable to accept new values because of write-backlog congestion. Concurrent producers blocked by the same congestion episode do not multiply this value.
      Returns:
      the cumulative congestion time in milliseconds

    • getMaximumCongestionTime

      public long getMaximumCongestionTime()
      Description copied from interface: JournalMBean
      Determine the longest continuous FlashJournal congestion episode.
      Returns:
      the maximum congestion time in milliseconds

    • getCurrentCongestionTime

      public long getCurrentCongestionTime()
      Description copied from interface: JournalMBean
      Determine the elapsed time of the current FlashJournal congestion episode, or zero if the journal is not congested.
      Returns:
      the current congestion time in milliseconds

    • resetStatistics

      public void resetStatistics()
      Description copied from class: AbstractJournalRM
      Reset the journal statistics.
      Specified by:
      resetStatistics in interface JournalMBean
      Overrides:
      resetStatistics in class AbstractJournalRM

    • findOldestActiveJournalFile

      protected FlashJournalRM.JournalFile findOldestActiveJournalFile()
      Return the oldest journal file that is still active. Appending, congested, full, and evacuating files are active; garbage and discarded files are not.
      Returns:
      the oldest active journal file, or null
      Since:
      26.1.0.0

    • onCollectorScanCompleted

      protected void onCollectorScanCompleted()
      Description copied from class: AbstractJournalRM
      Called after the collector completes a scan and publishes its results.
      Overrides:
      onCollectorScanCompleted in class AbstractJournalRM

    • getDirectoryFile

      public File getDirectoryFile()
      Determine the directory used to store the underlying disk files for the journal.
      Returns:
      the directory that is used to hold the journal files, or null if the default temporary directory is used

    • getPurgeDelayMillis

      public long getPurgeDelayMillis()
      Return the delay since the last touch of a tmp file until it is eligible for purging. Tmp files are purged at startup of the FlashJournalRM to clean up tmp files from a previous instance of the FlashJournalRM.
      Returns:
      the delay in milliseconds

    • getName

      protected String getName()
      Returns the name value to be used in MBean Objectname key "name".
      Specified by:
      getName in class AbstractJournalRM
      Returns:
      the name

    • getWriterTimeout

      protected long getWriterTimeout()
      Return the number of milliseconds the Writer daemon can be unresponsive prior to considering it timed out.
      Returns:
      the number of milliseconds the Writer can be unresponsive

    • getDependencies

      public FlashJournalRM.Dependencies getDependencies()
      Return the Dependencies object.
      Overrides:
      getDependencies in class AbstractJournalRM
      Returns:
      the journal dependencies object

    • cloneDependencies

      public FlashJournalRM.DefaultDependencies cloneDependencies(AbstractJournalRM.Dependencies deps)
      Clone the Dependencies object.
      Specified by:
      cloneDependencies in class AbstractJournalRM
      Parameters:
      deps - the populated Dependencies object
      Returns:
      the cloned Dependencies object

    • onDependencies

      protected void onDependencies(AbstractJournalRM.Dependencies deps)
      Process the Dependencies after they have been injected into this object.
      Overrides:
      onDependencies in class AbstractJournalRM
      Parameters:
      deps - the populated Dependencies object

    • startThreads

      protected void startThreads()
      Create and start the various threads used by this Journal Resource Manager.
      Overrides:
      startThreads in class AbstractJournalRM

    • purgeOldTempFiles

      protected void purgeOldTempFiles()
      Delete any old temp files that weren't cleaned up properly by a previous are configured to be "deleted on exit", it is possible that the temp files would not have been deleted if the JVM crashed or if there was a fault at the physical machine level.

    • stopThreads

      protected void stopThreads()
      Stop the various threads used by this Journal Resource Manager.
      Overrides:
      stopThreads in class AbstractJournalRM

    • calculateCurrentLoadFactor

      protected double calculateCurrentLoadFactor()
      Calculate current load factor for compaction (GC).
      Overrides:
      calculateCurrentLoadFactor in class AbstractJournalRM
      Returns:
      calculated load factor

    • shouldNotifyCollector

      protected boolean shouldNotifyCollector()
      Check if the Collector daemon needs to be notified to come out of sleep
      Overrides:
      shouldNotifyCollector in class AbstractJournalRM
      Returns:
      true if collector daemon should be notified

    • getDescription

      protected String getDescription()
      Format the object attributes into a String for inclusion in the String returned from the AbstractJournalRM.toString() method.
      Overrides:
      getDescription in class AbstractJournalRM
      Returns:
      a comma-delimited String listing the attributes of this object in the form "attribute=value"

    • getJournalFile

      protected FlashJournalRM.JournalFile getJournalFile(int nFileId)
      Obtain a JournalFile by its ID.
      Overrides:
      getJournalFile in class AbstractJournalRM
      Parameters:
      nFileId - the file id (0-511)
      Returns:
      the JournalFile for that file id, or null if there is currently no JournalFile for that file id

    • instantiateJournalFile

      protected FlashJournalRM.JournalFile instantiateJournalFile(int nFile)
      Factory: Instantiate a JournalFile or subclass thereof.
      Specified by:
      instantiateJournalFile in class AbstractJournalRM
      Parameters:
      nFile - the file number in the range 0-511 (inclusive)
      Returns:
      a new JournalFile instance

    • createTemporaryJournalFile

      protected File createTemporaryJournalFile() throws IOException
      Create the temporary file underlying a FlashJournal file.
      Returns:
      the temporary file
      Throws:
      IOException - if the file cannot be created
      Since:
      26.1.0.0

    • deleteJournalFile

      protected boolean deleteJournalFile(File file)
      Delete the file underlying a FlashJournal file.
      Parameters:
      file - the file to delete
      Returns:
      true if the file was deleted
      Since:
      26.1.0.0

    • claimJournalFileDeletion

      protected boolean claimJournalFileDeletion(FlashJournalRM.JournalFile file)
      Attempt to claim exclusive ownership of deleting the file underlying a FlashJournal file.
      Parameters:
      file - the journal file
      Returns:
      true if deletion ownership was acquired
      Since:
      26.1.0.0

    • instantiateBufferPool

      protected FlashJournalRM.BufferPool instantiateBufferPool()
      Factory: Instantiate a pool of buffers.
      Returns:
      a BufferPool or subclass thereof

    • getBufferPool

      protected FlashJournalRM.BufferPool getBufferPool()
      Obtain the BufferPool instance.
      Returns:
      the BufferPool for this FlashJournalRM

    • instantiatePreparerDaemon

      protected FlashJournalRM.PreparerDaemon instantiatePreparerDaemon()
      Factory: Instantiate a PreparerDaemon.
      Returns:
      a PreparerDaemon or a sub-class thereof

    • getPreparerDaemon

      protected FlashJournalRM.PreparerDaemon getPreparerDaemon()
      Obtain the PreparerDaemon instance.
      Returns:
      the PreparerDaemon for this FlashJournalRM

    • instantiateWriterDaemon

      protected FlashJournalRM.WriterDaemon instantiateWriterDaemon()
      Factory: Instantiate a WriterDaemon.
      Returns:
      a WriterDaemon or a sub-class thereof

    • getWriterDaemon

      protected FlashJournalRM.WriterDaemon getWriterDaemon()
      Obtain the WriterDaemon instance.
      Returns:
      the WriterDaemon for this FlashJournalRM

    • getWriterStatistics

      protected FlashJournalRM.WriterStatistics getWriterStatistics()
      Obtain the current writer statistics.
      Returns:
      the current writer statistics
      Since:
      26.1.0.0

    • getEvacuationMask

      protected long getEvacuationMask()
      Obtain the bit mask that is used to identify tickets that need to be evacuated for a particular file id.
      Specified by:
      getEvacuationMask in class AbstractJournalRM
      Returns:
      the bit mask for the file index within a ticket and whatever other bits are necessary to ensure that a ticket refers to that file

    • extractFileId

      protected int extractFileId(long lTicket)
      Extract a file index (a file ID) from the passed ticket.
      Specified by:
      extractFileId in class AbstractJournalRM
      Parameters:
      lTicket - a "ticket" that was created by the Journal
      Returns:
      the file index for the value represented by that ticket

    • extractOffset

      protected long extractOffset(long lTicket)
      Extract a file offset from the passed ticket.
      Specified by:
      extractOffset in class AbstractJournalRM
      Parameters:
      lTicket - a "ticket" that was created by the Journal
      Returns:
      the file offset for the value represented by that ticket

    • extractLength

      protected int extractLength(long lTicket)
      Extract a file offset from the passed ticket.
      Specified by:
      extractLength in class AbstractJournalRM
      Parameters:
      lTicket - a "ticket" that was created by the Journal
      Returns:
      the file offset for the value represented by that ticket

    • encodeTicket

      protected long encodeTicket(int nFile, long of, int cb)
      Encode a file number, offset and length into a "ticket".
      Specified by:
      encodeTicket in class AbstractJournalRM
      Parameters:
      nFile - a file number (aka a file index)
      of - a value offset
      cb - a value length
      Returns:
      a "ticket" containing the file number, offset and length