Package com.tangosol.io.journal

Class RamJournalRM

  • All Implemented Interfaces:
    Disposable, BinaryStoreManager, ClassLoaderAware, JournalMBean, Controllable, AutoCloseable
    public class RamJournalRM
extends AbstractJournalRM
    A RamJournalRM manages memory buffers for journal-based storage in memory, and acts as a shared resource for any number of journals that share a common configuration. While the RamJournalRM can be used by itself, it is intended to be used with a FlashJournalRM instance: So that large objects can be stored using flash; to allow for spill-over when the amount of total memory allocated to the RAM journal is used; and to handle the case when the journal "garbage collection" is temporarily not able to keep up with demand.

    To use the Resource Manager, configure it using Dependency Injection by passing in the populated dependency object at instantiation.. Once configured, start the Resource Manager via the AbstractJournalRM.start() method, and then obtain BinaryStore instances via the AbstractJournalRM.createBinaryStore() method. When a BinaryStore instance is no longer required, dispose of it via the Disposable interface, and similarly stop() or AbstractJournalRM.dispose() of the Resource Manager when it is no longer needed to ensure that any resources it allocated are cleaned up and released.

    The limits on the journal are as follows:

    • Values are limited by default to 16KB (and maximum 4MB);
    • An individual buffer (i.e. a journal "file") is limited by default to 2MB (and maximum 2GB);
    • A journal is composed of up to 512 files;
    • The total memory used by the journal is limited to 1GB by default (and maximum 64GB).

    Note that with a flash journal backing up the RAM journal, the limits have a different meaning:

    • Values that exceed the size limit will automatically be delegated to the flash journal;
    • All data that is attempted to be written when the total memory allocated to the RAM journal has been used will automatically be delegated to the flash journal.

    Note that, outside of the initial configuration, there is no difference in the use of the RAM journal with or without the flash journal; the combination of the RAM journal with the flash journal is transparent to the clients, with the only obvious difference being the significantly reduced possibility of an exception due to the RAM being exhausted.

    Since:
    Coherence 3.7
    Author:
    cp/cf 2010-06-24

    • Field Detail

      • SHIFT_RAM_FLAG

        protected static final int SHIFT_RAM_FLAG
        The location of the RAM flag within the ticket.
        See Also:
        Constant Field Values

      • MASK_RAM_FLAG

        protected static final long MASK_RAM_FLAG
        The 64-bit bitmask for the RAM flag.

        The RAM flag is the bit that signifies that the ticket was produced by the RAM Journal Resource Manager, as opposed to either a "real" compact ticket or a delegated flash ticket.

        See Also:
        Constant Field Values

      • SHIFT_FILE_ID

        protected static final int SHIFT_FILE_ID
        The location of the file id within the ticket.
        See Also:
        Constant Field Values

      • MASK_FILE_ID

        protected static final long MASK_FILE_ID
        The 64-bit bitmask for the file index. The 9 bits from 61-53.
        See Also:
        Constant Field Values

      • SHIFT_OFFSET

        protected static final int SHIFT_OFFSET
        The location of the value-offset within the ticket.
        See Also:
        Constant Field Values

      • MASK_OFFSET

        protected static final long MASK_OFFSET
        The 64-bit bitmask for the value offset. The 31 bits 52-22.
        See Also:
        Constant Field Values

      • SHIFT_LENGTH

        protected static final int SHIFT_LENGTH
        The location of the value-length within the ticket.
        See Also:
        Constant Field Values

      • MASK_LENGTH

        protected static final long MASK_LENGTH
        The 64-bit bitmask for the value length. The 22 bits 21-0.
        See Also:
        Constant Field Values

      • TYPE_NAME

        protected static final String TYPE_NAME
        A simple type name to be used as part of MBean Objectnames.
        See Also:
        Constant Field Values

      • m_jrnlrm

        protected FlashJournalRM m_jrnlrm
        A reference to a flash journal resource manager to use to read/write to/from flash for data that doesn't fit in memory.

      • m_flash

        protected RamJournalRM.FlashConsumer m_flash
        The conduit between this RAM journal resource manager and the flash journal resource manager.

    • Method Detail

      • getBacklogCount

        public int getBacklogCount()
        Determine the number of serialized values that have not yet been persisted to disk.

        This property is specific to a FlashJournalRM implementation.

        Returns:
        the number of serialized values queued to be written

      • getBacklogSize

        public int getBacklogSize()
        Determine the total size in bytes of the serialized values that have not yet been persisted to disk. This value is also referred to as the "backlog".

        This property is specific to a FlashJournalRM implementation.

        Returns:
        the number of bytes queued to be written

      • getMaxBacklogSize

        public int getMaxBacklogSize()
        Determine the maximum allowable size, in bytes, of the backlog; when the backlog reaches this level, writes are delayed until the backlog drops below its maximum.

        This property is specific to a FlashJournalRM implementation.

        Returns:
        the maximum allowable size of the backlog

      • getPoolSize

        public int getPoolSize()
        Determine the size (in bytes) of the buffers that are currently available in the pool. This is not a measurement of how many buffers are currently in use or how many have been allocated.

        This property is specific to a FlashJournalRM implementation.

        Returns:
        the total size in bytes of all of the buffers that are currently in the buffer pool

      • getMaxTotalRam

        public long getMaxTotalRam()
        Determine the total amount, in bytes, of RAM that will be used for the Journal.

        This property is specific to a RamJournalRM implementation.

        Returns:
        the maximum number of bytes that will be allocated for Journal storage

      • isNioRam

        public boolean isNioRam()
        Determine if the journal will use direct buffers (NIO RAM) instead of on-heap buffers (byte arrays).

        This property is specific to a RamJournalRM implementation.

        Returns:
        true iff the journal is configured to use NIO "direct buffer" RAM

      • getMaxPoolSize

        public long getMaxPoolSize()
        Determine the maximum size (in bytes) of the buffers that can be held by the pool. This is not a limit of how many buffers can be allocated, since some may be in use at any time, but rather how many will be held by the pool (i.e. recycled) as they are released.
        Returns:
        the maximum size in bytes of all of the buffers that the buffer pool can hold onto

      • getBufferSize

        public int getBufferSize()
        Determine the size of the buffers used to write a chunk of data at a time to an underlying journal file.

        This property is specific to a FlashJournalRM implementation.

        Returns:
        the size, in bytes, of each buffer

      • getHighFileCount

        public int getHighFileCount()
        Determine the high file count for this Journal. Compulsory compaction (GC) occurs when the journal file count reaches the high file count.

        This property is specific to a FlashJournalRM implementation.

        Returns:
        the high file count for this journal

      • getFlashJournalRM

        public FlashJournalRM getFlashJournalRM()
        Obtain the FlashJournalRM used to store large values or whatever values don't fit in the configured amount of RAM.
        Returns:
        the FlashJournalRM used by this RamJournalRM, or null if none

      • getName

        protected String getName()
        Returns the name value to be used in MBean Objectname key "name".
        Specified by:
        getName in class AbstractJournalRM
        Returns:
        the name

      • startThreads

        protected void startThreads()
        Create and start the various threads used by this Journal Resource Manager.
        Overrides:
        startThreads in class AbstractJournalRM

      • getDescription

        protected String getDescription()
        Format the object attributes into a String for inclusion in the String returned from the AbstractJournalRM.toString() method.
        Overrides:
        getDescription in class AbstractJournalRM
        Returns:
        a comma-delimited String listing the attributes of this object in the form "attribute=value"

      • getCollectorLoadFactor

        public double getCollectorLoadFactor()
        Determine the load factor threshold at which files become eligible for garbage collection (compaction). The load factor threshold is the ratio between the portion of the file used for live data and the total size of the file, and is expressed as a double value in the interval (0.0, 1.0).

        When the portion of released data exceeds (1 - threshold), the file becomes eligible for compaction (garbage collection).

        Specified by:
        getCollectorLoadFactor in interface JournalMBean
        Overrides:
        getCollectorLoadFactor in class AbstractJournalRM
        Returns:
        the compaction factor threshold between 0.0 and 1.0

      • shouldNotifyCollector

        protected boolean shouldNotifyCollector()
        Check if the Collector daemon needs to be notified to come out of sleep
        Overrides:
        shouldNotifyCollector in class AbstractJournalRM
        Returns:
        true if collector daemon should be notified

      • getMaxJournalFiles

        protected int getMaxJournalFiles()
        Determine the maximum number of journal files.
        Overrides:
        getMaxJournalFiles in class AbstractJournalRM
        Returns:
        the maximum number of simultaneous journal files that will be used to store the contents of this journal

      • getMinCollectorSleepMillis

        protected long getMinCollectorSleepMillis()
        Determine the minimum sleep cycle between collections.
        Overrides:
        getMinCollectorSleepMillis in class AbstractJournalRM
        Returns:
        the minimum number of milliseconds to sleep between collections

      • isDedupEnabled

        protected boolean isDedupEnabled()
        Determine if this journal resource manager should de-dup the keys that its JournalBinaryStore instances are managing.
        Overrides:
        isDedupEnabled in class AbstractJournalRM
        Returns:
        true iff this journal resource manager should routinely cause its JournalBinaryStore instances to de-dup their keys

      • isSingleEvacuation

        protected boolean isSingleEvacuation()
        Determine if this journal resource manager should only evacuate one single JournalFile (at the most) per collection cycle.
        Overrides:
        isSingleEvacuation in class AbstractJournalRM
        Returns:
        true iff this journal resource manager should only evacuate the emptiest JournalFile instead of all the JournalFile instances that qualify for evacuation

      • getFlashJournal

        protected Journal getFlashJournal()
        Obtain the Journal used to read/write flash.
        Returns:
        the flash Journal or null if there is none

      • getSurvivorFile

        protected AbstractJournalRM.JournalFile getSurvivorFile()
        Return a AbstractJournalRM.JournalFile based on the context of the calling thread or null if it is determined that an exhaustive evacuation will not reclaim sufficient memory. This method should only be invoked in a "depleted" state when:
        • the maximum number of files is used
        • all those files are non-appending
        When called on a client thread, any operation against this file will block until the full compaction is performed. There is no guarantee that compaction results in available memory thus an out of memory condition may still occur.

        When called on the CollectorDaemon thread this method returns a normal JournalFile, with the only exception that it will be in excess to the maximum number of permitted journal files and used exclusively by the CollectorDaemon to perform evacuation in a depleted state.

        Overrides:
        getSurvivorFile in class AbstractJournalRM
        Returns:
        either a JournalFile whose usage triggers a full collection or a JournalFile used exclusively by the CollectorDaemon. null may be returned if it is determined that an exhaustive evacuation will not reclaim sufficient memory

      • instantiateConsumer

        protected RamJournalRM.FlashConsumer instantiateConsumer()
        Factory: Instantiate a FlashConsumer object.
        Returns:
        an instance of FlashConsumer or a subclass thereof.

      • getFlashConsumer

        protected RamJournalRM.FlashConsumer getFlashConsumer()
        Obtain the FlashConsumer object used to read/write flash.
        Returns:
        the FlashConsumer instance, or null if there is no flash configured to use

      • getJournalFile

        protected RamJournalRM.JournalFile getJournalFile​(int nFileId)
        Obtain a JournalFile by its ID.
        Overrides:
        getJournalFile in class AbstractJournalRM
        Parameters:
        nFileId - the file id (0-511)
        Returns:
        the JournalFile for that file id, or null if there is currently no JournalFile for that file id

      • instantiateJournalFile

        protected RamJournalRM.JournalFile instantiateJournalFile​(int nFile)
        Factory: Instantiate a JournalFile or subclass thereof.
        Specified by:
        instantiateJournalFile in class AbstractJournalRM
        Parameters:
        nFile - the file number in the range 0-511 (inclusive)
        Returns:
        a new JournalFile instance

      • isCompact

        protected boolean isCompact​(long lTicket)
        Determine if the ticket is in the compact form.
        Overrides:
        isCompact in class AbstractJournalRM
        Parameters:
        lTicket - a "ticket" that was created by the Journal
        Returns:
        true iff the ticket is a compact ticket

      • isRam

        protected boolean isRam​(long lTicket)
        Determine if the ticket is in the RAM form.
        Parameters:
        lTicket - a "ticket" that was returned previously by the Journal
        Returns:
        true iff the ticket is a RAM ticket

      • isFlash

        protected boolean isFlash​(long lTicket)
        Determine if the ticket is neither in compact or RAM form.
        Parameters:
        lTicket - a "ticket" that was returned previously by the Journal
        Returns:
        true iff the ticket is from an underlying FlashJournalRM

      • getEvacuationMask

        protected long getEvacuationMask()
        Obtain the bit mask that is used to identify tickets that need to be evacuated for a particular file id.
        Specified by:
        getEvacuationMask in class AbstractJournalRM
        Returns:
        the the bit mask for the file index within a ticket and whatever other bits are necessary to ensure that a ticket refers to that file

      • extractFileId

        protected int extractFileId​(long lTicket)
        Extract a file index (a file ID) from the passed ticket.
        Specified by:
        extractFileId in class AbstractJournalRM
        Parameters:
        lTicket - a "ticket" that was created by the Journal
        Returns:
        the file index for the value represented by that ticket

      • extractOffset

        protected long extractOffset​(long lTicket)
        Extract a file offset from the passed ticket.
        Specified by:
        extractOffset in class AbstractJournalRM
        Parameters:
        lTicket - a "ticket" that was created by the Journal
        Returns:
        the file offset for the value represented by that ticket

      • extractLength

        protected int extractLength​(long lTicket)
        Extract a file offset from the passed ticket.
        Specified by:
        extractLength in class AbstractJournalRM
        Parameters:
        lTicket - a "ticket" that was created by the Journal
        Returns:
        the file offset for the value represented by that ticket

      • encodeTicket

        protected long encodeTicket​(int nFile,
                            long of,
                            int cb)
        Encode a file number, offset and length into a "ticket".
        Specified by:
        encodeTicket in class AbstractJournalRM
        Parameters:
        nFile - a file number (aka a file index)
        of - a value offset
        cb - a value length
        Returns:
        a "ticket" containing the file number, offset and length

      • getTicketDescription

        protected String getTicketDescription​(long lTicket)
        Format the information encoded in a ticket into a debug string.
        Overrides:
        getTicketDescription in class AbstractJournalRM
        Parameters:
        lTicket - the ticket value
        Returns:
        a description of the contents of the ticket value

      • instantiateBufferPool

        protected RamJournalRM.BufferPool instantiateBufferPool()
        Factory: Instantiate a pool of buffers.
        Returns:
        a BufferPool or subclass thereof

      • getBufferPool

        protected RamJournalRM.BufferPool getBufferPool()
        Obtain the BufferPool instance.
        Returns:
        the BufferPool for this FlashJournalRM