Class FlashJournalRM
- java.lang.Object
-
- com.tangosol.io.journal.AbstractJournalRM
-
- com.tangosol.io.journal.FlashJournalRM
-
- All Implemented Interfaces:
Disposable
,BinaryStoreManager
,ClassLoaderAware
,JournalMBean
,Controllable
,AutoCloseable
public class FlashJournalRM extends AbstractJournalRM
A FlashJournalRM manages disk I/O for temporary journal-based files, and acts as a shared resource for any number of journals that share a common configuration.To use the Resource Manager, configure it using Dependency Injection by passing in the populated dependency object at instantiation. Once configured, start the Resource Manager via the
AbstractJournalRM.start()
method, and then obtainBinaryStore
instances via theAbstractJournalRM.createBinaryStore()
method. When a BinaryStore instance is no longer required, dispose of it via theDisposable
interface, and similarlystop()
orAbstractJournalRM.dispose()
of the Resource Manager when it is no longer needed to ensure that any resources it allocated are cleaned up and released.While the Resource Manager is running, there will be one or more files created in the specified temporary directory (or the default temporary directory if none is specified). The number of files is not related to the number of BinaryStore instances, but rather to the total amount of data that the Resource Manager is managing at any given time. Additionally, the Resource Manager uses three daemon threads to help manage the data: "Journal-Preparer", "Journal-Writer" and "Journal-Collector". These threads respectively lay out the data to be written, write the data, and perform asynchronous GC of the storage files.
The limits on the journal are as follows:
- Values are limited by default to 64MB;
- An individual file is limited by default to 2GB (and maximum 4GB);
- A journal is composed of up to 512 files;
- A journal is thus limited by default to 1TB, with a theoretical maximum of 2TB
- Since:
- Coherence 3.7
- Author:
- cp/cf 2010-06-10
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected static class
FlashJournalRM.Buffer
A Buffer is a handy shell with some helper methods around a byte[] and a ByteBuffer.protected static class
FlashJournalRM.BufferPool
The BufferPool class recycles up to a maximum number of Buffer objects.protected static class
FlashJournalRM.CongestionPolicy
A CongestionPolicy defines how to behave when the currently activeFlashJournalRM.JournalFile
becomes congested.static class
FlashJournalRM.DefaultDependencies
The DefaultDependencies class provides a default implementation of Dependencies.static interface
FlashJournalRM.Dependencies
The Dependencies interface provides a FlashJournalRM object with its external dependencies.protected static class
FlashJournalRM.FailCongestionPolicy
AFlashJournalRM.CongestionPolicy
that fails, by throwing an unchecked exception, whenFlashJournalRM.FailCongestionPolicy.onCongestion(com.tangosol.io.journal.FlashJournalRM.JournalFile)
is invoked.protected class
FlashJournalRM.JournalFile
A JournalFile is created for each OS file used to store the journal contents.protected class
FlashJournalRM.PreparerDaemon
The PreparerDaemon takes the enqueued Binary values for the JournalFile objects, and lays them out in Buffer objects from the BufferPool, enqueuing them for the WriterDaemon to write.protected static class
FlashJournalRM.WaitCongestionPolicy
AFlashJournalRM.CongestionPolicy
which blocks the current thread on theFlashJournalRM.JournalFile
's monitor until awoken.protected class
FlashJournalRM.WriterDaemon
The WriterDaemon takes the enqueued Buffer objects from the PreparerDaemon, and appends them to a JournalFile.-
Nested classes/interfaces inherited from class com.tangosol.io.journal.AbstractJournalRM
AbstractJournalRM.CollectorDaemon, AbstractJournalRM.JournalImpl
-
-
Field Summary
Fields Modifier and Type Field Description protected static FlashJournalRM.FailCongestionPolicy
FAIL_CONGESTION_ACTION
AFlashJournalRM.CongestionPolicy
which fails, by throwing an unchecked exception, when the file is deemed congested.protected AtomicInteger
m_cbBacklog
Total amount of data enqueued to be written.protected FlashJournalRM.CongestionPolicy
m_congestionPolicy
TheFlashJournalRM.CongestionPolicy
to be used when in a congested state.protected FlashJournalRM.PreparerDaemon
m_daemonPreparer
The "rendering" thread that renders (copies and encrypts) the queued binaries into buffers to be written.protected FlashJournalRM.WriterDaemon
m_daemonWriter
The "writing" thread that just does appends of prepared buffers to journal files.protected File
m_dirTemp
Directory to store the temporary files.protected boolean
m_fLoggedDirectory
Set to true after the directory location has been logged.protected ConcurrentMap<Long,Binary>
m_mapLocators
Map of locators to Binary values that have been queued to be written.protected FlashJournalRM.BufferPool
m_pool
The BufferPool of Buffer objects that are used to format data to append to the journal.protected static long
MASK_FILE_ID
The 64-bit bitmask for the file index.protected static long
MASK_LENGTH
The 64-bit bitmask for the value length.protected static long
MASK_OFFSET
The 64-bit bitmask for the value offset.protected static int
SHIFT_FILE_ID
The location of the file id within the ticket.protected static int
SHIFT_LENGTH
The location of the value-length within the ticket.protected static int
SHIFT_OFFSET
The location of the value-offset within the ticket.protected static int
SHIFT_OFFSET_RAW
From an encoding point of view, the raw offset is shifted four bits less than necessary, because the four least significant bits of the offset are always zero (due to paragraph alignment).protected static String
TYPE_NAME
A simple name to be used as part of MBean ObjectNames.protected static FlashJournalRM.WaitCongestionPolicy
WAIT_CONGESTION_ACTION
The default congestion policy which blocks the client thread until awoken by theFlashJournalRM.WriterDaemon
.-
Fields inherited from class com.tangosol.io.journal.AbstractJournalRM
DEDUPE_ENABLED, m_ajournalfile, m_cJournalFiles, m_daemonCollector, m_dblFileSizeFactor, m_dependencies, m_dflHighestLoadFactor, m_jrnlfile, m_loader, m_nState, m_setJournals, MASK_COMPACT_FLAG, MASK_COMPACT_LENGTH, SHIFT_COMPACT_FLAG, SHIFT_COMPACT_LENGTH, STATE_CONFIGURED, STATE_INITIAL, STATE_RUNNING, STATE_STOPPED, STATE_STOPPING
-
-
Constructor Summary
Constructors Constructor Description FlashJournalRM(Cluster cluster, FlashJournalRM.Dependencies deps)
Construct a FlashJournalRM.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected double
calculateCurrentLoadFactor()
Calculate current load factor for compaction (GC).FlashJournalRM.DefaultDependencies
cloneDependencies(AbstractJournalRM.Dependencies deps)
Clone the Dependencies object.protected long
encodeTicket(int nFile, long of, int cb)
Encode a file number, offset and length into a "ticket".protected int
extractFileId(long lTicket)
Extract a file index (a file ID) from the passed ticket.protected int
extractLength(long lTicket)
Extract a file offset from the passed ticket.protected long
extractOffset(long lTicket)
Extract a file offset from the passed ticket.int
getBacklogCount()
Determine the number of serialized values that have not yet been persisted to disk.int
getBacklogSize()
Determine the total size in bytes of the serialized values that have not yet been persisted to disk.protected FlashJournalRM.BufferPool
getBufferPool()
Obtain the BufferPool instance.int
getBufferSize()
Determine the size of the buffers used to write a chunk of data at a time to an underlying journal file.FlashJournalRM.Dependencies
getDependencies()
Return the Dependencies object.protected String
getDescription()
Format the object attributes into a String for inclusion in the String returned from theAbstractJournalRM.toString()
method.File
getDirectoryFile()
Determine the directory used to store the underlying disk files for the journal.protected long
getEvacuationMask()
Obtain the bit mask that is used to identify tickets that need to be evacuated for a particular file id.int
getHighFileCount()
Determine the high file count for this Journal.protected FlashJournalRM.JournalFile
getJournalFile(int nFileId)
Obtain a JournalFile by its ID.int
getMaxBacklogSize()
Determine the maximum allowable size, in bytes, of the backlog; when the backlog reaches this level, writes are delayed until the backlog drops below its maximum.long
getMaxPoolSize()
Determine the maximum size (in bytes) of the buffers that can be held by the pool.long
getMaxTotalRam()
Determine the total amount, in bytes, of RAM that will be used for the Journal.protected String
getName()
Returns the name value to be used in MBean Objectname key "name".int
getPoolSize()
Determine the size (in bytes) of the buffers that are currently available in the pool.protected FlashJournalRM.PreparerDaemon
getPreparerDaemon()
Obtain the PreparerDaemon instance.long
getPurgeDelayMillis()
Return the delay since the last touch of a tmp file until it is eligible for purging.protected FlashJournalRM.WriterDaemon
getWriterDaemon()
Obtain the WriterDaemon instance.protected long
getWriterTimeout()
Return the number of milliseconds the Writer daemon can be unresponsive prior to considering it timed out.protected FlashJournalRM.BufferPool
instantiateBufferPool()
Factory: Instantiate a pool of buffers.protected FlashJournalRM.JournalFile
instantiateJournalFile(int nFile)
Factory: Instantiate a JournalFile or subclass thereof.protected FlashJournalRM.PreparerDaemon
instantiatePreparerDaemon()
Factory: Instantiate a PreparerDaemon.protected FlashJournalRM.WriterDaemon
instantiateWriterDaemon()
Factory: Instantiate a WriterDaemon.boolean
isNioRam()
Determine if the journal will use direct buffers (NIO RAM) instead of on-heap buffers (byte arrays).protected void
onDependencies(AbstractJournalRM.Dependencies deps)
Process the Dependencies after they have been injected into this object.protected void
purgeOldTempFiles()
Delete any old temp files that weren't cleaned up properly by a previous are configured to be "deleted on exit", it is possible that the temp files would not have been deleted if the JVM crashed or if there was a fault at the physical machine level.protected boolean
shouldNotifyCollector()
Check if the Collector daemon needs to be notified to come out of sleepprotected void
startThreads()
Create and start the various threads used by this Journal Resource Manager.void
stop()
Hard-stop the controllable service.protected void
stopThreads()
Stop the various threads used by this Journal Resource Manager.-
Methods inherited from class com.tangosol.io.journal.AbstractJournalRM
close, compact, configure, createBinaryStore, createJournal, destroyBinaryStore, dispose, encodeTicket, ensureCurrentJournalFile, extractBinary, getBinaryStoreCount, getCluster, getCollectorDaemon, getCollectorLoadFactor, getCollectorTimeout, getCompactionCount, getContextClassLoader, getCurrentCollectorLoadFactor, getExhaustiveCompactionCount, getExhaustiveCompactionTime, getFileCount, getGuardian, getHighestLoadFactor, getMaxCollectorSleepMillis, getMaxFileSize, getMaxJournalFiles, getMaxJournalFilesNumber, getMaxJournalSize, getMaxValueSize, getMinCollectorSleepMillis, getState, getStateDescription, getSurvivorFile, getTicketDescription, getTotalDataSize, getTotalFileSize, instantiateCollectorDaemon, instantiateJournal, isCompact, isDedupEnabled, isRunning, isSingleEvacuation, iterateJournalFiles, iterateJournals, registerJournal, registerMBean, resetStatistics, setContextClassLoader, setState, shutdown, start, toString, unregisterJournal, unregisterMBean
-
-
-
-
Field Detail
-
SHIFT_FILE_ID
protected static final int SHIFT_FILE_ID
The location of the file id within the ticket.- See Also:
- Constant Field Values
-
MASK_FILE_ID
protected static final long MASK_FILE_ID
The 64-bit bitmask for the file index. The 9 bits from 62-54.- See Also:
- Constant Field Values
-
SHIFT_OFFSET
protected static final int SHIFT_OFFSET
The location of the value-offset within the ticket.- See Also:
- Constant Field Values
-
SHIFT_OFFSET_RAW
protected static final int SHIFT_OFFSET_RAW
From an encoding point of view, the raw offset is shifted four bits less than necessary, because the four least significant bits of the offset are always zero (due to paragraph alignment).- See Also:
- Constant Field Values
-
MASK_OFFSET
protected static final long MASK_OFFSET
The 64-bit bitmask for the value offset. The 28 bits 53-26.Note that the least significant four bits of the offset MUST ALWAYS be zero, because they are NOT encoded into the ticket -- hence the 16-byte paragraph boundary for entries.
- See Also:
- Constant Field Values
-
SHIFT_LENGTH
protected static final int SHIFT_LENGTH
The location of the value-length within the ticket.- See Also:
- Constant Field Values
-
MASK_LENGTH
protected static final long MASK_LENGTH
The 64-bit bitmask for the value length. The 26 bits 25-0.- See Also:
- Constant Field Values
-
TYPE_NAME
protected static final String TYPE_NAME
A simple name to be used as part of MBean ObjectNames.- See Also:
- Constant Field Values
-
WAIT_CONGESTION_ACTION
protected static final FlashJournalRM.WaitCongestionPolicy WAIT_CONGESTION_ACTION
The default congestion policy which blocks the client thread until awoken by theFlashJournalRM.WriterDaemon
.
-
FAIL_CONGESTION_ACTION
protected static final FlashJournalRM.FailCongestionPolicy FAIL_CONGESTION_ACTION
AFlashJournalRM.CongestionPolicy
which fails, by throwing an unchecked exception, when the file is deemed congested.
-
m_congestionPolicy
protected volatile FlashJournalRM.CongestionPolicy m_congestionPolicy
TheFlashJournalRM.CongestionPolicy
to be used when in a congested state.
-
m_dirTemp
protected File m_dirTemp
Directory to store the temporary files.
-
m_fLoggedDirectory
protected boolean m_fLoggedDirectory
Set to true after the directory location has been logged.
-
m_cbBacklog
protected AtomicInteger m_cbBacklog
Total amount of data enqueued to be written.
-
m_mapLocators
protected ConcurrentMap<Long,Binary> m_mapLocators
Map of locators to Binary values that have been queued to be written. A "locator" is a ticket without an encoded length.
-
m_pool
protected FlashJournalRM.BufferPool m_pool
The BufferPool of Buffer objects that are used to format data to append to the journal.
-
m_daemonPreparer
protected FlashJournalRM.PreparerDaemon m_daemonPreparer
The "rendering" thread that renders (copies and encrypts) the queued binaries into buffers to be written.
-
m_daemonWriter
protected FlashJournalRM.WriterDaemon m_daemonWriter
The "writing" thread that just does appends of prepared buffers to journal files.
-
-
Constructor Detail
-
FlashJournalRM
public FlashJournalRM(Cluster cluster, FlashJournalRM.Dependencies deps)
Construct a FlashJournalRM.- Parameters:
cluster
- theCluster
for which theFlashJournalRM
is storing datadeps
- the Dependencies object
-
-
Method Detail
-
stop
public void stop()
Hard-stop the controllable service. UseControllable.shutdown()
for normal service termination. Calling this method for a service that has already stopped has no effect.- Specified by:
stop
in interfaceControllable
- Overrides:
stop
in classAbstractJournalRM
-
getMaxTotalRam
public long getMaxTotalRam()
Determine the total amount, in bytes, of RAM that will be used for the Journal.This property is specific to a
RamJournalRM
implementation.- Returns:
- the maximum number of bytes that will be allocated for Journal storage
-
isNioRam
public boolean isNioRam()
Determine if the journal will use direct buffers (NIO RAM) instead of on-heap buffers (byte arrays).This property is specific to a
RamJournalRM
implementation.- Returns:
- true iff the journal is configured to use NIO "direct buffer" RAM
-
getMaxBacklogSize
public int getMaxBacklogSize()
Determine the maximum allowable size, in bytes, of the backlog; when the backlog reaches this level, writes are delayed until the backlog drops below its maximum.This property is specific to a
FlashJournalRM
implementation.- Returns:
- the maximum allowable size of the backlog
-
getBacklogSize
public int getBacklogSize()
Determine the total size in bytes of the serialized values that have not yet been persisted to disk. This value is also referred to as the "backlog".This property is specific to a
FlashJournalRM
implementation.- Returns:
- the number of bytes queued to be written
-
getBacklogCount
public int getBacklogCount()
Determine the number of serialized values that have not yet been persisted to disk.This property is specific to a
FlashJournalRM
implementation.- Returns:
- the number of serialized values queued to be written
-
getBufferSize
public int getBufferSize()
Determine the size of the buffers used to write a chunk of data at a time to an underlying journal file.This property is specific to a
FlashJournalRM
implementation.- Returns:
- the size, in bytes, of each buffer
-
getMaxPoolSize
public long getMaxPoolSize()
Determine the maximum size (in bytes) of the buffers that can be held by the pool. This is not a limit of how many buffers can be allocated, since some may be in use at any time, but rather how many will be held by the pool (i.e. recycled) as they are released.- Returns:
- the maximum size in bytes of all of the buffers that the buffer pool can hold onto
-
getPoolSize
public int getPoolSize()
Determine the size (in bytes) of the buffers that are currently available in the pool. This is not a measurement of how many buffers are currently in use or how many have been allocated.This property is specific to a
FlashJournalRM
implementation.- Returns:
- the total size in bytes of all of the buffers that are currently in the buffer pool
-
getHighFileCount
public int getHighFileCount()
Determine the high file count for this Journal. Compulsory compaction (GC) occurs when the journal file count reaches the high file count.This property is specific to a
FlashJournalRM
implementation.- Returns:
- the high file count for this journal
-
getDirectoryFile
public File getDirectoryFile()
Determine the directory used to store the underlying disk files for the journal.- Returns:
- the directory that is used to hold the journal files, or null if the default temporary directory is used
-
getPurgeDelayMillis
public long getPurgeDelayMillis()
Return the delay since the last touch of a tmp file until it is eligible for purging. Tmp files are purged at startup of the FlashJournalRM to clean up tmp files from a previous instance of the FlashJournalRM.- Returns:
- the delay in milliseconds
-
getName
protected String getName()
Returns the name value to be used in MBean Objectname key "name".- Specified by:
getName
in classAbstractJournalRM
- Returns:
- the name
-
getWriterTimeout
protected long getWriterTimeout()
Return the number of milliseconds the Writer daemon can be unresponsive prior to considering it timed out.- Returns:
- the number of milliseconds the Writer can be unresponsive
-
getDependencies
public FlashJournalRM.Dependencies getDependencies()
Return the Dependencies object.- Overrides:
getDependencies
in classAbstractJournalRM
- Returns:
- the journal dependencies object
-
cloneDependencies
public FlashJournalRM.DefaultDependencies cloneDependencies(AbstractJournalRM.Dependencies deps)
Clone the Dependencies object.- Specified by:
cloneDependencies
in classAbstractJournalRM
- Parameters:
deps
- the populated Dependencies object- Returns:
- the cloned Dependencies object
-
onDependencies
protected void onDependencies(AbstractJournalRM.Dependencies deps)
Process the Dependencies after they have been injected into this object.- Overrides:
onDependencies
in classAbstractJournalRM
- Parameters:
deps
- the populated Dependencies object
-
startThreads
protected void startThreads()
Create and start the various threads used by this Journal Resource Manager.- Overrides:
startThreads
in classAbstractJournalRM
-
purgeOldTempFiles
protected void purgeOldTempFiles()
Delete any old temp files that weren't cleaned up properly by a previous are configured to be "deleted on exit", it is possible that the temp files would not have been deleted if the JVM crashed or if there was a fault at the physical machine level.
-
stopThreads
protected void stopThreads()
Stop the various threads used by this Journal Resource Manager.- Overrides:
stopThreads
in classAbstractJournalRM
-
calculateCurrentLoadFactor
protected double calculateCurrentLoadFactor()
Calculate current load factor for compaction (GC).- Overrides:
calculateCurrentLoadFactor
in classAbstractJournalRM
- Returns:
- calculated load factor
-
shouldNotifyCollector
protected boolean shouldNotifyCollector()
Check if the Collector daemon needs to be notified to come out of sleep- Overrides:
shouldNotifyCollector
in classAbstractJournalRM
- Returns:
- true if collector daemon should be notified
-
getDescription
protected String getDescription()
Format the object attributes into a String for inclusion in the String returned from theAbstractJournalRM.toString()
method.- Overrides:
getDescription
in classAbstractJournalRM
- Returns:
- a comma-delimited String listing the attributes of this object in the form "attribute=value"
-
getJournalFile
protected FlashJournalRM.JournalFile getJournalFile(int nFileId)
Obtain a JournalFile by its ID.- Overrides:
getJournalFile
in classAbstractJournalRM
- Parameters:
nFileId
- the file id (0-511)- Returns:
- the JournalFile for that file id, or null if there is currently no JournalFile for that file id
-
instantiateJournalFile
protected FlashJournalRM.JournalFile instantiateJournalFile(int nFile)
Factory: Instantiate a JournalFile or subclass thereof.- Specified by:
instantiateJournalFile
in classAbstractJournalRM
- Parameters:
nFile
- the file number in the range 0-511 (inclusive)- Returns:
- a new JournalFile instance
-
instantiateBufferPool
protected FlashJournalRM.BufferPool instantiateBufferPool()
Factory: Instantiate a pool of buffers.- Returns:
- a BufferPool or subclass thereof
-
getBufferPool
protected FlashJournalRM.BufferPool getBufferPool()
Obtain the BufferPool instance.- Returns:
- the BufferPool for this FlashJournalRM
-
instantiatePreparerDaemon
protected FlashJournalRM.PreparerDaemon instantiatePreparerDaemon()
Factory: Instantiate a PreparerDaemon.- Returns:
- a PreparerDaemon or a sub-class thereof
-
getPreparerDaemon
protected FlashJournalRM.PreparerDaemon getPreparerDaemon()
Obtain the PreparerDaemon instance.- Returns:
- the PreparerDaemon for this FlashJournalRM
-
instantiateWriterDaemon
protected FlashJournalRM.WriterDaemon instantiateWriterDaemon()
Factory: Instantiate a WriterDaemon.- Returns:
- a WriterDaemon or a sub-class thereof
-
getWriterDaemon
protected FlashJournalRM.WriterDaemon getWriterDaemon()
Obtain the WriterDaemon instance.- Returns:
- the WriterDaemon for this FlashJournalRM
-
getEvacuationMask
protected long getEvacuationMask()
Obtain the bit mask that is used to identify tickets that need to be evacuated for a particular file id.- Specified by:
getEvacuationMask
in classAbstractJournalRM
- Returns:
- the the bit mask for the file index within a ticket and whatever other bits are necessary to ensure that a ticket refers to that file
-
extractFileId
protected int extractFileId(long lTicket)
Extract a file index (a file ID) from the passed ticket.- Specified by:
extractFileId
in classAbstractJournalRM
- Parameters:
lTicket
- a "ticket" that was created by the Journal- Returns:
- the file index for the value represented by that ticket
-
extractOffset
protected long extractOffset(long lTicket)
Extract a file offset from the passed ticket.- Specified by:
extractOffset
in classAbstractJournalRM
- Parameters:
lTicket
- a "ticket" that was created by the Journal- Returns:
- the file offset for the value represented by that ticket
-
extractLength
protected int extractLength(long lTicket)
Extract a file offset from the passed ticket.- Specified by:
extractLength
in classAbstractJournalRM
- Parameters:
lTicket
- a "ticket" that was created by the Journal- Returns:
- the file offset for the value represented by that ticket
-
encodeTicket
protected long encodeTicket(int nFile, long of, int cb)
Encode a file number, offset and length into a "ticket".- Specified by:
encodeTicket
in classAbstractJournalRM
- Parameters:
nFile
- a file number (aka a file index)of
- a value offsetcb
- a value length- Returns:
- a "ticket" containing the file number, offset and length
-
-