com.nt.udc.util

Class ObjectStorageFile

java.lang.Object

com.nt.udc.util.ObjectStorageFile

Direct Known Subclasses:

NARKeyStorageFile, NARStorageFile, TimeStampedRecordStorageFile

public abstract class ObjectStorageFile
extends java.lang.Object

This class implements the basics of file-based Object storage. For the most part, it is similar to a Hashtable class, except that the Object is not stored in memory - it is stored and accessed from disk. The Object must be able to convert to/from a byte[] object in order to use this class. It is highly recommended that this NOT be done with an ObjectStream because of the performance implications. Also, the key for a given Object is the offset at which it sits in the file. If you would like to use a different key, use the FileHashtable class with this class, as that class maintains an Object key -> offset table. You may also create your own key -> offset mapping through use of the FileMap interface.

This class can also restore an old table, as defined in the constructor.

Implementation:

Since the table is on disk, any new or updated Objects must be appended to the file. This causes the file to grow rather large, seeing that an old or outdated Object cannot be removed from the table. A "removed" or "outdated" Object is simply marked for later removal, during the "compression" stage. The "compression" takes place when a pre-defined number of "outdated" Objects have been reached. This number must be defined in the constructor, and is known as the compression threshold. Keep in mind that a lower compression threshold causing compression to happen more often, but results in the disk table being smaller. Likewise, a larger compression threshold causing a less frequent compression, thereby improving performance, but allowing a larger disk table.

This class must be extended to handle the specifics of the Objects it is storing.

Through use of the "setupWalker" and "walkFile" methods, a user may step through the file by valid Object. This means that all "outdated" Objects will be skipped. Note that the walker supports only one active walk at a time, and the user must use "setupWalker" to reset the walker. It will not reset on its own.

Field Summary

Fields

Modifier and Type	Field and Description
protected ObjectStorageAllocator	allocator
protected int	currentStorageSize
protected byte[]	headers
protected byte[]	headers_no_flag
protected static byte[]	invalid_flags
protected static int	INVALID_OBJ
protected static long	MAX_PARTITION_SIZE
protected long	maxPartitionSize
static int	OBJ_HEADER
protected java.io.RandomAccessFile	objectFile RandomAccessFile that actually stores the Objects.
protected boolean	partitioned
protected long	time_used
protected static byte[]	valid_flags
protected static int	VALID_OBJ

Constructor Summary

Constructors

Constructor and Description
ObjectStorageFile(int count, java.io.File fileName, boolean newFile) Constructor for the storage manager.

Method Summary

Modifier and Type	Method and Description
long	add(java.lang.Object obj) Adds a new Object to the table.
protected long	addBytes(byte[] newBytes, ObjectStorageInfo info) Adds a set of bytes to the file, and returns the offset where they are stored.
java.lang.String	allocatorReport()
void	close() Closes a storage file.
void	drop()
java.lang.Object	get(long offset) Get the Object at the given offset.
protected byte[]	getBytes(long offset) Returns the bytes at the given offset.
int	getNumObjects() Returns the number of Objects in the storage file.
java.io.File	getObjectFile() Returns the File object associated with this ObjectStorageFile.
java.lang.Object[]	getSomeObjects(int numObjects) Returns a given set of Objects that exist in the file.
protected long[]	getValidOffsets() Returns a list of offsets for all the valid Objects in the file.
boolean	isEmpty()
boolean	isNewFile() Returns whether this is a new storage file.
protected void	markDeletedBytes(long offset)
protected abstract java.lang.Object	objectFromBytes(byte[] bytes) This method should return an Object, as created by the given bytes.
protected abstract byte[]	objectToBytes(java.lang.Object obj) This method should return the Object in its byte[] format.
void	open() Opens a storage file.
java.lang.Object	remove(long offset, boolean bRetValue) Remove the Object at the given offset.
protected byte[]	removeBytes(long offset) Removes the bytes at the given offset, and returns them.
byte[]	removeDelayedParsing(long offset, boolean bRetValue) Same as remove(), except that objectFromBytes() is not called.
long[]	restoreFile() Rebuilds the storage file offsets from a existing table storage file.
void	setPartition(long partitionSize)
protected void	setupWalker() Sets up the file walker to start at the beginning of the file.
protected void	setupWalker(long offset) Sets up the file walker to start at the given offset.
java.lang.String	toString() Converts the storage file to String representation.
void	toString(java.io.OutputStream os) Writes the String representation of this storage file to the given OutputStream.
long	update(java.lang.Object obj, long offset) Update a prior Object with a new Object.
protected long	walkFile() Walks the file to the next valid Object, and returns its offset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

objectFile

protected java.io.RandomAccessFile objectFile

RandomAccessFile that actually stores the Objects.

OBJ_HEADER

public static final int OBJ_HEADER

See Also:

Constant Field Values

allocator

protected ObjectStorageAllocator allocator

currentStorageSize

protected int currentStorageSize

VALID_OBJ

protected static final int VALID_OBJ

See Also:

Constant Field Values

INVALID_OBJ

protected static final int INVALID_OBJ

See Also:

Constant Field Values

time_used

protected long time_used

MAX_PARTITION_SIZE

protected static final long MAX_PARTITION_SIZE

See Also:

Constant Field Values

partitioned

protected boolean partitioned

headers

protected byte[] headers

invalid_flags

protected static final byte[] invalid_flags

valid_flags

protected static final byte[] valid_flags

headers_no_flag

protected byte[] headers_no_flag

maxPartitionSize

protected long maxPartitionSize

Constructor Detail

ObjectStorageFile

public ObjectStorageFile(int count,
                         java.io.File fileName,
                         boolean newFile)

Constructor for the storage manager.

Parameters:

count - Compression threshold. (see class description)

fileName - File name to use for the table.

newFile - Is this file name for a new table, or should this restore an old table?

Method Detail

setPartition

public void setPartition(long partitionSize)

getObjectFile

public java.io.File getObjectFile()

Returns the File object associated with this ObjectStorageFile.

Returns:

File, representing this ObjectStorageFile

add

public long add(java.lang.Object obj)
         throws java.io.IOException

Adds a new Object to the table.

Parameters:

obj - Add this Object to the table.

Returns:

Offset at which this Object exists.

Throws:

java.io.IOException

get

public java.lang.Object get(long offset)
                     throws java.io.IOException

Get the Object at the given offset.

Parameters:

offset - to retrieve the Object.

Returns:

The desired Object.

Throws:

java.io.IOException

remove

public java.lang.Object remove(long offset,
                               boolean bRetValue)
                        throws java.io.IOException

Remove the Object at the given offset.

Parameters:

offset - Offset of the Object.

bRetValue - Whether it will return the removed object or not

Throws:

java.io.IOException

removeDelayedParsing

public byte[] removeDelayedParsing(long offset,
                                   boolean bRetValue)
                            throws java.io.IOException

Same as remove(), except that objectFromBytes() is not called. This is exclusively for use by multi-threaded EP and AP.

Parameters:

offset -

bRetValue -

Returns:

The byte[] that should have been passed to objectFromBytes() to convert to original object with.

Throws:

java.io.IOException

update

public long update(java.lang.Object obj,
                   long offset)
            throws java.io.IOException

Update a prior Object with a new Object.

Parameters:

obj - New Object.

offset - Offset of the old Object.

Returns:

Offset of the new Object.

Throws:

java.io.IOException

restoreFile

public long[] restoreFile()
                   throws java.io.IOException

Rebuilds the storage file offsets from a existing table storage file.

Returns:

An array of longs, representing the new offsets.

Throws:

java.io.IOException

open

public void open()
          throws java.io.IOException

Opens a storage file.

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Closes a storage file.

Throws:

java.io.IOException

drop

public void drop()

isNewFile

public boolean isNewFile()

Returns whether this is a new storage file.

Returns:

true, if this is using a new storage file

getNumObjects

public int getNumObjects()
                  throws java.io.IOException

Returns the number of Objects in the storage file.

Returns:

Number of Objects in the storage file.

Throws:

java.io.IOException

getValidOffsets

protected long[] getValidOffsets()
                          throws java.io.IOException

Returns a list of offsets for all the valid Objects in the file.

Returns:

array of valid offsets

Throws:

java.io.IOException

setupWalker

protected void setupWalker()
                    throws java.io.IOException

Sets up the file walker to start at the beginning of the file.

Throws:

java.io.IOException

setupWalker

protected void setupWalker(long offset)
                    throws java.io.IOException

Sets up the file walker to start at the given offset.

Throws:

java.io.IOException

walkFile

protected long walkFile()
                 throws java.io.IOException

Walks the file to the next valid Object, and returns its offset.

Returns:

offset of new valid Object, or -1 if no valid offset was found

Throws:

java.io.IOException

getSomeObjects

public java.lang.Object[] getSomeObjects(int numObjects)
                                  throws java.io.IOException

Returns a given set of Objects that exist in the file. Subsequent calls to this method will eventually return all the Objects in the file. Note that getting ALL the Objects at once, and storing them in memory, may cause "out of memory" issues. Therefore, it is recommended that you deal with the Objects a little bit at a time. When using this method, be sure to call it until ALL of the Objects have been retrieved. The method will only reset its pointer to the top of the file once all of the current Objects have been retrieved.

Parameters:

numObjects - Number of Objects to return.

Returns:

Set of Objects from the file. Null, if no more Objects exist.

Throws:

java.io.IOException

toString

public java.lang.String toString()

Converts the storage file to String representation.

Overrides:

toString in class java.lang.Object

Returns:

Storage table, in String representation.

toString

public void toString(java.io.OutputStream os)
              throws java.io.IOException

Writes the String representation of this storage file to the given OutputStream.

Parameters:

os - OutputStream to write to.

Throws:

java.io.IOException

allocatorReport

public java.lang.String allocatorReport()

addBytes

protected long addBytes(byte[] newBytes,
                        ObjectStorageInfo info)
                 throws java.io.IOException

Adds a set of bytes to the file, and returns the offset where they are stored.

Parameters:

newBytes - Bytes to store

Returns:

offset of bytes

Throws:

java.io.IOException

getBytes

protected byte[] getBytes(long offset)
                   throws java.io.IOException

Returns the bytes at the given offset.

Parameters:

offset - offset to retrieve the bytes

Returns:

bytes at the offset

Throws:

java.io.IOException

markDeletedBytes

protected void markDeletedBytes(long offset)
                         throws java.io.IOException

Throws:

java.io.IOException

removeBytes

protected byte[] removeBytes(long offset)
                      throws java.io.IOException

Removes the bytes at the given offset, and returns them.

Parameters:

offset - offset of bytes to remove

Returns:

bytes at offset

Throws:

java.io.IOException

isEmpty

public boolean isEmpty()

objectFromBytes

protected abstract java.lang.Object objectFromBytes(byte[] bytes)

This method should return an Object, as created by the given bytes. If, for whatever reason, the Object could not be created from these bytes, this method should return null.

Parameters:

bytes - byte[] used to create the Object

Returns:

A valid Object, created from the bytes.

objectToBytes

protected abstract byte[] objectToBytes(java.lang.Object obj)

This method should return the Object in its byte[] format.

Parameters:

obj - Object to convert to bytes.

Returns:

bytes of the Object