com.oracle.bmc.hdfs.store

Class BmcDataStore

java.lang.Object

com.oracle.bmc.hdfs.store.BmcDataStore

All Implemented Interfaces:

AutoCloseable

public class BmcDataStore
extends Object
implements AutoCloseable

BmcDataStore is a facade to Object Store that provides CRUD operations for objects by Path references.

Statistics are updated only on successful operations, and not on attempted operations.

Constructor Summary

Constructors

Constructor and Description
BmcDataStore(BmcPropertyAccessor propertyAccessor, com.oracle.bmc.objectstorage.ObjectStorage objectStorage, String namespace, String bucket, FileSystem.Statistics statistics, OCIMonitorPluginHandler ociMonitorPluginHandler)

Method Summary

Modifier and Type	Method and Description
void	close()
static String	configureParquetCacheString(BmcPropertyAccessor propertyAccessor)
void	createDirectory(Path path) Creates a pseudo directory at the given path.
void	delete(Path path) Deletes the object at the given path.
Future<?>	deleteDirAsync(Path dir) Delete an directory using the common delete threadpool and return a future.
void	deleteDirectory(Path path) Deletes the directory at the given path.
Future<?>	deleteFileAsync(Path path) Delete an object using the common delete threadpool and return a future.
org.apache.commons.lang3.tuple.Pair<List<FileStatus>,String>	flatListDirectoryRecursive(Path path, String nextToken) A method to list all files/dirs in a given directory in a flat manner.
long	getBlockSizeInBytes() Gets the configured block size in bytes.
ContentSummary	getContentSummary(Path path)
FileChecksum	getFileChecksum(Path path)
FileStatus	getFileStatus(Path path) Returns the FileStatus for the object at the given path.
FileStatus	getFileStatus(Path path, boolean checkOnlyExists)
static int	getReadAheadSizeInBytes(BmcPropertyAccessor propertyAccessor)
int	getRecursiveDirListingFetchSize()
boolean	isDirectory(Path path, boolean checkOnlyExists) Checks if the specified path is a directory.
boolean	isEmptyDirectory(Path path) Tests to see if the directory at the given path is considered empty or not.
List<FileStatus>	listDirectory(Path path) Returns the status of each entry in the directory specified.
static ExecutorService	newSwingFixedThreadPool(int nThreads, ThreadFactory threadFactory, long threadsTimeoutInSeconds) Creates a thread pool that will be able to swing open threads between 0 to nThreads If any additional tasks come , they wait in the queue
FSInputStream	openReadStream(FileStatus status, Path path, int bufferSizeInBytes, FileSystem.Statistics statistics) Creates a new FSInputStream that can be used to read the object at the given path.
OutputStream	openWriteStream(Path path, int bufferSizeInBytes, Progressable progress) Creates a new OutputStream that can be written to in order to create a new file.
void	renameDirectory(Path sourceDirectoryPath, Path destinationDirectoryPath) Renames a directory to have a new location or name.
void	renameFile(Path source, Path destination) Renames an object from one name to another.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BmcDataStore

public BmcDataStore(BmcPropertyAccessor propertyAccessor,
                    com.oracle.bmc.objectstorage.ObjectStorage objectStorage,
                    String namespace,
                    String bucket,
                    FileSystem.Statistics statistics,
                    OCIMonitorPluginHandler ociMonitorPluginHandler)

Method Detail

getReadAheadSizeInBytes

public static int getReadAheadSizeInBytes(BmcPropertyAccessor propertyAccessor)

newSwingFixedThreadPool

public static ExecutorService newSwingFixedThreadPool(int nThreads,
                                                      ThreadFactory threadFactory,
                                                      long threadsTimeoutInSeconds)

Creates a thread pool that will be able to swing open threads between 0 to nThreads If any additional tasks come , they wait in the queue

Parameters:

nThreads - the number of threads in the pool

threadFactory - the factory to use when creating new threads

threadsTimeoutInSeconds - timeout Of threads in seconds

Returns:

the newly created thread pool

Throws:

NullPointerException - if threadFactory is null

IllegalArgumentException - if nThreads <= 0

configureParquetCacheString

public static String configureParquetCacheString(BmcPropertyAccessor propertyAccessor)

renameFile

public void renameFile(Path source,
                       Path destination)
                throws IOException

Renames an object from one name to another. This is a multi-step operation that consists of finding all matching objects, copying them to the destination, and then deleting the original objects.

Parameters:

source - The source to rename, assumed to exist.

destination - The destination, may not exist, will be overwritten

Throws:

IOException - if the operation cannot be completed.

renameDirectory

public void renameDirectory(Path sourceDirectoryPath,
                            Path destinationDirectoryPath)
                     throws IOException

Renames a directory to have a new location or name. This will move all objects that were considered part of the original directory to be in the new one (all objects that matched the path prefix). This is a multi-step operation that consists of finding all matching objects, copying them to the destination, and then deleting the original objects.

Note, source and destination should not be root TODO: destination could be?

Parameters:

sourceDirectoryPath - The source directory to rename, assumed to exist.

destinationDirectoryPath - The destination directory.

Throws:

IOException - if the operation cannot be completed.

close

public void close()

Specified by:

close in interface AutoCloseable

delete

public void delete(Path path)
            throws IOException

Deletes the object at the given path.

Parameters:

path - Path of object to delete.

Throws:

IOException - if the operation cannot be completed.

deleteFileAsync

public Future<?> deleteFileAsync(Path path)
                          throws IOException

Delete an object using the common delete threadpool and return a future.

Parameters:

path - Path of the object to delete asynchronously.

Throws:

IOException

deleteDirAsync

public Future<?> deleteDirAsync(Path dir)
                         throws IOException

Delete an directory using the common delete threadpool and return a future.

Parameters:

dir - Path of the directory to delete asynchronously.

Throws:

IOException

getRecursiveDirListingFetchSize

public int getRecursiveDirListingFetchSize()

deleteDirectory

public void deleteDirectory(Path path)
                     throws IOException

Deletes the directory at the given path.

Parameters:

path - Path of object to delete.

Throws:

IOException - if the operation cannot be completed.

createDirectory

public void createDirectory(Path path)
                     throws IOException

Creates a pseudo directory at the given path.

Parameters:

path - The path to create a directory object at.

Throws:

IOException - if the operation cannot be completed.

isEmptyDirectory

public boolean isEmptyDirectory(Path path)
                         throws IOException

Tests to see if the directory at the given path is considered empty or not.

Parameters:

path - The directory path.

Returns:

true if the directory is empty, false if not.

Throws:

IOException - if the operation could not be completed.

listDirectory

public List<FileStatus> listDirectory(Path path)
                               throws IOException

Returns the status of each entry in the directory specified.

Parameters:

path - The directory path.

Returns:

A list of file statuses, or empty if the directory was empty.

Throws:

IOException - if the operation could not be completed

flatListDirectoryRecursive

public org.apache.commons.lang3.tuple.Pair<List<FileStatus>,String> flatListDirectoryRecursive(Path path,
                                                                                               String nextToken)
                                                                                        throws IOException

A method to list all files/dirs in a given directory in a flat manner. This is done without using any delimiters in the OSS list objects API.

Parameters:

path - The path to the directory for which the listing needs to be done.

nextToken - This is the token string in order to continue with a next page of results. It should be passed null in the first call to start fresh.

Returns:

A pair containing list of FileStatus objects and a possible token to the next page.

Throws:

IOException

getContentSummary

public ContentSummary getContentSummary(Path path)
                                 throws IOException

Throws:

IOException

getFileStatus

public FileStatus getFileStatus(Path path)
                         throws IOException

Returns the FileStatus for the object at the given path.

Parameters:

path - The path to query.

Returns:

The file status, null if there was no file at this location.

Throws:

IOException - if the operation could not be completed.

getFileStatus

public FileStatus getFileStatus(Path path,
                                boolean checkOnlyExists)
                         throws IOException

Throws:

IOException

isDirectory

public boolean isDirectory(Path path,
                           boolean checkOnlyExists)
                    throws IOException

Checks if the specified path is a directory.

Parameters:

path -

Returns:

true if path exists and is a directory, false otherwise.

Throws:

IOException

openReadStream

public FSInputStream openReadStream(FileStatus status,
                                    Path path,
                                    int bufferSizeInBytes,
                                    FileSystem.Statistics statistics)

Creates a new FSInputStream that can be used to read the object at the given path.

Parameters:

status - The file status for this file.

path - The path to open.

bufferSizeInBytes - The buffer size in bytes (may not be used).

statistics - The FileSystem.Statistics instance to publish metrics into.

Returns:

A new input stream to read from.

openWriteStream

public OutputStream openWriteStream(Path path,
                                    int bufferSizeInBytes,
                                    Progressable progress)

Creates a new OutputStream that can be written to in order to create a new file.

Parameters:

path - The path for the new file.

bufferSizeInBytes - The buffer size in bytes can be configured by setting the config key BmcConstants.MULTIPART_PART_SIZE_IN_MB_KEY. Default value is 128MiB which comes from OCI Java SDK UploadConfiguration

progress - Progressable instance to report progress updates to.

Returns:

A new output stream to write to.

getFileChecksum

public FileChecksum getFileChecksum(Path path)
                             throws IOException

Throws:

IOException

getBlockSizeInBytes

public long getBlockSizeInBytes()

Gets the configured block size in bytes.

Returns:

Block size in bytes.