1. Learn About Storage Modernization with Oracle Cloud Infrastructure Object Storage
  2. Migrate

Migrate

The solution used here is to separate the large NFS share logically into two areas:

  • An area called an “operational NFS” share containing only data in flux, yet to be processed, and for post-processing and re-processing.
  • A temporary staging area which can be cleaned out regularly.

For the temporary area of the NFS store, an OCI Object Storage bucket is created, with the intent of mirroring the large directory structure within the NFS archive using standard object/path naming.

Note that no changes are made to the NFS structure aside from the expectation that the archive directory (20TB+) will be copied to Object Storage eventually. Data in the operational area is processed, checked, and held in the temporary staging area as before. Once finalized, data is moved to the Object-based archive and the temporary staging area is cleaned up. The mechanism for copying objects from the local operational NFS share to the archive bucket can be RCLONE or OCIFS.

For the archive part of the NFS File System, we will use the object bucket that was created earlier. Our intent is to mirror the large archive from NFS to Object Storage with the same path-like structure. To do this, RCLONE commands similiar to the following are used: 
cd archive
rclone copy . <the_rclone_config>:BucketName/

Note that depending on how the application is developed, the existing processes can continue to write to the existing locations until a cut-over takes place. At that point the the entire archive has been copied, application code is reading from object storage, and new files are generated to the new archive location.While the exact order of cutover events is going to depend on each case, at the end of the migration the main archive of NFS-based files can be deleted. At that point the billing will reflect the savings. Batch processing can occur, and through standard shell scripts, large hierarchies of files can be copied to Object Storage via RCLONE.

While the exact details of order of operations is implementation-specific, at the end of the process, the archive area of the NFS share is cleaned up once the archive files are safely in Object Storage. Each batch run causes the creation of new archive files, which are copied to Object Storage via RCLONE prior to deletion.