19 Migrate Object Storage
- If you are using the Oracle Cloud Infrastructure Storage Software Appliance, you can use Cloud Sync to migrate data to Oracle Cloud Infrastructure Storage Gateway.
- If you are not using Oracle Cloud Infrastructure Storage Software Appliance and if the total amount of object storage data does not exceed the guidelines for this type of data migration, then you can use Rclone to migrate data from Oracle Cloud Infrastructure Object Storage Classic to Oracle Cloud Infrastructure Object Storage. For more information on how to determine if your environment is a candidate for this migration strategy, contact your Oracle support or consulting representative.
Migrate Storage Appliance Data to Oracle Cloud Infrastructure Storage Gateway
Oracle Cloud Infrastructure Storage Gateway is the recommended gateway for Oracle Cloud Infrastructure Object Storage. Oracle recommends that you migrate from Oracle Cloud Infrastructure Storage Software Appliance backed by containers in Oracle Cloud Infrastructure Object Storage Classic to Oracle Cloud Infrastructure Storage Gateway that persists data in object storage buckets in Oracle Cloud Infrastructure.
Migration steps can take a considerable amount of time depending upon the number of files/directories and size of files, available resources (CPU, memory, network bandwidth, disk throughput, etc.). While scheduling the migration process, consider the estimated time required to complete the migration. If you abort the migration process for any reason, you can restart the process.
To migrate data from an Archive container, you must first restore the archived objects. For more information, see Restoring Archived Objects in Using Oracle Cloud Infrastructure Object Storage Classic.
Before You Begin
Before you begin migrating your workloads, verify that you have a cloud account with Identity Cloud Service to access Oracle Cloud Infrastructure, and then complete the planning checklist. This section provides information to help you complete the planning checklist.
Identify the following information for each storage software appliance that you want to migrate.
| Information Required | Details |
|---|---|
| Is the appliance is running in Compute Classic or in your on-premises environment? | |
| Version of the storage software appliance running in Compute Classic or in your on-premises environment | |
| Link to access the management console and password | |
| Size of the cache storage | |
| Total number of file systems |
Identify the following information for every file system in the storage software appliance that you want to migrate.
| Information Required | Details |
|---|---|
| Number of files in the file system | |
| Total storage space used | |
| Is the file system is encrypted? If yes, note down the location where you have downloaded the encryption keys. |
Get Details of the Source Environment
Identify the user name, password, authentication URL, and REST Endpoint URL for the Oracle Cloud Infrastructure Object Storage Classic account from which you want to migrate data.
Get Details of the Appliance Instance
Before gathering details of the instance on which Oracle Cloud Infrastructure Storage Software Appliance is installed, identify if the appliance is installed on a Compute Classic instance or in your on-premise environment. If you have set up multiple appliance instances, identify the following information for every appliance instance that you want to migrate: the public IP address of the instance that hosts the application, link to access the management console and the password. Use the public IP address to connect to the appliance instance using SSH and to log in to the management console of the appliance.
Get Details of an Appliance Installed on a Compute Classic Instance
If the appliance is installed on a Compute Classic host, then to gather the required information perform the following steps:
-
On the Compute Classic host go to the directory where you have downloaded the appliance provisioning tool and run the following command:
./fscs.sh -i config_fileIn this command,
config_fileis the full path and name of the appliance configuration file. -
The tool prompts you to enter the password for the Compute Classic user specified in the configuration file. Enter the password.
The tool validates the password and displays the IP addresses of the instance, as shown in the following example:
Extracting IP addresses ... Private IP address: 10.196.35.245 Public IP address: 203.0.113.48 -
The password for the
adminuser to access the management console of the appliance.If you don’t remember the password, you can reset it. See Changing the Admin Password for the Appliance in Using Oracle Cloud Infrastructure Storage Software Appliance.
-
Access the management console. In your web browser, go to
https://public_IP_address_of_appliance_instance, and then enter the admin password.The available filesystems are displayed. Note down the number of the filesystems that you want to migrate.
Get Details of an Appliance Installed in the On-Premises Environment
If the appliance is installed on a Compute
Classic host, use the oscsa command-line tool to manage Oracle Cloud Infrastructure Storage Software Appliance. oscsa commands might fail, if you don't run these commands as a root user. To gather the required information, perform the following steps:
-
If the appliance instance is in your on-premises environment, log in to the appliance host and then run the following commands:
oscsa infoNote down the link to access the management console from the output. Example:
Management Console: https://myApplianceHost.example.com:32775 If you have already configured an OSCSA FileSystem via the Management Console,you can access the NFS share using the following port. NFS Port: 32774Note down the link to access the management console and the NFS port from the output.
If the appliance instance is not running, then run the following command to bring up the appliance and get the required information.
oscsa up -
Run the following command to identify the version of the storage appliance.
oscsa versionNote down the version. While migrating the on-premise storage appliance, you may install a storage appliance on a Compute Classic instance. The version of the storage appliance that you install on the Compute Classic instance must be same as the on-premise storage appliance that you want to migrate.
-
Access the management console. In your web browser, go to
https://myApplianceHost.example.com:32775, and then enter the admin password.The available file systems are displayed. Note down the name and number of the file systems that you want to migrate.
Get Details of a File System
Each Oracle Cloud Infrastructure Storage Software Appliance instance can have multiple file systems. Gather the following details for every file system that you want to migrate.
To view the details of a file system, log in to the management console, and click the name of the file system:
Prepare the Target Environment
Before migrating the file systems, set up the target environment. To prepare the target environment, install and configure an instance of the Oracle Cloud Infrastructure Storage Gateway. You will use this instance after you complete the migration process. Test the target environment and resolve issues, if any. You have to test the environment only once before migrating the first file system. Mount a test file system on the storage gateway instance, and then check if you can successfully read and write to object storage.
Additional Steps to Prepare the On-Premises Environments
If you are migrating object storage data from a storage appliance in the cloud to a storage gateway in the cloud, you don't need to perform any additional steps to prepare the environments. However, if you are migrating object storage data from a storage appliance in your on-premises environment to a storage gateway in your on-premises environment, consider performing the following additional steps to prepare the environment.
Even if the appliance instance is available on-premises, you don't need to perform these additional steps if you are connecting to Oracle Cloud through a low latency, high throughput connectivity, such as Oracle Cloud Infrastructure FastConnect. In such a scenario, you can use the appliance instance in your on-premises environment to perform the migration.
Prepare Source Appliance for Migration
- Install and configure an Oracle Cloud Infrastructure Storage Software Appliance instance on a Compute Classic instance.
- Create a file system with the same name and properties (such as storage tier, encryption, and other options) as the file system being migrated. Set appropriate NFS options, which is available in advanced options, to ensure security.
- Do not connect the file system. Before connecting the file system you must ensure that all pending uploads to the appliance are complete. You'll also need to disconnect the existing appliance instance. You'll connect the file system later in the migration process.
Prepare Target Gateway for Migration
- Install and configure Oracle Cloud Infrastructure Storage
Gateway on a compute instance in Oracle Cloud
Infrastructure.
You 'll use this Oracle Cloud Infrastructure Storage Gateway instance during the migration process. Create this instance only if you want to use an Oracle Cloud Infrastructure instance to complete the migration instead of using an instance in your on-premises environment.
- Create a file system with the same name and properties as the file system being migrated. Use standard tier bucket along with object life cycle policy and not the archive tier bucket.
- Set appropriate NFS options, which is available in advanced options, to ensure security.
- Connect the Oracle Cloud Infrastructure Storage Gateway file system.
Start Migrating a File System
The following steps are required for migrating a single file system. Repeat these steps for each file system that you want to migrate. Depending upon the availability of resources, you may migrate multiple file systems in parallel.
Validate
After object storage data migration is complete, perform the following steps to determine if the migration was completed successfully.
- After the migration is complete, check if any errors occurred during the migration process. Cloud Sync records error messages in a file. You can download this file from the management console.
- Compare the file inventory in the source and target environments. You can list the files in the source and target environment and compare them.
Post Migration Requirements for On-premises Environment
Perform the following steps only if you are migrating an appliance instance which is installed in your on-premises environment and you have used a temporary storage gateway instance which is installed in the cloud for the purpose of migration. After migrating the file system to the temporary storage gateway instance in the cloud, connect the file system to an on-premises storage gateway instance.
Delete the Source File System and Appliance
After the migration is complete and you've validated the Oracle Cloud Infrastructure Storage Gateway file system, you can clean up unused resources.
Migrate Object Storage Using Rclone
Note:
Before you use the procedures in this document, review the amount of object storage data to be migrated. Consult with your Oracle representative to be sure the amount of data you want to migrate is supported by this migration method.
Considerations for Data Migration Using Rclone
Before you start your migration, consider the following factors that could have an impact on your migration process.
- You can't copy multiple containers at a time. You can copy only one container at a time. However, you can copy one or more objects at a time.
- Metadata and policies aren't copied. This includes:
- Custom metadata on objects and containers
- Cross-Origin Resource Sharing (CORS) settings
- Object immutability
- ACLs on containers
- Container quotas
- Container replication policies
- Server side encryption can be enabled on containers. Rclone can successfully copy containers where server side encryption is enabled.
- If you have large objects in your Oracle Cloud Infrastructure Object Storage Classic account, then you should check if the object as well as its segments are stored in the same container. When the same container has the object and as well as its segments, Rclone detects this and copies the object correctly without duplicating data. However, when the segments are in one container and the object manifest is in a different container (for example, if you upload a large object using the File Transfer Manager command-line interface), then Rclone can't detect the duplicate object. In such cases, ensure that you don't copy the container that has the segments or if you copy it, you delete it later to avoid duplicating data.
- There are some differences in the rules for naming containers in Oracle Cloud Infrastructure Object Storage Classic versus buckets in Oracle Cloud Infrastructure Object Storage. Container names can include UTF-8 characters subject to a maximum of 1061 bytes, while bucket names can include only upper or lower case letters, numbers, hyphens, underscores, and periods. Bucket names are also subject to a maximum of 256 characters. Due to these differences, you might need to rename some containers before you copy them.
- There are also some differences in the rules for naming objects in Oracle Cloud Infrastructure Object Storage Classic versus Oracle Cloud Infrastructure Object Storage. In Oracle Cloud Infrastructure Object Storage Classic you can specify object names with UTF-8 characters subject to a maximum of 1061 bytes, while in Oracle Cloud Infrastructure Object Storage you can specify object names with UTF-8 characters subject to a maximum of 1024 bytes and up to 1024 characters.
Consider using the following rclone options in the following scenarios:
dry-run:Use this option to validate a migration before start to copy data. Amongst other benefits, this option allows you to check that the specified bucket name is valid.includes, excludes,orfiltering:Use these options to include or exclude files to be copied, based on patterns or size.s3-upload-cutoff:Use this option to copy large objects.progress:Use this option to generate a real-time overview of the transfer.transfers:Use this option to utilize your network bandwidth better and increase throughput. You will need to tune this value based on the available bandwidth for your compute shape.
For more information about Oracle Cloud Infrastructure Object Storage Classic, see https://docs.oracle.com/en/cloud/iaas/storage-cloud/index.html and for information about Oracle Cloud Infrastructure Object Storage see https://docs.cloud.oracle.com/iaas/Content/Object/Concepts/objectstorageoverview.htm.
Before You Begin
Before you begin, complete the following prerequisites.
- Validate the existing object storage data and back up the object storage data in the source environment.
- Create a new object storage destination in your target environment. For information about creating a bucket, see Managing Buckets in Oracle Cloud Infrastructure documentation. You cannot change the name of the bucket after you create it, so carefully consider the naming conventions and if you want the bucket name to match the name of the container available at the source.
- To migrate data from an Archive container, you must first restore the archived objects. For more information, see Restoring Archived Objects in Using Oracle Cloud Infrastructure Object Storage Classic.
Get Details of Your Source Environment
Identify the user name, password, authentication URL, and REST Endpoint URL for the Oracle Cloud Infrastructure Object Storage Classic account from which you want to migrate data.
To identify the authentication URL and REST Endpoint URL: