About Storage Modernization with Oracle Cloud Infrastructure Object Storage
This solution playbook will help you understand how to design and execute the migration of long term storage from local or NFS filesystems to Oracle Cloud Infrastructure Object Storage (OCI Object Storage). A solution like this can help reduce costs as well as apply retention, lifecycle rules, and pre-authenticated requests as added features.
In this solution playbook, we are using a customer use case to highlight the starting point for implementing a successful modernization effort.
We often talk about cloud migration and modernization in the same conversation, going through all of the methods, services, and offerings available. The conversation often entails doing a phased approach, where an entire data center is systematically moved to the cloud, lifted and shifted as-is, and then everything else follows suit.
Once the initial phase is complete, application modernization often loses its importance when compared against security, monitoring, current and ongoing integration challenges. It also highlights a shift from work that Infrastructure teams can do to that of Application teams. Budgets, timing, and other priorities take precedence, and thus applications are left to run as-is in the cloud. In order to realize the true potential savings the cloud can provide, enterprises should look to technologies such as object storage in order to save on cost.
File archives are a great place to start, as they can be transitioned from NFS to OCI Object Storage with minimal development work and testing. Implementing a solution such as this one can save on the order of 10-50x on storage, depending on the use case. Enterprises have realized that operating data centers is becoming more of a liability, and could result in unseen additional costs, with the growing number of potential attacks, loss of service, and competitors innovating constantly.
This should make using cloud-native services such as OCI Object Storage a priority, to save on cost, protect the business, share the burden of operating a wide array of disparate workloads with a hyperscaler, and more.
Understand Post-Migration Challenges
The key here was to focus on a post-migration pain point and redesign a large enough piece of the application to be impactful, yet small enough to tackle within a single design sprint. This way, the business realizes both savings and peace of mind, while keeping development and testing costs low.
For the use case in this solution playbook, the cost of shared file storage (NFS) became an issue in the cloud, and the application’s original design became the reason that it wasn’t so easy to change. During the on-premises to cloud migration project, we talked about Object Storage as a cheaper and more reliable alternative to File Storage, on paper about 10x in savings. Adding backups and replication, that 10x is likely higher. Efficient features such as Cross-region replication, Retention Lock, and Lifecycle Policies can all work together to make Object Storage the basis for a cost-effective, reliable, and secure system on which to store important documents. However, when the application is designed around the NFS file system for document storage, and expects POSIX semantics, things become harder.
The application that was modernized in this use case is a standard 3-tier application, but with several external components that have to run a coordinated and cpu-intense process to generate customer bills, post them into an organized directory structure, and catalog them for download and long-term storage. These PDF and other files were stored on a large NFS file system with very a specific file naming pattern, in order to be accessed by a generated path. Yet another application was built around Apache HTTP server, using the long term storage area of this NFS share as its document root, such that the generated URLs from the application could be used to download files from any point over the last 2 years. Finally, files older than a certain age could be removed from the online archive, but still could be requested by auditors seeking records.
Thus, all files up to 10 years old were required to be kept on the NFS filesystem, essentially costing the business more money each day new files were generated. The problem existed across several different applications, so the cost issue would only get worse over time.
Leverage OCI Object storage
Object Storage is perfect for files that are not changing frequently. This is in contrast to NFS, which focuses on mixed-use shared storage.
By taking advantage of several Object Storage design elements and some specific features of OCI Object Storage Service, we can increase availability and reduce cost for suitable workloads.
In this use case, files created for medium term access and long term archive are a good fit. These files are likely written once, and have the need to be stored for months or years without change. In fact, the business may want to ensure that they are immutable for a period of time.
Overall, here is a list of basic reasons why Object Storage has advantages over traditional file storage for these types of files.
- Availability: Object Storage is a regional service, meaning that it is not tied to a single Availability Domain.
- Search: Using object metadata is likely more useful than relying solely on file names and POSIX-style find commands.
- Retention Rules: An entire bucket can be ensured not to change once objects are written to ensure immediate compliance.
- Storage Tiers: Object storage tiering (either auto or manual) leads to a huge reduction in cost for infrequently accessed or business rule required retention objects.
- Lifecycle Policies: Tuning the movement between storage tiers and auto-deletion (cleanup) after retention will save on storage and administration.
- Replication: Easily and automatically replicating an entire bucket to another region can increase availability and access to data.
- Cost: Properly constructed and maintained object storage costs much less than duplicated and cluttered NFS file systems.
NFS is still useful for applications which require a mounted, shared, POSIX-style filesystem. The customer still required NFS for shared storage, but its use was limited to "operational" files, as opposed to "archive" files. The solution described here covers how to set up and access Object Storage, and how the application was modified to use both NFS storage and the new Object Storage archive created for long-term storage.
Architecture
This architecture shows the design and execution to move long term storage to OCI Object Storage, helping to reduce cost and apply retention, lifecycle rules, and pre-authenticated requests as added features.
The following images depict the architecture "before" and "after" implementation. Oracle File System Service (FSS) is used for a large shared filesystem. On this filesystem, which had been migrated from an on-premises datacenter, the application components use batch processing to generate archive files on an ongoing basis. Thus, the same NFS filesystem holds both the application elements required to do the intermediate processing (scripts, temp files, and so on) and the actual file archive, stored in a hierarchy that must be maintained for up to 10 years, per business requirements.
Once set up, OCI Object Storage buckets are used to host the archive portion of what NFS was doing. Permissions for reading and writing the bucket are narrowly defined, and retention and lifecycle rules are established, in order to be prepared for a large influx of data. The large hierarchy of archive files is copied to OCI Object Storage, and the batch processing is refactored to place new archives on the new object bucket. Access mechanisms were also refactored slightly in order to access the files from object storage, in a way which prevented the rest of the application and end customers from having to make changes to how they access these archives.
The following diagram illustrates the architecture before implementation:
oci-object-storage-modernization-arch-oracle1.zip
The following diagram illustrates the architecture after implementation:
oci-object-storage-modernization-arch-oracle.zip
- Tenancy Admins - Full Access
- App Admins - Limited Access without object read
- Read only - Bucket inspect without object read
- Statements added per dynamic group
- Narrowly defined for specific resources
- List of instance OCIDs
- Instance compartment OCID
- 3650 days (10 years)
- locked
e: Lifecycle Rules:
- After 90 days - Infrequent storage
- After 180 days - Archive storage
- After 3651 days - Delete
This architecture supports the following components:
- Object storage
Oracle Cloud Infrastructure Object Storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.
- File storage
The Oracle Cloud Infrastructure File Storage service provides a durable, scalable, secure, enterprise-grade network file system. You can connect to a File Storage service file system from any bare metal, virtual machine, or container instance in a VCN. You can also access a file system from outside the VCN by using Oracle Cloud Infrastructure FastConnect and IPSec VPN.
- Identity and Access Management (IAM)
Oracle Cloud Infrastructure Identity and Access Management (IAM) is the access control plane for Oracle Cloud Infrastructure (OCI) and Oracle Cloud Applications. The IAM API and the user interface enable you to manage identity domains and the resources within the identity domain. Each OCI IAM identity domain represents a standalone identity and access management solution or a different user population.
Considerations for Identity Management
Using private object storage buckets implies setting up proper permissions for use. By default, user groups and dynamic groups are not typically granted access to wide permission sets, such as object-family, unless it is scoped to a compartment.
Before embarking on this solution, it is worth ensuring that only the groups you want to have access to objects storage have permissions. One thing that is extremely helpful here is following the CIS Landing Zone methodology around restricting access. When implementing this solution, we discuss the creation of dynamic groups, so it is worth understanding both the compartment structure of your tenancy, as well as the concepts that the landing zone discusses. It is also worth reading up on OCI Policy Syntax, including how to narrowly define a dynamic group and a policy statement.
Although both RCLONE and OCIFS support standard OCI API Keys as an authentication mechanism, we chose Instance Principals and Dynamic Groups to authenticate. This improves the overall security posture - it is not required to create, distribute, or rotate keys. Instead, separate dynamic groups are utilized, in order to ensure the most narrow permissions possible for each dynamic group. For example, allowing bucket creation could be allowed by policy for the RCLONE dynamic group, whereas only object read is allowed for the Apache dynamic group.
Considerations for Archive Storage
To save on costs and utilize the lowest cost tier of OCI Object Storage, this solution implemented Lifecycle Rules, which move objects to the Infrequent Tier, and then to the Archive tier after a period of time after creation.
Once the objects are archived, they cannot be directly reclassified to Standard Tier. Because of the offline nature of Archived Object Storage, a process needs to be developed in which an audit (business process) could be requested, certain files pulled from the archive, and then the files become accessible for a time-bound period.
Once again, using the inherent features of object storage, files can be recalled temporarily, copied out to a temporary location (another bucket), and exposed externally using Pre-authenticated Requests (PARs). These are obscure URLs (security by obscurity) that allow access to specific files in a bucket, and can expire after a set amount of time, revoking access.
During the recall period, the files can be copied to new standard-tier buckets, and then they automatically revert back to Archive mode, and no maintenance is needed. RCLONE and OCI CLI, Java, REST or Python can be used similarly for the main solution, again given access to a specific audit bucket.