Use Artifact Snapshots to Protect Your OCI Kubernetes Engine Clusters from Disaster

To ensure business continuity in the event of disasters, you'll want to implement a disaster recovery (DR) strategy for applications running on Kubernetes cluster that provides data protection and enables you to quickly switch to a standby system with minimal loss of data and productivity. Despite the tremendous change that Kubernetes adoption implies for IT system’s architecture, a Kubernetes system presents similar DR paradigms as a traditional application (Oracle Java SE, Oracle Java EE, and so on). You must maintain a consistent and an as up-to-date as possible copy of your primary system in a secondary location that can resume workloads should a disaster cause downtime in the primary region.

Oracle Maximum Availability Architecture (Oracle MAA) provides recommendations and utilities that enable you to recover in disaster scenarios affecting a location and forcing the redirection of workloads to a replica site. The focus of this content is Kubernetes configuration replication for applications. Applications running on Kubernetes clusters depend on many different components to operate, including control plane nodes, worker nodes, load balancers, and storage. At the same time, the runtime data generated by applications running on Kubernetes presents the same challenges as traditional applications—during runtime applications may generate, read and update persist data. This solution playbook provides recommendations to replicate the configuration of an application running on Kubernetes. Runtime data’s disaster protection is out of the scope of this document and should be treated exactly in the same way as in traditional applications running on application servers, including the following:

Avoid polyglot persistence. Using different types of persistent stores for runtime data is an almost impossible to solve problem, per the Backup Availability Consistency (BAC) Theorem.
Use a single store for all the different data types, microservices and applications with dependencies, as much as possible.
See Oracle MAA best practices for Oracle Database for disaster protection for your runtime data.

In addition, you must protect the Kubernetes cluster control plane. Use the appropriate etcd snapshots to avoid corruptions, failures, and to provide a flashback to working clusters. Although Oracle MAA provides best practices for control plane protection against disasters, it is out of the scope of this document to describe the required techniques in that area.

Before You Begin

There are several Oracle MAA technical briefs that describe how to set up a disaster recovery (DR) system for traditional middleware systems. These documents detail the disaster protection requirements for the external infrastructure components (such as storage, load balancers, and database) that Kubernetes applications use.

Review the following for more details:

Architecture

This architecture shows the disaster recovery (DR) system's topology for the Kubernetes cluster.

All runtime, configuration, and metadata information residing in the primary database is replicated from Region 1 to Region 2 with Oracle Autonomous Data Guard. The required Kubernetes (K8s) cluster configuration is replicated through ETCD snapshots for control plane protection and with YAML snapshots for application configuration protection. You can use artifact snapshots or You can use etcd copies or artifact snapshots for application-specific configuration protection for application-specific configuration protection. See Kubernetes clusters restore based on etcd snapshots for more details. The images that the container uses are hosted in registries, either local to each cluster or in external repositories (images are not considered a Kubernetes cluster configuration by themselves).

Note:

Setting up Oracle Autonomous Data Guard for the runtime database is out of the scope of this document.

Description of kubernetes-multiregion-dr.png follows

Description of the illustration kubernetes-multiregion-dr.png

kubernetes-multiregion-dr-oracle.zip

This architecture supports the following components:

Region
An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
Load balancer
The Oracle Cloud Infrastructure Load Balancing service provides automated traffic distribution from a single entry point to multiple servers in the back end.
Dynamic routing gateway (DRG)
The DRG is a virtual router that provides a path for private network traffic between VCNs in the same region, between a VCN and a network outside the region, such as a VCN in another Oracle Cloud Infrastructure region, an on-premises network, or a network in another cloud provider.
Data Guard
Oracle Data Guard and Oracle Active Data Guard provide a comprehensive set of services that create, maintain, manage, and monitor one or more standby databases and that enable production Oracle databases to remain available without interruption. Oracle Data Guard maintains these standby databases as copies of the production database by using in-memory replication. If the production database becomes unavailable due to a planned or an unplanned outage, Oracle Data Guard can switch any standby database to the production role, minimizing the downtime associated with the outage. Oracle Active Data Guard provides the additional ability to offload read-mostly workloads to standby databases and also provides advanced data protection features.
Oracle Real Application Clusters (Oracle RAC)
Oracle RAC enables you to run a single Oracle Database across multiple servers to maximize availability and enable horizontal scalability, while accessing shared storage. User sessions connecting to Oracle RAC instances can failover and safely replay changes during outages, without any changes to end user applications.
Registry
Oracle Cloud Infrastructure Registry is an Oracle-managed registry that enables you to simplify your development-to-production workflow. Registry makes it easy for you to store, share, and manage development artifacts, like Docker images. The highly available and scalable architecture of Oracle Cloud Infrastructure ensures that you can deploy and manage your applications reliably.
Kubernetes Engine
Oracle Cloud Infrastructure Kubernetes Engine (OCI Kubernetes Engine or OKE) is a fully managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud. You specify the compute resources that your applications require, and Kubernetes Engine provisions them on Oracle Cloud Infrastructure in an existing tenancy. OKE uses Kubernetes to automate the deployment, scaling, and management of containerized applications across clusters of hosts.
Kubernetes cluster
A Kubernetes cluster is a set of machines that run containerized applications. Kubernetes provides a portable, extensible, open source platform for managing containerized workloads and services in those nodes. A Kubernetes cluster is formed of worker nodes and control plane nodes.
Kubernetes worker node
A Kubernetes worker node is a worker machine that runs containerized applications within a Kubernetes cluster. Every cluster has at least one worker node.
Kubernetes control plane
A Kubernetes control plane manages the resources for the worker nodes and pods within a Kubernetes cluster. The control plane components detect and respond to events, perform scheduling, and move cluster resources. The following are the control plane components:
- kube-apiserver: Runs the Kubernetes API server.
- etcd: Distributed key-value store for all cluster data.
- kube-scheduler: Determines which node new unassigned pods will run on.
- kube-controller-manager: Runs controller processes.
- cloud-controller-manager: Links your cluster with cloud-specific API.
Ingress Controller
An Ingress controller is a component that runs in a Kubernetes cluster and manages the Ingress resources. It receives traffic from the external network, routes it to the correct service, and performs load balancing and SSL termination. The Ingress controller typically runs as a separate pod in the cluster and can be scaled independently from the services it manages.
KUBE-Endpoint API
The KUBE-Endpoint API is the kube-apiserver component of the Kubernetes control plane. It runs the Kubernetes API server.
ETCD Backup
ETCD Backup is a backup of etcd component of the Kubernetes control plane. The etcd contains the distributed key-value store for all cluster data. It's important to create an ETCD Backup to recover Kubernetes clusters for disaster recovery.
YAML Snapshots
A YAML snapshot is a point-in-time copy of the (yaml) files containing the definition of the artifacts in a Kubernetes cluster. The snapshot is a tar file that you can use to restore those artifacts in the same or a different Kubernetes cluster.

Considerations for Kubernetes Disaster Protection

When implementing disaster protection for Kubernetes, consider the following:

Symmetric disaster recovery (DR): Oracle recommends using the exact same resource capacity and configuration in primary and secondary. The Kubernetes namespaces involved should have similar resources available, such as the number of worker nodes (and their hardware capacity) and other infrastructure (shared storage, load balancers, databases, and so on). The resources on which the Kubernetes cluster in the secondary region depend, must be able to keep up with the same workloads as primary. Also, the two systems must be consistent functionally with the exact same services on which the restored system depends on, side cars, configuration maps (CMs) must be used in both locations.
Images present a similar paradigm to binaries: Images don't change as frequently as the Kubernetes configuration and you might not need to update images with every Kubernetes cluster replication. The images used by the primary system must be the same as the ones used in the secondary system or inconsistencies and failure may take place. However, image replication is out of the scope of this playbook. There are multiple strategies that you can use to maintain a consistent use of images between two locations, including the following:
- Save images in primary and load to secondary’s worker nodes. This approach is very easy to implement but incurs in management overhead. Using container registries has considerable benefits and saving images locally makes it more difficult to manage versions and updates.
- Images can reside in totally external Container registries in different regions or data centers from the ones used by primary and standby. External products and libraries are maintained by third parties and their availability is typically implicit in their releases.
- Images can reside in Container Registries located in primary and standby. Each region gets updated in parallel when a new version of an image is released. This provides better control over the software used but incurs in higher management overhead. It requires duplicating images and managing the credentials to access two different registries. CI/CD tools are typically used for this approach.

Although this playbook presents an example using Oracle Cloud Infrastructure, the recommendations are generic to custom Kubernetes clusters installed in on-premises systems. You can use the steps and scripts provided between a primary Kubernetes cluster running in OCI Kubernetes Engine (OKE) and a secondary cluster running in an on-premises or custom Kubernetes cluster. You can also use the steps and scripts between a primary Kubernetes cluster running in OKE and a secondary cluster running also in OKE, or between two on-premises or custom Kubernetes clusters.

About Required Products and Roles

This solution requires the following products and roles:

Oracle Cloud Infrastructure Kubernetes Engine (Kubernetes Engine or OKE) cluster
Bastion node capable of managing the kubernetes system
Oracle Cloud Infrastructure (OCI)

This playbook is based on using OCI regions and resources for the primary and secondary regions. However, this solution is also applicable for Kubernetes clusters that are not located on OCI.

These are the roles needed for each service.

Service Name: Role	Required to ...
Oracle Cloud Infrastructure: `admin`	provision and setup resources and services if you're using one or more OCI regions.
Kubernetes Engine cluster (primary): `administrator`	run all of the scripts.
Kubernetes Engine (primary) nodes: OS user with execute permissions and ssh permissions to secondary	run the following scripts: `maak8-get-all-artifacts.sh` `maak8DR-apply.sh`
Kubernetes Engine cluster (secondary): `administrator`	run all of the scripts.
Kubernetes Engine (secondary) nodes: OS user with execute permissions	run the following scripts: `removeyamlblock.sh` `apply-artifacts.sh` `maak8-push-all-artifacts.sh`

See Oracle Products, Solutions, and Services to get what you need.

Change Log

This log lists significant changes:

July 12, 2023

Revised the architecture diagram.
Enhanced the content.

Oracle Cloud Service Names Catalog

Conrefs for cloud services that have been released (GA) or are in development are imported from PDB varcat entries. Do not edit this file. To submit a change, use JIRA issue DPS-9286.

Oracle Cloud Infrastructure Zero Trust Packet Routing

Zero Trust Packet Routing

Oracle Autonomous Database for Developers - Container Image

Kubernetes Engine

OKE

Oracle Cloud Infrastructure Kubernetes Engine

OCI Speech

Autonomous Database for Developers - Container Image

Roving Edge Infrastructure

Oracle Interconnect for Google Cloud

Oracle Database@Google Cloud

OCI GoldenGate

Oracle Cloud Infrastructure GoldenGate

Autonomous Database for Developers

Oracle Roving Edge Infrastructure

GDK for Micronaut

Graal Development Kit for Micronaut

Oracle Exadata Cloud Infrastructure

OCI Data Integration

Oracle Database@Azure

OCI Ops Insights

Oracle Cloud Infrastructure Ops Insights

OCI Secure Desktops

Oracle Cloud Infrastructure Secure Desktops

Oracle Cloud Infrastructure Logging Analytics

OCI Document Understanding

Oracle Cloud Infrastructure Document Understanding

Oracle Cloud Infrastructure Speech

Oracle Cloud Infrastructure Language

OCI Generative AI

Oracle Cloud Infrastructure Generative AI

Oracle Interconnect for Azure

OCI Process Automation

Oracle Cloud Infrastructure Process Automation

Oracle Retail Merchandising Foundation Cloud Service

OCI Block Volumes

OCI Cache with Redis

Oracle Cloud Infrastructure Cache with Redis

OCI Database with PostgreSQL

Oracle Cloud Infrastructure Database with PostgreSQL

OCI Load Balancing

OCI Certificates

Oracle Cloud Infrastructure Certificates

Verrazzano

Oracle Verrazzano Enterprise Container Platform

Oracle Cloud VMware Solution

OCI FastConnect

OCI Compute

OCI Full Stack Disaster Recovery Service

OCI API Gateway

Oracle Cloud Infrastructure API Gateway

OCI File Storage

OCI Network Firewall

Oracle Cloud Infrastructure Network Firewall

OCI Storage Gateway

OCI Audit

OCI Monitoring

Compute Cloud@Customer

OCI Logging

Oracle Cloud Infrastructure Logging

OCI Bastion

Oracle Compute Cloud@Customer

OCI Identity and Access Management

OCI Object Storage

OCI Streaming

OCI Notifications

OCI Events

Oracle Database Zero Data Loss Autonomous Recovery Service

Oracle Cloud Infrastructure Container Instances

OCI Container Instances

Oracle Cloud Native Environment

OCI Lake

Oracle Cloud Infrastructure Lake

Oracle Autonomous Database on Dedicated Exadata Infrastructure

Oracle Internet of Things (IoT)

Oracle IoT Service Monitoring for Connected Assets Cloud Service

Oracle IoT Production Monitoring Cloud Service

Oracle IoT Connected Worker Cloud Service

Oracle IoT Asset Monitoring Cloud Service

Oracle IoT

Oracle Internet of Things

Oracle Cloud Infrastructure Government Cloud

Oracle IoT Fleet Monitoring Cloud Service

Oracle Fusion Data Intelligence

Fusion Data Intelligence

OCI Functions

Oracle Cloud Infrastructure Bastion

Oracle Cloud Infrastructure Cloud Shell

Oracle Cloud Observability and Management Platform

Oracle Managed Database as a Service

Oracle Exadata Database Service on Dedicated Infrastructure

Cache Database

cache database

Oracle Zero Latency Cache database

Oracle Exadata Database Service on Cloud@Customer

Oracle Coherence

Oracle Key Vault

OCI Vault

Oracle Cloud Migrations

Oracle Exalogic

Exalogic

Oracle WebLogic Server

Oracle Cloud Infrastructure Health Checks

OCI Health Checks

OCI Dedicated Region

Oracle Cloud Infrastructure Dedicated Region

OracleDB for Azure

Import a File into Oracle Content Management

Oracle Cloud Infrastructure Streaming

Oracle Cloud Infrastructure Queue

Oracle Cloud Infrastructure Workflow

Oracle Autonomous Database Serverless

Oracle Autonomous Data Guard

Oracle E-Business Suite

E-Business Suite

Oracle Private Cloud Appliance

Private Cloud Appliance

Oracle Cloud Infrastructure Data Labeling

OCI Data Labeling

Oracle Cloud Infrastructure Vision

OCI Vision

Oracle Communications Session Border Controller

Oracle Cloud Infrastructure DevOps

OCI DevOps

Oracle Cloud Infrastructure Artifacts Registry

OCI Artifacts Registry

OCI Registry

Oracle Zero Latency Cache Cloud Service

Oracle Zero Latency Cache

ZLC

Oracle Autonomous Database

Autonomous Database

Oracle Data Guard

Data Guard

Oracle Database Service for Microsoft Azure

Oracle Interconnect for Microsoft Azure

Oracle APLG

Oracle Cloud Infrastructure for Microsoft Azure

OCI for Azure

Oracle Fleet Monitoring

Oracle Base Database Service

Oracle Machine Learning for R

Oracle Machine Learning for SQL

Oracle Machine Learning for Python

Oracle CX Commerce

Oracle Commerce

Oracle Cloud Infrastructure Process Automation

Oracle Access Governance

Oracle Cloud Infrastructure Lakehouse

Oracle Cloud Infrastructure AI Services

Oracle MySQL HeatWave

Disaster Recovery

Recovery Service

Oracle Database Autonomous Recovery Service

Fusion CX Analytics

Oracle Fusion CX Analytics

Oracle Cloud EPM Planning

Oracle EPM Planning

Oracle Analytics Semantic Modeler

Semantic Modeler

Fusion SCM Analytics

Oracle Fusion SCM Analytics

Oracle Functional Setup Manager

Functional Setup Manager

Oracle Fusion Cloud Financial Management

Oracle Financial Management

Oracle Fusion Cloud Applications Suite

Fusion Applications Suite

Oracle Fusion Cloud Project Management

Oracle Project Management

Oracle Fusion Cloud Procurement

Oracle Procurement

Oracle Fusion Cloud Self Service Procurement

Oracle Self Service Procurement

Oracle Fusion Cloud Supply Chain Planning

Oracle Supply Chain Planning

Oracle Fusion Cloud Supply Planning

Oracle Manufacturing

Oracle Fusion Cloud Order Management

Oracle Order Management

Oracle Fusion Cloud Enterprise Performance Management

Oracle Cloud Enterprise Performance Management

Oracle Fusion Cloud Inventory Management

Oracle Inventory Management

Oracle Maintenance

Oracle Fusion Cloud Product Lifecycle Management

Oracle Product Lifecycle Management

Oracle Fusion Cloud Product Management

Oracle Product Management

Oracle Service

Oracle Marketing

Oracle Configure, Price, Quote

Oracle Cloud HCM

Oracle Talent Management

Oracle Cloud SCM

Oracle Cloud ERP

Oracle Managed Cloud Self-Service Platform

Oracle WebLogic Server for Oracle Cloud Infrastructure

Query Service

Oracle Query Service

Oracle WebLogic Server for OCI

OTBI

Oracle Transactional Business Intelligence

Oracle B2C Service

Oracle Fusion Service

Oracle Fusion Cloud Service

Process Automation

APEX Service

Oracle APEX Application Development

Oracle SOA Suite on Marketplace

Oracle Machine Learning Notebooks

Oracle MySQL Database Service

Oracle Cloud Infrastructure Notifications

Oracle WebLogic Server for OKE

Oracle NetSuite Analytics Warehouse

NetSuite Analytics Warehouse

Oracle Fusion HCM Analytics

Fusion HCM Analytics

Publisher

Oracle Analytics Publisher

Autonomous JSON Database

Oracle Autonomous JSON Database

Oracle Cloud Infrastructure Full Stack Disaster Recovery Service

Oracle Cloud Infrastructure Vault

Oracle Cloud Infrastructure Events

Oracle Cloud Infrastructure Functions

Oracle Warehouse Management Cloud Service

Oracle WebLogic Server for Oracle Cloud Infrastructure Kubernetes Engine

Oracle Analytics

Oracle Cloud Infrastructure Data Integration

Oracle Cloud Infrastructure Data Migration

Oracle Cloud Infrastructure Data Science

Data Flow

Data Catalog

Oracle Visual Builder Studio

Fusion Data Intelligence

Oracle Fusion Cloud Applications

Fusion Applications

Essbase

Fusion ERP Analytics

Oracle Fusion ERP Analytics

Oracle Fusion Data Intelligence

Data Science

Data Integration

Data Migration

Analytics Cloud

API Platform Cloud Service

Autonomous Data Warehouse

Autonomous Transaction Processing

Big Data Cloud

Big Data Service

Cloud SQL

Compute Classic

Console

Container Classic

Oracle Fusion Cloud Configure, Price, Quote

Data Loader

Data Modeler

Oracle Analytics Desktop

Database Backup Cloud Service

Database Classic Cloud Service

Day by Day

Digital Assistant

Exadata Express

FastConnect Classic

Free Database Service

global header

GoldenGate for Big Data

Mobile Hub

MySQL Cloud Service

OPCM

Oracle Analytics Cloud

Oracle Analytics Cloud - Classic

Oracle Analytics Cloud – Essbase

Oracle Analytics Day by Day

Oracle Analytics Synopsis

Oracle API Manager Cloud Service

Oracle Application Container Cloud Service

Oracle APEX

Oracle Autonomous Data Warehouse

Oracle Autonomous Transaction Processing

Oracle BI Cloud Service

Oracle BI Cloud Service Data Sync

Oracle BI Cloud Service REST API

Oracle BI Day by Day Mobile

Oracle BI EE

Oracle BI Enterprise Edition

Oracle BI Presentation Catalog

Oracle Big Data Cloud

Oracle Big Data Cloud at Customer

Oracle Big Data Cloud Service

Oracle Big Data Preparation Cloud Service

Oracle Big Data Service

Oracle Big Data SQL Cloud Service

Oracle Blockchain Platform

Oracle Business Intelligence

Oracle Business Intelligence Cloud Service

Oracle Business Intelligence Data Visualization

Oracle Business Intelligence Day by Day Mobile

Oracle Cache Cloud Service

Oracle Cloud

Oracle Cloud Application Services

Oracle Cloud at Customer

Oracle Cloud Infrastructure

Oracle Cloud Infrastructure Archive Storage

Oracle Cloud Infrastructure Audit

Oracle Cloud Infrastructure Block Volumes

Oracle Cloud Infrastructure Classic

Oracle Cloud Infrastructure Compute

Oracle Cloud Infrastructure Compute Classic

OCI Kubernetes Engine

Oracle Cloud Infrastructure Container Registry Classic

Oracle Cloud Infrastructure Container Service Classic

Oracle Cloud Infrastructure Database

Oracle Cloud Infrastructure FastConnect

Oracle Cloud Infrastructure FastConnect Classic

Oracle Cloud Infrastructure File Storage

Oracle Cloud Infrastructure Identity and Access Management

Oracle Cloud Infrastructure Load Balancing

Oracle Cloud Infrastructure Load Balancing Classic

Oracle Cloud Infrastructure Monitoring

Oracle Cloud Infrastructure Networking

Oracle Cloud Infrastructure Object Storage

Oracle Cloud Infrastructure Object Storage Classic

Oracle Cloud Infrastructure Ravello Service

Oracle Cloud Infrastructure Registry

Oracle Cloud Infrastructure Resource Manager

Oracle Cloud Infrastructure Storage Gateway

Oracle Cloud Infrastructure Web Application Firewall

Oracle Cloud Marketplace

Oracle Cloud Partner Portal

Oracle Cloud Platform Services

Oracle Cloud service

Oracle Cloud Service

Oracle Cloud Service for .NET

Oracle Cloud SQL

Oracle Cloud Stack

Oracle Fusion Cloud Configure, Price, Quote

Oracle Container Engine for Kubernetes

Oracle Content Management

Oracle Advertising and Customer Experience

Oracle Advertising and CX

Oracle Customer Insights and Engagement Cloud Service

Oracle Data as a Service for Customer Service

Oracle Data as a Service for Marketing

Oracle Data as a Service for Sales

Oracle Cloud Infrastructure Data Catalog

Oracle Data Cloud

Oracle Cloud Infrastructure Data Flow

Oracle Data Hub Cloud Service

Oracle Data Integration Platform Cloud

Oracle Data Integration Platform Cloud, Enterprise

Oracle Data Integration Platform Cloud, Governance

Oracle Data Integration Platform Cloud, Standard

Oracle Data Integrator

Oracle Data Integrator Cloud

Oracle Data Intelligence Platform Cloud

Oracle Data Lake Engine

Oracle Data Safe

Oracle Data Security Cloud Service (DSCS)

Oracle Data Visualization Cloud Service

Oracle Analytics Desktop

Oracle Database Backup Cloud Service

Oracle Database Cloud Schema Service

Oracle Database Classic Cloud Service

Oracle Database Exadata Cloud at Customer

Oracle Exadata Database Service

Oracle Database Exadata Express Cloud Service

Oracle Database Free Cloud Service

Oracle Developer Cloud Service

Oracle Developer Cloud Service (Traditional)

Oracle Digital Assistant

Oracle Enterprise Asset Management

Oracle Enterprise Data Quality

Oracle Fusion Cloud Enterprise Resource Planning

Oracle Essbase

Oracle Event Hub Cloud Service

Oracle Event Hub Cloud Service - Dedicated

Oracle Event Hub Cloud Service - Platform

Oracle Event Hub Cloud Service - Topics

Oracle Fusion Cloud Field Service

Oracle Financial Consolidation and Close Cloud Service

Oracle Fusion Cloud Financials

Oracle Fusion Cloud Applications

Oracle Fusion Cloud Manufacturing

Oracle Fusion Middleware Control

Oracle Fusion Cloud Human Resources

Oracle Human Resources

Oracle Fusion Cloud Human Resources

Oracle GoldenGate

Oracle GoldenGate Cloud Service

Oracle GoldenGate for Big Data

Oracle Fusion Cloud Human Capital Management

Oracle Identity Cloud Service

Oracle Identity Management

Oracle Integration

Oracle Integration Classic

Oracle Integration Management Service

Oracle Integration Cloud Service

Oracle Intelligent Track and Trace

Oracle Internet of Things (IoT) Asset Monitoring Cloud Service

Oracle Internet of Things (IoT) Connected Worker Cloud Service

Oracle Internet of Things (IoT) Fleet Monitoring Cloud Service

Oracle Internet of Things (IoT) Production Monitoring Cloud Service

Oracle Internet of Things (IoT) Service Monitoring for Connected Assets Cloud Service

Oracle Fusion Cloud IoT Intelligent Applications

Oracle Java Cloud Service

Oracle Java Cloud Service - SaaS Extension

Oracle Java Cloud Service - SaaS Extension Control

Oracle Java SE Cloud Service

Oracle Java SE Monitoring and Management Cloud Service

Oracle JavaScript Extension Toolkit

Oracle JET

Oracle Key Management Cloud Service

Oracle Machine Learning

Oracle Fusion Cloud Maintenance

Oracle Managed File Transfer Cloud Service

Oracle Management Cloud

Oracle CX Marketing

Oracle Messaging Cloud Service

Oracle Mobile Cloud Service

Oracle Mobile Cloud, Enterprise

Oracle Mobile Hub

Oracle MySQL Cloud Service

Oracle Nimbula

Oracle Node Cloud Service

Oracle NoSQL Database Cloud Service

Oracle PaaS

Oracle Planning and Budgeting Cloud Service

Oracle Platform as a Service

Oracle Intelligent Advisor

Oracle Process Cloud Service

Oracle Fusion Cloud Procurement

Oracle Product Value Chain Cloud

Oracle Fusion Cloud Project Management

Oracle Public Cloud Services

Oracle Fusion Cloud Sales Automation

Oracle Sales Automation

Oracle Sales Cloud user interface

Oracle Fusion Cloud B2C Service

Oracle Shared File Service

Oracle SOA

Oracle SOA Cloud Service

Oracle Social Data and Insight Cloud Service

Oracle SQL Developer

Oracle Fusion Cloud Supply Chain & Manufacturing

Oracle Talent Management Cloud

Oracle Fusion Cloud Talent Management

Oracle Taleo Enterprise Cloud Service

Oracle Transactional Business Intelligence Enterprise

Oracle Transportation Management Cloud Service

Oracle Value Chain Execution Cloud

Oracle Visual Builder

Oracle WebLogic Server for OCI

OTBI Enterprise

PaaS

Platform as a Service

Presentation Services

Process

Ravello

Registry Classic

SaaS

Schema Service

Software as a Service

SSI