Modern App Development - Data-Centric

Data-centric applications are core to modern enterprises, which routinely collect vast amounts and variety of operational data from business systems and devices, running billions of transactions per day. Data integrity and data consistency across all data is central to running any business. Databases have traditionally simplified these data-centric operations at scale.

Data-centric applications utilize the application services (queueing, events, caching, machine learning, etc.) and data services of the modern converged database. This pattern is particularly suitable for applications where the data is key to the functionality.
By using the application and data services of the converged database, the application is greatly simplified since it no longer needs to integrate and orchestrate data movement across multiple heterogeneous services.

Data-centric applications access, store, and process data all the time, and data gravity pulls app services closer to the data, near the data services in the database. It's much easier to process and analyze data securely and at scale in the database. In contrast, conventional wisdom is to extract data, process it externally, and then store it back into special-purpose data stores. This leads to complexity across the spectrum, often causing scalability problems and data security breaches.

Design Principles

  • Use converged databases with full featured support across all data and use application services built in to the converged Oracle Database

    Build data-centric applications using the converged Oracle Database features in Oracle Autonomous Database. This lets you store all the data types in one technology, instead of fragmentting your data in multiple single-purpose databases. Support transaction processing and data warehousing across all data types (JSON, key-value, relational, text, spatial, and graph) in a converged database to keep all data consistent at all times.

    This architecture is simpler, higher performance, and more secure. The higher performance results from eliminating the need to transfer and transform data among disparate single-purpose data stores. And all the data enjoy the comprehensive security mechanisms of the Oracle Autonomous Database.

    In addition to storing all your data, the Oracle Autonomous Database can also offer a converged application architecture. Instead of using a highly distributed architecture that calls distinct services for search, events, machine learning, and caching, a data-centric application can simply use the application services provided by the database that is already used to store the data. Use the text search, event system, and machine learning built into the database. Deploy Active Data Guard as a consistent, self-refreshed read cache.

    Data-centric applications that are event-driven should use Transactional Event Queues (TEQ) in Oracle Autonomous Database. This keeps the event state in the database and also provides transactional messaging.

    Data gathered through business transactions, IoT, and streams can have a single source of truth, for deeper insights with rich query processing in Oracle Autonomous Database.

    Oracle Autonomous Database can elastically scale using Oracle Real Application Clusters (RAC). In addition, you can use Oracle Sharding for massive scale-out and geographically distributed global database and data residency use cases.

  • Keep application tier stateless

    Keep the application and mid-tiers statelss by storing all state in in the Oracle Autonomous Database. Keeping the application tiers statelss simplifies life cycle management, high availability, and disaster recovery.

    Applications can maintain cached state outside the database. Any such state must be capable of being reconstituted from the state stored in the database.

  • Build applications as services that communicate through APIs

    Use REST APIs (Oracle REST Data Services) for external interfaces. UI and client-side business logic use REST to make requests to the server-side logic to invoke services from external applications (for example, integration workflows). Manage the REST APIs across services with Oracle Cloud Infrastructure API Gateway.

    The services themselves should be built with bounded contexts and loose coupling, communicating with Oracle Transactional Event Queues (TEQ) messaging in Oracle Autonomous Database using Java Message Service (JMS) APIs. Independent services communicate through messaging APIs.

  • Eliminate single points of failure through automated data replication and failure recovery

    Keeping the application state in the database allows us to simplify data replication and failure recovery. Since all the state is in the database, the entire application can take advantage of the high availability and disaster recovery capabilities of the Oracle Autonomous Database to eliminate planned and unplanned downtime. Furthermore, this approach eliminates the difficulty of recovering the entire application to a consistent point from multiple replication technologies.

    The Oracle Autonomous Database supports RAC and Autonomous Data Guard in an active-active architecture in which both the primary and backup systems can be used. Autonomous Data Guard will automatically replicate data within and across OCI regions for disaster recovery.

  • Use fully managed services to eliminate complexity across application development, runtimes and data management

    Use a fully-managed Oracle Autonomous Database and services in the database to simplify the development and deployment of data-centric applications. Oracle Autonomous Database is a fully and autonomously managed database with a set of rich applications and data services. Use either the shared or the dedicated managed service, based on your deployment requirements of service isolation and performance needs. The shared service is multitenant and works for most use cases. Applications that require strong isolation and high performance should use the dedicated service.

  • Instrument end-to-end monitoring and tracing

    You can use multiple monitoring tools for data-centric apps, to monitor data-tier events and metrics for application debugging and optimization. Most monitoring is at the system or app tier, making data-tier monitoring a secondary consideration. Integrate the application services and data services in the database, when possible, to get a consolidated view of the data associated with each application state. Troubleshoot problems related to a lack of data consistency, and the application code written to handle these inconsistencies, with confidence by using end-to-end monitoring for data-centric applications. Use Oracle Cloud Infrastructure Monitoring in Oracle Autonomous Database for database-level metrics, Application Performance Monitoring for application tracing and database tracing, and the Oracle Autonomous Database monitoring console for Automatic Workload Repository (AWR)-style reports for the database.

  • Use lightweight open-source frameworks and mature programming languages

    Develop data-centric applications in Java, deploy them in Docker containers, manage them with Oracle Cloud Infrastructure Container Engine for Kubernetes, and use the data integrity and rich analytical queries in SQL. By using more application services in the database, the need for other programming languages is reduced. Using JavaScript and PL/SQL procedures pushes much of the application processing to the data. Also, machine learning notebooks for Python run the machine learning models in the database without requiring a separate service.

  • Package and ship applications and data in containers

    To build data-centric applications, use Docker containers for applicationss and database containers for data. A database container usually contains many pluggable databases (PDBs), and this works well with the microservices model, where a microservice can isolate its data in a PDB. Use Oracle Cloud Infrastructure Container Engine for Kubernetes to manage application containers, register images in Oracle Cloud Infrastructure Registry, and use Oracle Database Kubernetes operators to unify the management of applicationss and the database with Kubernetes. Oracle Database provisioning and management can be simplified by including it in the singular setup with Kubernetes and kubectl commands. As an alternative, use CLI commands or Oracle Cloud Infrastructure Resource Manager to manage Oracle Autonomous Database.

  • Automate build, test, and deployment

    Oracle Database is containerized and has a container database that contains multiple PDBs. Build data-centric applications with rapid provisioning of databases (PDBs) in Oracle Autonomous Database for microservices, and shorten the deployment and upgrade cycles for data-centric applications by using the OCI DevOps service. Keep application service containers and data in PDBs aligned (for example, one PDB per microservice) to prevent fragmented and complex data access patterns that delay testing, development, and deployment and often result in an inconsistent state.

    Treat declarative configurations and programs (metadata and SQL) as code. Metadata changes must be described, labeled, and tracked in a source control system, and applied incrementally.

  • Implement a defense-in-depth approach to secure applications and data

    Use the security features built into Oracle Autonomous Database for data-centric applications. For access and authentication controls for the data tier, use HTTPS/TLS for external access to the load balancer and from the load balancer to each microservice. Store database user credentials in Oracle Database Vault. Use authentication and authorization where necessary. Other common security practices include rotation of wallets and passwords, Oracle Cloud Infrastructure Vault storage of secrets and wallets, code scan and analysis to reduce security vulnerabilities, reduction of surface area by overall microservices design, and using Oracle Data Safe to measure security posture.

Architecture

Oracle Autonomous Database provides various application and data services to use for data-centric applications. The database provides a rich set of application development services for data-centric applications: low code, CI/CD of metadata, data loading, data caching, text search, graph analysis, spatial analysis, machine learning, workflows, transforms, microservices patterns, event handling and notifications, and others. The data services provide security, data defense, high availability, and scaling, along with the ability to store JSON and other types of data.

For cases where application services aren’t available in the database, build services by using Helidon or Micronaut, using Java, and accessing the database through JDBC connection pooling.

The following diagram illustrates this architecture pattern.


The data-centric architecture has the following components:

  • Oracle Autonomous Database

    Oracle Autonomous Database runs natively on Oracle Cloud Infrastructure while providing workload-optimized cloud services for transaction processing and data warehousing. Reduce operational costs by up to 90% with a multimodel converged database and machine learning-based automation for full lifecycle management.

  • Oracle REST Data Services (ORDS)

    Oracle REST Data Services (ORDS) bridges HTTPS and the Oracle Database. A mid-tier Java application, ORDS provides a Database Management REST API, SQL Developer Web, a PL/SQL Gateway, SODA for REST, and the ability to publish RESTful Web Services for interacting with the data and stored procedures in your Oracle Database.

  • Oracle Application Express (APEX)

    APEX is a low-code development platform that enables you to build scalable, secure enterprise applications, with world-class features, that can be deployed anywhere. Oracle APEX is the world's most popular low-code platform for enterprise applications.

  • Oracle Transactional Event Queues (TEQ)

    TEQ and Advanced Queuing (AQ) are robust and feature-rich message queuing systems integrated with Oracle Database. When web, mobile, IoT, and other data-driven and event-driven applications stream events, or communicate with each other as part of a workflow, producer applications enqueue messages and consumer applications dequeue messages.

  • Oracle JSON Document Database

    Oracle Database fully supports schemaless application development using the JSON data model. This allows for a hybrid development approach: all of the schema flexibility and speedy application development of NoSQL document stores, combined with all of the enterprise-ready features in Oracle Database.

  • Oracle's Spatial Database

    Oracle's spatial database is included in Oracle's converged database, allowing developers and analysts to get started easily with location intelligence analytics and mapping services. It enables Geographic Information System (GIS) professionals to successfully deploy advanced geospatial applications. Organizations can manage different types of geospatial data, perform hundreds of spatial analytic operations, and use interactive map visualization tools with the spatial features in Oracle Autonomous Database and Oracle Database.

  • Oracle Graph Studio

    Oracle Autonomous Database includes Graph Studio, with one-click provisioning, integrated tooling, and security. Graph Studio automates graph data management and simplifies modeling, analysis, and visualization across the graph analytics lifecycle.

  • Oracle Machine Learning

    Oracle Machine Learning, included in Oracle Autonomous Database, uncovers hidden patterns and insights in enterprise data, generating new value for the business. Oracle Machine Learning accelerates the creation and deployment of machine learning models for data scientists using reduced data movement, AutoML technology, and simplified deployment.

  • Container Engine for Kubernetes

    Oracle Cloud Infrastructure Container Engine for Kubernetes is a fully managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud. You specify the compute resources that your applications require, and Container Engine for Kubernetes provisions them on Oracle Cloud Infrastructure in an existing tenancy. Container Engine for Kubernetes uses Kubernetes to automate the deployment, scaling, and management of containerized applications across clusters of hosts.

  • Virtual cloud network (VCN) and subnet

    A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.

  • Load balancer

    The Oracle Cloud Infrastructure Load Balancing service provides automated traffic distribution from a single entry point to multiple servers in the back end.

  • API gateway

    Oracle API Gateway service enables you to publish APIs with private endpoints that are accessible from within your network, and which you can expose to the public internet if required. The endpoints support API validation, request and response transformation, CORS, authentication and authorization, and request limiting.

Considerations

These are some additional considerations to keep in mind.

  • Design for Modern Three-Tier Model

    The classic three-tier model has evolved. Most application logic now runs in the client using JavaScript, which enables fast user interfaces by avoiding network roundtrips to cloud data centers. This is beneficial for cloud providers, who can shift the CPU cost of running application logic to clients’ smartphones or desktops.

    Design your application for a lightweight mid-tier. Maximize client-side processing by using JavaScript. Server-side application processing should be done in the database, using database-resident services such as caching, workflow, integration, search, events, and queues. The mid-tier should have minimal (ideally, zero) application logic, no application state, and be primarily used as a front end to proxy and pool connections to the database.

  • Adopt Metadata-Driven Low-Code Approach

    Specify application logic declaratively where possible, and write code only where needed.

    Use database metadata to describe the model that drives everything. Development is declarative, starting with the database data model or spreadsheet. Use the metadata to describe business logic, business rules, application execution, and customizations. And the model is easily customizable (by Oracle and end-users) because it is driven “live” by application metadata.

    This approach works directly against data in the database by using SQL, JSON, or graph. It also provides consistency across all the data, and eliminates the complexity, overhead, and bugs of external Object Relational (OR) mapping software.

  • Apply CI/CD Methodologies to Metadata

    Treat declarative configurations and programs (metadata and SQL) as code. Metadata changes must be described, labeled, tracked in a source control system, and applied incrementally. Some tools also support diffs and rollback. Use tools like Liquibase and Flyway. Both are open source, and Liquibase is integrated into Oracle SQL Developer Command Line (SQLcl).

  • Customize Using Metadata Instead of Code

    A metadata-driven approach simplifies customizations. It makes it easier to add custom fields and attributes on the UI that are stored and searchable, and customizations can survive application upgrades.

    A metadata-driven approach also simplifies upgrades that involve applying or merging the customizations with changes in the base release. Avoid customization via code because it’s harder to merge when there are updates. It’s much easier to diff and merge over a data dictionary than processing lines of code.

Alternatives and Antipatterns

Consider the alternatives to the architecture described in this pattern, and avoid attempting to implement antipatterns.

Alternatives:

  • OCI-managed services for specialized databases along with other OCI application development and integration managed services
  • Specialized databases with Oracle GoldenGate replication for consistency across multiple data stores

Antipatterns:

  • Keeping state in the mid-tier for applications
  • Using at-least-once messaging and relying on developers to write de-duplication code
  • Moving data between specialized data stores for specific processing