1. System Overview
  2. Introduction to Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack

1 Introduction to Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack

Oracle Exadata Database Machine is engineered to be the highest performing and most available platform for running Oracle Database.

Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack are hardware systems that consist of multiple components. This chapter provides an overview of Exadata systems, and usage information.

Note:

For ease of reading, the name "Oracle Exadata Rack" is used when information refers to both Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack.

Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack are described in the following topics:

1.1 Oracle Exadata Architecture

Oracle Exadata is a modern architecture featuring scale-out industry-standard database servers, scale-out intelligent storage servers, and an extremely high speed internal RDMA Network Fabric that connects the database and storage servers.

Oracle Exadata runs all types of database workloads including Online Transaction Processing (OLTP), Data Warehousing, and a consolidation of mixed workloads. Oracle Exadata powers and protects the most important databases, and is the ideal foundation for a consolidated database cloud.

Oracle Exadata includes all the hardware needed to run Oracle Database. The database servers, storage servers and network are preconfigured, pretuned and pretested by Oracle engineers, eliminating the weeks of effort typically required to deploy a high performance system. Extensive end-to-end testing ensures all components work together and there are no performance bottlenecks or single points of failure that can affect the complete system. Oracle Exadata also includes the Oracle Exadata System Software, which runs on both database servers and storage servers. Unique software algorithms in Oracle Exadata System Software implement database intelligence in storage, PCI-based flash, and the RDMA Network Fabric network to deliver higher performance and high capacity at lower costs than other platforms.

Parent topic: Introduction to Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack

1.2 Oracle Exadata Configurations

Oracle Exadata can come in a specific configuration, for example the half rack, or use an elastic configuration with a custom number of database servers and storage servers.

Oracle Exadata uses a scale-out architecture for both database servers and storage servers. As Oracle Exadata grows, more database CPUs, storage and networking are added in a balanced fashion, ensuring scalability without bottlenecks.

Early Oracle Exadata models had fixed configurations, such as quarter-rack or half-rack.

Contemporary Oracle Exadata systems use custom configurations called elastic configurations. Elastic configurations enable you to build an Oracle Exadata Rack containing a custom number of database servers and storage servers that meet your needs. See Elastic Configuration for details.

In addition to upgrading within a rack, multiple racks can be connected using the integrated RDMA Network Fabric to form larger configurations. For example, a system composed of four full Exadata racks is essentially four times more powerful than a single rack system. The four racks provide quadruple the I/O throughput, storage capacity and processors. The multi-rack system can be configured as a single large system or logically partitioned for consolidation of multiple databases.

When larger storage capacity is required, Oracle Exadata Storage Expansion Rack is available. Oracle Exadata Storage Expansion Rack enables growth of storage capacity and bandwidth for any Oracle Exadata. It is designed for database deployments that require very large amounts of data including historical or archival data, backups, documents, images, XML, large objects (LOBs), and so on. Oracle Exadata Storage Expansion Rack starts with a base configuration that includes four storage servers, and you can add as many additional storage servers as you need, up to the physical capacity of the rack. The servers in the Oracle Exadata Storage Expansion Rack connect to Oracle Exadata using the integrated RDMA Network Fabric.

All Oracle Exadata implementations use time-tested configuration standards, so customers benefit from the experience of other users who have deployed Oracle Exadata for their mission-critical applications.

Oracle Exadata runs standard Oracle Database software. Any applications that run with Oracle Database can be seamlessly migrated to Oracle Exadata with no changes to the application.

Parent topic: Introduction to Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack

1.2.1 Elastic Configuration

Elastic configuration enables you to customize Oracle Exadata Rack within set limits. For example, using elastic configuration, Oracle Exadata Database Machine X9M-2 can have 0 to 22 database servers, 0 to 19 Exadata Storage Servers, or a combination of database servers and Exadata Storage Servers.

Note:

Oracle Exadata Rack collectively refers to Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack.

You can use Oracle Exadata Configuration Assistant (OECA) to quickly and easily define an elastic configuration. OECA automatically validates each Oracle Exadata Rack configuration and produces a report that contains the associated environmental specifications (power requirements, cooling requirements, rack weight, and so on).

OECA is available on Oracle Technology Network: 

https://www.oracle.com/database/technologies/oeca-download.html

Parent topic: Oracle Exadata Configurations

1.3 Oracle Exadata Database Machine Performance Features

Oracle Exadata Database Machine not only provides high performance and scalability but also unique technology.

The Exadata database servers can offload data-intensive SQL operations to Oracle Exadata Storage Servers. Moving the SQL processing to Oracle Exadata Storage Servers enables data filtering and processing to occur immediately and in parallel across all storage servers as data is read from disk. Exadata storage offload reduces database server CPU consumption and greatly reduces the amount of data moved between the storage and database servers.

Compressing data can provide a dramatic reduction in the storage consumed for large databases. Oracle Exadata Storage Server provides a very advanced compression capability called Hybrid Columnar Compression. Hybrid Columnar Compression enables the highest levels of data compression and provides tremendous cost savings and performance improvements due to reduced I/O. On conventional systems, enabling high data compression has the drawback of reducing performance. Oracle Exadata Database Machine is able to offload decompression overhead into a large number of processors in Oracle Exadata Storage Servers so most workloads run faster using Hybrid Columnar Compression than they do without it. Hybrid Columnar Compression combines the compression and analytic performance benefits of column storage while avoiding the dramatic slowdown that pure columnar stores experience for drill-down operations.

Oracle Exadata Database Machine uses the latest PCIe flash technology rather than flash disks. PCIe flash greatly accelerates performance by placing flash directly on the high speed PCIe bus rather than behind slow disk controllers and directors. Refer to Hardware Components of Oracle Exadata for information about flash card configurations and capacity.

Exadata Smart Flash Cache automatically caches frequently accessed data in PCIe flash while keeping infrequently accessed data on disk drives. This provides the performance of flash with the capacity and cost of disk. Exadata Smart Flash Cache understands database workloads and when to avoid caching data that will never be reused or fit in the cache. Oracle Database and Oracle Exadata System Software allow the user to provide directives at the database table, index and segment level to ensure that specific data is retained in flash. Tables can be retained in flash without the need to move the table to different tablespaces, files or LUNs as would have to be done with traditional storage and flash disks.

Note:

Oracle Exadata Database Machine does not include any Oracle software licenses. Appropriate licensing of the following software is required when Oracle Exadata Database Machinee is used as a database server:

  • Oracle Database

  • Oracle Exadata System Software

In addition, Oracle recommends that the following software is licensed:

  • Oracle RAC

  • Oracle Partitioning

Parent topic: Introduction to Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack

1.4 Oracle Exadata Rack Limits and Restrictions

Parent topic: Introduction to Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack

1.4.1 Restrictions for Modifying Oracle Exadata Rack Hardware

Modification of Oracle Exadata Rack is generally not required or recommended, and unpermitted modifications can result in loss of warranty and support. However, the following hardware modifications are permitted:

  • Customers may perform the following modifications to the included Management Network Switch, which supports the administration network. Modification is only allowed after delivery of the Oracle Exadata Rack, and the customer bears responsibility for all costs and issues that may arise from any change.

    Customers may:

    • Replace the Management Network Switch with an equivalent customer-supplied 1U Gigabit Ethernet switch.

    • Reconfigure the Management Network Switch settings and firmware.

    • Remove the Management Network Switch and cable the administration network connections directly to an external customer-supplied switch or patch panel.

  • Subject to validation using Oracle Exadata Configuration Assistant (OECA), customers may install additional equipment into unused rack slots.

    Use the ADD EQUIPMENT button in OECA to specify the additional equipment details.

    Additional equipment may only be installed after the resulting configuration is validated using OECA.

Related Topics

Parent topic: Oracle Exadata Rack Limits and Restrictions

1.4.2 Restrictions for Modifying the Oracle Exadata Software Components

The following restrictions apply to software modifications to Oracle Exadata Rack. Violating these restrictions can result in loss of warranty and support.

  • Oracle Exadata System Software and the operating systems cannot be modified, and you cannot install any additional software or agents on the Oracle Exadata Storage Servers.

  • You cannot update the firmware directly on the Oracle Exadata Storage Servers. The firmware is updated as part of an Oracle Exadata Storage Server patch.

  • You can update the firmware of the other components of Oracle Exadata Rack.

    • You can update the OS and firmware versions on the Management Network Switch to meet their data center requirements.

    • You can update the firmware of the components of the database servers provided they comply with the restrictions documented in My Oracle Support Doc ID 888828.1, and its related notes.

    • You can update the firmware of the RDMA Network Fabric switches provided they comply with the validated versions documented in My Oracle Support Doc ID 888828.1, and its related notes.

  • You may load additional software on the database servers. However, to ensure best performance, Oracle discourages adding software except for agents, such as backup agents and security monitoring agents, on the database servers.

    Loading non-standard kernel modules on the operating system of the database servers is allowed but discouraged. Oracle will not support questions or issues with the non-standard modules. If a server crashes, and Oracle suspects the crash may have been caused by a non-standard module, then Oracle support may refer you to the vendor of the non-standard module or ask that the issue be reproduced without the non-standard module. Modifying the database server operating system other than by applying official patches and upgrades is not supported.

  • RDMA Network Fabric-related packages should always be maintained at the officially supported release.

  • You can install Oracle Enterprise Manager Cloud Control Management Agents or Oracle Enterprise Manager Grid Control Management Agents on the database servers to monitor the system.

  • Installing Oracle Enterprise Manager Ops Center agents to monitor the system is not allowed.

  • Oracle Exadata Deployment Assistant (OEDA) is used to specify the physical network interfaces that you want to bond for the client network and the additional networks, if configured. OEDA generates bonded network interfaces that amalgamate two physical network interfaces.

    Manual changes to the OEDA-generated bonding configuration are allowed but discouraged. Oracle will not support questions or issues associated with non-standard bonding configurations. In any event, bonding configurations having fewer than two networking interfaces are not permitted.

Related Topics

Parent topic: Oracle Exadata Rack Limits and Restrictions

1.4.3 Restrictions for Connecting Oracle Exadata with Other Hardware, Systems, or Networks

The following restrictions apply to hardware and software modifications to Oracle Exadata Rack. Violating these restrictions can result in loss of warranty and support.

  • Oracle Exadata Storage Expansion Rack can only be connected to Oracle Exadata Database Machine or Oracle SuperCluster, and only supports databases running on the database servers on Oracle Exadata Database Machine or Oracle SuperCluster.

  • Standalone Oracle Exadata Storage Servers can only be connected to Oracle Exadata Database Machine or Oracle SuperCluster, and only support databases running on the database servers of Oracle Exadata Database Machine or Oracle SuperCluster.

  • Customers cannot connect USB devices that draw more than 100 mA of power to the database servers.

  • Customers cannot connect USB devices to the Oracle Exadata Storage Servers except as documented in the Oracle Exadata documentation. In those documented situations, the USB device should not draw more than 100 mA of power.

  • Only switches specified for use in Oracle Exalogic Elastic Cloud, Oracle Big Data Appliance, and Oracle SuperCluster may be connected to the Oracle Exadata Rack RDMA Network Fabric network. It is not supported to connect other RDMA Network Fabric switches, including third-party switches, to the Oracle Exadata Rack RDMA Network Fabric network. Only the RDMA Network Fabric networking topologies specified in this guide and Oracle Exadata Database Machine Extending and Multi-Rack Cabling Guide are supported, and any other RDMA Network Fabric network topology is not supported.

    Note:

    When interconnecting racks, all racks must use the same type of RDMA Network Fabric, either InfiniBand or RDMA over Converged Ethernet (RoCE).

  • You may connect external servers, that are not part of Oracle Engineered Systems, to the RDMA Network Fabric switches in Oracle Exadata. However, it is your responsibility to upgrade and maintain the compatibility of the RDMA Network Fabric software of the external servers with the RDMA Network Fabric software release for Oracle Exadata. Oracle highly recommends that you maintain the same release of RDMA Network Fabric software and operating system on the external server as on Oracle Exadata. If an RDMA Network Fabric problem is encountered and an external server is connected, then you may be asked to remove the external server and reproduce the problem.

  • You can connect the database servers to external (non-Exadata) storage using iSCSI or NFS subject to the following:

    • The storage is allowed for application input and output files, Data Pump export files and logs, and files associated with backup and recovery operations.
    • The use of non-Oracle storage for Oracle Database files is not supported.
    • The Fibre Channel over Ethernet (FCoE) protocol is not supported.

Related Topics

Parent topic: Oracle Exadata Rack Limits and Restrictions

1.4.4 Maximum Number of Database Processes

This topic describes the maximum number of database processes on each Exadata database server or virtual machine guest.

The tables in this topic contain the maximum number of database processes supported on each Exadata database server.

In all cases, the best practice is to keep the process count below the specified values. Also, if a subset of the database workload uses parallel queries, the effective limit will be between the values in the "Maximum Number of Processes with No Parallel Queries" and "Maximum Number of Processes with All Running Parallel Queries" columns.

The following table shows the maximum number of database processes supported on each physical Exadata database server in a bare-metal system configuration.

Table 1-1 Maximum Number of Database Processes on a Physical Database Server

Machine Type RDMA Network Fabric Type Maximum Number of Processes with No Parallel Queries Maximum Number of Processes with All Running Parallel Queries

2-socket

RoCE Network Fabric

25,000

16,000

8-socket

 RoCE Network Fabric

64,000

44,000

2-socket

 InfiniBand Network Fabric

16,000

14,000

8-socket

 InfiniBand Network Fabric

64,000

44,000

The following table shows the maximum number of database processes supported on each virtual machine (VM) guest (Oracle VM DomU or KVM guest) in a virtualized system configuration.

Table 1-2 Maximum Number of Database Processes on a Virtual Machine Guest

Machine Type RDMA Network Fabric Type Maximum Number of Processes with No Parallel Queries Maximum Number of Processes with All Running Parallel Queries

2-socket

 RoCE Network Fabric without Exadata Secure RDMA Fabric Isolation configured

25,000

16,000

2-socket

 RoCE Network Fabric with Exadata Secure RDMA Fabric Isolation configured

50,000

32,000

2-socket

 InfiniBand Network Fabric

16,000

14,000

Use the following best practices to optimize resource utilization while supporting a high process count:

  • Establish database connections using a set of Oracle Net listeners instead of using local bequeath connections.

  • On each database server, the number of Oracle Net listeners should be a multiple of the number of server CPU sockets. For example, on an 8-socket database server, you should configure at least eight listeners.

  • Evenly spread the Oracle Net listeners across the available CPU sockets.

    For example, assuming the listener.ora file is configured correctly for eight listeners (named LISTENER0 - LISTENER7), you can use the following script to start eight listeners on an 8-socket server with each listener bound to a different CPU socket: 

    #!/bin/bash
export ORACLE_HOME=/u01/app/oracle/product/19.0.0.0/dbhome_1
for socket in `seq 0 7`
do
 numactl --cpunodebind=${socket} $ORACLE_HOME/bin/lsnrctl start LISTENER${socket}
done

  • Use Oracle Net connection rate limiting to control connection storms and enhance system stability.

    To avoid excessive client connection timeouts and server-side errors, limit the connection rate on each database server to 400 connections per second. In other words, the sum of the rate_limit control parameter should not exceed 400 across all Oracle Net listeners on each database server.

    See Connection Rate Limiter Parameters in Oracle Database Net Services Reference.

  • Use the following recommended Oracle Net listener control parameter settings to avoid Oracle Net connection errors, such as TNS-12514.

    • MAX_ALL_CONNECTIONS_listener_name=4096

    • MAX_REG_CONNECTIONS_listener_name=2048

    See Control Parameters in Oracle Database Net Services Reference.

Parent topic: Oracle Exadata Rack Limits and Restrictions