1 Introduction to Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack

Oracle Exadata Database Machine is engineered to be the highest performing and most available platform for running Oracle Database.

Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack are hardware systems that consist of multiple components. This chapter provides an overview of Exadata systems, and usage information.

Note:

For ease of reading, the name "Oracle Exadata Rack" is used when information refers to both Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack.

Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack are described in the following topics:

1.1 Oracle Exadata Database Machine Architecture

Oracle Exadata Database Machine is a modern architecture featuring scale-out industry-standard database servers, scale-out intelligent storage servers, and an extremely high speed internal RDMA Network Fabric that connects the database and storage servers.

Oracle Exadata Database Machine runs all types of database workloads including Online Transaction Processing (OLTP), Data Warehousing, and a consolidation of mixed workloads. Oracle Exadata Database Machine powers and protects the most important databases, and is the ideal foundation for a consolidated database cloud.

Oracle Exadata Database Machine includes all the hardware needed to run Oracle Database. The database servers, storage servers and network are preconfigured, pretuned and pretested by Oracle engineers, eliminating the weeks of effort typically required to deploy a high performance system. Extensive end-to-end testing ensures all components work together and there are no performance bottlenecks or single points of failure that can affect the complete system. Oracle Exadata Database Machine also includes the Oracle Exadata System Software, which runs on both database servers and storage servers. Unique software algorithms in Oracle Exadata System Software implement database intelligence in storage, PCI-based flash, and the RDMA Network Fabric network to deliver higher performance and high capacity at lower costs than other platforms.

1.2 Oracle Exadata Database Machine Configurations

Oracle Exadata Database Machine can come in a specific configuration, for example the half rack, or use an elastic configuration with a custom number of database servers and storage servers.

Oracle Exadata Database Machine uses a scale-out architecture for both database servers and storage servers. As Oracle Exadata Database Machine grows, more database CPUs, storage and networking are added in a balanced fashion, ensuring scalability without bottlenecks. The first Oracle Exadata Database Machine models (X2 and X3) had fixed configurations, such as the half rack or the quarter rack.

Now you can upgrade Oracle Exadata Database Machine to custom configurations called "elastic configurations". Elastic configurations enable you to build an Oracle Exadata Rack containing a custom number of database servers and storage servers that meet your needs. See Elastic Configurations for details.

In addition to upgrading within a rack, multiple racks can be connected using the integrated RDMA Network Fabric to form larger configurations. For example, a system composed of four Oracle Exadata Full Racks is four times as powerful as a single rack system. The four racks provide quadruple the I/O throughput, storage capacity and processors. It can be configured as a large single system or logically partitioned for consolidation of multiple databases. Oracle Real Application Clusters (Oracle RAC) can dynamically add more processing power, and Oracle Automatic Storage Management (Oracle ASM) can dynamically add more storage.

When larger storage capacity is required Oracle Exadata Storage Expansion Rack is available. Oracle Exadata Storage Expansion Rack enables growth of storage capacity and bandwidth for any Oracle Exadata Database Machine. It is designed for database deployments that require very large amounts of data including historical or archival data, backups, documents, images, XML, large objects (LOBs), and so on. Oracle Exadata Storage Expansion Rack starts with the quarter rack size, and you can add as many additional storage servers as you need. The servers in the Oracle Exadata Storage Expansion Rack connect to Oracle Exadata Database Machine using the integrated RDMA Network Fabric. Configuration does not require set up of logical unit numbers (LUNs) or mount points. Storage is configured and added to the database online using a command line interface.

All Oracle Exadata Database Machines are configured identically so customers benefit from the experience of other users who have deployed Oracle Exadata Database Machine for their mission-critical applications.

Oracle Exadata Database Machine runs standard Oracle Database. Any applications that run with Oracle Database can be seamlessly migrated to Oracle Exadata Database Machine with no changes to the application.

1.2.1 Elastic Configurations

Elastic configurations allow Oracle Exadata Racks to have customer-defined combinations of database servers and Exadata Storage Servers. For example, Oracle Exadata Database Machine X5-2 Elastic Configuration elastic configuration can have 0 to 22 database servers, 0 to 19 Exadata Storage Servers, or a combination of database servers and Exadata Storage Servers.

Note:

For ease of reading, the name "Oracle Exadata Rack" is used when information refers to both Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack.

To build an Oracle Exadata system with a custom number of database and storage servers, use Oracle Exadata Configuration Assistant (OECA). OECA is available on Oracle Technology Network:

http://www.oracle.com/technetwork/database/exadata/oeca-download-2817713.html

OECA simplifies the elastic configuration process. OECA facilitates scoping and analyzing elastic configurations and reporting environmental specifications.

1.3 Oracle Exadata Database Machine Performance Features

Oracle Exadata Database Machine not only provides high performance and scalability but also unique technology.

The Exadata database servers can offload data-intensive SQL operations to Oracle Exadata Storage Servers. Moving the SQL processing to Oracle Exadata Storage Servers enables data filtering and processing to occur immediately and in parallel across all storage servers as data is read from disk. Exadata storage offload reduces database server CPU consumption and greatly reduces the amount of data moved between the storage and database servers.

Compressing data can provide a dramatic reduction in the storage consumed for large databases. Oracle Exadata Storage Server provides a very advanced compression capability called Hybrid Columnar Compression. Hybrid Columnar Compression enables the highest levels of data compression and provides tremendous cost savings and performance improvements due to reduced I/O. On conventional systems, enabling high data compression has the drawback of reducing performance. Oracle Exadata Database Machine is able to offload decompression overhead into a large number of processors in Oracle Exadata Storage Servers so most workloads run faster using Hybrid Columnar Compression than they do without it. Hybrid Columnar Compression combines the compression and analytic performance benefits of column storage while avoiding the dramatic slowdown that pure columnar stores experience for drill-down operations.

Oracle Exadata Database Machine uses the latest PCIe flash technology rather than flash disks. PCIe flash greatly accelerates performance by placing flash directly on the high speed PCIe bus rather than behind slow disk controllers and directors. Refer to Hardware Components of Oracle Exadata Database Machine for information about flash card configurations and capacity.

Exadata Smart Flash Cache automatically caches frequently accessed data in PCIe flash while keeping infrequently accessed data on disk drives. This provides the performance of flash with the capacity and cost of disk. Exadata Smart Flash Cache understands database workloads and when to avoid caching data that will never be reused or fit in the cache. Oracle Database and Oracle Exadata System Software allow the user to provide directives at the database table, index and segment level to ensure that specific data is retained in flash. Tables can be retained in flash without the need to move the table to different tablespaces, files or LUNs as would have to be done with traditional storage and flash disks.

Note:

Oracle Exadata Database Machine does not include any Oracle software licenses. Appropriate licensing of the following software is required when Oracle Exadata Database Machinee is used as a database server:

  • Oracle Database

  • Oracle Exadata System Software

In addition, Oracle recommends that the following software is licensed:

  • Oracle RAC

  • Oracle Partitioning

1.4 Oracle Exadata Rack Limits and Restrictions

1.4.1 Restrictions for Modifying Oracle Exadata Rack Hardware

Modification of Oracle Exadata Rack is generally not required or recommended, and unpermitted modifications can result in loss of warranty and support. However, the following hardware modifications are permitted:

  • Customers may perform the following modifications to the included Cisco Ethernet switch, which supports the management network. Modification is only allowed after delivery of the Oracle Exadata Rack, and the customer bears responsibility for all costs and issues that may arise from any change.

    Customers may:

    • Replace the switch with an equivalent customer-supplied 1U Gigabit Ethernet switch.

    • Reconfigure the supplied Cisco Ethernet switch settings and firmware.

    • Remove the supplied Cisco Ethernet switch and cable the management network connections directly to an external customer-supplied switch or patch panel.

  • Subject to validation using Oracle Exadata Configuration Assistant (OECA), customers may install additional equipment into unused rack slots.

    Use the ADD EQUIPMENT button in OECA to specify the additional equipment details.

    Additional equipment may only be installed after the resulting configuration is validated using OECA.

1.4.2 Restrictions for Modifying the Oracle Exadata Database Machine Software Components

The following restrictions apply to software modifications to Oracle Exadata Rack. Violating these restrictions can result in loss of warranty and support.

  • Oracle Exadata System Software and the operating systems cannot be modified, and you cannot install any additional software or agents on the Oracle Exadata Storage Servers.

  • You cannot update the firmware directly on the Oracle Exadata Storage Servers. The firmware is updated as part of an Oracle Exadata Storage Server patch.

  • You can update the firmware of the other components of Oracle Exadata Rack.

    • You can update the IOS and firmware versions on the Ethernet switch to meet their data center requirements.

    • You can update the firmware of the components of the database servers provided they comply with the restrictions documented in My Oracle Support Doc ID 888828.1, and its related notes.

    • You can update the firmware of the RDMA Network Fabric switches provided they comply with the validated versions documented in My Oracle Support Doc ID 888828.1, and its related notes.

  • You may load additional software on the database servers. However, to ensure best performance, Oracle discourages adding software except for agents, such as backup agents and security monitoring agents, on the database servers.

    Loading non-standard kernel modules on the operating system of the database servers is allowed but discouraged. Oracle will not support questions or issues with the non-standard modules. If a server crashes, and Oracle suspects the crash may have been caused by a non-standard module, then Oracle support may refer you to the vendor of the non-standard module or ask that the issue be reproduced without the non-standard module. Modifying the database server operating system other than by applying official patches and upgrades is not supported.

  • RDMA Network Fabric-related packages should always be maintained at the officially supported release.

  • You can install Oracle Enterprise Manager Cloud Control Management Agents or Oracle Enterprise Manager Grid Control Management Agents on the database servers to monitor the system.

  • Installing Oracle Enterprise Manager Ops Center agents to monitor the system is not allowed.

1.4.3 Restrictions for Connecting Oracle Exadata Database Machine with Other Hardware, Systems, or Networks

The following restrictions apply to hardware and software modifications to Oracle Exadata Rack. Violating these restrictions can result in loss of warranty and support.

  • Oracle Exadata Storage Expansion Rack can only be connected to Oracle Exadata Database Machine or Oracle SuperCluster, and only supports databases running on the database servers on Oracle Exadata Database Machine or Oracle SuperCluster.

  • Standalone Oracle Exadata Storage Servers can only be connected to Oracle Exadata Database Machine or Oracle SuperCluster, and only support databases running on the database servers of Oracle Exadata Database Machine or Oracle SuperCluster.

  • Customers cannot connect USB devices that draw more than 100 mA of power to the database servers.

  • Customers cannot connect USB devices to the Oracle Exadata Storage Servers except as documented in the Oracle Exadata Database Machine documentation. In those documented situations, the USB device should not draw more than 100 mA of power.

  • Only switches specified for use in Oracle Exalogic Elastic Cloud, Oracle Big Data Appliance, and Oracle SuperCluster may be connected to the Oracle Exadata Rack RDMA Network Fabric network. It is not supported to connect other RDMA Network Fabric switches, including third-party switches, to the Oracle Exadata Rack RDMA Network Fabric network. Only the RDMA Network Fabric networking topologies specified in this guide and Oracle Engineered Systems Extending and Multi-Rack Cabling Guide are supported, and any other RDMA Network Fabric network topology is not supported.

    Note:

    When interconnecting racks, all racks must use the same type of RDMA Network Fabric, either InfiniBand or RDMA over Converged Ethernet (RoCE).
  • You may connect external servers, that are not part of Oracle Engineered Systems, to the RDMA Network Fabric switches in Oracle Exadata Database Machine. However, it is your responsibility to upgrade and maintain the compatibility of the RDMA Network Fabric software of the external servers with the RDMA Network Fabric software release for Oracle Exadata Database Machine. Oracle highly recommends that you maintain the same release of RDMA Network Fabric software and operating system on the external server as on Oracle Exadata Database Machine. If an RDMA Network Fabric problem is encountered and an external server is connected, then you may be asked to remove the external server and reproduce the problem.

  • You can connect the database servers to external (non-Exadata) storage using iSCSI or NFS subject to the following:

    • The storage is allowed for application input and output files, Data Pump export files and logs, and files associated with backup and recovery operations.
    • The use of non-Oracle storage for Oracle Database files is not supported.
    • The Fibre Channel over Ethernet (FCoE) protocol is not supported.

Related Topics

1.4.4 Maximum Number of Database Processes

This topic describes the limits relating to the maximum number of database processes on each Exadata database node or virtual machine guest.

The following table shows the maximum number of database processes supported on each Exadata database node. The best practice is to keep the process count below these values. If a subset of your workload is running parallel queries, the maximum database process count will be between the values in the "Maximum Number of Processes with No Parallel Queries" column and the "Maximum Number of Processes with All Running Parallel Queries" column.

Table 1-1 Maximum Number of Database Processes on a Database Node

Machine Type RDMA Network Fabric Bonding Type Maximum Number of Processes with No Parallel Queries Maximum Number of Processes with All Running Parallel Queries

8-socket (X2-8, X3-8)

Active passive

28,500

25,000

8-socket (X4-8, and later)

Active bonding

64,000

44,000

2-socket (X2-2, X3-2)

Active passive

12,500

10,000

2-socket (X4-2, and later)

Active bonding

16,000

14,000

The following table shows the maximum number of database processes supported on each virtual machine (VM) guest (Oracle VM DomU or KVM guest). The best practice is to keep the process count below these values. If a subset of your workload is running parallel queries, the maximum database process count will be between the "Maximum Number of Processes with No Parallel Queries" column and the "Maximum Number of Processes with All Running Parallel Queries" column.

Table 1-2 Maximum Number of Database Processes on a Virtual Machine Guest

Machine Type RDMA Network Fabric Bonding Type Maximum Number of Processes with No Parallel Queries Maximum Number of Processes with All Running Parallel Queries

2-socket (X2-2, X3-2)

Active passive

11,500

8,000

2-socket (X4-2, and later)

Active bonding

23,000

14,000

The following best practices should be followed to ensure optimal resource utilization at high process count:

  • Application-initiated Oracle foregrounds should be established through a set of Oracle listeners running on the Exadata database nodes instead of using local bequeath connections.

  • The number of listeners should be at least as high as the number of database node CPU sockets, and every database node CPU socket should run the same number of listeners. For example, on an Oracle Exadata X5-8 database node, eight listeners could be configured, one per database node CPU socket.

  • Listeners should spawn Oracle processes evenly across database node CPU sockets. This can be done by specifying the socket they will run on at startup time. For example, assuming the listener.ora file is configured correctly for listeners 0 through 7, the following script could be used to spawn eight listeners on an X5-8 database node, each on a different socket:

    #!/bin/bash
    export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_1
    for socket in `seq 0 7`
    do
     numactl --cpunodebind=${socket} $ORACLE_HOME/bin/lsnrctl start LISTENER${socket}
    done
  • Listener connection rate throttling should be used to control login storms and provide system stability at high process counts.

  • The total number of connections established per second, in other words, the sum of rate_limit for all listeners, should not be more than 400 to avoid excessive client connection timeouts and server-side errors.