Performance improvement is an iterative process. Removing the first bottleneck (a point where resource contention is at its highest) might not lead to performance improvement immediately because another bottleneck might be revealed that has an even greater performance impact on the system. For this reason, the Oracle performance method is iterative. Accurately diagnosing the performance problem is the first step toward ensuring that the changes you make to the system will result in improved performance.
Performance problems generally result from a lack of throughput, unacceptable user or job response time, or both. The problem might be localized to specific application modules, or it might span the entire system. Before looking at any database or operating system statistics, it is crucial to get feedback from the most important components of the system: the users of the system and the people ultimately paying for the application. Getting feedback from users makes determining the performance goal easier, and improved performance can be measured in terms of real business goals, rather than system statistics.
The Oracle performance method can be applied until performance goals are met or deemed impractical. Because this process is iterative, it is likely that some investigations will be made that have little impact on the performance of the system. It takes time and experience to accurately pinpoint critical bottlenecks in a timely manner. Automatic Database Diagnostic Monitor (ADDM) implements the Oracle performance method and analyzes statistics to provide automatic diagnosis of major performance problems. Using ADDM can significantly shorten the time required to improve the performance of a system, and it is the method used in this guide.
This chapter discusses the Oracle Database performance method and contains the following sections:
Database statistics provide information about the type of load on the database and the internal and external resources used by the database. To accurately diagnose performance problems with the database using ADDM, statistics must be available.
Oracle Database generates many types of cumulative statistics for the system, sessions, and individual SQL statements. Oracle Database also tracks cumulative statistics about segments and services. Automatic Workload Repository (AWR) automates database statistics gathering by collecting, processing, and maintaining performance statistics for database problem detection and self-tuning purposes.
By default, the statistics gathering process repeats every hour and results in an AWR snapshot, which is a set of data for a specific time that is used for performance comparisons. The delta values captured by the snapshot represent the changes for each statistic over the time period. Statistics gathered by AWR are queried from memory. The gathered data can be displayed in both reports and views.
Gathering database statistics using AWR is enabled by default and is controlled by the
STATISTICS_LEVEL initialization parameter. The
STATISTICS_LEVEL parameter should be set to
ALL to enable statistics gathering by AWR. The default setting is
TYPICAL. Setting the
STATISTICS_LEVEL parameter to
BASIC disables many Oracle Database features, including AWR, and is not recommended. For information about the
STATISTICS_LEVEL initialization parameter, see Oracle Database Reference.
CONTROL_MANAGEMENT_PACK_ACCESS initialization parameter should be set to
DIAGNOSTIC+TUNING (default) or
DIAGNOSTIC to enable automatic database diagnostic monitoring. Setting
NONE disables many Oracle Database features, including ADDM, and is strongly discouraged. For information about the
CONTROL_MANAGEMENT_PACK_ACCESS initialization parameter, see Oracle Database Reference.
Time model statistics measure the time spent in the database by operation type. The most important time model statistic is database time, or DB time. Database time represents the total time spent in database calls, and is an indicator of the total instance workload. As shown in Figure 2-1, database time makes up a portion of an application's overall user response time.
A session is a specific connection of a user to an Oracle Database instance through a user process. Database time is calculated by aggregating the CPU time and wait time of all user sessions not waiting for idle wait events (user sessions that are not idle). For example, a user session may involve an online transaction made at a bookseller's Web site consisting of the actions shown in Figure 2-2.
Query for novels by author
The user performs a search for novels by a particular author. This action causes the application to perform a database query for novels by the author.
Browse results of query
The user browses the returned list of novels by the author and accesses additional details, such as user reviews and inventory status. This action causes the application to perform additional database queries.
Add item to cart
After browsing details about the novels, the user decides to add one of the novels to the shopping cart. This action causes the application to make a database call to update the shopping cart.
The user completes the transaction by checking out, using the address and payment information previously saved at the bookseller's Web site from a previous purchase. This action causes the application to perform various database operations to retrieve the user's information, add a new order, update the inventory, and generate an E-mail confirmation.
For each of the preceding actions, the user makes a request to the database, as represented by the down arrow in Figure 2-2. The CPU time spent by the database processing the request and the wait time spent waiting for the database are considered DB time, as represented by the shaded areas. After the request is completed, the results are returned to the user, as represented by the up arrow. The space between the up and down arrows represents the total user response time for processing the request, which contains other components besides DB time, as illustrated in Figure 2-1.
The objective of database tuning is to reduce the time that users spend performing actions on the database, or reducing database time. In this way, you can improve the overall response time of user transactions on the application.
Wait events are incremented by a session to indicate that the session had to wait for an event to complete before being able to continue processing. When a session has to wait while processing a user request, the database records the wait by using one of a set of predefined wait events. The events are then grouped into wait classes, such as User I/O and Network. Wait event data reveals symptoms of problems that might be affecting performance, such as latch, buffer, and I/O contention.
The Active Session History (ASH) statistics are samples of session activity in the database. The database samples active sessions every second and stores them in a circular buffer in the System Global Area (SGA). Any session that is connected to the database and using CPU, or is waiting for an event that does not belong to the idle wait class, is considered an active session. By capturing only active sessions, a manageable set of data is represented. The size of the data is directly related to the work being performed, rather than the number of sessions allowed on the system.
Using the DB time example described in "Time Model Statistics", samples of session activity are collected from the online transaction made at the bookseller's Web site, represented as vertical lines below the horizontal arrow in Figure 2-3.
The light vertical lines represent samples of inactive session activity that are not captured in the ASH statistics. The bold vertical lines represent samples of active sessions that are captured at:
7:38, while novels by the author are being queried
7:42, while the user is browsing the query results
7:50, when one of the novels is added to the shopping cart
7:52, during the checkout process
Table 2-1 lists the ASH statistics that are collected for the active sessions, along with examples of the session ID (SID), module, SQL ID, session state, and wait events that are sampled.
Performance tuning using the Oracle performance method is driven by identifying and eliminating bottlenecks in the database, and by developing efficient SQL statements. Database tuning is performed in two phases: proactively and reactively.
In the proactive tuning phase, you must perform tuning tasks as part of your daily database maintenance routine, such as reviewing ADDM analysis and findings, monitoring the real-time performance of the database, and responding to alerts.
In the reactive tuning phase, you must respond to issues reported by the users, such as performance problems that may occur for only a short duration of time, or performance degradation to the database over a period of time.
SQL tuning is an iterative process to identify, tune, and improve the efficiency of high-load SQL statements.
Applying the Oracle performance method involves the following:
Performing pre-tuning preparations, as described in "Preparing the Database for Tuning"
Tuning the database proactively on a regular basis, as described in "Tuning the Database Proactively"
Tuning the database reactively when performance problems are reported by the users, as described in "Tuning the Database Reactively"
Identifying, tuning, and optimizing high-load SQL statements, as described in "Tuning SQL Statements"
To improve the performance of your database, you will need to apply these principles iteratively.
This section lists and describes the steps that must be performed before the database can be properly tuned.
Get feedback from users.
Determine the scope of the performance project and subsequent performance goals, and determine performance goals for the future. This process is key for future capacity planning.
Check the operating systems of all systems involved with user performance.
Check for hardware or operating system resources that are fully utilized. List any overused resources as possible concerns for later analysis. In addition, ensure that all hardware is functioning properly.
Ensure that the
CONTROL_MANAGEMENT_PACK_ACCESS initialization parameter is set to
DIAGNOSTIC+TUNING (default) or
DIAGNOSTIC to enable ADDM.
"Gathering Database Statistics Using the Automatic Workload Repository" for information about configuring AWR
This section lists and describes the steps required to keep the database properly tuned on a regular basis. These tuning procedures are considered proactive and should be performed as part of your daily maintenance of Oracle Database.
Review the ADDM findings, as described in Chapter 3, "Automatic Database Performance Monitoring".
ADDM automatically detects and reports on performance problems with the database, including most of the "Common Performance Problems Found in Oracle Databases". The results are displayed as ADDM findings on the Database Home page in Oracle Enterprise Manager (Enterprise Manager). Reviewing these findings enables you to quickly identify the performance problems that require your attention.
Implement the ADDM recommendations, as described in Chapter 3, "Automatic Database Performance Monitoring".
ADDM automatically provides a list of recommendations for reducing the impact of the performance problem with each ADDM finding. Implementing a recommendation applies the suggested changes to improve the database performance.
Monitor performance problems with the database in real time, as described in Chapter 4, "Monitoring Real-Time Database Performance".
The Performance page in Enterprise Manager enables you to identify and respond to real-time performance problems. By drilling down to the appropriate pages, you can identify and resolve performance problems with the database in real time, without having to wait until the next ADDM analysis.
Respond to performance-related alerts, as described in Chapter 5, "Monitoring Performance Alerts".
The Database Home page in Enterprise Manager enables you to view performance-related alerts generated by the system. Typically, these alerts reveal performance problems whose resolutions will improve the performance of your database.
Validate that the changes made have produced the desired effect, and verify that the users perceive performance improvements.
Repeat these steps until your performance goals are met or become impossible to achieve because of other constraints.
This section lists and describes the steps required to tune the database based on user feedback. This tuning procedure is considered reactive. Perform this procedure periodically when performance problems are reported by the users.
Run ADDM manually to diagnose current and historical database performance when performance problems are reported by the users, as described in Chapter 6, "Manual Database Performance Monitoring".
This task is useful if you want to analyze current database performance before the next ADDM analysis, or to analyze historical database performance when you were not proactively monitoring the system.
Resolve transient performance problems, as described in Chapter 7, "Resolving Transient Performance Problems".
The Active Session History (ASH) reports enable you to analyze transient performance problems with the database that are short-lived and do not appear in the ADDM analysis.
Resolve performance degradation over time, as described in Chapter 8, "Resolving Performance Degradation Over Time".
The Automatic Workload Repository (AWR) Compare Periods report enables you to compare database performance between two periods of time, and resolve performance degradation that may happen from one time period to another.
Validate that the changes made have produced the desired effect, and verify that the users perceive performance improvements.
Repeat these steps until your performance goals are met or become impossible to achieve due to other constraints.
This section lists and describes the steps required to identify, tune, and optimize high-load SQL statements.
Identify high-load SQL statements, as described in Chapter 9, "Identifying High-Load SQL Statements".
Use the ADDM findings and the Top SQL section to identify high-load SQL statements that are causing the greatest contention.
Tune high-load SQL statements, as described in Chapter 10, "Tuning SQL Statements".
You can improve the efficiency of high-load SQL statements by tuning them using SQL Tuning Advisor.
Optimize data access paths, as described in Chapter 11, "Optimizing Data Access Paths".
Analyze the SQL performance impact of SQL tuning and other system changes, as described in Chapter 12, "Analyzing SQL Performance Impact".
You can analyze the performance impact of your SQL tuning activities or other system changes on a given SQL workload by using SQL Performance Analyzer.
Repeat these steps until all high-load SQL statements are tuned for greatest efficiency.
This section lists and describes common performance problems found in Oracle databases. By following the Oracle performance method, you should be able to avoid these problems. If you have these problems, then repeat the steps in the Oracle performance method, as described in "Using the Oracle Performance Method", or consult the appropriate section that addresses these problems:
Is the application performing poorly because the system is CPU-bound? Performance problems caused by CPU bottlenecks are diagnosed by ADDM, as described in Chapter 3, "Automatic Database Performance Monitoring". You can also identify CPU bottlenecks by using the Performance page in Enterprise Manager, as described in "Monitoring CPU Utilization".
Undersized memory structures
Are the Oracle memory structures—such as the System Global Area (SGA), Program Global Area (PGA), and buffer cache—adequately sized? Performance problems caused by undersized memory structures are diagnosed by ADDM, as described in Chapter 3, "Automatic Database Performance Monitoring". You can also identify memory usage issues by using the Performance page in Enterprise Manager, as described in "Monitoring Memory Utilization".
I/O capacity issues
Is the I/O subsystem performing as expected? Performance problems caused by I/O capacity issues are diagnosed by ADDM, as described in Chapter 3, "Automatic Database Performance Monitoring". You can also identify disk I/O issues by using the Performance page in Oracle Enterprise Manager, as described in "Monitoring Disk I/O Utilization".
Suboptimal use of Oracle Database by the application
Is the application making suboptimal use of Oracle Database? Problems such as establishing new database connections repeatedly, excessive SQL parsing, and high levels of contention for a small amount of data (also known as application-level block contention) can degrade the application performance significantly. Performance problems caused by suboptimal use of Oracle Database by the application are diagnosed by ADDM, as described in Chapter 3, "Automatic Database Performance Monitoring". You can also monitor top activity in various dimensions—including SQL, session, services, modules, and actions—by using the Performance page in Enterprise Manager, as described in "Monitoring User Activity".
Is the database performing suboptimally due to a high degree of concurrent activities in the database? A high degree of concurrent activities might result in contention for shared resources that can manifest in the forms of locks or waits for buffer cache. Performance problems caused by concurrency issues are diagnosed by ADDM, as described in Chapter 3, "Automatic Database Performance Monitoring". You can also identify concurrency issues by using Top Sessions in Enterprise Manager, as described in "Monitoring Top Sessions".
Database configuration issues
Is the database configured optimally to provide desired performance levels? For example, is there evidence of incorrect sizing of log files, archiving issues, excessive number of checkpoints, or suboptimal parameter settings? Performance problems caused by database configuration issues are diagnosed by ADDM, as described in Chapter 3, "Automatic Database Performance Monitoring".
Short-lived performance problems
Are users complaining about short-lived or intermittent performance problems? Depending on the interval between snapshots taken by AWR, performance problems that have a short duration may not be captured by ADDM. You can identify short-lived performance problems by using the Active Session History report, as described in Chapter 7, "Resolving Transient Performance Problems".
Degradation of database performance over time
Is there evidence that the database performance has degraded over time? For example, are you or your users noticing that the database is not performing as well as it was 6 months ago? You can generate an AWR Compare Periods report to compare the period when the performance was poor to a period when the performance is stable to identify configuration settings, workload profile, and statistics that are different between these two time periods. This technique will help you identify the cause of the performance degradation, as described in Chapter 8, "Resolving Performance Degradation Over Time".
Inefficient or high-load SQL statements
Are any SQL statements using excessive system resources that impact the system? Performance problems caused by high-load SQL statements are diagnosed by ADDM, as described in Chapter 3, "Automatic Database Performance Monitoring" and "Identification of High-Load SQL Statements Using ADDM Findings". You can also identify high-load SQL statements by using Top SQL in Enterprise Manager, as described in "Identifying High-Load SQL Statements Using Top SQL". After they have been identified, you can tune the high-load SQL statements using SQL Tuning Advisor, as described in Chapter 10, "Tuning SQL Statements".
Are any database objects the source of bottlenecks because they are continuously accessed? Performance problems caused by object contention are diagnosed by ADDM, as described in Chapter 3, "Automatic Database Performance Monitoring". You can also optimize the data access path to these objects using SQL Access Advisor, as described in Chapter 11, "Optimizing Data Access Paths".
Unexpected performance regression after tuning SQL statements
Is the performance of SQL statements degrading after they have been tuned? Tuning SQL statements may cause changes to execution plans of SQL statements, resulting in a significant impact on SQL performance. In some cases, the changes may result in the improvement of SQL performance. In other cases, the changes may cause SQL statements to regress, resulting in a degradation of SQL performance. Before making changes on a production system, you can analyze the performance impact from tuning SQL statements on a test system by using SQL Performance Analyzer, as described in Chapter 12, "Analyzing SQL Performance Impact".