| Oracle® Enterprise Manager Cloud Control Administrator's Guide 12c Release 1 (12.1.0.1) Part Number E24473-01 |
|
|
PDF · Mobi · ePub |
| Oracle® Enterprise Manager Cloud Control Administrator's Guide 12c Release 1 (12.1.0.1) Part Number E24473-01 |
|
|
PDF · Mobi · ePub |
This chapter covers the following topics:
Installation Best Practices for Enterprise Manager High Availability
Managing Multiple Hosts and Deploying a Remote Management Repository
Converting the Enterprise Manager Repository from Single Instance to RAC
Configuring Management Service to Management Repository Communication
Configuring Standby Database for the Enterprise Manager Repository
The following sections document best practices for installation and configuration of each Cloud Control component.
The Management Agent is started manually. It is important that the Management Agent be automatically started when the host is booted to insure monitoring of critical resources on the administered host. To that end, use any and all operating system mechanisms to automatically start the Management Agent. For example, on UNIX systems this is done by placing an entry in the UNIX /etc/init.d that calls the Management Agent on boot or by setting the Windows service to start automatically.
Once the Management Agent is started, the watchdog process monitors the Management Agent and attempts to restart it in the event of a failure. The behavior of the watchdog is controlled by environment variables set before the Management Agent process starts. The environment variables that control this behavior follow. All testing discussed here was done with the default settings.
EM_MAX_RETRIES – This is the maximum number of times the watchdog will attempt to restart the Management Agent within the EM_RETRY_WINDOW. The default is to attempt restart of the Management Agent 3 times.
EM_RETRY_WINDOW - This is the time interval in seconds that is used together with the EM_MAX_RETRIES environmental variable to determine whether the Management Agent is to be restarted. The default is 600 seconds.
The watchdog will not restart the Management Agent if the watchdog detects that the Management Agent has required restart more than EM_MAX_RETRIES within the EM_RETRY_WINDOW time period.
The Management Agent persists its intermediate state and collected information using local files in the $AGENT_HOME/$HOSTNAME/sysman/emd sub tree under the Management Agent home directory.
In the event that these files are lost or corrupted before being uploaded to the Management Repository, a loss of monitoring data and any pending alerts not yet uploaded to the Management Repository occurs.
At a minimum, configure these sub-directories on striped redundant or mirrored storage. Availability would be further enhanced by placing the entire $AGENT_HOME on redundant storage. The Management Agent home directory is shown by entering the command 'emctl getemhome' on the command line, or from the Management Services and Repository tab and Agents tab in the Cloud Control console.
The Management Service contains results of the intermediate collected data before it is loaded into the Management Repository. The loader receive directory contains these files and is typically empty when the Management Service is able to load data as quickly as it is received. Once the files are received by the Management Service, the Management Agent considers them committed and therefore removes its local copy. In the event that these files are lost before being uploaded to the Management Repository, data loss will occur. At a minimum, configure these sub-directories on striped redundant or mirrored storage. When Management Services are configured for the Shared Filesystem Loader, all services share the same loader receive directory. It is recommended that the shared loader receive directory be on a clustered file system like NetApps Filer.
Installing all the Cloud Control components on a single host is an effective way to initially explore the capabilities and features available to you when you centrally manage your Oracle environment.
A logical progression from the single-host environment is to a more distributed approach, where the Management Repository database is on a separate host and does not compete for resources with the Management Service. The benefit in such a configuration is scalability; the workload for the Management Repository and Management Service can now be split. This configuration also provides the flexibility to adjust the resources allocated to each tier, depending on the system load.
In this more distributed configuration, data about your managed targets travels along the following paths so it can be gathered, stored, and made available to administrators by way of the Grid Control console:
Administrators use the Grid Control console to monitor and administer the targets just as they do in the single-host scenario described in Deploying Cloud Control Components on a Single Host.
Management Agents are installed on each host on the network, including the Management Repository host and the Management Service host. The Management Agents upload their data to the Management Service by way of the Management Service upload URL, which is defined in the emd.properties file in each Management Agent home directory. The upload URL uploads the data directly through the Oracle HTTP Server.
The Management Repository is installed on a separate machine that is dedicated to hosting the Management Repository database. The Management Service uses JDBC connections to load data into the Management Repository database and to retrieve information from the Management Repository so it can be displayed in the Grid Control console. This remote connection is defined in the Administration Server and can be accessed and changed through emctl commands.
The Management Service communicates directly with each remote Management Agent over HTTP by way of the Management Agent URL. The Management Agent URL is defined by the EMD_URL property in the emd.properties file of each Management Agent home directory. As described in Deploying Cloud Control Components on a Single Host, the Management Agent includes a built-in HTTP listener so no Oracle HTTP Server is required on the Management Agent host.
In Enterprise Manager release 12c, the installation does not create a new database. You must install a database which should be on the same host as Enterprise Manager.
When you install all the Cloud Control components on a single host, the management data travels along the following paths:
Administrators use the Grid Control console to monitor and administer the managed targets that are discovered by the Management Agents on each host. The Grid Control console uses the following URL to connect to the Oracle HTTP Server:
https://host1.acme.com:7799/em
The Management Service retrieves data from the Management Repository as it is requested by the administrator using the Grid Control console.
The Management Agent loads its data (which includes management data about all the managed targets on the host, including the Management Service and the Management Repository database) by way of the Oracle HTTP Server upload URL. The Management Agent uploads data directly to Oracle HTTP Server. The default port for the upload URL is 1159 (if it is available during the installation procedure). The upload URL is defined by the REPOSITORY_URL property in the following configuration file in the Management Agent home directory:
AGENT_HOME/sysman/config/emd.properties (UNIX) AGENT_HOME\sysman\config\emd.properties (Windows)
See Also:
For more information about the Oracle Enterprise Manager directory structure (AGENT_HOME directory in particular), see the Oracle® Enterprise Manager Cloud Control Advanced Installation and Configuration Guide.The Management Service uses JDBC connections to load data into the Management Repository database and to retrieve information from the Management Repository so it can be displayed in the Grid Control console. The Management Repository connection details can be listed and changed by using the following emctl commands:
emctl config oms -list_repos_details emctl config oms -store_repos_details
See Also:
"Reconfiguring the Oracle Management Service" on page 34-9 for more information on modifying the Management Repository connection information.The Management Service sends data to the Management Agent by way of HTTP. The Management Agent software includes a built-in HTTP listener that listens on the Management Agent URL for messages from the Management Service. As a result, the Management Service can bypass the Oracle HTTP Server and communicate directly with the Management Agent. If the Management Agent is on a remote system, no Oracle HTTP Server is required on the Management Agent host.
The Management Service uses the Management Agent URL to monitor the availability of the Management Agent, submit Enterprise Manager jobs, and other management functions.
The Management Agent URL can be identified by the EMD_URL property in the following configuration file in the Management Agent home directory:
AGENT_HOME/sysman/config/emd.properties (UNIX) AGENT_HOME\sysman\config\emd.properties (Windows)
For example:
EMD_URL=https://host1.acme.com:3872/emd/main
In addition, the name of the Management Agent as it appears in the Grid Control console consists of the Management Agent host name and the port used by the Management Agent URL.
In larger production environments, you may find it necessary to add additional Management Service installations to help reduce the load on the Management Service and improve the efficiency of the data flow.
Note:
When you add additional Management Service installations to your Cloud Control configuration, be sure to adjust the parameters of your Management Repository database. For example, you will likely need to increase the number of processes allowed to connect to the database at one time. Although the number of required processes will vary depending on the overall environment and the specific load on the database, as a general guideline, you should increase the number of processes by 40 for each additional Management Service.For more information, see the description of the PROCESSES initialization parameter in the Oracle Database Reference.
Understanding the Flow of Management Data When Using Multiple Management Services
In a multiple Management Service configuration, the management data moves along the following paths:
Administrators can use one of two URLs to access the Grid Control console. Each URL refers to a different Management Service installation, but displays the same set of targets, all of which are loaded in the common Management Repository. Depending upon the host name and port in the URL, the Grid Control console obtains data from the Management Service (by way of the Oracle HTTP Server) on one of the Management Service hosts.
Each Management Agent uploads its data to a specific Management Service, based on the upload URL in its emd.properties file. That data is uploaded directly to the Management Service by way of Oracle HTTP Server.
Whenever more than one Management Service is installed, it is a best practice to have the Management Service upload to a shared directory. This allows all Management Service processes to manage files that have been uploaded from any Management Agent. This protects from the loss of any one Management Server causing a disruption in upload data from Management Agents.
Configure this functionality from the command line of each Management Service process as follows:
emctl config oms loader -shared <yes|no> -dir <load directory>
Important:
By adding a load balancer, you can avoid the following problems:Should the Management Service fail, any Management Agent connected to it cannot upload data.
Because user accounts only know about one Management Service, users lose connectivity should the Management Service go down even if the other Management Service is up.
See Section 33.3, "High Availability Configurations" for information regarding load balancers.
Note:
If the software library is being used in this environment, it should be configured to use shared storage in the same way as the shared Management Service loader. To modify the location for the software library:Click the Deployments tab on the Enterprise Manager Home page.
Click the Provisioning subtab.
On the Provisioning page, click the Administration subtab.
In the Software Library Configuration section, click Add to set the Software Library Directory Location to a shared storage that can be accessed by any host running the Management Service.
Each Management Service communicates by way of JDBC with a common Management Repository, which is installed in a database on a dedicated Management Repository host. Each Management Service uses the same database connection information, defined in the emgc.properties file, to load data from its Management Agents into the Management Repository. The Management Service uses the same connection information to pull data from the Management Repository as it is requested by the Grid Control console.
Any Management Service in the system can communicate directly with any of the remote Management Agents defined in the common Management Repository. The Management Services communicate with the Management Agents over HTTP by way of the unique Management Agent URL assigned to each Management Agent.
The Management Agent URL is defined by the EMD_URL property in the emd.properties file of each Management Agent home directory. Each Management Agent includes a built-in HTTP listener so no Oracle HTTP Server is required on the Management Agent host.
Once you configure the Management Repository, the next step is to install and configure the Enterprise Manager Cloud Control mid-tier, the Management Services, for greater availability. Before discussing steps that add mid-tier redundancy and scalability, note that the Management Service itself has a built-in restart mechanism based on the Oracle Weblogic Node Manager and the Oracle Process Management and Notification Service (OPMN). These services will attempt to restart a Management Service that is down. It is advisable to run OPMN and Node Manager as operating system services, so that they restart automatically if their host machine is restarted.
If you are managing a large environment with multiple Management Services and Management Repository nodes, then consider installing the Management Services on hardware nodes that are different from Management Repository nodes (Figure 33–3). This allows you to scale out the Management Services in the future.
Also consider the network latency between the Management Service and the Management Repository while determining the Management Service install location. The distance between the Management Service and the Management Repository may be one of the factors that affect network latency and hence determine Enterprise Manager performance.
If the network latency between the Management Service and Management Repository tiers is high or the hardware available for running Enterprise Manager is limited, then the Management Service can be installed on the same hardware as the Management Repository (Figure 33–4). This allows for Enterprise Manager high availability, as well as keep the costs down.
If you plan ahead, you can configure your Enterprise Manager deployment for high availability by choosing the correct options during the first Management Service install. You can also retrofit the MAA best practices into an existing Enterprise Manager deployment configured initially using the default install options.
Once your first Management Service is setup for high availability, there are two paths to setting up your additional Management Services for high availability:
Installing a fresh additional Management Service as per MAA best practices
Retrofitting MAA best practices on existing Additional Management Service
In either of the two cases, the following considerations should be noted:
The additional Management Service should be hosted in close network proximity to the Management Repository database for network latency reasons.
Configure the same path on all Management Services for the directory used for the shared filesystem loader.
Additional Management Services should be installed using the same OS user and group as the first Management Service. Proper user equivalence should be setup so that files created by each Management Service on the shared loader directory can be accessed and modified by the other Management Service processes.
Adjust the parameters of your Management Repository database. For example, you will likely need to increase the number of processes allowed to connect to the database at one time. Although the number of required processes will vary depending on the overall environment and the specific load on the database, as a general guideline, you should increase the number of processes by 40 for each additional Management Service.
Install the additional Management Service using the OUI installer. The additional Management Service will inherit most of the HA configuration from the first Management Service. Post installation, do the following step to complete the HA configuration:
Update the SLB configuration by adding the additional Management Service to the different pools on the SLB. Setup monitors for the new Management Service.
Once you have the additional Management Service installed, use the following steps to copy over the configuration from the first Management Service.
Export the configuration from the first Management Service using emctl exportconfig oms -dir <location for the export file>
Copy over the exported file to the additional Management Service
Shutdown the additional Management Service
Import the exported configuration on the additional Management Service using emctl importconfig oms -file <full path of the export file>
Restart the additional Management Service
Setup EMCLI using emcli setup -url=https://slb.example.com/em -username sysman -password <sysman password> -nocertvalidate
Resecure the Management Agent that is installed with the additional Management Service to upload to SLB using emctl secure agent -emdWalletSrcUrl https://slb.example.com:<upload port>/em
Update the SLB configuration by adding the additional Management Service to the different pools on the SLB. Setup monitors for the new Management Service. Modify the ssl.conf file to set the Port directive to the SLB virtual port used for UI access.
The Management Service for Cloud Control has a high availability feature called the Shared Filesystem Loader. In the Shared Filesystem Loader, management data files received from Management Agents are stored temporarily on a common shared location called the shared receive directory. All Management Services are configured to use the same storage location for the shared receive directory. The Management Services coordinate internally and distribute among themselves the workload of uploading files into the Management Repository. Should a Management Service go down, its workload is taken up by surviving Management Services. You must choose a shared receive directory that is accessible by all the Management Services using redundant file storage.
During the first Management Service installation, the shared receive directory can be configured out of the box by passing SHARED_RECEIVE_DIRECTORY_LOCATION=<shared recv directory> option to runInstaller (setup.exe on Windows). Oracle recommends that this location be outside the Instance Home and Middleware Home locations.
If not configured during installation, the Shared Filesystem Loader can also be configured post-installation by running the following emctl command on every Management Service:
emctl config oms loader -shared yes -dir <shared recv directory>
Note:
If shared filesystem Loader is configured on the first Management Service, any additional management service that is installed later will inherit the shared filesystem loader configuration. Therefore, ensure that the shared recv directory is available on the additional Management Service prior to running install.Consider the following while configuring Shared Filesystem Loader on Windows.
On Windows platforms, the Enterprise Manager install may configure the Management Service to run as a service using 'LocalSystem' account. This is a local account and will typically not have access to the network drive for the shared filesystem that requires domain authentication. To resolve this issue, configure the Management Service to run as a domain user as follows:
Go to the Control Panel and then open the Services panel.
Double click the appropriate service (Oracleoms11gProcessManager).
Change the 'Log on as' user from the 'Local System Account' to This Account.
Specify the domain user that has access to shared network drive.
Click OK.
Do not use local drive letters for mapped network shares while configuring the shared filesystem on Windows. Use UNC locations instead.
emctl config oms loader -shared yes -dir \\\\<host>\\<share-name>\\<recv-dir> for example emctl config oms loader -shared yes -dir \\\\hostA\\vol1\\recv
Note the use of double backslashes while specifying the directory location.
Note:
User equivalence should be set up properly across OMS so that files created by one OMS on the loader directory are modifiable by other OMS.Since software library location has to be accessed by all Management Services, considerations similar to shared fileystem loader directory apply here too. The configuration of software library is not covered during install. It needs to be configured post-install using the Enterprise Manager Console:
On the Enterprise Manager home page, click the Deployments tab.
Click the Provisioning subtab.
On the Provisioning page, click the Administration subtab.
In the Software Library Configuration section, click Add to set the Software Library Directory Location to a shared storage that can be accessed by any host running the Management Service.
This section describes the guidelines for setting up a Server Load Balancer (SLB) to balance the agent and browser traffic to multiple Management Services. This is a two step process:
Configure the SLB
Make needed changes on the Management Services
Use the following table as reference for setting up the SLB with Cloud Control Management Services.
Table 32-1 Management Service Ports
| Cloud Control Service | TCP Port | Monitor Name | Persistence | Pool Name | Load Balancing | Virtual Server Name | Virtual Server Port |
|---|---|---|---|---|---|---|---|
|
Secure Upload |
1159 |
mon_gcsu1159 |
None |
pool_gcsu1159 |
Round Robin |
vs_gcsu1159 |
1159 |
|
Agent Registration |
4889 |
mon_gcar4889 |
Active Cookie Insert |
pool_gcar4889 |
Round Robin |
vs_gcar4889 |
4889 |
|
Secure Console |
7799 |
mon_gcsc7799 |
Source IP |
pool_gcsc7799 |
Round Robin |
vs_gcsc443 |
443 |
|
Unsecure Console (optional) |
7788 |
mon_gcuc7788 |
Source IP |
pool_gcuc7788 |
Round Robin |
vs_gcuc80 |
80 |
Use the administration tools that are packaged with your SLB. A sample configuration follows. This example assumes that you have two Management Services running on host A and host B using the default ports as listed in Table 33–1.
Create Pools
A pool is a set of servers grouped together to receive traffic on a specific TCP port using a load balancing method. Each pool can have its own unique characteristic for a persistence definition and the load-balancing algorithm used.
Table 32-2 Pools
| Pool Name | Usage | Members | Persistence | Load Balancing |
|---|---|---|---|---|
|
pool_gcsu1159 |
Secure upload |
HostA:1159 HostB:1159 |
None |
Round Robin |
|
pool_gcar4889 |
Agent registration |
HostA:4889 HostB:4889 |
Active cookie insert; expiration 60 minutes |
Round Robin |
|
pool_gcsc7799 |
Secured console access |
HostA:7799 HostB:7799 |
Source IP; expiration 60 minutes |
Round Robin |
|
pool_gcuc7788 (optional) |
Unsecured console access |
HostA:7788 HostB:7788 |
Source IP; expiration 60 minutes |
Round Robin |
Create Virtual Servers
A virtual server, with its virtual IP Address and port number, is the client addressable hostname or IP address through which members of a load balancing pool are made available to a client. After a virtual server receives a request, it directs the request to a member of the pool based on a chosen load balancing method.
Create Monitors
Monitors are used to verify the operational state of pool members. Monitors verify connections and services on nodes that are members of load-balancing pools. A monitor is designed to check the status of a service on an ongoing basis, at a set interval. If the service being checked does not respond within a specified timeout period, the load balancer automatically takes it out of the pool and will choose the other members of the pool. When the node or service becomes available again, the monitor detects this and the member is automatically accessible to the pool and able to handle traffic.
Table 32-4 Monitors
| Monitor Name | Configuration | Associate With |
|---|---|---|
|
mon_gcsu1159 |
Type: https Interval: 60 Timeout: 181 Send String: GET /em/upload Receive String: Http Receiver Servlet active! |
HostA:1159 HostB:1159 |
|
mon_gcar4889 |
Type: http Interval: 60 Timeout: 181 Send String: GET /em/genwallet Receive String: GenWallet Servlet activated |
HostA:4889 HostB:4889 |
|
mon_gcsc7799 |
Type: https Interval: 5 Timeout: 16 Send String: GET /em/console/home HTTP/1.0\n Receive String: /em/console/logon/logon;jsessionid= |
HostA:7799 HostB:7788 |
|
mon_gcuc7788 (optional) |
Type: https Interval: 5 Timeout: 16 Send String: GET /em/console/home HTTP/1.0\n Receive String: /em/console/logon/logon;jsessionid= |
HostA:7788 HostB:7788 |
Note: If you have SSO configured, use the following alternate definitions for the mon_gcsc7799 and mon_gcuc7788 monitors.
Table 32-5 Monitors for SSO Configuration
| Monitor Name | Configuration | Associate With |
|---|---|---|
|
mon_gcsc7799 |
Type: https Interval: 5 Timeout: 16 Send String: GET /em/genwallet Receive String: GenWallet Servlet activated |
HostA:7799 HostB:7788 |
|
mon_gcuc7788 (optional) |
Type: https Interval: 5 Timeout: 16 Send String: GET /em/genwallet Receive String: GenWallet Servlet activated |
HostA:7788 HostB:7788 |
Note:
F5 SLB monitors expect the "Receive String" within the first 5120 characters of a response. For SSO environments, the "Receive String" may be returned at some point beyond the 5120 limit. The monitor will not function in this situation.Perform the following steps:
Resecure the Management Service
By default, the service name on the Management Service-side certificate uses the name of the Management Service host. Management Agents do not accept this certificate when they communicate with the Management Service through a load balancer. You must run the following command to regenerate the certificate on the first Management Service:
emctl secure -oms -sysman_pwd <sysman_pwd> -reg_pwd <agent_reg_password> -host slb.example.com -secure_port 1159 -slb_port 1159 -slb_console_port 443 [-lock] [-lock_console]
Resecure all Management Agents
Management Agents that were installed prior to SLB setup, including the Management Agent that comes with the Management Service install, would be uploading directly to the Management Service. These Management Agents will not be able to upload after SLB is setup. Resecure these Management Agents to upload to the SLB by running the following command on each Management Agent:
emctl secure agent –emdWalletSrcUrl https://slb.example.com:<upload port>/em
The following sections describe reconfiguration and tuning changes you can make to the Management Agent after you have installed Enterprise Manager. Refer to the following sections for more information:
Configuring the Management Agent to Use a New Management Service
Changing the Management Agent Port
Controlling the Amount of Disk Space Used by the Management Agent
About the Management Agent Watchdog Process
Setting the Management Agent Time Zone
Adding Trust Points to the Management Agent Configuration
When you install the Management Agent on a managed host, you associate the Management Agent with a particular Management Service. The Management Agent uses the Management Service URL address and port to identify and communicate with the Management Service.
After you install the Management Agent, you can later reconfigure the Management Agent so it is associated with a different Management Service. Reconfiguring the Management Agent requires no changes to the Management Service. The reconfigured Management Agent will begin communicating with the new Management Service after the Management Agent is restarted.
If you are associating the Management Agent with a new Management Service that is locked in secure mode, then first associate the Management Agent with the new Management Service and then secure the Management Agent.
To associate the Management Agent with a new Management Service after you have installed the Management Agent:
Stop the Management Agent.
Locate the emd.properties file in the Management Agent home directory:
AGENT_HOME/sysman/config/emd.properties
Use a text editor to open the file and locate the REPOSITORY_URL property.
Modify the value for the REPOSITORY_URL property so it references the new Management Service. For example:
REPOSITORY_URL=http://mgmthost2.acme.com:4889/em/upload
Modify the value for the emdWalletSrcUrl property so it references the new Management Service. For example, if the new Management Service is on a host called mgmthost2.acme.com , modify the property as follows:
emdWalletSrcUrl=http://mgmthost2.acme.com:4889/em/wallets/emd
Save your changes and close the emd.properties file.
To ensure that the Management Agent is no longer holding any specific data or settings from the previous Management Service, delete all the files in the following directories:
AGENT_HOME/sysman/emd/upload/ AGENT_HOME/sysman/emd/state/ AGENT_HOME/sysman/emd/collection/* AGENT_HOME/sysman/emd/lastupld.xml AGENT_HOME/sysman/emd/agntstmp.txt AGENT_HOME/sysman/emd/blackouts.xml AGENT_HOME/sysman/emd/protocol.ini
Note that this action removes all user-defined metrics (UDM)s and custom changes to metric and policy collections.
Note:
You can use the
emctl clearstate agent command to delete the files in the state directory.Restart the Management Agent.
To secure the Management Agent of the new Management Service, use the following command:
emctl secure agent [registration password] [-emdWalletSrcUrl <url>]
The Management Agent uses a predefined port number to receive requests from the Management Service. This port number is defined by default when you install the Management Agent on a managed host. If you later need to modify this port, you can use the following procedure. You might need to modify this port number if you have existing software that uses the default Management Agent port.
To change the Management Agent port:
Stop the Management Agent.
Locate the emd.properties file in the Management Agent home directory:
AGENT_HOME/sysman/config/emd.properties
Use a text editor to open the file and locate the EMD_URL property.
For example:
EMD_URL=http://managed_host1.acme.com:1813/emd/main
Modify the port number in the EMD_URL property so the Management Agent uses a new unused port on the managed host.
For example:
EMD_URL=http://managed_host1.acme.com:1913/emd/main
You can also use the netstat command to check for the unused port:
On Windows:
netstat -an | findstr <new port number>
On UNIX:
netstat -an | grep <new port number>
Start the Management Agent.
Note:
After the changed URL is processed, the old Management Agent should not have any targets. Oracle recommends that you remove the old Management Agent from the Management Service to ensure that there are no unwanted targets in the Cloud Control console .Oracle designed the Management Agent to work within a set of disk space limits. These limits prevent the Management Agent from using too much disk space and causing performance or resource issues on your enterprise systems. However, if disk space becomes an issue, you can adjust the default settings that are used to control the amount of disk space used by the Management Agent.
As the Management Agent on a particular host gathers management data about the targets on the host, it saves the collected data on the local disk until the data is uploaded to the Management Repository. The Management Agent saves this collected data and metadata in the following directory:
AGENT_HOME/sysman/emd/upload
By default, the Management Agent will save up to 50MB of collected data in the upload directory. If the amount of collected data exceeds 50MB, data collection is stopped temporarily until the data is uploaded to the repository and more disk space becomes available.
To verify how much space is available:
Use the emctl status agent command. For example:
Available disk space on upload filesystem : 1.18% Collection Status : Disabled by Upload Manager Last successful heartbeat to OMS : 2007-07-31 11:22:07
Investigate the <AGENT_HOME>/sysman/log/emagent.trc file. The file will have errors such as :
24.995519 MB Data. 34.06% of disk used. Disabling collections. 2006-10-19 10:41:23 Thread-19 WARN collector: Disable collector
In addition, the Management Agent checks to be sure that the percentage of disk space currently in use on the local disk does not exceed 98 percent. If this value is exceeded, the Management Agent stops collecting data and stops saving information to the Management Agent log and trace files.
You can modify these default settings as follows:
Stop the Management Agent.
Locate the emd.properties file in the Management Agent home directory:
AGENT_HOME/sysman/config/emd.properties
Use a text editor to open the file and modify the entries shown in Table 34–1.
Save your changes and exit the file.
Restart the Management Agent.
Table 32-6 Properties for Controlling the Disk Space Used by the Management Agent
| Property | Explanation |
|---|---|
|
Use this property in the emd.properties file to specify the maximum number of megabytes (MB) used by the collected data in the Management Agent upload directory. When this limit is exceeded, the Management Agent will stop collecting additional management data until the next upload to the Management Repository reduces the amount of collected data in the upload directory. |
|
|
Use this property in the The Management Agent will begin collecting data again when the percentage of disk space in use falls to less than the percentage specified in the |
|
|
UploadMaxDiskUsedPctFloor |
Use this property in the
|
|
UploadMaxNumberXML |
Use this property in the |
The Management Agent is the Enterprise Manager component that gathers the data you need to manage your enterprise efficiently. As a result, Enterprise Manager includes software that keeps track of the Management Agent processes and makes sure the Management Agent stays running.
For example, if the Management Agent quits unexpectedly, this self-monitoring process—referred to as the watchdog process—will restart the Management Agent automatically.
In most situations, the watchdog process works in the background and requires no configuration or maintenance. The watchdog process is controlled by the emwd.pl script located in the following directory of the Management Agent home directory:
AGENT_HOME/bin
You can identify the watchdog process by using the following commands:
$PROMPT> ps -ef | grep emwd
In today's global economy, it is not uncommon for the systems you manage to reside in multiple locations throughout the world. For example, if your company headquarters are in New Hampshire, USA, you may need to manage systems that reside in California, Canada, and in Europe.
As Enterprise Manager collects monitoring data from Management Agents running on these remote systems, it is important that the data is correlated accurately. A software failure on a machine in Ontario, Canada might be the cause of a performance problem on a machine in Hoboken, New Jersey.
To correlate this data, it is important that Enterprise Manager obtains the correct time zone for each Management Agent that you install. The following sections describe how the Management Agent obtains the time zone and how to correct the problem if the time zone for a Management Agent is incorrect:
Understanding How the Management Agent Obtains Time Zone Information
Resetting the Time Zone of the Management Agent Due to Inconsistency of Time Zones
Troubleshooting Management Agent Time Zone Problems
When you install the Management Agent, the software attempts to obtain the current time zone of the host computer. If successful, the installation procedure updates the agentTZRegion property setting in the following configuration file:
AGENT_HOME/sysman/config/emd.properties
The agentTZRegion property can be set to any of the values listed in the following file, which is installed in the Management Agent home directory:
AGENT_HOME/sysman/admin/supportedtzs.lst
To reconfigure a different time zone, perform the following steps. These steps assume that the original time zone used was EST and the target time zone is CST.
Set the environment correctly.
On Windows XP
- From the Start Menu, access the Control Panel. Click Date and Time, then click the Time Zone tab.
- Select GMT-06:00 Central Time (US & Canada) from the list.
- Click OK.
- Open a command line screen (cmd.exe).
- Set the following environment variables:
SET TZ=CST SET ORACLE_HOME=< your oracle home directory > SET PATH=%ORACLE_HOME%\bin;%PATH%
On UNIX
- Login to your UNIX server as the Oracle user.
- Set the following environment variables:
$ export TZ=CST $ export ORACLE_HOME=< your oracle home directory > $ export PATH=%ORACLE_HOME%\bin;%PATH%
Execute the following commands:
On Windows
%ORACLE_HOME%\bin\emctl config agent getTZ %ORACLE_HOME%\bin\emctl stop iasconsole %ORACLE_HOME%\bin\emctl resetTZ agent
On UNIX
$ORACLE_HOME/bin/emctl config agent getTZ $ORACLE_HOME/bin/emctl stop iasconsole $ORACLE_HOME/bin/emctl resetTZ agent
Delete all the files under the following directory:
On Windows
%ORACLE_HOME%\sysman\logs
On UNIX
$ORACLE_HOME$/sysman/logs
Start the console again.
On Windows
%ORACLE_HOME%\bin\emctl start iasconsole %ORACLE_HOME%\bin\emctl status iasconsole %ORACLE_HOME%\bin\emctl status agent %ORACLE_HOME%\bin\emctl config agent getTZ
On UNIX
$ORACLE_HOME\bin\emctl start iasconsole $ORACLE_HOME\bin\emctl status iasconsole $ORACLE_HOME\bin\emctl status agent $ORACLE_HOME\bin\emctl config agent getTZ
Check the timestamp in the log file.
Check the em.properties file. The agentTZRegion parameter should now look like this:
On Windows
%ORACLE_HOME%\sysman\config\em.properties agentTZRegion=America/Chicago
On UNIX
$ORACLE_HOME/sysman/config/em.properties agentTZRegion=America/Chicago
You need to reset the time zone of the Management Agent when both of the following situations are true:
The Management Agent has been running with a particular time zone
Subsequently a change occurs to the time zone of the host where the Management Agent is running
To propagate the time zone change to the emd.properties file, perform the following:
Execute the following script:
ORACLE_HOME/bin/emctl resetTZ agent
This script updates ORACLE_HOME/sysman/config/emd.properties so that the value of agentTZRegion matches that of the current time zone setting of the machine.
Note:
The location of theemd.properties file depends on the Control Console being used:
For the Database Control Console, the location is usually: ORACLE_HOME/<host>_<sid>/sysman/config
For the Application Server Control Console, the location is: ORACLE_HOME/sysman/config
For the Cloud Control Management Agent, the location is ORACLE_HOME/sysman/config
For the Real Application Cluster central Management Agent, the location is usually: ORACLE_HOME/<host>/sysman/config
In addition, this command prompts you to run a script against the Enterprise Manager Repository. You must log in to the database as the Enterprise Manager repository user and run the script mgmt_target.set_agent_tzrgn. An example follows:
SQL> exec mgmt_target.set_agent_tzrgn('em.oracle.com:1830','PST8PDT');
SQL> commit;
SQL> exit
em.oracle.com:1830 represents the name of the emd target.
Sometimes, during the Management Agent installation, the time zone detected by the Management Agent configuration tool is not recognized by the Management Agent. In other words, the time zone obtained by the configuration tool is not listed in the Management Agent list of supported time zones.
This problem prevents the Management Agent from starting and results in an error similar to the following:
Could not determine agent time zone. Please refer to the file:
ORACLE_HOME/sysman/admin/supportedtzs.lst and pick a timezone region with a
standard offset of +5:0 from GMT and update the property 'agentTZRegion' in the
file: ORACLE_HOME/sysman/config/emd.properties
This error appears in one of the log files shown in Table 34–2, depending upon which Enterprise Manager product you are using.
Table 32-7 Location of Time Zone Error in the Enterprise Manager Log Files
| If you are using... | Look for the Time Zone Error in This File... |
|---|---|
|
Grid Control console |
|
|
Application Server Control Console |
|
|
Database Control Console |
|
To configure the Management Agent to use a valid time zone:
Enter the following command in the Management Agent home directory to identify the time zone currently being used by the host computer:
AGENT_HOME/bin/emctl config agent getTZ
Note the time zone that is returned by the emctl config agent getTZ command.
This is the time zone of the host computer.
Use a text editor to open the following file in the Management Agent home directory:
AGENT_HOME/sysman/admin/supportedtzs.lst
This file contains a list of all the time zones supported by the Management Agent.
Browse the contents of the supportedtzs.lst file and note the supported time zone closest to the time zone of the host computer.
Use a text editor to open the following Management Agent configuration file:
AGENT_HOME/sysman/config/emd.properties
Locate the following property near the end of the emd.properties file:
agentTZRegion=
Set the value of this property to the time zone you identified as closest to the host time zone in the supportedtzs.lst file.
For example:
agentTZRegion=Europe/Warsaw
Save your changes and close the emd.properties file.
You should now be able to start the Management Agent without generating the error in the log file.
Perform these steps to add the relevant security certificate:
Obtain the certificate, which is in Base64encoded X.509 (.CER) format, in the b64SiteCertificate.txt file. (The file name may be different in your configuration.) An example of the contents of the file is as follows:
------BEGIN CERTIFICATE-------------- MIIDBzCCAnCgAw... ...... base 64 certificate content ..... ------END CERTIFICATE-----------------
In the Oracle Home of the Management Agent monitoring the wallet, run the following command to add the certificate to the Management Agent:
${ORACLE_HOME}/bin/mkwallet -i welcome
${ORACLE_HOME}/sysman/config/monwallet
${ORACLE_HOME}/sysman/config/b64SiteCertificate.txt NZDST_CLEAR_PTP
Consider the following before installing the standby Management Services.
Oracle recommends that this activity be done during a lean period or during a planned maintenance window. When new Management Services are installed on the standby site, they are initially configured to connect to the Management Repository database on the primary site. Some workload will be taken up by the new Management Service. This could result in temporary loss in performance if the standby site Management Services are located far away from the primary site Management Repository database. However there would be no data loss and the performance would recover once the standby Management Services are shutdown post configuration.
The shared storage used for the shared filesystem loader and software library must be made available on the standby site using the same paths as the primary site.
Install the first standby Management Service using the following steps:
Copy the emkey to the Management Repository by running the following command on the first Management Service on the primary site:
emctl config emkey -copy_to_repos
Perform a software-only install by running the installer with the following arguments:
runInstaller -noconfig -validationaswarnings
Apply one-off patches
<OMS Oracle Home>/install/oneoffs/apply_NewOneoffs.pl <OMS Oracle Home> OC9321514,9207217
Configure the Management Service by running omsca in standby mode. Choose a different domain name for the standby. For example, if the primary WebLogic domain is GCDomain, choose GCDomainStby.
omsca standby -EM_DOMAIN_NAME GCDomainStby -nostart
When prompted for Management Repository details, provide the Primary database details.
Configure the virtualization add on by running the following command:
addonca -oui -omsonly -name vt -install gc
Configure the Management Agent that comes with the Management Service install by running:
<Agent Oracle Home>/bin/agentca -f
Export the configuration from the primary Management Service using:
emctl exportconfig oms -dir <location for the export file>
Copy over the exported file to the standby Management Service
Import the exported configuration on the standby Management Service using:
emctl importconfig oms -file <full path of the export file>
Install addional standby Management Services as follows:
Specify the primary database details and the standby administration server details on the installer screens. Post installation, do the following steps to complete the HA configuration.
Do a software only install by running the installer with following arguments:
runInstaller -noconfig -validationaswarnings
Apply one-off patches
<OMS Oracle Home>/install/oneoffs/apply_NewOneoffs.pl <OMS Oracle Home> OC9321514,9207217
Configure the Management Service by running omsca. When prompted for Management Repository details, provide the Primary database details. When prompted for Administration Server details, provide the standby administration server details.
omsca add –nostart
Configure the virtualization add on by running the following command
addonca -oui -omsonly -install gc
Configure the Management Agent that comes with the Management Service install by running:
<Agent Oracle Home>/bin/agentca -f
Export the configuration from the primary Management Service using:
emctl exportconfig oms -dir <location for the export file>
Copy over the exported file to the standby Management Service
Import the exported configuration on the standby Management Service using:
emctl importconfig oms -file <full path of the export file>
Validate your installation and complete the setup as follows:
Update the standby SLB configuration by adding the standby Management Services to the different pools on the SLB. Setup monitors for the new Management Service.
Make the standby Management Services point to the standby Management Repository database by running the following command on the first standby Management Service:
emctl config oms -store_repos_details -repos_conndesc <connect descriptor of standby database> -repos_user sysman -no_check_db
Shut down all Management Services by running the following command on each Management Service:
emctl stop oms -all
This section provides a general reference for Cloud Control administrators who want to configure Enterprise Manager Cloud Control in Cold Failover Cluster (CFC) environments.
The following conditions must be met for Cloud Control to fail over to a different host:
The installation must be done using a Virtual Host Name and an associated unique IP address.
Install on a shared disk/volume which holds the binaries, the configuration and the runtime data (including the recv directory).
Configuration data and metadata must also failover to the surviving node.
Inventory location must failover to the surviving node.
Software owner and time zone parameters must be the same on all cluster member nodes that will host this Oracle Management Service (OMS).
To override the physical host name of the cluster member with a virtual host name, software must be installed using the parameter ORACLE_HOSTNAME. For inventory pointer, the software must be installed using the command line parameter -invPtrLoc to point to the shared inventory location file, which includes the path to the shared inventory location.
If you are using an NFS mounted volume for the installation, please ensure that you specify rsize and wsize in your mount command to prevent running into I/O issues.
For example:
oms.acme.com:/u01/app/share1 /u01/app/share1 nfs rw,bg,rsize=32768,wsize=32768,hard,nointr,tcp,noac,vers=3,timeo=600 0 0
Note:
Any reference to shared failover volumes could also be true for non-shared failover volumes which can be mounted on active hosts after failover.You can set up the virtual host name and virtual IP address by either allowing the clusterware to set it up, or manually setting it up yourself before installation and startup of Oracle services. The virtual host name must be static and resolvable consistently on the network. All nodes participating in the setup must resolve the virtual IP address to the same host name. Standard TCP tools such as nslookup and traceroute can be used to verify the host name. Validate using the following commands:
nslookup <virtual hostname>
This command returns the virtual IP address and full qualified host name.
nslookup <virtual IP>
This command returns the virtual IP address and fully qualified host name.
Be sure to try these commands on every node of the cluster and verify that the correct information is returned.
Storage can be managed by the clusterware that is in use or you can use any shared file system (FS) volume as long as it is not an unsupported type, such as OCFS V1. The most common shared file system is NFS.
Note:
If the OHS directory is on a shared storage, the LockFile directive in the httpd.conf file should be modified to point to a local disk, otherwise there is a potential for locking issues.Some operating system versions require specific operating system patches be applied prior to installing 11gR1. The user installing and using the 11gR1 software must also have sufficient kernel resources available. Refer to the operating system's installation guide for more details.Before you launch the installer, certain environment variables need to be verified. Each of these variables must be identically set for the account installing the software on ALL machines participating in the cluster:
OS variable TZ
Time zone setting. You should unset this variable prior to installation
PERL variables
Variables such as PERL5LIB should also be unset to avoid association to the incorrect set of PERL libraries
The user and group of the software owner should be defined identically on all nodes of the cluster. This can be verified using the 'id' command:
$ id -a
uid=550(oracle) gid=50(oinstall) groups=501(dba)
Use the following steps to set up shared inventory:
Create your new ORACLE_HOME directory.
Create the Oracle Inventory directory under the new oracle home:
$ cd <shared oracle home>
$ mkdir oraInventory
Create the oraInst.loc file. This file contains the Inventory directory path information needed by the Universal Installer.
vi oraInst.loc
Enter the path information to the Oracle Inventory directory and specify the group of the software owner as the oinstall user. For example:
inventory_loc=/app/oracle/product/11.1/oraInventory
inst_group=oinstall
Refer to the following steps when installing the software:
Create the shared disk location on both the nodes for the software binaries
Install WebLogic Server. For information on installing WebLogic Server, refer to Oracle Enterprise Manager Cloud Control Basic Installation Guide.
Point to the inventory location file oraInst.loc (under the ORACLE_BASE in this case), as well as specifying the host name of the virtual group. For example:
$ export ORACLE_HOSTNAME=lxdb.acme.com $ runInstaller -invPtrloc /app/oracle/share1/oraInst.loc ORACLE_HOSTNAME=lxdb.acme.com -debug
Install Oracle Management Services on cluster member Host1 using the option, "EM install using the existing DB"
Continue the remainder of the installation normally.
Once completed, copy the files oraInst.loc and oratab to /etc. Also copy /opt/oracle to all cluster member hosts (Host2, Host3, and so on).
On Windows environments, an additional step is required to copy over service and keys required by the Oracle software. Note that these steps are required if your clustering software does not provide a shared windows registry.
Using regedit on the first host, export each Oracle service from under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services.
Using regedit on the first host, export HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE.
Use regedit to import the files created in step 1 and 2 to the failover host.
Ensure that you start your services in the proper order. Use the order listed below:
Establish IP address on the active node
Start the TNS listener (if it is part of the same failover group)
Start the database (if it is part of the same failover group)
Start Cloud Control using emctl start oms
Test functionality
In case of failover, refer to the following steps:
Establish IP on failover box
Start TNS listener using the command lsnrctl start if it is part of the same failover group
Start the database using the command dbstart if it is part of the same failover group
Start Cloud Control using the command emctl start oms
Test the functionality
Before installing Enterprise Manager, you should prepare the database, which will be used for setting up Management Repository. Install the database using Database Configuration Assistant (DBCA) to make sure that you inherit all Oracle install best practices.
Configure Database
For both high availability and scalability, you should configure the Management Repository in the latest certified database version, with the RAC option enabled. Check for the latest version of database certified for Enterprise Manager from the Certify tab on the My Oracle Support website.
Choose Automatic Storage Management (ASM) as the underlying storage technology.
When the database installation is complete:
Go to $ORACLE_HOME/rbdms/admin directory of the database home and execute the 'dbmspool.sql'
This installs the DBMS_SHARED_POOL package, which will help in improving throughput of the Management Repository.
Install Enterprise Manager
While installing Enterprise Manager using Oracle Universal Installer (OUI), you will be presented with the option for configuring the Management Repository using an existing database.
There are some parameters that should be configured during the Management Repository database install (as previously mentioned) and some parameters that should be set after the Management Service has been installed.
Start by installing Management Agents on each Management Repository RAC node. Once the Management Agents are installed and the Management Repository database is discovered as a target, the Enterprise Manager console can be used to configure these best practices in the Management Repository.
These best practices fall in the area of:
Configuring Storage
Configuring Oracle Database 11g with RAC for High Availability and Fast Recoverability
Enable ARCHIVELOG Mode
Enable Block Checksums
Configure the Size of Redo Log Files and Groups Appropriately
Use a Flash Recovery Area
Enable Flashback Database
Use Fast-Start Fault Recovery to Control Instance Recovery Time
Enable Database Block Checking
Set DISK_ASYNCH_IO
The details of these settings are available in Oracle Database High Availability Best Practices.
Use the MAA Advisor for additional high availability recommendations that should be applied to the Management Repository. To access the MAA Advisor:
On the Database Target Home page, locate the High Availability section.
Click Details next to the Console item.
In the Availability Summary section of the High Availability Console page, click Details located next to the MAA Advisor item.
The scenario used in the following conversion process provisions a two-node cluster, creates an additional physical standby database that uses Oracle ASM or some type of shared storage on the new server, and converts the standby database to an Oracle RAC database. Finally, a switchover operation is performed to move the new Oracle RAC standby database to the primary database role.
Note:
The MAA recommended best practice is to add a standby database during this process even if there is already a standby database in place. This is to maintain the same level of protection from disasters during all phases of this process, with no compromise to the existing availability solution. Thus, if the starting configuration contained a standby database, the ending configuration would contain an Oracle RAC primary database and two single-instance standby databases. After the switchover is performed, either of the single-instance physical standby databases can be removed with no negative impact.Conversion Tasks
Task 1: Provision Oracle Clusterware, ASMTask 2: Create a Single Instance Physical Standby Database on the New Cluster , and Oracle RAC
Task 3: Prepare the Environment Prior to Conversion to Oracle RAC
Task 4: Convert a Physical Standby Database to Oracle RAC
Task 5: Perform a switchover and Enable Additional Threads
This section describes using Deployment Procedures to provision a two-node cluster running Oracle Clusterware, Oracle ASM, and Oracle RAC. If you have already provisioned Oracle RAC software onto the new server, skip to Task 2 to create the physical standby database.
The process described in this section uses gold images of Oracle Clusterware, Oracle ASM, and Oracle RAC that were pre-created and stored in the Software Library.
Also, see “Provisioning Oracle RAC Using Gold Image” in the Oracle Enterprise Manager Administrator's Guide for Software and Server Provisioning and Patching for additional prerequisite information.
Perform the following steps:
In Cloud Control, click the Deployments tab.
On the Deployments page, in the Deployment Procedure Manager section, click RAC Provisioning Procedures.
On the Deployment Procedure Manager page, in the Procedure subtab, select to run the Deployment Procedure for Oracle Clusterware and Oracle RAC that you created previously in the “Install Software and Upload Components” section.
Note: Ensure the customized copy of the Deployment Procedure uses the appropriate commands (for example, sudo) for step execution.
Click Schedule Deployment. Enterprise Manager Cloud Control displays the Select Source page of the Deployment Procedure.
4. On the Select Source page, do the following:
a. In the Select Source section, select Select from Software Library.
b. In the Source for Clusterware section, click the torch icon and select the Component that has the gold image of Oracle Clusterware.
c. In the Source for RAC section, click the torch icon and select the Component that has the gold image of Oracle RAC.
d. In the Source for ASM section, choose whether or not you want to deploy Oracle ASM. The MAA team chose to deploy Oracle ASM using the same component for the ASM Oracle home as was used for the Oracle RAC Oracle home.
Note: Ensure that you select only Components that are in "Active" status. Once you select the component name, the application automatically displays the component location.
5. On the Select Hosts page, perform the following:a. In the Hosts to Include in Cluster section, click Add and select the target hosts that should form the cluster.By default, the Show Suitable Hosts option is selected and the table lists only those hosts that are best suited for provisioning. If you do not find the host you want to add, then select Show All Hosts to view a complete list of hosts.By default, the procedure automatically pre-fills the Private Host Name and Virtual Host Name fields with values. Ensure the correct Virtual Host Name for the VIP is used. If necessary, edit them to specify values that match your environment. Optionally, you can also specify IP addresses. For example:b. In the Hosts to Include in Cluster section, configure the private and public network interfaces by clicking Select Interfaces. By default, the interfaces that have the same name and subnet for the selected target hosts are displayed. You can also choose Show Interface List to view all the interfaces for the selected target hosts. Select one of the existing interfaces or specify a completely new one. Click OK.c. In the Network Interface Configuration section, review the details of the private and public interfaces.Click Next.6. On the Credentials/Schedule page:a. In the Hosts section, specify the Host Credentials (username and password that will be used to access the server during installation) to be used for the target.You can opt to retain the default selection (Preferred) so that the preferred credentials stored in the Management Repository are used, or override the preferred credentials to explicitly specify the host credentials.b. In the Schedule section, schedule the Deployment Procedure to run either immediately or later.c. Click Next.7. On the Configure Cluster page:a. In the Cluster Name and Location section, review the default name and location details provided for Oracle Clusterware, Oracle RAC Database, and Oracle ASM and edit them if necessary. For example:In this example, the default cluster name is based on the host cluster name you provided in the Agent Deploy application in Enterprise Manager Cloud Control, while deploying Management Agents on a cluster. The scratch location is a temporary location on the target host where temporary files are placed before provisioning and configuring Oracle RAC.For security purposes, the clusterware configuration sets the ownership of Oracle Clusterware home and all its parent directories to be owned by root. Hence, Oracle recommends that you install Oracle Clusterware outside the Oracle base of the Oracle RAC home.b. In the Database Details section, by default, a starter database is created. However, because this environment will be used to create a standby database, no starter database is needed. Deselect the Create Starter Database checkbox.c. In the ASM Instance Details section (that appears only if you had selected to deploy Oracle ASM), provide the password for the SYS user and the Oracle ASM disk string. For example:Click Next.
8. On the Storage page, do the following:
a. On the Shared Storage Configuration section, specify locations in the partition name and the mount location fields for the Oracle Cluster Registry (OCR), voting disks, and data files. Also, select the mount format and a storage device for storing data. While partition name is the path to the location where the device is installed, mount location is the mount point that represents the partition location.
If you are using raw devices, you can select the Clear raw devices checkbox under the table to clear the devices as a part of the installation. Doing so ensures the installation does not fail due to information remaining on the devices from previous installations.b. In the Advanced Options section, select a checkbox for ASM redundancy: None, Normal, or High.
Click Next.
9. On the Advance Configuration page, do the following:
a. In the Configure the Bonding Interface (Private Interconnect) section, if necessary. select Configure Bonding Interface to configure the bonding interface. For more information about the individual settings, click Help in the Enterprise Manager console.
b. In the Sysctl File Configuration section, select Configure Sysctl file if you want to configure the sysctl.conf file. Specify the mode of editing the system configuration file and the location of the reference system configuration file used for modifying the kernel parameters. For more information about the individual settings, click Help in the Enterprise Manager console.
10. On the “Configure Oracle Home“ page, you can optionally choose to install and initiate the configuration manager to receive security updates:
• If the host where the database is being provisioned has a direct connection to the Internet, then specify an e-mail address and My Oracle Support password to install and initiate the configuration manager. An e-mail address is required so that security updates and install updates can be sent from My Oracle Support.
• If the host where the database is being provisioned has an indirect connection to the Internet through a proxy server, then specify an e-mail address and My Oracle Support password, and then in the Connection Details section, specify the proxy server details.
Click Next.
11. On the Review page, review the details you have provided for provisioning Oracle RAC and click Finish to submit the job to Enterprise Manager.
After the Deployment Procedure ends successfully you can verify the cluster configuration on the Target tab. In the following MAA example you can verify the clusterware home and the hosts included in the cluster:
The scenario used in the following steps creates a new physical standby database using the Oracle RAC home on the new cluster. This new standby database will then be converted to Oracle RAC during Task 4. During standby creation, create the database files on shared storage to facilitate conversion to Oracle RAC later. The MAA recommendation is to use Oracle ASM managed storage.
Follow the instructions in <link to creating a standy database> to create a single-instance physical standby database on some form of shared storage using the newly installed Oracle RAC home that you created in Task 1. In this example, the new physical standby database is called racsby. The standby database is created on Oracle ASM managed storage.
Prepare the environment prior to conversion to Oracle RAC by performing the following tasks:
1. Verify that the STANDBY_FILE_MANAGEMENT initialization parameter is set to AUTO on both the primary database and on the standby database that is to be converted.
2. Run the $ORACLE_HOME/rdbms/admin/catclust.sql script on the primary database to install the cluster database views into the environment.
Note: This step is required only if the configuration does not already include a RAC database.
3. Manually create a second undo tablespace on the primary database to support the new database instance being created. The following SQL statement is an example:
create undo tablespace undotbs2 datafile '/u01/app/oracle/oradata/gcprim/undotbs02.dbf' size 500M autoextend on retention guarantee;
Note: This step is required only if there are not enough undo tablespaces already created to support the new Oracle RAC database. You must create an undo tablespace for each new database instance to be added. The Convert to Cluster wizard (executed in Task 4) will display an error if there are not enough undo tablespaces created.
4. Remove the standby database to be converted from the Data Guard broker configuration.
Note: It is necessary to temporarily remove the new standby database from broker management in order to update the broker configuration file initialization parameters. The next step describes how to change the parameters to re-create the broker configuration files on Oracle ASM shared storage (instead of the current location on file system storage).
Perform the following steps to prepare the standby database for conversion to an Oracle RAC standby database. In the following examples, the standby database to be converted is called racsby.
1. In Cloud Control, click the Targets tab, and then click the Databases subtab.
2. On the Databases page, select the primary database (gcprim) from the table.
3. On the gcprim database home page, click Primary in the High Availability section.
4. On the Data Guard page for gcprim, select the standby database that is to be converted (this is the racsby database in the MAA examples), and click Remove. For example:
You should click the check box to “Preserve the destination corresponding to this standby” to continue shipping redo data to the standby.
The Remove Standby Database wizard asks you to confirm the removal and then proceeds to remove the racsby standby database from the broker configuration.
Note: Removing the standby database from the broker configuration only removes the entries from the broker configuration files. The removal does not delete the database from the Data Guard configuration. In fact, the current state of the physical standby database does not change.
5. Modify the following broker initialization parameters for the racsby standby database by using Initialization Parameters option on the Server tab:
• The broker configuration files must be on shared storage when used in conjunction with a RAC database. Edit the DG_BROKER_CONFIG_FILE1 and DG_BROKER_CONFIG_FILE2 parameter values to set them to a location on Oracle ASM shared storage. The recommended Oracle ASM location is in the same diskgroup as the datafiles for the database. Note this directory must exist prior to adding the standby database back into the broker configuration.
• Set the DG_BROKER_START initialization parameter to FALSE to disable the broker. This is necessary for the parameter changes for the broker configuration file locations to take effect.
Click Save to File.
6. Add the racsby standby database back into the broker configuration. This step completes the action of relocating the broker configuration files into the new Oracle ASM shared storage locations.
a. In Cloud Control click Targets, and then click Databases.
b. On the Databases page, select the primary database (gcprim in this example).
c. On the Database Instance page for the primary database, click Primary in the list under the High Availability section of the page.
d. On the Data Guard page for the primary database, click Add Standby Database to invoke the Add Standby Database wizard.
Note: The following steps automatically re-enable the broker management of the racsby standby database.
The following sequence of steps is similar to the process described in Module 1: Create a Physical Standby Database except you will be enabling broker management only for an existing standby database rather than creating a new standby. Respond to the wizard dialog as follows:
• On the Add Standby Database page, select “Manage an existing standby database” and click Continue.
• On the Select Existing Standby Database page, select the standby database from the table. In this example, this is the racsby database. Click Next.
• On the Configuration page, click Next.
• On the Review page, review the configuration settings and click Finish to add the standby database back into the broker configuration. The following screenshot shows the broker configuration running with the gcprim primary database and the two physical standby databases: gcsby and racsby.
This task converts the newly created physical standby database into an Oracle RAC database, with no downtime occurring to the primary database.
Figure 4 shows the state of the configuration after you perform the steps to convert the new racsby physical standby database to an Oracle RAC standby database. The configuration contains a single-instance primary database, a single-instance physical standby database, and an Oracle RAC physical standby database.
Figure 4: Intermediate State 2 for Module 2 After Conversion to Oracle RAC
Perform the following steps to convert the racsby standby database to Oracle RAC:
1. In Cloud Control click Targets, and then click Databases.
2. On the Databases page, select the standby database (racsby in our example) from the table and then on the standby database home page, click the Server subtab.
3. On the Server page in the Change Database section, click Convert to Cluster. The Convert to Cluster Database wizard starts.
4. Respond to the Convert to Cluster Database wizard, as follows:
a. On the Cluster Credentials page, specify the credentials for the Oracle RAC and Oracle ASM homes.
Note: In the Information section of the page, you can see the wizard recognizes that the racsby database is a standby database for the gcprim primary database.
On this page, only login credentials were needed to be supplied because the database was already configured to use Oracle RAC and Oracle ASM homes.
Click Next.
b. On the Hosts page, select the hosts from the table on which you want to run the converted Oracle RAC database. Note: The current host for the standby database is always selected.
Click Next.
c. On the Options page, select to use either an existing listener or create a new listener, and specify a prefix to be used to name the cluster database instance (ORACLE_SID).
d. On the Shared Storage page, specify the data file and FLASH_RECOVERY_AREA storage locations. If the database is to be converted to Oracle RAC in-place (that is, the files are already located on shared storage), then you can use the existing locations. Otherwise specify the target disk groups for the data files and the FLASH_RECOVERY_AREA.
e. Review the settings and click Submit to run the Convert Cluster Database job in Enterprise Manager.
5. After the Convert Cluster Database job has completed successfully, remove the original (non Oracle RAC) standby database definition (racsby) from Cloud Control.
Warning: When you remove the old database from Cloud Control, all the monitoring history is deleted. Remove a database only when this data is no longer needed.
a. In Cloud Control, click Targets.
b. On the Databases page, notice that there are two entries for same standby database: one is for the original single-instance standby database and the other is for the new clustered standby database. Select the single-instance standby database definition from the table and click Remove.
This section describes how to perform a switchover to complete the database conversion to Oracle RAC and to enable the new thread on the Oracle RAC database.
In the MAA example used in this section, the primary database is gcprim and has a single-instance standby database with the database unique name gcsby. The primary database is running on a file system. The new Oracle RAC database unique name is racsby and resides on.
Oracle ASM. After the switchover, racsby is the Oracle RAC primary database, and gcprim is a single-instance physical standby database.Perform a Switchover
Perform the following steps to switchover the newly converted Oracle RAC standby database (racsby) to run in the primary database role.
1. In Cloud Control, click Targets and then click the All Targets subtab.
2. Select the gcprim database hyperlink.
3. Select Details in the High Availability section.4. On the Data Guard page, select the Oracle RAC physical standby database that you want to switch to the primary database role, and click Switchover.
After the switchover completes, the Data Guard page shows the roles have been switched between racsby (now an Oracle RAC primary database), and gcprim (now a single-instance physical standby database).
Enable Additional Threads
Manually enable additional threads on the Oracle RAC primary database to make this a two-node Oracle RAC database. For example:
SQL> ALTER DATABASE ENABLE PUBLIC THREAD 2;
Database altered.
Note: Repeat this step for each additional thread added.
Figure 5 shows the configuration that results after you have completed the steps to perform a switchover and enable the additional database threads. The configuration contains an Oracle RAC primary database and two single-instance physical standby databases.
Figure 5: Intermediate State 3 for Module 2 after Switchover
At this point, you can optionally remove the additional physical standby database. Figure 6 shows the ending configuration containing an Oracle RAC primary database and one single-instance physical standby database.
Management Service processes need to be configured to communicate with each node of the RAC Management Repository in a redundant fashion.
Note that Real Application Cluster (RAC) nodes are referred to by their virtual IP (vip) names. The service_name parameter is used instead of the system identifier (SID) in connect_data mode and failover is turned on. Refer to Oracle Database Net Services Administrator's Guide for details.
Configure the repository connect descriptor by running the emctl command from any Management Service:
emctl config oms -store_repos_details -repos_conndesc '"(DESCRIPTION= (ADDRESS_LIST=(FAILOVER=ON) (ADDRESS=(PROTOCOL=TCP)(HOST=node1-vip.example.com)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=node2-vip.example.com)(PORT=1521))) (CONNECT_DATA=(SERVICE_NAME=EMREP)))"' -repos_user sysman
After making the previous change, run the following command to make the same change to the monitoring configuration used for the Management Services and Repository target: emctl config emrep -conn_desc
The starting point of this step is to have the primary site configured as per Cloud Control MAA guidelines. The following steps lay down the procedure for setting up the standby Management Repository database.
Prepare Standby Management Repository hosts for Data Guard
Install a Management Agent on each of the standby Management Repository hosts. Configure the Management Agents to upload by the SLB on the primary site. Install CRS and Database software on the standby Management Repository hosts. The version used must be the same as that on the primary site.
Prepare Primary Management Repository database for Data Guard
If the primary Management Repository database is not already configured, enable archive log mode, setup flash recovery area and enable flashback database on the primary Management Repository database.
Create Physical Standby Database
In Enterprise Manager, the standby Management Repository database must be physical standbys. Use the Enterprise Manager Console to setup a physical standby database in the standby environment. Note that Enterprise Manager Console does not support creating a standby RAC database. If the standby database has to be RAC, configure the standby database using a single instance and then use the Convert to RAC option from Enterprise Manager Console to convert the single instance standby database to RAC. Also, note that during single instance standby creation, the database files should be created on shared storage to facilitate conversion to RAC later.
Note that the Convert to RAC option is available for Oracle Database releases 10.2.0.5, 11.1.0.7, and above. Oracle Database release 11.1.0.7 requires patch 8824966 for the Convert to RAC option to work.
Add Static Service to Listener
To enable Data Guard to restart instances during the course of broker operations, a service with a specific name must be statically registered with the local listener of each instance. The value for the GLOBAL_DBNAME attribute must be set to a concatenation of <db_unique_name>_DGMGRL.<db_domain>. For example, in the LISTENER.ORA file:
SID_LIST_LISTENER=(SID_LIST=(SID_DESC=(SID_NAME=sid_name)
(GLOBAL_DBNAME=db_unique_name_DGMGRL.db_domain)
(ORACLE_HOME=oracle_home)))
Enable Flashback Database on the Standby Database
Verify Physical Standby
Verify the Physical Standby database through the Enterprise Manager Console. Click the Log Switch button on the Data Guard page to switch log and verify that it is received and applied to the standby database.
While high availability typically protects against local outages such as application failures or system-level problems, disaster tolerance protects against larger outages such as catastrophic data-center failure due to natural disasters, fire, electrical failure, evacuation, or pervasive sabotage. For Maximum Availability, the loss of a site cannot be the cause for outage of the management tool that handles your enterprise.
Maximum Availability Architecture for Enterprise Manager mandates deploying a remote failover architecture that allows a secondary datacenter to take over the management infrastructure in the event that disaster strikes the primary management infrastructure.
Setting up disaster recovery for Enterprise Manager essentially consists of installing a standby Management Repository Database, a standby Management Service and a standby Server Load Balancer and configuring them to automatically startup when the primary components fail.
Prerequisites
The primary site must be configured as per Cloud Control MAA guidelines described in previous sections. This includes Management Services fronted by an SLB and all Management Agents configured to upload to Management Services by the SLB.
· The standby site must be similar to the primary site in terms of hardware and network resources to ensure there is no loss of performance when failover happens.
· There must be sufficient network bandwidth between the primary and standby sites to handle peak redo data generation.
· Configure storage used by the software library to be replicated at the primary and standby site. In the event of a site outage, the contents of this storage must be made available on the standby site using hardware vendor disk level replication technologies.
· For complete redundancy in a disaster recovery environment, a second load balancer must be installed at the standby site. The secondary SLB must be configured in the same fashion as the primary. Some SLB vendors (such as F5 Networks) offer additional services that can be used to pass control of the Virtual IP presented by the SLB on the primary site to the SLB on the standby site in the event of a site outage. This can be used to facilitate automatic switching of Management Agent traffic from the primary site to the standby site.
Consider the following before installing the standby Management Services.
· Oracle recommends that this activity be done during a lean period or during a planned maintenance window. When new Management Services are installed on the standby site, they are initially configured to connect to the Management Repository database on the primary site. Some workload will be taken up by the new Management Service. This could result in temporary loss in performance if the standby site Management Services are located far away from the primary site Management Repository database. However there would be no data loss and the performance would recover once the standby Management Services are shutdown post configuration.
· The shared storage used for the software library must be made available on the standby site using the same paths as the primary site.
Install the first standby Management Service using the following steps:
Copy the emkey to the Management Repository by running the following command on the first Management Service on the primary site:
emctl config emkey -copy_to_repos
Export the configuration from the first Management Service on the primary site using:
emctl exportconfig oms -dir <location for the export file>
After the configuration is exported, do not make any configuration changes to the primary site still the standby management service is configured.
Install a Management Agent on the standby host if one does not already exist.
Perform a software-only install of the Enterprise Manager software using a modified version of the “Add Management Service” Deployment Procedure.
Navigate to Enterprise -> Provisioning and Patching -> Procedure Library.
Select Add Management Service procedure and click on the “Create Like” button.
Go to the Procedure Steps tab and select and disable the steps - “Configure Management Service”, “Targets Discovery” and “Post Configuration Tasks”.
Save the modified deployment procedure and use it to install the Enterprise Manager software on the standby OMS host.
After the Deployment Procedure completes, delete the file emInstanceMapping.properties from <OMS Oracle Home>/sysman/config on the standby OMS host.
Configure the Management Service by running omsca in standby mode. Choose a different domain name for the standby. For example, if the primary WebLogic domain is GCDomain, choose GCDomainStby.
omsca standby –EM_DOMAIN_NAME GCDomainStby –NM_USER nodemanager -AS_USERNAME weblogic –nostart
When prompted for the Administration Server host and EM Instance host, enter the standby OMS hostname (or accept the default).
When prompted for the passwords, provide the same passwords as the primary site.
When prompted for Management Repository details, provide the Primary database details.
Configure the required plugins by running the following command:
pluginca -action deploy -isFirstOMS true -plugins <plugin-list> -oracleHome <oms oracle home> -middlewareHome <wls middleware home>
where plugin-list is the list of plugins returned by the SQL query
SELECT epv.plugin_id, epv.version FROM em_plugin_version epv, em_current_deployed_plugin ecp WHERE epv.plugin_type NOT IN ( 'BUILT_IN_TARGET_TYPE' , 'INSTALL_HOME') AND ecp.dest_type='2' AND epv.plugin_version_id = ecp.plugin_version_id;
and is a comma separate list in the following format: <plugin-id>=<plugin-version>,<plugin-id>=<plugin-version>,…
Example: "oracle.sysman.empa=12.1.0.1.0,oracle.sysman.mos=12.1.0.1.0,oracle.sysman.emas=12.1.0.1.0,oracle.sysman.emfa=12.1.0.1.0,oracle.sysman.db=12.1.0.1.0,oracle.sysman.emct=12.1.0.1.0,oracle.sysman.vt=12.1.0.1.0,oracle.sysman.ssa=12.1.0.1.0"
Copy over the configuration exported from the Primary Management Service in step 2 above to the standby Management Service host. Import the exported configuration on the standby Management Service using:
emctl importconfig oms -file <full path of the export file>
Note this command emits a warning about a failed export and prompts for confirmation to proceed. The warning can be ignored by entering "y" to proceed.
Note this command will start the Management Service.
Stop the Management Service but leave the Administration Server running using:
emctl stop oms
Add the standby Weblogic Domain and associated targets:
The standby Weblogic Domain and associated targets can be added using the Guided Discovery process via the Setup -> Add Target -> Add Targets Manually page, selecting 'Oracle Fusion Middleware' from the 'Target Types' drop-down. Use the secure port (typically 7101) and, under 'Advanced', set the JMX Protocol to “t3s”.
Note that the Weblogic targets except the Administration Server will be shown as down as the standby OMS is down at this stage.
If you have Single Sign On configured on the primary site, follow the same steps to configure SSO on the standby OMS.
If you have Real User Experience Insight, AD4J Manager or BI Publisher configured on the primary site, follow the same steps to configure them on the standby OMS.
It is recommended that your standby site be similar in configuration as your primary site. This means configuring multiple OMS on your standby site, similar to your primary site. Install additional standby Management Services as per the procedure listed below under “Additional Standby Management Services”.
If, however, you choose to start with a single OMS on the standby site initially, you may skip this step and continue with the next section “Validating your installation and Complete the Setup”. If you decide to add an additional standby OMS later after having run the “Validating your installation and Complete the Setup” steps, the steps listed under “Additional Standby Management Services” can be followed after executing the following additional steps:
Start up the standby Administration Server by running the following command on the first standby Management Service:
emctl start oms –admin_only
Additional Standby Management Services
Export the configuration from the first Management Service on the primary site using:
emctl exportconfig oms -dir <location for the export file>
After the configuration is exported, do not make any configuration changes to the primary site still the standby management service is configured.
Install a Management Agent on the standby host.
Perform a software-only install of the Enterprise Manager software using a modified version of “Add Management Service” Deployment Procedure.
Navigate to Enterprise -> Provisioning and Patching -> Procedure Library.
Select Add Management Service procedure and click on “Create Like” button.
Go to the Procedure Steps tab and select and disable the steps - “Configure Management Service”, “Targets Discovery” and “Post Configuration Tasks”.
Save the modified deployment procedure and use it to install the Enterprise Manager software on the standby OMS host.
After the Deployment Procedure completes, delete the file emInstanceMapping.properties from <OMS Oracle Home>/sysman/config on the standby OMS host.
Configure the Management Service by running omsca.
omsca add –nostart
When prompted for Management Repository details, provide the Primary database details.
When prompted for Administration Server details, provide the standby administration server details.
Configure the required plugins by running the following command:
pluginca -action deploy –isFirstOMS false -plugins <plugin-list> -oracleHome <oms oracle home> -middlewareHome <wls middleware home>
where plugin-list is the list of plugins returned by the SQL query above and is a comma separate list in the following format: <plugin-id>=<plugin-version>,<plugin-id>=<plugin-version>,…
Example "oracle.sysman.empa=12.1.0.1.0,oracle.sysman.mos=12.1.0.1.0,oracle.sysman.emas=12.1.0.1.0,oracle.sysman.emfa=12.1.0.1.0,oracle.sysman.db=12.1.0.1.0,oracle.sysman.emct=12.1.0.1.0,oracle.sysman.vt=12.1.0.1.0,oracle.sysman.ssa=12.1.0.1.0"
Copy over the configuration exported from the Primary Management Service in step 1 above to the standby Management Service host. Import the exported configuration on the standby Management Service using:
emctl importconfig oms -file <full path of the export file>
Note this command emits a warning about a failed export and prompts for confirmation to proceed. The warning can be ignored by entering "y" to proceed.
Note this command will start the Management Service.
Stop the Management Service using:
emctl stop oms
Refresh the standby domain target from the console. This will present a guided workflow to discover and add the new managed server and associated targets.
If you have Single Sign On configured on the primary site, follow the same steps to configure SSO on the standby OMS.
If you have Real User Experience Insight, AD4J Manager or BI Publisher configured on the primary site, follow the same steps to configure them on the standby OMS.
Validating Your Installation and Complete the Setup
Update the standby SLB configuration by adding the standby Management Services to the different pools on the SLB. Setup monitors for the new Management Service.
The starting point of this step is to have the primary site configured as per Cloud Control MAA guidelines. Please refer to this document <need a link here to detailed steps of configuring a Physical Standby> for details steps. Note the following points.
The Standby Management Repository database must be a Physical Standby. Logical standby are not supported.
Use the Enterprise Manager itself to setup a physical standby database in the standby environment. Note that Enterprise Manager Console does not support creating a standby RAC database. If the standby database has to be RAC, configure the standby database using a single instance and then use the Convert to RAC option from Enterprise Manager Console to convert the single instance standby database to RAC. Also, note that during single instance standby creation, the database files should be created on shared storage to facilitate conversion to RAC later.
Note that the Convert to RAC option is available for Oracle Database releases 10.2.0.5, 11.1.0.7, and above. Oracle Database release 11.1.0.7 requires patch 8824966 for the Convert to RAC option to work.
Add Static Service to Listener
To enable Data Guard to restart instances during the course of broker operations, a service with a specific name must be statically registered with the local listener of each instance. The value for the GLOBAL_DBNAME attribute must be set to a concatenation of <db_unique_name>_DGMGRL.<db_domain>. For example, in the LISTENER.ORA file:
SID_LIST_LISTENER=(SID_LIST=(SID_DESC=(SID_NAME=sid_name)
(GLOBAL_DBNAME=db_unique_name_DGMGRL.db_domain)
(ORACLE_HOME=oracle_home)))
To allow re-instate of an old primary database as a standby database after a failover, flashback database must be enabled. Hence do so for both the primary and the standby databases.
To allow Enterprise Manager to monitor a Physical Standby database (which is typically in a mounted state), specify sysdba monitoring privileges. This can be specified either during the Standby creation wizard itself or post creation by modifying the Monitoring Configuration for the standby database target.
|
 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Legal Notices |
|