This chapter describes operations that you can perform after you have configured the Oracle Business Intelligence topology, including monitoring, scaling, and backing up the enterprise deployment.
This chapter contains the following topics:
Section 13.2, "Starting and Stopping Oracle Business Intelligence"
Section 13.5, "Manually Failing Over the Administration Server to APPHOST2"
Section 13.6, "Performing Backups and Recoveries for Enterprise Deployments"
After configuring the Oracle Business Intelligence enterprise deployment, use the information in this chapter to manage the deployment. This chapter includes topics on typical operations like starting, stopping, monitoring, and patching the deployment
At some point, you might need to expand the topology by scaling it up, or out. See Section 13.4, "Scaling Enterprise Deployments" for information about these tasks.
Back up the topology before and after any configuration changes. See Section 13.6, "Performing Backups and Recoveries for Enterprise Deployments" for information.
This chapter also documents solutions for possible known issues that might occur after you have configured the topology in Section 13.9, "Troubleshooting Enterprise Deployments."
To start Oracle Business Intelligence, you must always start the Managed Servers first, before the system components. In addition, any time that you restart the Managed Servers, you must restart the system components.
For additional information, see "Starting and Stopping Oracle Business Intelligence" in Oracle Fusion Middleware System Administrator's Guide for Oracle Business Intelligence Enterprise Edition.
This section contains the following topics:
Section 13.2.1, "Starting and Stopping Oracle Business Intelligence Managed Servers"
Section 13.2.2, "Starting and Stopping Oracle Business Intelligence System Components"
Perform the following steps to stop, start, or restart Oracle Business Intelligence Managed Servers:
Log in to the Oracle WebLogic Server Administration Console.
Expand the Environment node in the Domain Structure window.
Click Servers. The Summary of Servers page is displayed.
Select the Oracle Business Intelligence Managed Server that you want to manage (for example, bi_server1, bi_server2, and so on).
Perform one of the following actions:
To stop the Managed Server, click Stop.
To start the Managed Server, click Start.
To restart the Managed Server, first click Stop and wait until the server is completely stopped. Then, select the Managed Server again and click Start.
Perform the following steps to stop, start, or restart Oracle Business Intelligence system components:
Log in to Oracle Enterprise Manager Fusion Middleware Control.
Expand the Business Intelligence node in the Farm_domain_name window.
Click coreapplication.
On the Business Intelligence Overview page, click Stop, Start, or Restart.
For information on monitoring the Oracle Business Intelligence topology, see "Monitoring Service Levels" in Oracle Fusion Middleware System Administrator's Guide for Oracle Business Intelligence Enterprise Edition.
See also "Diagnosing and Resolving Issues in Oracle Business Intelligence" in Oracle Fusion Middleware System Administrator's Guide for Oracle Business Intelligence Enterprise Edition for information about Oracle Business Intelligence log files, including rotating and managing logs.
You can scale up or scale out the Oracle Business Intelligence enterprise topology, as follows:
When you scale up the topology, you add additional system components to one of the existing nodes in the enterprise topology.
When you scale out the topology, you add a new node to the topology with a Managed Server and set of system components.
This section includes the following topics:
Section 13.4.1, "Scaling Up the Oracle Business Intelligence Topology"
Section 13.4.2, "Scaling Out the Oracle Business Intelligence Topology"
Note:
To scale out and up the SOA subsystem used by I/PM, refer to the SOA enterprise deployment topology documentation.
This procedure assumes that you already have an enterprise topology that includes two nodes, with a Managed Server and a full set of system components on each node. To scale up the topology, you increase the number of system components that are running on one of the existing nodes.
Note that it is not necessary to run multiple Managed Servers on a given node.
Perform the following steps to scale up the Oracle Business Intelligence enterprise topology:
Log in to Fusion Middleware Control.
Expand the Business Intelligence node in the Farm_domain_name window.
Click coreapplication.
Click Capacity Management, then click Scalability.
Click Lock and Edit Configuration.
Change the number of BI Servers, Presentation Services, or JavaHosts using the arrow keys.
Click Apply, then click Activate Changes.
Click Overview, then click Restart.
When scaling out the topology, you add a new Managed Server and set of system components to a new node in the topology (APPHOST3). This procedure assumes that you already have an enterprise topology that includes two nodes, with a Managed Server and a full set of system components on each node.
Before performing the steps in this section, check that you meet these requirements:
There must be existing nodes that are running Oracle Business Intelligence Managed Servers within the topology.
The new node (APPHOST3) can access the existing home directories for Oracle WebLogic Server and Oracle Business Intelligence.
When an ORACLE_HOME or WL_HOME is shared by multiple servers in different nodes, it is recommended that you keep the Oracle Inventory and Middleware home list in those nodes updated for consistency in the installations and application of patches. To update the oraInventory in a node and "attach" an installation in a shared storage to it, use ORACLE_HOME/oui/bin/attachHome.sh. To update the Middleware home list to add or remove a WL_HOME, edit the MW_HOME/.home file. See the steps in Section 13.4.2.1, "Scale-out Procedure for Oracle Business Intelligence".
You must ensure that all shared storage directories are available on the new node. Ensure that all shared directories that are listed in Section 4.3.4, "Recommended Directory Locations" are available on all nodes, except for the ORACLE_INSTANCE directory and the domain directory for the scaled out Managed Server.
Also, if you use shared storage for the identity keystore and trust keystore that hold the host name verification certificates, then ensure that the shared storage directory is accessible from the scaled-out node (APPHOST3). If you use local directories for the keystores, then follow the steps in Section 10.3, "Enabling Host Name Verification Certificates for Node Manager" to create and configure a local identity keystore for the scaled-out node.
For example, mount the following directories:
Transaction Log directory
JMS Persistence Store
Global Cache
BI Presentation Catalog
BI Repository Publishing directory
BI Publisher Catalog
BI Publisher Configuration Keystore (certs)
MW_HOME
Perform the following steps to scale out Oracle Business Intelligence on APPHOST3:
On APPHOST3, mount the existing Middleware home, which includes the Oracle Business Intelligence installation and (optionally, if the domain directory for Managed Servers in other nodes resides on shared storage) the domain directory, and ensure that the new node has access to this directory, similar to the rest of the nodes in the domain.
To attach ORACLE_HOME in shared storage to the local Oracle Inventory, execute the following command:
APPHOST3> cd ORACLE_COMMON_HOME/oui/bin/ APPHOST3> ./attachHome.sh -jreLoc MW_HOME/jdk
To update the Middleware home list, create (or edit, if another WebLogic installation exists in the node) the MW_HOME/.home file and add ORACLE_BASE/product/fmw to it.
Run the Configuration Assistant from one of the shared Oracle homes, using the steps in Section 9.3, "Scaling Out the Oracle Business Intelligence System on APPHOST2" as a guide.
Note:
Before running the Configuration Assistant, ensure that the appropriate JDK version is installed on the same directory as for the first two nodes. If the appropriate JDK is not installed, then the Configuration Assistant likely fails to start.
Scale out the system components on APPHOST3, using the steps in Section 9.3.2, "Scaling Out the System Components" as a guide.
Configure the bi_server3 Managed Server by setting the Listen Address and disabling host name verification, using the steps in Section 9.4, "Configuring the bi_server2 Managed Server" as a guide.
Configure JMS for BI Publisher, as described in Section 9.5.3.4, "Configuring JMS for BI Publisher."
Set the location of the default persistence store for bi_server 3, as described in Section 9.6.1, "Configuring a Default Persistence Store for Transaction Recovery."
Configure Oracle HTTP Server for APPHOST3VHN1, using the steps in Section 8.4.1, "Configuring Oracle HTTP Server for the Administration Server and the bi_servern Managed Servers" as a guide.
Start the bi_server3 Managed Server and the system components that are running on APPHOST3. See Section 13.2, "Starting and Stopping Oracle Business Intelligence" for details.
Add System Properties to the Server Start Tab for Oracle RTD, as described in Section 9.5.2.2, "Adding System Properties to the Server Start Tab."
Configure server migration for the new node, as described in the following sections:
To validate the configuration, access the following URLs:
Access http://APPHOST3VHN1:9704/analytics to verify the status of bi_server3.
Access http://APPHOST3VHN1:9704/wsm-pm to verify the status of Web Services Manager. Click Validate Policy Manager. A list of policies and assertion templates that are available in the data is displayed.
Note: The configuration is incorrect if no policies or assertion templates are displayed.
Access http://APPHOST3VHN1:9704/xmlpserver to verify the status of the BI Publisher application.
Access http://APPHOST3VHN1:9704/ui to verify the status of the Oracle RTD application.
Access http://APPHOST3VHN1:9704/mapviewer to verify the status of MapViewer.
Access http://APPHOST3VHN1:9704/aps/Essbase to verify the status of the Oracle Essbase application.
Access http://APPHOST3VHN1:9704/aps/SmartView to verify the status of the SmartView application.
Access http://APPHOST3VHN1:9704/workspace to verify the status of the Workspace application.
Access http://APPHOST3VHN1:9704/hr to verify the status of the Financial Reporting application.
Access http://APPHOST3VHN1:9704/calcmgr/index.htm to verify the status of the Calculation Manager application.
Oracle recommends using host name verification for the communication between Node Manager and the servers in the domain. This requires the use of certificates for the different addresses that communicate with the Administration Server and other servers. See Chapter 10, "Setting Up Node Manager for an Enterprise Deployment" for further details.
In case a node fails, you can fail over the Administration Server to another node. This section describes how to fail over the Administration Server from APPHOST1 to APPHOST2, and contains the following topics:
Section 13.5.2, "Validating Access to APPHOST2 Through Oracle HTTP Server"
Section 13.5.3, "Failing the Administration Server Back to APPHOST1"
Note the following assumptions:
The Administration Server is configured to listen on ADMINVHN, and not on ANY address.
The Administration Server is failed over from APPHOST1 to APPHOST2, and the two nodes have these IP addresses:
APPHOST1: 100.200.140.165
APPHOST2: 100.200.140.205
ADMINVHN: 100.200.140.206. This is the VIP where the Administration Server is running, assigned to ethX:Y, available in APPHOST1 and APPHOST2.
The domain directory where the administration server is running in APPHOST1 is on a shared storage and is mounted also from APPHOST2.
Oracle WebLogic Server and Oracle Fusion Middleware components have been installed in APPHOST2 (that is, the same paths for ORACLE_HOME and MW_HOME that exist on APPHOST1 are also available on APPHOST2).
Perform the following steps to fail over the Administration Server to a different node (APPHOST2):
Stop the Administration Server, if it is running.
Migrate the IP address to the second node using the following steps:
Run the following command as root on APPHOST1 (where X:Y is the current interface used by ADMINVHN):
APPHOST1> /sbin/ifconfig ethX:Y down
Run the following command on APPHOST2:
APPHOST2> /sbin/ifconfig interface:index IP_address netmask netmask
For example:
APPHOST2> /sbin/ifconfig eth0:1 10.200.140.206 netmask 255.255.255.0
Note:
Ensure that the netmask and interface to be used match the available network configuration in APPHOST2.
Update the routing tables using arping. For example:
APPHOST2> /sbin/arping -b -A -c 3 -I eth0 100.200.140.206
Start Node Manager in APPHOST2, using the instructions in Step 3 of Section 8.3.3, "Starting the Administration Server on APPHOST1."
Start the Administration Server on APPHOST2, using the instructions in Step 4 of Section 8.3.3, "Starting the Administration Server on APPHOST1."
Test that you can access the Administration Server on APPHOST2, as follows:
Ensure that you can access the Administration Console at http://ADMINVHN:7001/console.
Check that you can access and verify the status of components in Fusion Middleware Control at http://ADMINVHN:7001/em.
Perform the steps that are described in Section 8.4.5, "Validating Access Through Oracle HTTP Server" to check that you can access the Administration Server when it is running on APPHOST2.
This step checks that you can fail back the Administration Server; that is, stop it on APPHOST2 and run it on APPHOST1 again. Perform the following steps to migrate ADMINVHN back to the APPHOST1 node:
Ensure that the Administration Server is not running.
Run the following command as root on APPHOST2.
APPHOST2> /sbin/ifconfig ethX:Y down
Run the following command on APPHOST1:
APPHOST1> /sbin/ifconfig interface:index IP_Address netmask netmask
For example:
APPHOST1> /sbin/ifconfig eth0:1 100.200.140.206 netmask 255.255.255.0
Note:
Ensure that the netmask and interface that you use matches the available network configuration in APPHOST1.
Update the routing tables using arping. For example:
APPHOST1> /sbin/arping -b -A -c 3 -I ethZ 100.200.140.206
Start the Administration Server again on APPHOST1, as described in Step 4 of Section 8.3.3, "Starting the Administration Server on APPHOST1."
Test that you can access the Administration Console at http://ADMINVHN:7001/console.
Check that you can access and verify the status of components in Fusion Middleware Control at http://ADMINVHN:7001/em.
If you encounter problems with Administration Server failover, then see Section 13.9.2, "Administration Server Fails to Start After a Manual Failover" for additional information.
See "Backup and Recovery Recommendations for Oracle Business Intelligence" in Oracle Fusion Middleware Administrator's Guide for full information about backing up and recovering Oracle Business Intelligence.
See "Patching Oracle Business Intelligence Systems" in Oracle Fusion Middleware System Administrator's Guide for Oracle Business Intelligence Enterprise Edition for more information about Oracle Business Intelligence patching.
Much of the EDG production deployment involves firewalls. Because database connections are made across firewalls, Oracle recommends that the firewall be configured so that the database connection is not timed out. For Oracle Real Application Clusters (Oracle RAC), the database connections are made on Oracle RAC VIPs and the database listener port. You must configure the firewall to not time out such connections. If such a configuration is not possible, then set the*SQLNET.EXPIRE_TIME=n* parameter in the ORACLE_HOME/network/admin/sqlnet.ora file on the database server, where n is the time in minutes. Set this value to less than the known value of the timeout for the network device (that is, a firewall). For Oracle RAC, set this parameter in all of the Oracle home directories.
This section includes the following topics:
Section 13.9.1, "Page Not Found When Accessing BI Applications Through Load Balancer"
Section 13.9.2, "Administration Server Fails to Start After a Manual Failover"
Section 13.9.3, "Error While Activating Changes in Administration Console"
Section 13.9.4, "bi_server Managed Server Not Failed Over After Server Migration"
Section 13.9.5, "bi_server Managed Server Not Reachable From Browser After Server Migration"
Section 13.9.6, "OAM Configuration Tool Does Not Remove URLs"
Section 13.9.7, "Users Redirected to Login Screen After Activating Changes"
Section 13.9.8, "Users Redirected to Home Page After Activating Changes"
Section 13.9.11, "Missing JMS Instances on BI Publisher Scheduler Diagnostics Page"
Section 13.9.12, "BI Publisher Jobs in Inconsistent State After Managed Server Shutdown"
Section 13.9.13, "JMS Instance Fails In a BI Publisher Scheduler Cluster"
Section 13.9.14, "Configuring MapViewer when the Administrator Belongs to a Custom Group"
Problem: A 404 "page not found" message is displayed in the web browser when you try to access Oracle Business Intelligence applications (such as Oracle BI Presentation Services, BI Publisher, and Oracle RTD) using the load balancer address. The error is intermittent and the Managed Servers for Oracle Business Intelligence are shown as "Running" in the Administration Console.
Solution: Even when the Managed Servers for Oracle Business Intelligence are up and running, some of the applications that are contained in them might be in Admin, Prepared, or other states different from Active. The applications might be unavailable while the Managed Server is running. Check the Deployments page in the Administration Console to verify the status of the affected application. It should be in "Active" state. Check the output log for the Managed Server for errors that pertain to that application and try to start it from the Deployments page in the Administration Console.
Problem: The Administration Server fails to start after the Administration Server node fails and manual failover to another nodes is performed. The Administration Server output log reports the following:
<Feb 19, 2009 3:43:05 AM PST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory: ORACLE_BASE/admin/edg_domain/aserver/edg_domain/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>
Solution: When restoring a node after a node crash and using shared storage for the domain directory, you might see this error in the log for the Administration Server due to unsuccessful lock cleanup. To resolve this error, remove the file ORACLE_BASE/admin/domain_name/aserver/domain_name/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok.
Problem: Activation of changes in Administration Console fails after changes to a server's start configuration have been performed. The Administration Console reports the following when clicking "Activate Changes":
An error occurred during activation of changes, please see the log for details. [Management:141190]The commit phase of the configuration update failed with an exception: In production mode, it's not allowed to set a clear text value to the property: PasswordEncrypted of ServerStartMBean
Solution: This might happen when start parameters are changed for a server in the Administration Console. In this case, provide user name and password information in the server start configuration in the Administration Console for the specific server whose configuration was being changed.
Problem: After reaching the maximum restart attempts by local Node Manager, Node Manager in the failover node tries to restart it, but the server does not start. The server seems to be failed over as reported by Node Manager's output information. The VIP used by the bi_server Managed Server is not enabled in the failover node after Node Manager tries to migrate it (if config in the failover node does not report the VIP in any interface). Executing the command "sudo ifconfig $INTERFACE $ADDRESS $NETMASK" does not enable the IP in the failover node.
Solution: The rights and configuration for sudo execution do not prompt for a password. Verify the configuration of sudo with the system administrator so that sudo works without a password prompt.
Problem: Server migration is working (bi_server Managed Server is restarted in the failed over node), but the Virtual_Hostname:9704/analytics URL cannot be accessed in the web browser. The server has been "killed" on its original host, and Node Manager in the failover node reports that the VIP has been migrated and the server started. The VIP that is used by the Managed Server for bi_server cannot be pinged from the client's node (that is, the node where the browser is being used).
Solution: The arping command that is executed by Node Manager to update ARP caches did not broadcast the update properly. In this case, the node is not reachable to external nodes. Either update the nodemanager.properties file to include the MACBroadcast or execute a manual arping:
/sbin/arping -b -q -c 3 -A -I INTERFACE ADDRESS > $NullDevice 2>&1
where INTERFACE is the network interface where the virtual IP is enabled and ADDRESS is the virtual IP address.
Problem: The OAM Configuration Tool has been used and a set of URLs was added to the policies in Oracle Access Manager. One of multiple URLs had a typo. Executing the OAM Configuration Tool again with the correct URLs completes successfully; however, when accessing Policy Manager, the incorrect URL is still there.
Solution: The OAM Configuration Tool adds new URLs to existing policies only when executed with the same app_domain name. To remove a URL, use the Policy Manager Console in OAM. Log on to the Access Administration site for OAM, click My Policy Domains, and click the created policy domain (bifoundation_domain). Click the Resources tab, and then remove the incorrect URLs.
Problem: After configuring Oracle HTTP Server and LBR to access the Administration Console, some activation changes cause the redirection to the login screen for the Administration Console.
Solution: This is the result of the console attempting to follow changes to port, channel, and security settings as a user makes these changes. For certain changes, the console might redirect to the Administration Server's listen address. Activation is completed regardless of the redirection. It is not required to log in again; users can simply update the URL to admin.mycompany.com/console/console.portal and directly access the home page for the Administration Console.
Note:
This problem does not occur if you have disabled tracking of the changes that are described in this section.
Problem: After configuring OAM, some activation changes cause redirection to the Administration Console home page (instead of the context menu where the activation was performed).
Solution: This is expected when OAM SSO is configured and the Administration Console is set to follow configuration changes (redirections are performed by the Administration Server when activating some changes). Activations complete regardless of this redirection. For successive changes not to redirect, access the Administration Console, choose Preferences, then Shared Preferences, and deselect the "Follow Configuration Changes" check box.
Problem: Attempts to start a Managed Server that uses the Java Object Cache, such as OWSM Managed Servers, fail. The following errors are shown in the logs:
J2EE JOC-058 distributed cache initialization failure J2EE JOC-043 base exception: J2EE JOC-803 unexpected EOF during read.
Solution: Another process is using the same port that JOC is attempting to obtain. Either stop that process, or reconfigure JOC for this cluster to use another port in the recommended port range.
Problem: You are experiencing out-of-memory issues on Managed Servers.
Solution: Increase the size of the memory heap that is allocated for the Java Virtual Machine to at least one gigabyte:
Log in to the Administration Console.
Click Environment, then Servers.
Click a Managed Server name.
Open the Configuration tab.
Open the Server Start tab in the second row of tabs.
Include the memory parameters in the Arguments box. For example:
-Xms256m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=1024m
Note:
The memory parameter requirements might differ between various JVMs (Sun, JRockit, or others).
Save the configuration changes.
Restart all running Managed Servers.
In some cases, only one JMS instance is visible on the BI Publisher Scheduler diagnostics page, rather than all instances in the cluster. This issue is most likely caused by clocks being out of sync. See Section 2.5, "Clock Synchronization" for more information on the importance of synchronizing clocks on all nodes in the cluster.
Before stopping the Managed Server on which BI Publisher is running, it is a best practice (but not mandatory) to wait for all running BI Publisher jobs to complete, or to cancel any unfinished jobs using the Report Job History page. Otherwise, the shutdown might cause some jobs to incorrectly stay in a running state.
On rare occasions, a JMS instance is missing from a BI Publisher Scheduler cluster. To resolve this issue, restart the BI Publisher application from the Oracle WebLogic Server Administration Console.
Perform the following steps to restart the BI Publisher application:
Log in to the Administration Console.
Click Deployments in the Domain Structure window.
Select bipublisher(11.1.1).
Click Stop.
After the application stops, click Start.
If the WebLogic domain administration account uses a different group name then Administrators (the default), you must update the MapViewer weblogic.xml file on all nodes to include the actual group name.
Perform the following steps to update the MapViewer weblogic.xml file with the custom group name:
Open the MapViewer weblogic.xml file for editing in the following directory:
ORACLE_HOME/bifoundation/jee/mapviewer.ear/web.war/WEB-INF
In the weblogic.xml file, locate the following lines:
<security-role-assignment> <role-name>map_admin_role</role-name> <principal-name>Administrators</principal-name> </security-role-assignment> <security-role-assignment> <role-name>secure_maps_role</role-name> <principal-name>Administrators</principal-name> </security-role-assignment>
Replace the two occurrences of Administrators with the actual administration group name (for example, BIAdministrators). For example:
<security-role-assignment> <role-name>map_admin_role</role-name> <principal-name>BIAdministrators</principal-name> </security-role-assignment> <security-role-assignment> <role-name>secure_maps_role</role-name> <principal-name>BIAdministrators</principal-name> </security-role-assignment>
Save and close the file.
Restart the MapViewer application, as follows:
Log in to the Administration Console.
Click Deployments in the Domain Structure window.
Select mapviewer(11.1.1).
Click Stop.
After the application has stopped, click Start.
Repeat these steps on all nodes.