10 Managing Enterprise Deployments

This chapter provides information about operations that you can perform after you have set up your topology, including monitoring, scaling, and backing up your enterprise deployment.

This chapter contains the following topics:

10.1 Starting and Stopping Oracle Business Intelligence

To start Oracle Business Intelligence, you must always start the Managed Servers first, before the system components. In addition, any time the Managed Servers are restarted, the system components must be restarted also.

For additional information, see "Starting and Stopping Oracle Business Intelligence" in Oracle Fusion Middleware System Administrator's Guide for Oracle Business Intelligence Enterprise Edition.

This section contains the following topics:

10.1.1 Starting and Stopping Oracle Business Intelligence Managed Servers

Follow these steps to stop, start, or restart Oracle Business Intelligence Managed Servers:

  1. Log in to the Oracle WebLogic Server Administration Console.

  2. Expand the Environment node in the Domain Structure window.

  3. Click Servers. The Summary of Servers page is displayed.

  4. Select the Oracle Business Intelligence Managed Server you want to manage (for example, bi_server1, bi_server2, and so on).

  5. Perform one of the following actions:

    • To stop the Managed Server, click Stop.

    • To start the Managed Server, click Start.

    • To restart the Managed Server, first click Stop and wait until the server is completely stopped. Then, select the Managed Server again and click Start.

10.1.2 Starting and Stopping Oracle Business Intelligence System Components

Follow these steps to stop, start, or restart Oracle Business Intelligence system components:

  1. Log in to Oracle Enterprise Manager Fusion Middleware Control.

  2. Expand the Business Intelligence node in the Farm_domain_name window.

  3. Click coreapplication.

  4. On the Business Intelligence Overview page, click Stop, Start, or Restart.

10.2 Monitoring Enterprise Deployments

For information on monitoring the Oracle Business Intelligence topology, see "Monitoring Service Levels" in Oracle Fusion Middleware System Administrator's Guide for Oracle Business Intelligence Enterprise Edition.

See also "Diagnosing and Resolving Issues in Oracle Business Intelligence" in Oracle Fusion Middleware System Administrator's Guide for Oracle Business Intelligence Enterprise Edition for information about Oracle Business Intelligence log files, including rotating and managing logs.

10.3 Scaling Enterprise Deployments

You can scale up or scale out the Oracle Business Intelligence enterprise topology, as follows:

  • When you scale up the topology, you add additional system components to one of the existing nodes in your enterprise topology.

  • When you scale out the topology, you add a new node to your topology with a Managed Server and set of system components.

This section includes the following topics:

Note:

To scale out and up the SOA subsystem used by I/PM, refer to the SOA enterprise deployment topology documentation.

10.3.1 Scaling Up the Oracle Business Intelligence Topology

This procedure assumes that you already have an enterprise topology that includes two nodes, with a Managed Server and a full set of system components on each node. To scale up the topology, you increase the number of system components running on one of your existing nodes.

Note that it is not necessary to run multiple Managed Servers on a given node.

To scale up the Oracle Business Intelligence enterprise topology:

  1. Log in to Fusion Middleware Control.

  2. Expand the Business Intelligence node in the Farm_domain_name window.

  3. Click coreapplication.

  4. Click Capacity Management, then click Scalability.

  5. Click Lock and Edit Configuration.

  6. Change the number of BI Servers, Presentation Servers, or JavaHosts using the arrow keys.

  7. Click Apply, then click Activate Changes.

  8. Click Overview, then click Restart.

10.3.2 Scaling Out the Oracle Business Intelligence Topology

When scaling out the topology, you add a new Managed Server and set of system components to a new node in your topology (APPHOST3). This procedure assumes that you already have an enterprise topology that includes two nodes, with a Managed Server and a full set of system components on each node.

Prerequisites

Before performing the steps in this section, check that you meet these requirements:

  • There must be existing nodes running Oracle Business Intelligence Managed Servers within the topology.

  • The new node (APPHOST3) can access the existing home directories for Oracle WebLogic Server and Oracle Business Intelligence.

  • When an ORACLE_HOME or WL_HOME is shared by multiple servers in different nodes, it is recommended that you keep the Oracle Inventory and Middleware home list in those nodes updated for consistency in the installations and application of patches. To update the oraInventory in a node and "attach" an installation in a shared storage to it, use ORACLE_HOME/oui/bin/attachHome.sh. To update the Middleware home list to add or remove a WL_HOME, edit the User_Home/bea/beahomelist file. See the steps below.

  • You must ensure that all shared storage directories are available on the new node. Ensure that all shared directories listed in Section 2.3.2, "Recommended Locations for the Different Directories" are available on all nodes, except for the ORACLE_INSTANCE directory and the domain directory for the scaled out Managed Server.

  • If you are using shared storage for the identity keystore and trust keystore that hold your host name verification certificates, make sure that the shared storage directory is accessible from the scaled-out node (APPHOST3). If you are using local directories for your keystores, follow the steps in Section 7.3, "Enabling Host Name Verification Certificates for Node Manager" to create and configure a local identity keystore for the scaled-out node.

10.3.2.1 Scale-out Procedure for Oracle Business Intelligence

Perform these steps to scale out Oracle Business Intelligence on APPHOST3:

  1. On APPHOST3, mount the existing Middleware home, which should include the Oracle Business Intelligence installation and (optionally, if the domain directory for Managed Servers in other nodes resides on shared storage) the domain directory, and ensure that the new node has access to this directory, just like the rest of the nodes in the domain.

  2. To attach ORACLE_HOME in shared storage to the local Oracle Inventory, execute the following command:

    APPHOST3> cd ORACLE_COMMON_HOME/oui/bin/
APPHOST3> ./attachHome.sh -jreLoc ORACLE_BASE/product/fmw/jrockit_160_<version>

    To update the Middleware home list, create (or edit, if another WebLogic installation exists in the node) the MW_HOME/bea/beahomelist file and add ORACLE_BASE/product/fmw to it.

  3. Run the Configuration Assistant from one of the shared Oracle homes, using the steps in Section 6.1, "Scaling Out the BI System on APPHOST2" as a guide.

  4. Scale out the system components on APPHOST3, using the steps in Section 6.2, "Scaling Out the System Components" as a guide.

  5. Configure the bi_server3 Managed Server by setting the Listen Address and disabling host name verification, using the steps in Section 6.4, "Configuring the bi_server2 Managed Server" as a guide.

  6. Perform additional Oracle Real-Time Decisions configuration steps for the bi_server3 Managed Server on APPHOST3, using the steps in Section 6.5.2, "Additional Configuration Tasks for Oracle Real-Time Decisions" as a guide.

  7. Configure JMS for Oracle BI Publisher, as described in Section 6.5.3.5, "Configuring JMS for Oracle BI Publisher."

  8. Configure Oracle BI Office on APPHOST3, as described in Section 6.5.4, "Additional Configuration Tasks for Oracle BI Office."

  9. Set the location of the default persistence store for bi_server 3, as described in Section 6.6, "Configuring a Default Persistence Store for Transaction Recovery."

  10. Configure Oracle HTTP Server for APPHOST3VHN1, using the steps in Section 5.9.2, "Configuring Oracle HTTP Server for the bi_servern Managed Servers" as a guide.

  11. Start the bi_server3 Managed Server and the system components running on APPHOST3. See Section 10.1, "Starting and Stopping Oracle Business Intelligence" for details.

  12. Set up server migration for the new node, as described in the following sections:

  13. To validate the configuration, access the following URLs:

    • Access http://APPHOST3VHN1:9704/analytics to verify the status of bi_server3.

    • Access http://APPHOST3VHN1:9704/wsm-pm to verify the status of Web Services Manager. Click Validate Policy Manager. A list of policies and assertion templates available in the data is displayed.

      Note: The configuration is incorrect if no policies or assertion templates appear.

    • Access http://APPHOST3VHN1:9704/xmlpserver to verify the status of the Oracle BI Publisher application.

    • Access http://APPHOST3VHN1:9704/ui to verify the status of the Oracle Real-Time Decisions application.

  14. Oracle recommends using host name verification for the communication between Node Manager and the servers in the domain. This requires the use of certificates for the different addresses communicating with the Administration Server and other servers. See Chapter 7, "Setting Up Node Manager" for further details.

10.4 Performing Backups and Recoveries

See "Backup and Recovery Recommendations for Oracle Business Intelligence" in Oracle Fusion Middleware Administrator's Guide for full information about backing up and recovering Oracle Business Intelligence.

10.5 Patching Enterprise Deployments

See "Patching Oracle Business Intelligence Systems" in Oracle Fusion Middleware System Administrator's Guide for Oracle Business Intelligence Enterprise Edition for more information about Oracle Business Intelligence patching.

10.6 Troubleshooting

This section covers the following topics:

10.6.1 Page Not Found When Accessing BI Applications Through Load Balancer

Problem: A 404 "page not found" message is displayed in the Web browser when you try to access Oracle Business Intelligence applications (such as Oracle BI Presentation Services, Oracle BI Publisher, and Oracle Real-Time Decisions) using the load balancer address. The error is intermittent and BI servers appear as "Running" in the Administration Console.

Solution: Even when the BI Managed Servers are up and running, some of the applications contained in them may be in Admin, Prepared, or other states different from Active. The applications might be unavailable while the BI server is running. Check the Deployments page in the Administration Console to verify the status of the affected application. It should be in "Active" state. Check the BI server's output log for errors pertaining to that application and try to start it from the Deployments page in the Administration Console.

10.6.2 Administration Server Fails to Start After a Manual Failover

Problem: Administration Server fails to start after the Administration Server node failed and manual failover to another nodes is performed. The Administration Server output log reports the following:

<Feb 19, 2009 3:43:05 AM PST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory: ORACLE_BASE/admin/edg_domain/aserver/edg_domain/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>

Solution: When restoring a node after a node crash and using shared storage for the domain directory, you may see this error in the log for the Administration Server due to unsuccessful lock cleanup. To resolve this error, remove the file ORACLE_BASE/admin/domain_name/aserver/domain_name/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok.

10.6.3 Error While Activating Changes in Administration Console

Problem: Activation of changes in Administration Console fails after changes to a server's start configuration have been performed. The Administration Console reports the following when clicking "Activate Changes":

An error occurred during activation of changes, please see the log for details.
 [Management:141190]The commit phase of the configuration update failed with an exception:
In production mode, it's not allowed to set a clear text value to the property: PasswordEncrypted of ServerStartMBean

Solution: This may happen when start parameters are changed for a server in the Administration Console. In this case, provide username/password information in the server start configuration in the Administration Console for the specific server whose configuration was being changed.

10.6.4 bi_server Managed Server Not Failed Over After Server Migration

Problem: After reaching the maximum restart attempts by local Node Manager, Node Manager in the failover node tries to restart it, but the server does not come up. The server seems to be failed over as reported by Node Manager's output information. The VIP used by the bi_server Managed Server is not enabled in the failover node after Node Manager tries to migrate it (if config in the failover node does not report the VIP in any interface). Executing the command "sudo ifconfig $INTERFACE $ADDRESS $NETMASK" does not enable the IP in the failover node.

Solution: The rights and configuration for sudo execution should not prompt for a password. Verify the configuration of sudo with your system administrator so that sudo works without a password prompt.

10.6.5 bi_server Managed Server Not Reachable From Browser After Server Migration

Problem: Server migration is working (bi_server Managed Server is restarted in the failed over node), but the Virtual_Hostname:9704/analytics URL cannot be accessed in the Web browser. The server has been "killed" in its original host and Node Manager in the failover node reports that the VIP has been migrated and the server started. The VIP used by the bi_server Managed Server cannot be pinged from the client's node (that is, the node where the browser is being used).

Solution: The arping command executed by Node Manager to update ARP caches did not broadcast the update properly. In this case, the node is not reachable to external nodes. Either update the nodemanager.properties file to include the MACBroadcast or execute a manual arping:

/sbin/arping -b -q -c 3 -A -I INTERFACE ADDRESS > $NullDevice 2>&1

Where INTERFACE is the network interface where the virtual IP is enabled and ADDRESS is the virtual IP address.

10.6.6 OAM Configuration Tool Does Not Remove URLs

Problem: The OAM Configuration Tool has been used and a set of URLs was added to the policies in Oracle Access Manager. One of multiple URLs had a typo. Executing the OAM Configuration Tool again with the correct URLs completes successfully; however, when accessing Policy Manager, the incorrect URL is still there.

Solution: The OAM Configuration Tool only adds new URLs to existing policies when executed with the same app_domain name. To remove a URL, use the Policy Manager Console in OAM. Log on to the Access Administration site for OAM, click My Policy Domains, and then click the created policy domain (bifoundation_domain). Click the Resources tab, and then remove the incorrect URLs.

10.6.7 Users Redirected to Login Screen After Activating Changes

Problem: After configuring Oracle HTTP Server and LBR to access the Administration Console, some activation changes cause the redirection to the login screen for the Administration Console.

Solution: This is the result of the console attempting to follow changes to port, channel, and security settings as a user makes these changes. For certain changes, the console may redirect to the Administration Server's listen address. Activation is completed regardless of the redirection. It is not required to log in again; users can simply update the URL to admin.mycompany.com/console/console.portal and directly access the home page for the Administration Console.

Note:

This problem will not occur if you have disabled tracking of the changes described in this section.

10.6.8 Users Redirected to Home Page After Activating Changes

Problem: After configuring OAM, some activation changes cause redirection to the Administration Console home page (instead of the context menu where the activation was performed).

Solution: This is expected when OAM SSO is configured and the Administration Console is set to follow configuration changes (redirections are performed by the Administration Server when activating some changes). Activations should complete regardless of this redirection. For successive changes not to redirect, access the Administration Console, choose Preferences, then Shared Preferences, and deselect the "Follow Configuration Changes" check box.

10.6.9 Configured JOC Port Already in Use

Problem: Attempts to start a Managed Server that uses the Java Object Cache, such as OWSM Managed Servers, fail. The following errors appear in the logs:

J2EE JOC-058 distributed cache initialization failure
J2EE JOC-043 base exception:
J2EE JOC-803 unexpected EOF during read.

Solution: Another process is using the same port that JOC is attempting to obtain. Either stop that process, or reconfigure JOC for this cluster to use another port in the recommended port range.

10.6.10 Out-of-Memory Issues on Managed Servers

Problem: You are experiencing out-of-memory issues on Managed Servers.

Solution: Increase the size of the memory heap allocated for the Java VM to at least one gigabyte:

  1. Log in to the Administration Console.

  2. Click Environment, then Servers.

  3. Click a Managed Server name.

  4. Open the Configuration tab.

  5. Open the Server Start tab in the second row of tabs.

  6. Include the memory parameters in the Arguments box, for example:

    -Xms256m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=1024m

    Note:

    The memory parameter requirements may differ between various JVMs (Sun, JRockit, or others).

  7. Save the configuration changes.

  8. Restart all running Managed Servers.

10.7 Other Recommendations

This section covers the following topics:

10.7.1 Preventing Timeouts for SQLNet Connections

Much of the EDG production deployment involves firewalls. Because database connections are made across firewalls, Oracle recommends that the firewall be configured so that the database connection is not timed out. For Oracle Real Application Clusters (Oracle RAC), the database connections are made on Oracle RAC VIPs and the database listener port. You must configure the firewall to not time out such connections. If such a configuration is not possible, set the*SQLNET.EXPIRE_TIME=n* parameter in the ORACLE_HOME/network/admin/sqlnet.ora file on the database server, where n is the time in minutes. Set this value to less than the known value of the timeout for the network device (that is, a firewall). For Oracle RAC, set this parameter in all of the Oracle home directories.

10.7.2 Auditing

Oracle Fusion Middleware Audit Framework is a new service in Oracle Fusion Middleware 11g, designed to provide a centralized audit framework for the middleware family of products. The framework provides audit service for platform components such as Oracle Platform Security Services (OPSS) and Oracle Web Services. It also provides a framework for JavaEE applications, starting with Oracle's own JavaEE components. JavaEE applications will be able to create application-specific audit events. For non-JavaEE Oracle components in the middleware, such as C or JavaSE components, the audit framework also provides an end-to-end structure similar to that for JavaEE applications.

Figure 10-1 is a high-level architectural diagram of the Oracle Fusion Middleware Audit Framework.

Figure 10-1 Audit Event Flow

Description of Figure 10-1 follows
Description of "Figure 10-1 Audit Event Flow"

The Oracle Fusion Middleware Audit Framework consists of the following key components:

  • Audit APIs: These are APIs provided by the audit framework for any audit-aware components integrating with the Oracle Fusion Middleware Audit Framework. During run time, applications may call these APIs, where appropriate, to audit the necessary information about a particular event happening in the application code. The interface allows applications to specify event details such as username and other attributes needed to provide the context of the event being audited.

  • Audit Events and Configuration: The Oracle Fusion Middleware Audit Framework provides a set of generic events for convenient mapping to application audit events. Some of these include common events such as authentication. The framework also allows applications to define application-specific events.

    These event definitions and configurations are implemented as part of the audit service in Oracle Platform Security Services. Configurations can be updated through Enterprise Manager (UI) and WLST (command-line tool).

  • Audit Bus-stop: Bus-stops are local files containing audit data before they are pushed to the audit repository. In the event where no database repository is configured, these bus-stop files can be used as a file-based audit repository. The bus-stop files are simple text files that can be queried easily to look up specific audit events. When a DB-based repository is in place, the bus-stop acts as an intermediary between the component and the audit repository. The local files are periodically uploaded to the audit repository based on a configurable time interval.

  • Audit Loader: As the name implies, the audit loader loads the files from the audit bus-stop into the audit repository. In the case of platform and JavaEE application audit, the audit loader is started as part of the JavaEE container start-up. In the case of system components, the audit loader is a periodically spawned process.

  • Audit Repository: The audit repository contains a predefined Oracle Fusion Middleware Audit Framework schema, created by Repository Creation Utility (RCU). Once configured, all the audit loaders are aware of the repository and upload data to it periodically. The audit data in the audit repository is expected to be cumulative and will grow over time. Ideally, this should not be an operational database used by any other applications; rather, it should be a standalone RDBMS used for audit purposes only. In a highly available configuration, Oracle recommends that you use an Oracle Real Application Clusters (Oracle RAC) database as the audit data store.

  • Oracle Business Intelligence Publisher: The data in the audit repository is exposed through predefined reports in Oracle Business Intelligence Publisher. The reports allow users to drill down the audit data based on various criteria. For example:

    • Username

    • Time range

    • Application type

    • Execution context identifier (ECID)

For more introductory information for the Oracle Fusion Middleware Audit Framework, see the "Introduction to Oracle Fusion Middleware Audit Framework" chapter in the Oracle Fusion Middleware Security Guide.

For information on how to configure the repository for Oracle Fusion Middleware Audit Framework, see the "Configuring and Managing Auditing" chapter in the Oracle Fusion Middleware Security Guide.

The EDG topology does not include Oracle Fusion Middleware Audit Framework configuration. The ability to generate audit data to the bus-stop files and the configuration of the audit loader will be available after the products are installed. The main consideration is the audit database repository where the audit data is stored. Because of the volume and the historical nature of the audit data, it is strongly recommended that customers use a separate database from the operational store or stores being used for other middleware components.