11 Managing the Topology

This chapter describes some operations that you can perform after you have set up the topology. These operations include monitoring, scaling, and backing up your topology.

This chapter contains the following sections:

Section 11.1, "Monitoring the Topology"
Section 11.2, "Configuring UMS Drivers"
Section 11.3, "Scaling the Topology"
Section 11.4, "Performing Backups and Recoveries"
Section 11.5, "Troubleshooting"
Section 11.6, "Best Practices"

11.1 Monitoring the Topology

For information on monitoring the topology, see chapter 15 of the Oracle Fusion Middleware Administrator's Guide for Oracle WebCenter.

11.2 Configuring UMS Drivers

UMS driver configuration is not automatically propagated in a SOA cluster. This implies that users need to:

Apply the configuration of UMS drivers in each and every one of the servers in the EDG topology that is using the driver.
When server migration is used, servers are moved to a different node's domain directory. It is necessary to pre-create the UMS driver configuration in the failover node. The UMS driver configuration file location is:
```
ORACLE_BASE/admin/<domain_name>/mserver/<domain_name>/servers/<server_name>/tmp/_WL_user/<ums_driver_name>/*/configuration/driverconfig.xml
```
(where '*' represents a directory whose name is randomly generated by WLS during deployment, for example, "3682yq").

In order to create the file in preparation for possible failovers, users can force a server migration and copy the file from the source node.

It is required to restart the driver for these changes to take effect (that is, for the driver to consume the modified configuration). To restart the driver:

Log on to the Oracle WebLogic Administration console.
Expand the environment node on the navigation tree.
Click on Deployments.
Select the driver.
Click Stop->When work completes and confirm the operation.
Wait for the driver to transition to the "Prepared" state (refresh the administration console page, if required).
Select the driver again, and click Start->Servicing all requests and confirm the operation.

Make sure that you verify in Oracle Enterprise Manager Fusion Middleware Control that the properties for the driver have been preserved.

11.3 Scaling the Topology

You can scale out and or scale up the enterprise topology. When you scale up the topology, you add new managed servers to nodes that are already running on one or more managed servers. When you scale out the topology, you add new managed servers to new nodes.

This section covers includes the topics:

Section 11.3.1, "Scaling Up the Topology (Adding Managed Servers to Existing Nodes)"
Section 11.3.2, "Scaling Out the Topology (Adding Managed Servers to New Nodes)"
Section 11.3.3, "Scaling Out the Oracle Content Server (OCS) Topology"

11.3.1 Scaling Up the Topology (Adding Managed Servers to Existing Nodes)

In this case, you already have a node that runs a managed server configured with WebCenter components or a managed server with WSM-PM. The node contains a WebLogic Server home and an Oracle Fusion Middleware WebCenter home in shared storage.

You can use the existing installations (WebLogic Server home, Oracle Fusion Middleware home, and domain directories) for creating new Oracle WebCenter and WLS_WSM servers. There is no need to install WLS or WebCenter binaries in a new location, or to run pack and unpack.

Follow these steps for scaling up the topology:

Using the Administration Console, clone WLS_Spaces1 or WLS_Portlet1 or WLS_Services1 or WLS_WSM1 into a new managed server. The source managed server to clone should be one that already exists on the node where you want to run the new managed server.

To clone a managed server, complete these steps:
1. Select Environment and then Servers from the Administration Console.
2. Select the managed server to clone (for example, WLS_Spaces1 or WLS_Portlet1).
3. Select Clone.
Name the new managed server (SERVER_NAME)n, where n is a number to identify the new managed server.
For the listen address, assign the host name or IP to use for this new managed server.
If the Managed Server being scaled out is WLS_Services, then follow these extra steps:
1. Start the Managed Server at least once, to create the Managed Servers deployment directories for Oracle Wiki. The Managed Server can then be shut down.
2. Create a soft link from this new servers config directories to that of the shared Wiki Server directories:
```
$ ln -s <DOMAIN_HOME>/servers/WLS_ServicesN/stage/owc_wiki/11.1.1.1.0/owc_wiki/attachments/ /shared/owc_wiki/attachments
$ ln -s <DOMAIN_HOME>/servers/WLS_ServicesN/stage/owc_wiki/11.1.1.1.0/owc_wiki/templates/ /shared/owc_wiki/templates
```
Reconfigure the Oracle HTTP Server module with the new member in the cluster. See Oracle HTTP Server configuration.
Add the New Managed Server to the Java Object Cache Cluster (see Section 6.13, "Setting Up the Java Object Cache").

11.3.2 Scaling Out the Topology (Adding Managed Servers to New Nodes)

In scaling out your topology, you add new managed servers configured with Oracle WebCenter applications and or WSM-PM to new nodes.

Before performing the steps in this section, check that you meet these requirements:

In your topology, there are existing nodes running managed servers configured with Oracle WebCenter applications and or WSM-PM.
The new node can access the existing home directories for WebLogic Server and Oracle WebCenter. You use the existing installations in shared storage for creating a new managed server. You do not need to install WebLogic Server or WebCenter binaries in a new location but you do need to run pack and unpack to bootstrap the domain configuration in the new node.

Follow these steps for scaling out the topology:

On the new node, mount the existing FMW Home, which should include the WebCenter installation and the domain directory, and ensure that the new node has access to this directory, just like the rest of the nodes in the domain.
To attach ORACLE_HOME in shared storage to the local Oracle Inventory, execute the following command:
```
SOAHOSTn>cd ORACLE_BASE/product/fmw/soa/
SOAHOSTn>./attachHome.sh -jreLoc ORACLE_BASE/fmw/jrockit_160_05_R27.6.2-20
```
To update the Middleware home list, create (or edit, if another WebLogic installation exists in the node) the $HOME/bea/beahomelist file and add ORACLE_BASE/product/fmw to it.
Log in to the Oracle WebLogic Administration Console.
Create a new machine for the new node that will be used, and add the machine to the domain.
Update the machine's Node Manager's address to map the IP of the node that is being used for scale out.
Using the Administration Console, clone WLS_Spaces1/WLS_Portlet1/WLS_Services1/WLS_WSM1 into a new managed server and name it WLS_(SERVER_TYPE)n, where n is a number.

The steps assume that you are adding a new server to node n, where no managed server was running.
For the listen address of the managed server, assign the host name or IP to use for the new managed server.
If the Managed Server being scaled out is WLS_Services then follow these extra steps:
1. Start the Managed Server at least once, to create the Managed Servers deployment directories for Oracle Wiki. The Managed Server can then be shut down.
2. Create a soft link from this new servers config directories to that of the shared Wiki Server directories:
```
$ ln -s <DOMAIN_HOME>/servers/WLS_ServicesN/stage/owc_wiki/11.1.1.1.0/owc_wiki/attachments/ /shared/owc_wiki/attachments
$ ln -s <DOMAIN_HOME>/servers/WLS_ServicesN/stage/owc_wiki/11.1.1.1.0/owc_wiki/templates/ /shared/owc_wiki/templates
```
Reconfigure the Oracle HTTP Server module with the new member in the cluster (see Oracle HTTP Server configuration).
Start Node Manager on the new node: using the installation in shared storage from the already existing nodes, start Node Manager passing as parameter the host name of the new node:
```
NEW_NODE> WL_HOME/server/bin/startNodeManager <new_node_ip>
```
If you used the paths shown in Section 4.1.1, "Installing Oracle WebLogic Server," WL_HOME would be ORACLE_BASE/wls.
Start and test the just added managed server from the Administration Console.

Access the application on the newly created managed server (http://HOST:port/webcenter or http://host:port/wsm-pm). The application should be functional.
Add the new managed server to the Java Object Cache Cluster (see Section 6.13, "Setting Up the Java Object Cache").

11.3.3 Scaling Out the Oracle Content Server (OCS) Topology

Each Content Server node is installed using the cluster installer which can be found in <content_server_dir>/bin on the shared disk.

Installing Another Content Server Node

From the new node (SOAHOSTn), run the command (all on one line):

Installer
-set-ClusterNodeIntradocDir=<ocs-stub>
-set-ClusterNodeName=<nodename>
-set-ClusterNodeAddress=<ip_of_node>
-set-ClusterBinDirRule=<local|shared>
ConfigureClusterNode
ConfigureAdminClusterNode

For <nodename>, specify SOAHOSTn.

For <stellent-stub>, use a local directory, for example, /u01/app/oracle/product/ucm.

For Set-ClusterBinDirRule, specify local.

Post-Installation Steps

After setting up the nodes, add the following lines to the intradoc.cfg file in the <stellent-stub>/bin and <stellent-stub>/admin/etc directories:

DisableSharedCacheChecking=true 
ClusterNodeName=<nodename> 
ClusterGroup=<cluster_name> 
SocketServerAddress=<server_IP_address>

<nodename> is the identifier for the node.

<cluster_name> is the identifier for the cluster.

All nodes should have a different node name, but should have the same cluster group name; for example, the lines on SOAHOSTn might look like the following:

DisableSharedCacheChecking=true
ClusterNodeName=SOAHOSTn
ClusterGroup=ucmcluster
SocketServerAddress=<IP_of_SOAHOSTn>

Starting the Servers

Start Oracle Content Server and Admin Server on each node using the binaries in <ocs-stub>/bin.

Verify that the log and PID files are created in the <ocs-stub>/admin/etc and <ocs-stub>/etc directories.

Configuring the Load Balancer

You must reconfigure the load balancer to act as a socket load balancer to this new host as well. This will be used to make socket connections from the Oracle HTTP Server as well as WebCenter. Example:

Virtual host on load balancer: wcinternal.mycompany.com:9054

Maps to: WCHOST1:9054, WCHOST2:9054, SOAHOSTn:9054

Load-balancing method: round-robin

11.4 Performing Backups and Recoveries

Table 11-1 lists the static artifacts to back up in the 11g Oracle WebCenter enterprise deployment.

Table 11-1 Static Artifacts to Back Up in the 11g Oracle WebCenter Enterprise Deployment

Type	Host	Location	Tier
ORACLE HOME (DB)	RAC Database hosts - CUSTDBHOST1 and CUSTDBHOST2	The location is user-defined	Directory Tier
MW HOME (SOA + WC)	SOAHOST1 and SOAHOST2 - SOA WCHOST1 and WCHOST2 - WC	ORACLE_BASE/product/fmw on all hosts	Application Tier
ORACLE HOME (OHS)	WEBHOST1 and WEBHOST2	ORACLE_BASE/admin/<instance_name>	Web Tier
ORACLE HOME (OCS)	WCHOST1 and WCHOST2	On shared disk: /share/oracle/ucm On each host, local files at ORACLE_HOME/ucm	Application Tier
Installation- related files		OraInventory, <user_home>/bea/beahomelist, oraInst.loc, oratab

Table 11-2 lists the runtime artifacts for back up in the 11g Oracle WebCenter enterprise deployment.

Table 11-2 Run-Time Artifacts to Back Up in the 11g Oracle WebCenter Enterprise Deployment

Type	Host	Location	Tier
DOMAIN HOME	SOAHOST1 SOAHOST2 WCHOST1 WCHOST2	ORACLE_BASE/admin/<domain_name>/mserver/<domain_name>	Application Tier
Application artifacts (ear and war files)	SOAHOST1 SOAHOST2 WCHOST1 WCHOST2	Look at all the deployments through admin console and get all the application artifacts	Application Tier
OHS INSTANCE HOME	WEBHOST1 and WEBHOST2	On WEBHOST1, ORACLE_HOME/ohs_1/instances/instance1 On WEBHOST2, ORACLE_HOME/ohs_2/instances/instance2	Web Tier
OHS OCS configuration files	WEBHOST1 and WEBHOST2	On each host, at /share/oracle/ucm, which is a local file system.	Web Tier
RAC databases	CUSTDBHOST1 and CUSTDBHOST2	The location is user-defined	Directory Tier
OCS content repository		Database-based	Directory Tier

For more information on backup and recovery of Oracle Fusion Middleware components, see Oracle Fusion Middleware Administrator's Guide.

Note:

ORACLE_HOME should be backed up if any changes are made to the XEngine configuration that are part of your B2B setup. These files are located under ORACLE_HOME/soa/thirdparty/edifecs/XEngine. To back up ORACLE_HOME, execute the following command:

SOAHOST1> tar -cvpf fmwhomeback.tar ORACLE_BASE/product/fmw

11.5 Troubleshooting

This section covers the following topics:

Section 11.5.1, "Admin Server Fails to Start After a Manual Failover"
Section 11.5.2, "Error While Activating Changes in Administration Console"
Section 11.5.3, "OAM Configuration Tool Does Not Remove URLs"
Section 11.5.4, "Portlet Unavailable After Database Failover"
Section 11.5.5, "Redirecting of Users to Login Screen After Activating Changes in Administration Console"
Section 11.5.6, "Redirecting of Users to Administration Console's Home Page After Activating Changes to OAM"
11.5.1 Admin Server Fails to Start After a Manual Failover

Problem: Admin Server fails to start after the Admin Server node failed and manual failover to another nodes is performed. The Admin Server output log reports the following:
```
<Feb 19, 2009 3:43:05 AM PST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory: ORACLE_BASE/admin/soadomain/aserver/soadomain/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>
```
Solution: When restoring a node after a node crash and using shared storage for the domain directory, you may see this error in the log for the admin server due to unsuccessful lock cleanup. To resolve this error, remove the file ORACLE_BASE/admin/<domain_name>/aserver/<domain_name>/servers/AdminServer/data/ldap/ldapfiles/ EmbeddedLDAP.lok.
11.5.2 Error While Activating Changes in Administration Console

Problem: Activation of changes in Administration Console fails after changes to a server's start configuration have been performed. The Administration Console reports the following when clicking "Activate Changes":
```
An error occurred during activation of changes, please see the log for details.
 [Management:141190]The commit phase of the configuration update failed with an exception:
In production mode, it's not allowed to set a clear text value to the property: PasswordEncrypted of ServerStartMBean
```
Solution: This may happen when start parameters are changed for a server in the Administration Console. In this case, either provide username/password information in the server start configuration in the Administration Console for the specific server whose configuration was being changed, or remove the <password-encrypted></password-encrypted> entry in the config.xml file (this requires a restart of the admin server).
11.5.3 OAM Configuration Tool Does Not Remove URLs

Problem: The OAM Configuration Tool has been used and a set of URLs was added to the policies in Oracle Access Manager. One of multiple URLs had a typo. Executing the OAM Configuration Tool again with the correct URLs completes successfully; however, when accessing Policy Manager, the incorrect URL is still there.

Solution: The OAM Configuration Tool only adds new URLs to existing policies when executed with the same app_domain name. To remove a URL, use the Policy Manager Console in OAM. Log on to the Access Administration site for OAM, click on My Policy Domains, click on the created policy domain (SOA_EDG), then on the Resources tab, and remove the incorrect URLs.
11.5.4 Portlet Unavailable After Database Failover

Problem: While creating a page inside WebCenter Spaces, if you add a portlet to the page and a database failover occurs, an error component may added to the page with the following message showing on it:
```
"Error"
"Portlet unavailable"
```
This message remains even if you refresh the page or log out and back in again.

Solution: To resolve this issue, delete the component and add it again.
11.5.5 Redirecting of Users to Login Screen After Activating Changes in Administration Console

Problem: After configuring OHS and LBR to access the Oracle WebLogic Administration Console, some activation changes cause the redirection to the login screen for the admin console.

Solution: This is the result of the console attempting to follow changes to port, channel, and security settings as a user makes these changes. For certain changes, the console may redirect to the Admin Server's listen address. Activation is completed regardless of the redirection. It is not required to log in again; users can simply update the URL to soa.mycompany.com/console/console.portal and directly access the home page for the Administration Console.

11.5.6 Redirecting of Users to Administration Console's Home Page After Activating Changes to OAM

Problem: After configuring OAM, some activation changes cause the redirection to the Administration Console's home page (instead of the context menu where the activation was performed).

Solution: This is expected when OAM SSO is configured and is the result of the redirections performed by the Administration Server. Activation is completed regardless of the redirection. If required, users may "manually" navigate again to the desired context menu.

11.6 Best Practices

This section covers the following topics:

Section 11.6.1, "Preventing Timeouts for SQLNet Connections"
Section 11.6.2, "Setting AUTOEXTEND for Tablespaces for Processing More Than 30,000 Files"
Section 11.6.3, "Auditing"

11.6.1 Preventing Timeouts for SQLNet Connections

Much of the EDG production deployment involves firewalls. Because database connections are made across firewalls, Oracle recommends that the firewall be configured so that the database connection is not timed out. For Oracle Real Application Clusters (RAC), the database connections are made on Oracle RAC VIPs and the database listener port. You must configure the firewall to not time out such connections. If such a configuration is not possible, set the*SQLNET.EXPIRE_TIME=n* parameter in the ORACLE_HOME/network/admin/sqlnet.ora file on the database server, where n is the time in minutes. Set this value to less than the known value of the timeout for the network device (that is, a firewall). For RAC, set this parameter in all of the Oracle home directories.

11.6.2 Setting AUTOEXTEND for Tablespaces for Processing More Than 30,000 Files

After processing more than 30,000 files, tablespace does not extend. You must enable AUTOEXTEND for the tablespace when it reaches its limit. The best practice is to set AUTOEXTEND for tablespace before you commence a process to the recommended value. See also Oracle Database Administrator's Guide for more information on autoextending tablespaces.

11.6.3 Auditing

Oracle Fusion Middleware Audit Framework is a new service in Oracle Fusion Middleware 11g, designed to provide a centralized audit framework for the middleware family of products. The framework provides audit service for platform components such as Oracle Platform Security Services (OPSS) and Oracle Web Services. It also provides a framework for JavaEE applications, starting with Oracle's own JavaEE components. JavaEE applications will be able to create application-specific audit events. For non-JavaEE Oracle components in the middleware such as C or JavaSE components, the audit framework also provides an end-to-end structure similar to that for JavaEE applications.

Figure 11-1 is a high-level architectural diagram of the Oracle Fusion Middleware Audit Framework.

Figure 11-1 Audit Event Flow

Description of "Figure 11-1 Audit Event Flow"

The Oracle Fusion Middleware Audit Framework consists of the following key components:

Audit APIs

These are APIs provided by the audit framework for any audit-aware components integrating with the Oracle Fusion Middleware Audit Framework. During runtime, applications may call these APIs where appropriate to audit the necessary information about a particular event happening in the application code. The interface allows applications to specify event details such as username and other attributes needed to provide the context of the event being audited.
Audit Events and Configuration

The Oracle Fusion Middleware Audit Framework provides a set of generic events for convenient mapping to application audit events. Some of these include common events such as authentication. The framework also allows applications to define application-specific events.

These event definitions and configurations are implemented as part of the audit service in Oracle Platform Security Services. Configurations can be updated through Enterprise Manager (UI) and WLST (command-line tool).
Audit Bus-stop

Bus-stops are local files containing audit data before they are pushed to the audit repository. In the event where no database repository is configured, these bus-stop files can be used as a file-based audit repository. The bus-stop files are simple text files that can be queried easily to look up specific audit events. When a DB-based repository is in place, the bus-stop acts as an intermediary between the component and the audit repository. The local files are periodically uploaded to the audit repository based on a configurable time interval.
Audit Loader

As the name implies, audit loader loads the files from the audit bus-stop into the audit repository. In the case of platform and JavaEE application audit, the audit loader is started as part of the JavaEE container start-up. In the case of system components, the audit loader is a periodically spawned process.
Audit Repository

Audit Repository contains a pre-defined Oracle Fusion Middleware Audit Framework schema, created by Repository Creation Utility (RCU). Once configured, all the audit loaders are aware of the repository and upload data to it periodically. The audit data in the audit repository is expected to be cumulative and will grow overtime. Ideally, this should not be an operational database used by any other applications - rather, it should be a standalone RDBMS used for audit purposes only. In a highly available configuration, Oracle recommends that you use an Oracle Real Application Clusters (RAC) database as the audit data store.
Oracle Business Intelligence Publisher

The data in the audit repository is exposed through pre-defined reports in Oracle Business Intelligence Publisher. The reports allow users to drill down the audit data based on various criteria. For example:
- Username
- Time Range
- Application Type
- Execution Context Identifier (ECID)

For more introductory information for the Oracle Fusion Middleware Audit Framework, see the "Introduction to Oracle Fusion Middleware Audit Framework" chapter in the Oracle Fusion Middleware Security Guide.

For information on how to configure the repository for Oracle Fusion Middleware Audit Framework, see the "Configuring and Managing Auditing" chapter in the Oracle Fusion Middleware Security Guide.

The EDG topology does not include Oracle Fusion Middleware Audit Framework configuration. The ability to generate audit data to the bus-stop files and the configuration of the audit loader will be available once the products are installed. The main consideration is the audit database repository where the audit data is stored. Because of the volume and the historical nature of the audit data, it is strongly recommended that customers use a separate database from the operational store or stores being used for other middleware components.

11 Managing the Topology

11.1 Monitoring the Topology

11.2 Configuring UMS Drivers

11.3 Scaling the Topology

11.3.1 Scaling Up the Topology (Adding Managed Servers to Existing Nodes)

11.3.2 Scaling Out the Topology (Adding Managed Servers to New Nodes)

11.3.3 Scaling Out the Oracle Content Server (OCS) Topology

11.4 Performing Backups and Recoveries

11.5 Troubleshooting

11.5.1 Admin Server Fails to Start After a Manual Failover

11.5.2 Error While Activating Changes in Administration Console

11.5.3 OAM Configuration Tool Does Not Remove URLs

11.5.4 Portlet Unavailable After Database Failover

11.5.5 Redirecting of Users to Login Screen After Activating Changes in Administration Console

11.5.6 Redirecting of Users to Administration Console's Home Page After Activating Changes to OAM

11.6 Best Practices

11.6.1 Preventing Timeouts for SQLNet Connections

11.6.2 Setting AUTOEXTEND for Tablespaces for Processing More Than 30,000 Files

11.6.3 Auditing