10 Performing Maintenance and Troubleshooting

This chapter contains basic system maintenance and database troubleshooting information.

Basic Maintenance

This section contains information about the basic maintenance tasks for Oracle Communications MetaSolv Solution (MSS). See MetaSolv Solution System Administrator's Guide for full system maintenance information

Starting the MSS System

When starting MSS, you must start components in the following order:

  • Database

  • Application server

  • MSS client (user workstation)

The startup scripts are located in the domain directory. The script name includes the server name. For example, if a server is named mslv01, it will have a startup script named startmslv01.sh in the domain directory.

To start MSS, run the following scripts located in the domain directory:

  • For Unix Linux:

    startWebLogic.sh
    
  • For Windows:

    startWebLogic.cmd
    

    Starts the administration server if you are running in a domain that has a separate administration server. Start the administration server before starting the managed server(s).

  • For Unix Linux:

    start<server>.sh
    
  • For Windows:

    start<server>.cmd
    

    Starts an application server (either managed or single) running MSS.

If you are running an administration server and a managed server on the same machine, after the managed server is started, the administration server can be shut down to release resources that the process would otherwise consume. However, the administration server must be running to allow access to the management console or to perform any administration function.

Uninstalling MSS

To uninstall MSS from an application server:

  1. Start the administration server and log in to the WebLogic Server Administration Console.

  2. To undeploy the application, do the following:

    1. From the Domain Structure tree, click Deployments.

    2. In the Change Center pane, click Lock & Edit.

    3. From the Summary of Deployments pane, click the check box for nur.ear (or cluster-nur.ear).

    4. Click Stop and select Force Stop Now.

    5. Click Delete.

    6. Click Yes to delete the deployment.

    7. In the Change Center pane, click Activate Changes.

  3. Repeat step 2 for each version of ASR or LSR that is installed.

  4. To delete the data sources, do the following:

    1. From the Domain Structure tree, expand Services, expand JDBC, and then click Data Sources.

    2. In the Change Center pane, click Lock & Edit.

    3. From the Summary of JDBC Data Sources pane, click the check box for each data source that is to be deleted (refer to the list below).

      • Services > JDBC > Data Sources > mslvDataSource

      • Services > JDBC > Data Sources > mslvDbTraceDataSource

      • Services > JDBC > Data Sources >mslvNoneTxDataSource

      • Services > JDBC > Data Sources >mslvProcDataSource

      • Services > JDBC > Data Sources >mslvPSDataSource

      • Services > JDBC > Data Sources >mslvWSDataSource

    4. Click Delete.

    5. Click Yes to delete the data sources.

    6. In the Change Center pane, click Activate Changes.

  5. In the file system on the machine, delete the MSS installation directory.

Changing an IP Address

When an IP address for a server changes, adjustments must be made in the configuration. The following sections indicate how to handle this change.

Changing an IP Address for Clustered Servers

The IP address or DNS name can be used to identify a machine's listen address. Oracle recommends using the DNS name.

When the IP address changes on a server's machine, you must modify the machine's IP address in the Oracle WebLogic server if it is used as the server listen address, the gateway.ini, integration.xml, tbs.ini, tbs_util.ini, and any other files that identify the server machine using the IP address.

Changing an IP Address for Single or Administration Servers

Before the IP address changes, start the server and change the listen address to the new IP address, then implement the IP address change and restart the server.

Change the startserver.sh file to show the new address:

ADMIN_URL=http://<Admin server IP address:port number>/

If you use the DNS name of the server's host machine rather than the IP address, there is no effect when the IP address changes.

Troubleshooting Database Issues

This section contains information about the causes and resolutions to common database problems so you can troubleshoot in MSS.

ORA-12519, TNS: No Appropriate Service Handler Found

Problem

Listener refused the connection with the following error:

ORA-12519, TNS: no appropriate service handler found

Cause

You receive this error message when activating the data sources (DS) or when starting the server.

Resolution

Check the DB parameter for the number of processes. The number of processes should be high enough to hold the number of connections that you want to create in addition to the WL internal processes.

ORA-01979: missing or invalid password for role WOTSTWTWWOO or ADMIN_ROLE

Problem

During upgrade, you receive the following error:

ORA-01979: missing or invalid password for role "WOTSTWTWWOO" or "ADMIN_ROLE"

Cause

You receive this error message when activating the data sources (DS) or when starting the server.

Resolution

Change the role passwords.

Only the database administrator can change passwords for the roles ADMIN_ROLE and WOTSTWTWWOO by running the following stored procedure:

===========pl/sql should be run by dba for changing role's password==========
DECLARE
  C_NAME VARCHAR2(200);
  C_PASSWORD VARCHAR2(200);
BEGIN
  C_NAME := 'role_name'; /*specify the role name, ADMIN_ROLE OR WOTSTWTWWOO*/
  C_PASSWORD := 'password'; /*specify the password*/
  SP_CREDSTORE_CHG_ROLE_PWD(C_NAME, C_PASSWORD);
END;

ORA-39384: Warning: User <USERNAME> Has Been Locked And The Password Expired During Import

Problem

After DataPump import, the users with 10g style verifiers are locked, the passwords are expired, and the following error message is displayed:

ORA-39384: Warning: User <USERNAME> Has Been Locked And The Password Expired During Import

Cause

Because 10g style verifiers are no longer supported in 12.2, DataPump marks such users as locked and expires the password when they are imported.

Resolution

After the import execution, unlock the user and set a new value in the target database by running the following command:

ALTER USER <username> ACCOUNT UNLOCK;
ALTER USER <username> IDENTIFIED BY <password>;

SQL*Plus Quits During the Upgrade

SQL*Plus may unexpectedly quit while running the upgrade under the following scenarios:

  • The ASAP, EBOND, EDI, JOB, and SYS user IDs do not exist on the instance.

  • You entered an incorrect password.

  • ASAP does not have DBA-level authority.

  • The database is not at the appropriate version level when starting the upgrade.

  • You do not have the standard tablespaces data and indexes.

  • The full directory path for the SQL script files does not include the trailing slash, you entered an invalid path, or you do not have write access to the directory.

  • The free space checks did not pass (minimum available tablespace free for the largest contiguous space and total free space). Check the output log (upgenv.log) for the exit reason, correct the tablespace limitation, and run the _upgmss.sql script again.

Restarting a Failed Upgrade Attempt With the Incorrect Mode

When failure occurs before the main upgrade sequence has begun (for example, during the interactive upgrade sequence), use your original selection of either upgrade mode 1 or 2 to restart the upgrade. Do not use upgrade mode 3 to restart the upgrade if a failure occurs during the interactive upgrade sequence.

Upgrade Failure and Losing Audits

If upgrade mode 1 was run more than once, all audits are lost and you must start the upgrade again from the beginning.

Database Graphics Not Displaying Correctly

Check to make sure you have run the TBSGraphicLoad.exe located in the same directory as the MSS executable on the client workstation. This executable contains new and updated graphics for the MSS database. It does not need to be run by every user. It only has to run one time on the database for the release.

No Database Log Files

When _upgradetbs.sql runs, you are prompted for the directory path for the script files that are used to do the upgrade. The path must end with a trailing slash. The slash is different for UNIX/Linux and Windows:

/ for UNIX and Linux

\ for Windows

If the slash is not included, no log files are written.

Troubleshooting MSS Installation Issues

This section describes how to troubleshoot some of the issues that you may encounter during the MSS installation process.

Destination Unreachable Error in AppServerLog.xml in IPv6 Environment

After you install MSS in an IPv6 environment, you may receive the following error in the AppServerLog.xml file:

Error initializing the MetaSolv Solution Sender:
javax.naming.CommunicationException:
t3:// 2606:b400:2010:484b:f13b:4672:38b7:5af9:9065:
Destination 2606:b400:2010:484b:f13b:4672:38b7:5af9,
9065 unreachable; nested exception is:
java.net.ConnectException: Connection refused: no further information;
No available router to destination [Root exception is java.net.ConnectException:
t3:// 2606:b400:2010:484b:f13b:4672:38b7:5af9:9065:
Destination 2606:b400:2010:484b:f13b:4672:38b7:5af9,
9065 unreachable; nested exception is:
java.net.ConnectException: Connection refused: no further information;
No available router to destination]
at weblogic.jndi.internal.ExceptionTranslator.
toNamingException(ExceptionTranslator.java:40)

This message is thrown if the Integration Server, Event Server and System Task Server are enabled. If this error repeatedly occurs (even after the server is up and running successfully), you must enable the HTTP Tunneling option through the WebLogic administrative console. To enable HTTP Tunneling perform the following:

  1. Login to the WebLogic administrative console.

  2. Select Home, Summary of Services, and then Summary of Servers.

  3. Select AdminServer.

  4. Enable the Tunneling option.

Once the tunneling option is set, you can ignore the single issuance of the error message.

Generation Problem or Error for the Nameservice.ior File

During the installation of MSS, you may have an issue on the creation of the NameService.ior file and no error messages existing in the log files. To resolve this issue, you need to ensure that the following permissions are on the directories and files within the MSS home directory structure.

Another error can occur when starting the server with any one of the CORBA servers enabled, and the server can throw the following error in the mss.log file:

Tried and failed to open file: MSLV_HOME/m63domain/appserver/ior/NameService.ior
SEVERE Exception while converting string to objectorg.omg.CORBA.BAD_PARAM:
Invalid or unreadable URL/IOR: file:///MSLV_HOME/m63domain/appserver/ior/NameService.ior
vmcid: 0x0  minor code: 0  completed: No

Table 10-1 lists the directories and permissions that you need to address where MSLV_Home is the directory for the MSS home directory and MSS_Server_Name is the appropriate MSS server name.

Table 10-1 Required Directory and File Permissions

Directory/File Owner Permissions Group and Other Permissions

MSLV_Home/MSS_Server_Name/appserver

Read, Write, Execute

Read, Execute

MSLV_Home/MSS_Server_Name/appserver/ior

Read, Write, Execute

Read, Execute

MSLV_Home/MSS_Server_Name/jacORB

Read, Write, Execute

Read, Execute

MSLV_Home/MSS_Server_Name/jacORB/startMSLVorb.sh

Read, Write, Execute

Read, Execute

MSLV_Home/MSS_Server_Name/jacORB/bin

Read, Write, Execute

(You also need to set these permissions for all the sub-directories and files.)

Read, Execute

These problems can occur because of the existing permissions on the files and directories in the environment. Once you set the permissions correctly, restart your server, the IOR file should exist in the MSLV_Home/MSS_Server_Name/appserver/ior directory, and no open file error should exist in the mss.log file.

Web Service Deployment Warning Message

You can ignore the following warning message that appears in the appserver.mss.log file during Web Service deployment:

<Warning> <com.sun.xml.ws.wsdl.PayloadQNameBasedOperationFinder> <BEA-000000> <Non unique body parts! In a port, as per BP 1.1 R2710 operations must have unique operation signature on the wire for successful dispatch. Methods [getServiceLocationRequest, getEntityByKeyRequest] have the same request body block {http://www.openuri.org/}getEntityByKeyRequest. Method dispatching may fail, runtime will try to dispatch using SOAPAction. Another option is to enable AddressingFeature to enabled runtime to uniquely identify WSDL operation using wsa:Action header.>

This Warning message is related to getServiceLocationRequest and getEntityByKeyRequest APIs, because both the APIs use the same request structure. Because customers have used these two APIs on their integrations in prior versions of MSS, such as 6.2.1, these two APIs are also available in MSS 6.3.0, for customers migrating to the MSS 6.3.0 release.

Access Problem for the WebLogic Console and MSS Application Using IPv6 Address

After you create the domain and install the MSS application using IPv6 address, you may encounter the following issues:

  • You are unable to access the WebLogic Server Administration Console using Internet Explorer and the following error message appears:

    The proxy could not connect to the destination in time. Please verify the site you are attempting to access and retry.
    
  • You are unable to log in to the MSS application and the following error message appears:

    The user ID you entered is not valid. Please use a valid User ID or ask your manager for assistance.
    

To resolve both the issues, do the following:

  1. Start Internet Explorer.

  2. From the Tools menu, select Internet Options.

    The Internet Options window appears.

  3. Click the Connections tab, and then click LAN Settings.

    The Local Area Network (LAN) Settings window appears.

  4. Under the Automatic configuration section and the Proxy server section, deselect all the check boxes.

  5. Click OK, which returns you to the Internet Options window.

  6. Click OK.

The Runtime Information Page is Not Displayed

The Runtime Information Page is not displayed and the following error message appears:

Error 500--Internal Server Error
From RFC 2068 HyperText Transfer Protocol -- HTTP/1.1:
10.5.1 500 Internal Server Error
The server encountered an unexpected condition which prevented it from fulfilling the request.

And the MSS error log reflects the following:

javax.servlet.jsp.JspException: No collection found
at org.apache.struts.taglib.logic.IterateTag.doStartTag(IterateTag.java:281)
at jsp_servlet.__runtimeinfo._jspService(__runtimeinfo.java:594)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
Truncated. see log file for complete stacktrace

To resolve this issue, you must enable HTTP tunneling in the WebLogic Server Administration Console.

Enabling HTTP Tunneling

To enable HTTP tunneling:

  1. Log in to the WebLogic Server Administration Console.

  2. In the Domain Structure tree, expand Environment, and then click Servers.

    The Summary of Servers page appears.

  3. Click a server (for example, m63ditu1server).

    The Settings for ServerName page is displayed.

  4. Click the Protocols tab.

  5. Under Change Center, click Lock & Edit.

  6. On the HTTP tab, do the following

  7. Select the Enable Tunneling check box.

  8. Click Save.

  9. In the Change Center of the Administration Console, click Activate Changes, which activates these changes.

Check Box To Select the Server Name is Not Displayed in the MSS Installer

If you try to connect to the WebLogic Server by specifying an IPv6 address in the Admin Host field of the MSS installer, the check box to select the server name is not displayed under the domain tree.

Figure 10-1 Check Box to Select the Server Name

Description of Figure 10-1 follows
Description of "Figure 10-1 Check Box to Select the Server Name"

To resolve this issue, ensure that the IPv6 address of the Admin Host is added in the hosts file of the Operating System, as follows:

2606:b400:2010:484b:f13b:4672:38b7:5af9  slc04wro.example.com  slc04wrolocalhost

On Windows, the hosts file is typically located at C:\Windows\System32\drivers\etc\. On Unix and Solaris, the hosts file is located at /etc/hosts.

My Desktop Page is Not Displayed

If you install and deploy MSS on a single server using IPv6 address with SSL enabled, the My Desktop page is not displayed in the MSS application.

To resolve this issue, you must enable HTTP tunneling in the WebLogic Server Administration Console. See "Enabling HTTP Tunneling" for more information.

Number Format Exception When Starting the Proxy Server

When starting the proxy server, you may receive the following number format exception:

java.lang.NumberFormatException: For input string: "b400:2010:484b:f13b:4672:38b7:5af9"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at weblogic.servlet.proxy.HttpClusterServlet$Server.parsePorts
(HttpClusterServlet.java:2042)
at weblogic.servlet.proxy.HttpClusterServlet$Server.<init>
(HttpClusterServlet.java:1898)
at weblogic.servlet.proxy.HttpClusterServlet$ServerList.<init>
(HttpClusterServlet.java:2078)
at weblogic.servlet.proxy.HttpClusterServlet.init
(HttpClusterServlet.java:141)

To resolve this issue, ensure that you specify the IPv6 addresses within [] (brackets) in the WebLogicCluster init parameter of the HttpClusterServlet in the web.xml of the proxy server.

For example:

<init-param>
  <param-name>WebLogicCluster</param-name>
  <param-value> [IPv6_address1]:port1|[IPv6_adrress2]:port2</param-value>
</init-param>

Unable to Log In to the MSS Application

When logging in to the MSS application, you may receive the following error:

Unable to communicate with application server selected.

To resolve this issue, do the following:

  • If you are connecting to an IPv6 proxy server for accessing MSS client application, ensure that you specify the proxy server's IPv6 address in the client machine's etc/hosts file, as follows:

    IPv6address                 ipv6machine@example.com    ipv6machine
    
  • Install the self-signed certificate on the client machine. See "Installing the Self-Signed Certificate on the Client Machine" for more information.

Updates to the web.xml File of the Proxy Server are Not Reflected

Your updates are not reflected in the web.xml file of the proxy server.

To resolve this issue, do the following before starting the proxy server for the first time:

  • Make your changes in the web.xml file

  • Specify the IPv6 addresses within [] (brackets) in the WebLogicCluster init parameter of the HttpClusterServlet in the web.xml file

  • Add the SecureProxy init parameter to the HttpClusterServlet in the web.xml file for a cluster that has SSL enabled

If you have already started the proxy server and still unable to see your changes in the web.xml file, redeploy the Oracle Cluster Proxy web application using the WebLogic Server Administration Console.

Updates to orb.properties for SSL and AIX Platform

If you create a domain using SSL on an AIX platform, you must make additional updates to the orb.properties file. Ensure the following property values are set:

  1. Open the file MSLV_Home\mslv01\jacORB\orb.properties

    where:

    • MSLV_Home is the directory in which the MSS software is installed

    • mslv01 is the server home directory

  2. Provide the appropriate values for the following properties:

    jacorb.security.jsse.server.key_manager_algorithm=IbmX509 
    jacorb.security.jsse.server.trust_manager_algorithm=IbmX509 
    jacorb.security.jsse.client.key_manager_algorithm=IbmX509 
    jacorb.security.jsse.client.trust_manager_algorithm=IbmX509
    

Managed Server Startup Warning When Servers on Different Machines

When the admin server and the managed server are on different machines, a warning can occur during a cluster installation. You can encounter the following warning when trying to start the managed server:

####<Warning> <DeploymentService> <BEA-290074> <Deployment service servlet received file download request for file>

To resolve this issue, you add the following java option:

-Dweblogic.data.canTransferAnyFile=true

to the startWeblogic.sh file on the managed server.

DLL Error when Connecting to the Database

A DLL error can occur if an incorrect version of the Oracle Database Client was installed and you run any of the following applications:

  • Location and Routing Gateway

  • NPA Split Utility

  • Data Selection

  • Pre-Migration Analysis Tool

  • MetaSolv Solution Utilities

The DLL error is the following:

Cannot make a connection to the database.
Oracle library OCI.DLL or ORAC803.DLL could not be loaded.

This error can occur if the 64-bit version of the Oracle Database Client was installed instead of the 32-bit version. See MSS Planning Guide for more information on software version requirements.

Window Response Delay from MSS in SSL Environment After No Activity

When running MSS in an SSL environment, it is possible to experience a response delay when moving outside the application and then returning back to MSS. If you leave the MSS application for a few minutes (with no activity), and then later attempt to move to a new MSS window, then there can be a delay of few seconds before it responds. Also, the window header title can include “Not Responding" for few seconds.

This issue occurs due to the idle connection time-out set in the Oracle WebLogic console. To resolve this issue, log in to the WebLogic console, and under Server->Protocol->General, set the option for the idle connection time-out to a higher number. The default is 60 seconds, and after 60 seconds of no activity, it drops the connection to the client. Increase the idle connection time-out value to the appropriate value for your requirements.

CORBA Error During Integration Between MSS and Network Integrity

During the integration between MSS and Network Integrity, when a connection is made to the MSS application from Network Integrity through the CORBA server, Network Integrity returns the following error message:

Can't establish Corba connection with MSS
IDL:omg.org/CosNaming/NamingContext/NotFound1.0.

To resolve this issue, do the following:

  1. Navigate to the MSLV_Home/appserver/gateway directory and edit the gateway.ini file.

  2. Uncomment the INFRASTRUCTURESERVER entry in the [Servers] section.

Known Issues with Reconfiguration Wizard (Only if Reconfiguration Wizard is used to Migrate WebLogic Domain)

After executing the WebLogic Reconfiguration wizard to migrate from 12.2.1.2 to 12.2.1.3/12.2.1.4 domain, the start scripts to start up the MetaSolv application server/cluster throws error "No such file or directory" and the server fails to start.

To resolve this issue, follow the Knowledge Article 2607740.1 - After Execute WebLogic Reconfiguration Wizard Receive "No Such File or Directory" Errors When Starting the Server(s) on the My Oracle Support website:

https://support.oracle.com/epmos/faces/DocumentDisplay?id=2607740.1