This chapter provides guidelines to help you troubleshoot problems with your Oracle Communications Billing and Revenue Management (BRM) system. You can find information about interpreting error messages, diagnosing common problems, and contacting Oracle Technical Support.
Before you read this chapter, you should be familiar with how BRM works. See "BRM System Architecture" in BRM Concepts.
For information on problems related to poor performance, see "Improving BRM Performance".
When any problems occur, it is best to do some troubleshooting before you contact Oracle Technical Support:
You know your installation better than Oracle Technical Support does. You know if anything in the system has been changed, so you are more likely to know where to look first.
Troubleshooting skills are important. Relying on Technical Support to research and solve all of your problems prevents you from being in full control of your system.
If you have a problem with your BRM system, ask yourself these questions first, because Oracle Technical Support will ask them of you:
What exactly is the problem? Can you isolate it? For example, if an account causes Customer Center to crash on one computer, does it give the same result on another computer? Or if users cannot authenticate, is it all services or just one service? If it is an IP problem, does it affect all users or just those on a specific POP or a specific type of terminal server?
Oracle Technical Support needs a clear and concise description of the problem, including when it began to occur.
What do the log files say?
This is the first thing that Oracle Technical Support asks for. Check the error log for the BRM component you are having problems with. If you are having problems connecting to BRM, start with the log for the CM.
Have you read the documentation?
Look through the list of common problems and their solutions in "Diagnosing Some Common Problems with BRM".
Has anything changed in the system? Did you install any new hardware or new software? Did the network change in any way? Does the problem resemble another one you had previously? Has your system usage recently jumped significantly?
Is the system otherwise operating normally? Has response time or the level of system resources changed? Are users complaining about additional or different problems?
If the system is completely dead, check the basics: Can you run testnap successfully? Can you access Oracle outside of BRM? Are any other processes on this hardware functioning normally?
If the error message points to a configuration problem, check the configuration file (pin.conf or properties) for the troublesome component. If you find that the solution requires reconfiguring the component, stop all processes for that component, change the configuration, and stop and restart the component.
For more information, see:
If you still cannot resolve the problem, contact Oracle Technical Support as described in "Getting Help with BRM Problems".
BRM error log files provide detailed information about system problems. If you are having a problem with BRM, look in the log files.
Log files include errors that must be managed and errors that do not need immediate attention (for example, invalid logins). To manage log files, you should make a list of the important errors for your system, as opposed to errors that do not need immediate attention.
BRM records system activity in a set of log files, one for each component or application. If a component of your BRM system has a problem, you can find the error message in the log file for that component. For information on locating a log file, see "Using Logs to Monitor Components".
BRM error messages use this syntax:
[severity] [date_&_time] [host_name] [program]:[pid] [file]:[line] [correlation_id] [message]
where:
severity is the severity of the error message. It can be one of the following values:
E: Error. An error indicates that a component of your BRM system is not operating correctly. This is the most severe type of problem.
W: Warning. Warnings indicate that the system cannot perform a task because the database contains incorrect or inconsistent data. The system continues to operate, but you should investigate and resolve problems with the data immediately.
D: Debug. Debugging messages, which indicate problems with an application, are typically used by application developers to diagnose errors in custom applications. You see these messages only if you set error reporting to the highest level. See "Setting the Reporting Level for Logging Messages".
date_&_time is the date and time the message was logged.
host_name is the name of the computer generating the message. If several machines are sharing a log file using Network File System (NFS), or if all log files are stored in a central location, use this information to pinpoint the machine with the problem.
program is the name of the program (or process) generating the log message. This information helps resolve problems in billing utilities, for example, because all billing utilities use the same log file.
pid is the process ID for the process generating the log message. This information helps resolve problems in components, such as Connection Managers (CMs) and Data Managers (DMs), that might have many processes running in parallel with the same name (program).
file is the name of the source file where the error was detected. Technical Support uses this information when diagnosing system problems.
line is the line number in the source file where the error was detected. Technical Support uses this information when diagnosing system problems.
correlation_ID is the identifier to correlate the log messages from all BRM components related to a single client operation. This information can be used to sort error messages and to identify the set of error messages generated from a single operation. See "Logging External User Information in Error Logs".
message is a detailed description of the error condition. Part of the message often includes the error type, location, and code, which you can use to interpret the error. See "Reference Guide to BRM Error Codes".
An error often produces a cluster of error messages in the log file. The Facilities Modules (FMs), especially, tend to generate cascading messages. To resolve the error, isolate the group of messages, as defined by their common correlation ID, date/time, and process ID, and look at the first one in the series. The error location for that message generally indicates the source of the problem. Then find the last message text in the first error, to identify the operation that was associated with the error. Always consider whether an error could have been caused by something happening in a downstream process.
When using an external application to connect to BRM, if a request from the external application fails, BRM logs the BRM user, the external user and the external correlation ID in the correlation_ID of the log header, to identify the original request from the external application.
The additional information in the correlation_ID uses the following syntax:
BRM_user::external_user:external_correlation_ID
Where:
BRM_user: Specifies the user with which BRM client connects to the CM.
external_user: Specifies the user from the external system connecting to BRM.
external_correlation_ID: Specifies the correlation ID that an external system passes to BRM to correlate between an operation within its system and the corresponding operations in BRM.
For example:
2:CT1255:Account_Manager:1948:1684:63:1063403309:14:root.0.0.0.1::user1:123456789
Note:
If the external application does not provide the external user and external correlation ID, the correlation_ID displays empty strings.For example:
2:CT1255:Account_Manager:1948:1684:63:1063403309:14:root.0.0.0.1:::
The following examples show the typical process for evaluating and interpreting error messages to resolve problems with BRM.
A BRM client application fails and displays an error message.
Look in the application's log file. The file shows the following error message:
E Fri Sep 12 14:50:05 2003 db2.corp:12602 sample_app.c:173 2:CT1255:Account_Manager:1948:1684:63:1063403309:14:root.0.0.0.1::user1:123456789 op_cust_create_acct error [location=pin_errloc_dm class= errno= field num= recid=<0> reserved=<0>]
The message shows that:
At 014:50:05 the system returned an error.
The host name is db2.corp.
The file name is sample_app.c.
The line of code is 173.
The correlation ID is
2:CT1255:Account_Manager:1948:1684:63:1063403309:14:root.0.0.0.1::user1:123456789
There was a problem creating an account.
The error was first found in the DM.
Check the Data Manager log file (dm_oracle.pinlog) for an error message that occurred at the same time and has the same correlation ID, in this case Fri Sep 12 14:50:5 and
2:CT1255:Account_Manager:1948:1684:63:1063403309:14:root.0.0.0.1::user1:123456789
E Fri Sep 12 14:50:05 2003 db2.corp dm:12250 dm_subr.c(1.58):351
2:CT1255:Account_Manager:1948:1684:63:1063403309:14:root.0.0.0.1::user1:123456789
ORACLE error: do_sql_insert: oexec: code 1653, op 4, peo 1
=ORA-01653: unable to extend table PIN.EVENT_T by 512 in
tablespace PIN00"insert into event_t ( poid_db, poid_id1,
poid_id0, poid_type, aobj_db,
The error message shows an Oracle error, with the Oracle code 1653.
Consult the Oracle documentation. Code 1653 indicates that there is a problem with growing an extent. Because the error message reported that BRM was unable to extend one of the tables, you can deduce that the problem is that there is no more room in the database and you must increase its size, as explained in the Oracle documentation.
When you try to add a new account in Customer Center, you see the following message:
A problem occurred while attempting to create an account. The error is likely related to some invalid field, missing required information, or duplicate information. Please check fields carefully, and try again.
Check the cm.pinlog file, which shows the following error message:
D Mon Apr 18 12:41:17 2003 turnip cm:16798 fm_bill_purchase.c:493 2:CT1255:Account_Manager:1992:1940:63:1050694800:5:root.0.0.0.1::user1:123456789 qnty (200.000000) > max (1.000000)... E Mon Apr 18 12:41:17 2003 turnip cm:16798 fm_bill_purchase.c:196 2:CT1255:Account_Manager:1992:1940:63:1050694800:5:root.0.0.0.1::user1:123456789 op_bill_purchase error [location=<PIN_ERRLOC_FM:5> class=<PIN_ERRCLASS_APPLICATION:4> errno=<PIN_ERR_BAD_VALUE:46> field num=<PIN_FLD_PURCHASE_MAX:4,237> recid=<0> reserved=<0>]
The debugging message shows that the account creation routine is trying to purchase a product. The error indicates that the purchase quantity (200) is more than the allowed quantity of the product (1). Because the account creation processes use deals to purchase products, this error probably means that the deal has been defined incorrectly. Look in the price list to check the maximum allowable purchase amount for the product.
You cannot start the DM.
Check the dm_oracle.pinlog file, which shows the following error message:
E THU Sep 11 00:30:49 2003 kauai dm:29349 dm_main.c(1.74):1723 2:CT1255:dm:28492:1:0:1063265316:0:root.0.0.0.1::user1:123456789 DM master dm_die:"bad bind(2)", errno 125
This error message indicates that the DM cannot initiate itself. Usually, errno followed by a number means that a system message is associated with this error. You can check the error file: /usr/include/sys/errno.h. In this case, error 125 is listed as ”EADDRINUSE: Address already in use”. In other words, the DM process is trying to use a port that is already in use by another process.
An error in the application log file indicates the error location is the DM.
Check the dm_oracle.pinlog file, which shows the following error message:
E WED Aug 18 01:40:07 2003 kauai dm:402.354 dm_subr.c(1.80):481 2:CT1255:dm:28509:1:0:1061195411:7:root.0.0.0.1::user1:123456789 ORACLE error: do_sql_insert: obndrv: code 1036, op 28, peo 0 =ORA-01036: illegal variable name/number was binding ":poid_DB" buf 0x195b180, bufl 5, ftype 5
The message shows an Oracle error, number 1036, which you can investigate in the Oracle documentation by using the oerr command.
% oerr ora 1036 01036, 00000, "illegal variable name/number" // *Cause: Unable to find bind context on user side // *Action: Make sure that the variable being bound is in the sql statement.
The obndrv function is looking for the variable :poid_DB in the SQL statement, but the error says that it is not there. For information on how to gather the SQL statements generated by the Oracle database or DM, see "Increasing the Level of Reporting for a DM".
This section lists some common problems with BRM and shows you how to diagnose the error messages and resolve the problems.
One of the log files (log or pinlog) for the DM or CM has a reference to ”bad bind” and ”errno 13”.
Cause
The port number specified in the configuration file (dm_port or cm_ptr entry) is incorrect.
Another possibility is that the port number is below 1023, and the CM, CMMP, or DM was not started as root. System processes that use port numbers below 1023 must be started as root. If you use a port number greater than 1024, you do not have to start the process as root.
Solution
Edit the configuration file for the component to specify an unassigned port number above 1023, such as 1950.
The log file for the DM or CM has a reference to ”bad bind” and ”errno 125”.
Cause
Duplicate port number. Some other process is already using the port.
Solution
Edit the configuration file for the component to specify an unassigned port number above 1023, such as 1950.
When you look at the processes running, you see the master Oracle DM and front ends running but no back end running.
Causes
The database name configuration entry (sm_database) for the DM points to the wrong database name. (The error message shows which database name the DM is trying to connect to.)
The Oracle password configuration entry (sm_pw) is missing.
The Oracle tnsnames file is missing or incorrect.
The oracle_sid or oracle_Home environment variable is set incorrectly.
The Oracle DM is spawning too many back-end processes simultaneously for the IPC or BEQ protocol to handle.
Solutions
Enter the correct database name and Oracle user name and password for the BRM database in the configuration file for the Oracle DM and restart the DM.
Create a valid Oracle tnsnames file and check the environment variables.
If you are using the IPC or BEQ protocol, configure the Oracle DM to wait a specified amount of time before spawning or respawning a new back-end process. To do this, add the following entry to the Oracle DM configuration file (BRM_home/sys/dm_oracle/pin.conf):
- dm dm_restart_delay DelayTime
Note:
Adding a delay increases the Oracle DM startup time.where DelayTime is the amount of time, in microseconds, the Oracle DM should wait before spawning a new back-end process. Set DelayTime to the smallest possible time that fixes your connection problems. As a guideline, start with 1000000 microseconds (1 second) and then decrease the time until you find the optimal setting for your system.
While loading the CDRs using the direct path load option, an error stating that the index is in an unusable state occurs.
Cause
While IREL processes the CDRs using the direct path loading option, it updates the indexes. However, as the index is being updated, another application, for example, pin_monitor_balance would also access the same index partition.
Solution
Configure the dm_sql_retry entry in the pin.conf file. This is specified as an integer value that indicates the number of times an SQL statement is to be retried if this error occurs.
Note:
This is not a mandatory parameter to be set in the pin.conf file. The default behavior is to not try running the SQL statement if the error occurs.You run the stop script, but the script fails. You find a reference to ”permission denied” in the log file for the component.
Cause
You do not have permission to stop the BRM system.
Solution
Log in as root or as the user who started the BRM system.
You run the stop script, but the script fails. You find a reference to ”no pid file” in the log file for the component.
Cause
BRM cannot find the .pid file.
Solution
Identify the process ID for the component you want to stop, and then stop the process manually. See "Starting and Stopping the BRM System".
When you try to start a client application, you get an error message advising you of ”problems connecting to the database.”
Cause
The CM might not be set to handle the number of current client sessions.
Solution
Set the cm_max_connects entry in the configuration file for the CM to a number larger than the number of client sessions you anticipate. Then restart the CM. See "Starting and Stopping the BRM System".
An application cannot connect to BRM, and the log file for the application (which might be default.pinlog in the current directory) shows the error ”PIN_ERR_NAP_CONNECT_FAILED(27).”
Causes
The configuration file (pin.conf) for the application might be pointing to the wrong CM.
The CM is not running.
The CM is not set to handle this many connections.
No TCP sockets are available on the client or CM machine, perhaps because you used many sockets recently and the sockets have not been released from their two-minute wait period after the connections were closed.
Solutions
Open the configuration file for the application and check the entries that specify the CM.
Check for CM processes. See "Checking the Number and ID of a BRM Process".
Set the cm_max_connects entry in the configuration file for the CM to a number larger than the number of application sessions you anticipate. Then restart the CM.
Wait a few minutes to see if the sockets are freed up.
On Solaris: To see how many sockets are available:
netstat -n -f inet -p tcp | wc -l
On HP-UX IA64: To see how many sockets are available:
netstat -n -f inet | grep ’^tcp | wc -l
On Linux: To see how many sockets are available:
netstat -n -A inet -t | wc -l
On AIX: To see how many sockets are available:
netstat -n -f inet | grep tcp
If the resulting number is close to 65535, there are too many socket connections for a single IP address on this machine.
You might find a message similar to the following:
DMfe #3: dropped connect from 111.122.123.1:45826, too full W Thu Aug 06 13:58:05 2001 portalhost dm:17446 dm_front.c(1.47):1498
Cause
There are not enough connections allowed for the DM.
Solution
Use the dm_max_per_fe parameter in the DM configuration file to increase the number of CM connections allowed.
Install and configure an additional DM.
Your BRM system stops responding, or Oracle reports deadlocking messages.
Cause
The DM might have too few back ends for the type of BRM activity.
Solution
Configure the DM with more back ends. For example, provide at least two DM back ends for each customer service representative. For more guidelines on setting the number of back ends, see "Improving Data Manager and Queue Manager Performance".
The Oracle DM (dm_oracle) waits indefinitely for a response from the Oracle database.
Cause
If there is a problem with the Oracle database, dm_oracle might hang when it attempts to connect to the database.
Solution
Set the database_request_timeout_duration parameter in the dm_oracle configuration file (BRM_home/sys/dm_oracle/pin.conf):
- dm database_request_timeout_duration milliseconds
where milliseconds is the number of milliseconds the DM waits for a response. For example:
- dm database_request_timeout_duration 10000
If the database does not respond during the wait period and you are using Oracle RAC, the DM times out and then makes one attempt to connect to another Oracle database instance.
If this pin.conf parameter is not specified or is set to 0, the connection attempt does not time out.
Note:
If you are using a single database or a multischema system without Oracle RAC, the DM attempts to connect to the same database schema again. In this case, the timeout setting is useful only if you are experiencing temporary network problems.Important:
If you are using Oracle RAC, the tnsnames.ora file must be configured correctly for the reconnection to work.The DM will not start, and the error log file for the DM refers to ”bad shmget” or ”bad shmat” and ”errno 12.” Or, when the system is running, the CM or an application shows the error ”PIN_ERR_NO_MEM” in its log file.
Causes
The DM or another queuing-based daemon did not have enough shared memory to complete the operation. This is caused by one or more of the following conditions:
Other processes are using all of the shared memory.
There are too many CM processes.
There are memory leaks in the CM or its FMs.
On Solaris: The shared memory segment allocated by one of the DM processes has not been cleaned up properly, leaving a sizeable chunk of memory allocated but unused. This condition, rare in normal operation, can be caused by the following activities:
Repeated starting and stopping of the system.
Stopping the DMs manually, especially by using kill -9.
On Solaris: The shared memory configuration for the system is less than the shared memory set for BRM.
Solutions
To check for memory leaks, use ps with the vsz flag at two or more intervals to see changes in shared memory.
On Solaris:
ps -eo pid,vsz,f,s,osz,pmem,comm | egrep 'cmldml [application]'
On HP-UX IA64:
ps -eo pid,vsz,flags,state,sz,pmem,comm | egrep 'cmldml [application]'
On Linux:
ps -eo pid,vsz,f,s,sz,pmem,comm | egrep 'cmldml [application]'
On AIX:
ps -eo pid,vsz,dpgsz,THREAD,comm |egrep 'cmldml [application]'
For a CM, vsz should grow only until an operation has passed through the CM and then stay constant. For example, if vsz is growing during billing or RADIUS Manager operations, there is a memory leak.
To check for and clean up unused memory on Solaris:
Stop all DM processes. See "Starting and Stopping the BRM System".
Confirm that there are no DM processes running. See "Checking the Number and ID of a BRM Process".
Run df -k to check swap space usage. Confirm that the available space is very low.
Run ipcs -ma to show the shared memory segments that have been allocated but not used recently. A shared memory segment is probably abandoned when you see the following conditions:
Number of attaches (NATTCH) is 0
KEY is 0 (and not using a special dm_shmkey)
Creator process ID (CPID) is gone
Last detach time (DTIME) has a value
Run ipcrm -m segment_id on each of the unused segments to free up the space.
Run df -k again to confirm that the available swap space has been cleared.
Stop and restart the DM processes. See "Starting and Stopping the BRM System".
To increase the system shared memory on Solaris, open the /etc/system file and set the shminfo_shmmax configuration parameter to a value greater than the value of dm_shmsize in the DM configuration file (pin.conf). Stop and restart the computer.
Example /etc/system file for a 64 MB system:
set shmsys:shminfo_shmmax=37748736 set shmsys:shminfo_shmmin=1 set shmsys:shminfo_shmmni=100 set shmsys:shminfo_shmseg=10 set semsys:seminfo_semmns=200 set semsys:seminfo_semmni=70
In this example, the shared memory segment has been set to 36 MB (1048576 times 36).
To check for and clean up unused memory on Linux:
Stop all DM processes. See "Starting and Stopping the BRM System".
Confirm that there are no DM processes running. See "Checking the Number and ID of a BRM Process".
Run df -k to check swap space usage. Confirm that the available space is very low.
Run ipcs -ma to show the shared memory segments that have been allocated but not used recently.
Run ipcs -mac to show the shared memory segments that have been allocated along with the corresponding user information.
Run ipcs -mat to show the shared memory segments that have been allocated detach timing information.
Note:
In steps 4, 5, and 6, a shared memory segment is probably abandoned when you see the following conditions:Number of attaches (NATTCH) is 0
KEY is 0 (and not using a special dm_shmkey)
Creator process ID (CPID) is gone
Last detach time (DTIME) has a value
Run ipcrm -m segment_id on each of the unused segments to free up the space.
Run df -k again to confirm that the available swap space has been cleared.
Stop and restart the DM processes. See "Starting and Stopping the BRM System".
To increase the system shared memory on Linux, open the /etc/sysctl.conf file and set the shminfo_shmmax configuration parameter to a value greater than the value of dm_shmsize in the DM configuration file (pin.conf). Stop and restart the computer.
To check for and clean up unused memory on AIX:
Stop all DM processes. See "Starting and Stopping the BRM System".
Confirm that there are no DM processes running. See "Checking the Number and ID of a BRM Process".
Run df -k to check swap space usage. Confirm that the available space is very low.
Run ipcs -ma to show the shared memory segments that have been allocated but not used recently.
Run ipcs -mac to show the shared memory segments that have been allocated along with the corresponding user information.
Run ipcs -mat to show the shared memory segments that have been allocated detach timing information.
Note:
In steps 4, 5, and 6, a shared memory segment is abandoned when you see the following conditions:Number of attaches (NATTCH) is 0
KEY is 0 (and not using a special dm_shmkey)
Creator process ID (CPID) is gone
Last detach time (DTIME) has a value
Run ipcrm -m segment_id on each of the unused segments to free up the space.
Run df -k again to confirm that the available swap space has been cleared.
Stop and restart the DM processes. See "Starting and Stopping the BRM System".
When using GUI applications such as Pricing Center, Suspense Manager, and Customer Center or batch applications such as Invoice formatter, you may sometimes receive ”java.lang.OutOfMemoryError: Java heap space” error messages.
Cause
The Java application does not have enough memory to complete the operation.
Solution
Increase the maximum heap size used by the Java Virtual Machine (JVM). The exact amount varies greatly with your needs and system resources.
The heap size is controlled by the -Xmx size entry in the Java application startup script. By default, the -Xmx size entry is not present in the startup line. To increase the maximum heap size, add this entry and a number (in megabytes) to the application startup line. The following example adds a 1024 MB maximum heap size to the class:
java -Xmx1024m class
Note:
Increasing the heap size may degrade the performance of other processes if insufficient resources are available. You must adjust the heap size based on your application needs and within your system's limits.The error log file for the DM for your Oracle database refers to ”No memory for...”, such as ”No memory for list in pini_flist_grow.” You suspect memory problems, but your system has sufficient memory for the environment.
Cause
The DM is not configured to use sufficient shared memory.
Solution
Open the DM configuration file (BRM_home/sys/dm_oracle/pin.conf).
Increase the size of the dm_bigsize and dm_shmsize parameters. Follow the guidelines in the configuration file for editing these entries.
Save the configuration file.
Stop and restart the DM.
Even though the billing processes are running, BRM is not producing billing data.
Cause
There are too few back ends for the DM. Because billing daemons run in parallel, you must have at least one DM back end for each billing program thread, plus one back end for the master thread searches.
Solution
Edit the dm_n_be entry in the DM configuration file (pin.conf) to add more back ends to the DM, and then stop and restart the DM. See "Configuring DM Front Ends and Back Ends".
Running the billing scripts puts an inordinately heavy load on the computer, and processing the accounts takes a long time.
Cause
An index is missing or unbalanced; or in Oracle, an index is in the CHOOSE Optimizer mode and statistics are out of date.
Solution
Rebuild the BRM indexes before you run the billing scripts. See "Rebuilding Indexes" in BRM Installation Guide.
Cause
BRM is trying to send a welcome email message, but the Email DM (dm_email) is not running.
Solution
Start the Email DM, or disable the welcome email message.
To start the Email DM, see "Sending Email to Customers Automatically" in BRM Managing Customers.
To disable the welcome message, see "Disabling the Welcome Message" in BRM Managing Customers.
Cause
The load_config utility tried to load the contents of XML configuration files into configuration (/config/*) objects in the BRM database, but the contents are not loaded.
Solution
Set the ORACLE_HOME environment variable to the BRM database client library path; for example, /tools/CGBU/contrib/Linux/x86_64/packages/oracle/db/12.2.0.1.0.
If you cannot resolve your problems with BRM, contact Oracle Technical Support.
Problems can often be fixed simply by shutting down BRM and restarting the computer that the BRM system runs on. See "Starting and Stopping the BRM System".
If that does not solve the problem, the first troubleshooting step is to look at the error log for the application or process that reported the problem. See "Using Error Logs to Troubleshoot BRM". Be sure to observe "General Checklist for Resolving Problems with BRM" before reporting the problem to Oracle Technical Support.
If "General Checklist for Resolving Problems with BRM" does not help you resolve the problem, write down the pertinent information:
A clear and concise description of the problem, including when it began to occur.
Relevant portions of the relevant log files.
Relevant configuration files (pin.conf or properties).
Recent changes in your system, even if you do not think they are relevant.
List of all BRM components, ServicePaks, FeaturePaks, and patches installed on your system.
Note:
You can collect BRM information by using the pinrev script. For information, see "Checking BRM Component Version Numbers".
You can collect pipeline information by using the piperev script. For information, see "Checking Pipeline Component Version Numbers".
When you are ready, report the problem to Oracle Technical Support.