Oracle® Fusion Applications Administrator's Troubleshooting Guide 11g Release 6 (11.1.6) Part Number E25450-05 |
|
|
PDF · Mobi · ePub |
This chapter describes common problems that you might encounter when using Oracle Metadata Repository (MDS Repository) and explains how to solve them. This chapter contains the following topics:
Section 9.1, "Introduction to Troubleshooting MDS Repository"
Section 9.3, "Applications Performing Slowly Due to MDS Repository"
Section 9.4, "Diagnosing Exceptions with Oracle Metadata Repository"
Some procedures in this chapter reference content in the Oracle Fusion Middleware guides. These guides describe using Fusion Middleware Control. These procedures also apply to Fusion Applications Control.
This section provides guidelines and a process for using the information in this chapter. Using the following guidelines and process will focus and minimize the time you spend resolving problems.
Guidelines
When using the information in this chapter, Oracle recommends:
After performing any of the solution procedures in this chapter, immediately retrying the failed task that led you to this troubleshooting information. If the task still fails when you retry it, perform a different solution procedure in this chapter and then try the failed task again. Repeat this process until you resolve the problem.
Making notes about the solution procedures you perform, symptoms you see, and data you collect while troubleshooting. If you cannot resolve the problem using the information in this chapter and you must log a service request, the notes you make will expedite the process of solving the problem.
Process
Follow the process outlined in Table 9-1 when using the information in this chapter. If the information in a particular section does not resolve your problem, proceed to the next step in this process.
Table 9-1 Process for Using the Information in this Chapter
Step | Section to Use | Purpose |
---|---|---|
1 |
Get started troubleshooting the MDS Repository. The procedure in this section quickly addresses a wide variety of problems. |
|
2 |
Section 9.3 through Section 9.4 |
Perform problem-specific troubleshooting procedures. These sections describe:
|
3 |
Use My Oracle Support to get additional troubleshooting information about Oracle Fusion Applications or Oracle SOA Suite. My Oracle Support provides access to several useful troubleshooting resources, including Knowledge Base articles and Community Forums and Discussions. |
|
4 |
Log a service request if the information in this chapter and My Oracle Support does not resolve your problem. You can log a service request using My Oracle Support at |
For general instructions on managing the metadata for Oracle Fusion Middleware components in the Oracle Metadata Repository, see the "Managing the Metadata Repository" chapter in the Oracle Fusion Middleware Administrator's Guide.
Use the following tools to troubleshoot Oracle Metadata Repository issues:
Use the MDS Repository commands described in the "Metadata Services (MDS) Custom WLST Commands" section of the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
Use the following script from the Oracle WebLogic Server domain home directory, which is based on WLST commands:
From the fusionapps
Middleware subdirectory, start the WLST:
(UNIX) FA_MW_HOME/oracle_common/common/bin/wlst.sh (Windows) FA_MW_HOME\oracle_common\common/bin\wlst.cmd
where DOMAIN_HOME
is located in the following locations:
(UNIX) APPLICATIONS_CONFIG/instance/domains/host/domain_name (Windows) APPLICATIONS_CONFIG\instance\domains\host\domain_name
Connect to Oracle WebLogic Server.
Use WLST commands for the MDS Repository. For example:
deleteMetadata
, exportMetdata
, importMetadata
, and purgeMetadata
to manage application metadata
registerMetadataDBRepository
to register a database-based MDS repository.
importMAR
to import an application MAR
Use the force=true
argument to force the import of all document. If you do not use the with force
argument, only documents that have changed will be imported.
createMetadataLabel
, deleteMetadataLabel
, listMetadataLabels
, promoteMetadataLabel
, and purgeMetadataLabels
to manage labels
Modify configuration and runtime MBean properties:
Modify the AutoPurgeTimeToLive
and MaximumCacheSize
configuration MBean property in Fusion Applications Control. See the "Changing MDS Configuration Attributes for Deployed Applications" section in the Oracle Fusion Middleware Administrator's Guide.
Modify the clearCache
runtime MBean property in Fusion Applications Control:
From the navigation pane, expand the farm and then Application Deployments.
Click the application that uses MDS Repository.
In the Java EE Application home page, from the Application Deployment menu, choose MDS Configuration.
The MDS Configuration page displays the page.
In the Advanced Configuration section, click Runtime MBean Browser.
In the System MBean Browser page, expand Application Defined MBeans.
The System MBean Browser displays Application Defined MBeans > oracle.mds.lcm > Server: Managed Server > Application: Application Name > MDSAppRuntime > MDSAppRuntime and displays the Application Defined MBeans: MDSAppRuntime:MDSAppRuntime page.
Click the Operations tab.
Click the clearCache.
Click Invoke.
View the logs. Right-click the domain under WebLogic Domain and choose View Log Messages. For more information, see the "Viewing Log Files and Their Messages Using Fusion Middleware Control" chapter in the Oracle Fusion Middleware Administrator's Guide.
From the navigation pane, expand the farm, WebLogic Domain, Managed_Server.
Choose Logs > View Log Messages.
If the problem still cannot be solved, increase the log level of the system to debug the transactions. To simplify troubleshooting, it is recommended that you enable the following parent loggers at the TRACE32 (FINEST) level.
To change logger levels, perform the following steps:
From the navigation pane, expand the farm > WebLogic Domain > Managed_Server.
Choose Logs > Log Configuration.
In the Logger Name column, expand the oracle and then oracle.mds to display the loggers.
In the Oracle Diagnostic Logging Level (Java Level) column, change the logging level to TRACE:32.
- oracle.mds.query for detailed diagnostics for SQL statements, bind variables, and timing information about MDS database queries
- oracle.mds.custmerge for detailed diagnostics when applying MDS customization instructions
- oracle.mds.jsp for trace JSP integration with MDS
- oracle.mds.dbstore for detailed logging about all operations done at the persistence level for the database-based MDS repository
- oracle.mds.sandbox for debug level information for all the sandbox operations.Use this logger when troubleshooting creation of a sandbox, managing the sandbox metadata, applying a sandbox or any other sandbox lifecycle-related operations. Use this logger in conjunction with the oracle.mds.dbstore tracer logger.
- oracle.mds.coretxn for detailed logging for MDSTransaction
methods. MDSTransaction
is an internal class which is associated with an MDSSession
that is being used to make changes to metadata. Enabling this tracer includes diagnostics to help verification that the same MDSSession
is not being used by multiple threads which while not illegal is unusual and can cause problems if done incorrectly.
- oracle.mds.pdcache for trace-document cache access and invalidation. Shows labels being used for lookups and versions being returned from the cache.
- oracle.mds.remotechange for detailed tracing for remote notifications, polling thread and synchronous polling on flushChanges()
. This logger can be useful to check that remote notifications, for example, configuration is correct and changes are being distributed.
- oracle.mds.corecache to trace core cache access and invalidation.
- oracle.mds.jsf for MDSFaceletsHelper
diagnostics for Facelets integration with MDS.
- oracle.mds.rename for detailed diagnostics for listing dependency references during rename operation.
- oracle.mds.custupdate for detailed diagnostics for change events handling and optimization, and creation of MDS customization instructions during update of MDS customizations.
- oracle.mds.custmerge for detailed diagnostics when applying MDS customization instructions
- oracle.mds.mnfe for tracing of MetadataNotFoundException
and isStale()
evaluation.
The change should take effect within a few minutes. Note that in a production system, setting the trace at a fine-grained level can result in a large amount of output that must be diagnosed. You can alternately use selective tracing that provides a way to get a detailed, on-disk trace selectively (for example, by user name, thereby eliminating trace output for other users).
To activate selective tracing, right-click the domain under WebLogic Domain and choose Logs > Selective Tracing.
Note that Selecting Tracing does not display as an option when you right-click an Administration Server or Managed Server and choose Logs.
From the Option Name list, choose the type of selective trace (for example, based on user name), and start the trace.
When the problem has been reproduced, disable the trace and view the output to narrow down the issue.
For more information on selective tracing, see the "Configuring and Using Selective Tracing" section of Oracle Fusion Middleware Administrator's Guide.
Review the error logs (from Fusion Applications Control) for more information on the error.
Cross layer, server, and family functionality can be correlated through the execution context ID (ECID) (for example, you can look up the composite instance for a given expense report by correlating all the log entries with the ECID associated with that expense report transaction). For more information, see the "Correlating Messages Across Log Files and Components" section of Oracle Fusion Middleware Administrator's Guide.
Problem
If the application is performing slowly, there could be an issue with the MDS Repository.
Solution 1
To check the MDS Repository performance and adjust the cache size:
Check performance metrics for cache performance.
Navigate to the application's home page by expanding the farm, then Application Deployments. Then, select an application.
The application's home page is displayed.
From the Application Deployment menu, choose Performance Summary.
The Performance Summary page displays.
Click Show Metric Palette.
Expand MDS Metrics and select IOs Per Metadata Object Get and IOs Per MO Content Get.
Monitor the performance of these metrics. Typically, values close to 1 indicate poor cache performance.
Modify the MaximumCacheSize
configuration MBean property to -1
. See the "Changing MDS Configuration Attributes for Deployed Applications" section in the Oracle Fusion Middleware Administrator's Guide.
Solution 2
To resolve this problem from the MDS Repository database:
Capture an Automatic Workload Repository (AWR) report. See the "Gathering Database Statistics Using the Automatic Workload Repository" section in the Oracle Database 2 Day + Performance Tuning Guide.
Check if any specific MDS SQL is taking too long. Analyze log (if needed with oracle.mds logger set to FINER
) an or DMS data from DMSSpy.
Regather the Oracle Database statistics by executing the GATHER_SCHEMA_STATS
procedure in the DBMS_STATS
PL/SQL package from SQL as follows:
EXEC DBMS_STATS.GATHER_SCHEMA_STATS(
ownname => schema_owner,
estimate_ percent => dbms_stats.auto_sample_size,
method_opt => 'for all columns size auto',
cascade => true);
EXEC DBMS_STATS.GATHER_TABLE_STATS(
ownname =>'schema_Owner',
tabname =>'MDS_PATHS',
method_opt =>'FOR COLUMNS SIZE 1 PATH_FULLNAME');
where schema_owner
is the name of the schema, typically, FUSION_MDS
.
For information about the GATHER_SCHEMA_STATS
procedure, see "GATHER_SCHEMA_STATS Procedures" in the Oracle Database PL/SQL Packages and Types Reference.
If performance does not improve after collecting statistics, then flush the shared pool to clear the execution plan for the database and generate a new query plan with the following SQL command:
ALTER SYSTEM FLUSH SHARED_POOL; ALTER SYSTEM FLUSH BUFFER_CACHE;
If you notice continuous growth in metadata in Oracle Database tables, it could be the purge not executed or labels are preventing metadata from being purged.
Use these queries to identify which of the above is the cause.
To determine how many versions can be purged in soa-infra
:
SELECT count( * ) FROM mds_paths path, mds_txn_locks, mds_partitions WHERE path_low_cn <= lock_txn_cn AND path_low_cn > 0 AND NOT EXISTS (SELECT label_cn from mds_labels WHERE path.path_low_cn <= label_cn AND (path.path_high_cn > label_cn OR path.path_high_cn IS NULL) AND label_partition_id = path.path_partition_id) AND path_high_cn IS NOT NULL AND path_partition_id = lock_partition_id and path_partition_id = partition_id AND partition_name = 'soa-infra'
To determine how many versions are held by labels:
select label_name, label_cn, (SELECT count( * ) FROM mds_paths path, mds_partitions WHERE path_high_cn IS NOT NULL and path_low_cn <= label_cn AND path_low_cn > 0 and path_high_cn > label_cn AND path_partition_id = partition_id and partition_name=:B1) versHeld, txn_time from mds_labels, mds_transactions, mds_partitions where label_partition_id=txn_partition_id and label_cn=txn_cn and label_partition_id=partition_id and partition_name=:B1 order by versHeld desc
To determine partition-wise statistics of tip and non-tip, purgeable versions:
select docs.*, labels.count_labels, can_be_purged, round(100*(can_be_purged/docs.total)) pct_can_be_purged from (select partition_name, count(*) total, sum(decode(PATH_HIGH_CN, null, 1, 0)) tip, sum(decode(PATH_HIGH_CN, null, 0, 1)) non_tip, round(100*(sum(decode(PATH_HIGH_CN, null, 0, 1))/count(*))) percent_non_tip from MDS_PATHS x, MDS_PARTITIONS y where x.PATH_PARTITION_ID = y.PARTITION_ID and path_type = 'DOCUMENT' group by partition_name) docs, (select PARTITION_name, count(label_name) count_labels from MDS_LABELS l, MDS_PARTITIONS p where p.partition_id = l.label_partition_id(+) group by PARTITION_name, partition_id) labels, (SELECT partition_name, count (*) can_be_purged FROM MDS_PATHS path, MDS_TXN_LOCKS, MDS_PARTITIONS p WHERE partition_id = path_partition_id and path_low_cn <= lock_txn_cn AND path_low_cn > 0 AND NOT EXISTS (SELECT label_cn from MDS_LABELS WHERE path.path_low_cn <= label_cn AND (path.path_high_cn > label_cn OR path.path_high_cn IS NULL) AND label_partition_id = path.path_partition_id) AND path_high_cn IS NOT NULL AND path_partition_id = lock_partition_id and path_type = 'DOCUMENT' GROUP BY partition_name ) can_purge WHERE docs.partition_name = labels.partition_name AND docs.partition_name = can_purge.partition_name(+) order by total
Execute purgeMetadata
to cleanup the content. You may need to reclaim the space manually in some cases. See the "Tuning Database Repository" section of the Oracle Fusion Middleware Performance and Tuning Guide.
If large portion of non-tip versions are held by labels, consider removing those (older) labels using purgeMetadataLabels
and re-run purgeMetadata
.
Problem 1
The following error code is seen in the diagnostic.log
file:
MDS-01273: The operation on the resource {0} failed because source metadata store mapped to the namespace {1} is read only.
For example:
ReadOnlyStoreException: MDS-01273: @ The operation on the resource /oracle/apps/cdm/foundation/parties/flex/orgContactNew/test.xsd failed because source @ metadata store mapped to the namespace / BASE DEFAULT is read only Ensure that adf-config.xml contains a mapping that covers the resource and is mapped to DBMEtadataStore
The diagnostic.log
file is stored in the following directories:
(UNIX) DOMAIN_HOME/servers/server_name/logs/server-name-diagnostic.log (Windows) DOMAIN_HOME\servers\server_name\logs\server-name-diagnostic.log
This error occurs because the resource is not mapped in adf-config.xml
. When it is not mapped, it defaults to ClasspathStore
, which is read-only.
Solution 1
To resolve this problem:
Check persistence configuration in adf-config.xml
. Specifically, check the repository pointing to the namespace should have its metadata-store-class
set to oracle.mds.persistence.stores.db.DBMetadataStore
. For example:
<mdsC:adf-mds-config>
<mds-config>
...
<persistence-config>
<metadata-namespaces>
<namespace path="/oracle/apps/cdm/foundation/parties/flex/orgContact" metadata-store-usage="globalStore-by-adfconfigfilter"/>
</metadata-namespaces>
<metadata-store-usages>
<metadata-store-usage id="globalStore-by-adfconfigfilter" default-cust-store="true" deploy-target="true">
<metadata-store class-name="oracle.mds.persistence.stores.db.DBMetadataStore">
<property name="repository-name" value="mds-ApplicationMDSDB"/>
<property name="partition-name" value="FAGlobal"/>
<property name="jndi-datasource" value="jdbc/mds/mds-ApplicationMDSDBDS"/>
</metadata-store>
</metadata-store-usage>
....
</persistence-config>
</mds-config>
</mdsC:adf-mds-config>
In the adf-config.xml
or the customization document, ensure the read-only-mode
element is set to true
.
Problem 2
The following error code is seen in the diagnostic.log
file:
MDS-00013: no metadata found for metadata object "{0}"
For example:
file:oracle.mds.core.MetadataNotFoundException: MDS-00013: no metadata found formetadata object "</a/b/document.xml>
The diagnostic.log
file is stored in the following directories:
(UNIX) DOMAIN_HOME/servers/server_name/logs/server-name-diagnostic.log (Windows) DOMAIN_HOME\servers\server_name\logs\server-name-diagnostic.log
Solution 2
To resolve this problem:
Check if the namespace to store mappings in adf-config.xml
mapping this MO to an Oracle Database store. For example:
<mdsC:adf-mds-config>
<mds-config>
...
<persistence-config>
<metadata-namespaces>
<namespace path="/oracle/apps/cdm/foundation/parties/flex/orgContact" metadata-store-usage="globalStore-by-adfconfigfilter"/>
</metadata-namespaces>
<metadata-store-usages>
<metadata-store-usage id="globalStore-by-adfconfigfilter" default-cust-store="true" deploy-target="true">
<metadata-store class-name="oracle.mds.persistence.stores.db.DBMetadataStore">
<property name="repository-name" value="mds-ApplicationMDSDB"/>
<property name="partition-name" value="FAGlobal"/>
<property name="jndi-datasource" value="jdbc/mds/mds-ApplicationMDSDBDS"/>
</metadata-store>
</metadata-store-usage>
....
</persistence-config>
</mds-config>
</mdsC:adf-mds-config>
Based on the mapping, perform the appropriate action
If it is mapped to a database store, use the exportMetadata
command to check if the metadata document is present in the MDS Repository partition. For more information about this command, see the "exportMetadata" section of the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
If it is not mapped, then the document must be in classpath. You may need to bounce the server.
Set TRACE
log level for logger oracle.mds.mnfe.
See Section 9.2.3.
Problem 3
The following error code is seen in the diagnostic.log
file:
MDS-01330: unable to load MDS configuration document
Typically, there will be nested exceptions that describe the actual reason for failure. For example:
MDSConfigurationException : parseADFConfigurationMDS-0133
This error occurs because the application configuration has an Oracle Metadata Repository repository stub, but Oracle Metadata Repository repository is not specified at deployment time.
This error can also occur when the repository is configured incorrectly or if the Oracle Database is down. For example:
MDSConfigurationException encountered in parseADFConfigurationMDS-01330: unable to load MDS configuration document MDS-01329: unable to load element "persistence-config" MDS-01370: MetadataStore configuration for metadata-store-usage "OWSM_TargetRepos" is invalid. MDS-01377: Unable to get database connection from data source configured with JNDI name "jdbc/mds/owsm". weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection
The diagnostic.log
file is stored in the following directories:
(UNIX) DOMAIN_HOME/servers/server_name/logs/server-name-diagnostic.log (Windows) DOMAIN_HOME\servers\server_name\logs\server-name-diagnostic.log
Solution 3
To resolve this problem, from Fusion Applications Control:
Check the MDS Configuration page for the application.
Fix configuration and re-deploy the application.
See the "Changing MDS Configuration Attributes for Deployed Applications" section in the Oracle Fusion Middleware Administrator's Guide.
Problem 4
During runtime, an exception stack displays with a concurrent modification exception:
java.util.ConcurrentModificationException
This error occurs during runtime, when working with objects using JEDI or customizing using Page Composer. Essentially, the same document at the same layer is being customized by multiple users.
Solution 4
Resolve the concurrency.