8 Troubleshooting the Oracle Metadata Repository

This chapter describes common problems that you might encounter when using Oracle Metadata Repository (MDS Repository) and explains how to solve them. This chapter contains the following topics:

• Introduction to Troubleshooting MDS Repository

• Getting Started with Troubleshooting and Logging Basics for Oracle Metadata Repository

• Applications Performing Slowly Due to MDS Repository

• Diagnosing Exceptions with Oracle Metadata Repository

Some procedures in this chapter reference content in the Oracle Fusion Middleware guides. These guides describe using Fusion Middleware Control. These procedures also apply to Fusion Applications Control.

8.1 Introduction to Troubleshooting MDS Repository

This section provides guidelines and a process for using the information in this chapter. Using the following guidelines and process will focus and minimize the time you spend resolving problems.

Guidelines

When using the information in this chapter, Oracle recommends:

• After performing any of the solution procedures in this chapter, immediately retrying the failed task that led you to this troubleshooting information. If the task still fails when you retry it, perform a different solution procedure in this chapter and then try the failed task again. Repeat this process until you resolve the problem.

• Making notes about the solution procedures you perform, symptoms you see, and data you collect while troubleshooting. If you cannot resolve the problem using the information in this chapter and you must log a service request, the notes you make will expedite the process of solving the problem.

Process

Follow the process outlined in Table 8-1 when using the information in this chapter. If the information in a particular section does not resolve your problem, proceed to the next step in this process.

Table 8-1 Process for Using the Information in this Chapter

Step	Section to Use	Purpose
1	Section 8.2	Get started troubleshooting the MDS Repository. The procedure in this section quickly addresses a wide variety of problems.
2	Section 8.3 through Section 8.4	Perform problem-specific troubleshooting procedures. These sections describe: Possible causes of the problems Solution procedures corresponding to each of the possible causes
3	Section 13.1	Use My Oracle Support to get additional troubleshooting information about Oracle Fusion Applications or Oracle SOA Suite. My Oracle Support provides access to several useful troubleshooting resources, including Knowledge Base articles and Community Forums and Discussions.
4	Section 13.1	Log a service request if the information in this chapter and My Oracle Support does not resolve your problem. You can log a service request using My Oracle Support at https://support.oracle.com.

8.2 Getting Started with Troubleshooting and Logging Basics for Oracle Metadata Repository

For general instructions on managing the metadata for Oracle Fusion Middleware components in the Oracle Metadata Repository, see the "Managing the Metadata Repository" chapter in the Oracle Fusion Middleware Administrator's Guide.

Use the following tools to troubleshoot Oracle Metadata Repository issues:

• WLST Commands

• MBean Properties in Fusion Applications Control

• Logging

8.2.1 WLST Commands

Use the MDS Repository commands described in the "Metadata Services (MDS) Custom WLST Commands" section of the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.

Use the following script from the Oracle WebLogic Server domain home directory, which is based on WLST commands:

1. From the fusionapps Middleware subdirectory, start the WLST:

   (UNIX) FA_MW_HOME/oracle_common/common/bin/wlst.sh
   (Windows) FA_MW_HOME\oracle_common\common/bin\wlst.cmd
   

   where DOMAIN_HOME is located in the following locations:

   (UNIX) APPLICATIONS_CONFIG/instance/domains/host/domain_name
   (Windows) APPLICATIONS_CONFIG\instance\domains\host\domain_name
   

2. Connect to Oracle WebLogic Server.

3. Use WLST commands for the MDS Repository. For example:

   • deleteMetadata, exportMetdata, importMetadata, and purgeMetadata to manage application metadata

   • registerMetadataDBRepository to register a database-based MDS repository.

   • importMAR to import an application MAR

     Use the force=true argument to force the import of all document. If you do not use the with force argument, only documents that have changed will be imported.

   • createMetadataLabel, deleteMetadataLabel, listMetadataLabels, promoteMetadataLabel, and purgeMetadataLabels to manage labels

8.2.2 MBean Properties in Fusion Applications Control

Modify configuration and runtime MBean properties:

• Modify the AutoPurgeTimeToLive and MaximumCacheSize configuration MBean property in Fusion Applications Control. See the "Changing MDS Configuration Attributes for Deployed Applications" section in the Oracle Fusion Middleware Administrator's Guide.

• Modify the clearCache runtime MBean property in Fusion Applications Control:

  1. From the navigation pane, expand the farm and then Application Deployments.

  2. Click the application that uses MDS Repository.

  3. In the Java EE Application home page, from the Application Deployment menu, choose MDS Configuration.

     The MDS Configuration page displays the page.

  4. In the Advanced Configuration section, click Runtime MBean Browser.

  5. In the System MBean Browser page, expand Application Defined MBeans.

     The System MBean Browser displays Application Defined MBeans > oracle.mds.lcm > Server: Managed Server > Application: Application Name > MDSAppRuntime > MDSAppRuntime and displays the Application Defined MBeans: MDSAppRuntime:MDSAppRuntime page.

  6. Click the Operations tab.

     
     Description of the illustration mdsmbean.gif
     
     

  7. Click the clearCache.

  8. Click Invoke.

8.2.3 Logging

View the logs. Right-click the domain under WebLogic Domain and choose View Log Messages. For more information, see the "Viewing Log Files and Their Messages Using Fusion Middleware Control" chapter in the Oracle Fusion Middleware Administrator's Guide.

1. From the navigation pane, expand the farm > WebLogic Domain > Managed_Server.

2. Choose Logs > View Log Messages.

If the problem still cannot be solved, increase the log level of the system to debug the transactions. To simplify troubleshooting, it is recommended that you enable the following parent loggers at the TRACE32 (FINEST) level.

To change logger levels, perform the following steps:

1. From the navigation pane, expand the farm > WebLogic Domain > Managed_Server.

2. Choose Logs > Log Configuration.

3. In the Logger Name column, expand the oracle and then oracle.mds to display the loggers.

4. In the Oracle Diagnostic Logging Level (Java Level) column, change the logging level to TRACE:32.

   - oracle.mds.query for detailed diagnostics for SQL statements, bind variables, and timing information about MDS database queries

   - oracle.mds.custmerge for detailed diagnostics for SQL statements, bind variables, and timing information about MDS database queries

   - oracle.mds.jsp for trace JSP integration with MDS

   - oracle.mds.dbstore for detailed logging about all operations done at the persistence level for the database-based MDS repository

   - oracle.mds.sandbox for debug level information for all the sandbox operations.Use this logger when troubleshooting creation of a sandbox, managing the sandbox metadata, applying a sandbox or any other sandbox lifecycle-related operations. Use this logger in conjunction with the oracle.mds.dbstore tracer logger.

   - oracle.mds.coretxn for detailed logging for MDSTransaction methods. MDSTransaction is an internal class which is associated with an MDSSession that is being used to make changes to metadata. Enabling this tracer includes diagnostics to help verification that the same MDSSession is not being used by multiple threads which while not illegal is unusual and can cause problems if done incorrectly.

   - oracle.mds.pdcache for trace-document cache access and invalidation. Shows labels being used for lookups and versions being returned from the cache.

   - oracle.mds.remotechange for detailed tracing for remote notifications, polling thread and synchronous polling on flushChanges(). This logger can be useful to check that remote notifications, for example, configuration is correct and changes are being distributed.

   - oracle.mds.corecache to trace core cache access and invalidation.

   - oracle.mds.jsf for MDSFaceletsHelper diagnostics for Facelets integration with MDS.

   - oracle.mds.rename for detailed diagnostics for listing dependency references during rename operation.

   - oracle.mds.custupdate for detailed diagnostics for change events handling and optimization, and creation of MDS customization instructions during update of MDS customizations.

   - oracle.mds.custmerge for detailed diagnostics when applying MDS customization instructions

   - oracle.mds.mnfe for tracing of MetadataNotFoundException and isStale() evaluation.

   The change should take effect within a few minutes. Note that in a production system, setting the trace at a fine-grained level can result in a large amount of output that must be diagnosed. You can alternately use selective tracing that provides a way to get a detailed, on-disk trace selectively (for example, by user name, thereby eliminating trace output for other users).

5. To activate selective tracing, right-click the domain under WebLogic Domain and choose Logs > Selective Tracing.

   Note that Selecting Tracing does not display as an option when you right-click an Administration Server or Managed Server and choose Logs.

6. From the Option Name list, choose the type of selective trace (for example, based on user name), and start the trace.

7. When the problem has been reproduced, disable the trace and view the output to narrow down the issue.

   For more information on selective tracing, see the "Configuring and Using Selective Tracing" section of Oracle Fusion Middleware Administrator's Guide.

8. Review the error logs (from Fusion Applications Control) for more information on the error.

   Cross layer, server, and family functionality can be correlated through the execution context ID (ECID) (for example, you can look up the composite instance for a given expense report by correlating all the log entries with the ECID associated with that expense report transaction). For more information, see the "Correlating Messages Across Log Files and Components" section of Oracle Fusion Middleware Administrator's Guide.

8.3 Applications Performing Slowly Due to MDS Repository

Problem

If the application is performing slowly, there could be an issue with the MDS Repository.

Solution 1

To check the MDS Repository performance and adjust the cache size:

1. Check performance metrics for cache performance.

   1. Navigate to the application's home page by expanding the farm, then Application Deployments. Then, select an application.

      The application's home page is displayed.

   2. From the Application Deployment menu, choose Performance Summary.

      The Performance Summary page displays.

   3. Click Show Metric Palette.

   4. Expand MDS Metrics and select IOs Per Metadata Object Get and IOs Per MO Content Get.

   5. Monitor the performance of these metrics. Typically, values close to 1 indicate poor cache performance.

2. Modify the MaximumCacheSize configuration MBean property to -1. See the "Changing MDS Configuration Attributes for Deployed Applications" section in the Oracle Fusion Middleware Administrator's Guide.

Solution 2

To resolve this problem from the MDS Repository database:

1. Capture an Automatic Workload Repository (AWR) report. See the "Gathering Database Statistics Using the Automatic Workload Repository" section in the Oracle Database 2 Day + Performance Tuning Guide.

2. Check if any specific MDS SQL is taking too long. Analyze log (if needed with oracle.mds logger set to FINER) an or DMS data from DMSSpy.

3. Regather the Oracle Database statistics by executing the GATHER_SCHEMA_STATS procedure in the DBMS_STATS PL/SQL package from SQL as follows:

   EXEC DBMS_STATS.GATHER_SCHEMA_STATS(
   ownname => schema_owner, 
   estimate_ percent => dbms_stats.auto_sample_size,  
   method_opt => 'for all columns size auto', 
   cascade => true);
   

   EXEC DBMS_STATS.GATHER_TABLE_STATS(
   ownname =>'schema_Owner',
   tabname =>'MDS_PATHS',
   method_opt =>'FOR COLUMNS SIZE 1 PATH_FULLNAME');
   

   where schema_owner is the name of the schema, typically, FUSION_MDS.

   For information about the GATHER_SCHEMA_STATS procedure, see "GATHER_SCHEMA_STATS Procedures" in the Oracle Database PL/SQL Packages and Types Reference.

4. If performance does not improve after collecting statistics, then flush the shared pool to clear the execution plan for the database and generate a new query plan with the following SQL command:

   ALTER SYSTEM FLUSH SHARED_POOL;
   ALTER SYSTEM FLUSH BUFFER_CACHE;
   

5. If you notice continuous growth in metadata in Oracle Database tables, it could be the purge not executed or labels are preventing metadata from being purged.

   1. Use these queries to identify which of the above is the cause.

      To determine how many versions can be purged in soa-infra:

      SELECT count( * ) FROM mds_paths path, mds_txn_locks, mds_partitions WHERE path_low_cn <= lock_txn_cn AND path_low_cn > 0
      AND NOT EXISTS (SELECT label_cn from mds_labels
                      WHERE path.path_low_cn <= label_cn
                      AND (path.path_high_cn > label_cn OR path.path_high_cn IS NULL)
                      AND label_partition_id = path.path_partition_id)        
               AND path_high_cn IS NOT NULL     
               AND path_partition_id = lock_partition_id
               and path_partition_id = partition_id        
               AND partition_name = 'soa-infra'
      

      To determine how many versions are held by labels:

      select label_name, label_cn, (SELECT count( * )
      FROM mds_paths path, mds_partitions
      WHERE path_high_cn IS NOT NULL
      and path_low_cn <= label_cn
      AND path_low_cn > 0
      and path_high_cn > label_cn
      AND path_partition_id = partition_id
      and partition_name=:B1) versHeld, txn_time from mds_labels, mds_transactions, mds_partitions
      where label_partition_id=txn_partition_id
        and
      label_cn=txn_cn
      and label_partition_id=partition_id
      and partition_name=:B1 order by versHeld desc
      

      To determine partition-wise statistics of tip and non-tip, purgeable versions:

      select docs.*, labels.count_labels, can_be_purged,
             round(100*(can_be_purged/docs.total)) pct_can_be_purged from
        (select partition_name, count(*) total, sum(decode(PATH_HIGH_CN, null, 1, 0)) tip,
         sum(decode(PATH_HIGH_CN, null, 0, 1)) non_tip,
         round(100*(sum(decode(PATH_HIGH_CN, null, 0, 1))/count(*))) percent_non_tip
         from MDS_PATHS x, MDS_PARTITIONS y
         where x.PATH_PARTITION_ID = y.PARTITION_ID
         and path_type = 'DOCUMENT'
         group by partition_name)
      docs,
        (select PARTITION_name, count(label_name) count_labels
         from MDS_LABELS l, MDS_PARTITIONS p
         where p.partition_id = l.label_partition_id(+)
         group by PARTITION_name, partition_id)
      labels,
        (SELECT partition_name, count (*) can_be_purged
        FROM MDS_PATHS path, MDS_TXN_LOCKS, MDS_PARTITIONS p
        WHERE partition_id = path_partition_id
        and path_low_cn <= lock_txn_cn AND path_low_cn > 0
        AND NOT EXISTS (SELECT label_cn from MDS_LABELS
                        WHERE path.path_low_cn <= label_cn
                        AND (path.path_high_cn > label_cn OR path.path_high_cn IS NULL)
                        AND label_partition_id = path.path_partition_id)
        AND path_high_cn IS NOT NULL
        AND path_partition_id = lock_partition_id
        and path_type = 'DOCUMENT'
        GROUP BY partition_name )
      can_purge
      WHERE docs.partition_name = labels.partition_name
      AND   docs.partition_name = can_purge.partition_name(+)
      order by total
      

   2. Execute purgeMetadata  to cleanup the content. You may need to reclaim the space manually in some cases. See the "Tuning Database Repository" section of the Oracle Fusion Middleware Performance and Tuning Guide.

   3. If large portion of non-tip versions are held by labels, consider removing those (older) labels using purgeMetadataLabels and re-run purgeMetadata.

8.4 Diagnosing Exceptions with Oracle Metadata Repository

Problem 1

The following error code is seen in the diagnostic.log file:

MDS-01273: The operation on the resource {0} failed because source metadata store mapped to the namespace {1} is read only.

For example:

ReadOnlyStoreException: MDS-01273:
@ The operation on the resource /oracle/apps/cdm/foundation/parties/flex/orgContactNew/test.xsd failed because source
@ metadata store mapped to the namespace / BASE DEFAULT is read only
Ensure that adf-config.xml contains a mapping that covers the resource and is mapped to DBMEtadataStore

The diagnostic.log file is stored in the following directories:

(UNIX) DOMAIN_HOME/servers/server_name/logs/server-name-diagnostic.log
(Windows) DOMAIN_HOME\servers\server_name\logs\server-name-diagnostic.log

This error occurs because the resource is not mapped in adf-config.xml. When it is not mapped, it defaults to ClasspathStore, which is read-only.

Solution 1

To resolve this problem:

1. Check persistence configuration in adf-config.xml. Specifically, check the repository pointing to the namespace should have its metadata-store-class set to oracle.mds.persistence.stores.db.DBMetadataStore. For example:

   <mdsC:adf-mds-config>
   <mds-config>
   ...
   <persistence-config>
   <metadata-namespaces>
   <namespace path="/oracle/apps/cdm/foundation/parties/flex/orgContact" metadata-store-usage="globalStore-by-adfconfigfilter"/>
   </metadata-namespaces>
   <metadata-store-usages>
   <metadata-store-usage id="globalStore-by-adfconfigfilter" default-cust-store="true" deploy-target="true">
   <metadata-store class-name="oracle.mds.persistence.stores.db.DBMetadataStore">
   <property name="repository-name" value="mds-ApplicationMDSDB"/>
   <property name="partition-name" value="FAGlobal"/>
   <property name="jndi-datasource" value="jdbc/mds/mds-ApplicationMDSDBDS"/>
   </metadata-store>
   </metadata-store-usage>
   ....
   </persistence-config>
   </mds-config>
   </mdsC:adf-mds-config>
   

2. In the adf-config.xml or the customization document, ensure the read-only-mode element is set to true.

Problem 2

The following error code is seen in the diagnostic.log file:

MDS-00013: no metadata found for metadata object "{0}"

For example:

file:oracle.mds.core.MetadataNotFoundException: MDS-00013: no metadata found formetadata object "</a/b/document.xml>

The diagnostic.log file is stored in the following directories:

(UNIX) DOMAIN_HOME/servers/server_name/logs/server-name-diagnostic.log
(Windows) DOMAIN_HOME\servers\server_name\logs\server-name-diagnostic.log

Solution 2

To resolve this problem:

1. Check if the namespace to store mappings in adf-config.xml mapping this MO to an Oracle Database store. For example:

   <mdsC:adf-mds-config>
   <mds-config>
   ...
   <persistence-config>
   <metadata-namespaces>
   <namespace path="/oracle/apps/cdm/foundation/parties/flex/orgContact" metadata-store-usage="globalStore-by-adfconfigfilter"/>
   </metadata-namespaces>
   <metadata-store-usages>
   <metadata-store-usage id="globalStore-by-adfconfigfilter" default-cust-store="true" deploy-target="true">
   <metadata-store class-name="oracle.mds.persistence.stores.db.DBMetadataStore">
   <property name="repository-name" value="mds-ApplicationMDSDB"/>
   <property name="partition-name" value="FAGlobal"/>
   <property name="jndi-datasource" value="jdbc/mds/mds-ApplicationMDSDBDS"/>
   </metadata-store>
   </metadata-store-usage>
   ....
   </persistence-config>
   </mds-config>
   </mdsC:adf-mds-config>
   

2. Based on the mapping, perform the appropriate action

   • If it is mapped to a database store, use the exportMetadata command to check if the metadata document is present in the MDS Repository partition. For more information about this command, see the "exportMetadata" section of the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.

   • If it is not mapped, then the document must be in classpath. You may need to bounce the server.

3. Set TRACE log level for logger oracle.mds.mnfe. See Section 8.2.3.

Problem 3

The following error code is seen in the diagnostic.log file:

MDS-01330: unable to load MDS configuration document

Typically, there will be nested exceptions that describe the actual reason for failure. For example:

MDSConfigurationException : parseADFConfigurationMDS-0133

This error occurs because the application configuration has an Oracle Metadata Repository repository stub, but Oracle Metadata Repository repository is not specified at deployment time.

This error can also occur when the repository is configured incorrectly or if the Oracle Database is down. For example:

MDSConfigurationException encountered in parseADFConfigurationMDS-01330: unable to
load MDS configuration document
MDS-01329: unable to load element "persistence-config"
MDS-01370: MetadataStore configuration for metadata-store-usage "OWSM_TargetRepos" is invalid.
MDS-01377: Unable to get database connection from data source configured with JNDI name "jdbc/mds/owsm".
weblogic.common.resourcepool.ResourceDeadException:
0:weblogic.common.ResourceException: Could not create pool connection

The diagnostic.log file is stored in the following directories:

(UNIX) DOMAIN_HOME/servers/server_name/logs/server-name-diagnostic.log
(Windows) DOMAIN_HOME\servers\server_name\logs\server-name-diagnostic.log

Solution 3

To resolve this problem, from Fusion Applications Control:

1. Check the MDS Configuration page for the application.

2. Fix configuration and re-deploy the application.

See the "Changing MDS Configuration Attributes for Deployed Applications" section in the Oracle Fusion Middleware Administrator's Guide.

Problem 4

During runtime, an exception stack displays with a concurrent modification exception:

java.util.ConcurrentModificationException

This error occurs during runtime, when working with objects using JEDI or customizing using Page Composer. Essentially, the same document at the same layer is being customized by multiple users.

Solution 4

Resolve the concurrency.