17 Troubleshoot Oracle Metadata Services Repository

The Oracle WebLogic Scripting Tool (WLST) is a command-line scripting environment that can be used to create, manage, and monitor Oracle WebLogic Server domains.

You can manage Oracle Metadata Service (MDS) Repository by using the MDS Repository commands described in the Metadata Services (MDS) Custom WLST Commands section of the WLST Command Reference for WebLogic Server.

To manage application metadata in the MDS repository, perform the following steps:

To modify configuration and runtime MBean properties, perform the following steps:

• Modify the AutoPurgeTimeToLive and MaximumCacheSize configuration MBean property in Fusion Applications Control. See the Changing MDS Configuration Attributes for Deployed Applications section in the Administering Oracle Fusion Middleware.

• Invoke the clearCache runtime MBean operation in Fusion Applications Control. If it seems that metadata updates are not being picked up, invoke this operation to see if it is related to caching. Note that the use of this operation has a short-term performance impact while the cache is re-populated.

  To invoke the clearCach runtime MBean operation in Fusion Applications Control, perform the following steps:

  1. From the navigation pane, expand the farm, then expand Application Deployments.

  2. Click the application that uses MDS Repository.

  3. In the Java EE Application home page, from the Application Deployment menu, choose MDS Configuration.

     The MDS Configuration page appears.

  4. In the Advanced Configuration section, click Runtime MBean Browser link.

  5. In the System MBean Browser page, expand Application Defined MBeans, oracle.mds.lcm, Server: Managed Server, Application: Application Name, MDSAppRuntime, and MDSAppRuntime.

     The System MBean Browser displays the Application Defined MBeans: MDSAppRuntime:MDSAppRuntime page.

  6. Click the Operations tab, as shown in the following figure.

     Figure 17-1 Expansion of MDSAppRuntime in the System MBean Browser

     
     
     

  7. Click the clearCache.

  8. Click Invoke.

To modify logging attributes, use WebLogic Script Tools (WLST) commands. See Fusion Middleware WebLogic Scripting Tool Command Reference. Alternatively, the logs can be viewed and the logger levels can be changed.

To view the logs, perform the following steps:

1. Right-click the domain under WebLogic Domain, then choose View Log Messages.

2. From the navigation pane, expand the farm, then expand WebLogic Domain, and Managed_Server.

3. Choose Logs, then View Log Messages.

If the problem still cannot be solved, increase the log level of the system to debug the transactions.

To simplify troubleshooting, it is recommended to enable the loggers listed in the following steps at the TRACE:32 (FINEST) level. (See exception below related to oracle.mds).

To change logger levels, use the setLogLevel WLST command or perform the following steps:

1. From the navigation pane, expand the farm, then expand WebLogic Domain, and then Managed_Server.

2. Choose Logs, and then Log Configuration.

3. In the Logger Name column, expand the oracle, and then oracle.mds to display the loggers. Typically oracle.mds is set at FINER for initial investigation to get an overall view. Multiple loggers can be enabled and one or more of the FINEST loggers can subsequently be added to drill down into more detail.

   Warning: Setting oracle.mds to FINEST (rather than FINER) will enable a huge amount of tracing as it effectively switches on all of the oracle.mds. loggers at FINEST.

4. In the Oracle Diagnostic Logging Level (Java Level) column, enable tracing for the following loggers, depending on the functional area to be investigated. For example, change the logging level to TRACE:32 for the following loggers:

   • oracle.mds.query - for detailed diagnostics for SQL statements, bind variables, and timing information about MDS database queries.

   • oracle.mds.jsp - for trace JSP integration with MDS.

   • oracle.mds.dbstore - for detailed logging about all operations done at the persistence level for the database-based MDS repository.

   • oracle.mds.sandbox - for debug level information for all the sandbox operations. Use this logger when troubleshooting creation of a sandbox, managing the sandbox metadata, applying a sandbox or any other sandbox lifecycle-related operations. Use this logger in conjunction with the oracle.mds.dbstore logger.

   • oracle.mds.coretxn - for detailed logging for MDSTransaction methods. MDSTransaction is an internal class which is associated with an MDSSession that is being used to make changes to metadata. Enabling this logger includes diagnostics to help verification that the same MDSSession is not being used by multiple threads, which while not illegal is unusual and can cause problems if done incorrectly.

   • oracle.mds.pdcache - to trace document cache access and invalidation. Shows labels being used for lookups and versions being returned from the cache.

   • oracle.mds.remotechange - for detailed tracing on polling thread, synchronous polling on flushChanges() and optional distributed notifications feature (not enabled for Fusion Applications).

   • oracle.mds.corecache - to trace core cache access and invalidation.

   • oracle.mds.rename - for detailed diagnostics for listing dependency references during rename operation.

   • oracle.mds.custupdate - for detailed diagnostics for change events handling and optimization, and creation of MDS customization instructions during update of MDS customizations.

   • oracle.mds.custmerge - for detailed diagnostics when applying MDS customization instructions.

   • oracle.mds.mnfe - for tracing of MetadataNotFoundException and isStale() evaluation.

   The change should take effect within a few minutes. In a production system, setting the trace at a fine-grained level can result in a large amount of output that must be diagnosed. Alternatively, use selective tracing that provides a way to get a detailed, on-disk trace selectively (for example, by user name, thereby eliminating trace output for other users).

5. To activate selective tracing, right-click the domain under WebLogic Domain, then choose Logs, and then Selective Tracing.

   Note that Selecting Tracing does not display as an option when you right-click an Administration Server or Managed Server and choose Logs.

6. From the Option Name list, choose the type of selective trace (for example, based on user name), and then start the trace.

7. When the problem has been reproduced, disable the trace and view the output to narrow down the issue.

   For more information on selective tracing, see the Configuring and Using Selective Tracing section of Administering Oracle Fusion Middleware.

8. Review the error logs (from Fusion Applications Control) for more information on the error.

   Cross layer, server, and family functionality can be correlated through the execution context ID (ECID) (for example, look up the composite instance for a given expense report by correlating all the log entries with the ECID associated with that expense report transaction). For more information, see the Correlating Messages Across Log Files and Components section of Administering Oracle Fusion MiddlewareGuide.

To check the MDS Repository performance and adjust the cache size, perform the following steps:

1. Check performance metrics for cache performance.

   1. Navigate to the application's home page and expand the farm, then Application Deployments, and then select an application.

      The application's home page appears.

   2. From the Application Deployment menu, choose Performance Summary.

      The Performance Summary page appears.

   3. Click Show Metric Palette.

   4. Expand MDS Metrics and select IOs Per Metadata Object Get and IOs Per MO Content Get.

   5. Monitor the value of these metrics. A value of 0 is ideal since it indicates that all requests are being serviced from the cache.

      Increasing the cache size may help reduce these metrics. However, a value of 0 may not be achievable in many situations due to the following reasons:

      • Metadata updates

      • User customizations that are only cached for the duration of the user session at most

      • IO operations that were not due to calls from the Metadata Object APIs

2. Modify the MaximumCacheSize configuration MBean property, gradually increasing it in size to see if that improves performance. See the Changing MDS Configuration Attributes for Deployed Applications section in the Administering Oracle Fusion Middleware.

To resolve this problem from the MDS Repository database, perform the following steps:

1. Capture an Automatic Workload Repository (AWR) report. See the Gathering Database Statistics Using the Automatic Workload Repository section in the Oracle Database 2 Day + Performance Tuning Guide.

   Check if any specific MDS SQL is taking too long.

2. Regather the Oracle Database statistics for each MDS schema by executing the GATHER_SCHEMA_STATS procedure in the DBMS_STATS PL/SQL package from SQL, as a privileged database user, such as SYS.

   execute dbms_stats.gather_schema_stats(
   ownname =>'schemaOwner',
   options => 'Gather Auto');
   

   Replace <schemaOwner> with the name of the schema, for example FUSION_MDS. Also, place the entire command in a single line at the time of execution.

3. Analyze the DMS data from DMSSpy.

4. If there is continuous growth in metadata in Oracle Database tables, it could be due to the purge not being executed or labels are preventing metadata from being purged.

   1. Use the following queries to identify the cause of the problem:

      1. Determine how many versions can be purged in partitions. In the following examples, :B1 is a bind variable.

      2. Define the variable as the appropriate partition name. For example, Oracle ADF metadata is stored in the FAGlobal partition and Oracle SOA Suite metadata is stored in the soa_infra partition.

         SELECT count( * ) FROM mds_paths path, mds_txn_locks, mds_partitions WHERE path_low_cn <= lock_txn_cn AND path_low_cn > 0
         AND NOT EXISTS (SELECT label_cn from mds_labels
                         WHERE path.path_low_cn <= label_cn
                         AND (path.path_high_cn > label_cn OR path.path_high_cn IS NULL)
                         AND label_partition_id = path.path_partition_id)        
                  AND path_high_cn IS NOT NULL     
                  AND path_partition_id = lock_partition_id
                  and path_partition_id = partition_id        
                  AND partition_name =:B1
         

      3. To determine how many document versions in the repository are held by labels:

         select label_name, label_cn, (SELECT count( * )
         FROM mds_paths path, mds_partitions
         WHERE path_high_cn IS NOT NULL
         and path_low_cn <= label_cn
         AND path_low_cn > 0
         and path_high_cn > label_cn
         AND path_partition_id = partition_id
         and partition_name=:B1) versHeld, txn_time from mds_labels, mds_transactions, mds_partitions
         where label_partition_id=txn_partition_id
           and
         label_cn=txn_cn
         and label_partition_id=partition_id
         and partition_name=:B1 order by versHeld desc
         

      4. To determine partition-wise statistics of tip and non-tip, purgeable versions:

         select docs.*, labels.count_labels, can_be_purged,
                round(100*(can_be_purged/docs.total)) pct_can_be_purged from
           (select partition_name, count(*) total, sum(decode(PATH_HIGH_CN, null, 1, 0)) tip,
            sum(decode(PATH_HIGH_CN, null, 0, 1)) non_tip,
            round(100*(sum(decode(PATH_HIGH_CN, null, 0, 1))/count(*))) percent_non_tip
            from MDS_PATHS x, MDS_PARTITIONS y
            where x.PATH_PARTITION_ID = y.PARTITION_ID
            and path_type = 'DOCUMENT'
            group by partition_name)
         docs,
           (select PARTITION_name, count(label_name) count_labels
            from MDS_LABELS l, MDS_PARTITIONS p
            where p.partition_id = l.label_partition_id(+)
            group by PARTITION_name, partition_id)
         labels,
           (SELECT partition_name, count (*) can_be_purged
           FROM MDS_PATHS path, MDS_TXN_LOCKS, MDS_PARTITIONS p
           WHERE partition_id = path_partition_id
           and path_low_cn <= lock_txn_cn AND path_low_cn > 0
           AND NOT EXISTS (SELECT label_cn from MDS_LABELS
                           WHERE path.path_low_cn <= label_cn
                           AND (path.path_high_cn > label_cn OR path.path_high_cn IS NULL)
                           AND label_partition_id = path.path_partition_id)
           AND path_high_cn IS NOT NULL
           AND path_partition_id = lock_partition_id
           and path_type = 'DOCUMENT'
           GROUP BY partition_name )
         can_purge
         WHERE docs.partition_name = labels.partition_name
         AND   docs.partition_name = can_purge.partition_name(+)
         order by total
         

   2. Execute the purgeMetadata WLST command to clean up the content. In some cases, it is necessary to reclaim the space manually. See the Tuning the Database Repository section of the Tuning Performance.

   3. If a large portion of non-tip versions are held by labels, consider removing those (older) labels using Fusion Middleware Control.

      If a label on which a sandbox is based is deleted, the sandbox becomes unusable, so using the purgeMetadataLabels WLST command can be risky. However, the Manage Labels page in Fusion Middleware Control automatically hides sandbox labels to avoid accidentally deleting sandbox labels. To delete a label that contains no sandboxes:

      1. Expand the farm, then Metadata Repositories.

      2. Select the repository.

      3. On the repository home page, select a partition, and then click Manage Labels.

      4. On the Manage Labels page, search for a label.

      5. By default, labels associated with sandboxes and deployed applications are not shown. To display those labels, select Sandboxes or Deployment or both. Fusion Middleware Control will not let a label associated with a sandbox be deleted.

      6. Select the label, and then click Delete Selected.

      7. In the confirmation box, click OK.