Skip Headers
Oracle® Content Services Administrator's Guide
10g Release 1 (10.1.1)

Part Number B14493-01
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Master Index
Master Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
View PDF

10 Oracle Content Services Maintenance and Tuning

This chapter provides important information about ongoing system maintenance, performance tuning, and recovery. As with any production system, your implementation of Oracle Content Services should include a basic disaster recovery plan.

This chapter includes the following topics:

Backup and Recovery

Planning for failures is one of the most important jobs of any system administrator or DBA. Be sure to implement a daily or weekly backup plan that meets the needs of your business and operations environment. Take advantage of the Oracle database backup capabilities, built right into the database.

Always back up the system before upgrading, migrating new data, or making other major changes. See Oracle Collaboration Suite Administrator's Guide, as well as Oracle Database Backup and Recovery Basics, for additional information.

Note:

In addition to the Oracle Content Services schema, there are two "special" schemas that ensure secure connectivity to other systems. When you back up your system, make sure to include these schemas.

The special schema names are derived from the Oracle Content Services schema name. For example, if the Oracle Content Services schema name is CONTENT, the additional schemas are CONTENT$CM and CONTENT$ID.

Service Configurations and Java Memory Sizing

In Oracle Content Services, the default service configurations specify the maximum number of sessions which can connect to the service. This is to reduce the likelihood of experiencing java.lang.OutOfMemory errors in OC4J_Content.default_island.1 or in application.log.

In previous releases, the default service configurations allowed an unlimited number of sessions. Due to this change, you may now see the following errors:

If you see either of these errors, change the service configuration from Small to Medium or from Medium to Large, or create your own custom service configuration. If you use the Large service configuration, or if you create your own custom service configuration, you must adjust your -Xmx setting.

If you see java.lang.OutOfMemory errors in your OC4J_Content.default_island.1 or application.log files, then you also need to adjust your -Xmx setting.

Table 10-1 describes factors that might require you to change the -Xmx setting.

Table 10-1 Xmx Settings

Service Configuration Setting for IFS.SERVICE. Maximum ConcurrentSessions Expected PCCU Recommended size for Xmx (Java maximum memory) Need to change the default Xmx setting of 256MB?

Small

40

25

64 MB

No

Medium

70

45

162 MB

No

Large

200

125

430 MB

Yes


Note:

The term PCCU refers to Peak Concurrent Connected Users. PCCU is the number of users who are signed in to Oracle Content Services and have performed an operation during the peak hour of the day. If you do not know how many users that is likely to be, assume 10% of your entire Oracle Content Services named user population.

See "Managing Service Configurations" for additional information about creating and changing service configurations.

Calculating Xmx Settings

A general guideline for calculating the Xmx setting is:

Xmx = PCCU * 2.8MB

Alternatively, you can use the following equation to determine a more precise value:

Xmx = (PCCU * 1.6 sessions per PCCU * 1MB per session) + (DATACACHE.Size * 3KB per data cache object) + (8MB + (CONNECTIONPOOL.READONLY.MaximumSize + CONNECTIONPOOL.WRITEABLE.MaximumSize)) + (20% JVM overhead for garbage collection)

The maximum value for the Xmx depends on your operating system. On Linux operating systems, the setting cannot exceed 2GB. On Solaris operating systems, the setting cannot exceed 4GB. Oracle recommends that the Xmx setting should not exceed 2GB for Oracle Content Services.

See "Modifying Node Configurations" for more information about how to change the Xmx setting.

Adjusting Service Configuration Settings

If you expect your peak concurrent connected users (PCCU) to exceed 125, you should create your own service configuration using the following recommendations:

MaximumConcurrentSessions = 1.6 * PCCU DATACACHE.Size = 400 * PCCU DATACACHE.EmergencyTrigger = 0.80 * DATACACHE.Size DATACACHE.UrgentTrigger = 0.75 * DATACACHE.Size DATACACHE.NormalTrigger = 0.65 * DATACACHE.Size DATACACHE.PurgeTarget = 0.55 * DATACACHE.Size CONNECTIONPOOL.WRITEABLE.MaximumSize = 0.05 * PCCU CONNECTIONPOOL.WRITEABLE.TargetSize = 0.04 * PCCU CONNECTIONPOOL.WRITEABLE.MinimumSize = 5 CONNECTIONPOOL.READONLY.MaximumSize = 0.05 * PCCU CONNECTIONPOOL.READONLY.TargetSize = 0.04 * PCCU CONNECTIONPOOL.READONLY.MinimumSize = 5

The other settings in the service configuration do not generally need to be adjusted.

Performance Tuning

Performance is typically affected by network input/output (I/O), hard-disk drive I/O, memory (random access memory) I/O, or some combination of these three or other factors. Adjusting one of the factors sometimes moves the bottleneck to a new location, so you must approach the tuning task in a logical manner.

The performance tips in this section cover the basics and include:

See Oracle Database Performance Tuning Guide for complete information.

Running the Oracle Content Services analyze.sql Script

Oracle Content Services uses the Oracle database's Cost-Based Optimizer (CBO) to determine the most efficient way to execute SQL statements. For the CBO to work properly, the Oracle Content Services analyze.sql script should be run as part of regular Oracle Content Services operations, especially after large volume changes to the data, such as after users have loaded a large number of files into the instance. This script generates statistics about the distribution of data in Oracle Content Services so that the CBO can choose the most efficient way to execute SQL statements. For more information about the Cost-Based Optimizer, see Oracle Database Performance Tuning Guide.

Run the script during non-busy periods to avoid impeding performance for users.

The analyze.sql script, which makes calls to the DBMS_STATS package, exports schema statistics to a backup table, so you can restore statistics later, if necessary, as discussed in "Restoring Prior Statistics" in the following section. To run the script, enter the following at the command line:

cd ORACLE_HOME/content/admin/sql
sqlplus content_services_schema/password@connect_string @analyze.sql content_services_schema

This script may take a while to run, especially if Oracle Content Services contains a large number of documents.

Restoring Prior Statistics

Before gathering new statistics, the analyze.sql script exports backup statistics to the IFS_BACKUP_STATS table, marking the set of statistics with a timestamp. You can query the table for existing saved sets by executing this SQL statement:

SQL> select distinct statid from ifs_backup_stats;

This query returns a list of all statistics by statid (the date and time stamp). For example:

STATID
------------------------------
01-MAY-02 02:15.36
04-MAY-02 20:00.15
08-MAY-02 02:15.48
11-MAY-02 06:21.40
11-MAY-02 20:15.37

You can then restore the statistics from a day and time when you know your performance was better. For example, if you find that after using the statistics from your 8:00 pm running of analyze that performance is worse, then you can restore your statistics from earlier that day using:

SQL> @import_backup_stats.sql user_name '08-MAY-02 06:21.40'

By restoring the statistics, you are directing the CBO to revert to the way it previously executed SQL statements.

Providing Adequate Storage to Improve Performance

The largest consumption of disk space occurs on the disks that actually contain the documents residing in Oracle Content Services, namely the Indexed Medias and Non-Indexed Medias tablespaces. This section explains how the documents are stored and how to calculate the amount of space those documents will require.

Document Storage and Sizing Issues

BLOBs provide for transactional semantics much like the normal data stored in a database. To meet the criteria of transactional semantics, BLOBs must be broken down into smaller pieces which are individually modifiable and recoverable. These smaller pieces are referred to as chunks. Chunks are actually a group of one or more sequential database blocks from a tablespace that contains a BLOB column.

Both database blocks and chunk information within those blocks (BlockOverhead) impose some amount of overhead for the stored data. BlockOverhead is presently 60 bytes per block and consists of the block header, the BLOB header, and the block checksum. Oracle Content Services configures its BLOBs to have a 32 K chunk size. As an example, assume that the DB_BLOCK_SIZE parameter of the database is set to 8192 (8 K). A chunk would require four contiguous blocks and impose an overhead of 240 bytes. The usable space within a chunk would be 32768-240=32528 bytes.

Each document stored in Oracle Content Services will consist of some integral number of chunks. Using the previous example, a 500K document will actually use 512000/32528=15.74=16 chunks. Sixteen chunks will take up 16*32 K = 524288 bytes. The chunking overhead for storing this document would then be 524288-512000=12288 bytes which is 2.4 percent of the original document's size. The chunk size used by Oracle Content Services is set to optimize access times for documents. Note that small documents, less than one chunk, will incur a greater disk space percentage overhead since they must use at least a single chunk.

Another structure required for transactional semantics on BLOBs is the BLOB Index. Each BLOB index entry can point to eight chunks of a specific BLOB object (NumLobPerIndexEntry = 8). Continuing the example, whereas a 500 K document takes up 16 chunks, two index entries would be required for that object. Each entry takes 46 bytes (LobIndexEntryOverhead) and is then stored in an Oracle B*Tree index, which in turn has its own overhead depending upon how fragmented that index becomes.

The last factor affecting BLOB space utilization is the PCTVERSION parameter used when creating the BLOB column. For information about how PCTVERSION works, please consult Oracle Database SQL Reference.

Oracle Content Services uses the default PCTVERSION of 20 percent for the BLOB columns it creates. This reduces the possibility of "ORA-22924 snapshot too old" errors occurring in read consistent views. By default, a minimum of a 20 percent increase in chunking space must be added in to the expected disk usage to allow for persistent PCTVERSION chunks.

For large systems where disk space is an issue, set the PCTVERSION to 1 to reduce disk storage requirements. This may be done at any time in a running system with these SQL commands:

alter table odmm_contentstore modify lob (globalindexedblob) (pctversion 1);
alter table odmm_contentstore modify lob (intermediablob) (pctversion 1);
alter table odmm_contentstore modify lob (intermediablob_t) (pctversion 1);
alter table odmm_nonindexedstore modify lob (nonindexedblob2) (pctversion 1);

The steps for calculating BLOB tablespace usage are as follows:

  1. Calculate the number of chunks a file will take up by figuring the number of blocks per chunk and then subtracting the BlockOverhead (60 bytes) from the chunk size to get the available space per chunk.

  2. Divide the file size by the available space per chunk to get the number of chunks.

    chunks = roundup(FileSize/(ChunkSize-((ChunkSize/BlockSize) * BlockOverhead)))
    
    

    For example, if FileSize = 100,000, ChunkSize = 32768, Blocksize = 8192, and BlockOverhead = 60, then Chunks = roundup (100000 /(32768 - ((32768 / 8192) * 60)))= 4 Chunks.

  3. Calculate the amount of disk space for a file by multiplying the number of chunks times the chunk size and then multiplying that result by the PCTVERSION factor. Then add the space for NumLobPerIndexEntry (8) and LobIndexEntryOverhead (46 bytes).

    FileDiskSpaceInBytes = roundup(chunks*ChunkSize*PctversionFactor) +
    roundup(chunks/NumLobPerIndexEntry*LobIndexEntryOverhead)
    
    

    Continuing from the preceding example, chunks = 4, ChunkSize = 32768, PctversionFactor = 1.1, NumLobPerIndexEntry = 8, and LobIndexEntryOverhead = 46, so FileDiskSpaceInBytes = roundup (4 * 32768 * 1.1) + (roundup(4/8) * 46) = 144226 FileDiskSpaceInBytes.

  4. Calculate the total disk space used for file storage by summing up the application of these formulas for each file to be stored in the BLOB.

    TableSpaceUsage = the total of FileDiskSpaceInBytes for all files stored
    
    

Oracle Content Services creates multiple BLOB columns. The space calculation must be made for each tablespace based upon the amount of content that will qualify for storage in each tablespace.

Oracle Content Services Metadata and Infrastructure

The Oracle Content Services server keeps persistent information about the file system and the contents of that file system in database tables. These tables and their associated structures are stored in the Oracle Content Services Primary tablespace. These structures are required to support both the file system and the various protocols and APIs that make use of that file system. The administration and planning tasks of this space should be very similar to operations on a normal Oracle database installation.

You should plan for approximately 6 K of overhead for each document to be used from this tablespace, or about 2 percent of the overall content. If there is a significant number of custom metadata, such as attributes, subclasses or categories, this overhead should be much larger.

The initial disk space allocated for the primary tablespace is approximately 50 MB for a default installation. Of the 50 MB, 16 MB is actually used at the completion of installation. This includes instantiations for all required tables and indexes and the metadata required for the 700+ files that are loaded into Oracle Content Services as part of the installation. Different tables and indexes within this tablespace will grow at different rates depending on which features of Oracle Content Services get used in a particular installation.

Analyzing Performance Problems

After ensuring that you have run statistics properly and have enough free hard-disk drive to support the tablespaces, you may still have performance problems. If that is the case, you must determine whether the performance bottleneck is caused by the Oracle database, by Oracle Content Services, or by other factors.

To isolate the problem, start by looking at which processes are running and how many resources they are consuming.

  1. Run top (on UNIX) or Task Manager (on Windows platforms) as you reproduce the problem.

  2. Determine whether a Java process, the Oracle shadow process, I/O, or a combination is the bottleneck during that time.

If the Database Is the Bottleneck

If the bottleneck is the Oracle shadow process, use the Statspack utility to determine the SQL statement which is causing the largest number of buffer gets, and run Explain Plan on it.

If you see full table scans, then that may be the cause of the problem; the optimizer may not be choosing an appropriate plan. Report that problem to Oracle Support. Additional work must be done to isolate the problem.

For more information about the Statspack utility and Explain Plan, see Oracle Database Performance Tuning Guide.

If the Java Processes Are the Bottleneck

You may have too little memory. For example, if you see any java.lang.OutOfMemoryError errors in your log files, increase your Maximum Memory (Xmx) settings for that JVM. See "Modifying Node Configurations" for more information about changing the Xmx setting.

If users are complaining about poor response times, and top (on UNIX) or its equivalent (for example, Task Manager on Windows platforms), shows a Java process running at 100 percent of a CPU for a minute or longer, then the Xmx setting for Java may be too small.

  1. Turn on verbose garbage collection (verbosegc). To do this, edit the Java Properties of the node configuration. See Table 6-2, "Node Configuration Properties" for more information.

    In the node log file, output related to garbage collection appears as follows:

    [Full GC 1476K->1476K(2112K), 0.0549430 secs]
    
    

    A Full GC occurs when the Garbage Collector has exhausted all available memory in the nursery, and has to go into the rest of the heap to reclaim memory.

  2. If Full GCs generally occur more than once every 10 minutes (not just after startup), increase your Xmx settings for that JVM.

Viewing Cache Statistics and Changing Cache Settings

If the bottleneck is an Oracle Content Services Java process, start by checking the percentage of cache hits for the Oracle Content Services service using the Oracle Collaboration Suite Control, as follows:

  1. On the Content Services Home page, click the name of the node you want to manage.

  2. Click the name of the service. Typically, this will be IfsDefaultService. The Service page appears.

  3. Scroll to the Performance section and click Committed Data Cache Statistics. The Committed Data Cache Statistics page appears, showing real-time data for Cache Size, Cache Puts, Cache Removes, Cache Purges, Cache Purge Cycles, Cache Lookups, and Cache Hits.

    The goal is to have a high percentage of Cache Hits; as much as 100 percent is possible. If the percentage of Cache Hits for the service is less than 98 percent, the size of the Committed Data Cache may be too small.

    Because the Statistics Agent captures the real-time data, you can also see prior statistics by viewing the node log or application log. You can also configure this agent to write statistics to a document stored in the Oracle Content Services repository. See "Statistics Agent" for information about the Statistics Agent.

  4. To change the runtime Cache settings, return to the Service page and click Committed Data Cache Administration in the Administration section.

  5. Proportionately increase all Cache settings (Cache Capacity, Normal Purge Trigger, Urgent Purge Trigger, Emergency Purge Trigger, Purge Target) and click Apply.

    This will increase your memory usage on the Applications tier computer by approximately 3 KB for each object. For example, if you increase cache capacity by 5000, your memory usage will grow by 15 MB.

To make the changes permanent, update the service configuration. See "Modifying Service Configurations" for more information.

Viewing Connection Pool Statistics and Changing Connection Pool Settings

Check the target and maximum number of connections for the Read-Only and Writable connection pools using the Oracle Collaboration Suite Control, as follows:

  1. On the Content Services Home page, click the name of the node you want to manage.

  2. Click the name of the service. Typically, this will be IfsDefaultService. The Service page appears.

  3. Scroll to the Performance section and click Connection Pool Statistics.

    You should increase the "Target max. number of connections" and "Absolute max. number of connections" if any of the following is true:

    • "Failed allocation" is greater than zero.

    • "Total Connections" is more than two higher than "Target max number of connections."

    • "Deferred allocations" is greater than 5 percent and "Average time to allocate" is more than 10 milliseconds.

    Because the Statistics Agent captures the real-time data, you can also see prior statistics by viewing the node log or application log. You can also configure this agent to write statistics to a document stored in the Oracle Content Services repository. See "Statistics Agent" for information about the Statistics Agent.

  4. To change the runtime Connection Pool settings, return to the Service page and click Connection Pool Administration in the Administration section.

  5. Increase the "Target max. number of connections" and "Absolute max. number of connections" and click Apply.

    Each additional Target or Absolute connection will use approximately 8 MB for each connection on the Applications tier and 1 MB for each connection on the database.

To make the changes permanent, update the service configuration. See "Modifying Service Configurations" for more information.