CSCU (CIndex to SQL Conversion Utility) Home Page

Oracle® Imaging and Process Management

CIndex to SQL Conversion Utility (CSCU)
Version 7.7.1.0

October 2007

Oracle

Oracle Imaging and Process Management CIndex to SQL Conversion Utility, Release 7.7.1.0

Copyright © 1998, 2007, Oracle. All rights reserved.

Primary Author: Amy Willard

The Programs (which include both the software and documentation) contain proprietary information; they are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright, patent, and other intellectual and industrial property laws. Reverse engineering, disassembly, or decompilation of the Programs, except to the extent required to obtain interoperability with other independently created software or as specified by law, is prohibited.

The information contained in this document is subject to change without notice. If you find any problems in the documentation, please report them to us in writing. This document is not warranted to be error-free. Except as may be expressly permitted in your license agreement for these Programs, no part of these Programs may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose.

If the Programs are delivered to the United States Government or anyone licensing or using the Programs on behalf of the United States Government, the following notice is applicable:

U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the Programs, including documentation and technical data, shall be subject to the licensing restrictions set forth in the applicable Oracle license agreement, and, to the extent applicable, the additional rights set forth in FAR 52.227-19, Commercial Computer Software--Restricted Rights (June 1987). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065.

The Programs are not intended for use in any nuclear, aviation, mass transit, medical, or other inherently dangerous applications. It shall be the licensee's responsibility to take all appropriate fail-safe, backup, redundancy and other measures to ensure the safe use of such applications if the Programs are used for such purposes, and we disclaim liability for any damages caused by such use of the Programs.

Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

The Programs may provide links to Web sites and access to content, products, and services from third parties. Oracle is not responsible for the availability of, or any content provided on, third-party Web sites. You bear all risks associated with the use of such content. If you choose to purchase any products or services from a third party, the relationship is directly between you and the third party. Oracle is not responsible for: (a) the quality of third-party products or services; or (b) fulfilling any of the terms of the agreement with the third party, including delivery of products or services and warranty obligations related to purchased products or services. Oracle is not responsible for any loss or damage of any sort that you may incur from dealing with any third party.


Overview

This help file contains information regarding the CIndex to SQL Conversion Utility (CSCU).

Introduction

The CINDEX to SQL Conversion Utility (CSCU) converts CINDEX files to SQL tables, and must be run prior to upgrading to IBPM 7.6 and later.

The conversion utility, CSCU.exe, requires that the system have Cndx32s.dll from a previous installation of Acorde or IBPM. When running multiple instances of CSCU.exe on multiple machines, make sure that this file is on every machine that will be running CSCU.EXE and in the path.

CSCU has the following functionality.

  • Start, stop, restart and schedule jobs.

  • Run on multiple machines at the same time

  • May be run on any computer with access to the SQL database and to the CINDEX directory

  • Status information is displayed of the current progress

  • Log files are kept detailing all operations

Limitations

The following are limitations for CSCU.

CSCU does not track storage object deletions. Do not run CSCU on live indexes where storage object removals or modifications will take place. It is recommended that CSCU NOT be run against a live system.

CSCU was tested with a minimum version of ODBC. Choose a computer to run CSCU that has at least the following version of the ODBC drivers installed.

  • Oracle must be at least version 9.2.0.6.3.
  • SQL must be at least 2000.81.9031.14 or later.

If necessary, the Oracle ODBC drivers can be downloaded via Oracle Tech-Net at http://www.oracle.com, and the SQL Server ODBC drivers can be downloaded from the appropriate MDAC version which is located at http://www.microsoft.com/downloads/details.aspx?familyid=9ad000f2-cae7-493d-b0f3-ae36c570ade8&languageid=f49e8428-7071-4979-8a67-3cffcb0c2524&displaylang=en

Installation Instructions

The CSCU installation package is available on your IBPM 7.6 (and later) CD in the directory named \CSCU. The installation installs the CSCU utility and the help file.

CAUTION: OTSCHED.DLL for IBPM 7.6 has not been tested with previous versions of Acorde or IBPM, and may be incompatible with earlier versions of the product. Please install CSCU on a machine where the 7.6 OTSCHED.DLL will not interfere with your current version of Acorde or IBPM.

When the user inserts the IBPM 7.6 release CD on any computer that previously had Acorde or IBPM installed on it, the AUTORUN program will automatically prompt the user to install CSCU. However, if you wish to install CSCU on a computer that has never had Acorde or IBPM installed, or if you wish to install CSCU manually, follow these steps:

  1. On a computer with ODBC connectivity, create an ODBC connection to your database. This can be done via the ODBC Data Source Administrator (ODBCAD32.EXE).
  2. Ensure that the computer you are installing to has read, write and modify access to your CINDEX directories.
  3. Run setup.exe from the CD from the CSCU directory and follow the installation instructions.
  4. Run CSCU.EXE from Start | Programs | Stellent IBPM | CSCU | CSCU to start the program. See the Usage Instructions topic for more information on how to use CSCU, and the Planning topic for more information about making a successful conversion to IBPM 7.6.


Conversion Planning

This topic contains planning considerations before using the CIndex to SQL Conversion Utility (CSCU).

Introduction

CSCU has been designed for the fastest operation possible, and even supports multiple conversion processes running simultaneously. Most installations will not need this capability, but you should be aware of it in case your installation has more than 1/2 billion objects stored.

It is strongly recommended that careful analysis and testing be performed on your Acorde or IBPM system before attempting an actual production conversion. Make sure you have a complete backup of all the data files before starting any conversion or upgrade.

Considerations

It is important to determine answers to the following questions when planning for a storage index conversion using CSCU.

  • How much data space will be necessary?
  • What will be the impact on the database?
  • How long will the conversion take?
  • How many objects per year will be added to our IBPM system, and how will that effect database growth planning?
  • Are there any special considerations regarding Batches?

It is important to answer all of these questions prior to an actual conversion. Here are some methods to determine answers to these important questions.

"How much data space will be necessary?"

The amount of database space necessary for the converted indexes will be equal or less than the amount you are currently using for your CINDEX files. The reason is that CINDEX does not reclaim deleted record space. Therefore, if your system is very fluid in the number of objects added and deleted, then your database space requirements will be much less than your current CINDEX space requirements. However, if your system has few if any deleted objects, then the database space will be approximately equal to the amount used by your CINDEX storage indexes.

The conversion process will require additional transaction space as well. Transaction space required during the testing of a 14GB conversion has been:

  • 2.5 GB for SQL Server
  • 30 GB for Oracle (Change UNDO tablespace if retention is set to default. Set retention to zero if UNDO tablespace is limited. Rollback segments may also need to be increased.)

The best way to determine for sure the amount necessary, is to do a partial or complete test conversion (see Test Conversion below) and measure the amount actually used.

"What will the impact on the database be?"

Moving the storage indexes into the database will impact the database in three ways:

First, your database will use more disk space. Previously, all storage indexes were kept separately in CINDEX files. Now, however, all storage indexes will be kept in your database. In some instances, this may be hundreds of millions of extra rows! In test cases, we have found that 123.25 million rows required 9.25 GB of disc space.

Secondly, the database conversion process will move the CINDEX content into the database as fast as physically possible. This will have an immediate short term impact on the database of high activity and rapid growth. Therefore, it is advisable to perform a test conversion (see Test Conversion below) to see how much space is required by the conversion process. Depending upon the outcome of the test conversion, you may need to increase your processing power and/or your available storage space.

Thirdly, once the conversion is complete there will be additional long term impacts on the database resulting from daily access including: document retrievals, new document storage, and document disposition. If possible, after a test conversion is completed, a simulation test of the load encountered under normal operations can be performed to gauge the configuration settings required in your environment.

"How long will the conversion take?"

The answer to this depends upon many factors such as the speed of your database processor, the speed of your network, the speed of the access to the CINDEX files and the number of CSCU instances running. In test cases, we have found a conversion speed of 123.25 million rows converted in 5.5 hours (100 MB network, CINDEX files local to the client, 2.8 GHz dual processor XEON for the SQL Server database, 2.4 GHz dual processor XEON for the client). However, your results could vary greatly from these. The best way to determine how long a conversion will require is to perform a test conversion and measure the results. See Test Conversion below for more information.

"How many objects per year will be added to the system, and how will that effect database growth planning?"

The easiest way to determine this answer is to perform a "SELECT COUNT(*) from OBJECTLIST" from your production system back-end database, which will tell you approximately the number of objects you have in your system at any given time. Divide this number by the number of months you have been live with IBPM (or Acorde), and you will have the average number of objects per month. You can use this number to plan for the growth of your system to allow enough space for the ST_PAGES table after the conversion is complete.

"Are there any special considerations regarding Batches?"

Prior to CIndex removal in the IBPM 7.6 release, batch names were stored case sensitive in the batch.dat file. In the IBPM 7.6 release, batch names are stored all in upper case in the Imaging database. This has the effect of making the batch names case insensitive.

To ensure success converting existing batches, please verify that your batch.dat job does not contain any duplicate batch name references (i.e. TEST, test, Test). If duplicate batch name references exist in the batch.dat job, it is recommended that all batches be indexed into the IBPM database prior to using CSCU to convert the batch.dat job.

If it is not possible to index existing batches prior to conversion, the eReset utility must be used to rename batches with duplicate batch names. Failure to remove duplicate batch name references from the batch.dat prior to processing the conversion job with CSCU will result in errors and a failure to insert duplicate batch name references in the database. The eReset utility is installed on all existing Storage Servers or Name Services servers.

Test Conversion

A test conversion is an excellent method to answer many of your questions, plus ensure that your final on-line conversion will go smoothly. To perform a test conversion, follow these steps.

  • Make a copy of your CINDEX storage index files (normally located at ...\Acordesv\disc\indexes directory) to some temporary location. You'll have to temporarily bring down your IBPM services to make the copy.
  • Next, create a temporary database space for your test conversion. If you want to test the entire conversion, make the database space large enough to hold the entire converted information. This will prevent the database from extending the space automatically during the processing.
  • Start CSCU and convert one or more of the CINDEX "jobs" (see the Usage Description topic for more details).
  • After the conversion is complete, be sure to check the error logs to ensure there were no errors. If errors appear in the logs, check them out and correct them before performing your final conversion.

Use this test run to determine conversion speed, database space utilization, database processor impact, network utilization, etc.

Sample Conversion Road Map

The following is a sample conversion road map... a way to navigate the complex jungle of converting your storage indexes from CINDEX to a Relational Database, or RDB. Use this road map to help you avoid the pitfalls and snares the unwary may encounter.

Two or Three Months Before Actual Conversion

  • Do a partial test conversion using one or two of your CINDEX files (page files, such as ########.DAT would be best)
  • Discuss with your DBA the conversion process, including the amount of extra database space you anticipate needing, the amount of growth in database space per month, and the amount of additional processing power required to handle that processing. Much of this information will be gleaned from the test conversion performed above.
  • Acquire any additional storage space, network bandwidth, processing power, or computer memory needed to support the conversion process (CSCU) and/or the additional daily processing anticipated once all users are using IBPM 7.6.
  • Schedule a full test conversion with your DBA and network professionals to be performed after the additional required equipment has been installed and verified to be in proper working order.

One Month Before Actual Conversion

  • Perform a full test conversion over a weekend. Your DBA should be on hand to measure RDB performance and CPU, memory and space usage. Your network professionals should be on hand to measure network utilization. You should ensure that you can get through the entire conversion in one weekend and your RDB is properly configured. Test for performance improvement by running and timing one, and then two, instances of CSCU on your network.
  • Inspect and correct any and all conversion errors. Since every CINDEX record in every CINDEX file will be examined, if there are any CINDEX corruption errors, CSCU will find them.
  • Correct any system failures or resource insufficiencies found, and schedule the final conversion with your DBA and network professionals.
  • Clean up test conversion from the RDB. See "How to roll out the effects of CSCU on the database" for more information.

The Actual Conversion

In all cases, it is very important that you have a known good backup of your databases, your CINDEX files AND your DSMS source files before starting the conversion.

Even though you may have a very large IBPM installation, you will be surprised at how quickly the conversion progresses, especially if you have multiple copies of CSCU converting at the same time. Follow these steps to complete your conversion.

  • Confirm you have a good backup.
  • Install the CSCU package (see Installation Instructions).
  • Extend your database space to allow for the storage indexes. Make sure you have sufficient temporary DB space as well as live DB space.
  • Bring down your existing Acorde or IBPM system.
  • Convert the non page files, such as BATCH.DAT and CTRL.DAT and REGISTER.DAT. Converting these first, and ensuring that they are converted successfully will avoid have to re-convert page files if BATCH.DAT fails to convert (see "Common Conversion Errors" below)
  • Convert the storage index page (########.DAT) files.
  • Apply indexes after all conversions are complete from one and only one instance of CSCU.
  • Inspect CSCU error logs and correct any conversion errors. See Common Conversion Errors below for a list of errors that are most likely to occur, as well as suggested corrective actions.
  • Rename the directory of your CINDEX files (DON'T DELETE THEM!) Renaming the directory will ensure that if any Storage Servers or Name Services are not updated properly (see next step), then they will not start up, and will not accidentally modify the (now converted) CINDEX files.
  • Install and upgrade your system to IBPM 7.6.
  • Bring up the new IBPM 7.6 system.
  • Test a random set of images to ensure that they can be retrieved and that new filings can be performed.

What to do if the Conversion Fails due to CSCU errors

  • Don't panic. If the conversion fails for some reason, no data has been lost or corrupted. The CSCU utility does not delete or change any existing information, it only converts from CINDEX to RDB.
  • Determine the cause of the failure. Has only one CINDEX file failed? Which one (see "Common Conversion Errors" below)? Are you are out of RDB storage space or temporary table space? Use your discovered answers to help plan and execute a successful conversion the next time.
  • If the error cannot be corrected and conversion completed, then you must clean up the database and schedule a conversion at a later date. See "How to roll out the effects of CSCU on the database" for more information.

What to do if the Conversion Fails due to Other Issues

  • Once again, don't panic. The CSCU utility does not delete or change any existing information, it only converts from CINDEX to RDB.
  • Investigate the cause of the failure and correct.
  • In almost all cases, CSCU can be restarted and will pick right back up where it left off. If you have time to complete the conversion in the time allowed (before users again use the system), then it is best to restart CSCU.
  • If you do not have time to complete the conversion before the users must use the system, then you must clean up the database and schedule a conversion at a later date. See "How to roll out the effects of CSCU on the database" for more information.

Common Conversion Errors

The following are the most common errors you will encounter when using CSCU.

  • Problem: CSCU does not complete processing before the users need to get back on the system.
    Correction    : Investigate and correct cause of slow conversion. You should obtain an insert rate of at least 4,000 rows per second for one instance of CSCU. If you are not achieving at least that rate, determine the cause of the slow down and correct. If you are achieving that conversion rate or greater, then investigate adding instances of CSCU on other machines, increasing the CPU processing power on your RDB, increasing the network bandwidth between your CSCU client(s) and the CINDEX files and the RDB, or scheduling the conversion over a longer weekend.
  • Problem: The database runs out of storage data space or temporary table space. Correction: Increase the database storage or temporary table space.
  • Problem: BATCH.DAT is unable to convert due to duplicate rows inserted into the ST_BATCHES table.
    Correction:     This is due to the addition a unique index on the BATREF column. The old BATCH.DAT did not ensure that this column was unique. To correct this, please see the above topic "Are there any special considerations regarding Batches?"

How to roll out the effects of CSCU on the database

If CSCU does not complete or fails, and you must roll out changes it has made to the database, then the following tables should be dropped:

Tables: SU_CSCUJOBS, ST_PAGE, ST_BATCHES, ST_CTRL, ST_MAP3XIBPM, and ST_VOLUMES


Usage

Usage Instructions

This topic contains instructions on how to use the CIndex to SQL Conversion Utility (CSCU).

Tour of CSCU User Interface

CSCU has the following four distinct sections of the user interface.

  • A conversion jobs section, which lists all of the jobs that are available for conversion, their current status, as well as five buttons for interaction with the conversion jobs.
  • An error handling and logging section, which allows the user to specify how errors will be handled by CSCU, should they occur.
  • A status section that displays the current status of the active conversion job.
  • A program control section.

The Conversion Jobs Section

In the conversion jobs section, there is a list of all conversion jobs and the current status of each job.

  • Job Name - This is the file name of the CINDEX job.
  • Status - This is the current status of the job. The values for status are:

Complete: The conversion job is complete and has been successfully processed.
Incomplete: The conversion job has not been started.
Processing: The conversion job has been started and is still processing.
Error: The job encountered one or more errors and could not be completed. Note: In order to retry the conversion of any job that has a status of "Error" you must first Manage (see below) the job to inspect and reset the job status.

  • File Date - This is the last access date of this CINDEX job.
  • Size - This is the size, in bytes, of the CINDEX job.
  • Machine - When a job is being processed (see Processing above), then this column will have the machine name of the CSCU installation which is currently processing this job.
  • Error Code - When a conversion job encounters a problem, this column contains the error code that is generated.
  • Desc. - This column contains the corresponding description, if available, for the returned error code.

The following five buttons control operations on the conversion jobs.

  • Select All - This button is used to select all the jobs at once. Use the Refresh button to unselect all jobs.
  • Refresh - This button is used to get the current status of all jobs. Note: Use this feature to remotely monitor many conversion jobs from a central location.
  • Convert - This button is used to convert selected jobs.
  • Manage - This button is used to inspect and view detailed information about a conversion job, and to reset jobs with an Error status back to an Incomplete status for re-processing.
  • Verify - This button is used to perform a verification of a conversion job. This verification will test every CINDEX record against what is contained in the database to ensure that records are not missing and have not been changed.

The Error Handling and Logging Section

If an error happens, there are three ways to handle the error.

  • Log the error and continue with the same job (default).
  • Log the error and skip to the next conversion job.
  • Log the error and stop all processing of conversion jobs.

If you are running CSCU on a computer where IBPM Services have previously been installed, then the log files will be in the same location where you configured them to be for the services, and using the same name (i.e. YYYYMMDD.LOG). However, if you are running CSCU on a computer that has never had IBPM services installed, then the log file will be located in C:\StellentIBPM\LOG directory. If you wish to change this location manually, then the setting may be changed using REGEDIT to modify the value of HKEY_LOCAL_MACHINE\SOFTWARE\OPTIKA\COMMON\EVENTLOGPATH.

The Status Section

In this section, the user can see the progress of the current job. The progress section includes the following information.

  • Progress - There are two controls to the right of the word Progress. The first control is a progress bar that shows what percentage of the job has been completed. The second control lists the job name (i.e. CINDEX file name) that the program is currently processing.
  • Status - This is a text message indicating what the system is currently doing. The message will indicate information like Idle or Processing, 50% complete.
  • Elapsed Time - This value is the number of hours, minutes and seconds since the current job started.
  • Estimated Remaining Time - This value is an estimate of the number of hours, minutes and seconds until the current job is complete.
  • Processed Records - This is the number of records the job has processed in the current job.
  • Estimated Records - This is an estimate of the total number of records the program may have to process in the current job. This is an estimate only, and represents the maximum number of rows to be processed.
  • Inserted Rows - This is the total number of rows that have been inserted into the current SQL table.
  • Failed Rows - This is the total number of row inserts that have failed for the current job.

The Program Control Section

This section consists of four buttons:

  • Indexes - After all jobs have been converted, this button must be selected to apply indexes to the database tables. If the user attempts to exit the program and all jobs are complete, they will be warned that indexes must be applied to successfully convert the CINDEX files.
  • Schedule - This button allows the user to set a schedule for conversion operation. The standard scheduling plug-in will be used. By default, the schedule used will run immediately and will run 24 hours per day. When the program enters a time period where it should not be running, then the Processing Message will be set to something like, CSCU waiting until HH:MM to continue processing.
  • Help - This button displays help documentation for this utility.
  • Exit/Stop - Allows the user to stop a conversion job, or exit CSCU if no conversion job is currently processing.

Normal Usage Scenario

Here is an example of how to successfully use CSCU. Please refer also to Planning for more information.

  • Make a known valid backup of your CINDEX files and database in case a later restore is necessary.
  • Install CSCU from the release CD. See Installation Instructions for more information.
  • Make a copy of your CINDEX index files. It will be necessary to stop IBPM Services to copy the CINDEX files, and then restart your system.
  • Disable all modification security settings from your on-line system to ensure that no storage objects are modified or deleted during conversion. This step is only necessary if conversion requires more time than your system can be down during conversion.
  • Start and login to CSCU.
  • Create a schedule for CSCU to run conversions (use the schedule button). By default, CSCU is scheduled to run all the time.
  • Select one conversion job to convert.
  • Click on Convert to start the conversion.
  • After the conversion, use the status information about the conversion job to determine the following:
    (1) CSCU is generating the expected through-put,
    (2) allocated database space is sufficient to convert the entire system and
    (3) gauge the amount of time that will be required to complete the entire conversion job.
  • If you have any failures, investigate the cause of the conversion failure before proceeding. Make appropriate corrections in the operating environment (such as DB backup times, server down times, etc) and restart the conversion. You may have to manage a conversion job (see Manage button above) before proceeding.
  • Convert the other jobs by using one or more installations of CSCU. CSCU is designed to safely work in tandem with one or more installations of CSCU. Although the user only needs one copy of the utility, it must be installed on each separate machine where CSCU conversions are to be processed. However, performing a test conversion of 1 to several jobs will help determine how much time your conversion will require.
  • Once a complete conversion pass has been performed, if you have been adding content to your production system during conversion, re-copy the index files and restart CSCU. Verify and convert any files that have not yet been converted. Note: Adding content to your production system during CSCU conversion is discouraged and requires great care to ensure no content is missed during conversion.
  • Inspect the error logs and correct all errors.
  • After all jobs have been converted, select one and only one installation of CSCU and click on the Indexes button to apply indexes to the database tables.

You are ready to upgrade the system to the 7.6 IBPM software.


CSCU Frequently Asked Questions

This topic includes some frequently asked questions and some benchmark information.

FAQ

How long will CSCU take to convert my storage indexes?

While your network infrastructure, computer system hardware and IBPM or Oracle I/PM services configuration may vary, we have made every attempt to do performance testing on a typical end-user system. We strongly suggest a test conversion for proper conversion planning.

See the section of this page titled, Benchmarks. See also the How Long will the Conversion Take and Test Conversion sections of the Planning topic.

Can I run two (or more) copies of CSCU at the same time?

Yes. However, you should first test and measure with one instance of CSCU before adding more instances of CSCU. If your RDB system or network is saturated with only one copy of CSCU running, then adding more instances of CSCU may actually slow down conversion. You cannot run two instances of CSCU on the same computer because CSCU has been tuned for optimal network performance for database I/O with only one instances of CSCU running on one computer.

What can I do to decrease the time it takes CSCU to complete a conversion?

There are several things you should explore to increase CSCU performance, including:

  • Locate all CINDEX files locally (on the same computer) where CSCU is running
  • Do not attempt to run CSCU on an active system. It will fail.
  • Increase RDB processing power
  • Choose the fastest client computer to run CSCU on, with the fastest possible hard drive
  • Place CSCU on the same local sub-net and network switch as your RDB system
  • Add an additional instance of CSCU (see notes below about splitting CINDEX files across two or more instances of CSCU)
  • Increase your network bandwidth

You said to copy the CINDEX files locally to CSCU. Is it possible to split CINDEX files across two or more instances of CSCU?

It is possible to split CIndex files across two or more instances of CSCU, but more care is required to configure the conversion jobs that each instance will process. From each copy of CSCU, you must take care to only select those jobs which are local to the instance of CSCU that you are currently configuring.

Do NOT select all jobs for processing when using multiple instances of CSCU! This will cause conversion failures because each instance of CSCU shares a common control table, SU_CSCUJOBS. If one instance of CSCU begins processing a job that is already being converted by another instance of CSCU, an error will be generated because the job can not be accessed. In this scenario, it is likely that all conversion jobs will have an ERROR status.

How much space will the storage indexes require in my RDB system?

A very rough estimate is that they will require less than or equal to the same amount as your CINDEX files do. However, since CINDEX does not reclaim deleted rows, then depending on the amount of object deletion and modification, your RDB system will actually require less space than your CINDEX files do. Before performing a conversion, users will see an estimate of required database space.

When can I remove my CINDEX files?

You should not remove your CINDEX files until you are certain that your new IBPM 7.6 product is operational, and that stored images are retrieving properly. Moreover, you should never remove your CINDEX files until you are sure that they are backed up properly and known to be valid.

How does IBPM 7.6 using a RDB system compare (speed-wise) to IBPM 7.5 using CINDEX?

Very favorably. IBPM 7.6 is within 2 milliseconds, average, in retrieval speeds, and may actually out-perform CINDEX for larger installations due to better RDB system scalability.

What should I do with my "open file manager" software, now that I can backup all of the storage indexes using a normal database backup?

It is no longer necessary to manage open files during a backup because the CINDEX files are not being added to or modified as of IBPM 7.6.

What is the "Schedule" button for, and when would it be used?

The Schedule button was built into CSCU at a time when its performance was slower and converting a live Acorde or IBPM system was in consideration. It allows CSCU to perform conversions during certain hours of operation. However, converting live systems is strongly discouraged. CSCU conversion speeds have been greatly improved, and unless your system has more than 1/2 billion records in your CINDEX files (such as 1/2 billion objects) there should be no need to convert a live system. Converting a live system can be done, but requires fairly complex operations and should be avoided whenever possible. If you feel you must convert an Acorde or IBPM system during normal working hours, please contact Customer Support for further assistance.

CSCU has an "estimated" records to be converted and an "actual records." Why are these numbers always different?

CINDEX files do not keep track of actual records, but rather keeps track of the total number of records ever added to them. Therefore, the estimated records is a quick approximation of how many records may be converted. The actual records value is tracking the records as they are being converted.

I've noticed that sometimes the "Initializing Job" status displayed in CSCU takes a long time. Why is that?

This is because the job you are converting has actually tried to be converted before, and in order for CSCU to perform a conversion on this job, it must first determine the first and last records in the file. In some instances, CSCU may search for a long time trying to find the last record in the file. When this happens, the "Initializing Job" status may be displayed for several minutes.

I see the following error when I try to run CSCU: "Incorrect version of ODBC driver! Oracle must be at least 9.2.0.6.3, or SQL must be at least 2000.81.9031.14. Please upgrade your ODBC driver version." How can I correct this problem?

Your system does not have the correct version of the ODBC drivers installed. Because CSCU is meant to be used before installing the 7.6 or later version of IBPM, it is best to install ODBC drivers if necessary. The CSCU installation does not automatically change your ODBC drivers. Before installing the drivers, ensure that installing a different MDAC driver on the computer chosen to run CSCU will not adversely effect your previously installed Acorde or IBPM product or any other critical application, and install the correct version of the ODBC drivers.

If necessary, the Oracle ODBC drivers can be downloaded via Oracle Tech-Net at http://www.oracle.com, and the SQL Server ODBC drivers can be downloaded from the appropriate MDAC version which is located at http://www.microsoft.com/downloads/details.aspx?familyid=9ad000f2-cae7-493d-b0f3-ae36c570ade8&languageid=f49e8428-7071-4979-8a67-3cffcb0c2524&displaylang=en

Benchmarks

Disclaimer: Each system upgrade will be unique. These benchmarks represent a reasonable upgrade environment.

The following benchmark information was gathered to provide guidelines to help customers estimate the resources and time required to perform the conversion of the CIndex storage control files as part of the upgrade to IBPM 7.6.

The Benchmarking was performed on two computers: one hosting Microsoft SQL Server, and the other the 7.6 upgrade utility CSCU. The CIndex files were located on the same machine as the CSCU utility to achieve the highest performance.

SQL Server | CSCU Utility
2.8 GHz dual Xeon, 2 GB RAM, 147 GB SCSI hard drives (raid 0), 100 Mb network, Windows 2000 Hyper threading disabled | 2.4 GHz dual Xeon 1 GB RAM 136 GB Fujitsu SCSI hard drives 100 Mb network Windows 2000 Hyper threading enabled

The test data for this benchmark was a customer’s set of 69 CIndex storage control files totaling 14 GB of data. The average file sizes were as follows:

File Type | Size (bytes)
Page | 184,465,967
Register | 210,944
Control | 4,096
Map3x | 2,938,908,672
Batch | 4,096

The total amount of time required to convert these files was 5 hours and 33 minutes. The impact on the database was to increase its total size by 9.5 GB.

CSCU inserts the new data into the database in blocks of 1000 records at a time (per transaction). The auto-expanding transaction space grew to 2.5 GB during the conversion.

Metric | Comments
42 Mb / Minutes | Total 14 GB conversion took 5 hours and 33 minutes
81% active records | Percentage of records in CIndex files that were active*
68% DB equiv. | Percentage of CIndex size expected in database
121 bytes / record | Average record size of new records created
6,172 records / sec | Average conversion speed
60% CPU | Utilization of the SQL Server’s CPU
70% CPU | Utilization of the conversion computer’s (CSCU) CPU
Negligible | Memory usage on both SQL and CSCU computers

*As documents are deleted from the IBPM system their associated CIndex records are deleted. These deleted records remain in the CIndex files taking up space. This percentage represents the number of active records versus the total number of active plus deleted.

The upgrade was retried with two CSCU utilities working in parallel on the CIndex files separated into two groups. The CIndex files remained located on the first CSCU computer. The second CSCU computer accessed the files across the network. Introducing a second instance of CSCU decreased the conversion time by 25%.

Sample Interpolation

Given 2 GB of CIndex files to be converted in a similar environment one could expect the following results:

Value | Result | Calculation
Conversion Time | 47 min | 2 GB / 42 Mg/min
Database Space | 1.36 GB | 2 GB * 68%


7.6 Open Issues CSCU

CSCU Open Issues

This topic contains known issues with the shipping version of CSCU 7.6. This information is a duplicate of the information related to open issues with CSCU found in the ReleaseDocs.CHM.

Service -- Utilities

76575 -- CSCU MUST RUN STAND ALONE - CSCU is designed to run stand-alone. Running CSCU while the IBPM or Acorde services are running will result in an error "-32 (34016)" being logged to the error log.

76783 -- DO NOT CHANGE DATE TIME WHILE CSCU IS RUNNING - If the date or time on the local computer is changed while CSCU is running, the scheduler tool does not execute CSCU when it was scheduled to run. Do not reset the date or time on a machine while CSCU is running.

77125 -- SCHEDULE DAILY - The default schedule for CSCU is daily. However, the daily bar is not highlighted.

77568 -- INVALID RECORDS DURING CONVERSION - The CINDEX to SQL Conversion Utility (CSCU) does not perform validation on the CINDEX records read from the various CINDEX files. Therefore, if a CINDEX record is invalid, it will be converted into the SQL database in the same format as it existed in the CINDEX files. If a CINDEX record violates a SQL Database constraint (such as an attempt to insert a blank column into a non-null column), then a database exception will be caught and logged into the service log files.

77599 -- CSCU NETWORK CONNECTIONS - The CINDEX to SQL Conversion Utility (CSCU) can not detect a network connection failure. It will process network connection failures as if the CINDEX record was unavailable. Therefore, CSCU should be run with all CINDEX files local to the CSCU utility if possible. If multiple copies of CSCU are required, duplicate copies of the CINDEX files may be made and converted.

77926 -- RUNNING CSCU - To operate properly, CSCU must be run from an account with rights to access the registry keys to check the MDAC version and the ODBC database drivers. These keys are in HKEY_LOCAL_MACHINE/SOFTWARE/Microsoft/DataAccess.

78334 -- MULTIPLE ORACLE ODBC DRIVER ON CSCU MACHINE - If multiple Oracle ODBC drivers are installed on a system, CSCU will always detect and use the oldest ODBC version. To correct, remove the oldest ODBC driver.

78335 -- CREATING INDEXES WITH CSCU - When running the COLD to SQL Conversion Utility (CSCU) the indexes button should be pressed on one and only one instances of CSCU. Clicking "Index" on multiple instances of CSCU will result in both stating that indexes have been created, when in fact only one instance creates the indexes.

78396 -- MANUALLY SELECT DATABASE FOR CSCU - The CINDEX to SQL Conversion Utility (CSCU) will not detect if the selected database is an IMAGING database. To successfully convert the storage CINDEX file, the IMAGING database must be manually selected.