C H A P T E R  5

Troubleshooting the Database Migration

This chapter provides troubleshooting information for the migration of the database from the Sun Java System Content Delivery Server, version 2005Q4 to the Sun Java System Content Delivery Server, version 5.1.

This chapter covers the following topics:


Messages

Database migration and verification scripts provide several levels of logging and tracing to expedite the troubleshooting process. The following message levels are used:

All messages except those in the Debug category are printed to the console and logged into the $CDS_HOME/dist/cds/logs/migration.log file. All messages, including Debug messages are logged into the $CDS_HOME/dist/cds/logs/migration.trace.log file.

Typically, when a migration process fails, a few error messages and exception traces are printed to both the console and the log files. To understand the reason for the failure, and gather maximum information about the problem, it is important to scan the process output and log file from the beginning to end to identify the original error message.

Provide the screen output, the log file, and the log trace file when you contact support due to a failed migration attempt. Depending on the information in those files, the contents of your label directory or a copy of your database might be required to fully investigate and correct the problem.

The following sections describe Error and Warning messages in more detail.



Note - The %s symbol in the body of messages in the following section is a placeholder for data displayed in the actual message.



Error Messages

Some error messages could indicate a problem with the configuration of the migration scripts or with the environment (for example, Content Delivery Server or the Oracle server). This section provides a list of error messages with detailed descriptions and the most probable cause for the error. Error messages can chain as cascaded modules report exceptions on other modules that they were running. Pay attention to all error messages produced by a failed execution and to the messages associated with exceptions.



Note - Error messages that begin with IE . are internal error messages. These messages are displayed when a code assumption has been violated. Such an occurrence could indicate that either the database is corrupted or that there is a migration logic failure. You must contact support to resolve such errors.


CONFIGURATION Exception!

Description: Occurs any time the configuration is invalid. This error can be caused by incorrect command line options or configuration files.

Action: Examine the entire migration process output to obtain more information about the error.

Exception while running %sException from task %sThread %s failedDelaying abnormal exit due to running threadsException while parsing argumentsConfiguration exceptionVerification problemInitialization problemDB failure, query : %s, params %sMigration exceptionWhile executing SQL : %sException in bootstrap()Initialization failedSTEP %s FAILEDcmd process returned code %simport returned code %sMR ’%s’ interrupts due to pending exceptions !Exception while running shutdown task %sUnexpected null %s (%s)Thread stopped due to pending exceptionWhile processing edition %sExecution failed on matrix %s, prefix %s, matrix %sexception while parsing argumentsConfiguration error

Description: These messages are follow-up error message to another error.

Action: Examine the entire migration process output to identify the original error message.

VERIFICATION OF %s FAILEDVERIFICATION FAILEDTarget query returned no rowsSource query returned no rowsExceeded allowed number of failures (%s) for column %sFAILED verification of %sCounters don’t match for prefix %s, table %s. Expected: %s, found: %sTarget query returned no row for %s,PK %sVerification failed while comparing column %s, at primary key %s

Description: Verification failure.

Action: Examine the migration output to determine the actual causes for the verification failure and to identify the tables where the failure is found. Contact support to determine what action is required to correct the problem.

Failed to login to source database as catalog owner userFailed to find vending user %s from %sServer code must be specifiedOperation can be either read OR writeServer type must be either catalog OR vendingoperation can be ’read’ or ’write’ only, and must be specifiedserver type can be ’catalog’ or ’vending’ only, and must be specifiedvending code myst be specified for ’vending’ server typeError parsing configuration file %sExpected root ’dbmigration’ elementcode attribute must be set and not empty for <server> tagFound two or more <server> elements with code %sServer code ’%s’ was not found in config fileVending code ’%s’ not found in configuration for server ’%s’Catalog event source ’%s’ is not definedMultiple <catalog> tags for server code ’%s’Multiple <label> tags for server code ’%s’Multiple <vending> tags with the same code ’%s’ found for server code ’%s’Source database config must be specifiedTarget database config must be specifiedCan’t read source database configuration file ’%s’Can’t read target database configuration file ’%s’Can’t find configuration file %sNo deployment name specified for the <catalog> tagDeployment ’%s’, directory ’%s’, can not be read or is not a directoryCannot access deployment configuration file ’%s’Expected property file (%s) does not exist<vending> tag must have ’code’ attribute present and not emptyNon directory specified for search %s<event_pk> tag should contain valid long value, got:%sValid pricing option name should be specified in <change_option_price> elementValid new price should be specified for <change_option_price> element<source_account> tag expected for <vending code="%s">No location specified for the labelFailed to find deployment file %smin attribute value is not a numbermax attribute value is not a numberNo value specified for coverage elementcoverage value is not a double number : %scoverage value must be in range of [0..100]Unknown size character %s in <max_file_size> tagbad file size specified in <max_file_size> : %sbad integer specified in <max_index> : %sCannot find SOURCE vending schema %sCannot find SOURCE event schema %sFailed to create directory %s%s exists and not a directory%s already populated. Verify you don’t need existing data and remove this pathLabel %s does not existLabel %s is not a directoryPath %s does not exist. Are you sure %s was created ?%s is not a directoryPreparation for %s was not completed successfullyThis migration step was already successfully completed. If you are sure you would like to retry it, remove the following file: %sDirectory is not writeable: %sCannot find target vending schema %s, in %Cannot find SOURCE catalog schema %sCannot find SOURCE fuilfillment schema %sCannot find target catalog schema %sNo vending server %s found in the databaseMore than one vending server %s found in the databaseCan’t establish connection as a system user. Does DBConf have SystemUser and SystemPassword tags ?

Description: Error is caused by a misconfigured file. The migration configuration file or some of the deployment configuration files are invalid.

Action: Correct the configuration file and restart the migration.

Missing primary key %s in table %s

Description: Probable cause is data corruption.

Action: Contact support for help in identifying and fixing the corrupted database.

The catalog data has not yet been submitted to the database. You must complete catalog migration before doing this step

Description: The migration steps were executed in the wrong order.

Action: Execute migration steps in the order specified in this guide. See Chapter 4 for more information.

DATA file doesn’t exist for %s:%sDUMP file doesn’t exist for %s:%s(%s)Expected to find constraints file %sAt least one record expected to be updated for %s

Description: Inconsistent label directory.

Action: Verify that no modifications were done to the label directory between different migration steps. If the problem persists, contact support.

migration is installed incorrectlyUnknown CDS DB version %sSource Database versions mismatch : found %s, expected %s

Description: Most likely caused by an incorrectly installed Content Delivery Server or migration patch.

Action: Verify that the installation is intact and that all migration patches were installed properly. If the problem persists, contact support.

export process failed on %sFailed import for %s_%sIMPORT process did not leave or left an empty log file ’%s’%s failed (sqlldr rc is %s)%s sqlldr did not leave log file. This means id had not done anything

Description: An external Oracle process failed.

Action: Examine the trace file for messages left by the corresponding external process. Examine the external process log files available in the label directory. These errors often happen with the ORACLE_HOME variable is not set, or the PATH variable does not contain the path to the Oracle binary directory. Contact support if the root cause of the failure cannot be determined.

Other error messages usually indicate that a problem exists in the migration program code. You might need to contact customer support to resolve the problem. In most cases, the errors are due to an unexpected state of the database or to a customized server environment. Make sure you provide customer support with both the log and the trace files. It is also helpful if you provide the label directory contents along with the exported database on which the migration was performed.


Warning Messages

Warning messages are indicators of irregularities in the database that were resolved during the migration process.

Interrupted while waiting for process output

Description: The thread that was watching for output generated by an external process received an unexpected interrupt. The external process might have unexpectedly died, or another migration thread requested all threads to be interrupted.

Action: No action required.

Process watcher caught exception

Description: The thread that was watching for output generated by an external process received an unexpected exception. The external process might have unexpectedly died or closed one of its output channels.

Action: No action required.

SQL warning: %s

Description: An SQL query submitted to the Oracle server returned a warning message.

Action: Examine the warning message to determine if any action is necessary.

Unknown tag %s found within tag %sNo tags are expected within tag ’%s’

Description: The migration configuration file is structurally incorrect.

Action: Review and revise the migration configuration file.

Caught exception trying to retrieve environment

Description: An exception occurred during migration when trying to retrieve environment variables for debugging purposes.

Action: Examine the exception message and trace to determine if any action is necessary.

Terminating task due to pending exceptionsTerminating thread due to pending exceptionsExiting due to external error conditions, destroying current process

Description: An executing thread detected an exception on another thread. This message is to inform you that the current task is not complete. The migration will fail due to the thread that received the original exception. An error message or other messages are also displayed.

Action: Follow the action for the error messages.

Missing entry with primary key %s in table %s

Description: The process that loads the BLOB entries into the database found a record that had BLOB in the source database but is missing in the target database.

Action: Contact support to verify if any action is necessary.

Null capability id 16 for one of the derived editions of edition %s.Poorly formatted capability value for id 16, for one of the derived editions of edition %s : %s.Edition %s has different capability value for capability %s. Old=%s, New=%s.

Description: The database contained an incorrect record or no record at all for one of the editions’s Supported Devices capability. This indicates that the database is probably corrupted.

Action: Contact support to verify what, if any, action is necessary.

Inconsistency in LCM table, found a mix of temporary/final records

Description: The logic used to generate the new LARGE_CAPABILITY_MAP table has failed to produce the required set of records.

Action: Contact support to verify what, if any, action is necessary.

Model to instance mapping count doesn’t match expectations. Expected : %s, found %s

Description: The resubmission and edition reverification process might have been compromised.

Action: Contact support to verify if any action is necessary.

Specified number of threads is not an integer

Description: The value of the <threads> tag in the migration configuration file is invalid.

Action: Review the migration configuration file and correct the error.

No verification coverage provided for %s, count=%s, defaulting to %s.

Description: The migration configuration file specified incomplete verification coverage.

Action: Review the migration configuration file and correct the problem.

Source fulflilment_request table reports %s entries per this vending account. %s entries, however, were copied from the source database. Absolute delta : %s%s entries must be deleted from SUBSCRIBER_GIFT table. Actually deleted %s entries from SUSBCRIBER_GIFT table%s entries must be deleted from SUBSCRIBER_PURCHASE table. Actually deleted %s entries from SUBSCRIBER_PURCHASE table.

Description: During migration, inconsistencies were detected in the FULFILLMENT_REQUEST table. This is usually found when records in this table do not match all the records in the SUBSCRIBER_PURCHASE and SUBSCRIBER_GIFT tables.

Action: Contact support to verify if any action is necessary.

Category item %s has no instrumented editions

Description: When this edition was resubmitted, no instrumented editions were produced by the submission workflow. This error most likely occurred because this content stopped being eligible for any existing devices.

Action: Examine the corresponding content item and all of it’s edition to determine why it’s resubmission has failed.

Category item %s will be deleted as there are other category items for class %sCategory item %s is marked ’Unstocked’ as there are no other category items for class %s

Description: During category item processing for the Vending Manager database, the class with these category items had editions that could no longer be stocked. Migration removes all such editions, until only one is left. This one is then marked as Unstocked, as at least one item must represent the content class in the Vending Database.

Action: none

Target query returned no rows for ’%s’, PK %s

Description: Verification expects that all rows that existed in the source database are present in the target database, however the verifying matrix indicates that if they are not present, the condition is not fatal. This error usually indicates that some rows in this table were omitted during migration due to data corruption.

Action: Contact support to verify if any action is necessary

Deployment implements different migration version that does core migration code. Developer portal version: %s, core version : %s

Description: The migration file might be misconfigured, probably caused by an incorrectly applied migration patch. Check the instruction for all applied patches to verify they were properly followed.

Action: Contact support if the problem cannot be resolved by reapplying the migration patches.

 Edition %s doesn’t have content data

Description: Database corruption. During migration, an edition was detected that had no binaries associated with it.

Action: Contact support to verify if any action is necessary

 Edition %s failed to verify

Description: When the specified edition was passed into the verification workflow, the verification rejected it. The database still stored the original edition, but no derived editions were created.

Action: Examine the corresponding content item and all of its edition to determine why its resubmission failed.

 Edition %s doesn’t have any devices matched to it.

Description: After verification, the edition was not determined to be capable with any of the currently known devices. The edition probably became incapable for any device if device profiles changed or if the edition changed during the reinstrumentation process.

Action: Examine the edition in question and check why did it stop matching any devices.


Troubleshooting Failed Steps

This section provides information to help you recover from problems you might encounter when migrating Content Delivery Server database.

If any of the read steps fail during the migration, you need to perform the following actions:

1. Clear the condition that caused the error.

2. Remove the label contents associated with the read step.

3. Restart the step.

The location for the label contents for the catalog sever is label_dir/catalog. The location for the vending server is label_dir/vending/target_vending_name.

If any of the write steps fail during the migration, the database could be left in a partially populated state. You need to reinitialize it before attempting to rerun the write step. To reinitialize the database, use the cdsi db commands described in Preliminary Migration Steps in Chapter 1 with the modifications given here.

If the catalog write step fails, a reinitialization of the entire database must be performed. To reinitialize the entire database, run the following commands:

# cdsi db users
# cdsi db schemas_noc

Before running the cdsi db users command, make sure that there are no open connections made by any software to any of the database prefixes. You can specify a custom database configuration file as a third parameter to the cdsi command.

If a catalog write step fails, you only need to reinitialize the affected catalog database. Leave other vending and catalog databases intact. To reinitialize the catalog database, run the following command:

# cdsi db users -cs

If a vending write step fails, you only need to reinitialize the affected vending database. Leave other vending and catalog databases intact. To reinitialize the vending database, run the following command:

# cdsi db schemas_noc -vs name 

name is the name of the Vending Manager server that you need to reinitialize (the one you are using for Content Delivery Server version 5.1).

Failure to clean the corresponding database after a failed write step can result in inconsistent behavior when the write step is repeated. The database migration might not detect such behavior. This behavior is evident in the exceptionally long processing time of the migration. The ultimate result is that the migration never completes.

To detect the problem, attach a tracer to the import process to see if it is executing properly. Proper execution consists of a series of read and write system calls. If no system calls are issued within one minute, there is a problem with the import process and the migration must be aborted.

A migration process can end in one of the following ways:

Be aware that when the process is interrupted, not all processes spawned by the migration process are terminated. The system administrator can identify such processes and terminate them manually. These processes can be Oracle import or export programs or Oracle SQL loader programs. Even when terminated manually, the processes could leave open connections to the Oracle database server. Having open connections prevents the corresponding database from being reset using the cdsi db commands. Refer to Oracle documentation for information on how to close open sessions.