Sun Java System Calendar Server 6 2005Q4 Administration Guide

Chapter 22 Troubleshooting

This chapter covers some of the troubleshooting techniques you can use to determine if there is a problem with your system and what is causing it. The chapter contains the following topics:

Turning on Debugging Information

While there is no one ics.conf parameter that puts the whole system in “debug mode”, this section describes some ways to get useful debug information:


Note –

Be sure to turn off excess logging and monitoring when not needed as it will negatively impact performance.


Increase Logging Level

Use the parameter shown in the following table to increase the verbosity of logging:

Parameter 

Description and Default Value 

logfile.loglevel

Set to DEBUG to get all levels logged, including CRITICAL, ALERT, ERROR, WARNING, NOTICE, and INFORMATION. This applies to all logs.

For more information about the different logs available, see Using Calendar Server Log Files.

Enable Logging Access to the LDAP Cache

To log all accesses of the LDAP data cache and print out the log (report) set the ics.conf parameters shown in the following table:

Parameter 

Description and Default Value 

local.ldap.cache.stat.enable

Specifies whether or not to log the access to the LDAP data cache and to print statistics in the log file. The default is “no” (no statistics logged). Set to “yes” to enable logging of statistics.

For performance enhancement, use this only in debug mode. 

local.ldap.cache.stat.interval

Specifies the interval in seconds when each statistics report is written to the log file. The default is “1800” seconds (30 minutes).

This is only active if logging is enabled. Decreasing the interval can help you pinpoint problems. Increasing the interval helps decrease system load. 

Clearing the LDAP Cache

There is currently no logic in Calendar Server to expire LDAP cache data. You must manually remove the contents of the ldap_cache directory and restart Calendar Server.

ProcedureTo Clear the LDAP Cache

  1. Stop Calendar Server.

  2. Remove all files in the /var/opt/SUNWics5/csdb/ldap_cache directory, but do not remove the ldap_cache directory itself.

  3. Restart Calendar Server.

Monitor the System Using Calendar Server Utilities

Use the following Calendar Server utilities to monitor your system:

For more information on the Calendar Server utilities, see Appendix D, Calendar Server Command-Line Utilities Reference.

Troubleshooting LDAP Issues

If you are creating a hosted environment for the first time, you must create the DC tree in LDAP by adding the appropriate entries for domains, containers, users, and resources. If the DC tree does not already exist when using a Calendar Server utility, such as cscal, you might see the following error message: "Initialization failed .... exiting".

Be sure that your DC tree contains at least one (default) domain under the DC tree root. Create the DC tree structure using instructions found in Creating New Hosted Domains.

Troubleshooting Migration Utilities

Calendar Server offers several utilities for migrating calendar databases and LDAP directories. This section contains the following topics:

What to do Before Calling Technical Support

In general, if you have trouble using the migration utilities, you should contact technical support, having already gathered the following information:

Where to Find the Migration Utilities

The various migration utilities and their documentation can be found at the locations indicated in the list that follows:

Schema Migration Utility (commdirmig

This utility is bundled with Delegated Administrator, which is a separately installable component. It migrates your LDAP directory from Schema 1 to Schema 2. For information about this utility, see theSun Java System Communications Services 6 2005Q4 Schema Migration Guide.

Calendar Server 5 to Calendar Server 6 Migration Utility (cs5migrate

Technical support offers a migration bundle that includes the utility and the documentation for it.

Calendar Server Migration Utility (csmig)

This utility is installed with Calendar Server. Documentation can be found in Chapter 4, Database Migration Utilities, which includes a troubleshooting section. Running this utility is necessary if you are using hosted domains and the LDAP Calendar Lookup Database (CLD) plug-in.

Calendar Server Virtual Domain Migration Utility (csvdmig)

This utility is installed with Calendar Server. Documentation can be found in Chapter 4, Database Migration Utilities. Use this utility to prepares your calendar database and LDAP directory entries for hosted domains.

Calendar Server 2 to Calendar Server 6 Migration Utility (ics2migrate)

This utility is installed with Calendar Server. Documentation can be found in Chapter 4, Database Migration Utilities. Use this utility to migrate your Calendar Server 2 databases to be compatible with Calendar Server 5.

Netscape Calendar Server 4 to Calendar Server 5 Migration Utility (ncs4migrate)

This utility is available only from technical support. The utility package includes documentation. This utility migrates Netscape Calendar Server 4 to Calendar Server 5. These migrations tend to require special attention because of the lack of uniformity in the source database. It is not unusual for a lot of manual This utility is available only from technical support. The utility package includes documentation. This utility migrates Netscape Calendar Server 4 to Calendar Server 5. These migrations tend to require special attention. It is not unusual for a lot of work on the source file to be necessary before the utility can be run. You might consider using Professional Services to help you plan your migration.

Troubleshooting Calendar Server

This section covers various troubleshooting ideas for non-database problems. The following topics are covered in this section:


Tip –

In addition, there is a trouble shooting section for SSL in the SSL chapter:

Troubleshooting SSL


Pinging a Calendar Service

To verify that a service is listening on a specified port number, use the cstool utility ping command. Pinging a service does not verify that a service is actually running but indicates if it can accept a socket connection.

Service Options for cstool

The Calendar Server service options are:

http

HTTP Service (cshttpd)

admin

Administration Service (csadmind)

ens

Event Notification Service (enpd)


Note –

You cannot ping the DWP service (csdwpd), or Notification Service (csnotifyd).


cstool Example

For example, to ping the machine with the host name calserver to see if the cshttpd service is listening on port 80:

cstool -p 80 -h calserver ping http

By default, cstool waits 120 seconds for a response; however, you can change by value by using the -t timeout option.

For the complete utility reference material, refer to Appendix D, Calendar Server Command-Line Utilities Reference.


Note –

To run cstool, Calendar Server must be running.


ProcedureFixing start-cal Problems

If not all of the calendar services started when you issued start-cal, the ones that did start must be stopped before restarting. For example, if enpd, csnotifyd, and csadmind started, but not cshttpd, then enpd, csnotifyd, and csadmind must be stopped.

To start calendar services:

  1. Log in as a user who has administrative rights to the system where Calendar Server is running.

  2. Use start-cal to stop and then restart services. For example:

    cal_svr_base/SUNWics5/cal/sbin/start-cal

    start-cal issues a stop-cal command first before starting the various calendar services.

  3. If stop-cal fails to stop, there might be some child processes that failed to stop. To handle this, see Fixing stop-cal Problems.

Fixing stop-cal Problems

There are two separate issues to consider when Calendar Server shuts down:

ProcedureTo Stop Child Processes

After issuing stop-cal, it is possible that some child processes were not stopped. For example, stop-cal might stop the cshttpd parent process but not any cshttpd child processes. In this situation, you must stop the remaining Calendar Server processes individually, using the following procedure:

  1. Log in as a user who has administrative rights to the system where Calendar Server is running.

  2. Determine the process ID (PID) of the remaining Calendar Server processes by entering a ps command for each service:


    ps -elf | grep cs-process
    

    where cs-process is enpd, csnotifyd, csdwpd, csadmind, or cshttpd. For example:


    ps -elf | grep cshttpd
  3. Using the PID of each process that is still running, enter a kill -15 command to kill the process. For example: kill -15 9875

  4. Enter each ps command again to make sure that all Calendar Server processes are stopped.


    If a Calendar Server process is still running, 
       enter a kill -9 command to kill it. 
    For example: kill -9 9875

    Note –

    On Linux systems with Calendar Server running, if you search for calendar processes using the ps command, the results might appear confusing. In Linux, the ps command returns the list of threads running rather than the list of processes. There is no known workaround to display only the processes.


ProcedureTo Recover After an Improper Shutdown

If Calendar Server was not properly shutdown, perform the following steps:

  1. Perform the steps in the previous procedure, Fixing stop-cal Problems.

  2. Manually delete all files in the LDAP data cache database directory.

    These left over files could cause database corruptions. To delete the files:

    1. Change to the LDAP data cache directory.

      The default is /opt/SUNWics5/csdb/ldap_cache, but use the directory pointed to by the local.ldap.cache.homedir.path parameter in the ics.conf file.

    2. Remove all files in the directory.

      For example: rm *.*

    3. Check to make sure all files were removed.

      For example: ls

  3. Restart Calendar Server.

    cal_svr_base/SUNWics5/cal/sbin/start-cal

    For instructions on how to configure LDAP data caching, see Configuring Calendar Server for LDAP. For more information about the LDAP data cache, see theSun Java System Communications Services 6 2005Q4 Deployment Planning Guide.

Can't Connect to Back-end Server

  1. Ping the back-end server to see if it is responding.

    If it is responding, go to step 3. If it is not responding, determine why it is failing, and when it is functioning again, proceed to

  2. Clear the CLD cache. See Clearing the CLD Cache.

    If you are using the CLD cache option and you have updated a server name for an ics.conf parameter, you should clear the CLD cache to remove the server names. An out-of-date entry in the CLD cache can prevent a front-end server from establishing a connection to the correct back-end server or cause Calendar Server not to find a calendar after it have been moved.

  3. Restart Calendar Server.

Can’t Find Calendar

If you are using the CLD cache option and you have moved one or more calendars to different back-end servers (or changed the name of the back-end server), perform the following steps:

  1. Be sure that you followed the procedure for moving calendars found at:

    Managing User Calendars.

  2. Clear the CLD cache. See Clearing the CLD Cache.

    The CLD cache will be out of date if you moved one or more calendars to different back-end servers. To refresh it, you need to clear the cache so it will be rebuilt.

Get “Unauthorized” When Trying to Log In Using Proxy Authentication

  1. Verify that service.http.allowadminproxy is set to “yes”.

  2. Verify that the admin-user has Calendar Server administrator privileges.

  3. Verify that the admin-password is correct.

  4. Verify that the calendar-user is a valid Calendar Server user.

Troubleshooting Searches that Don’t Complete Properly

The nsslapd-sizelimit and nsLookthroughLimit attributes in your LDAP directory server configuration must be large enough so that searches complete properly. If nsSizeLimit is not large enough, truncation can occur and no results will be displayed. If nsLookthroughLimit is not large enough the search may not complete.

This section covers the following topics:

ProcedureTo Determine if Limit Attributes Have Appropriate Values

  1. To determine if these attributes are set to appropriate values, try the following command:

    ldapsearch -b "base" "(&(icscalendarowned=*user*)(objectclass=icsCalendarUser))"

    where base is the LDAP base DN of the directory server where the user and resource data for Calendar Server is located, and user is the value that an end user can enter in the search dialog in the user interface.

  2. If the LDAP server returns an error, the nsSizeLimit or the nsLookthroughLimit parameters might not be large enough.

ProcedureTo Set the Limit Attributes to Appropriate Values

The DN for these attributes is:

dn: cn=config,cn=ldbm databases,cn=plug ins,cn=config

  1. Use ldapmodify to dynamically set the value of nsLookthroughLimit.

    You do not have to stop and restart Directory Server to change this attribute.

    The default value is 5000. You might want to increase this value if searches are not reporting results. However, this could slow down the LDAP server.

    It is possible to set the limit to -1, which causes no limit to be used. However, do this with caution as it could conceivably cause the system to hang.

  2. If you want to set nsslapd-sizelimit to a higher value, you must perform the following steps:

    1. Stop the Directory Server.

    2. Edit the dse.ldif file.

    3. Restart the Directory Server.


      Note –

      For information on how to use ldapmodify and edit the dse.ldif file, see Directory Server documentation found at:

      http://docs.sun.com/coll/1316.1


Turning Off Those Annoying Daily Messages from csstored

The start-cal command launches the csstored process by default, even if it is unconfigured. An unconfigured csstored process will emit a message stating it is unconfigured, every 24 hours, on every machine where csstored is running.

Disable the message by preventing csstored from running unconfigured. To disable the csstored process from running, set the following ics.conf parameter as shown for each machine where the message is being generated:

service.store.enable=”no”

Be careful not to disable the process on the machines where you have csstored configured to make automatic backups.

Dealing with Database Issues

This section covers various issues involving the calendar server databases:

Finding Berkeley Database Tools

Many of the troubleshooting steps you will want to take require having access to the Berkeley database utility programs. While a version of these utility programs is available in the Calendar Server bundle, they are not supported. You might want to obtain more information directly from Sleepycat Software (http://www.sleepycat.com.

This section covers the following topics:

To Access the Berkeley Database Utilities

Set and export the LD_LIBRARY_PATH environment variable to reflect the following directory:

cal_svr_base/SUNWics5/cal/tools/unsupported/bin/

List of Available Tools

The following table lists some of the commonly used Berkeley database tools (utility programs).

Berkeley Database Tools 

Description 

db_archive

Writes the path names of log files that are no longer in use to the standard output, one pathname per line. 

db_checkpoint

A daemon process that monitors the database log and periodically calls the checkpoint routine to checkpoint it. 

db_deadlock

Traverses the database environment lock region and aborts a lock request each time it detects a deadlock or a lock request that has timed out. 

db_dump

Writes the specified file to standard output in a flat-text format understood by the db_load utility.

db_load

Reads from the standard input and loads it into the database file specified. If the file does not already exist it creates it. 

db_printlog

Debugging utility that dumps log files in human-readable format. 

db_recover

Restores the database to a consistent state after an unexpected application, database, or system failure. 

db_stat

Displays statistics for the database environment. 

db_verify

Verifies the structure of one or more files and the databases they contain. 

ProcedureTo Detect and Fix Database Deadlocks

If the Berkeley database is in a deadlock state, you must reset the database. It is important to detect this condition as early as possible.

To enable the system to periodically check the databases to detect a deadlock state and inform the Administrator:

  1. Log in as an administrator with permission to change the configuration.

  2. Change to the /etc/opt/SUNWics5/cal/config directory.

  3. Save your old ics.conf file by copying and renaming it.

  4. Edit the ics.conf, if necessary, to have the following value:

    local.caldb.deadlock.autodetect=”yes”


    Note –

    When this parameter is set to “yes”, the db_deadlock daemon is launched that will monitor the lock region.


Detecting Database Corruption

Calendar database corruption can be caused by various reasons: system resource contention, hardware failures, application errors, database failures, and of course human error. This section describes how to detect calendar database corruption:

Database Corruption Basics

No one can guarantee corruption free databases. But you can minimize data loss and operational downtime. Closely monitoring the database and calendar server is key to detecting corruption early. Frequent and complete backups are the key to recovering from corruption once it is found.

There are two levels of corruption possible in a calendar database:

Monitoring Log Files

Monitor the Calendar Server log files, including the alarm logs, for any error messages that might indicate database corruption. For information about the log files, refer to Using Calendar Server Log Files.

You should inspect the log files on a regular basis for ALERT, CRITICAL, ERROR, and WARNING level errors and, if found, examine the events for possible problems with the operation of Calendar Server. The NOTICE and INFORMATION level log events are generated during normal operation of Calendar Server and are provided to help you monitor server activity.

Never remove any transaction log files in the database directory. The transaction log files contain the transaction updates (additions, modifications, or deletions), and removing them can corrupt the calendar database beyond recovery.


Note –

When requesting technical support for Calendar Server, you might be asked to provide the log files for help in resolving problems.


Using csmonitor

Use the csmonitor utility to monitor Calendar Server. It will send an alert email to the administrator if it detects problems, such as more than one transaction log file, or a shortage of disk space for the calendar database. For more information, see csmonitor.

ProcedureTo Check for Calendar Database Corruption

Use the check command to scan for corruptions in the calendar database, including calendar properties (calprops) and events and todos (tasks). If the check command finds an inconsistency that cannot be resolved, it reports the situation in its output.

The check command does not check for corruption in the alarm or group scheduling engine (GSE) databases.

  1. Log in as a user who has administration rights to the system where Calendar Server is installed.

  2. Calendar Server can be either running or stopped; however, if possible, stop Calendar Server.

  3. Make a copy of your calendar database, if you haven’t already done so.

    Copy only the database (.db) files. You don’t need to copy any share (__db.*) or log (log.*) files.

  4. Change to the cal_svr_base/SUNWics5/cal/sbin directory.

    For example, on Solaris Operating Systems for the default directory, enter:


    cd /opt/SUNWics5/cal/sbin
  5. Run the check command on the copy of your calendar database:


    ./csdb check dbdir /tmp/check.out 

    If you don’t specify dbdir, check uses the database in the current directory.

    The check command can generate a lot of information, so consider redirecting all output, including stdout and stderr, to a file (as shown in the example).

  6. When check has finished, review the output file. If your database is corrupted, run the rebuild command.

    (See Rebuilding a Corrupted Calendar Database.)

Preventing Service Interruptions When Your Database is Corrupted (Read-only Mode)

This sections covers how to keep your corrupted database accessible while you are in recovery mode and includes the following topics:

Using Read-only Mode

If you are encountering database corruption, one way to prevent service interruptions is to put your database in read-only mode. This mode allows end users to read database entries, but does not allow additions, modifications, or deletions. If an end user attempts to add, modify or delete any calendar data, the system gives an error message. In addition, administrator tools that add, modify or delete calendar events and todos will not work while the database is in read-only mode.


Note –

If the database is corrupted to the point that it can’t be read, you must interrupt service long enough to restore a backup. The quickest way to restore a backup is to have a good hot backup. See Before You Restore.


ProcedureTo Put a Database in Read-only Mode

  1. While it isn’t necessary, you might choose to stop calendar services momentarily to prevent further corruption of the database.

    To stop calendar services:

    cal_svr_base/SUNWics5/cal/sbin/stop-cal

  2. At a command line, change to the directory where the ics.conf is located:

    cd /etc/opt/SUNWics5/config

  3. Specify read-only mode for the calendar database:

    caldb.berkeleydb.readonly=”yes”

  4. When you have completed editing the ics.conf file, restart Calendar Server:

    cal_svr_base/SUNWics5/cal/sbin/start-cal

    You must restart the services in order for the ics.conf changes to take effect.

Handling Common Database Failures

This section covers a few of the common database failures and includes some suggested remedies. It contains the following topics:

Procedurecsadmind Won’t Start or Crashes During Startup

Since csadmind is the service that handles both the group scheduling engine (GSE) and the alarm dispatch engine, this could have been caused by offending entries in the GSE queue or the alarm queue.

Remedies:

  1. If csadmind is not running, issue stop-cal immediately.

    Leaving calendar server running could cause transaction logs to accumulate, which could further corrupt the database, and could take much longer to reconcile the transaction log files to the database.

  2. Try restarting csadmind again (issue start-cal again).

    If it starts successfully, make sure the two queues are functioning by:

    1. Checking the GSE queue using csschedule.

    2. Checking the alarm queue using dbrig.

      For instructions on running csschedule and dbrig, see Appendix D, Calendar Server Command-Line Utilities Reference.

  3. If csadmind crashes with a dump, analyze the pstack.

    If you notice any GSE related functions in the trace (they will have the letters GSE in them), look at the first entry in the GSE queue and the referenced entry in the events database. Most of the time, the event referred to in the GSE entry is the offending entry. To fix this problem:

    1. Remove the GSE entry using csschedule.

    2. Remove the offending event from the database using cscomponents.

      For instructions on running csschedule and cscomponents, see Appendix D, Calendar Server Command-Line Utilities Reference.

  4. If the entries are not corrupted, then it could be a special case that the calendar server could not handle.

    Take the following steps:

    1. Take a calendar environment snapshot of the corrupted database, and contact customer support.

      To create an environmental backup:

      1. Use the db_checkpoint utility found at:

        cal_svr_base/SUNWics5/cal/tools/unsupported/bin/db_checkpoint

      2. Run db_archive -s.

        Use the -s option to identify all the database files and copy them to a removable medium, such as CD, or DVD, or tape.

      3. Run db_archive -l.

        Use the -loption to identify all the log files and copy unapplied log files to a removable-medium device.

    2. To avoid service interruptions, place your calendar database into a read-only state temporarily, and revert to a hot backup copy.

      • Placing your calendar database into a read-only state temporarily prevents any add, modify or delete transactions from taking place. End users will get an error message when they try to add, modify or delete any calendar data. Administrator tools that add, modify or delete calendar events and todos also will not work while the database is in read-only mode.

        To put your calendar database in read-only mode, edit the ics.conf file and set the following parameter to “yes”, as shown:

        caldb.berkeleydb.readonly=”yes”

      • Revert to a hot backup copy, using the instructions found in Restoring an Automatic Backup Copy.

        With csstored configured and enabled, a hot backup is available that should be within minutes of being up-to-date. You should always verify your hot backup copy to make sure it is not corrupt also. (Run db_verify.)

  5. If all else fails, perform the dump and reload procedure to see if it can salvage the database.

    This procedure is described in Using the Dump and Load Procedure to Recover a Calendar Database.

ProcedureServices Hung, and End Users Can’t Connect–Orphaned Locks

This condition may be caused by a control thread, which holds a Berkeley DB database page lock, quitting without releasing the lock. To confirm the problem, run pstack on cshttpd processes and csadmind. (pstack is a standard UNIX utility found at: /usr/bin/pstack) It should show threads that are waiting to acquire a lock.

To fix the problem, restart Calendar Server, as follows:

  1. Change to the directory where start-cal resides.

    cd cal_svr_base/SUNWics5/cal/sbin

  2. Issue the start-cal command.

    ./start-cal

Procedurecsdb rebuild Never Finishes–Database Looping

Database looping is usually caused by corruption in the database files. Since it is a database corruption, it can be unrecoverable. There are several options:

  1. Revert to the hot backup.

    If the corruption occurred recently, you can use one of your hot backups.

  2. Use your catastrophe archival recovery process.

    For a suggested process, see Restoring an Automatic Backup Copy.

  3. Use the dump and reload procedure, Using the Dump and Load Procedure to Recover a Calendar Database.

Rebuilding a Corrupted Calendar Database

This section describes how to use the csdb rebuild command and contains the following topics:

rebuild Overview

The rebuild command scans a calendar database and checks the calendar properties (calprops) events and todos (tasks) for corruption. If the rebuild command finds an inconsistency, it generates a rebuilt calendar database (.db files) in the cal_svr_base/SUNWics5/cal/sbin/rebuild_db directory.

The rebuild command without the -g option rebuilds all databases except the group scheduling engine (GSE) database. To also rebuild the GSE database, include the -g option.

To determine if the GSE database has any entries, run the csschedule -v list command and then let the GSE finish processing the entries before you run the rebuild command.

ProcedureTo Rebuild a Calendar Database

  1. Log in as a user who has administration rights to the system where Calendar Server is installed.

  2. Stop Calendar Server.

  3. Make a copy of your calendar databases, placing them into the /tmp/db directory.

    Copy the database (.db) files and the log (log.*) files. You don’t need to copy any share (__db.*) files.

  4. Change to the cal_svr_base/SUNWics5/cal/sbin directory.

    For example, on Solaris Operating Systems, for the default directory, enter:


    cd /opt/SUNWics5/cal/sbin

    Note –

    If disk space is a problem for the sbin directory, run the rebuild command in a different directory.


  5. Run the rebuild command on the copy of your calendar database:


    ./csdb rebuild /tmp/db /tmp/

    If you don’t specify a database path, rebuild uses the current directory. The /tmp/ parameter species the destination directory for the rebuilt database.

    To also rebuild the GSE database, include the -g option.

    The rebuild command can generate a lot of information, so consider redirecting all output, including stdout and stderr, to a file.


    Note –

    Always rebuild your calendar database using the latest backup copy.

    However, if you have experienced a significant loss of data and you have periodically backed up your database and have more than one copy available, rebuild from the latest copy to the oldest one. (The only drawback is that calendar components that were deleted will reappear in the rebuilt database.)

    For example, if you have three sets of backup calendar database files in directories db_0601, db_0615, and db_0629, run the rebuild command in the following sequence:


    ./csdb rebuild db_0629 
    ./csdb rebuild db_0615 
    ./csdb rebuild db_0601

    The rebuild command then writes the rebuilt database to the cal_svr_base/SUNWics5/cal/sbin/rebuild_db directory.


  6. When rebuild has finished, review the output in the rebuild.out file.

    If the rebuild was successful, the last line in the rebuild.out file should be:


    Calendar database has been rebuilt
  7. After you have verified that rebuild was successful in the previous step, copy the rebuilt database (.db) files from the rebuild_db directory to your production database.

  8. If you have any share (__db.*) or log (log.*) files from the corrupted database, move them to another directory.

  9. Restart Calendar Server.

Sample Rebuild Output

The following example shows the command and the output that it generated:


# ./csdb -g rebuild
Building calprops based on component information.
Please be patient, this may take a while...
Scanning events database...
512 events scanned
Scanning todos database...
34 todos scanned
Scanning events database...
512 events scanned
Scanning todos database...
34 todos scanned
Scanning deletelog database...
15 deletelog entries scanned
Scanning gse database...
21 gse entries scanned
Scanning recurring database...
12 recurring entries scanned
Successful components db scan
Calendar database has been rebuilt
Building components based on calprops information.
Please be patient, this may take a while...
Scanning calprops database to uncover events...
25 calendars scanned
Scanning calprops database to uncover todos...
25 calendars scanned
Successful calprops db scan
Calendar database has been rebuilt
            

Note –

The preceding sample output shows the events and the todos databases scanned twice each. This is not an error. It scans the first time to verify the information in the calendar properties database and then scans again to make sure calendar properties database is accessible.


Using the Dump and Load Procedure to Recover a Calendar Database

This sections contains the following topics:

Dump and Load Overview

Use the dump and load procedure to try to recover a corrupted database. The dump and load procedure uses the Berkeley database db_dump and db_load utilities, which Calendar Server includes in the following directory:


cal_svr_base/SUNWics5/cal/tools/unsupported/bin

The db_dump utility reads a database file and writes the database entries to an output file, using a format that is compatible with the db_load utility.

For documentation about the db_dump and db_load utilities, refer to the Sleepycat Software Web site:

http://www.sleepycat.com/docs/utility/index.html

Your success in recovering a database using the db_dump and db_load utilities depends on the degree of corruption of your database. You might need to try several db_dump options before you successfully recover your database. If your database is severely corrupted, however, recovery might not be possible, and you might need to revert to the last good hot backup or archive backup of your database.


Note –

Before you perform the dump and load procedure, your calendar database must be Berkeley DB version 3.2.9, or later. If you have an earlier version, first run the cs5migrate utility to upgrade your calendar database.

For the most up to date version of cs5migrate, call Sun technical support.


ProcedureTo Perform the Dump and Load Procedure

  1. Log in as the user and group under which Calendar Server is running, such as icsuser and icsgroup, or as superuser (root).

  2. Stop Calendar Server, if necessary.

  3. Backup your corrupted database using a utility such as csbackup, the Sun StorEdge Enterprise BackupTM software, or Legato Networker®.

    For more information refer to Chapter 17, Backing Up and Restoring Calendar Server Data.

  4. Dump each corrupted database file using the db_dump utility.

    The database files are ics50calprops.db, ics50journals.db, ics50alarms.db, ics50events.db, ics50todos.db, and ics50gse.db.

    Run db_dump using the following options, in order, until your database is recovered (or until you determine that the database can’t be recovered):

    • No options for minor database corruption.

    • -r option for moderate database corruption.

    • -R option for severe database corruption. The -R option dumps more data than the -r option, including partial and deleted records, from the corrupted database.

      For example, to run db_dump with the -r option:


      db_dump -r ics50events.db \> ics50events.db.txt
  5. Load the output file into a new database file using the db_load utility.

    For example:


    db_load new.ics50events.db < ics50events.db.txt

    If db_load reports an odd number of keys or data entries, edit the db_dump output file, and remove the odd key or data entries. Then run db_load again.

  6. Repeat the previous two steps for the other corrupted database files.

    That is, run db_dump for the other corrupted database files.

  7. Rebuild the recovered database files using the csdb rebuild command, as described in Rebuilding a Corrupted Calendar Database.

    When rebuild has finished, review the output in the output file. If the rebuild was successful, the last line in the rebuild.out file should be:


    Calendar database has been rebuilt

    If the csdb rebuild command was not successful, dump your database using the next db_dump option (-r or -R).

    If the db_dump -R option does not recover your corrupted database, contact your Sun Microsystems technical support or sales account representative for assistance. In the meantime, you might need to revert to the last good backup of your database.

Restoring an Automatic Backup Copy

If you have used the automatic backup feature described in Chapter 10, Configuring Automatic Backups (csstored),, you can use the hot backup copy when your live database is corrupted.

This sections covers how to restore the two different automatic backups:

Before You Restore

Before you restore a backup, be sure that you have:

ProcedureTo Restore a Hot Backup

Hot backups should be your first choice of backup when your live database is corrupted. To restore a hot backup, follow these steps:

  1. Identify any log files that were unapplied or open for writing in the corrupted live database directory.

  2. Close the log that was open for writing. It contains the most recent transactions.

  3. Create a new (recovery) directory.

  4. Copy the current hot backup copy into the new recovery database directory.

  5. Copy the log.* files from your corrupted live database directory into your new recovery database directory.

  6. If you are keeping an archive copy of the database, copy the logs that had not been applied to the live database into the archive directory, so your archive backup copy will be complete.

  7. Run db_recover with the -c -h options specified against the new recovery database.

    For example, if your new recovery directory is called recoverydb, then the command would be as follows:

    db_recover -c -h recoverydb

  8. Leave the log.* files in the new recovery directory.

    The db_recover program applied the log files to the new recovery databases, but starting with version 42, the Berkeley DB expects them to remain.

  9. Run db_verify against the database files in the new recovery directory.

    For instructions, see To Check for Calendar Database Corruption.

  10. Run csdb -v list against the new recovery directory.

  11. If the new recovery directory passed all three preceding recovery steps, replace the old corrupted live database with the new recovery database.

  12. Copy the new live database into your hot backup directory to function as the new snapshot.

    All new logs will be applied to this copy until the next regular snapshot is taken.

  13. Start Calendar Server.

  14. If the new recovery directory failed any of the steps, identify an uncorrupted older hot backup as follows:

    1. Working backward through your hot backups, find the most recent copy that is not corrupted by running db_verify and csdb -v list on each in turn.

    2. The first hot backup copy that passes can be restored to your live database directory.

      Replace the corrupted live database with the clean hot backup, as described in To Restore a Hot Backup. (Be sure to read Before You Restore first.)

    3. If none of your hot backups work and you do not have archive backups to try, call technical support. If you do have archive backups, follow the procedure that followsTo Restore an Archive Backup. (See also, Before You Restore.)

ProcedureTo Restore an Archive Backup

If you do not have an uncorrupted hot backup, but have archive backups and their transaction logs, you can restore the most current uncorrupted version of the archived database by performing the following steps:

  1. Identify any log files that were unapplied or open for writing in the corrupted live database directory.

  2. Close the log that was open for writing. It contains the most recent transactions.

  3. Create a new (recovery) directory.

  4. Copy the most recent archive copy and its log files into the new recovery database directory.

  5. Copy any unapplied log.* files from your corrupted live database directory into your new recovery database directory.

  6. Run db_recover with the -c -h options specified against the new recovery database.

    For example, if your new recovery directory is called recoverydb, then the command would be as follows:

    db_recover -c -h recoverydb

  7. Leave the log.* files in the new recovery directory.

    The db_recover program applied the log files to the new recovery databases, but starting with version 4.2, Berkeley DB expects the log files to still be there.

  8. Run db_verify against the database files in the new recovery directory.

    For instructions, see To Check for Calendar Database Corruption.

  9. Run csdb -v list against the new recovery directory.

  10. If the new recovery directory passed all three preceding recovery steps, replace the old corrupted live database with the new recovery database.

  11. Copy the new live database into your hot backup directory to function as the new snapshot.

  12. Start Calendar Server.

  13. If the new recovery directory failed any of the steps, identify an uncorrupted older archive backup as follows:

    1. Working backward through your archive backup copies, find the most recent copy that is not corrupted by running the three recovery programs against each of them in turn: db_recover -c-h, db_verify and csdb -v list.

    2. The first archive copy that passes can be restored to your live database directory.

      Replace the corrupted live database with the clean archive backup, as shown in To Restore an Archive Backup.

    3. If none of your archive backups work, call technical support.

Repairing Custom Backup Scripts

This section includes the following topics:

Berkeley Tools Now Compiled with a Dynamic Library

If you have created a custom backup script using the Berkeley database tools, such as db_recover, you may find that it will no longer work after upgrading to Calendar Server. The reason for this is that the earlier versions of Calendar Server compiled the tools with a static library. The tools are now compiled with a dynamic library, libdb-4.2.so.

To Repair a Custom Backup Script

To use the new dynamic library with your existing custom scripts, set the following global variable as shown:

LD_LIBRARY_PATH=libdb-4.2.so