test

Sun Java System Calendar Server 6.3 Administration Guide

Chapter 22 Troubleshooting Calendar Server 6.3 Software

This chapter covers how to set up logging and what to do for some common problems.

The chapter contains the following topics:

22.1 Turning on Debugging Information for Calendar Server 6.3 Software

This section contains conceptual information and instructions on using logs and debugging to troubleshoot problems with your Calendar Server deployment.

While there is no one ics.conf parameter that puts the whole system in “debug mode”, this section describes some ways to get useful debug information:

Note –

Be sure to turn off excess logging and monitoring when not needed as it will negatively impact performance.

22.1.1 Increase Logging Level

Use the parameter shown in the following table to increase the verbosity of logging:

Parameter 

Description and Default Value 

logfile.loglevel

Set to DEBUG to get all levels logged, including CRITICAL, ALERT, ERROR, WARNING, NOTICE, and INFORMATION. This applies to all logs.

22.1.2 Enable Logging Access to the LDAP Cache

To log all accesses of the LDAP data cache and print out the log (report) set the ics.conf parameters shown in the following table:

Parameter 

Description and Default Value 

local.ldap.cache.stat.enable

Specifies whether or not to log the access to the LDAP data cache and to print statistics in the log file. The default is “no” (no statistics logged). Set to “yes” to enable logging of statistics.

For performance enhancement, use this only in debug mode. 

local.ldap.cache.stat.interval

Specifies the interval in seconds when each statistics report is written to the log file. The default is “1800” seconds (30 minutes).

This is only active if logging is enabled. Decreasing the interval can help you pinpoint problems. Increasing the interval helps decrease system load. 

22.1.3 Clearing the LDAP Cache

There is currently no logic in Calendar Server to expire LDAP cache data. You must manually remove the contents of the ldap_cache directory and restart Calendar Server.

ProcedureTo Clear the LDAP Cache

  1. Stop Calendar Server.

  2. Remove all files in the /var/opt/SUNWics5/csdb/ldap_cache directory, but do not remove the ldap_cache directory itself.

  3. Restart Calendar Server.

22.1.4 WCAP Command and HTTP Access Logging

Two configuration parameters that facilitate debugging enable logging of incoming commands and HTTP accesses. Add one or both of these parameters to the ics.conf file to activate the logging:

Caution – Caution –

The log files can grow very quickly and fill the available disk space. Monitor them carefully to avoid problems. Choose a period of low activity on your systems to run with these commands enabled. Performance will drop noticeably if run during peak hours. Always disable the two commands when you are finished troubleshooting.

22.1.5 Monitor the System Using the Calendar Server 6.3 csstats Utility

Use the csstats list command to display statistical information from counter objects defined in the counter.conf file.

For more information on the csstats utility, see Appendix D, Calendar Server Command-Line Utilities Reference.

22.2 Troubleshooting LDAP Issues

This section contains conceptual information on troubleshooting LDAP issues.

If you are creating a multiple domain environment for the first time, you must create the DC tree in LDAP by adding the appropriate entries for domains, containers, users, groups and resources. If the DC tree does not already exist when using a Calendar Server utility, such as cscal, you might see the following error message: "Initialization failed .... exiting".

Be sure that your DC tree contains at least one (default) domain under the DC tree root. Create the DC tree structure using instructions found in 13.2 Creating New Calendar Server Domains.

22.3 Troubleshooting Migration Utilities

Calendar Server offers several utilities for migrating calendar databases and LDAP directories.

This section contains the following topics:

22.3.1 What to do Before Calling Technical Support

In general, if you have trouble using the migration utilities, you should contact technical support.

Before calling, gather the following information:

22.3.2 Where to Find the Migration Utilities

The various migration utilities and their documentation can be found at the locations indicated in the list that follows:

Schema Migration Utility (commdirmig

This utility is bundled with Delegated Administrator, which is a separately installable component. It migrates your LDAP directory from Schema version 1 to Schema version 2. For information about this utility, see theSun Java Communications Suite 5 Schema Migration Guide.

Calendar Server 6.2 to 6.3 migration utilitycsmigrate

This utility can be found in the sbin directory after the software is installed.

Calendar Server 5 to Calendar Server 6.2 migration utility (cs5migrate

This utility can be found in the sbin directory after the software is installed.

Calendar Server multiple domain database preparation utility (csmig)

This utility can be found in the sbin directory after the software is installed.

Documentation for this utility can be found in Chapter 3, Database Migration Utilities for Calendar Server 6.3, which includes a troubleshooting section.

Calendar Server non-domain to multiple domain migration utility (csvdmig)

This utility can be found in the sbin directory after the software is installed.

Documentation for this utility can be found in Chapter 3, Database Migration Utilities for Calendar Server 6.3. Use this utility to prepares your calendar database and LDAP directory entries for multiple domains.

Note –

Always run csmig before csvdmig.

Calendar Server 2 to Calendar Server 6 Migration Utility (ics2migrate)

This utility is installed with Calendar Server. Documentation can be found in Chapter 3, Database Migration Utilities for Calendar Server 6.3. Use this utility to migrate your Calendar Server 2 databases to be compatible with Calendar Server 5.

Netscape Calendar Server 4 to Calendar Server 5 Migration Utility (ncs4migrate)

This utility is available only from technical support. The utility package includes documentation. This utility migrates Netscape Calendar Server 4 to Calendar Server 5. These migrations tend to require special attention because of the lack of uniformity in the source database. It is not unusual for a lot of manual This utility is available only from technical support. The utility package includes documentation. This utility migrates Netscape Calendar Server 4 to Calendar Server 5. These migrations tend to require special attention. It is not unusual for a lot of work on the source file to be necessary before the utility can be run. You might consider using Professional Services to help you plan your migration.

22.4 Non-Database Troubleshooting for Calendar Server

This section covers various troubleshooting ideas for non-database problems.

The following topics are covered in this section:

Tip –

In addition, there is a trouble shooting section for SSL in the SSL chapter:

7.2 Troubleshooting SSL for Calendar Server 6.3 Software

22.4.1 One cshttpd Process is Accepting Too Many Connections and Taking 100% of CPU Time

If one cshttpd process is accepting too many connections and taking 100% of CPU time, you might have disabled load balancing. To re-enable it, change the value of ics.conf parameter service.http.loadbalancing to "yes".

ProcedureFixing start-cal Problems

If not all of the calendar services started when you issued start-cal, the ones that did start must be stopped before restarting. For example, if enpd, csnotifyd, and csadmind started, but not cshttpd, then enpd,, csnotifyd, and csadmind must be stopped.

To start calendar services:

  1. Log in as an administrator with configuration privileges.

  2. Issue the stop-cal command.

  3. If the stop-cal command fails to stop all Calendar Server services, there might be some child processes still running. To handle this, see 22.4.2 Fixing stop-cal Problems.

  4. Once you are sure all Calendar Server processes are stopped, use the start-cal command to start all services. For example:

    cal-svr-base/SUNWics5/cal/sbin/start-cal

22.4.2 Fixing stop-cal Problems

This section contains some conceptual information and instructions for fixing stop-cal problems.

There are two separate issues to consider when Calendar Server shuts down:

ProcedureTo Stop Child Processes

After issuing stop-cal, it is possible that some child processes were not stopped. For example, stop-cal might stop the cshttpd parent process but not any cshttpd child processes. In this situation, you must stop the remaining Calendar Server processes individually, using the following procedure:

  1. Log in as a user who has administrative rights to the system where Calendar Server is running.

  2. Determine the process ID (PID) of the remaining Calendar Server processes by entering a ps command for each service:

    ps -elf | grep cs-process

    where cs-process is enpd, csnotifyd, csdwpd, csadmind, or cshttpd. For example:

    ps -elf | grep cshttpd

  3. Using the PID of each process that is still running, enter a kill -15 command to kill the process. For example: kill -15 9875

  4. Enter each ps command again to make sure that all Calendar Server processes are stopped.


    If a Calendar Server process is still running, 
   enter a kill -9 command to kill it. 
For example: kill -9 9875
    Note –

    On Linux systems with Calendar Server running, if you search for calendar processes using the ps command, the results might appear confusing. In Linux, the ps command returns the list of threads running rather than the list of processes. There is no known workaround to display only the processes.

ProcedureTo Recover After an Improper Shutdown

If Calendar Server was not properly shutdown, perform the following steps:

  1. Perform the steps in the previous procedure, 22.4.2 Fixing stop-cal Problems.

  2. Manually delete all files in the LDAP data cache database directory.

    These left over files could cause database corruptions. To delete the files:

    1. Change to the LDAP data cache directory.

      The default is /opt/SUNWics5/csdb/ldap_cache, but use the directory pointed to by the local.ldap.cache.homedir.path parameter in the ics.conf file.

    2. Remove all files in the directory.

      For example: rm *.*

    3. Check to make sure all files were removed.

      For example: ls

  3. Restart Calendar Server.

    cal-svr-base/SUNWics5/cal/sbin/start-cal

    For instructions on how to configure LDAP data caching, see 4.8 Configuring LDAP for Calendar Server Version 6.3. For more information about the LDAP data cache, see theSun Java Communications Suite 5 Deployment Planning Guide.

22.4.3 Can't Connect to Back-End Server

  1. Ping the back-end server to see if it is responding.

    If it is not responding, determine why it is failing. When it is functioning again, proceed to the next step in this task.

  2. Clear the CLD cache. See 12.5 Clearing the CLD Cache in Calendar Server Version 6.3.

    If you are using the CLD cache option and you have updated a server name for an ics.conf parameter, you should clear the CLD cache to remove the server names. An out-of-date entry in the CLD cache can prevent a front-end server from establishing a connection to the correct back-end server or cause Calendar Server not to find a calendar after it have been moved.

  3. Stop the server with the stop-cal command.

  4. Restart Calendar Server using start-cal.

22.4.4 Can’t Find Calendar

If you are using the CLD cache option and you have moved one or more calendars to different back-end servers (or changed the name of the back-end server), might not be able to see the calendars on the new server.

    If this happens, perform the following steps:

  1. Clear the CLD cache. See 12.5 Clearing the CLD Cache in Calendar Server Version 6.3.

    The CLD cache will be out of date if you moved one or more calendars to different back-end servers. To refresh it, you need to clear the cache so it will be rebuilt.

  2. If that fails, confirm that you followed the correct procedure for moving calendar. This information can be found at:

    15.6 Managing User Calendars.

    Then, clear the cache.

22.4.5 Can't Create Calendar on Back-End Machine

If you try to create a calendar on a designated back-end machine, and you get the following error message: Invalid DWP Host Server, it means one of two things. Either your server is not configured properly, or the calendar owner has already been assigned to a different back-end server.

This section contains information on how to fix these two problems:

22.4.5.1 Back-End Machine Not Configured Properly

Look at the ics.conf file for the back-end server in question.

Verify that the following settings exist: 

service.dwp.enable = "yes"
caldb.cld.type = "directory"
local.hostname = "back-end hostname"

22.4.5.2 Calendar Owner Assigned to a Different Back-End Machine

Look at the user's LDAP entry and see if there is an icsDWPHost attribute present. The value of icsDWPHost must match the back—end server name on which you are attempting to create the calendar. You can not create a calendar for this user on a different back-end server.

22.4.6 Get “Unauthorized” When Trying to Log In Using Proxy Authentication

This section contains suggestions for possible reasons for failure. Follow the suggested steps and retry the login.

  1. Perform one or more of the following steps to fix this error:

    • Verify that service.http.allowadminproxy is set to “yes”.

    • Verify that the admin-user has Calendar Server administrator privileges.

    • Verify that the admin-password is correct.

    • Verify that the calendar-user is a valid Calendar Server user.

  2. Retry the log in.

22.4.7 Troubleshooting Searches that Don’t Complete Properly

This section contains conceptual information and instructions for troubleshooting searches that don't complete properly.

The nsslapd-sizelimit and nsLookthroughLimit attributes in your LDAP Directory Server configuration must be large enough so that searches complete properly. If nsSizeLimit is not large enough, truncation can occur and no results will be displayed. If nsLookthroughLimit is not large enough the search may not complete.

This section covers the following topics:

ProcedureTo Determine if Limit Attributes Have Appropriate Values

  1. To determine if these attributes are set to appropriate values, try the following command:

    ldapsearch -b "base" "(&(icscalendarowned=*user*)(objectclass=icsCalendarUser))"

    where base is the LDAP base DN of the directory server where the user and resource data for Calendar Server is located, and user is the value that an end user can enter in the search dialog in the user interface.

  2. If the LDAP server returns an error, the nsSizeLimit or the nsLookthroughLimit parameters might not be large enough.

ProcedureTo Set the Limit Attributes to Appropriate Values

The DN for these attributes is:

dn: cn=config,cn=ldbm databases,cn=plug ins,cn=config

  1. Use ldapmodify to dynamically set the value of nsLookthroughLimit.

    You do not have to stop and restart Directory Server to change this attribute.

    The default value is 5000. You might want to increase this value if searches are not reporting results. However, this could slow down the LDAP server.

    It is possible to set the limit to -1, which causes no limit to be used. However, do this with caution as it could conceivably cause the system to hang.

  2. If you want to set nsslapd-sizelimit to a higher value, you must perform the following steps:

    1. Stop the Directory Server.

    2. Edit the dse.ldif file.

    3. Restart the Directory Server.

      Note –

      For information on how to use ldapmodify and edit the dse.ldif file, see Directory Server documentation found at:

      http://docs.sun.com/coll/1316.1

22.5 Dealing with Calendar Server Database Issues

This section covers various issues involving the Calendar Server (Berkeley Database) databases:

This section contains the following topics:

22.5.1 Finding Berkeley Database Tools

Many of the troubleshooting steps you will want to take require having access to the Berkeley database utility programs. While a version of these utility programs is available in the Calendar Server bundle, they are not supported. You might want to obtain more information directly from Sleepycat Software (http://www.oracle.com/database/berkeley-db/index.html.

This section covers the following topics:

22.5.1.1 To Access the Berkeley Database Utilities

Set and export the LD_LIBRARY_PATH environment variable to reflect the following directory:

cal-svr-base/SUNWics5/cal/tools/unsupported/bin/

22.5.1.2 List of Available Tools

The following table lists some of the commonly used Berkeley database tools (utility programs).

Berkeley Database Tools 

Description 

db_archive

Writes the path names of log files that are no longer in use to the standard output, one pathname per line. 

db_checkpoint

A daemon process that monitors the database log and periodically calls the checkpoint routine to checkpoint it. 

db_deadlock

Traverses the database environment lock region and aborts a lock request each time it detects a deadlock or a lock request that has timed out. 

db_dump

Writes the specified file to standard output in a flat-text format understood by the db_load utility.

db_load

Reads from the standard input and loads it into the database file specified. If the file does not already exist it creates it. 

db_printlog

Debugging utility that dumps log files in human-readable format. 

db_recover

Restores the database to a consistent state after an unexpected application, database, or system failure. 

db_stat

Displays statistics for the database environment. 

db_verify

Verifies the structure of one or more files and the databases they contain. 

ProcedureDetecting and Fixing Database Deadlocks

If the Berkeley database is in a deadlock state, you must reset the database. It is important to detect this condition as early as possible.

To enable the system to periodically check the databases to detect a deadlock state and inform the Administrator:

  1. Log in as an administrator with permission to change the configuration.

  2. Stop Calendar Server services by issuing the stop-cal command.

  3. Change to the /etc/opt/SUNWics5/cal/config directory.

  4. Save your old ics.conf file by copying and renaming it.

  5. Edit the ics.conf, if necessary, to have the following value:

    local.caldb.deadlock.autodetect=”yes”

    Note –

    When this parameter is set to “yes”, the db_deadlock daemon is launched that will monitor the lock region.

22.5.2 Detecting Database Corruption

Calendar database corruption can be caused by various reasons: system resource contention, hardware failures, application errors, database failures, and of course human error. This section describes how to detect calendar database corruption:

22.5.2.1 Database Corruption Basics

No one can guarantee corruption free databases. But you can minimize data loss and operational downtime. Closely monitoring the database and calendar server is key to detecting corruption early. Frequent and complete backups are the key to recovering from corruption once it is found.

There are two levels of corruption possible in a calendar database:

22.5.2.2 Monitoring Log Files

Monitor the Calendar Server log files, including the alarm logs, for any error messages that might indicate database corruption.

You should inspect the log files on a regular basis for ALERT, CRITICAL, ERROR, and WARNING level errors and, if found, examine the events for possible problems with the operation of Calendar Server. The NOTICE and INFORMATION level log events are generated during normal operation of Calendar Server and are provided to help you monitor server activity.

Never remove any transaction log files in the database directory. The transaction log files contain the transaction updates (additions, modifications, or deletions), and removing them can corrupt the calendar database beyond recovery.

Note –

When requesting technical support for Calendar Server, you might be asked to provide the log files for help in resolving problems.

ProcedureTo Check for Calendar Database Corruption

Use the check command to scan for corruptions in the calendar database, including calendar properties (calprops) and events and todos (tasks). If the check command finds an inconsistency that cannot be resolved, it reports the situation in its output.

The check command does not check for corruption in the alarm or group scheduling engine (GSE) databases.

  1. Log in as a user who has administration rights to the system where Calendar Server is installed.

  2. Calendar Server can be either running or stopped; however, if possible, stop Calendar Server.

  3. Make a copy of your calendar database, if you haven’t already done so.

    Copy only the database (.db) files. You don’t need to copy any share (__db.*) or log (log.*) files.

  4. Change to the cal-svr-base/SUNWics5/cal/sbin directory.

    For example, on Solaris Operating Systems for the default directory, enter:

    cd /opt/SUNWics5/cal/sbin

  5. Run the check command on the copy of your calendar database:

    ./csdb check dbdir /tmp/check.out

    If you don’t specify dbdir, check uses the database in the current directory.

    The check command can generate a lot of information, so consider redirecting all output, including stdout and stderr, to a file (as shown in the example).

  6. When check has finished, review the output file. If your database is corrupted, run the rebuild command.

    (See 22.5.6 Rebuilding a Corrupted Calendar Database.)

22.5.3 Dealing with Transaction Log Files Suddenly Very Large and Numerous

It is possible that your automatic purge configuration settings do not properly account for the client user interface your end users prefer. The sudden appearance of large numbers of large transaction log files can be simply the result of a long delay in purging Delete log records. If this delay is purposefully done to accommodate users of the Connector for Microsoft Outlook or the Sync Tool, then the appearance of the large and numerous transaction log files is to be expected. No further action is required. Eventually the system will catch up. However, if your end users are using the Communications Express client, returning the automatic purge settings to their defaults should fix the problem.

22.5.4 Preventing Service Interruptions When Your Database is Corrupted (Read-only Mode)

This sections covers how to keep your corrupted database accessible while you are in recovery mode and includes the following topics:

22.5.4.1 Using Read-only Mode

If you are encountering database corruption, one way to prevent service interruptions is to put your database in read-only mode. This mode allows end users to read database entries, but does not allow additions, modifications, or deletions. If an end user attempts to add, modify or delete any calendar data, the system gives an error message. In addition, administrator tools that add, modify or delete calendar events and todos will not work while the database is in read-only mode.

Note –

If the database is corrupted to the point that it can’t be read, you must interrupt service long enough to restore a backup. The quickest way to restore a backup is to have a good hot backup. See 22.5.8.1 Before You Restore.

ProcedureTo Put a Database in Read-only Mode

  1. Log in as an administrator with permission to change the configuration.

  2. Stop Calendar Server services by issuing the stop-cal command.

  3. At a command line, change to the directory where the ics.conf is located:

    cd /etc/opt/SUNWics5/config

  4. Specify read-only mode for the calendar, by setting the parameter as follows:

    caldb.berkeleydb.readonly=”yes”

  5. Restart Calendar Server by issuing the start-cal command.

    cal-svr-base/SUNWics5/cal/sbin/start-cal

    You must restart the services in order for the ics.conf changes to take effect.

22.5.5 Handling Common Database Failures

This section covers a few of the common database failures and includes some suggested remedies. It contains the following topics:

Procedurecsadmind Won’t Start or Crashes During Startup

Since csadmind is the service that handles both the group scheduling engine (GSE) and the alarm dispatch engine, this could have been caused by offending entries in the GSE queue or the alarm queue.

Remedies:

  1. If csadmind is not running, issue the stop-cal command immediately.

    Leaving calendar server running could cause transaction logs to accumulate, which could further corrupt the database, and could take much longer to reconcile the transaction log files to the database.

  2. Verify that all Calendar Server processes are stopped.

    For instructions on how to verify that all processes are stopped, see To Stop Child Processes.

  3. Try restarting csadmind again by issuing the start-cal -csadmind command.

    If it starts successfully, make sure the two queues are functioning by performing the following steps:

    1. Checking the GSE queue using csschedule.

    2. Checking the alarm queue using dbrig.

      For instructions on running csschedule and dbrig, see Appendix D, Calendar Server Command-Line Utilities Reference.

  4. If csadmind crashes with a dump, analyze the pstack.

    If you notice any GSE related functions in the trace (they will have the letters GSE in them), look at the first entry in the GSE queue and the referenced entry in the events database. Most of the time, the event referred to in the GSE entry is the offending entry. To fix this problem:

    1. Remove the GSE entry using csschedule.

    2. Remove the offending event from the database using cscomponents.

      For instructions on running csschedule and cscomponents, see Appendix D, Calendar Server Command-Line Utilities Reference.

  5. If the entries are not corrupted, then it could be a special case that the calendar server could not handle.

    Take the following steps:

    1. Take a calendar environment snapshot of the corrupted database, and contact customer support.

      To create an environmental backup:

      1. Use the db_checkpoint utility found at:

        cal-svr-base/SUNWics5/cal/tools/unsupported/bin/db_checkpoint

      2. Run db_archive -s.

        Use the -s option to identify all the database files and copy them to a removable medium, such as CD, or DVD, or tape.

      3. Run db_archive -l.

        Use the -l option to identify all the log files and copy unapplied log files to a removable-medium device.

    2. To avoid service interruptions, place your calendar database into a read-only state temporarily, and revert to a hot backup copy.

      • Placing your calendar database into a read-only state temporarily prevents any add, modify or delete transactions from taking place. End users will get an error message when they try to add, modify or delete any calendar data. Administrator tools that add, modify or delete calendar events and todos also will not work while the database is in read-only mode.

        To put your calendar database in read-only mode, edit the ics.conf file and set the following parameter to “yes”, as shown:

        caldb.berkeleydb.readonly=”yes”

      • Revert to a hot backup copy, using the instructions found in 22.5.8 Restoring an Automatic Backup Copy.

        With csstored configured and enabled, a hot backup is available that should be within minutes of being uptodate. You should always verify your hot backup copy to make sure it is not corrupt also. (Run db_verify.)

  6. If all else fails, perform the dump and reload procedure to see if it can salvage the database.

    This procedure is described in 22.5.7 Using the Dump and Load Procedure to Recover a Calendar Database.

ProcedureServices Hung, and End Users Can’t Connect–Orphaned Locks

This condition may be caused by a control thread, which holds a Berkeley DB database page lock, quitting without releasing the lock. To confirm the problem, run pstack on cshttpd processes and csadmind. (pstack is a standard UNIX utility found at: /usr/bin/pstack) It should show threads that are waiting to acquire a lock.

To fix the problem, restart Calendar Server, as follows:

  1. Change to the directory where start-cal resides.

    cd cal-svr-base/SUNWics5/cal/sbin

  2. Issue the start-cal command.

    ./start-cal

Procedurecsdb rebuild Never Finishes–Database Looping

Database looping is usually caused by corruption in the database files. Since it is a database corruption, it can be unrecoverable. There are several options:

  1. Revert to the hot backup.

    If the corruption occurred recently, you can use one of your hot backups.

  2. Use your catastrophe archival recovery process.

    For a suggested process, see 22.5.8 Restoring an Automatic Backup Copy.

  3. Use the dump and reload procedure, 22.5.7 Using the Dump and Load Procedure to Recover a Calendar Database.

22.5.6 Rebuilding a Corrupted Calendar Database

This section describes how to use the csdb rebuild command and contains the following topics:

22.5.6.1 rebuild Overview

The rebuild command scans a calendar database and checks the calendar properties (calprops) events and todos (tasks) for corruption. If the rebuild command finds an inconsistency, it generates a rebuilt calendar database (.db files) in the cal-svr-base/SUNWics5/cal/sbin/rebuild_db directory.

The rebuild command without the -g option rebuilds all databases except the group scheduling engine (GSE) database. To also rebuild the GSE database, include the -g option.

To determine if the GSE database has any entries, run the csschedule -v list command and then let the GSE finish processing the entries before you run the rebuild command.

ProcedureTo Rebuild a Calendar Database

  1. Log in as a user who has administration rights to the system where Calendar Server is installed.

  2. Stop Calendar Server.

  3. Make a copy of your calendar databases, placing them into the /tmp/db directory.

    Copy the database (.db) files and the log (log.*) files. You don’t need to copy any share (__db.*) files.

  4. Change to the cal-svr-base/SUNWics5/cal/sbin directory.

    For example, on Solaris Operating Systems, for the default directory, enter:

    cd /opt/SUNWics5/cal/sbin

    Note –

    If disk space is a problem for the sbin directory, run the rebuild command in a different directory.

  5. Run the rebuild command on the copy of your calendar database:

    ./csdb rebuild /tmp/db /tmp/

    If you don’t specify a database path, rebuild uses the current directory. The /tmp/ parameter species the destination directory for the rebuilt database.

    To also rebuild the GSE database, include the -g option.

    The rebuild command can generate a lot of information, so consider redirecting all output, including stdout and stderr, to a file.

    Note –

    Always rebuild your calendar database using the latest backup copy.

    However, if you have experienced a significant loss of data and you have periodically backed up your database and have more than one copy available, rebuild from the latest copy to the oldest one. (The only drawback is that calendar components that were deleted will reappear in the rebuilt database.)

    For example, if you have three sets of backup calendar database files in directories db_0601, db_0615, and db_0629, run the rebuild command in the following sequence:


    ./csdb rebuild db_0629 
./csdb rebuild db_0615 
./csdb rebuild db_0601

    The rebuild command then writes the rebuilt database to the cal-svr-base/SUNWics5/cal/sbin/rebuild_db directory.

  6. When rebuild has finished, review the output in the rebuild.out file.

    If the rebuild was successful, the last line in the rebuild.out file should be:


    Calendar database has been rebuilt

  7. After you have verified that rebuild was successful in the previous step, copy the rebuilt database (.db) files from the rebuild_db directory to your production database.

  8. If you have any share (__db.*) or log (log.*) files from the corrupted database, move them to another directory.

  9. Restart Calendar Server.

22.5.6.2 Sample Rebuild Output

The following example shows the command and the output that it generated:


# ./csdb -g rebuild
Building calprops based on component information.
Please be patient, this may take a while...
Scanning events database...
512 events scanned
Scanning todos database...
34 todos scanned
Scanning events database...
512 events scanned
Scanning todos database...
34 todos scanned
Scanning deletelog database...
15 deletelog entries scanned
Scanning gse database...
21 gse entries scanned
Scanning recurring database...
12 recurring entries scanned
Successful components db scan
Calendar database has been rebuilt
Building components based on calprops information.
Please be patient, this may take a while...
Scanning calprops database to uncover events...
25 calendars scanned
Scanning calprops database to uncover todos...
25 calendars scanned
Successful calprops db scan
Calendar database has been rebuilt
Note –

The preceding sample output shows the events and the todos databases scanned twice each. This is not an error. It scans the first time to verify the information in the calendar properties database and then scans again to make sure calendar properties database is accessible.

22.5.7 Using the Dump and Load Procedure to Recover a Calendar Database

This sections contains the following topics:

22.5.7.1 Dump and Load Overview

Use the dump and load procedure to try to recover a corrupted database. The dump and load procedure uses the Berkeley database db_dump and db_load utilities, which Calendar Server includes in the following directory:

cal-svr-base/SUNWics5/cal/tools/unsupported/bin

The db_dump utility reads a database file and writes the database entries to an output file, using a format that is compatible with the db_load utility.

For documentation about the db_dump and db_load utilities, refer to the Sleepycat Software Web site:

http://www.sleepycat.com/docs/utility/index.html

Your success in recovering a database using the db_dump and db_load utilities depends on the degree of corruption of your database. You might need to try several db_dump options before you successfully recover your database. If your database is severely corrupted, however, recovery might not be possible, and you might need to revert to the last good hot backup or archive backup of your database.

Note –

Before you perform the dump and load procedure, your calendar database must be Berkeley DB version 3.2.9, or later. If you have an earlier version, first run the cs5migrate utility to upgrade your calendar database.

For the most up to date version of cs5migrate, call Sun technical support.

ProcedureTo Perform the Dump and Load Procedure

  1. Log in as the user and group under which Calendar Server is running, such as icsuser and icsgroup, or as superuser (root).

  2. Stop Calendar Server, if necessary.

  3. Backup your corrupted database using a utility such as csbackup, the Sun StorEdge Enterprise BackupTM software, or Legato Networker®.

    For more information refer to Chapter 17, Backing Up and Restoring Calendar Server Data.

  4. Dump each corrupted database file using the db_dump utility.

    The database files are ics50calprops.db, ics50journals.db, ics50alarms.db, ics50events.db, ics50todos.db, and ics50gse.db.

    Run db_dump using the following options, in order, until your database is recovered (or until you determine that the database can’t be recovered):

    • No options for minor database corruption.

    • -R option for moderate database corruption.

    • -R option for severe database corruption. The -R option dumps more data than the -r option, including partial and deleted records, from the corrupted database.

      For example, to run db_dump with the -r option:

      db_dump -r ics50events.db \> ics50events.db.txt

  5. Load the output file into a new database file using the db_load utility.

    For example:

    db_load new.ics50events.db < ics50events.db.txt

    If db_load reports an odd number of keys or data entries, edit the db_dump output file, and remove the odd key or data entries. Then run db_load again.

  6. Repeat the previous two steps for the other corrupted database files.

    That is, run db_dump for the other corrupted database files.

  7. Rebuild the recovered database files using the csdb rebuild command, as described in 22.5.6 Rebuilding a Corrupted Calendar Database.

    When rebuild has finished, review the output in the output file. If the rebuild was successful, the last line in the rebuild.out file should be:


    Calendar database has been rebuilt

    If the csdb rebuild command was not successful, dump your database using the next db_dump option (-r or -R).

    If the db_dump -R option does not recover your corrupted database, contact your Sun Microsystems technical support or sales account representative for assistance. In the meantime, you might need to revert to the last good backup of your database.

22.5.8 Restoring an Automatic Backup Copy

If you have used the automatic backup feature described in Chapter 9, Configuring Automatic Backups (csstored), you can use the hot backup copy when your live database is corrupted.

This sections covers how to restore the two different automatic backups:

22.5.8.1 Before You Restore

Before you restore a backup, be sure that you have:

ProcedureTo Restore a Hot Backup

Hot backups should be your first choice of backup when your live database is corrupted. To restore a hot backup, follow these steps:

  1. Identify any log files that were unapplied or open for writing in the corrupted live database directory.

  2. Close the log that was open for writing. It contains the most recent transactions.

  3. Create a new (recovery) directory.

  4. Copy the current hot backup copy into the new recovery database directory.

  5. Copy the log.* files from your corrupted live database directory into your new recovery database directory.

  6. If you are keeping an archive copy of the database, copy the logs that had not been applied to the live database into the archive directory, so your archive backup copy will be complete.

  7. Run db_recover with the -c -h options specified against the new recovery database.

    For example, if your new recovery directory is called recoverydb, then the command would be as follows:

    db_recover -c -h recoverydb

  8. Leave the log.* files in the new recovery directory.

    The db_recover program applied the log files to the new recovery databases, but starting with version 42, the Berkeley DB expects them to remain.

  9. Run db_verify against the database files in the new recovery directory. To run db_verify:

    1. Stop the Calendar Server using these commands.

      cd /opt/SUNWics5/cal/sbin

      ./stop-cal

    2. Create another copy of the Calendar Server database (csdb) using this command.

      cp -Rp /var/opt/SUNWics5/csdb /var/opt/SUNWics5/csdb.db_verify

    3. Run db_verify on the copy of csdb.

      Note –

      Do not run db_verify on the original csdb.

      LD_LIBRARY_PATH=/opt/SUNWics5/cal/lib
export LD_LIBRARY_PATH
cd /opt/SUNWics5/cal/tools/unsupported/bin
./db_verify -h /var/opt/SUNWics5/csdb.db_verify ics50alarms.db
./db_verify -h /var/opt/SUNWics5/csdb.db_verify ics50calprops.db
./db_verify -h /var/opt/SUNWics5/csdb.db_verify ics50events.db
./db_verify -h /var/opt/SUNWics5/csdb.db_verify ics50gse.db
./db_verify -h /var/opt/SUNWics5/csdb.db_verify ics50journals.db
./db_verify -h /var/opt/SUNWics5/csdb.db_verify ics50recurring.db
./db_verify -h /var/opt/SUNWics5/csdb.db_verify ics50todos.db
./db_verify -o -h /var/opt/SUNWics5/csdb.db_verify ics50deletelog.db
      Note –

      Run db_verify with -o option for ics50deletelog.db.

      If db_verify completes running successfully, you will not get any error messages. If the database file gets corrupted, it throws error messages. For example:


      ./db_verify -h /var/opt/SUNWics5/csdb.db_verify ics50todos.db
db_verify:Page 612: last item on page sorted greater than parent entry
db_verify: Page 612: incorrect next_pgno 885 found in leaf chain (should be 501)
db_verify: Page 0: page 501 encountered a second time on free list
db_verify: DB->verify: ics50todos.db: DB_VERIFY_BAD: Database verification failed

  10. Run csdb -v list against the new recovery directory.

  11. If the new recovery directory passed all three preceding recovery steps, replace the old corrupted live database with the new recovery database.

  12. Copy the new live database into your hot backup directory to function as the new snapshot.

    All new logs will be applied to this copy until the next regular snapshot is taken.

  13. Start Calendar Server.

  14. If the new recovery directory failed any of the steps, identify an uncorrupted older hot backup as follows:

    1. Working backward through your hot backups, find the most recent copy that is not corrupted by running db_verify and csdb -v list on each in turn.

    2. The first hot backup copy that passes can be restored to your live database directory.

      Replace the corrupted live database with the clean hot backup, as described in To Restore a Hot Backup. (Be sure to read 22.5.8.1 Before You Restore first.)

    3. If none of your hot backups work and you do not have archive backups to try, call technical support. If you do have archive backups, follow the procedure that followsTo Restore an Archive Backup. (See also, 22.5.8.1 Before You Restore.)

ProcedureTo Restore an Archive Backup

If you do not have an uncorrupted hot backup, but have archive backups and their transaction logs, you can restore the most current uncorrupted version of the archived database by performing the following steps:

  1. Identify any log files that were unapplied or open for writing in the corrupted live database directory.

  2. Close the log that was open for writing. It contains the most recent transactions.

  3. Create a new (recovery) directory.

  4. Copy the most recent archive copy and its log files into the new recovery database directory.

  5. Copy any unapplied log.* files from your corrupted live database directory into your new recovery database directory.

  6. Run db_recover with the -c-h options specified against the new recovery database.

    For example, if your new recovery directory is called recoverydb, then the command would be as follows:

    db_recover -c -h recoverydb

  7. Leave the log.* files in the new recovery directory.

    The db_recover program applied the log files to the new recovery databases, but starting with version 4.2, Berkeley DB expects the log files to still be there.

  8. Run db_verify against the database files in the new recovery directory.

    Steps in the To Restore a Hot Backup procedure, explains how to run db_verify.

  9. Run csdb -v list against the new recovery directory.

  10. If the new recovery directory passed all three preceding recovery steps, replace the old corrupted live database with the new recovery database.

  11. Copy the new live database into your hot backup directory to function as the new snapshot.

  12. Start Calendar Server.

  13. If the new recovery directory failed any of the steps, identify an uncorrupted older archive backup as follows:

    1. Working backward through your archive backup copies, find the most recent copy that is not corrupted by running the three recovery programs against each of them in turn: db_recover -c-h, db_verify, and csdb -v list.

    2. The first archive copy that passes can be restored to your live database directory.

      Replace the corrupted live database with the clean archive backup, as shown in To Restore an Archive Backup.

    3. If none of your archive backups work, call technical support.

22.5.9 Repairing Custom Backup Scripts

This section includes the following topics:

22.5.9.1 Berkeley Tools Now Compiled with a Dynamic Library

If you have created a custom backup script using the Berkeley database tools, such as db_recover, you may find that it will no longer work after upgrading to Calendar Server. The reason for this is that the earlier versions of Calendar Server compiled the tools with a static library. The tools are now compiled with a dynamic library, libdb-4.2.so.

22.5.9.2 To Repair a Custom Backup Script

To use the new dynamic library with your existing custom scripts, set the following global variable as shown:

LD_LIBRARY_PATH=libdb-4.2.so