This chapter describes how to troubleshoot your Oracle Communications Indexing and Search Service deployment.
This section explains how to use Oracle Communications Messaging Server and Indexing and Search Service log files to diagnose problems.
Because Indexing and Search Service does not see the original IMAP commands, you must use Messaging Server logs to view this information. You can use the Messaging Server telemetry logs to check on some IMAP searches, for example ESEARCH searches. For more information about telemetry logs, see the topic on checking user IMAP, POP, and Webmail sessions by using telemetry in Messaging Server System Administrator's Guide.
You can then use the host name, user name, and folder information in the Indexing and Search Service logs to match up with the corresponding IMAP command in the Messaging Server logs.
Location of the Messaging Server IMAP log: MessagingServer_home/logs/imap
When to use: If bootstrapping of accounts is failing, authentication might be a problem. Check the Messaging Server imap log. If your read-only message store administrative user (mail.imap.admin.username in the /opt/sun/comms/jiss/etc/jiss.conf file) is the problem, you might see errors similar to the following:
[Account Information: connect [172.20.241.110:56831] [Account Notice: badlogin: [172.20.241.110:56831] plain User not found
If the password of that user is incorrect (mail.imap.admin.password in the /opt/sun/comms/jiss/etc/jiss.conf file), you might see errors similar to the following:
[Account Information: connect [172.20.241.110:56854] [Account Notice: [172.20.241.110:56854] Password verification failed [Account Notice: badlogin: [172.20.241.110:56854] plain User not found
Also, ensure that this user is specified in store.indexeradmins (to give read-only message store administrative rights) and that you have restarted the imapd process, to pick up this change. Failure to do so causes errors similar to the following:
WARNING: Caught IssException: com.sun.comms.iss.common.IssException: BootStrap of account, User: testuser1 Host: mailhost.example.com Not authorized to login as specified user
You must add this read-only message store administrative user manually (that is, it is not automatically created) during the Indexing and Search Service installation process. For more information, see Indexing and Search Service Installation and Configuration Guide.
Location: /var/svc/log/*jiss*
When to use: If you receive the following error when trying to start Indexing and Search Service:
# /opt/sun/comms/jiss/bin/svc_control.sh start Starting utilSvc Starting indexSvc svcadm: Instance "svc:/application/jiss-indexSvc:default" is in maintenance state.
To determine the cause for indexSvc entering the maintenance state, check the /var/svc/log/application-jiss-indexSvc:default.log log file. There can be several possible causes, one of which can be that the Message Queue broker might not be running.
After determining the cause, fix the underlying issue (for example, start the Message Queue (JMQ) broker), and run the svc_control.sh stop command (to clear the maintenance state), followed by the svc_control.sh start command again.
Location: /var/imq/instances/imqbroker/log/log.txt
When to use: To investigate problems, such as authentication failures, from the IMQ brokers on either the Messaging Server or the Indexing and Search Service system.
For more information, see "Message Queue Consumer and Broker Log Messages".
Location: GlassFish_home/appserver/domains/domain1/log/server.log
When to use: Verify that Oracle GlassFish Server is receiving, authenticating, and servicing requests.
If you see a query from Messaging Server that is refused, ensure that the originating IP address is correctly specified in the mail.server.ip parameter in the jiss.conf file. If you do not see a query as expected from Messaging Server in the Indexing and Search Service GlassFish Server log, check the settings of the service.imap.indexer.*configutil parameters. If those settings are correct, try issuing a query directly from the Messaging Server and check the Indexing and Search Service GlassFish Server log again, for example:
# telnet isshost.example.com 8080 Trying 10.10.10.10... Connected to isshost.example.com. Escape character is '^]'. GET /rest/search?q=%20%2busername:user1%20%2Bhostname:mailhost.example.com&contentformat=simpleuid&format=atom HTTP/1.0 HTTP/1.1 200 OK X-Powered-By: Servlet/2.5 Server: Sun GlassFish Enterprise Server v2.1.1 Patch16 Content-Language: * Accept-Ranges: bytes Content-Type: text/xml;charset=UTF-8 Content-Length: 1781 Date: Thu, 14 Nov 2013 22:24:27 GMT Connection: close <?xml version="1.0" encoding="UTF-8"?> <feed xmlns="http://www.w3.org/2005/Atom" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"> <title>isshost.example.com search: +username:user1 +hostname:mailhost.example.com</title> <link>http://isshost.example.com/rest/search?q=+username:user1%20+hostname:isshost.example.com&format=atom</link> <updated>Thu Nov 14 22:24:27 UTC 2013</updated> <author><name>Oracle, Inc.</name></author><id>urn:uuid:9999999999999</id> <opensearch:totalResults>39</opensearch:totalResults> <opensearch:startIndex>0</opensearch:startIndex> <opensearch:itemsPerPage>10</opensearch:itemsPerPage> <opensearch:Query role="request" searchTerms="+username:user1 +hostname:isshost.example.com" searchPage="-1" /> <link rel="search" type="application/opensearchdescription+xml" href="http://isshost.example.com" /> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>1</id> </entry> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>2</id> </entry> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>3</id> </entry> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>4</id> </entry> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>5</id> </entry> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>6</id> </entry> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>7</id> </entry> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>8</id> </entry> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>9</id> </entry> <entry> <folder>Trash</folder> <uidvalidity>1195182036</uidvalidity> <id>10</id> </entry> </feed> Connection to isshost.example.com closed by foreign host.
For more information, see "Index Service Log Messages".
Location: /var/opt/sun/comms/jiss/logs/iss-jmqconsumer.log.0
When to use: Verify the status of the Messaging Server event consumer (jmqconsumer) to ensure that real-time updates are working (that is, arrival of new mail, deletion of mail, new folder creation, and so on).
The mail store JMQ broker is only accessed by jmqconsumer. JMQ broker connection problems with the Messaging Server message store appear in this log file. The following messages show that the JMQ broker is functioning:
Mar 24, 2009 10:55:22 AM com.sun.comms.iss.jmqconsumer.JMQConsumer$AsynchConsumer run INFO: Listening to Queue INDEXMS on mailhost.example.com:7676 Mar 24, 2009 12:43:28 PM com.sun.comms.iss.jmqconsumer.JMQConsumer$AsynchConsumer run INFO: Listening to Queue INDEXMS on mailhost.example.com:7676 Mar 24, 2009 12:44:07 PM com.sun.comms.iss.common.AccountStateMgrImpl setAccountState INFO: Set account user1@mailhost.example.com state to B Mar 24, 2009 1:12:41 PM com.sun.comms.iss.common.AccountStateMgrImpl setAccountState INFO: Set account user1@mailhost.example.com state to A
For more information, see "Message Queue Consumer and Broker Log Messages".
Location: /var/opt/sun/comms/jiss/logs/iss-jmqconsumer-stats.log.0
When to use: To check JMQ event statistics related to performance and patterns of service over long periods of time.
For more information, see "JMQConsumer Statistics: Output of listbacklog Option".
Location: /var/opt/sun/comms/jiss/logs/iss-indexsvc.log.0
When to use: To check issues with indexing during bootstrapping or during the indexing of new real-time events from Messaging Server.
For more information, see "Index Service Log Messages".
Location: /var/opt/sun/comms/jiss/logs/iss-indexsvc-stats.log.0
When to use: To check indexing statistics related to performance and patterns of service over long periods of time.
For more information, see "Index Service Statistics: Output of liststats Option".
Location: /var/opt/sun/comms/jiss/logs/iss-searchsvc.log.0
When to use: To check on the arrival and servicing of search requests.
For more information, see "Index Service Log Messages".
Location: /var/opt/sun/comms/jiss/logs/iss-searchsvc-stats.log.0
When to use: To check search statistics related to performance and patterns of search requests over long periods of time.
For more information, see "Index Service Statistics: Output of liststats Option".
Location: /var/opt/sun/comms/jiss/logs/iss-utilsvc.log.0
When to use: To check on the arrival and servicing of utility requests.
For more information, see "Index Service Log Messages".
Indexing and Search Service provides command-line tools to manage the Indexing and Search Service store. The primary tool is the issadmin.sh script. Other tools listed in this section are intended for specific problems. For more information, see "Indexing and Search Service Command-Line Utilities".
When to use: If an individual index directory is corrupted. Checks for consistency and corrects some types of errors at the Lucene data record level.
When to use: To inspect internal database structure at the Lucene data record level. This script runs from the command line and supports commands to search and display records in an individual index directory. Its features are similar to those of the luke.sh command.
When to use: To inspect internal database structure at the Lucene data record level. This script runs a graphical user interface (GUI) interface that corresponds to lucli.sh. To enable this script, you must manually download the jar files.
When to use: To manipulate index directories directly at the Lucene file level. This script is rarely used, because it requires detailed knowledge of internal store structure.
When to use: To search accounts by using the command-line interface. Can be used to check if an account contains the expected data from search queries that have failed.
When to use: To diagnose general problems, to recreate lost data from single or multiple accounts, and to list, create, delete, check, and sync accounts and folders.
When you suspect a problem with an account, for example, after finding an error message in the log files, you can print a summary of the account structure by using the --accountinfo option to the issadmin.sh command.
Output from the --accountinfo option includes:
Name of the account
Group to which the account is assigned
Disk space used by the account's group
Account state
Time that the account was created in the Indexing and Search Service store
Time that the state of the account was last changed
Counts of the number of emails in each folder and total for the account
Specific information about each folder by name (including the number of emails found containing attachments as indicated by the at: field, number of emails that produced attachment store files, typically thumbnails used for client display or content files under some configurations, and other information)
Folder names are case sensitive and should display as seen in the email client. A name that appears as a string of question marks usually indicates a problem with displaying other character sets (such as Korean or Chinese) on your terminal. Ensure that your terminal is configured to display UTF-8, for example:
# LANG=en_US.UTF-8 # export LANG
In addition to the information for the specific account, the header of the output displays some global characteristics of the store. The total size, total search, and total index values are always zero. Most of the other values are easy to understand. If any appear inconsistent (such as the number of accounts do not match the number you created less those you deleted), then investigate using the --listbrief, or --listaccounts, or both options. (These commands are more expensive and can take several minutes to complete when the store contains many accounts.) The dIndex memory locked field should usually be false, unless at some point you used the --lockmemoryindex option. Take special care using this option, because it can cause unexpected behavior in the store and search.
The email counts can have two forms: a single integer, or a pair of integers separated by a slash ("/"). The single integer form is the number of emails currently in the folder. The first integer of a pair also indicates the number of emails currently in the folder. The second integer might be smaller or larger than the first. Its presence indicates some emails were copied to or from the folder, and it is the number of emails whose copied contents is still associated with the folder. Immediately after an account is bootstrapped, --accountinfo should not show any folder with a count using the pair form. A folder with a count using the pair form indicates a problem with the bootstrap procedure. (After a user has been manipulating the contents of folder of an account, the pair form could be due to normal copying or moving message between folders.) If you suspect a problem, use the --checkfolder option or --checkaccount option to verify that there is no problem with the content of the account.
Output from the --checkaccount option shows whether the account information in the index matches (or is "in sync") with the information in the Messaging Server message store that the index depends upon. (The --checkfolder option performs the same kind of checking on a single folder.) If you suspect that an account has a problem, the --checkaccount option output indicates which folders are not synchronized. If the number of emails in any folder does not match the Messaging Server message store count, use the --sync option with the --checkaccount option to perform on-demand account update. This process usually synchronizes the account within minutes.
The --checkaccount option updates the number of emails in the index to match the Messaging Server message store. However, it does not perform a detailed verification of all information in the index against the Messaging Server message store, as that process is more time and resource intensive. Add the --detail modifier to the --checkaccount option to perform a detailed check. The flags level also checks for any email flag information that is out of sync, and when used with the --sync option, it corrects such problems. When you specify a level of status or full, the --detail modifier likely produces much more output. For example, you would see any indexing problems found with individual emails in the account or folder. Many of these status messages result from attachments containing data that cannot be interpreted properly. Sometimes the data format is inconsistent or other limitations on the indexing conversion process occur. These messages cannot be avoided when using the --sync option. These messages represent limitations on the kind of data that can be indexed and searched and might explain why some search queries are not able to return results as expected.
However, the output from the --detail status modifier can indicate other problems with the indexed data that can be corrected. If problems persist with the account after using --sync, examine them for clues about what might be wrong.
You can correct many problems with accounts by using the --sync option. If --sync fails to correct the account, there might be an internal consistency problem which requires other approaches. If the problem appears to be local to a specific folder, then using the --deletefolder option on the problem folder, followed by a bootstrap (using the --bootstrap --folder option to issadmin.sh), might correct it. Each folder can be corrected in turn, or, in the worse case, you must use the --deleteaccount option and rebootstrap the entire account. If you must perform such repairs, start by using the --setstate X option to ensure the account index is offline from the real-time event updates while you delete and rebootstrap folders or the account.
Output from the --checkstore option describes the overall state of the index store, in contrast to the --checkaccount and --checkfolder commands, which describe information about the structure of individual accounts. There are two parts to this information: one part is generated by the --detail dindex modifier and one by the --detail store modifier. Both parts appear when the --detail full modifier is used.
The output from using the --detail dindex modifier contains information about the master dIndex directory alone. The command checks for the records describing accounts and groups, looking for duplicate or missing records, and comparing their information for internal consistency between records.
The output from using the --detail store modifier contains information about the index store that holds the individual account group index directories. The command checks the information in the dIndex against what it finds in the index store directories for extra or missing group index directories, comparing the dIndex and store directories for consistency.
The --detail verbose modifier provides control over the message output generated by the --checkstore command. If you specify this option, the output may be quite verbose, as it generates explicit messages for each account or index group it checks. By default the message output is smaller. Use the --detail verbose option only if the default messages do not provide enough detail.
If you expect the output from the --checkstore command to be quite verbose, sometimes the --altoutput file option is required to avoid problems when using --checkstore with a store containing many accounts.
Only run the --checkstore command while the services are stopped. The --checkstore command checks dIndex consistency against the rest of the store, and so you must avoid activity that could modify dIndex. Depending on the store size and complexity, the --checkstore command with the --full option could take up to 40 minutes. Schedule such work when you can shut down the services.
The --sync option can be used with the --checkstore --detail dindex options to correct inconsistencies found in the dIndex. The --sync option might update the dIndex, and so is not permitted while the services are running. After running with --sync, check the output of the --checkstore command again. Some problems might not have been corrected by --sync, and require further intervention.
The --checkstore --sync command prompts you if you also specify the --prompt option, enabling you to step through the process. You can disable the prompting when the command begins. The --sync option can correct many inconsistencies, such as duplicate account document, duplicate folder documents, missing or orphaned account documents, and so forth. Use the --sync option with the --detail full option in most circumstances, to detect and correct as many problems as possible at once. Using both options together takes longer to complete, but it avoids partially correcting the store, which can lead to inconsistencies that are much harder to detect and correct.
When the --checkstore --sync command detects and corrects a problem, typically one or more records for accounts are removed from the dIndex and meta and content index directories. The result is that accounts are effectively deleted from the store. You must reboostrap such accounts after the --sync is complete. Account names that have been deleted by the --sync option are appended to the iss.store.dir/store/checkstore.accounts file. This file has the same format as that used by the --accountlist FILE option. This file is automatically read when the IndexService restarts and the accounts in it are added to the autobootstrap queue to be recreated, as long as the value of the configuration parameter iss.indexsvc.periodic.autobootstrap.enabled is true. Once the accounts are added to the autobootstrap queue, they are deleted from this file. The system then manages the accounts as part of the autobootstrap queue.
Commands such as issadmin.sh invoke services in the IndexService that is a separate running process. (You can also run the issadmin.sh command when the IndexService is not running.) When these commands are executed, they submit tasks to the IndexService, which performs the work, then they wait for the response. If you interrupt one of the commands (by using either Control-C at the command line or the kill command), the submitting process is stopped, but not the services that are active in the IndexService. These services continue to run to completion even though you have killed the submitting process. Some commands, like bootstrapping a large account or various uses of the --accountlist option, also can take a long time to complete, and you might need to interrupt these requests.
To stop a service in the IndexService after you interrupt a command, use the --listactiveservices option to determine which services are still active in the IndexService. Typical output for a host running Indexing and Search Service after interrupting a --checkaccount --sync command might resemble the following:
issadmin.sh --listactiveservices
Fri Aug 24 00:29:16 GMT 2012
active index services:
a:28323:checkaccount:sync:mailhost.example.com:jennifer:javaone2009:edoc:0:1:Fri_Aug_24_00_28_45_GMT_2012:w
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:2000:2001:Fri_Aug_24_00_29_13_GMT_2012:w
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:3000:3001:Fri_Aug_24_00_29_13_GMT_2012:w
a:Autosync:checkaccount:sync:mailhost.example.com:amam:INBOX:Fri_Aug_24_00_00_12_GMT_2012:w
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:5000:5001:Fri_Aug_24_00_29_13_GMT_2012:w
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:1000:1001:Fri_Aug_24_00_29_13_GMT_2012:w
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:Fri_Aug_24_00_28_45_GMT_2012:w
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:0:1:Fri_Aug_24_00_29_12_GMT_2012:w
a:28323:checkaccount:sync:mailhost.example.com:jennifer:javaone2009:Fri_Aug_24_00_28_45_GMT_2012:w
a:28323:checkaccount:1:jennifer:mailhost.example.com:Fri_Aug_24_00_28_44_GMT_2012:w
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:4000:4001:Fri_Aug_24_00_29_13_GMT_2012:w
a:Autosync:checkaccount:sync:mailhost.example.com:amam:INBOX:edoc:0:189816754:Fri_Aug_24_00_11_44_GMT_2012:w
Fri Aug 24 00:29:23 GMT 2012
This example shows that many threads are still active: Each line of output is a separate thread. Names that end with ":w" indicate that these threads are potentially writing to the index. (The a: indicates an issadmin.sh command. The number is a process ID used to identify all threads created under the same issadmin.sh command. Other fields are command specific. Threads created from the autosync and other features are also listed.)
By using the --stopservices command, you can stop any one of the named threads. The following command stops a single thread:
issadmin.sh --stopservice a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:Fri_Aug_24_00_ 28_45_GMT_2012:w
However, usually you would want to stop all threads in a group. The following example stops all threads whose names start with a:28323:
issadmin.sh --stopservice a:28323:
Any string that ends with a colon (:) can be used as a prefix/wildcard.
The --stopservice command generates the following output:
issadmin.sh --stopservice a:28323:
Fri Aug 24 00:29:43 GMT 2012
stop of a:28323: :
a:28323:checkaccount:sync:mailhost.example.com:jennifer:javaone2009:edoc:0:1:Fri_
Aug_24_00_28_45_GMT_2012:w writing service not canceled; marked to stop
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:2000:2001:
Fri_Aug_24_00_29_13_GMT_2012:w writing service not canceled; marked to stop
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:3000:3001:
Fri_Aug_24_00_29_13_GMT_2012:w writing service not canceled; marked to stop
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:5000:5001:
Fri_Aug_24_00_29_13_GMT_2012:w writing service not canceled; marked to stop
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:1000:1001:
Fri_Aug_24_00_29_13_GMT_2012:w writing service not canceled; marked to stop
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:Fri_Aug_24_00_
28_45_GMT_2012:w writing service not canceled; marked to stop
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:0:1:Fri_
Aug_24_00_29_12_GMT_2012:w writing service not canceled; marked to stop
a:28323:checkaccount:sync:mailhost.example.com:jennifer:javaone2009:Fri_Aug_24_00_
28_45_GMT_2012:w writing service not canceled; marked to stop
a:28323:checkaccount:1:jennifer:mailhost.example.com:Fri_Aug_24_00_28_44_GMT_
2012:w writing service not canceled; marked to stop
a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:4000:4001:
Fri_Aug_24_00_29_13_GMT_2012:w writing service not canceled; marked to stop
Fri Aug 24 00:29:53 GMT 2012
Threads that are "done" stop immediately. To avoid corrupting the index data, threads marked with ":w" must stop when they next reach a point where the index is no longer being written. This process can take a bit longer, but the threads usually stop fairly quickly.
However, because the --sync command continues to run while you are using these commands, more threads might be created by threads that have not stopped yet. Therefore, you must repeat the --listactiveservices and --stopservices commands, perhaps several times:
issadmin.sh --listactiveservices Fri Aug 24 00:29:58 GMT 2012 active index services: a:Autosync:checkaccount:sync:mailhost.example.com:amam:INBOX:Fri_Aug_24_00_00_12_GMT_2012:w a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:3000:3001:Fri_Aug_24_00_30_07_GMT_2012:w a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:Fri_Aug_24_00_28_45_GMT_2012:w a:28323:checkaccount:sync:mailhost.example.com:jennifer:javaone2009:Fri_Aug_24_00_28_45_GMT_2012:w a:28323:checkaccount:1:jennifer:mailhost.example.com:Fri_Aug_24_00_28_44_GMT_2012:w a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:2000:2001:Fri_Aug_24_00_30_07_GMT_2012:w a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:0:1:Fri_Aug_24_00_30_07_GMT_2012:w a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:4000:4001:Fri_Aug_24_00_30_07_GMT_2012:w a:Autosync:checkaccount:sync:mailhost.example.com:amam:INBOX:edoc:0:189816754:Fri_Aug_24_00_11_44_GMT_2012:w a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:5000:5001:Fri_Aug_24_00_30_07_GMT_2012:w a:28323:checkaccount:sync:mailhost.example.com:jennifer:javaone2009:edoc:0:1:Fri_Aug_24_00_30_00_GMT_2012:w a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:1000:1001:Fri_Aug_24_00_30_07_GMT_2012:w Fri Aug 24 00:30:08 GMT 2012 issadmin.sh --stopservice a:28323: Fri Aug 24 00:30:31 GMT 2012 stop of a:28323: : a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:3000:3001: Fri_Aug_24_00_30_07_GMT_2012:w writing service not canceled; marked to stop a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:Fri_Aug_24_00_ 28_45_GMT_2012:w writing service not canceled; marked to stop a:28323:checkaccount:sync:mailhost.example.com:jennifer:javaone2009:Fri_Aug_24_00_ 28_45_GMT_2012:w writing service not canceled; marked to stop a:28323:checkaccount:1:jennifer:mailhost.example.com:Fri_Aug_24_00_28_44_GMT_ 2012:w writing service not canceled; marked to stop a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:2000:2001: Fri_Aug_24_00_30_07_GMT_2012:w writing service not canceled; marked to stop a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:0:1:Fri_ Aug_24_00_30_07_GMT_2012:w writing service not canceled; marked to stop a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:4000:4001: Fri_Aug_24_00_30_07_GMT_2012:w writing service not canceled; marked to stop a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:5000:5001: Fri_Aug_24_00_30_07_GMT_2012:w writing service not canceled; marked to stop a:28323:checkaccount:sync:mailhost.example.com:jennifer:javaone2009:edoc:0:1:Fri_ Aug_24_00_30_00_GMT_2012:w writing service not canceled; marked to stop a:28323:checkaccount:sync:mailhost.example.com:jennifer:ubuntu-art:edoc:1000:1001: Fri_Aug_24_00_30_07_GMT_2012:w writing service not canceled; marked to stop Fri Aug 24 00:30:45 GMT 2012 issadmin.sh --listactiveservices Fri Aug 24 00:30:48 GMT 2012 active index services: a:Autosync:checkaccount:sync:mailhost.example.com:amam:INBOX:Fri_Aug_24_00_00_12_GMT_2012:w a:Autosync:checkaccount:sync:mailhost.example.com:amam:INBOX:edoc:0:189816754:Fri_Aug_24_00_11_44_GMT_2012:w Fri Aug 24 00:30:54 GMT 2012 issadmin.sh --listactiveservices Fri Aug 24 01:11:38 GMT 2012 active index services: none Fri Aug 24 01:11:40 GMT 2012
Eventually all threads stop, as seen when --listactiveservices shows no more threads running. In this case, the --sync command did not complete, so the state of the account and its contents are not as expected. You can proceed by using commands like --accountinfo to see what the account looks like and clean it up. Console output from the original --sync command is not produced, because the output is lost when you interrupt the command.
You might also check the index directories of the account for write.lock files. Depending on when the interrupt occurred, these files might not be cleaned up properly.
If you find that simple issadmin.sh commands (such as --liststats) hang and do not return in a few minutes, the Index Service might have failed in a way which requires you to shut down the services and restart them by using the svc_control.sh command. Both indexing and search service are temporarily suspended.
If you detect a hanging condition, you can use the following commands to help diagnose the problem before shutting down services:
To show the process ID (pid) of the IndexService component, run the jps command.
Save all output to a file for later examination:
jstack -F -l pid
To identify any index group directories that were open at the time of failure, run a command like the following:
find /var/opt/sun/comms/jiss/store -name "write.lock"
All write.lock files are found in the IndexSearch_home/index/store/locks directory, not the individual group index directories. The name of each file in this directory contains the number of the group to which it belongs.
Stop the services by using the svc_control.sh stop command.
The output from these commands identifies groups of accounts which might be causing problems. After shutting down the services, remove the write.lock files for each group and find accounts that might be affected by using issadmin.sh commands.
If possible, leave the services offline while you make repairs, then start the servers after repairs are complete.
To restore services quickly, use the --setstate I option to disable accounts that might be causing problems, then check these accounts after you restart the services to see if they need repair. Use the --accountinfo option on each account in such groups and look for any reason for the hanging.
After you restart services, if the --accountinfo header information does not look reasonable, or the hanging condition or other problems recur, the critical internal data structures might be damaged. For more information, see "Disaster Recovery".
The information in an Indexing and Search Service Store instance is organized into accounts. In the email environment, each account in the Indexing and Search Service store uses the same user name as the corresponding account in the Messaging Server message store from which its content is derived.
The following characters are invalid in Indexing and Search Service account names:
<space> $ ~ = # * + % ! , { } ( ) / \ < > ; : " ` [ ] & ?
If any of these characters appear in an account name, Indexing and Search Service does not create or bootstrap the index for that account, and search queries for such an account fail. For more information about account name restrictions in Messaging Server, see the topic on message store valid UIDs and folder names in Messaging Server System Administrator's Guide.
This section describes how to troubleshoot the Web Services Proxy (isshttpd).
When configuring the first Indexing and Search Service node to use LDAP to store the host mappings between Messaging Server and Indexing and Search Service hosts, the isshttpd service does not start until you run the isshttpmgr.sh -u file command. However, you cannot run this command before the setup -t isshttpd command because it requires configuration information provided in that command. Once the LDAP entries are created and replicated there should be no problem starting isshttpd on subsequent nodes.
You can create wget scripts to simulate Oracle Communication Convergence behavior to Indexing and Search Service.
Edit the /etc/wgetrc file to point to isshttpd with the following entries:
http_proxy = localhost:5559 ftp_proxy = localhost:5559
Run the following shell script while updating the password for indexeradmin, the user name, and the mailhost:
#!/bin/sh
[ -f /tmp/isshttpd.out ] && rm -f /tmp/isshttpd.out
[ -f /tmp/cookie.out ] && rm -f /tmp/cookie.out
host=localhost
mailhost=ms1.example.com
username=username
password=password
/usr/sfw/bin/wget \--save-cookies=/tmp/cookie.out \
\--keep-session-cookies \
\--server-response \
\--no-check-certificate \
\-O /tmp/isshttpd.out \
\--max-redirect 0 \
\--post-data="j_username=indexeradmin%3B$username&j_password=$password" \
\-t 1 \
"http://$host:8080/rest/j_security_check"
/usr/sfw/bin/wget \-O /tmp/isshttpd.out \
\--load-cookies=/tmp/cookie.out \
\--no-check-certificate \
\--server-response \
\-t 1 \
"http://$host:8080/rest/search?q=%2busername:$username%20%2bhostname:$mailhost%20%2bfolder:INBOX&c=100"
This shell script searches the inbox of the user provided through isshttpd. You can validated different mappings by using a different user and mailhost combination.
This section describes how to migrate your Indexing and Search Service accounts from Java 6 to Java 7.
If you used Java 7 to index the Indexing and Search Service store, then you do not need to migrate your Indexing and Search Service data. You can continue to update to future versions of Java 7, but you should not use any version of Java 6 with Indexing and Search Service.
The Java 7 release changed the Unicode representation of character strings compared to Java 6. If you created your Indexing and Search Services store using any Java 6 version, then updating to Java 7 may potentially cause some search queries to produce inaccurate results. For example, the search queries might return too few or too many matches, because of the changed data representation. To avoid this situation, you must reindex all or part of the Indexing and Search Service store data using Java 7. If you do not know what parts of the data are impacted, then you must reindex all accounts in the store.
Reindexing the Indexing and Search Service store can be very time consuming and disruptive to a production system. Depending on your data, the number of accounts impacted might be only a fraction of the total. Tools providing by Indexing and Search Service help you to manage indexing these accounts to minimize the Java 7 changes.
These Indexing and Search Service tools enable you to migrate to Java 7 by:
Using the autosync process to identify which accounts must be reindexed under Java 7
Halting all services, reconfiguring to use Java 7, and restarting services
Submitting the accounts to be indexed using Java 7 to the autobootstrap process
The first and last steps can take a long time depending on how many accounts and how much data must be analyzed or indexed. The problem of inaccurate search results is limited to the time between restarting the service under Java 7 and the completion of the autobootstrap for each individual account. In this way all accounts are available for searching while Indexing and Search Service performs the corrections for Java 7.
The iss.indexsvc.periodic.autosync.deepcheck.enable configuration parameter identifies which accounts to reindex for Java 7.
When you set the value of this parameter to true, the autosync processing checks for Java 7 character set problems in each folder in each account, and the folder state to be recorded in the index. When this parameter is set to false (the default), no checking for Java 7 problems occurs, and no folder state updates occur. The folder state indicates whether any email in the folder must be reindexed under Java 7.
You must enable Indexing and Search Service autosync for the iss.indexsvc.periodic.autosync.deepcheck.enable parameter to have effect. When iss.indexsvc.periodic.autosync.deepcheck.enable is set to true, autosync runs normally but performs the extra checking for Java 7 character set problems. This checking causes autosync to take at least twice as long as usual, and possibly much longer, to complete.
The following command monitors the progress of the deep checking:
issadmin.sh --checkstore --detail deepcheck
This command collects and summarizes the information recorded in the folder state fields across all accounts. The output includes lines that you can extract to create a file suitable for use with the --setautobootlist file command. You can run this command whether deepcheck is enabled or not. Its output shows how much the autosync has completed and how much more indexing is needed before the Java 7 migration is complete. For more information, see "Java 7 Migration Example".
After you create the list of accounts that must be indexed under Java 7 (by using the --checkstore--detail deepcheck command), you submit the accounts to the autobootstrap queue for reindexing using the following additional option:
issadmin.sh --setautobootlist file --reboot
file is the file of accounts to be reindexed in the usual --accountlist format as extracted from the --checkstore--detail deepcheck output. The --reboot option informs the autobootstrap processing to delete the account if it already exists before reindexing it. In this manner Indexing and Search Service can still search the prior version of the account while it is waiting to be reindexed.
The following example shows how to use Indexing and Search Service tools to migrate an Indexing and Search Service store from Java 6 to Java 7. This example assumes that you have created the Indexing and Search Service store and it is still running under Java 6.
Run the following command to determine how much data must be analyzed:
issadmin.sh --checkstore --detail deepcheck
The output of this command resembles the following:
Wednesday, January 29, 2014 12:12:04 PM PST checkIndex succeeded for: /var/iss/index//store/dIndex will continue using this copy Header Summary found: Store ID: ISSID_9527c3c6-9fc0-47dd-af40-26e8aa4d4c25 created: 20140129164752 default host name: isshost.example.com default export/import path: <none specified> total size of store instance: 0 total number of accounts: 55 total number of account groups: 55 last known consecutive group: 155 total search queries performed: 0 total search query failures: 0 total index events processed: 55 last backup performed: never last host number: 1 last user number: 55 attachment store enabled: true dIndex memory locked: false number of partitions (from acctMgr):0 number of partitions (from dIndex):0 Checking contents of groups: 55 group documents found, as expected no duplicate id records found in group documents 55 account documents found, as expected no duplicate id records found in account documents Checking folder status: 944 folders found, 0 have java 7 update issues, 0 are OK, 0 are Java 7 safe, 0 are Java 7 indexed, 944 have no status Time to check for group duplicates: 0 seconds Time to check for account duplicates: 0 seconds Checking group assignments of accounts: all account ids are assigned to groups, as expected account numbers consistent Time to check for group/account consistency: 0 seconds # deepcheck folder analysis: # number of accounts clear of problems: 0 # number of accounts safe from problems: 0 # number of accounts with java 7 update issues: 0 # number of accounts with java 7 indexed: 0 # number of accounts with no status: 55 # number of accounts with unknown status: 0 Wednesday, January 29, 2014 12:12:04 PM PST
Look at the section titled "Checking folder status" for the total folders in all accounts in the store, and those with status values marked. Because the deep checking has not yet begun, no folders have status yet. As the autosync checks each folder, these counts change over time.
Look at the last section titled "deepcheck folder analysis" for the status by accounts, which also shows that no status has been recorded. The information in this section grows as accounts are found that require reindexing.
Start the deep checking by setting the iss.indexsvc.periodic.autosync.deepcheck.enable configuration parameter to true.
The autosync configuration parameters must also be enabled when you refresh the configuration. See "Selecting Appropriate Autosync Configuration Values" for information on sizing the autosync configuration to reflect the increased overhead that the deep checking incurs.
As the autosync checks the accounts, run the --checkstore command again to watch the progress. Expect at least twice the usual time for autosync to process all the accounts, and perhaps much longer if the accounts contain a great deal of data. For an Indexing and Search Service Store containing 100,000 accounts, it could take several days to deep check every account. After the autosync has checked every account, the process continues as new email and other changes to the accounts occur. Running the --checkstore command at this point produces output similar to the following:
Wednesday, January 29, 2014 03:25:17 PM PST checkIndex succeeded for: /var/iss/index//store/dIndex will continue using this copy Header Summary found: Store ID: ISSID_9527c3c6-9fc0-47dd-af40-26e8aa4d4c25 created: 20140129164752 default host name: isshost.example.com default export/import path: <none specified> total size of store instance: 0 total number of accounts: 55 total number of account groups: 55 last known consecutive group: 155 total search queries performed: 0 total search query failures: 0 total index events processed: 55 last backup performed: never last host number: 1 last user number: 55 attachment store enabled: true dIndex memory locked: false number of partitions (from acctMgr):0 number of partitions (from dIndex):0 Checking contents of groups: 55 group documents found, as expected no duplicate id records found in group documents 55 account documents found, as expected no duplicate id records found in account documents Checking folder status: 944 folders found, 115 have java 7 update issues, 829 are OK, 0 are Java 7 safe, 0 are Java 7 indexed, 0 have no status Time to check for group duplicates: 0 seconds Time to check for account duplicates: 0 seconds Checking group assignments of accounts: all account ids are assigned to groups, as expected account numbers consistent Time to check for group/account consistency: 0 seconds # deepcheck folder analysis: # number of accounts clear of problems: 34 # number of accounts safe from problems: 0 # number of accounts with java 7 update issues: 21 # number of accounts with java 7 indexed: 0 # number of accounts with no status: 0 # number of accounts with unknown status: 0 # 3 folders in account test have java 7 update issues # folders:INBOX, Sent, info-ims ;;;;;isshost.example.com;test # 1 folder in account durga has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;durga # 1 folder in account test1 has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;test1 # 16 folders in account jennifer have java 7 update issues # folders:???, INBOX, Sent, bar/foo bar, convergence, folderhierarchy/a/1 # i18n, javaone2009, llnl.gov MIME tests, popAcct, problem attachment # stacy, ubuntu-art, |< |_| |\| -|-, time-exceeded-msgs, time-exceeded-msgs-2 ;;;;;isshost.example.com;jennifer # 1 folder in account jeffb has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;jeffb # 1 folder in account jeffa has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;jeffa # 3 folders in account jake have java 7 update issues # folders:Sent, INBOX, APOD ;;;;;isshost.example.com;jake # 1 folder in account test8 has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;test8 # 2 folders in account test9 have java 7 update issues # folders:INBOX, Sent ;;;;;isshost.example.com;test9 # 1 folder in account test5 has java 7 update issues # folders:bug13521409 ;;;;;isshost.example.com;test5 # 18 folders in account test2 have java 7 update issues # folders:MultipleAttachments, ODFAttachments, INBOX, MULTIPLEWordAttachment # MultiplePDFAttachment, OtherAttachment, PDFAttachment, PDFFIVE, PDFFOUR # PDFNew, PDFTHREE, PDFTWO, PowerPointAttachments, RTFAttachment, Sent # demonov19_2007, nestedFolders/nested1, nestedFolders/nested1/c ;;;;;isshost.example.com;test2 # 2 folders in account paul have java 7 update issues # folders:INBOX, Sent ;;;;;isshost.example.com;paul # 1 folder in account david has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;david # 1 folder in account anil has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;anil # 1 folder in account dev has java 7 update issues # folders:INBOX.old ;;;;;isshost.example.com;dev # 27 folders in account test10 have java 7 update issues # folders:DebuggingParseException, INBOX, support, forum, messaging # sac-interest, s10, sec list, ubuntu lists/bazaar ;;;;;isshost.example.com;test10 # 5 folders in account rick have java 7 update issues # folders:INBOX, Sent, a16new, demonov19_2007, partnos ;;;;;isshost.example.com;rick # 1 folder in account admin has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;admin # 4 folders in account lea have java 7 update issues # folders:Drafts, INBOX, Sent, leaemptyfolder/sub folder ;;;;;isshost.example.com;lea # 22 folders in account i18n have java 7 update issues # folders:INBOX, ??????/users-ru, 12May2011/?esky, ???/ubuntu-ko, ???/xen-japanese # 12May2011/???, 12May2011/?????, 12May2011/???, 12May2011/????, 12May2011/Deutsch # 12May2011/Espa?ol, 12May2011/Fran?ais, 12May2011/Polski, 12May2011/Portugu?s Brasileiro # 12May2011/Pycc???, Drafts, ??/ubuntu-zh, Espa?ol/ubuntu-ar, Trash # Sent, Deutsch/ubuntu-de, Fran?ais/users-fr ;;;;;isshost.example.com;i18n # 3 folders in account geetha have java 7 update issues # folders:INBOX, Sent, messaging ;;;;;isshost.example.com;geetha Wednesday, January 29, 2014 03:25:17 PM PST
Note how the "folder status" has changed. All folders have a status now, and some "have Java 7 update issues." At this point there are no folders marked as "Java 7 safe" or "Java 7 indexed" because the reindexing has not yet begun.
Note how the "deepcheck folder analysis" section has changed. You do not need to reindex the accounts marked "clear of problems". They do not contain any data that requires them to be rebootstrapped. You must reindex the accounts marked "with Java 7 update issues" under Java 7 to avoid search problems. The other totals should be zero after all accounts have been checked. However, you might see a small number of accounts or folders that still show "no status" after the autosync has completed processing all accounts. This situation could be due to accounts being added since the autosync cycle began, or accounts that are not in the Active state. (Autosync only processes accounts in the Active state, so if any accounts are in the Inactive, Bootstrap, or Unknown states, you should use the --listbrief command to identify them. You can then either change them to Active for autosync to find later, or delete them if unneeded, or keep them for manual processing after the Java 7 update.) The remaining part of the output shows the individual accounts that have Java 7 issues requiring reindexing. For each account, the number and names of the folders in which problems are detected are listed. This account information can give you a feel for how frequently problems were detected.
To generate the file of accounts to be submitted to autobootstrap, find the string ";;;" by using the grep command. Extract only the account records needed for the --setautobootlist file command into a file.
Once you have the file of accounts to reindex, you are ready to update to Java 7. Install Java 7 in addition to Java 6 on your host so that it is ready when you want to switch, and not delay the restart of the services.
Ensure that the iss.indexsvc.periodic.autobootstrap.enabled parameter is currently set to false.
This setting prevents autobootstrapping from running at this point.
Run the following command to set up the accounts to be autobootstrapped before you shut down Indexing and Search Service services:
issadmin.sh --setautobootlist file --reboot
Shut down Indexing and Search Service services.
After you shut down Indexing and Search Service services, change the following jiss.conf parameter to indicate the change to the Java 7 installation:
java.home
After making this change, restart Indexing and Search Service services.
Enable autobootstrapping by setting the iss.indexsvc.periodic.autobootstrap.enabled parameter to true.
If you preset the accounts in the autobootstrap queue, you should also enable the autobootstrap before restart. At this point all the accounts are using Java 7, but only the accounts that need reindexing should show any difference, and only if a search query happens to occur that triggers a character representational problem. From this point on, do not configure the store to use Java 6.
Refresh the Indexing and Search Service services:
issadmin --refresh
The autosync and deepcheck are still enabled after the update to Java 7. The folder status continues to be updated. Use the --checkstore command to monitor the progress of the reindexing. The following output shows Java 7 reindexing:
Wednesday, January 29, 2014 03:41:56 PM PST checkIndex succeeded for: /var/iss/index//store/dIndex will continue using this copy Header Summary found: Store ID: ISSID_9527c3c6-9fc0-47dd-af40-26e8aa4d4c25 created: 20140129164752 default host name: isshost.example.com default export/import path: <none specified> total size of store instance: 0 total number of accounts: 55 total number of account groups: 55 last known consecutive group: 155 total search queries performed: 0 total search query failures: 0 total index events processed: 69 last backup performed: never last host number: 1 last user number: 69 attachment store enabled: true dIndex memory locked: false number of partitions (from acctMgr):0 number of partitions (from dIndex):0 Checking contents of groups: 55 group documents found, as expected no duplicate id records found in group documents 55 account documents found, as expected no duplicate id records found in account documents Checking folder status: 944 folders found, 14 have java 7 update issues, 0 are OK, 192 are Java 7 safe, 738 are Java 7 indexed, 0 have no status Time to check for group duplicates: 0 seconds Time to check for account duplicates: 0 seconds Checking group assignments of accounts: all account ids are assigned to groups, as expected account numbers consistent Time to check for group/account consistency: 0 seconds # deepcheck folder analysis: # number of accounts clear of problems: 0 # number of accounts safe from problems: 34 # number of accounts with java 7 update issues: 7 # number of accounts with java 7 indexed: 14 # number of accounts with no status: 0 # number of accounts with unknown status: 0 # 3 folders in account test have java 7 update issues # folders:INBOX, Sent, info-ims ;;;;;isshost.example.com;test # 1 folder in account durga has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;durga # 1 folder in account test1 has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;test1 # 1 folder in account jeffa has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;jeffa # 3 folders in account jake have java 7 update issues # folders:Sent, INBOX, APOD ;;;;;isshost.example.com;jake # 1 folder in account test8 has java 7 update issues # folders:INBOX ;;;;;isshost.example.com;test8 # 4 folders in account lea have java 7 update issues # folders:Drafts, INBOX, Sent, leaemptyfolder/sub folder ;;;;;isshost.example.com;lea Wednesday, January 29, 2014 03:41:59 PM PST
The totals in the "folder status" are shifted to "Java 7 safe" and "Java 7 indexed" indicating folders had no previous problems or have been reindexed using Java 7 respectively.
The "deepcheck folder analysis" counts have changed, and the list of accounts needing reindexing has shrunk as the bootstrapping proceeds. Eventually all the accounts submitted for autobootstrap are indexed, and the --checkstore output resembles the following:
Wednesday, January 29, 2014 03:51:02 PM PST checkIndex succeeded for: /var/iss/index//store/dIndex will continue using this copy Header Summary found: Store ID: ISSID_9527c3c6-9fc0-47dd-af40-26e8aa4d4c25 created: 20140129164752 default host name: isshost.example.com default export/import path: <none specified> total size of store instance: 0 total number of accounts: 55 total number of account groups: 55 last known consecutive group: 155 total search queries performed: 0 total search query failures: 0 total index events processed: 76 last backup performed: never last host number: 1 last user number: 76 attachment store enabled: true dIndex memory locked: false number of partitions (from acctMgr):0 number of partitions (from dIndex):0 Checking contents of groups: 55 group documents found, as expected no duplicate id records found in group documents 55 account documents found, as expected no duplicate id records found in account documents Checking folder status: 944 folders found, 0 have java 7 update issues, 0 are OK, 155 are Java 7 safe, 789 are Java 7 indexed, 0 have no status Time to check for group duplicates: 0 seconds Time to check for account duplicates: 0 seconds Checking group assignments of accounts: all account ids are assigned to groups, as expected account numbers consistent Time to check for group/account consistency: 0 seconds # deepcheck folder analysis: # number of accounts clear of problems: 0 # number of accounts safe from problems: 34 # number of accounts with java 7 update issues: 0 # number of accounts with java 7 indexed: 21 # number of accounts with no status: 0 # number of accounts with unknown status: 0 Wednesday, January 29, 2014 03:51:08 PM PST
All folders and accounts show either "safe" or "indexed", indicating the update to Java 7 is complete.
Disable the deepcheck by setting the iss.indexsvc.periodic.autosync.deepcheck.enable parameter to false, to reduce the overhead of the autosync.
Refresh the configuration.
issadmin.sh --refresh
If the counts for other than "safe" and "indexed" are not zero, then there might be accounts that have not been corrected because they were not Active (as noted previously). Use the --listbrief command to identify any accounts that you might need to delete and reindex manually.
The previous example involved a small number of accounts (55). A production Indexing and Search Service store likely contains several thousand times more accounts and data, and so the --checkstore command might take a few minutes to complete instead of seconds as in this example. The time needed to deep check all accounts, and to reindex all accounts in which a Java 7 problem is detected, will also be long depending on the size of your data and how many problems are detected. Therefore, take care to configure the autosync and autobootstrap parameters to keep the overhead down and enable normal operation of Indexing and Search Service services. The following section provides guidelines for deciding how to balance the costs of the Java 7 upgrade with normal server operation.
Use the following guidelines to determine appropriate autosync configuration values for your deployment.
Check log files. Before enabling the deep check feature, examine the IndexSvc log files for an indication of the current autosync overhead. If autosync is already enabled, and the log level allows for INFO messages, look for messages containing the phrase "findAllCandidates: time to create list." These messages occur when autosync has completed a cycle of all accounts and generated a new list to start the process again. The time stamp of such messages can help you determine approximately how long it takes the server to cycle through all the accounts for autosync using current configuration parameters. This time duration is less than the lower bound for deep check processing, because deep checking is much more resource intensive that the typical autosync processing. Use this time duration to get an approximation for how long the deep check might take to complete.
Process accounts in parallel. The deep checking overhead is significant on both Indexing and Search Service and Messaging Server. Each email in each folder of each account must be scanned for Java 7 migration issues. Thus the load on the Messaging Server is comparable to a full bootstrap of the account, and somewhat less on the Indexing and Search Service server. Spread this load out over time by adjusting the configuration parameters to process only a small number of accounts in parallel simultaneously, to avoid incurring delays in the normal Indexing and Search Service search services. As shown in the previous example, use the --checkstore--detail deepcheck command to monitor progress to achieve the right balance between normal Indexing and Search Service processing and the deep check in autosync.
Set parameters for the first time running autosync. If you have not been running autosync, and do not have a way to estimate the full cycle duration, then set the autosync iss.indexsvc.periodic.autosync.count and iss.indexsvc.periodic.autosync.thread.count parameters to one third the default parameter values in the jiss.comf.template file to begin deep check processing. Observe the overhead and adjust these parameter values as appropriate to the load on the servers.
Start with small count and interval values. You can modify the autosync parameters in the jiss.conf file by running the issadmin.sh --refresh command without having to restart Indexing and Search Service. However, the effects of these parameters do not take effect immediately. The current work period must finish before the new values are applied. Thus the "count" and "interval" values should be kept relatively small until you have determined the load on the system so they can be refreshed quickly without incurring a long wait for the current autosync work or interval to complete.
Run the deep check off hours. Because the deep check processing might take a long time, run it during known times of low system load. The iss.indexsvc.periodic.autosync.deepcheck.enable parameter is refreshable, so you can turn it off or on as the load on the system varies. The work so far completed is recorded in the index, so no information is lost. However, completing the deep check takes much longer if not run all the time.
When to stop deep check. If, after running the deep check for a while, the --checkstore--detail deepcheck command output indicates a very high rate of accounts that require reindexing (say 80 to 90 percent of all accounts checked), you might consider stopping the deep check. Then you can simply shut down the server, update to Java 7, and rebootstrap all accounts in the index. Taking these steps avoids the extra overhead of the deep check, and completes the Java 7 update in less time. The downside is that you do not know which accounts might return inaccurate search results during the period before being indexed using Java 7.
Create your own bootstrap commands. The --checkstore--detail deepcheck command output includes comments showing those folders in an account which contain data that must be reindexed. Only the folders listed must actually be corrected. To reduce the amount of data to be reindexed, you can create your own list of --bootstrap commands that use the --folder name option on just the folders indicated. Creating your own list also reduces the time needed to complete the bootstrap. However, you must generate such a command list manually, which is harder to manage, and perhaps more error prone. Create your own list only in special cases, such as if the --checkstore--detail deepcheck command output shows that most accounts have the same single folder (such as INBOX) to be reindexed. You could use a sequence of commands such as the following to reindex each account XXXX, and complete the Java 7 update more quickly than the general autobootstrap procedure outlined previously.
issadmin.sh --user XXXX --setstate I issadmin.sh --user XXXX --deletefolder --folder INBOX issadmin.sh --user XXXX --bootstrap --folder INBOX
Any accounts which do not follow this pattern would have to be indexed individually based on which folders are needed to complete the update.
The --checkstore command detects and repair inconsistencies in the dIndex, and corrects problems in the account group index directories (meta and content). Some checks complete with little impact on the system, while others are performance intensive. You can configure the --checkstore detections that cause little performance impact to run automatically when the IndexService starts. Doing so helps to catch dIndex problems sooner rather than later. In addition, you can configure Indexing and Search Service to automatically repair some of these problems, resulting in fewer failures overall.
You can also configure other --checkstore capabilities, such as checking the specific account group index for consistency, to run during the periodic autosync --checkaccount processing. The overhead for checking each account is minor, but over time, autosync checks every account. The additional checking delays the autosync processing slightly, but results in fewer problems that --checkstore needs to fix later.
To automatically check the dIndex when the IndexService starts:
Set the following configuration parameter to true:
iss.indexsvc.periodic.autosync.checkstore.enabled
By default, the parameter is set to true.
To view Warning level log messages in the log, set the following parameter to true:
iss.indexsvc.periodic.autosync.enabled
During autosync checking for each account, an account's meta and content index directories are checked for existence and readability. This checking occurs every time autosync processes an account.
To automatically repair dIndex problems:
Ensure that the following configuration parameters are set to true:
iss.indexsvc.periodic.autosync.enabled iss.indexsvc.periodic.autosync.checkstore.enabled
Set the following configuration parameter to true:
iss.indexsvc.periodic.autosync.checkstore.sync.enabled
During autosync syncing for each account, the meta and content index directories are repaired for any account in need of correction.
Once the deep check processing has produced a list of accounts to be indexed under Java 7, estimate the reindexing time for those accounts to determine what configuration parameters to use for the autobootstrap processing. The account list that you use for the --setautobootlist file command can also be used in the --accountinfo command to find how many emails each account contains. However, if the number of accounts is large, this approach might not be practical.
As with the autosync parameters, start by using relatively small values for the autobootstrap "count," "interval," and "thread.count" parameters, so that you can quickly --refresh them as you monitor the autobootstrap process. The order of the accounts being bootstrapped is roughly the order of the names in your --setautobootlist file. Accounts later in the list have a larger likelihood of being searched before being rebootstrapped under Java 7. You can reorder the lines in the --setautobootlist file if you have any preference of which accounts you would rather have reindexed first. The order generated by the deep check is random.
After you have submitted the accounts for autobootstrap with the --setautobootlist file--reboot command, you can adjust the autobootstrap parameters based on how quickly the various accounts finish and the load on the system. If the autobootstrap load causes service to degrade too much, you can reduce the autobootstrap "count," "interval," and "thread.count" parameter values by using the --refresh command. You can even disable autobootstrapping as you think best. The list of accounts is retained unless you use the --unsetautobootlist file command to remove accounts not yet finished bootstrapping. If you "unset" and then later "set" any accounts, remember to use the --reboot option with the --setautobootlist file command.
Hardware failures, power loss, or software bugs can corrupt index store data. As a result, Indexing and Search Services might fail or stop responding.
When problems like these occur, normal operations might resume automatically, or you might need to perform significant intervention. The severity of and recovery from such failures depend on where the data corruption occurs.
The index store consists of two major parts:
The master directory (dIndex) - Contains information about the organization of accounts
The account group index directories - Contain data about individual emails and folders in each account
Recovery from failures in each of these parts requires different approaches. The following sections contain general information about disaster recovery approaches and specific scenarios.
The first high-level step for disaster recovery is to ensure that the dIndex is usable.
Before attempting recovery operations on the dIndex data, ensure that the servers are stopped.
Look for any write.lock file in the dIndex directory and remove it. The presence of a write.lock file means that dIndex was likely being written when the failure occurred, so it may not be reliable.
Before starting the services or any issadmin.sh commands, run the following command to see if its structure is consistent:
issadmin.sh --checkstore --detail dindex
If no problems are reported, then dIndex might not be seriously effected. If you see any warnings of the form:
WARNING: checkForCorruptdIndex: index path: /var/opt/sun/comms/jiss/index/store/dIndex dIndex is not clean, run checkIndex manually.
then the dIndex must be fixed or replaced by a backup copy to restore server functions.
Note:
This message is similar to the result of running the checkIndex.sh script on the dIndex.You can specify partitions to the dIndex (refer to "Improving Global Directory Index Performance"). If the value of configuration parameter iss.store.partitions.count is greater than zero, then in addition to the dIndex directory, there are dIndexNN directories that contain parts of the dIndex information. The --checkstore command checks each of the partitions as well. Each dIndexNN directory is backed up independently and therefore can be used to recover from corruption.
If checkIndex.sh problems are detected, make a backup copy of the entire dIndex directory to preserve the original data. If there are backup files in the store (named dIndex.backupA, dIndex.backupB, and dIndex.backupC), determine which is the most recently changed, and run checkIndex.sh on each to find if any are valid. If the checkIndex.sh output for one or more shows no sign of corruption, then decide whether the most recent backup is sufficiently current that you can use it to replace the corrupted dIndex. Your alternative is to run the checkIndex.sh command with the -fix option again on the dIndex. This action likely causes information loss in dIndex. In either case, using one of the backupA/B/C copies or using -fix probably means that some information has been lost. Depending on the nature of the corruption, using a recent backup might be a better approach, since the dIndex was in a known consistent state at the time the backup was created. However, there is no way to tell what data would be missing since the backup was created. (Any number of accounts might need to be synchronized again.)
If dIndex partitioning is being used, each individual dIndexXX directory can be replaced by its backup in a similar manner. The effect of such replacement is much more limited than when dIndex is replaced, because each partition only contains information about a subset of the accounts. To enable the services to be run again, every partition of the dIndex must be able to pass the checkIndex.sh test.
After these corrections to dIndex are complete, run the --checkstore --detail dindex command again. It should show no corruption, but may show other problems if any older backup copies have been incorporated.
After running the checkIndex.sh command to correct dIndex problems, if the --checkstore --detail dindex command still shows problems, run the following command to correct the store:
issadmin.sh --checkstore --detail full --sync
If your store contains 100,000 accounts or more, this --detail full command might take at least 30 minutes to complete, because the command must check each individual account group index. Completion time varies and depends upon the number of accounts and the amount of data in the store. This command also generates a lot of diagnostic logging to standard output. When this command completes, account names that have been deleted by the --sync option (to resolve conflicts) are appended to the iss.store.dir/store/checkstore.accounts file. This file has the same format as that used by the --accountlist FILE option. This file is automatically read when the IndexService restarts and the accounts in it are added to the autobootstrap queue to be recreated, as long as the value of the configuration parameter iss.indexsvc.periodic.autobootstrap.enabled is true. Once the accounts are added to the autobootstrap queue, they are deleted from this file. The system then manages the accounts as part of the autobootstrap queue. If you do not want to bootstrap any specific account in this file, remove the account from the file before restarting the services. You can also add other accounts to this file as needed.
After running the --sync command, run the following command to verify that the store is problem free:
issadmin.sh --checkstore --detail full
If errors still appear, retry the --sync command. If errors still persist, you should at least be able to restart the services. Accounts with errors might continue to have problems, but your system should perform better.
Note:
If running the first --checkstore --detail full --sync command does not correct all errors, contact Oracle Support with the output.The --detail full option is equivalent to the --detail dindex,store,group option. While you can run the --sync command on the dIndex and store separately (the group option only matters when the store option is specified), use the --detail full option instead of separate commands. Limiting the checking to one or the other option reduces the information available for the checking. Thus, the system might not fully diagnose some problems, resulting in not enough information to clean up all the extraneous data in the store.
Correct any problems shown in the output before attempting to start services. You might also be able to use the backup copy of dIndex to diagnose what was lost by using the lucli.sh tool, but this process is complicated.
When the data in dIndex is corrupted, the effect might be global or local. In the worst situations, services cannot be started, or basic issadmin.sh commands like --accountinfo, --listbrief, --listaccounts, or --liststats fail to produce expected output or hang.
Corrupted global data can be detected in the header information of the output from many common issadmin.sh commands such as --listbrief. For example, if the number of accounts or groups is reported incorrectly, some critical global data might be corrupted. In this case, the main action you can take for recovery is to recreate (bootstrap) the entire index from Messaging Server store data. This process can be lengthy, so you might want to take the time to determine if parts of the store can be salvaged.
You can run the issadmin.sh commands whether the Index and Search Service servers are started or stopped. Failures in issadmin.sh commands when the services are not running might indicate if the problem is global or local to only some accounts or groups.
If only some accounts or groups appear to be failing, use the following:
Disable those accounts using the --setstate I command. If the services start without failure, then the problem might be local and these accounts can be corrected while the rest of the services are active.
Try using the --deleteaccount or --deletefolder commands on suspicious accounts to see if the services can continue to operate. Sometimes services can be restored after you have removed a few accounts. You can then bootstrap these accounts again.
Ensure that the rest of the data is intact by using the --checkaccount or --checkfolder commands on every individual account (refer to the --accountlist option).
If the problems with dIndex appear to be corrected, you can use the --sync command to resolve any other problems in individual accounts.
If some issadmin.sh commands appear to work but dIndex is still corrupted, you can try the following approach to reduce the time that is required to recreate the store:
Use the --export command to export any accounts that you can.
Remove the corrupted store and start rebootstrapping.
Import into the new store any accounts that you can successfully export by using the --import command.
Use --checkaccount and --sync commands on every account to ensure that the data is current.
If the dIndex data is not corrupted but some accounts appear to be failing, then the data corruption might be localized in a set of account groups. Since each account group contains independent "meta" and "content" index directories, the damage might be limited to specific groups of accounts. You might be able to correct the damage without impacting the rest of the store, as described in the following steps:
With services stopped, start by searching for write.lock files in the rest of the store with a command such as the following:
find /var/opt/sun/comms/jiss/index/store/ -name write.lock
All write.lock files are found in the IndexSearch_home/index/store/locks directory, not the individual group index directories. The name of each file in this directory contains the number of the group to which it belongs.
Remove all the write.lock files that you find. Any group directories identified by such files are candidates for further investigation, because they are likely to be corrupted.
Use the checkIndex.sh command on each of the pair of index directories in each suspect group, using the same procedure as described in the preceding dIndex corruption section. (The last four digits of the group number are used to find the directories in the store; for example, for group 12345, look for directories with the path 23/45/index12345_meta and 23/45/index12345_content.) The individual group directories do not have backup files like the dIndex, so if the checkIndex.sh command indicates corruption, that index directory must be fixed by using the -fix option, which will likely cause data loss. (Sometimes, after running checkIndex.sh -fix, the group index might still not be clean. If the group index is not clean, try rerunning the -fix. More data is lost with each such command, but eventually the index should be cleaned. If an index cannot be cleaned by repeated --fix commands, then you must delete all files in that index directory and recreate the group. Refer to the section below about creating a new group for details.) If any of the checkIndex.sh commands indicate that data was lost, accounts in those groups are likely corrupted and need attention.
Run the issadmin.sh --checkstore --detail store command, and check its output for other accounts or groups which may show problems. (If your store contains more that 100000 accounts, the --detail store command might take a very long time (up to 40 minutes) to complete.)
Disable services to each suspect account by using the issadmin.sh --setstate I option. Then use the --accountinfo option, followed by --checkaccount or --checkfolder to diagnose account specific problems.
If necessary, try to delete any data that appears corrupted by using the --deleteaccount or --deletefolder options, and bootstrap or use the --sync option to try to correct the accounts.
If these attempts fail to correct the accounts, then a more global problem with dIndex might exist. The services might be able to run with these specific accounts disabled, but to restore full service to all accounts, you must correct the larger global problem. Running the issadmin.sh --checkstore --detail dindex command might help at this point. Some other approaches to consider include the following:
If you suspect that the problem might be related to a specific account group directory, you might be able to move the accounts in that group into another group by using either the --moveaccount or --export and --import command options. If either the group directories or dIndex is very badly damaged, these commands might also fail, or only transfer part of the account. Always use --checkaccount after moving or importing an account to see if it succeeded. Afterward, the --sync option might be able to correct the account.
If you are unable to create a group when trying to move an account, then dIndex is likely corrupted. Depending on how many accounts are configured as allowed per group, you might be able to move the accounts into existing groups, but not into a new group. Moving accounts into existing groups can allow the services to be resumed temporarily, but you can only correct this failure of dIndex by recreating the store again from the start.
As a last resort, bootstrapping a folder or account into a different group should restore the data for specific corrupted accounts. Always remember to delete the folder or account from the damaged group directory before attempting to bootstrap again.
Finally, all data corruption might not be evident using these techniques, because the structure of the data is correct but the data values have been modified. Search results might prove inconsistent or wrong. This kind of corruption might be less serious but harder to detect because it only effects the correctness of individual searches, not the operation of the entire store. You can restore services to the other accounts while fixing the affected folder or account. In this case, you can delete the suspect folders or accounts and bootstrap them while other services are running.
If one or both of the index directories of an account group (the "meta" and "content" directories) cannot be corrected by using the checkIndex.sh --fix command, then the only way to correct this group is to replace the contents of the corrupted account group index completely, and all data for all accounts in that group is lost.
To recreate an individual account group directory:
Create an empty account group.
Delete all files in the corrupted index directory.
Copy all files from the empty account group into the corrupted directory.
The accounts that were in that group can then be managed using the normal issadmin.sh commands, because the knowledge about those accounts still exists in the dIndex, although the account group index is empty.
A simple way to create an empty account group is to create a nonexistent account in a specific group, and then delete that account. This process creates the structure of an empty index directory that can then be copied to correct any number of group index directories that had to be deleted. When copying the empty account group directory, be sure to preserve the owner, group, and access rights of the empty account group directory (as with the "cp -p" command). To ensure that the normal default allocation of accounts to groups does not effect the empty group directories, use the --group option with the --createaccount command with a small group number, less than 100. (Group numbers less than 100 are not normally allocated by default, so that they can be used for administration purposes like this without interference from the rest of the services.)
Because recovery from a disaster can be so costly, you might want to perform some routine maintenance functions regularly to make recovery faster. For dIndex, backup copies of the entire dIndex are automatically made; these are named dIndex.backupA, dIndex.backupB, and dIndex.backupC and are located in the same directory as dIndex. These backup directories are rewritten periodically based on the value of the iss.store.account.optimizeinterval configuration parameter. (The path to the most recent backup appears in the header output for --accountinfo and other options of the issadmin.sh command.) These backups might allow you to recover some functionality if critical data is corrupted.
After the original bootstrap of accounts is complete, much of the global critical information in dIndex stays stable. Therefore, you might be able to temporarily replace the corrupted dIndex by a backup version while you check the rest of the system for failures. Depending how long ago the backup was made, you might be able to restore services in this manner for many users temporarily until you can complete a full rebuild of the store. However, doing so is not a fully reliable solution for the problem. You might need to perform a full rebootstrap of all accounts, although the system might be able to provide services to some accounts while the rest are being repaired.
You can back up individual accounts by using the --export option, and you can use the snapshot of the account to recover lost data by using the --import command. Then, after you have fixed any corruption, using the --checkaccount and --sync commands to correct the data might be quicker than a full rebootstrap of the account. However, the cost of exporting and importing each account in both time and disk space might be greater than is reasonable for large numbers of accounts.
The topics in this section contain guidelines for how to respond to specific severe problems.
Running out of disk space can create serious problems:
If the disk storage system containing the Indexing and Search Service store becomes filled, indexing services begin to fail.
If the disk containing the log files becomes filled, then further WARNING and SEVERE messages are also lost, so indications of what is wrong might also be missing. Some index data might be lost, and the index directories might be corrupted.
If you discover the disk has become full or is extremely close to full (98% capacity or more), immediately stop the services to prevent failing service requests from corrupting index data. Then recover space on the full disk until it is at least below 95% capacity. Follow the steps outlined in "Disaster Recovery" to determine if data corruption must be corrected before restarting the services.
If you detected the disk full problem and shut down services before any damage to the index occurred, you can restore services while you take steps to reduce the capacity of the disk further:
Some actions you can take to reduce the disk space for the store include:
Placing the log directory on a separate disk
Locating the export/import snapshot directory on a separate disk
Removing any defunct or inactive accounts from the Indexing and Search Service store
Using the disk space sizes from the output of the --accountinfo and --listaccounts commands, determine whether you should --export any accounts to other Indexing and Search Service store instances based on how much space would thus be recovered.
You might consider investing time to prevent the disk full condition, rather than spending more time and effort in recovering from a badly corrupted store. Monitoring the disk space left, for example with a cron job, would enable you to detect when a problem is imminent and to stop the services before serious damage can occur. If you allow enough margin, such as detecting when the disk is 85% to 90% full, then you can sometimes increase disk space as described previously without shutting down the services. How actively you check for problems depends on how much unused disk space your store usually contains, and how expensive a recovery from the disk full condition would be.
A warning message indicating "OutOfMemoryException" (OOME) might appear infrequently in the index log. This message is usually caused by an email that is unusually large or has a very complicated body structure, which causes an indexing service request to fail (typically the bootstrap or --sync commands). When this failure occurs, it might cause the index service process to hang, requiring you to stop and restart the service.
Whether or not the index service process hangs, the OOME message means that the account indicated in the message has suffered a failure, and the corresponding index data might be corrupted.
The general steps to begin recovery from an OOME message are:
Stop any running commands that affect the account. (See "Interrupting Commands" for details.)
Check the account group meta and content index directories of the damaged account for write.lock files.
All write.lock files are found in the IndexSearch_home/index/store/locks directory, not the individual group index directories. The name of each file in this directory contains the number of the group to which it belongs.
The presence of any write.lock files means that an IndexWriter was opened to the account, and data might have been written into the file when the OOME occurred. This IndexWriter must be cleared and any corresponding write.lock files must be deleted before the account can be corrected.
Shutting down the index service clears the IndexWriter, but service to all accounts is interrupted for some length of time. To avoid this service interruption, you can try the following general approach:
To clear the specific IndexWriter that was used, delete any write.lock files from the index meta and content directories.
To avoid losing all the data in the account index, run the --export command.
Delete the account.
Recreate the account and use the --import command to restore the account data. The account is likely out of sync with the Messaging Store. It might be wise to create the account in a separate new group to avoid future problems.
To avoid repeating the failure, you must prevent the email that caused the original OOME problem from being processed. Use the --ignorefolder command and specify the folder containing the offending email to prevent all emails in that folder from being indexed again.
Use the --sync or bootstrap commands to recover the remainder of the account data. Unfortunately, this approach causes the loss of all data in the folder being ignored, including data unrelated to the failure. If you can move emails that are causing the OOME to another empty folder on the Messaging Store, then you can avoid the loss of indexing of other emails in the folder. However, this approach might not be acceptable to your users.
OOME problems might not be repeatable, since memory use varies significantly depending on what else is happening on the system at the time. After the account is indexed with a folder marked ignored, you might want to attempt to --unignorefolder the single folder and bootstrap it again when the system is less loaded. This approach might not succeed. If another OOME occurs, then you must repeat the same recovery process. Using such an approach depends upon what and how much data in the ignored folder will not be indexed.
Also, any write.lock found in an account index group directory indicates that there is an IndexWriter actively writing to an index, but only a single IndexWriter is used by all accounts in any group index. If there is only one account assigned to the group, then you know the account causing the OOME is the one affected.
However, if you have assigned multiple accounts to a group, this procedure of recovering from OOME failure might cause interaction with those accounts as well. To ensure that the accounts are valid, use one of the following approaches:
Disable all accounts in the group (using --setstate I) and back them up with --export before starting to recover. After recovery is complete, you can restore these accounts and use the --sync command to bring them up to date.
Alternatively, you can use the --moveaccount command to move these accounts into other groups before attempting the recovery.
Always do --checkaccount after moving or restoring accounts to ensure that the resulting accounts are valid.
To rename an Indexing and Search Service host:
Change the host name in the operating system.
See your operating system documentation for more information.
In Indexing and Search Service, perform the following on the renamed host:
Edit the jiss.conf file to update the appropriate parameter (hostname, mail.imq, imq.host, or ldap.host) with the new host name.
To uninstall settings, run the following command:
IndexSearch_home/bin/setup -u
To update settings, run the following command:
IndexSearch_home/bin/setup -f IndexSearch_home/etc/jiss.conf
Topics in this section:
Why is my installation not picking up changes that I've made to the jiss.conf file?
How can I check for problems with indexing emails or attachments, or generating thumbnail images?
Is there a way to rotate the Indexing and Search Service logs?
What does a 403 error mean in the GlassFish Server server.log file?
Run the following command to see if changes have not been incorporated:
IndexSearch_home/bin/issadmin.sh --checkconfig
Then try the following command:
IndexSearch_home/bin/issadmin.sh --refresh
Run the following command again to see if the problem is corrected:
IndexSearch_home/bin/issadmin.sh --checkconfig
This command should update most but not all configuration changes. If changes are still not updated, then ensure that you stop and restart Indexing and Search Service after you make changes to the jiss.conf file. See "Stopping and Starting Indexing and Search Service".
Use the issadmin.sh command to check for synchronization:
IndexSearch_home/bin/issadmin.sh --user user --checkaccount
If the mailbox and index are not synchronized, run the issadmin.sh command with the --sync parameter to fix:
IndexSearch_home/bin/issadmin.sh --user user --checkaccount --sync
If the Messaging Server host's time and the Indexing and Search Service host's time are not synchronized, the JMQConsumer log shows large timing discrepancies. The time when the jmqconsumer service logged a message and the time when the message was actually sent could be significantly off or even negative, for example:
Mar 31, 2009 11:59:32 AM com.sun.comms.iss.jmqconsumer.JMQConsumer$EventQueueRunnable run INFO: Account amam@mailhost.example.com is active on Tue Mar 31 11:59:32 GMT 2009, processing event generated on Tue Mar 31 12:03:39 GMT 2009 with sequence number 8, time between generate and submit to index svc is -246813ms. Mar 31, 2009 11:59:53 AM com.sun.comms.iss.jmqconsumer.MessageHeaders displayJMSType
To fix this discrepancy, make sure both the Messaging Server system and the Indexing and Search Service system run an NTP daemon so that their times are synchronized.
Event notifications are not generated for actions generated from the Messaging Server reconstruct command. You must manually synchronize the accounts by using one of the following commands:
issadmin.sh --user user --checkaccount --sync
or
issadmin.sh --accountlist userlist --checkaccount --sync
Run the imqcmd command against the mail store IMQ broker, for example:
imqcmd list dst Username: admin Password: Listing all the destinations on the broker specified by: ------------------------- Host Primary Port ------------------------- localhost 7676 ------------------------------------------------------------------------- Name Type State Producers Consumers Msgs Total Count UnAck Avg Size ------------------------------------------------------------------------- INDEXMS Queue RUNNING 32 1 0 0 0.0 Successfully listed destinations.
Verify that INDEXMS exists and that producers (from Messaging Server) and one consumer (from Indexing and Search Service) are present.
Use the issadmin.sh command to view the status of an account:
issadmin.sh --user user --checkaccount --detail status
Problems might occur due to unsupported attachment types, errors in the attachment file (such as unreadable HTML or a faulty JPEG file that cannot be opened), or errors in the structure of the email. Look at the original email message to determine the specific problem.
Some errors could be due to text or plain attachments that are too large. You can increase the default setting of 2 MBytes by changing the iss.indexsvc.attachment.sizelimit parameter in the jiss.conf file. For example, the following entry increases the attachment size limit to 25 MBytes:
iss.indexsvc.attachment.sizelimit=25000000
If you increase the value of the iss.indexsvc.attachment.sizelimit parameter, then also increase the max heap size for indexSvc by using the java.args setting in the jiss.conf file. For example:
java.args=-Xmx4g
At present, you cannot rotate logs on demand. Instead, use the following procedure:
Disable the indexSvc service:
cd /var/opt/sun/comms/jiss/logs svcadm disable svc:/application/jiss-indexSvc:default
Move the log file to a new name:
mv iss-indexsvc.log.0 iss-indexsvc.log.1
Enable the indexSvc service:
svcadm enable svc:/application/jiss-indexSvc:default
Run the tail command to ensure that Indexing and Search Service is running and writing to the new log, for example:
tail -f iss-indexsvc.log.0
The following is an example excerpt from the GlassFish Server server.log file.
[#|2009-03-25T14:42:56.299-0700|INFO|sun-appserver9.1|org.restlet.Component (14487431)|_ThreadID=14;_ThreadName=httpSSLWorkerThread-8080-0;|2009-03-25 14:42:56 192.0.2.0 - 192.0.2.1 80 GET /rest/search q=%2busername:testuser%20%2bhostname:bc011.example.com%20%2bfolder:Junk %20%2bbody:"apple"&c=2147483647&contentformat=simpleuid&format=atom 403 337 - 3 http://host.example.com - -|#]
The last line in this example output shows that a 403 HTTP error (forbidden) was returned. This error occurs if the originating IP address (in this example, 192.0.2.0) is not stored in the mail.server.ip setting in the jiss.conf file. Indexing and Search Service currently grants access to the RESTful web service by IP address as a substitute for root user access.
If this error is occurring, ensure that you have correctly entered the IP address for the mail.server.ip setting. You can also enter a comma-delimited list of IP addresses when using multiple servers. If you change one or more IP addresses, make sure to stop and start the Indexing and Search Service services. For more information, see "Stopping and Starting Indexing and Search Service".