This chapter describes how to administer the Oracle Communications Indexing and Search Service store.
When you create an Indexing and Search Service account, it is uniquely assigned to a single Indexing and Search Service store instance, which contains all the account's index and data information. To manage the size of the store and location of accounts, you must be able to move an account from one store instance to another. This process consists of two steps:
Exporting the information from its current store instance
Importing the exported information into another store instance
The export step creates a snapshot of the account information in a file structure separate from the store instance. This structure can be used as an archive and as a backup. An exported account is therefore effectively a recovery from that backup.
The following terms apply to the process of exporting and importing accounts:
Account. Set of user information uniquely identified by "user" and "host" strings.
Account snapshot. All information for an account captured at a given time by the issadmin.sh --export command.
Snapshot repository. Directory containing zero or more account snapshots. If you do not specify a snapshot repository by using the --eximpath option to the issadmin.sh command, the system uses the default snapshot repository to save snapshots.
The following restrictions are applicable to exporting and importing Indexing and Search Service accounts:
When exporting and importing an account, ensure that the account state is set to Inactive.
This setting ensures the validity of information while the export and import is occurring.
You can only export or import one account per command.
However, you can use the --accountlist file option to export or import multiple accounts in a single command, if the values specified for all the other command line options used are the same for all accounts in the list (for example, options like --rehost and --eximpath).
During an export and import, you can move the account but not rename it.
The user name must not change during the export and import process. To move the account to a different host, run the issadmin.sh command with the --rehost option. If the user name changes, the index records cannot be reused. The user name is treated as a new account, which you must then bootstrap.
The administrator is responsible for managing the snapshot repository contents.
Indexing and Search Service does not contain any automatic process for deleting old export snapshots. As you add new snapshots, disk space is used until you perform a cleanup of old snapshots. The eximsummarylist summary file, located in the snapshot base directory, can also be cleaned up by simply editing it. The eximsummarylist file contains information for all the users in one snapshot directory, and a roll-up of all the export commands (as opposed to being overridden from a new issadmin.sh --export command). If you do edit eximsummarylist, ensure that you do not invalidate the basic structure, because locating the proper snapshot to --import depends on its integrity. This file also serves as a log record of each --export that occurred in the snapshot repository.
In the following example, the user mailbox is initially on mailhostA and the Indexing and Search Service account is on isshostA. You are moving the mailbox to mailhostB and the Indexing and Search Service account to isshostB.
From isshostA, set the account to Inactive and export the account.
(isshostA)cd IndexSearch_home/bin
(isshostA)issadmin.sh --user userA --host mailhostA.example.com --setstate I
(isshostA)issadmin.sh --export --eximpath /export/archive --user userA --host mailhostA.example.com
Move the mailbox from mailhostA to mailhostB.
(mailhostA)cd MessagingServer_home/bin
(mailhostA)rehostuser -u userA -d mailhostB.example.com
From isshostB, create the Indexing and Search Service account.
(isshostB)cd IndexSearch_home/bin
(isshostB)/issadmin.sh --createaccount --userA --host mailhostB.example.com
Rehost the account from isshostA to isshostB.
(isshostA)issadmin.sh --import --eximpath /net/isshostA.example.com/export/archive --user userA --host mailhostB.example.com --rehost mailhostA.example.com
From isshostB, check the account.
(isshostB)issadmin.sh --checkaccount --user userA --host mailhostB.example.com
If the checkaccount command did not return "in sync," perform a manual synchronization.
(isshostB)issadmin.sh --checkaccount --sync --user userA --host mailhostB.example.com
Set the account to Active.
(isshostB)issadmin.sh --setstate A --user userA --host mailhostB.example.com
(Optional) Delete the account on mailhostA.
(isshostA)issadmin.sh --deleteaccount --user userA --host mailhostA.example.com
Each account snapshot consists of a directory in the snapshot repository. This directory contains directories for the meta index, content index, dIndex records (a modified subset for this account), data store files, and a plain-text summary file describing the snapshot. The snapshot also appears as part of the overall exisummarylist file. This summary identifies a specific account within the snapshot repository.
A snapshot repository can contain any number of such account snapshot directories. It also contains a plain-text summary file (eximsummarylist) that is used to navigate multiple snapshot directories in the repository.
The --eximpath option can specify any available directory. If you do not specify a directory, the --export and --import options use iss.store.dir/snapshots as the default. You can override the default setting by using the --setdefaulteximpath option to the issadmin.sh command.
The process of rehosting a user from one Messaging Server instance to another is simple and automated. The issrehostuser.sh script and the rehostuser command work together to move the account from its Indexing and Search Service instance to a new one while at the same time the account is moved to its new Messaging Server instance. For more information about using the rehostuser command, see the Messaging Server documentation. See "issrehostuser.sh" for syntax description and options.
This section describes how to monitor the status of the Indexing and Search Service store by using the isssadmin.sh command with options such as --listaccounts, --listfolders, and --checkaccounts. See "issadmin.sh" for more information about these options.
To list the information of all accounts in the Indexing and Search Service store, including any accounts currently being bootstrapped, run the following command:
issadmin.sh --listaccounts
To verify that accounts have been bootstrapped:
View the files in the log directory (defined by iss.log.dir in the IndexSearch_home/etc/jiss.conf file) to check for any problems while bootstrapping the account.
Run either issadmin.sh--checkaccount or issadmin.sh--checkfolder to check the account for problems in specific emails that might have occurred during bootstrapping.
These commands compare the data in the index against the current state of the Messaging Server store for that account and list any problems found in the index. For serious problems, you can delete the account (or just a single folder) and bootstrap it again to correct inconsistencies. Sometimes the bootstrapping does not process attachment data completely, causing incomplete search results, but the rest of the account data can be searched.
Log files contain error and informational messages about the operation of the various Indexing and Search Service processes. The indexSvc, searchSvc, and jmqconsumer processes also log statistics about their progress since the server was last restarted in separately configured log files. These log files provide insight into overall service status. You can use these log files to detect use patterns and performance issues more easily than analyzing the individually logged informational messages.
To generate a statistic snapshot for the indexSvc and jmqconsumer processes, use the issadmin.sh command with the --liststats and --listbacklog commands, respectively. For more information about these options, see "issadmin.sh".
The following example output shows the kinds of information that the --liststats option can provide on a running indexSvc service. The output shows the general format of statistics that are periodically recorded in a log file. The first line indicates that the frequency is about every 10 minutes (600 seconds, the default). You can vary this frequency to between 15 and 60000 seconds by using the iss.indexsvc.statisticsinterval configuration parameter. The second line describes the various columns that appear in each line of the data that follows.
For details about specific lines of information, see the Notes that follow the output.
Following the "autosync" lines are summary statistics for the Buffer Manager used within Lucene to reuse various sizes of data buffers to reduce memory pressure. The two sets of buffers are configured and behave independently of each other. If either the "too big buffer created" or "too big ByteB created" fields are not zero, additional information is printed indicating what size buffer was requested that could not be cached. Such buffers are created and released to the heap (not cached or reused), and, if they occur frequently, might be a source of large memory overhead (inducing garbage collection).
Fri Jul 06 09:06:59 GMT+00:00 2012 common.LogUtils.indexStats runLoop INFO: 607290 ms since prior sample, server has been running for 1.1 hours)
count description prior count delta percentage rate/sec
2314883 basic docs created 2259890 54993 2 91 (Note 1)
14 document types created 14
table of doctypes:
950418 xemailxmetax 935619 14799 1 24 (Note 2)
941960 xemailxcontentx 928396 13564 1 22 (Note 3)
93150 xemailxattachx 91877 1273 1 2 (Note 4)
167272 affpd 157615 9657 6 16 (Note 5)
92335 afd 83389 8946 10 14 (Note 6)
22761 amd 20086 2675 13 4 (Note 7)
31430 atdd 28139 3291 11 5 (Note 8)
10390 asd 9731 659 6 1 (Note 9)
0 admd
1 sdc 1 (Note 10)
2 sda 2 (Note 10)
3442 sdb 3356 86 2 0 (Note 10)
1 dd 1 (Note 11)
1721 agd 1678 43 2 0 (Note 12)
186300 attachment type fields create 183754 2546 1 4 (Note 13)
21 attachment types created 21
table of attachment types:
57924 pdf 57402 522 0 0
2474 plain 1270 1204 94 1
14 audio 2 12 600 0
342 odf 206 136 66 0
1872 ssign 1038 834 80 1
2726 image 1390 1336 96 2
232 html 178 54 30 0
314 compress 108 206 190 0
1800 vcf 1270 530 41 0
112376 jpeg 110352 2024 1 3
40 xml 30 10 33 0
898 other 712 186 26 0
32 ppt 18 14 77 0
68 doc 36 32 88 0
4944 pgpsign 3848 1096 28 1
30 rtf 26 4 15 0
48 video 22 26 118 0
28 xls 16 12 75 0
10 applefile 2 8 400 0
24 iwork 6 18 300 0
52 vsd 4 48 1200 0
651138 single doc searches 627338 23800 3 39 (Note 14)
158960 single doc search found none 150370 8590 5 14 (Note 15)
22943 single map doc searches 22753 190 0 0 (Note 16)
221513 account docs from cache 207014 14499 7 24 (Note 17)
141091 folder flag docs from cache 127189 13902 10 23 (Note 18)
1641865 total docs cached 1608713 33152 2 55 (Note 19)
140844 account 138145 2699 1 4
864760 folder 855814 8946 1 14
0 folder flags (obsolete)
636261 folder flags partial 614754 21507 3 35
0 account map
441613 total docs removed from cache 441560 53 0 0 (Note 20)
485 account 459 26 5 0
485 folder 459 26 5 0
0 folder flags (obsolete)
440643 folder flags partial 440642 1 0 0
0 account map
238638 size of account doc cache 238556 82 0 0
1560726 size of folder doc cache 1560686 40 0 0
108530 size of flags partial cache 93945 14585 15 24
46266 # folder flag sets cached 40330 5936 14 9
0 size of account map doc cache
125886 calls to closeAccess 118205 7681 6 12 (Note 21)
35323 calls to delayCloseAccess 31590 3733 11 6 (Note 22)
9716 closeAccess skipped 9161 555 6 0 (Note 23)
158610 closeWriters done 148836 9774 6 16 (Note 24)
119321 groupNumberCache size 119278 43 0 0 (Note 25)
119246 groupAccountGroupCache size 119207 39 0 0 (Note 26)
0 group Cache remote updates (Note 27)
0 group Cache entries cleared (Note 28)
379680 # IndexReaders opened 354571 25109 7 41 (Note 29)
11 # opens took over 1 sec 11
4 # opens took over 10 secs 4
148684 # dIndex IndexReaders opened 140264 8420 6 14 (Note 30)
56756 # opens took over 1 sec 51016 5740 11 9
6214 # opens took over 10 secs 5772 442 7 0
table of event types:
231875 Total events since index init 223109 8766 3 14 (Note 31)
36294 NewMsg events 32894 3400 10 5
20033 UpdateMsg events 17938 2095 11 3
0 Rename events
0 Delete events
0 Create events
8788 Copy events 7486 1302 17 2
13206 ExpungeMsg events 11283 1923 17 3
31579 ChangeFlag events 31579
1969 Bootstrap events 1923 46 2 0
0 SetAcl events
0 Login events
0 Logout events
0 unknown events
autosync:
19437 candidates added 16739 2698 16 4 (Note 32)
10000 accounts checked 9400 600 6 0
8690 accounts found OK 8687 3 0 0
1310 accounts needed sync 713 597 83 0
993 account checking skipped 396 597 150 0
2 sync fail: other command 2
0 sync fail: missing folder
0 sync fail: authentication
0 sync fail: other reasons
autobootstrap: (Note 33)
0 candidates reboot by age
0 candidates from checkstore
500 accounts bootstrapped
simple buffer cache limits: tiny 4096 small 16384 medium 6144 large 64 xlarge 16
simple buffer cache sizes now: tiny 106 small 4661 medium 5511 large 33 xlarge 11
simple buffer sizes (in kb): tiny 128b small 2 medium 16 large 128 xlarge 1024
4894m reused buffers 4610m 283790246 6 472924
2517m tiny buffer reused 2375m 141769185 5 236252
2343m small buffer reused 2203m 139914905 6 233162
25007652 medium buffer reused 23564990 1442662 6 2404
8265133 large buffer reused 7602221 662912 8 1104
17208 xlarge buffer reused 16626 582 3 0
32186 created buffers 30709 1477 4 2
156 tiny buffer created 156
5009 small buffer created 5009
26977 medium buffer created 25500 1477 5 2
33 large buffer created 33
11 xlarge buffer created 11
0 too big buffer created
4894m requested buffers 4610m 283791724 6 472927
2517m tiny buffer requests 2375m 141769186 5 236252
2343m small buffer requests 2203m 139914905 6 233162
25034437 medium buffer requests 23590298 1444139 6 2406
8265158 large buffer requests 7602246 662912 8 1104
17215 xlarge buffer requests 16633 582 3 0
4894m cached buffers 4610m 283792417 6 472928
2517m tiny buffer cached 2375m 141769186 5 236252
2343m small buffer cached 2203m 139915043 6 233162
25012971 medium buffer cached 23568277 1444694 6 2407
8265158 large buffer cached 7602246 662912 8 1104
17215 xlarge buffer cached 16633 582 3 0
20473 not cached buffers 18857 1616 8 2
0 tiny buffer not cached
0 small buffer not cached
20473 medium buffer not cached 18857 1616 8 2
0 large buffer not cached
0 xlarge buffer not cached
0 unknown size buffer not cache
ByteBuffer cache limits: tiny 4096 small 16384 medium 6144 large 64 xlarge 32
ByteBuffer cache sizes now: tiny 128 small 4639 medium 5476 large 32 xlarge 20
ByteBuffer sizes (in kb): tiny 128b small 2 medium 16 large 128 xlarge 1024
41008324 reused ByteB 38397306 2611018 6 4351
1271295 tiny ByteB reused 1179606 91689 7 152
16413900 small ByteB reused 15250954 1162946 7 1938
23130463 medium ByteB reused 21785860 1344603 6 2240
176798 large ByteB reused 165554 11244 6 18
15868 xlarge ByteB reused 15332 536 3 0
33957 created ByteB 32525 1432 4 2
128 tiny ByteB created 128
4987 small ByteB created 4987
28790 medium ByteB created 27358 1432 5 2
32 large ByteB created 32
20 xlarge ByteB created 20
0 too big ByteB created
41041241 requested ByteB 38428791 2612450 6 4353
1271295 tiny ByteB requests 1179606 91689 7 152
16418375 small ByteB requests 15255429 1162946 7 1938
23158869 medium ByteB requests 21812834 1346035 6 2243
176822 large ByteB requests 165578 11244 6 18
15880 xlarge ByteB requests 15344 536 3 0
41017579 cached ByteB 38404592 2612987 6 4354
1271295 tiny ByteB cached 1179606 91689 7 152
16418027 small ByteB cached 15254979 1163048 7 1938
23135555 medium ByteB cached 21789085 1346470 6 2243
176822 large ByteB cached 165578 11244 6 18
15880 xlarge ByteB cached 15344 536 3 0
22355 not cached ByteB 20723 1632 7 2
0 tiny ByteB not cached
0 small ByteB not cached
22355 medium ByteB not cached 20723 1632 7 2
0 large ByteB not cached
0 xlarge ByteB not cached
0 unknown size ByteB not cached
Tue Oct 15 13:44:21 PDT 2013 common.LogUtils.indexStats runLoop INFO: net change
to AddDocsThreadsList size: 2 had been 0
Tue Oct 15 13:44:21 PDT 2013 common.LogUtils.indexStats runLoop INFO: net change
to SingleFolderList size: 2 had been 0
Tue Oct 15 13:44:21 PDT 2013 common.LogUtils.indexStats runLoop INFO: net change
to FolderThreadsList size: 4 had been 1
Tue Oct 15 13:44:21 PDT 2013 common.LogUtils.indexStats runLoop INFO: statistics
sleeping for 600 seconds
Notes:
Number of Lucene documents created since the server started. Following this line is the number of different type of documents, which are then formatted in a table.
Number of meta data documents created. Each email must have one meta document. Therefore, this number represents the number of emails processed since the server started.
Number of content documents created. The body of an email is recorded here. This number is usually less than or equal to the number of meta documents.
Number of attachment documents created. Each email attachment causes one such document to be created. Usually this count is much smaller than the previous two counts.
Number of account folder flag partial documents created. Flag information for each email in each folder of an account is grouped in these documents. Updated whenever email flag values change.
Number of account folder documents created. Each folder in each account requires one such document to record details such as name, current number of emails, and so on.
Number of account map documents created. This number is the basic account information record, containing identification data (host, user) plus account structure (number of folders, and so on).
Number of account transition date documents created. Used to track the time of major account state transitions (such as creation of new emails in account, and so on).
Number of account state documents created. Used to track each account state transition.
Number of store summary documents created. Three separate document types are created (a,b,c) to represent the data because different parts are usually updated at different frequencies depending on what processing occurs. Data includes global counts, fixed configuration parameters, and so on.
Dummy used for initializing empty index directories; always exactly one.
Number of account group documents created. Updated when accounts are assigned, removed, or reassigned to an index group.
Number of attachment type fields created. Content and attachment documents use these fields to identify the kind of attachments in the email. The table shows how many attachments of each kind were processed. Usually it is exactly twice the number of xemailxattachx documents created.
Number of times a particular document was searched for (such as affpd and amd doctypes).
Number of times a particular document was searched for and not found. This situation is expected at various times, such as when creating a new account, where the account record must be checked not already to exist, and so is expected not to be found before it is created.
Number of times a single document was searched for (usually afd and amd). It uses a different interface, so a restricted subset of the counts in the previous two lines.
Number of times an account document was found in the memory cache when looked up, thus avoiding the disk I/O overhead.
Number of times a folder document was found in the memory cache when looked up, thus avoiding the disk I/O overhead.
Number of documents added to the in-memory cache. Various kinds of documents are cached to avoid I/O overhead because they are frequently accessed and infrequently changed. Account, folder, and flag documents are counted separately.
Number of documents removed from in-memory cache. Cached documents are removed when data is stale, when it might also be necessary to update the index on disk.
Number of times access to an account was released to ensure that the account is no longer modified, and any index writers can be closed.
Number of times release of access to an account was delayed, causing a delay in the close of any open index writers. This process is useful when several real-time events for an account occur in quick succession, since the rapid close and reopen of the index is unnecessary overhead and a potentially large throughput bottleneck.
Number of times a delayedClose was skipped (for example, no close done). This situation occurs when real-time events occur so rapidly that a new event requiring the same index writer arrives before the previous delayed close has completed. Thus this number measures how often the close and reopen was avoided between subsequent events, which can speed up processing at the cost of delaying the disk index update.
Number of times any index writer was actually closed. Counts the close of the meta and content index writer for every account; indicates how frequently index directories on disk are being updated.
Number of group number to account associations cached. Assignment of any account to a group occurs when it is created or moved, and does not change often so this information is cached in memory to speed up lookup (avoids disk I/O). This line is primarily interesting only when creating, moving, importing, or deleting accounts to indicate progress.
Number of group documents currently cached. Group information is cached in memory because it changes infrequently after accounts have been bootstrapped and assigned into groups. This line is primarily interesting only when creating, moving, importing, or deleting accounts to indicate progress.
Number of times a group notification was sent to remote servers (such as index service to search service). This situation happens when major changes to the in-memory group cache occurs to signal remote servers to update their in-memory group caches. (This feature is disabled by default. See the iss.store.groupcache.enabled parameter for more information.)
Number of group cache entries sent in group notifications to remote servers. This number is a cumulative total of all group entries that are removed from the group cache. This feature is disabled by default. See the iss.store.groupcache.enabled parameter for more information.
Number of times any IndexReader was opened.
Number of times an IndexReader was opened for the dIndex directory. Included in the total for any IndexReader opened.
Total number of events processed by the index service since the store instance was created. Below this total is the break down of the events by type processed since the service was last started. These numbers restart at zero if the service is restarted, but the total accumulates across server restart.
Total number of accounts added to the autosync candidates list after the initial full candidate list was created. These are accounts where events were dropped or lost, which caused the system to add the account to the front of the list to be corrected soon.
Number of users scheduled for rebootstrapping due to account age (the default is one year) and the total number of autobootstrapped accounts in the statistics period.
The following example output shows the kind of information that is logged for the searchSvc service. The example below shows the general format of statistics that are periodically recorded in a log file. These numbers are totals accumulated since the service was started. The first line indicates that the frequency is about every 30 minutes (1800 seconds, the default). You can vary this frequency to between 15 and 180000 seconds by using the iss.searchsvc.statisticsinterval configuration parameter. The second line describes the various columns that appear in each line of the data that follows.
For details about specific lines of information, see the Notes that follow the output.
Following the "search time average" lines are summary statistics for the Buffer Manager used within Lucene to reuse various sizes of data buffers to reduce memory pressure. The two sets of buffers are configured and behave independently of each other. If either the "too big buffer created" or "too big ByteB created" fields are not zero, additional information is printed indicating what size buffer was requested that could not be cached. Such buffers are created and released to the heap (not cached/reused), and, if they occur frequently, might be a source of large memory overhead (inducing garbage collection).
Tue Oct 15 13:44:11 PDT 2013 common.LogUtils.searchStats runLoop INFO: 1800298 ms since prior sample, server has been running for 2.0 hours
count description prior count delta percentage rate/sec
54430 single doc searches 35200 19230 54 10 (Note 1)
0 single doc search found none
54430 single map doc searches 35200 19230 54 10 (Note 2)
0 account docs from cache
0 folder flag docs from cache
54395 calls to closeAccess 35174 19221 54 10 (Note 3)
0 calls to delayCloseAccess (Note 4)
0 closeAccess skipped (Note 5)
0 closeWriters done (Note 6)
17730 groupNumberCache size 13219 4511 34 2 (Note 7)
17730 groupAccountGroupCache size 13219 4511 34 2 (Note 8)
0 group Cache remote updates (Note 9)
0 group Cache entries cleared (Note 10)
158711 # IndexReaders opened 102557 56154 54 31 (Note 11)
5 # opens took over 1 sec 2 3 150 0
0 # opens took over 10 secs
54451 # dIndex IndexReaders opened 35212 19239 54 10 (Note 12)
1689 # opens took over 1 sec 368 1321 358 0
0 # opens took over 10 secs
search statistics:
54451 total queries 35211 19240 54 10 (Note 13)
0 query failed
0 query parse failed
0 query parse token failed
0 query rewritten
35327 total number of folder terms 21267 14060 66 7 (Note 14)
table of search folder values: (Note 15)
35327 "INBOX" 21267 14060 66 7
table of search field values: (Note 16)
19124 attachment-type 13944 5180 37 2
35327 folder 21267 14060 66 7
24668 subject 17570 7098 40 3
10659 body 3697 6962 188 3
search result counts: (Note 17)
20098 zero results 12303 7795 63 4
17897 1-10 results 11777 6120 51 3
11127 11-100 results 7535 3592 47 1
4760 101-1000 results 3234 1526 47 0
515 more than 1000 results 326 189 57 0
search time counts: (Note 18)
47516 less than 1 second 32447 15069 46 8
6830 1 less than 5 seconds 2725 4105 150 2
48 5 less than 10 seconds 2 46 2300 0
3 10 less than 20 seconds 1 2 200 0
0 20 less than 30 seconds
0 30 seconds or over
search time totals (ms): (Note 19)
20830416 less than 1 second 13658003 7172413 52 3984
11131942 1 less than 5 seconds 3751215 7380727 196 4099
290501 5 less than 10 seconds 12692 277809 2188 154
39714 10 less than 20 seconds 11836 27878 235 15
0 20 less than 30 seconds
0 30 seconds or over
search time average (ms): (Note 20)
438 less than 1 second 420 18 4 0
1629 1 less than 5 seconds 1376 253 18 0
6052 5 less than 10 seconds 6346
13238 10 less than 20 seconds 11836 1402 11 0
0 20 less than 30 seconds
0 30 seconds or over
simple buffer cache limits: tiny 4096 small 16384 medium 6144 large 64 xlarge 16
simple buffer cache sizes now: tiny 64 small 1090 medium 3295 large 8 xlarge 4
simple buffer sizes (in kb): tiny 128b small 2 medium 16 large 128 xlarge 1024
1199m reused buffers 756545319 442563958 58 245828
579180081 tiny buffer reused 369498075 209682006 56 116470
593141149 small buffer reused 371354650 221786499 59 123194
24863511 medium buffer reused 14524620 10338891 71 5742
1924536 large buffer reused 1167974 756562 64 420
0 xlarge buffer reused
4586 created buffers 2320 2266 97 1
64 tiny buffer created 64
1117 small buffer created 1117
3393 medium buffer created 1127 2266 201 1
8 large buffer created 8
4 xlarge buffer created 4
0 too big buffer created
1199m requested buffers 756547115 442566224 58 245829
579180081 tiny buffer requests 369498075 209682006 56 116470
593142010 small buffer requests 371355511 221786499 59 123194
24866712 medium buffer requests 14525555 10341157 71 5744
1924536 large buffer requests 1167974 756562 64 420
0 xlarge buffer requests
1199m cached buffers 756546029 442567185 58 245829
579180081 tiny buffer cached 369498075 209682006 56 116470
593141983 small buffer cached 371355268 221786715 59 123194
24866614 medium buffer cached 14524712 10341902 71 5744
1924536 large buffer cached 1167974 756562 64 420
0 xlarge buffer cached
0 not cached buffers
0 tiny buffer not cached
0 small buffer not cached
0 medium buffer not cached
0 large buffer not cached
0 xlarge buffer not cached
0 unknown size buffer not cache
ByteBuffer cache limits: tiny 4096 small 16384 medium 6144 large 64 xlarge 32
ByteBuffer cache sizes now: tiny 126 small 1087 medium 3295 large 8 xlarge 8
ByteBuffer sizes (in kb): tiny 128b small 2 medium 16 large 128 xlarge 1024
27492648 reused ByteB 16168752 11323896 70 6290
198471 tiny ByteB reused 123644 74827 60 41
2503419 small ByteB reused 1591446 911973 57 506
24746464 medium ByteB reused 14426276 10320188 71 5732
44294 large ByteB reused 27386 16908 61 9
0 xlarge ByteB reused
4653 created ByteB 2386 2267 95 1
128 tiny ByteB created 128
1115 small ByteB created 1115
3394 medium ByteB created 1127 2267 201 1
8 large ByteB created 8
8 xlarge ByteB created 8
0 too big ByteB created
27496261 requested ByteB 16170098 11326163 70 6291
198471 tiny ByteB requests 123644 74827 60 41
2504022 small ByteB requests 1592049 911973 57 506
24749474 medium ByteB requests 14427019 10322455 71 5733
44294 large ByteB requests 27386 16908 61 9
0 xlarge ByteB requests
27496132 cached ByteB 16169016 11327116 70 6291
198469 tiny ByteB cached 123644 74825 60 41
2503994 small ByteB cached 1591810 912184 57 506
24749375 medium ByteB cached 14426176 10323199 71 5734
44294 large ByteB cached 27386 16908 61 9
0 xlarge ByteB cached
0 not cached ByteB
0 tiny ByteB not cached
0 small ByteB not cached
0 medium ByteB not cached
0 large ByteB not cached
0 xlarge ByteB not cached
0 unknown size ByteB not cached
Tue Oct 15 13:44:11 PDT 2013 common.LogUtils.searchStats runLoop INFO: statistics sleeping for 1800 seconds
Notes:
Number of times a particular document was searched for (such as affpd and amd doctypes). Refer to corresponding IndexServer statistics for more detail.
Number of times a single document was searched for (usually afd and amd). It uses a different interface, so a restricted subset of the counts in the previous two lines.
Number of times access to an account was released. This process ensures that the account is no longer modified, so any index writers can be closed. Search does not write to any accounts.
Number of times release of access to an account was delayed, causing a delay in the close of any open index writers. Since search never writes to any accounts, this value should always be zero.
Number of times a delayedClose was skipped (for example, no close done). Because search never writes to any accounts, this value should always be zero.
Number of times any index writer was actually closed. Because search never writes to any accounts, this value should always be zero.
Number of group number to account associations cached. Assignment of any account to a group occurs when it is created or moved, and does not change often so this information is cached in memory to speed up lookup (avoids disk I/O). This line is primarily interesting only when creating, moving, importing, or deleting accounts to indicate progress.
Number of group documents currently cached. Group information is cached in memory because it changes infrequently after accounts have been bootstrapped and assigned into groups. This line is primarily interesting only when creating, moving, importing, or deleting accounts to indicate progress.
Number of times a group notification was sent to remote servers (such as index service to search service). Refer to corresponding IndexServer statistic for details.
Number of group cache entries sent in group notifications to remote servers. Refer to corresponding IndexServer statistic for details.
Number of times any IndexReader was opened.
Number of times an IndexReader was opened for the dIndex directory. Included in the total for any IndexReader opened.
Number of search queries received. The lines following this number show the counts of failures by various types, and the number of times a search query was rewritten into a different form for easier processing. (The rewritten count is of interest primarily to developers.) Note the number of successful searches is found by subtracting the failure counts from the total received.
Number of search queries that contained any folder term. Most searches specify a folder to search. Those searches that do not specify a folder search the entire account, across all folders.
Table of counts of searches with folder terms, grouped by name of folder. This table will have a line for each unique folder name searched. In this example only the INBOX folder was ever searched.
Table of counts of searches with each kind of term. The hostname and username terms are not counted because every search query must contain these terms.
Table of counts of searches based on how many results were returned.
Table of counts of searches based on how long each search took.
Table of total times taken by searches based on how long each took. Times are in milliseconds. This table shows how long the search server spends processing these types of search query.
Table of average times taken by searches based on how long each took. Times are in milliseconds. This table shows how long a typical search in each category took.
The following example output shows the kind of information that the --listbacklog command option can provide from a running JMQConsumer service. The example shows the general format of statistics that are periodically recorded in a log file. These numbers are totals accumulated since the service was started.
The first two lines report how often the service runs (about every 30 minutes (1800 seconds)), and how long the server has been running.
The next set of lines list the counts of backlogged events for each account with greater than zero events queued, grouped by account state. The number of lines in this set varies depending on how many accounts have events backlogged. Many of the lines indicating "Active" and "Inactive" accounts have been omitted in this example because they are so numerous. The summary of this information follows, starting with the totals for all events backlogged and for all users, with these totals then divided by account state.
After this summary, a table of counts is displayed, listing the number of events received, finished, failed, skipped, or dropped since the service started. Finally, the number of accounts whose state changed since the previous statistics were generated is logged, with the time required to generate these statistics.
For details about specific lines of information, see the Notes that follow the output.
Tue Oct 15 13:14:13 PDT 2013 common.LogUtils.jmqStats run INFO: queue check sleeping for 1800 seconds
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats runQueueCheck INFO: runQueueCheck: started,
server has been running for 2.5 hours
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: All users share host
sc11b.example.com unless explicitly present in following messages:
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:
3025193@colorone.org:2 3099348@colorone.org:1 3076509@colorone.org:1 3114789@colorone.org:1
3014263@colorone.org:2 3063475@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:
3100179@colorone.org:2 3021434@colorone.org:1 3026333@colorone.org:1 3087690@colorone.org:1
3050238@colorone.org:2 3049006@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:
3025356@colorone.org:1 3012582@colorone.org:1 3034605@colorone.org:1 3062376@colorone.org:5
3071166@colorone.org:1 3075015@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:
3080474@colorone.org:2 3059811@colorone.org:1 3080156@colorone.org:1 3088724@colorone.org:3
3012326@colorone.org:1 3010701@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:
3029686@colorone.org:3 3036588@colorone.org:1 3118407@colorone.org:1 3090770@colorone.org:5
3112683@colorone.org:1 3112397@colorone.org:2
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:
3098379@colorone.org:1 3109379@colorone.org:1 3075342@colorone.org:1 3057362@colorone.org:1
3005288@colorone.org:1 3003412@colorone.org:1
...
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Inactive:
3091024@colorone.org:1 3013443@colorone.org:1 3066022@colorone.org:2 3118394@colorone.org:1
3039528@colorone.org:1 3107080@colorone.org:3
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Inactive:
3013876@colorone.org:3 3031089@colorone.org:2 3116432@colorone.org:3 3016384@colorone.org:2
3097988@colorone.org:1 3031007@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Inactive:
3056847@colorone.org:3 3065067@colorone.org:2 3116299@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Bootstrap:
3099295@colorone.org:5 3107340@colorone.org:3 3030422@colorone.org:2 3060000@colorone.org:13
3116786@colorone.org:2 3045641@colorone.org:2
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Total: 28755 Users: 18556
Total active: 28577 Active Users: 18469 Total inactive: 151 Inactive Users: 81 Total bootstrap: 27
Bootstrap Users: 6 Total Unknown: 0 Unknown Users: 0
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO:
count description prior count delta percentage rate/sec
203129 events Received 132351 70778 53 39 (Note 1)
0 events unsupported
109975 events Finished 84252 25723 30 14 (Note 2)
0 events failed/Not Finished
488 events with no account 407 81 19 0 (Note 3)
285 events skipped from queues 227 58 25 0 (Note 4)
0 events dropped: noops
63582 events dropped: changeflags 29130 34452 118 19
0 events dropped: others
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats call INFO: 0 account states changed, took 460ms
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats runQueueCheck INFO: runQueueCheck: done
Notes:
Number of events read. This number is the total of all events processed, dropped, skipped, or backlogged since the service started.
Number of events successfully processed by the Index Service.
Number of events read for which no account is defined in the Index Service. These events are ignored.
Number of events skipped. These are events previously in the backlog that were not processed for a variety of reasons. (This number is different from the dropped events which were never backlogged.)
If you deploy a firewall between your Messaging Server and Indexing and Search Service hosts, you must open an additional port on the firewall for Message Queue (JMQ). That is, JMQ requires two ports. For more information, see "Connecting Through a Firewall" in Sun GlassFish Message Queue Administration Guide at:
http://docs.oracle.com/cd/E19226-01/821-0027/gcuhq/index.html
The Service Management Facility (SMF) enables you to manage the operating system and application services, such as Indexing and Search Service. SMF replaces the legacy init scripting start-up mechanism common in prior releases of Oracle Solaris. SMF improves the availability of a system by ensuring that essential system and application services run continuously even when hardware or software failures occur.
The following steps describe how to configure the iss.user to be able to manage Indexing and Search Service through SMF. In this example, the iss.user is defined to be jiss.
Log in as root.
Set up Indexing and Search Service so that its services are managed by the user jiss by adding the following line to the /etc/security/auth_attr file.
solaris.smf.manage.iss:::Manage ISS Service States::
Run the following command to add the user privilege.
usermod -A "solaris.smf.manage.iss" jiss
Run the svccfg command to configure the utilSvc service then refresh the configuration.
svccfg -s jiss-utilSvc setprop general/action_authorization=astring: 'solaris.smf.manage.iss' svccfg -s jiss-utilSvc setprop general/value_authorization=astring: 'solaris.smf.manage.iss' svcadm refresh jiss-utilSvc
Run the svccfg command to configure the indexSvc service then refresh the configuration.
svccfg -s jiss-indexSvc setprop general/action_authorization=astring: 'solaris.smf.manage.iss' svccfg -s jiss-indexSvc setprop general/value_authorization=astring: 'solaris.smf.manage.iss' svcadm refresh jiss-indexSvc
Run the svccfg command to configure the searchSvc service then refresh the configuration.
svccfg -s jiss-searchSvc setprop general/action_authorization=astring: 'solaris.smf.manage.iss' svccfg -s jiss-searchSvc setprop general/value_authorization=astring: 'solaris.smf.manage.iss' svcadm refresh jiss-searchSvc
Run the svccfg command to configure the jmqconsumer service then refresh the configuration.
svccfg -s jiss-jmqconsumer setprop general/action_authorization=astring: 'solaris.smf.manage.iss' svccfg -s jiss-jmqconsumer setprop general/value_authorization=astring: 'solaris.smf.manage.iss' svcadm refresh jiss-jmqconsumer
Verify the configuration.
su - jiss /usr/sbin/svcadm restart jiss-jmqconsumer /usr/sbin/svcadm restart jiss-searchSvc /usr/sbin/svcadm restart jiss-indexSvc /usr/sbin/svcadm restart jiss-utilSvc
The Global Directory Index, also referred to as the dIndex, controls the Indexing and Search Service store. This section describes how to improve dIndex performance.
On average, the dIndex directory of a typical Indexing and Search Service store instance containing 150,000 accounts requires about 400 MB of disk space. Read, write, and especially segment merge operations on such large index directories are memory intensive and put a heavy workload on the CPU. Processing is also slowed due to how disk writes are serialized, even on updates to separate accounts. To reduce processing overhead as the dIndex grows over time, Indexing and Search Service 1.0.5.18.0 and greater uses an alternate form of the dIndex, which distributes data for different accounts across multiple, smaller directories. This data distribution enables quicker access in parallel for independent account transactions. Using this dIndex alternative form, most dIndex data reads and updates require a smaller memory footprint, and fewer transactions need to be serialized.
The first implementation of the dIndex is known as "format 1." The form of the dIndex introduced in Indexing and Search Service 1.0.5.18.0 is known as "format 2." Both dIndex format 1 and 2 are fully functional, and either form can be used for any Indexing and Search Service instance independent of any other instance in a multiple Indexing and Search Service instance deployment. However, you can only use the format 2 dIndex with software installed from Indexing and Search Service 1.0.5.18.0 or greater.
Format 1 remains the default form of the dIndex. To use format 2, set the iss.store.partitions.count configuration parameter to a value greater than zero. You can only change this parameter when Indexing and Search Service services are stopped. Thus, this parameter is not refreshable. Once you have defined the value of this parameter for an instance, you can only change the format of the dIndex by using the issadmin.sh --converttoformat command.
You can set the partition count value in the jiss.conf file before creating a new Indexing and Search Service instance, and the Indexing and Search Service store can be populated by bootstrapping as before by using the value supplied. Once the dIndex has been created, usually the first time the Index Service has been started, the value is fixed in the dIndex, and can only be subsequently modified by using the --converttoformat command. You must convert any Indexing and Search Service store instances created before Indexing and Search Service 1.0.5.18.0 to use this feature.
The value of the iss.store.partitions.count parameter defines the number of dIndex directories created for the store. If the count is zero, only the format 1 dIndex is created. For any value N greater than zero, the format 2 dIndex is created. N additional directories are created in the same directory as the dIndex named dIndexXX, where XX takes the value 00 thru N-1. When the --converttoformat 2 command is used to convert an existing format 1 dIndex, account groups are distributed across the N dIndexXX directories more or less evenly to balance the load. Once assigned to a specific dIndexXX directory, the group does not move, but any account may be moved to a different group (as with the format 1 implementation).
Information for all accounts in each group is distributed to the same dIndexXX directory. The amount of information for each account might vary greatly, due to the amount of content and structure of the account (for example, how many emails and how many folders the account contains). If the distribution of accounts is unbalanced, you can move accounts between account groups after the --converttoformat 2 command completes by using the --moveaccount command. Each dIndexXX directory is independently backed up in the same manner as the dIndex directory. (Thus, if you need a backup to recover from data corruption in the dIndex, you only need to substitute backup files individually for those partitions exhibiting problems, limiting potential data loss.)
The choice of value for the iss.store.partitions.count parameter depends on the number of accounts, account groups, and amount of data in the dIndex and account group directories. Theoretically, the larger the partition count, the more potential parallel activity between accounts is possible. The memory needed to process each individual account should typically be less because only a fraction of the dIndex must be referenced for each transaction. However, if more transactions run in parallel, the cumulative memory use at any given time could be greater than without partitioning. For a format 1 dIndex size up to about 0.5 Gbyte, a partition count value of 5 to 10 typically reduces the average dIndexXX size enough to improve performance. For larger format 1 dIndex sizes up to 1 Gbyte and more, higher partition count values might be needed.
Note:
Limited tests have shown performance improvement plateaus rapidly after partition count 5 for a format 1 dIndex size of about 0.5 Gbyte. Size is not the only factor, so it is not possible to predict accurately how your performance is affected.The partition number for any group is displayed in the --accountinfo output (on the line containing the meta and content index sizes) whenever the partition count is not zero. The --converttoformat command takes up to about 10 minutes to run for most Indexing and Search Service store instances up to about 1 Gbyte of data, depending on your specific configuration and hardware.
Changing the partition count for a format 2 dIndex involves converting back to format 1, updating the iss.store.partitions.count parameter, then converting back to format 2 using a different partition count. To avoid confusion, remove old backup files that are no longer valid for the new partition count.
To change the partition count:
Convert the dIndex back to format 1 by editing the jiss.conf file and setting the iss.store.partitions.count parameter to 0.
Run the --converttoformat 1 command.
issadmin.sh --converttoformat 1
Convert the dIndex back to format 2 by editing jiss.conf and setting iss.store.partitions.count to a non zero partition.
Run the --converttoformat 2 command.
issadmin.sh --converttoformat 2
To check for backup files that are no longer valid, change to the directory specified by the iss.store.dir configuration parameter.
Remove unneeded backup files.
rm -rf dIndexNN.backup*