C Setting the Maximum Data Group Size

This appendix provides a detailed explanation of how the default maximum Data Browser group sizes can be increased to provide more accurate reporting of monitored traffic. It is strongly recommended that you carefully review the information presented before modifying the default settings.

Maximum Group Size and Condensation

In order to optimize performance, individual groups are not permitted to grow to unlimited sizes. Each main group table has a maximum permitted size. By default, this is 600 MB. There is no limit for session diagnostics information.

The maximum size of a group is managed by condensing. This is the process by which the number of rows within the group table are reduced by moving the least used data to a "other" group. For example, rows within the client-browser/version dimension might contain a small number of rows for rarely seen Mozilla Firefox versions. In this case, the number of rows within the table is reduced by replacing these with a single "other-firefox" row.

You might consider increasing the maximum size of the groups if the information you require is regularly being condensed. For example, required individual page names or user IDs are only reported as "other". Broadly speaking, this approach is recommended in the case of high levels of network traffic.

Setting the Maximum Group Size

If the maximum group size is set to more than 150% (that is, 900 MB), it is strongly recommended that the database is configured as a remote database. The maximum recommended group size is 300%^Foot 1 (that is, 1.8 GB). Be aware that response times to Data Browser queries may take longer to process if the maximum group size is increased.

It is important to note that custom dimensions have a significant impact on the condensing process. Incorrect configuration of custom dimensions can result in excessive condensing. For example, product names (or similar highly selective attributes) should not be configured as custom dimensions because that will skew the condensing process. Specifically, each page/product combination becomes a separate candidate for condensing, and only high frequency combinations will remain visible after condensing.

To determine the current data group sizes, issue the following command as the defined RUEI_USER user:

cop stats %date_filter

where date_filter specifies the date for which you want to view group sizes. Note that if you specify today's date, because the groups are still growing, the reported sizes may not represent their final sizes. Therefore, it is recommended that you issue the query for a previous day's date (for example, yesterday). A group's size is always reported as compressed and needs to be approximately doubled to determine its uncompressed size.

An example command and sample output would appear as follows:

cop stats %20091110

   DATA ROWS         DATA SIZE
data    desc      data     desc  cube name
----  ------   -------  -------  ----------
705532  565699  300.0 MB  30.0 MB  wg_visit_dy_20091110
918028   48727  303.0 MB   4.0 MB  wg_trans_dy_20091110
319163 1043110  169.0 MB  96.0 MB  wg_slowurl_dy_20091110
937011  222760  439.0 MB  11.0 MB  wg_sesdiag_dy_20091110
005411  005320    2.1 MB   0.3 MB  wg_service_dy_20091110
834063  126196  315.0 MB  20.0 MB  wg_page_dy_20091110
019008  001026    4.0 MB   5.0 MB  wg_kpi_dy_20091110
000345  004598    0.8 MB   0.6 MB  wg_keypage_dy_20091110
286384  967535  168.0 MB  88.0 MB  wg_failurl_dy_20091110
000378  005544    0.4 MB   0.2 MB  wg_failsrv_dy_20091110
632973 2175464  363.0 MB 151.0 MB  wg_failpg_dy_20091110

In the above example, it appears that the daily All pages (wg_page_dy), and two other groups have reached their maximum limits (300 MB of compressed data), and data is being condensed. Note that the Failed pages group (wg_failpg_dy), although it has over 300 MB of data, has not reached its limit of 1.4 million rows.

To set the maximum group size, issue the following commands:

$ su – moniforce
$ sqlplus /@$RUEI_DB_TNSNAME
SQL> update UXS_CONFIG set VALUE='900000000'
        where NAME='cube_max_size';

where the value specifies (in bytes) the maximum group size. In the above example, this is 900 MB (that is, 150%).

Note that the Failed URLs, Failed services, and Failed pages groups do not use the maximum group size setting. Instead, their size is controlled through the event_max_fail setting. This specifies the maximum number of rows that can added to the group's main database table during a 5-minute period. By default, this is 5000 rows. For the Slow URLs group, the event_max_slow setting is used, and specifies the number of the slowest URLs that are recorded within each 5-minute period. By default, this is 5000 rows.

Note that if you change the event_max_fail or the event_max_slow setting, you should also review the daily_max_fail setting. This specifies the maximum number of rows that the groups' tables can contain. This is derived from the formula 288 * event_max_fail. The default, is 1.4 million rows.

To activate the above changes, you need to issue the following commands:

$ sqlplus /@$RUEI_DB_TNSNAME
SQL> update UXS_CONFIG set VALUE='10000' where NAME='event_max_fail';
SQL> update UXS_CONFIG set VALUE='4320000' where NAME='daily_max_fail';

Note that the event_max_fail setting is limited to 10,000 rows.

Example - Increasing Maximum Group Size

Consider the case in which the maximum main group table size is increased to 900 MB (that is, 150% of the default maximum size). The maximum required database storage is indicated in Table C-0.

Table C-1 Default Required Database storage

Type of data retained	Default retention policy	Maximum group size (%)	DB space required per period (GB)	Total DB space required (GB)
Failed pages/URLs/services	15	150%	4.5	67.5
Daily	32	150%	3	96
Monthly	13	150%	3	39
Yearly	5	150%	3	15

Suites	15	150%	1.5	22.5

Additional overhead				10

Total required DB space				250

Reviewing the Effects of Changing the Maximum Group Size

After increasing the maximum group size setting, it is strongly recommended that you carefully review the effect of the change on the performance of the Reporter system. You should wait at least a full day, and select System, then Status, and the Data processing. Review the system load and determine whether the Reporter system can handle the additional processing overhead that increasing the maximum group size setting represents.

Important:

Increasing the data group sizes can have a significant impact on overall system performance. Therefore, any changes to data group sizes should be carefully planned, and performed in manageable steps.

Managing the Maximum Size of the Database

By default, the RUEI database is limited to a maximum size of 960 GB. This should be enough to meet your operational requirements. However, if the maximum database size is ever reached, processing automatically stops. You will then need to modify the Reporter data retention policies to reflect the size of available database storage, or increase the maximum database size, before processing can continue.

Footnote Legend

Footnote 1: Be aware that increasing the maximum group size will impact reporting performance. It is recommended that you incrementally increase the setting and verify that performance is acceptable.