|Oracle® Real User Experience Insight User's Guide
Release 11.1 for Linux x86-64
Part Number E22309-01
|PDF · Mobi · ePub|
This chapter describes how the reporting of monitored traffic can be fine optimized to meet your information requirements. This includes the specification of the cookie technologies used within your network environment, use of named Web server and client groups, as well as a number of advanced facilities, such as rule ordering and data retention policies.
You can open an overview of the monitored network traffic by selecting System, then Status, and then Data processing. This provides you with immediate information about hits, pages, and session processing, as well as the system load. An example is shown in Figure 12-1.
Note the Available resource usage (%) item on the Performance tab indicates the current processing level. If this approaches 100%, it means a lag in the processing of data is starting to occur, and it is no longer possible to process data in real time.
Be aware that because this facility is based on application logic, non-application traffic (such as suites, services, and SSOs), are not represented in the displayed reports.
Important:In order for RUEI to correctly report on monitored traffic, it is strongly recommended that you regularly review this traffic summary. If necessary, review the RUEI configuration accordingly. For example, add additional cookie technologies. In addition, if the system is unable to track sessions, proper tracking of user flows will also not be available because user flow reporting requires session tracking.
In order to accurately monitor your Web environment, RUEI needs to know and understand the cookie technology your Web site is using. This will either be a standard technology (such as ASP or ColdFusion), or a custom implementation. In the case of the latter, you will need to provide the system with information about it. Note that you can define a maximum of five cookie technologies for use when monitoring.
To specify your cookie technology, do the following:
Select Configuration, then Applications, and then Session tracking. Note that this option is only available to Administrators. The currently defined cookie settings are displayed. An example is shown in Figure 12-2.
Click Add new cookie or an existing cookie definition. A dialog similar to the one shown in Figure 12-3 appears.
Select the cookie technology used in your Web environment from the Cookie type menu. If you are using a non-standard technology, select "(custom)".
If you selected "(custom)", you are required to specify the name of the cookie used by your organization. Note that you can specify wildcard characters (*) as part of the cookie name.
Note:Cookie names are case sensitive.
If you select "(URL argument)", you are required to specify the name of the URL argument used by your organization. The use of URL arguments in session tracking is fully explained in Appendix B, "Cookie Structures". When ready, click Save.
Any changes made to this setting are applied after a short interval (typically, 5 - 10 minutes), and are then visible within the Reporter system shortly after this.
As mentioned earlier, session tracking is based on cookies. However, in certain circumstances, a cookie may not be suitable or available. For example, consider the following situations:
The cookie changes with every hit (for instance, this is the case with
The path set within the cookie only covers part of the application.
Do the following:
Add code similar to the following to the appropriate login page:
Note that the above code is for informational purposes only. You may need to modify it to meet your specific requirements.
Select Configuration, then Applications, and then Session tracking. Click Add new cookie. The dialog shown in Figure 12-4 appears.
To verify that your cookie configuration is being tracked correctly, do the following:
Clear all cookies in the browser.
(Re)login to the monitored application.
Perform a number of page views.
Logout out of the monitored application.
Wait for at least 10 minutes.
Open the RUEI Reporter environment, and select Browse data, open the All sessions group, select Session diagnostics, and locate the recorded session (by user ID or time). You can filter on applications.
Open the session and verify that there where more page views than just the login page. This verifies that the session ID is preserved after the login.
If you do not specify a cookie technology, then (by default) a combination of the client network and client browser is used to track sessions. However, in the event that this is not suitable for your environment, the client IP address can be used as an alternative tracking mechanism.
To specify the fallback session tracking mechanism, do the following:
Select Configuration, then Applications, and then Session tracking. The currently defined cookie settings are displayed. Click the currently defined session tracking fallback mechanism. The dialog shown in Figure 12-5 appears.
Use the Tracking mechanism menu to specify if a client network and browser combination should be used (the default), or the client IP address.
When ready, click Save. Any change you make takes effect immediately.
When considering which fallback mechanism to use, a general rule is that external-facing applications should use the default network/browser combination, while internal-facing applications should use client IP address. In the case of multiple users behind the same proxy server, the use of the default fallback mechanism is recommended. However, be aware this will result in all such users being recorded in one single session. The use of the client IP address mechanism is generally recommended in the following circumstances:
All users have a unique IP address. Note that for each application, you can specify if the client IP address should be retrieved from the TCP packet or a specific HTTP request header. This is described in Appendix O, "Monitoring NATed Traffic".
The organization enforces the use of a normalized browser. That is, a standard browser (such as Internet Explorer or Mozilla Firefox), with a standard version and plug-ins.
Some (or all) of the monitored applications are partially implemented in Java. Oracle E-Business Suite (EBS) is an example of such an application architecture. For these applications, the use of the client IP address mechanism prevents both Java and client requests appearing in the same reported session.
Important:The accurate specification of the cookie technologies used within your Web site is strongly recommended to ensure the accurate reporting of your network traffic.
In addition, you should that the cookie specified to track visitor sessions is not blinded. If it is, session creation based on the cookie will fail.
Optionally, you can use the Named servers facility to obtain more detailed insight into the visitors to your monitored Web sites. This facility allows you to assign ranges of server IP addresses (specified in the netmask) to a Web server group, and to individual Web servers. For example, a server group could be a department or data center, and the server name refers to specific Web servers within that group. In this way, you can easily identify the location of specific Web servers when problems (such as failed pages) occurred.
To use this facility, do the following:
Select Configuration, then General, and then Named servers. This option is only available to users with IT Analytical level access. The currently defined named servers are displayed. Click Add new server. The dialog shown in Figure 12-6 appears:
Use the fields within the dialog to specify a range of IP addresses or a specific IP address within a netmask, and the associated Web server and its group. When ready, click Save.
Optionally, you can click Upload list to merge a list of named servers with those that are currently defined. The file must contain only one entry per line, and the information for each server (as shown in Figure 12-6) must be tab-separated. Note that any definition in the merged file for an already defined named server overwrites its existing definition.
Any changes made to the named server groups are applied after a short interval (typically, 5-10 minutes), and are then visible within the Reporter system shortly after this.
The Web server information collected during monitoring can be viewed in the Data Browser via the All pages, Key pages, All functions, Failed functions groups, Failed URLs, Failed pages, and Slow URLs groups. The server IP identifies the specified IP addresses, and the server group refers to the group name. By zooming into a server group, you can view the individual Web server names that comprise the group. Zoom in again, and you can view the individual IP addresses assigned to that Web server.
In some instances, you want to be able to enhance the information associated with visitor IP addresses. This is especially useful when monitoring Intranet traffic and you want to be able to use your own client classification.
To use this facility, do the following:
Select Configuration, then General, and then Named clients. The currently defined named servers are listed. Click Add new client. This option is only available to IT users with Analytical level access. The dialog shown in Figure 12-7 appears.
Use the fields within the dialog to specify a range of IP addresses or a specific IP address within a netmask, the client, and their associated group (for example, company department). When ready, click Save.
Optionally, you can click Upload list to merge a list of named clients with those that are currently defined. The file must contain only one entry per line, and the information for each client (shown in Figure 12-7) must be tab-separated. Note that any definition in the merged file for an already defined named client overwrites its existing definition.
Any changes made to your defined named client groups are applied after a short interval (typically, 5-10 minutes), and are then visible within the Reporter system shortly after this.
Hit failures are recorded in the failed URL group. Because hit failures can occur for a wide variety of reasons, you can control what is recorded. For example, it is unlikely that you want incidents related to remote robot searches to be recorded. Do the following:
Select Configuration, then General, then Advanced settings, then URL reporting, and then Ignore failed URLs. Note that this option is only available to Administrators. The dialog shown in Figure 12-8 appears.
Specify any file names that should be ignored within the failed URL view. That is, they should not be seen as errors. Note that any directory information within file name definitions are ignored, and the defined files are also removed from the listed object URLs. Click Add to define a new file name that should be ignored. Click the Remove icon to the right of a defined file name to delete it from the list of files to be ignored.
Upon installation, two default files,
favicon.ico, are automatically configured. When ready, Click Save. Any changes to this setting are applied after 10 minutes. A short period after this time, the changes you have specified are visible in the Reporter interface.
You can control whether you want all, some, or no URL arguments recorded within the lowest level page URL dimension. Do the following:
Select Configuration, then General, then Advanced settings, then Page URL argument filtering, and then Page URL argument filtering. Note that this option is only available to Administrators. The dialog shown in Figure 12-9 appears.
Use the Argument filter menu to select the appropriate filter. The default is “allow-all”. That is, record all arguments. When ready, click Next.
If you selected the "allow-some" filter, the next dialog requires you specify which arguments should be recorded. Separate multiple arguments with an ampersand (&) symbol. When ready, click Next.
The new setting is applied after 10 minutes. Shortly after this time, the changes you have specified are visible in the Reporter interface.
Note:It is recommended that you make use of this facility if session or other random arguments are included in your page URLs. Otherwise, the content of page-based views (such as all pages or failed URLs) can become very large.
Within RUEI, session information is reported within the All sessions group. Here, information about a visitor session is available appropriately five minutes after the start of a session. By default, a visitor session is considered terminated if the visitor has been inactive for longer than the defined session idle time (by default, 60 minutes).
In order to optimize the reporting of sessions, the Session idle time advanced setting is available to specify the period (in minutes) of inactivity after which a visitor session is regarded as terminated. The default is 60 minutes.
Important:Because of the impact this setting can have on the performance of your installation, as well as the accuracy of the reported data, it is strongly recommended that you only change it under guidance from Customer Support.
In order to specify the idle time that should used when reporting sessions, do the following:
Select Configuration, then General, then Advanced settings, then Session processing, and then Session idle time. The dialog shown in Figure 12-10 appears.
Specify, in minutes, the period of visitor inactivity after which the session should be regarded as terminated. The default is 60 minutes. When ready, click Save.
Any change you make to this setting takes effect within five minutes.
By default, the order in which application, SSO profile, suite, and service filters are matched within RUEI is determined by the level of detail specified in the definition. That is, the definitions with the most information specified for them are applied first. However, sometimes you may want to modify the order in which filters are applied.
For example, you want to monitor network traffic for the domain "shop.oracle.com". You have defined two applications: one for the domain "shop*", and one for the domain "*oracle*. Because the string "*oracle*" is longer than the string "shop*", it is applied first. However, you want page identification for the "shop*" domain to take priority. You can use the rule ordering facility to override the default rule matching order, and specify the order in which pages for the required domains should be applied.
Note:It is recommended you use the default rule ordering, and that you define your applications, SSO profiles, suites, and services with sufficient information for them to be mutually exclusive.
To use the rule ordering facility, do the following:
Click the Configuration tab, select the Configuration menu option, and then the option Edit ruling orders. Note this option is only available to users with Full IT access permissions. The dialog shown in Figure 12-11 appears.
Use the Automatic rule ordering check box to specify whether the rule ordering is automatically derived from the currently defined applications, SSO profiles, suites, and services. As explained earlier, by default, the definitions with the most information specified for them are applied first. This is check box is automatically unchecked if you use the Up and Down controls to specify the order in which the rules should be applied. If you re-check it, the filter ordering is automatically reset to the default.
Note any changes you make are immediately put into effect. When ready, click Close.
Important:Be aware that if you modify the default rule ordering, and then define a new application, SSO profile, suite, or service, its associated filter is immediately placed at the bottom of the current rule ordering. Therefore, you should always review the rule ordering after the creation of new filters.
The availability of specific data within the Data Browser, as well as reports based on that data, depends on the amount of available disk space on the Collector and Reporter systems, as well as the amount of database space available on the Reporter system. This is illustrated in Figure 12-12.
Data gathered during monitoring is first written to log files, stored on the Collector system. These files are copied to, and processed by, the Reporter to populate the database that holds the multi-dimensional data structure viewable through the Data Browser and reports. These temporary log files are automatically removed from the Collector system after three days, and from the Reporter system (by default) after seven days. Note that data masking options (described in Section 13.5, "Masking User Information") can be specified to omit the logging of sensitive information.
The log files are used to created the Full session replay (FSR) data store. These files are regularly filtered to create the Error page replay (EPR) data store. The EPR files only contain information about failed events (that is, failed pages, objects, and function calls). Both the FSR and EPR data is held on the Collector system.
The size of the database user quota for the Reporter system is configurable during installation. By default, it is set to 200 GB. It is important to understand that data is consolidated when it is no longer required by the Reporter's defined retention policy. For example, by default, daily information about the last 32 days is retained. Daily information older than this is consolidated into the monthly information. Similarly, monthly information is consolidated into yearly information.
By default, RUEI keeps information on a daily, monthly, and yearly levels for 32 days, 13 months, and five years, respectively. Hence, for example, the oldest daily information will be dropped after 32 days. In addition, temporary log files are kept on the file system for approximately seven days. Be aware that a new RUEI installation will grow quickest during the first 32 days. After that time, the growth rate will slow. Of course, the growth rate depends on monitored traffic levels.
By default, information about failed URLs, pages, and service calls is kept for 15 days. If available, it can be viewed via the Session diagnostics replay facility (described in Section 4.1, "Introduction").
The settings described in the rest of this section allow you to optimize the disk and database utilization of your RUEI installation to meet your operational requirements.
To specify the data retention policies used by the Reporter system, do the following:
Select Configuration, then General, then Advanced settings, and then Reporter data retention policy. A screen similar to the one shown in Figure 12-13 appears.
As can be seen in Figure 12-13, every setting that has an impact on the database has a corresponding Database usage listing. This indicates the total database space (in gigabytes) currently used for the item, and the proportion this represents of the database's maximum permitted size. The projected database utilization (based on monitored traffic levels) is also indicated. Information about disk space utilization is available within the dialog boxes for individual settings.
Select the required setting. The settings shown in Table 12-1 are available.
Maximum database Size
Specifies (in gigabytes) the maximum amount of data allowed to be stored in the database. Note that you will need to specify the database SYSTEM user password to change this setting.
Failed event data retention
Specifies the period for which information about failed URLs, pages, and service calls is available. The default is for the last 15 days. If information is not available in the Session diagnostics replay, you may need to review this setting. Note this setting is linked to the Replay error page storage size setting (described in Section 13.7, "Defining Collector Data Retention Policies"). If you intend to increase the Failed event data retention setting, it is recommended you also increase the Error page replay storage size setting in order to facilitate this. Note also this setting has a high impact on disk space usage, and any change to it should be carefully considered in terms of anticipated network traffic.
Session diagnostics retention
Specifies the maximum number of days for which session diagnostics information is available. This facility is described in Section 4.1, "Introduction". The default is the last seven days, and the minimum is the last two days. This setting has an impact on database and disk space usage. The reported database usage is not included in the reported disk space usage.
Enriched data exchange retention
Specifies the maximum number of days for which information is available via the Enhanced data exchange facility. This facility is described in Appendix R, "Enriched Data Export Facility". The default is seven days. That is, data is available for the last six days, plus the current day. The maximum retention period depends on the available database storage capacity.
Be aware that this setting is applied on the full-day boundary (at approximately midnight). Therefore, if you decrease the number of days (for example, from seven to five), the setting change will take effect within 15 minutes, and the data for days five and six will be purged from the database.
Note that if set to one day, the previous day's data is deleted at approximately midnight, and only a limited amount of information is available for the current day. Therefore, in order to have access to the previous day's data after midnight, a retention period of at least two days must be specified.
KPI data exchange retention
Specifies the maximum number of days for which KPI information is available via the Enhanced KPI data exchange facility. This facility is described in Appendix R, "Enriched Data Export Facility". The default is 365 days. The maximum retention period depends on the available database storage capacity.
Daily data retention
Specifies the period for which daily information is available. The default is the last 32 days. The maximum period for which daily data may be kept depends on the monthly setting.
Monthly data retention
Specifies the period for which monthly information is available. The default is the last 13 months. The maximum period for which monthly data may be kept depends on the yearly setting.
Yearly data retention
Specifies the period for which yearly information is available. The default is the last five years. The minimum setting depends on the daily setting, while the minimum number depends on the monthly setting.
Maximum data group size
Specifies the maximum size to which data groups are permitted to grow. This setting is described in Section 12.9.3, "Setting the Maximum Size of the Failed Groups".
A dialog similar to the one shown in Figure 12-14 appears.
Use the dialog's control to specify the retention policy for the selected option.
For most settings, you can click Calculate to see the effect of your selection on database or disk space usage, as applicable.
When ready, click Save. Note that changes to disk space allocations take effect after approximately 10 minutes, while changes to database allocations only take effect after midnight.
Note:It is recommended that if you want to increase the amount of data kept, you start with the low-level data retention setting and work towards the high-level data retention setting. If you want to decrease the amount of data kept, start with the high-level data retention setting, and work towards the low-level data retention setting.
When specifying the high, medium, and low-level data retention settings, it is important to understand the dependency between stored days, months, and years. Use the following rules to calculate the required settings:
A month is assumed to have 30 days. The number of months that must be stored for a specified period of days is the number of days divided by 30 (rounded up to the next hole integer), plus one. For example, 33 days would require 33/30 (1.1 rounded up to 2), plus 1. Hence, three months.
The number of required years for a specified period of months is the number of months divided by 12 (rounded up to the next whole integer). For example, 11 months would require one year, while 13 months would require two years.
For example: 900 days, 31 months, and 3 years.
Note that configuration of Collector data retention policies is described in Section 13.7, "Defining Collector Data Retention Policies".
This section provides a detailed explanation of how the default maximum Data Browser group sizes can be increased to provide more accurate reporting of monitored traffic. It is strongly recommended that you carefully review the information presented before modifying the default settings.
In order to optimize performance, individual groups are not permitted to grow to unlimited sizes. Each main group table has a maximum permitted size. By default, this is 600 MB. The size of the Failed URLs, Failed services, Failed pages, and Slow URLs groups are controlled by a maximum number of rows that can be added to the group's main database table during a 5-minute period. By default, this is 5000 rows. This fully described in Section 12.9.3, "Setting the Maximum Size of the Failed Groups". There is no limit for session diagnostics information.
The maximum size of a group is managed by condensing. This is the process by which the number of rows within the group table are reduced by moving the least used data to a "other" group. For example, rows within the client-browser/version dimension might contain a small number of rows for rarely seen Mozilla Firefox versions. In this case, the number of rows within the table is reduced by replacing these with a single "other(firefox)" row.
You might consider increasing the maximum size of the groups if the information you require is regularly being condensed. For example, required individual page names or user IDs are only reported as "other". Broadly speaking, this approach is recommended in the case of high levels of network traffic.
If the maximum group size is set to more than 150% (that is, 900 MB), it is strongly recommended that the database is configured as a remote database. The maximum recommended group size is 300%Foot 1 (that is, 2.0 GB). Be aware that response times to Data Browser queries and dashboard updates may take longer to process if the maximum group size is increased.
It is important to note that custom dimensions have a significant impact on the condensing process. Incorrect configuration of custom dimensions can result in excessive condensing. For example, product names (or similar highly selective attributes) should not be configured as custom dimensions because that will skew the condensing process. Specifically, each page/product combination becomes a separate candidate for condensing, and only high frequency combinations will remain visible after condensing.
After increasing the maximum group size setting, it is strongly recommended that you carefully review the effect of the change on the performance of the Reporter system. You should wait at least a full day, and select System, then Status, and the Data processing. Review the system load and determine whether the Reporter system can handle the additional processing overhead that increasing the maximum group size setting represents.
Important:Increasing the data group sizes can have a significant impact on overall system performance. Therefore, any changes to data group sizes should be carefully planned, and performed in manageable steps.
As explained earlier, the Failed URLs, Failed services, and Failed pages groups do not use the maximum group size setting. Instead, their size is controlled through the
event_max_fail setting. This specifies the maximum number of rows that can added to the group's main database table during a 5-minute period. By default, this is 5000 rows. For the Slow URLs group, the
event_max_slow setting is used, and specifies the number of the slowest URLs that are recorded within each 5-minute period. By default, this is 5000 rows.
Note that if you change the
event_max_fail or the
event_max_slow setting, you should also review the
daily_max_fail setting. This specifies the maximum number of rows that the groups' tables can contain. This is derived from the formula 288 *
event_max_fail. The default, is 1.4 million rows.
To modify the above settings, issue the following commands:
$ sqlplus /@$RUEI_DB_TNSNAME SQL> update UXS_CONFIG set VALUE='10000' where NAME='event_max_fail'; SQL> update UXS_CONFIG set VALUE='4320000' where NAME='daily_max_fail';
Note that the
event_max_fail setting is limited to a maximum of 10,000 rows.
By default, information about the current (incomplete) period is always shown within selected periods that extend to the present time. In graphical visualizations, this is indicated with a dotted line. An example is shown in Figure 12-15.
To specify when incomplete periods should be reported, do the following:
Select Configuration, then General, then Advanced settings, then Data visualization, and then Current period reporting. The dialog shown in Figure 12-16 appears.
Select the visualization scheme to be used when reporting the current period. The options shown in Table 12-2 are available.
Specifies that all incomplete periods should be reported within all graphical visualizations, as well as value lists, reports, and exports. This is the default.
Specifies that no incomplete period should ever be reported.
Specifies that incomplete periods should only be reported within the specific visualizations. Note that the "Value list" option covers not only value lists within the Data Browser, but also reports and exports.
When ready, click Save. Any change you make to this setting takes effect immediately.
KPI and SLA values are reported to a certain level of precision. By default, this is two decimal places. However, you are free to modify this to reflect your reporting requirements. Do the following:
Select Configuration, then General, then Advanced settings, and then KPI and SLA precision reporting. Select the item whose reporting you want to modify. For example, SLA success. A dialog similar to the one shown in Figure 12-17 appears.
Specify the number of decimal places to which the selected item should be reported. When ready, click Save. Any change to these settings takes effect immediately.
As explained in Section 1.5, "Customizing Your Environment", users can customize the formatting settings used in their sessions. They can specify the characters used for the decimal point indicator and the thousand separator, and the date format that should be used. Administrators can also specify defaults for these settings on a system-wide basis by selecting System, then Maintenance, and then Formatting preferences.
Footnote LegendFootnote 1: Be aware that increasing the maximum group size will impact reporting performance. It is recommended that you incrementally increase the setting and verify that performance is acceptable.