1 Getting Started
What is RUEI?
The usage of web applications and services continues to grow. This includes not only the use of the Internet as a marketing channel, but also Extranet-based supply chain and back-office integration, and Intranet deployment of internal applications. It also includes the utilization of web services which implement clearly defined business functions. Applications can be accessed from mobile devices and there are many cloud based deployment options including on-premises, SaaS and hybrid. RUEI is designed for measuring, analyzing, and improving the availability and performance of all the deployment scenarios. To achieve this, RUEI is capable of performing data collection from network traffic, ADF servers, and data collection using Javascript browser instrumentation.
To view a visual demonstration about using RUEI, go to the following URL, and click Begin Video:
https://apex.oracle.com/pls/apex/f?p=44785:24:0::NO:24:P24_CONTENT_ID,P24_PREV_PAGE:5783,1
Data Collection
Figure 1-1 shows the Network Data Collector (available in previous RUEI releases) and Figure 1-2 shows the Tag Data Collector which is an option that allows you to collect data using Javascript and does not require network monitoring.
Table 1-1 outlines the different data collections that are available with RUEI.
Table 1-1 Data Collection Methods
Network | Tag | |
---|---|---|
Overview |
This option collects data that passes through the network and was the default option in previous releases and requires either a local or remote collector. It monitors all network traffic in promiscuous mode. |
This option, also called tag based monitoring, collects data by monitoring the request and processing of a specific web URL (the tag) which is inserted into all pages and requires either a local or remote collector. It monitors only the traffic related to a local IP address. |
Applications |
You must define an application. See Identifying and Reporting Web Pages chapter of User's Guide. |
You must define an onload object and use the generated javascript in your application. For more information, see Identifying and Reporting Web Pages in the Real User Experience Insight User Guide. |
Suites |
You must define a Suite. See Working With Suites and Web Services chapter of User's Guide. |
Only WebCenter Sites can be monitored using tag-based data collection. For more information, see Working With Suites and Web Services in Real User Experience Insight User Guide. |
Further Information |
Planning the Software Installation |
The options are further described in Planning the Software Installation.
The network data collection method is based on Network Protocol Analysis (NPA) technology. This method is 100% non-intrusive. Hence, it does not place any load on a web server, or require installing software agents that will impact performance. In addition, it does not require any change to the current application or infrastructure. When a new application release is deployed, or when an additional web server is added, there is no or very little change required to RUEI's monitoring environment. Typically, RUEI is installed before the web servers, behind a firewall in the DMZ (as shown in Figure 1-3).
Figure 1-3 How RUEI Collects Data with a Network Data Collector
When an object is requested by a visitor, RUEI sees the request and measures the time the web server requires to present the visitor with the requested object. At this point, RUEI knows who requested the page (the client IP), which object was requested, and from which server the object was requested (server IP).
When the web server responds and sends the requested object to the visitor. RUEI can see whether there is a response from the server, whether this response is correct, how much time the web server required to generate the requested object, and the size of the object. In addition, RUEI can also see whether the object was completely received by the visitor, or if the visitor aborted the download (that is, proof of delivery). RUEI can determine the time taken for the object to traverse the internet to the visitor and calculate the output between the visitor and the server (connection speed of the visitor).
Product Architecture
RUEI is based on a three-layer product architecture, as shown in Figure 1-4.
The monitored data packets are processed by the layers shown in Table 1-2.
Table 1-2 Product Architecture Layers
Layer | Description |
---|---|
Data Collection |
This layer is responsible for acquiring raw data and delivering it to the Data Processor layer. This data can be collected from multiple sources. The available attachment options are described later in this section. |
Data Processing |
This layer converts the raw data into the OLAP data sets. These comprise the multi-dimensional data structure that is viewable with the Data Browser. |
Data Presentation (Reporter) |
This layer is RUEI's analysis and reporting environment. This is a web-based information portal that can be accessed from any supported browser. |
Each of these layers can be deployed on the same system, or for scalability issues, on separate systems.
Security for Network Data Collection
To read HTTP(S) data streams, a proprietary software module reassembles TCP/IP packet streams. Because the network data collectors do not have an assigned IP number, and the software using these data collectors does not have a functional IP stack, RUEI is not able to respond to incoming traffic received on the data collectors. This makes RUEI "invisible" to the monitored networks, and completely secure.
Note:
Because of the non-intrusive way in which RUEI collects data, it is not possible for it to request retransmission in the event of an error on the measurement port.
Data collection can be configured to log encrypted data. To facilitate this, a copy of the web server's private SSL keys needs to be set up in the data collector. In addition, RUEI can be configured to omit logging of sensitive data in the arguments of POST requests of forms or content; so called data masking (or blinding).
Connection Options for Network Data Collection
RUEI supports the use of both copy portsFoot 1 and TAPsFoot 2 for monitoring network traffic (10/100 Mbps and 1/10 Gbps Ethernet connections are supported). Copy ports and TAPs are available for copper or fibre-based network infrastructures. While both devices allow non-intrusive monitoring of network traffic, there are differences between these two connection on options.
Monitoring SSL and Forms Traffic
Note:
SSL and Oracle Forms traffic are particularly sensitive to disruptions in the TCP packet stream. This is because they require state information to be maintained for the duration of the connection, and any lost packets can cause that information to be lost, preventing RUEI from accurately monitoring and reporting the connection.
Therefore, you should ensure that each Collector is connected to a reliable network device, such as a TAP. In addition, it is recommended that you regular review the information available through the Collector Statistics window (select System>Status>Collector Statistics for each collector node) to verify the integrity of the TCP packet stream. Keep a check on the reported TCP and SSL connection errors. Also, the Collector software needs direct access to the physical network interface and that a configuration where multiple servers share a single physical network interface, for example certain blade server types, may not work reliably. Contact your hardware vendor for any queries related to your configuration.
Copy Ports
Copy Port is a switch that builds the Layer 2 forwarding table based on the source MAC address that the switch receives. Once the forwarding table is built, the switch forwards the traffic that is destined for a MAC address directly to the corresponding port.
For example, after the web server MAC in Figure 1-5 is learned, unicast traffic from the browser to the web server is only forwarded to the web server port. Therefore, the Collector does not see this traffic.
Figure 1-5 Network Connection Using a Copy Port
![Description of Figure 1-5 follows Description of Figure 1-5 follows](img/span.png)
Description of "Figure 1-5 Network Connection Using a Copy Port"
In the configuration shown in the lower part of Figure 1-5, the Collector is attached to a port that is configured to receive a copy of every packet that the browser sends and receives. This port is called a copy port. Copy ports can copy traffic from any or all data ports to a single unused port and prevent bi-directional traffic on the port to protect against backflow or traffic into the network.
Activating a copy port on a switch can have a performance impact. Typically, copy ports support a wide range of configuration options. For more information, about these options, consult your switch documentation or contact the vendor.
TAPs
TAPs can be placed between any two network devices (such as routers and firewalls). Any monitoring device connected to a TAP receives the same traffic as if it were in-line, including all errors. This is achieved through the TAP duplicating all traffic on the link, and forwarding it to the monitoring port(s). The example shown in Figure 1-6 illustrates a typical TAP deployment for one Collector.
Important
Unlike copy ports, in the event of power failure, TAPs continue to allow data to flow between network devices. During heavy loads, copy ports are prone to packet loss. TAP devices are available for copper or fibre-based infrastructures. Moreover, they can be easily deployed when and where required, but without reconfiguration of switches or engineers needing to re-cable a network link. For these reasons, the use of TAPs is strongly recommended over that of copy ports.
Broadly speaking, there are three types of TAPs: network, regeneration, and aggregation TAPs. RUEI supports the use of network and regeneration TAPs. Aggregation taps are only supported if they maintain the ordering of packets in the packet stream. Reporting accuracy can be impacted when using aggregation taps if the monitor port gets saturated, resulting in packet loss and inaccurate timing information. When capturing data with a network TAP, the use of cascaded TAP configurations is not supported.
It is possible in RUEI to monitor and process data from multiple networks, by either deploying a tap on each network segment and connecting those to a central collector, or by deploying multiple collectors, one on each monitored segment. For more information, see Scaling Scenarios.
Installation Options
A RUEI system can be installed either through a Reporter or a Collector option. Each of these installation options is reviewed in the following sections.
Reporter
A Reporter system processes the data gathered by the Collectors attached to it. After processing, this data is stored in an Oracle database, referred to as the Reporter database. System users can review the collected data through a browser-based interface.
For RUEI to be able to accurately monitor network traffic, and report its results, it needs certain information about your network and application infrastructure. This includes how pages, service function calls, and end users will be identified, the scope of monitoring in your network environment, the monitoring of specific KPIs and SLAs, and the roles and permissions assigned to system users. This information is held in a separate Configuration database.
Collector
A Collector gathers data and submits the data to a Reporter. Multiple Collectors can be attached to the same Reporter. A direct connection is required between the Collector systems and the Reporter system. A collector can be either network based or tag based as described in Planning the Software Installation.
Each Reporter installation also contains a local Collector instance. The Reporter can be configured to just process information gathered by this local Collector (this is a single-server configuration), or to receive information from additional Collectors. The local Collector instance on the Reporter system can also be disabled if not required.
Local and Remote Database Installations
The data available through the Reporter system is stored in an Oracle database, called the Reporter database. The information required by RUEI in order to correctly monitor and report on your web infrastructure, such as information about monitored applications and system users, is held in a separate Configuration part of the database. The database can reside locally on the Reporter system, or on a remote database server (such as a database cluster).
The use of a remote database server provides a number of potential advantages over a locally installed database. In particular, it offers easier integration with existing security and back-up policies, as well as improved performance through the use of dedicated servers.
For supported database versions please see the certification list in My Oracle Support.
Scaling Scenarios
This section highlights the different deployment scenarios available to you. The selection of the most appropriate deployment scenario is primarily determined by the level of monitored network traffic, your reporting requirements, and the hardware specifications of your deployment systems.
Single-Server Deployment
This is the simplest deployment, and is suitable for monitoring web environments with low to medium levels of traffic. An example is shown in Figure 1-7.
In this deployment, a single system serves as both Collector and Reporter. As explained in the previous section, the Reporter database can reside locally on the Reporter system or on a remote database server.
Multiple-Server Deployment
The use of multiple servers may be considered when there is a need to monitor very high level of traffic. In addition, this deployment also provides the possibility of enhanced security. For example, by placing the Collector(s) outside the office network, while placing the Reporter system within the network. Figure 1-8 shows an example of a multiple-Collector deployment.
This features a deployment in which both data lines are monitored in the same reporting environment. This deployment assumes that the traffic on each line is mutually exclusive. It also illustrates a deployment used for security reasons. While the traffic from web servers A and B are monitored and reported, the traffic from web server C is not. This is also the reason why the Collectors are not placed above the switch. Collector instance on the Reporter system (system 1) is disabled.
For security reasons, it is recommended that access to the Reporter system is restricted to trusted IP ranges. Similarly, you may want to locate the Reporter system inside the internal network to maximize its security. The Collector's data gathering ports should be within the DMZ.
The application and infrastructure configuration information held in the database is maintained by the Reporter based on information provided by system users through its browser-based interface. Each Collector uses this information to determine how the data it gathers should be reported.
Server Requirements
The required minimum system specifications for the selected configuration (as explained in Installation Options) are described in the following sections.
Network Cards
It is recommended that you carefully consider the selection of network cards for your infrastructure. Depending on the connection option you selected in Connection Options for Network Data Collection, both copper and fibre-based network cards may be required. If necessary, consult both your network and systems management teams.
Network Cards Within Bonded Groups
Monitoring of network traffic using network cards that are part of a bonded group is not supported.
Note:
For more information about required and recommended system specifications, please contact Customer Support.
Single-Server Requirements
Table 1-3 Single-Server System Minimum Requirements
Element | Requirements |
---|---|
CPU |
64-bit Intel or AMD dual-CPU, dual-core processor (> 2 G Hz) or equivalent. |
Memory |
16 GB. |
Disk space |
|
Network interfaces |
When using a network-TAP deviceFoot 6, a minimum of three network interfaces are required:
|
GSM modem (optional) |
Optional support for a GSM modem to send text messages. The modem needs to be either GSM07.05 or GSM07.07 compatible. It can be connected through a serial or USB port. If USB is used, RUEI uses the first available port ( |
Footnote 3
To ensure acceptable performance of the RUEI installation, it is recommended to use high performance disk systems, with a minimum supported I/O rate of 70 MB/s. When monitoring high volumes of traffic, more powerful disk systems may be required. (Hardware) RAID-10 or equivalent storage configurations are strongly recommended.
Footnote 4
This may need to be increased if Enriched data exchange is enabled.
Footnote 5
The use of an NFS share for local data (that is, $RUEI_DATA
and $RUEI_HOME
) is not supported. This restriction does not apply to $RUEI_DATA
/processor/data
and $RUEI_DATA
/collector/wg/REPLAY
.
Footnote 6
When capturing data with a network-TAP device, the use of cascaded TAP configurations is not supported.
Reporter Requirements
Table 1-4 Reporter System Minimum Requirements
Element | Requirements |
---|---|
CPU |
64-bit Intel or AMD dual-CPU, dual-core processor (> 2 G Hz) or equivalent. |
Memory |
16 GB. |
Disk space |
|
Network interfaces |
A minimum of one network interface is required. |
GSM modem (optional) |
Optional support for a GSM modem to send text messages. The modem needs to be either GSM07.05 or GSM07.07 compatible. It can be connected through a serial or USB port. If USB is used, RUEI uses the first available port (ttyUSB0). Alternative methods of sending text messages are available (http/e-mail). |
Footnote 7
To ensure acceptable performance of the RUEI installation, it is recommended to use high performance disk systems, with a minimum supported I/O rate of 70 MB/s. When monitoring high volumes of traffic, more powerful disk systems may be required. (Hardware) RAID-10 or equivalent storage configurations are strongly recommended.
Footnote 8
This may need to be increased if Enriched data exchange is enabled.
Footnote 9
The use of an NFS share for local data (that is, $RUEI_DATA
and $RUEI_HOME
) is not supported. This restriction does not apply to $RUEI_DATA
/processor/data
and $RUEI_DATA
/collector/wg/REPLAY
.
Collector Requirements
The requirements for Collector systems are shown in Table 1-5.
Table 1-5 Collector System Minimum Requirements
Element | Requirement |
---|---|
CPU |
64-bit Intel or AMD dual-core processor or equivalent. |
Memory |
16 GB. |
Disk space |
Minimum 205 GB HDD free spaceFoot 10. |
Network interfaces |
When using a network-TAPFoot 11 device, a minimum of three network interfaces are required:
When using a network-copy port, a minimum of two network interfaces are required:
|
Footnote 10
The use of an NFS share for local data (that is, $RUEI_DATA
and $RUEI_HOME
) is not supported. This restriction does not apply to $RUEI_DATA
/processor/data
and $RUEI_DATA
/collector/wg/REPLAY
.
Footnote 11
Capturing data with a network-TAP device prevents the use of a cascaded TAPs configuration.
Footnote 12
For up and down stream traffic. The use of TAPs that integrate up and down stream traffic on one line (that is, link aggregation TAPs) is not recommended.
Deployment Best Practices
This section presents a best practices framework within which to optimize your RUEI deployment. It is recommended that you carefully review the following information.
Planning Your Deployment
It is important that the nature of the monitored network environment is clearly understood before deciding upon your RUEI deployment strategy. This includes not only the basic network connectivity, ports, addressing, and physical device requirements, but also a sound understanding of the monitored applications.
Moreover, before deploying RUEI, the basic traffic flows within the network must have been identified. This should include information about average and peak volumes of the traffic. Any physical deployment requirements (such as space limitations, distances, power planning, rack space and layout, or cabling) should be identified.
You can use the checklist presented in Installation Checklist to capture the information.
Forms-Based Traffic
If you are planning to monitor Forms-based traffic, the memory requirements may be higher than those outlined in Server Requirements. In this case, you should consider a split-server deployment.
Full Session Replay
If you are planning to make use of the Full Session Replay (FSR) facility, you need to configure additional storage capacity. This is explained in Full Session Replay Storage Requirements.
Encrypted Traffic
If a significant level of the monitored traffic is encrypted, it can increase the CPU overhead. In this case, it is recommended that you consider configuring additional CPUs or, alternatively, a split-server deployment.
Very High Levels of Traffic
When very high levels of traffic are being monitored (that is, more than 10 million page views per day), it is strongly recommended that you consider a split-server deployment. Alternatively, consider the use of a remote database server. The latter has the effect of significantly reducing (by up to 30%) the CPU overhead on the Reporter system. Monitored environments with more than 20 million page views per day should consider the use of both a split-server deployment and a remote database server.
Data Retention Policies
The availability of specific data within the Data Browser, as well as reports based on that data, depends on the amount of available disk space on the Collector and Reporter systems, as well as the amount of database space available on the Reporter system.
Data gathered during monitoring is first written to log files, stored on the Collector system. These files are copied to and processed by, the Reporter to populate the database that holds the multi-dimensional data structure viewable through the Data Browser and reports. These temporary log files are automatically removed from the Collector system after three days, and from the Reporter system (by default) after seven days.
The size of the database user quota for the Reporter system is configurable during installation. By default, it is set to 500 GB. It is important to understand that data is consolidated when it is no longer required by the Reporter's defined retention policy. For example, by default, daily information about the last 32 days is retained. Daily information older than this is consolidated into the monthly information. Similarly, monthly information is consolidated into yearly information.
RUEI maintains data at several aggregation levels, whose retention is configured in days. The following describes the various aggregation levels and their default values:
-
Instance: 8 days
-
5-minute: 15 days
-
Hourly: 32 days
-
Daily: 90 days
-
Monthly: 60 months
These numbers can be fine-tuned per category of data (app, suite, service, SLA) and beyond that per individual type (for example, All Pages or Failed Pages). The default value for enriched data exchange is 8 days for each type.
DB space is about 5 GB per period each. This is heavily dependent on load and diversity of traffic. You should occasionally check the reporter data retention screen, especially in the first month, to verify enough disk space is available.
Statistics data is configurable from the CLI. However, statistics retention is not configurable, while User flow completion and fusion product retention are only configurable from the command line.
Minimum and maximum values for data retention settings are automatically determined. Less-detailed aggregation levels must always have at least as much retention as more-detailed aggregation levels.
Be aware that a new RUEI installation will grow quickest during the first 32 days. The growth rate will slow down. The growth rate depends on monitored traffic levels.
Full Session Replay Storage Requirements
If you are planning to make use of the Full Session Replay (FSR) facility, you may need to configure additional storage capacity available to the Collector system. This should be a separate device (not a partition of the Collector server's existing hard drive), and made accessible to the RUEI file system. The procedure and the guidance on storage requirements, is described in the rest of this section. This procedure must be repeated for each Collector for which full session replay information is required.
Configuring Additional Storage for Full Session Replay
The procedure described below assumes that you have a fully operational system, and that FSR has been enabled. To configure the additional required storage, do the following:
-
Mount the device. For example, under
/mnt/external_storage
. -
Temporarily stop the Collector by running the following command:
appsensor stop wg
-
Move the
$APPSENSOR_HOME/wg/REPLAY
directory to the new device. In the above example, this is/mnt/external_storage
, and the result is that the replay files are now located in the/mnt/external_storage/REPLAY
directory. -
Create a symbolic link from
/mnt/external_storage/REPLAY
to$APPSENSOR_HOME/wg/REPLAY
. -
Restart the Collector by running the following command:
appsensor start wg
-
Calculate the required storage capacity. To do so, multiply the average number of daily page views by the average page size. Then, multiple this number by the number of days you wish full session replay data to be retained. Use Table 1-6 as guidance.
Table 1-6 Full Session Replay Storage Estimates
Low page weight (~10 Kb) Medium page weight (~50 Kb) High page weight (~100 Kb) Page views per day (millions)
Size per day (GB)
Disk I/O (MB/sec)
Size per day (GB)
Disk I/O (MB/sec)
Size per day (GB)
Disk I/O (MB/sec)
0.5
5
0.1
25
0.3
50
0.6
2
20
0.2
100
1.2
200
2.3
5
50
0.6
250
2.9
500
5.8
10
100
1.2
500
5.8
1000
11.6
20
200
2.3
1000
11.6
2000
23.1
50
500
5.8
2500
28.9
5000
57.9
Note:
Table 1-6 is intended for guidance only. It is strongly recommended that you regularly review average page sizes and daily page views, and adjust the required storage size as necessary.
Note:
Be aware that FSR functionality uses a significant number of non-sequential read operations. Please consult your hardware vendor for information on how to optimize your I/O performance.
-
Select Configuration>General>Advanced settings, and then Collector data retention policy. Click the Full session replay storage size (GB) setting. Specify (in gigabytes) the required storage size. The maximum that can be specified is 100 TB. When ready, click Save.
Memory Requirements
When calculating the amount of RAM required by your RUEI installation, it is recommended that you consider the following:
For each Collector system, each 100 concurrent hits require 2 MB, and each 1000 SSL connections require 1 MB. In addition, up to 600 MBps of network traffic can be buffered before individual TCP sessions start to be dropped. Up to 600 MBps should also be assumed for content checks (such as XPath queries and error strings). If you define a large number of content checks, or specify that they contain NLS character sets, the memory required may increase.
Software Requirements
The following GNU/Linux distributions are supported:
Oracle Linux
-
Oracle Linux 6, starting with version 6.5, 64-bit, both Intel and AMD compatible.
-
Oracle Linux 7, 64-bit, both Intel and AMD compatible.
-
Oracle Linux 8, 64-bit, both Intel and AMD compatible.
RedHat Enterprise Linux
-
RedHat Enterprise Linux 6, starting with version 6.5, 64-bit, both Intel and AMD compatible.
-
RedHat Enterprise Linux 7, 64-bit, both Intel and AMD compatible.
-
RedHat Enterprise Linux 8, 64-bit, both Intel and AMD compatible.
For supported database versions, please see the certification list in My Oracle Support.
Encrypting Sensitive Data
If sensitive data needs to be encrypted, you have the opportunity to encrypt your entire disk configuration during the disk partitioning phase of the Linux installation procedure.
Network Requirements
-
All server system clocks should be synchronized through NTP using UDP port 123.
-
Support DNS information requests over TCP and UDP port 53.
-
Support reports and e-mail alerts using TCP port 25.
-
Support SNMP traps on request from an SNMP Manager using UDP port 161/162.
-
The RUEI user interface is accessible over HTTPS port 443.
-
In the case of a remote database setup, access to TCP port 1521 is required between the Reporter and remote database server.
-
Each remote Collector system should be accessible by the Reporter system over TCP port 22. It is recommended all other ports be blocked.
-
If you are configuring a failover Reporter system (described in Configuring a Failover Reporter System), the primary and secondary Reporter systems need to be able to contact each other using ICMP.
-
If you are configuring a failover Collector system (described in Configuring a Failover Collector System), the primary and secondary Collector systems need to be able to contact each other using ICMP.
Collector-Reporter Bandwidths
The amount of data transferred between a remote Collector and the Reporter system largely depends on the type and level of network application traffic monitored by RUEI. In addition, the configuration of RUEI (such as defined functional errors, content checks, and page naming schemes) also influences the size of Collector files that need to be transferred to the Reporter system.
At peak times, the amount of data that needs to be transferred will be higher than during low traffic periods. The exact amount of the data transmission from a remote Collector to the Reporter system can only be determined after the actual RUEI deployment.
For an initial deployment, the following simple rule can be used: each 5 million daily page views will result in a peak transfer of approximately 125 MB at peak time, and approximately 1 GB per day. Hence, typically only a few percent of the actual monitored traffic will be stored by a Collector and transferred to the Reporter. When you want or need to minimize this data transfer, it is recommended that you minimize the amount of monitored HTTP traffic which is not required by RUEI. For example, by using a subnet or VLAN-filtered network.
Client Requirements
The workstations that will access the RUEI user interface must have one of the following browsers installed:
-
Mozilla Firefox 75 (and above)
-
Internet Explorer 11
-
Microsoft Edge
-
Safari 12 and 13.
-
Google Chrome 81 (and above).
JavaScript must be enabled. No other browser plug-ins are required.
In addition, the workstation should have a screen resolution of 1024 * 768 (or higher).
Note:
Ensure that any pop-up blocker within the browser has been disabled.
AJAX Support
RUEI uses AJAX to enhance its user interaction. Internet Explorer relies on the MSXML control to facilitate AJAX. The AJAX dependencies can trigger a security warning when using strict security settings.
Footnote Legend
Footnote 1:Copy ports are also known as Switched Port Analyzer (SPAN) ports which is a feature of Cisco switches.
Footnote 2:
Test Access Port (TAP) devices are provided by specialist vendors, such as NetOptics Inc.