This chapter describes the tuning measures you can take to enhance the overall performance of the Oracle Communications Network Integrity system.
Tuning Network Integrity improves the performance and throughput of the application and maximizes response time for the chosen hardware platform.
Network Integrity tuning tasks include:
This section provides recommended tuning adjustments for each operating system.
Edit the /etc/system file and set the following parameters:
set rlim_fd_cur=65535 set rlim_fd_max=65535 set shmsys:shminfo_shmmax=4294967295 set autoup=1 set tune_t_fsflushr=1
Edit the /lib/svc/method/net-init file and set the following parameters using the following commands:
ndd -set /dev/tcp tcp_conn_req_max_q 16384 ndd -set /dev/tcp tcp_conn_req_max_q0 16384 ndd -set /dev/tcp tcp_time_wait_interval 60000 ndd -set /dev/tcp tcp_xmit_hiwait 131072 ndd -set /dev/tcp tcp_recv_hiwait 131072 ndd -set /dev/tcp tcp_keepalive_interval 7200000 ndd -set /dev/tcp tcp_xmit_hiwaiter_def 1048576 ndd -set /dev/tcp tcp_ip_abort_interval 60000 ndd -set /dev/tcp tcp_rexmit_interval_initial 4000 ndd -set /dev/tcp tcp_rexmit_interval_min 3000 ndd -set /dev/tcp tcp_rexmit_interval_max 10000 ndd -set /dev/tcp tcp_smallest_anon_port 32768 ndd -set /dev/tcp tcp_naglim_def 1
Tune Solaris 10 for M series by editing the /etc/system file and set the following parameters:
set ddi_msix_alloc_limit=8 set ip:ip_soft_rings_cnt=16 set ip_squeue_soft_ring=1 set ip_threads_per_cpu=2
Tune Solaris 10 for CMT (T5440/T5240/T5140/T5220/T5120/T2000/T100), edit the /etc/system file and set the following parameters:
set ip:ip_soft_rings_cnt=16
Caution:
Reboot the server after making these changes.See the Solaris platform documentation for more information about tuning.
You can modify the following parameters for a Linux environment:
kernel.msgmni
The default message queue is 16 which supports 16 users.
Update the file /etc/sysctl.conf with the following entry:
kernel.msgmni=1024
Note:
mtu is the largest number of bytes that a packet can carry over a network. For better performance, set the parameter as:/sbin/ifconfig lo mtu 1500
net.ipv4.tcp_max_syn_backlog
This specifies the maximum number of remembered connection requests that have not received an acknowledgment from the connecting client.
Update the file /etc/sysctl.conf with the following entry:
net.ipv4.tcp_max_syn_backlog=8192
/etc/security/limits.conf
Update this file with the following parameters:
soft nofile 131072
hard nofile 131072
soft nproc 131072
hard nproc 131072
soft core unlimited
hard core unlimited
soft memlock 50000000
hard memlock 500000
Adjust the following parameters for an AIX environment.
Edit the .profile file, located in the directory where AIX is installed, to add the following entries:
AIXTHREAD_COND_DEBUG=OFF export AIXTHREAD_COND_DEBUG AIXTHREAD_MUTEX_DEBUG=OFF export AIXTHREAD_MUTEX_DEBUG AIXTHREAD_RWLOCK_DEBUG=OFF export AIXTHREAD_RWLOCK_DEBUG AIXTHREAD_SCOPE=S export AIXTHREAD_SCOPE LC__FASTMSG=true export LC__FASTMSG LDR_CNTRL=MAXDATA=0XB00000000@DSA export LDR_CNTRL
This section describes the Network Integrity database tuning parameters.
Note:
These parameters also apply to the Unified Inventory Management (UIM) system.To set the database initialization parameters, execute the following script as the SYS user:
alter system set db_file_multiblock_read_count=16 scope=spfile; alter system set distributed_lock_timeout=7200 scope=spfile; alter system set dml_locks=9700 scope=spfile; alter system set job_queue_processes=10 scope=spfile; alter system set log_buffer=31457280 scope=spfile; alter system set open_cursors=5000 scope=spfile; alter system set parallel_max_servers=640 scope=spfile; alter system set plsql_code_type=NATIVE scope=spfile; alter system set query_rewrite_integrity=STALE_TOLERATED scope=spfile; alter system set processes=3000 scope=spfile; alter system set sessions=4528 scope=spfile; alter system set transactions=4980 scope=spfile; shutdown immediate startup
Note:
The memory_target and max_memory_target parameters (which determine how much of RAM is allocated to the database SGA and PGA) should be set to as high a value as your database server platform allows.Oracle recommends a value of 8GB or higher.
Note:
This section also applies to the Unified Inventory Management (UIM) system.You can gather schema statistics to generate table and index statistics that the database engine query optimizer can use to select the best method for executing different SQL statements.
To gather schema statistics, execute the following command as the SYS user:
execute dbms_stats.gather_schema_stats(ownname => 'USERID',estimate_percent => 25, method_opt => 'for all indexed columns size auto', cascade => true);
Where USERID is the name of the primary schema for the application selected at the time of installation.
Note:
Oracle recommends that you run this command periodically (weekly, or monthly) as the database gets populated continuously.During Network Integrity installation, all indexes are created in the same tablespace as data records. It is a good idea for the index data to be on disks separate from the table data. If your disks are set up this way (that is, you are not using RAID), then you may want to create a tablespace just for indexes (for example NINDEX) and then relocate the existing indexes to this tablespace.
To create a tablespace just for indexes and then relocate the existing indexes to this tablespace:
Use a script similar to the one provided below to create a second script.
set lin 90 spool move_index.sql select 'alter index '||owner||'.'||index_name||' rebuild tablespace nindex;' from dba_indexes where owner='USERID' and not (index_name like 'SYS%') order by owner,index_name; spool off
The script creates the new script with the name move_index.sql.
Edit this new move_index.sql script to remove lines that are not alter index commands.
Execute the move_index.sql script as the SYS user.
You can improve the performance of frequently used Network Integrity specific SQL statements by creating new indexes.
Note:
The following script shows how to create four new indexes.You must create these four indexes to improve performance.
To create new indexes, execute the following script as the SYS user.
create index idx_logicaldevice_01 on logicaldevice (nativeemsname,entityclass) tablespace nindex compute statistics; create index idx_physicaldevice_01 on physicaldevice (nativeemsname,entityclass) tablespace nindex compute statistics; create index idx_disrootentityref_01 on disrootentityref (rootentityref,rootentityclass) tablespace nindex compute statistics; create index idx_specification_01 on specification (name) tablespace nindex compute statistics;
Frequently monitoring your database is important to ensure that none of your disks or tablespaces are run out of required space. Make sure you know where the audit and trace files are located so you can delete them before they fill the disk.
For more information on Oracle database management and monitoring activities, see Oracle Database Administrator's Guide.
Note:
Oracle recommends that you use the Oracle Enterprise Manager for performing most of your DBA activities, including managing disk space.Note:
Using RAID disks and faster disks for your Network Integrity database server gives you better performance and higher reliability.Keep the following in mind when selecting a JVM for running the application servers for Network Integrity:
Use the largest possible heap size that can be allocated based on the amount of RAM on your server.
The following environment variable names are used for the Network Integrity JVM in the rest of this section:
AS_USER_MEM_ARGS (JVM startup parameters for the Administration Server)
MS_USER_MEM_ARGS (JVM startup parameters for the Managed Server)
In the variables provided in this section, AS_ and MS_ represent Administration server and Managed server, respectively.
For the Administration server use the following variable:
export AS_USER_MEM_ARGS="JVM_Startup_Parameters" export USER_MEM_ARGS=$AS_USER_MEM_ARGS
For a Managed server use the following variable:
export MS_USER_MEM_ARGS="JVM_Startup_Parameters" export USER_MEM_ARGS=$MS_USER_MEM_ARGS
where "JVM_Startup_Parameters" are the JVM startup settings for your OS. For example, for 64-bit Sun Hotspot JVM Administration server on Solaris:
export MS_USER_MEM_ARGS="-d64 -Xms1g -Xmx20g -XX:PermSize=512m -XX:MaxPermSize=1012m -XX:GCTimeRatio=1 -XX:MaxGCPauseMillis=30000 -XX:+UseCompressedOops" export USER_MEM_ARGS=$MS_USER_MEM_ARGS
You can set these variables in the following two ways:
In one command session, export AS_USER_MEM_ARGS and launch the Administration server. Then, in a second command session, export
MS_USER_MEM_ARGS and launch the Managed servers.
Write your own startup scripts for the Administration and Managed servers.
This section provides the parameters for setting the JVM startup for Solaris.
64-bit Sun Hotspot JVM (assuming the maximum amount of RAM you can allocate is 21GB)
AS_USER_MEM_ARGS = "-d64 -Xms1g -Xmx1g -XX:PermSize=384m -XX:MaxPermSize=484m"
MS_USER_MEM_ARGS = "-d64 -Xms1g -Xmx20g -XX:PermSize=512m -XX:MaxPermSize=1012m -XX:GCTimeRatio=1 -XX:MaxGCPauseMillis=30000 -XX:+UseCompressedOops"
This section provides the parameters for setting the JVM startup for Linux.
64-bit Sun Hotspot JVM (assuming the maximum amount of RAM you can allocate is 21GB)
AS_USER_MEM_ARGS = "-d64 -Xms1g -Xmx1g -XX:PermSize=384m -XX:MaxPermSize=484m"
MS_USER_MEM_ARGS = "-d64 -Xms1g -Xmx20g -XX:PermSize=512m -XX:MaxPermSize=1012m -XX:GCTimeRatio=1 -XX:MaxGCPauseMillis=30000 -XX:+UseCompressedOops"
This section lists the recommended changes for tuning the WebLogic application server for Network Integrity.
To tune the WebLogic Administration Server for Network Integrity:
Log in to the Oracle WebLogic Server Administration Console using the Administrator credentials.
The Home screen of the Administration Console appears.
Under JDBC, under the main heading Services, select Data Sources.
The Summary of JDBC Data Sources screen appears.
In the Data Sources table, click JobDispatcherDS.
The Settings for JobDispatcherDS screen appears.
Select and display the Connection Pool tab.
Do the following:
In the Initial Capacity field, enter the value as 50.
In the Maximum Capacity field, enter the value as 100.
In the Capacity Increment field, enter the value as 2.
Come back to the Summary of JDBC Data Sources screen.
In the Data Sources table, click NIDataSource.
The Settings for NIDatasource screen appears.
Select the Transaction tab.
Configure the following parameters:
Select the XA Transaction Timeout check box.
In the XA Transaction Timeout field, enter 0.
Select the Connection Pool tab.
Do the following:
In the Initial Capacity field, enter the value as 50.
In the Maximum Capacity field, enter the value as 100.
In the Capacity Increment field, enter the value as 2.
Go back to Home.
Select Work Managers under Environment.
The Summary of Work Managers screen appears.
In the Global Work Managers, Request Classes and Constraints table, click NI-MaxThreadConstraint.
The Settings for NI-MaxThreadConstraint screen appears.
In the Count field, enter an appropriate value.
Keep the following in mind regarding the NI-MaxThreadConstraint parameter:
This is a critical tuning value. It directly affects the throughput of critical Network Integrity batch operations such as network scans and discrepancy detection. A value approximately equal to (max_heap_size-300mb)/160 is generally used initially. However, you can experiment with this throttle to find the perfect balance between a lower value (for example, 10) that reduces the chance of flooding the system with too many simultaneous tasks (which could result in a JVM out of memory error) and a higher value (for example, 70) that improves the throughput of batch operations. The optimum value depends on a variety of factors such as JVM type, heap size, RAM available, CPU number and speed, disk speed, database performance, and so on.
This is also a very important setting for the WebLogic Administration Server as the server might become unstable under out-of-memory conditions. If the NI-MaxThreadConstraint parameter is set too high, the WebLogic Administration Server may run out of memory trying to process too many requests at one time. You must tune for the worst case; the maximum parallelism of the largest network elements. The WebLogic Administration Server does offer some safeguards to prevent continued operation under overload conditions. You can configure the managed servers to automatically shut down a server under overload conditions. You do this by setting a Panic Action or a Failure Action on the Overload tab for the managed server(s) in the WebLogic Administration Server Console.
For more information, see the WebLogic Administration Server administration guide at the following location:
This parameter also affects interaction with external systems. For example, tune the UIM system to support up to NI-MaxThreadConstraint concurrent web service operations, to support uploading large quantities of resolved discrepancies to UIM. Configure the UIM InventoryTxDataSource with a maximum number of connections greater than the NI-MaxThreadConstraint value.
You must restart the managed server(s) after changing this value.
Note:
For information on tuning the WebLogic Administration Server for Oracle Communications Unified Inventory Management (UIM), see the related UIM documentation.This section provides recommendations on maximizing Network Integrity performance.
To support a large network of many thousands of network elements, performing tasks such as inventory imports, network scans, and discrepancy detection in parallel, maximizes performance.
You get parallelism for network scans and discrepancy detection automatically. The degree of parallelism is determined by the NI-MaxThreadConstraint value. For example, let's say you have a network scan (with discrepancy detection enabled) of 200 devices and NI-MaxThreadConstraint set to 50. When the scan starts, the first 50 devices are scanned. As each device completes, another device begins so that you always have 50 devices being scanned at the same time. When all 200 devices are scanned, then the discrepancy detection begins in the same way, the first 50 begin and completed ones are replaced with new ones to be scanned.
You do not get parallelism automatically for inventory imports though. For example, an inventory import scan of 200 devices imports those 200 devices in serial (regardless of the NI-MaxThreadConstraint value).
The best approach is to divide the 200 devices into N batches (where N is the number of CPU cores on your Network Integrity application server) and schedule them to all run at the same time. For example, if you have 8 CPU cores, you should create 8 inventory import scans, manually, of 25 devices each. This reduces the total time to import those devices in 8 parallel scans to approximately 20% of what it would take to import those devices serially in 1 large scan.
Note:
This approach is critical to maximizing the reconciliation throughput of the Network Integrity system and thereby supporting as large a network as possible.You can reduce chances of application server memory related problems due to too many concurrent work requests being generated by the Network Integrity UI and web services users by configuring the Shared Capacity for Work Managers parameter to have a much lower value.
To lower the value of the Shared Capacity for Work Managers parameter:
Log on to the Oracle WebLogic Administration Server.
The Home screen of the Administration Console appears.
Under Environment, select Servers.
The Summary of Servers screen appears.
In the Servers table, click the managed server, ManagedServerName, for which you want to configure the parameter.
The Settings for ManagedServerName screen appears.
Select the Overload tab to display it.
In the Shared Capacity For Work Managers field, enter a lower value.
Note:
The exact value depends on the capacity of your platform and the amount of scans and user requests being initiated.The following three timeout parameter settings affect the operation of Network Integrity jobs. If a job does not complete before these timeout values expire, then the job is abandoned.
Java Transaction API (JTA) timeout
This is the WebLogic console setting for the time limit of a java transaction.
To set the JTA timeout:
Log on to the Oracle WebLogic Administration Console.
The Home screen of the Administration Console appears.
Under Services, select JTA.
The Settings screen appears with the JTA tab displayed.
In the Timeout Seconds field, enter the appropriate value.
Note:
A safe value is 7200 seconds (2 hours). If you have extremely large devices in your UIM system, you may want to increase this value to something like 10800 (3 hours) or 144000 (4 hours).Stuck Thread Max Time
This is the Weblogic console setting for the length of time a thread can work for before the server considers it stuck.
To set the Stuck Thread Max Time value:
Log on to the Oracle WebLogic Administration Console.
The Home screen of the Administration Console appears.
Under Environment, select Servers.
The Summary of Servers screen appears.
In the Servers table, click the Managed_Server.
The Settings for Managed_Server screen appears.
Select the Tuning tab to display it.
In the Stuck Thread Max Time field, enter the appropriate value.
Note:
A safe value is 7200 seconds (2 hours). If you have extremely large devices in your UIM system, you may want to increase this value to something like 10800 (3 hours) or 144000 (4 hours).JobDispatcher TimeToLive
This is the Network Integrity Mbean property value for the time limit of a Network Integrity job.
Note:
These three parameters should have the same value.You can adjust the stuck thread timeout values using the WebLogic Server Administration Console.
Note:
Stuck threads add to the server load and could cause problems. Configuring the Stuck Thread Count value puts the server into a Failed state if too many threads are stuck. A value of 4 may be appropriate.