7 Policy-Based Scaling

WebLogic Server supports policy-based scaling, which is the ability to automatically execute a scaling operation on a dynamic cluster in response to a change in demand, or any other condition, that is monitored by a policy that has been configured in a domain-scope diagnostic system module. The following sections describe policy-based scaling in WebLogic Server and include an example of policy-based scaling using smart rules to track the average throughput in a cluster:

What is Policy-Based Scaling?

Policy-based scaling is based on policies and associated actions, leveraging the Policies and Actions component of the WebLogic Diagnostics Framework (WLDF). A policy sets the conditions under which a scaling operation on a dynamic cluster must occur, and when these conditions are met, the scaling action performs the scaling operation. For more information on the Policies and Actions component of WLDF, see Configuring Policies and Actions in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

WLDF provides two elastic actions, scale up and scale down, that you can assign to a policy to perform a scaling operation. For more information on the scale up and scale down actions, see Elastic Actions. Only one scaling action can be assigned to a given policy.

For example, Example 7-1 shows the WLST commands to create and configure a policy that, when triggered, executes a scale up action. Note the following:

  • The scale up action, scaleUp, is created. This action is configured to scale up the dynamic cluster, DynamicCluster, by one server instance.

  • The policy, named highMark, is configured to be evaluated every 10 seconds.

  • When the policy is triggered — that is, it is evaluated to true — the corresponding scale up action is executed.

Similarly, Example 7-2 shows the configuration of a policy that, when triggered, executes a scale down action. Note the following:

  • The scale down action, scaleDown, is created. This action is configured to scale down the dynamic cluster, DynamicCluster, by one server instance.

  • The policy, named LowMark, is configured to be evaluated every 10 seconds.

  • When the policy is triggered — that is, it is evaluated to true — the corresponding scale down action is executed.

Example 7-1 Creating a Policy to Scale Up a Dynamic Cluster

scaleUp=wn.lookupScaleUpAction('scaleUp')
if scaleUp == None:
  print "Creating scale up action"
  scaleUp=wn.createScaleUpAction('scaleUp')
scaleUp.setScalingSize(1)
scaleUp.setClusterName(DynamicCluster)
high=wn.createWatch('highMark')
high.setExpressionLanguage('EL')
high.getSchedule().setMinute('*')
high.getSchedule().setSecond('*/10')
high.getRuleType()
high.setAlarmType('AutomaticReset')
high.setAlarmResetPeriod(150000)
high.setRuleExpression("wls:ClusterHighThroughput('"+DynamicCluster+"', '10s', '30s', 250, 60)")
high.addNotification(scaleUp)
high.setEnabled(true)

Example 7-2 Creating a Policy to Scale Down a Dynamic Cluster

scaleDown=wn.lookupScaleDownAction('scaleDown')
if scaleDown == None:
  print "Creating scale down action"
  scaleDown=wn.createScaleDownAction('scaleDown')
scaleDown.setScalingSize(1)
scaleDown.setClusterName(DynamicCluster)
low=wn.createWatch('lowMark')
low.setExpressionLanguage('EL')
low.getSchedule().setMinute('*')
low.getSchedule().setSecond('*/10')
low.getRuleType()
low.setAlarmType('AutomaticReset')
low.setAlarmResetPeriod(60000)
low.setRuleExpression("wls:ClusterLowThroughput('"+DynamicCluster+"', '10s', '60s', 50, 75)")
low.addNotification(scaleDown)
low.setEnabled(true)

Note:

To configure automated elasticity for a dynamic cluster, you must create a domain-scope diagnostic system module in which you define the scaling policy, along with its corresponding elastic action, and then target that diagnostic module to the Administration Server.

Introducing Smart Rules

Smart rules are prepackaged functions with a set of configurable parameters that allow you to create complex policy expressions just by specifying the values for these configurable parameters.

A smart rule can be used as a predicate within a policy expression, either alone or with other predicates to create a more complex policy expression. Smart rules policy expressions are specified using the Java Expression Language (EL). See Configuring Smart Rule Based Policies in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

Smart rules can be used in policy-based scaling. Policy-Based Scaling Example shows the use of policies that use smart rules, wls:ClusterHighThroughput() and wls:ClusterLowThroughput(), in their rule expressions. These two smart rules are also shown in Example 7-1 and Example 7-2:

  • Example 7-1 shows the wls:ClusterHighThroughput() smart rule. This smart rule measures whether the average throughput in a cluster is increasing, as indicated by the average value of the ThreadPoolRuntimeMBean.Throughput attribute in each Managed Server.

    As specified in this example, this smart rule is triggered — that is, it is evaluated to true — if the average Throughput value, collected at 10-second intervals over the previous 30 seconds, is greater than or equal to 250 on at least 60 per cent of the servers in the dynamic cluster, DynamicCluster. When this smart rule is triggered, it executes the scale up action, scaleUp, which starts up a server instance in the cluster.

  • Example 7-2 shows the wls:ClusterLowThroughput() smart rule. This smart rule measures whether the average throughput in a cluster is decreasing, as indicated by the average value of the ThreadPoolRuntimeMBean.Throughput attribute in each Managed Server.

    As specified in this example, this smart rule is triggered — that is, it is evaluated to true — if the average Throughput value, collected at 10-second intervals over the previous 60 seconds, is less than 50 on at least 75 per cent of the servers in the dynamic cluster, DynamicCluster. When this smart rule is triggered, it executes the scale down action, scaleDown, which stops a server instance in the cluster.

For more information about creating and configuring smart rule-based policies using the WebLogic Server Administration Console and Fusion Middleware Control, see:

See Smart Rule Reference in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

Policy-Based Scaling Example

You can create policies that use the wls:HighThroughput() and wls:LowThroughput() smart rules to scale a dynamic cluster up under high load, and then scale the dynamic cluster back down when the demand decreases.

Note:

The examples in this section use WebLogic Server and collocated Oracle Traffic Director 12c on Windows.

Prerequisites

To successfully run this example, first prepare your environment and download the required files used in this example.

This section includes the following tasks:

Installing Required Software

To run this example, download and install the following software:

Downloading and Unpacking Required Files

The elasticity demo JAR files are available for download from the article Elasticity for Dynamic Clusters in the WebLogic Server Blog at the following location:

https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/File/540d5bc5bdb70b2b779130ee381cf061/otd_wls_elasticity_demo.zip

Unpack the following files from the elasticity JAR that are used in this example:

Note:

Save all of the following files in the same directory. This directory is referred to as DEMO_HOME in this example.

  • collocated_wls_otd_elasticity_domain.jar—the domain template used in this example. This domain template will set up your domain, including creating the necessary dynamic cluster, Oracle Traffic Director instances, and policies and actions and deploying the web.war application to the dynamic cluster.

  • unpack-elasticity-domain script—this script executes the unpack command to unpack the collocated_wls_otd_elasitcity_domain.jar file and create the domain, elasticity_domain, used in this example.

    The Windows version of this script is named unpack-elasticity-domain.cmd. The Unix version is unpack-elasticity-domain.sh.

  • Elasticity5MinuteDemo.jmx—the JMeter test plan used in this example.

  • Elasticity1HourDemo.jmx—optional, longer JMeter test plan.

Setting the Environment

After installing WebLogic Server and Oracle Traffic Director in the same ORACLE_HOME directory, run the setWLSEnv script command to set your environment variables for this example.

  1. Open a command window.

    Note:

    If you are running this demo on a Windows system, you may need to have administrator privileges. If so, open the command window as follows:

    1. Click Start.

    2. Right-click Command Prompt, then select Run as administrator.

      If Command Prompt is not listed in the Start Menu, enter command in the Search field, then right-click Command Prompt.

    3. Enter the administrator credentials, if requested.

  2. In the command window, change to the ORACLE_HOME/wlserver/server/bin directory.

  3. Run the setWLSEnv script. For example:

    Windows:

    ORACLE_HOME\wlserver\server\bin> setWLSEnv.cmd
    

    Unix:

    prompt> sh setWLSEnv.sh

Unpacking the Domain Template

Unpacking the domain template, collocated_wls_otd_elasticity_domain.jar, sets up your domain for this example, including the following:

  • Creates the elasticity_domain domain and directory.

  • Creates and configures the dynamic cluster, DynamicCluster, and four dynamic servers, DynamicCluster-1, DynamicCluster-2, DynamicCluster-3, and DynamicCluster-4.

  • Creates and configures the Oracle Traffic Director instance, elasticity-otd, including a new server pool, origin-server-pool-1, which is configured for round-robin load balancing and dynamic discovery.

  • Create and configures the smart rule-based policies, highMark and lowMark, and the elastic actions, scaleDown and scaleUp.

To unpack the domain template:

  1. Change to the directory in which you unpacked the demo JAR file. For example, on Windows:
    ORACLE_HOME\wlserver\server\bin> cd DEMO_HOME
    
  2. Run the unpack-elasticity-domain script. For example:

    Windows:

    DEMO_HOME> unpack-elasticity-domain.cmd
    

    Unix:

    DEMO_HOME> sh unpack-elasticity-domain.sh
    

See The Unpack Command in Creating Templates and Domains Using the Pack and Unpack Commands.

Starting the Administration Server and Node Manager

After you unpack the domain template, start the Administration Server and Node Manager by running the start-servers script as follows:

Windows:

DEMO_HOME> start-servers.cmd

Unix:

DEMO_HOME> sh start-servers.sh

The start-servers script also attempts to start the Oracle Traffic Director instance. However, when you run this script for the first time, you are likely to see the following error message:

Error starting otd_elasticity-otd_localhost, likely already running or not yet configured

This message is generated if the Oracle Traffic Director instance is not yet configured, as described in Starting the Oracle Traffic Director Instance.

Note:

The WebLogic Server Administration Console and Fusion Middleware Control for this demo are on port 20001. To information about displaying either of these consoles, see Starting the Oracle Traffic Director Instance and Viewing Dynamic Server Configuration and Activity.

See Starting and Stopping Servers in Administering Server Startup and Shutdown for Oracle WebLogic Server.

See Starting and Stopping Node Manager in Administering Node Manager for Oracle WebLogic Server.

Starting the Oracle Traffic Director Instance

You need to run the create-otd script the first time you run this demo. Subsequently, Oracle Traffic Director is started when you start the server instances as described in Starting the Administration Server and Node Manager.

To run the create-otd script, enter the following command in the command window in which you started the Administration Server and Node Manager:

Windows:

DEMO_HOME> create-otd.cmd

Unix:

DEMO_HOME> sh create-otd.sh

This script:

  • Creates a new Oracle Traffic Director instance, elasticity-otd, at port 20009.

  • Create a new server pool, origin-server-pool-1.

  • Configures the server pool for round-robin load balancing and dynamic discovery.

See Starting, Stopping, and Restarting Oracle Traffic Director Instances in Administering Oracle Traffic Director.

To view your Oracle Traffic Director instance using Fusion Middleware Control:

  1. Launch Fusion Middleware Control by entering the following URL in your browser's address bar:
    localhost:200001/em
    
  2. Log in by entering the administrator credentials, which are provided in the unpack-elasticity-domain script.
  3. From the WebLogic Domain menu, select Administration > OTD Configurations.
  4. In the Oracle Traffic Director Configurations table, select elasticity-otd.

    The Fusion Middleware Control console displays the main Oracle Traffic Directory monitoring page from which you can view statistics about server activity.

You can also test your Oracle Traffic Director instance by entering the following URL in your browser: http://localhost:20009/web/HelloServlet. Refresh the page several times to check that the load balancing is working. You should see the numbers on the page change.

To use WLST to check the state of the instance to make sure it is running, enter the following command:

state("elasticity-otd")

Viewing Dynamic Server Configuration and Activity

When you run the start-servers script, as described in Starting the Administration Server and Node Manager, the following WebLogic Server instances are started:

  • The Administration Server

  • Two server instances in the dynamic cluster named DynamicCluster:

    • DynamicCluster-1

    • DynamicCluster-2

You can use either the WebLogic Server Administration Console or Fusion Middleware Control to view the configuration of DynamicCluster and also the number of currently running server instances in the cluster as the demo runs.

For example, to view DynamicCluster in the WebLogic Server Administration Console:

  1. Launch the WebLogic Server Administration Console by entering the following URL in your browser's address bar:
    localhost:20001/console
    
  2. Log in by entering the administrator credentials, which are provided in the unpack-elasticity-domain script.
  3. In the console home page, click Servers.

    The Summary of Servers page is displayed, listing the servers that are currently running and shut down. You can refer to this page as the demo runs to see the state of server instances.

See Starting and Stopping Servers in Administering Server Startup and Shutdown for Oracle WebLogic Server.

Using JMeter to Drive the Demo

To execute the test plan from JMeter:

  1. Open a command prompt and change to the bin directory of the JMeter installation. For example, C:\JMeter\apache-jmeter-2.13\bin.
  2. Enter the following command to start JMeter in GUI mode:
    jmeter.bat
    
  3. In the JMeter console, select File > Open and select Elasticity5MinuteDemo.jmx.
  4. From the Options menu, make sure that Log Viewer is selected.
  5. In the navigation pane on the left, make sure Elasticity Ping Servlet Driver is selected, shown in Figure 7-1.

    Figure 7-1 JMeter Test Navigation Pane

    Description of Figure 7-1 follows
    Description of "Figure 7-1 JMeter Test Navigation Pane"
  6. Select Run > Start.

Using the Monitoring Dashboard to Monitor Scaling Operations

After you start the test plan in JMeter, you can use the WebLogic Server Administration Console Monitoring Dashboard to view the server activity generated by the JMeter test. You will see the number of running server instances ramp up to four in response to the load. When the scenario completes, the additional instances will be scaled down until the minimum number of two running server instances is reached. This entire scenario lasts for five minutes.

To access the Monitoring Dashboard:

  1. Launch the WebLogic Server Administration Console, as described in Viewing Dynamic Server Configuration and Activity, if it is not already running.
  2. Under Charts and Graphs, click Monitoring Dashboard.
  3. In the Monitoring Dashboard, click the start button, shown in Figure 7-2.

    Figure 7-2 Monitoring Dashboard Start Button

    Description of Figure 7-2 follows
    Description of "Figure 7-2 Monitoring Dashboard Start Button"
  4. As JMeter ramps up the volume of simulated client invocations on the dynamic cluster, scaling policies are triggered that execute scaling actions.

    Notice the throughput of the currently running server instances at the moment new server instances are started. Figure 7-3 shows example output. The graphs you obtain may vary depending on your machine's performance characteristics.

    Figure 7-3 Dynamic Server Scaling Activity

    Description of Figure 7-3 follows
    Description of "Figure 7-3 Dynamic Server Scaling Activity"

In Figure 7-3, notice the following:

  • The top four charts show the throughput on the Managed Server instances DynamicCluster-1, DynamicCluster-2, DynamicCluster-3, and DynamicCluster-4.

  • The server instances DynamicCluster-1 and DynamicCluster-2, shown in the top two charts, show throughput spikes that begin at approximately 2:42:00.

  • The chart for DynamicCluster-3 shows that this third server was started when the throughput spikes occurred on the server instances DynamicCluster-1 and DynamicCluster-2.

  • The bottom chart shows the number of active Managed Servers in the cluster, as indicated by the value of the AliveServerCount attribute of the ClusterRuntimeMBean instances. This chart shows approximately when a third and a fourth server instance are started.

  • The third server instance, DynamicCluster-3, is started in the interval between 2:42:00 and 2:43:30.

  • Notice that a fourth server instance, DynamicCluster-4, is started shortly after 2:46:00, to coincide with an increase in throughout on the first three server instances.

  • After a new server instance is started, notice the corresponding decrease in throughput on the other server instances.

Re-running and Stopping the Elasticity Demo

You can re-run the elasticity demo from the JMeter GUI, or you can stop the demo and all running server instances, including Oracle Traffic Director, by running the stop-servers script as follows:

Windows:

DEMO_HOME> stop-servers.cmd

Unix:

DEMO_HOME> sh stop-servers.sh