3 Starting and Stopping ECE

This chapter provides instructions for starting and stopping Oracle Communications Billing and Revenue Management Elastic Charging Engine (ECE) nodes.

Caution:

Restarts of the ECE system are not expected. Stopping all charging server nodes and restarting them results in loss of data. Do not stop all charging server nodes unless you want all data to be removed from Coherence caches. See "Restoring the ECE System" for more information.

About Starting and Stopping ECE

Starting and stopping ECE involves starting and stopping ECE nodes. An ECE node is a Java Virtual Machine (JVM) process running Elastic Charging Engine software. Most ECE nodes are members of the Coherence cluster. Nodes for ECE software processes that are not Coherence members typically need to be restarted after you make configuration changes to the software processes on a running system.

You use Elastic Charging Controller (ECC) to start and stop ECE nodes based on the role name assigned to them in the ECE topology file (ECE_home/oceceserver/config/eceTopology.conf where ECE_home is the directory in which ECE is installed). For example, charging server nodes have the role server; the command start server starts all charging server nodes in the cluster.

ECE must be in a usage-processing state to be able to process usage requests sent from the Elastic Charging Client. When you are initially starting the ECE system, you place ECE in a usage-processing state by running ECE nodes in a specific order:

Start the charging server nodes (start server).
Start the configLoader utility (start configLoader).
Start Pricing Updater (start pricingUpdater).
Start Customer Updater (start customerUpdater).

For example, when you first load your data into ECE, you do the following:

Start ECC.
```
./ecc
```
Run the following commands in the listed order.

These commands start the charging server nodes and load data into the ECE caches in the required order.
```
start
start configLoader
start pricingUpdater
start customerUpdater
```

Stopping all charging server nodes will result in loss of cache data. You can stop a specific charging server node with no loss of data in Coherence caches. See "Stopping Charging Server Nodes".

Non-charging server nodes can be stopped and restarted with no loss of cache data. See "Stopping Non-Charging Server Nodes".

See "Using the Elastic Charging Controller to Manage Nodes" for information about how to start and stop nodes using ECC.

Starting an ECE Standalone System

Start ECE nodes in the following order when starting an ECE standalone system:

Start the charging server nodes (start server).
Start the configLoader utility (start configLoader).
Start Pricing Updater (start pricingLoader).
Start Customer Updater (start customerLoader).

Starting and Stopping Diameter Gateway

Note:

This procedure assumes you have done the following:

Created Diameter Gateway instances by using the charging.diameterGatewayConfigurations MBean in a JMX editor.
Defined each instance in the ECE_home/oceceserver/config/eceTopology.conf file so that it can be started and stopped using ECC.

For information about creating Diameter Gateway instances and defining Diameter Gateway instances in your topology, see the post-installation tasks in BRM Elastic Charging Engine Installation Guide.

To start Diameter Gateway:

Verify that the Elastic Charging Server is running with all required data.

All ECE nodes used for Diameter credit-control processing must be running: charging server nodes, Pricing Updater, Customer Updater, External Manager (EM) Gateway, BRM Gateway, and Rated Event Formatter.

You must have at least one charging server node running.

All data, Oracle Communications Billing and Revenue Management (BRM) data and Pricing Design Center (PDC) pricing data, must be loaded into the ECE data grid.
Verify that the ECE nodes are in a usage-processing state:
1. Start the ECE charging servers (if they are not started).
2. Start a JMX editor, such as JConsole, that enables you to edit MBean attributes.
3. Connect to the ECE charging server node set to start CohMgt = true in the ECE_home/oceceserver/config/eceTopology.conf file.
  
  The eceTopology.conf file also contains the host name and port number for the node.
4. In the editor's MBean hierarchy, expand the ECE State Machine node.
5. Expand StateManager.
6. Expand Attributes.
7. Make sure the stateName attribute is set to UsageProcessing.
On the driver machine, change to the ECE_home/oceceserver/bin directory.
Start ECC:
```
./ecc
```
Do one of the following:
- To start all Diameter Gateway instances, run the following command:
```
start diameterGateway
```
- To start a specific Diameter Gateway instance, run the following command:
```
start Instance_Name
```
  where Instance_Name is the name of the Diameter Gateway instance as you defined it in the node-name column of the ECE_home/oceceserver/config/eceTopology.conf file.
At startup, Diameter Gateway connects to the network interface and port on which it listens for Diameter messages.

To stop Diameter Gateway, run one of the following commands from ECC:

To stop all Diameter Gateway instances, run the following command:
```
stop diameterGateway
```
To stop a specific Diameter Gateway instance, run the following command:
```
stop Instance_Name
```

For information about configuring Diameter Gateway, see the post-installation tasks in BRM Elastic Charging Engine Installation Guide.

Starting and Stopping RADIUS Gateway

Note:

This procedure assumes you have added RADIUS Gateway instances and configured them in your topology. See "Adding RADIUS Gateway Nodes for Authentication and Accounting" and "Configuring RADIUS Gateway Nodes" for more information.

To start RADIUS Gateway:

Verify that the Elastic Charging Server is running with all required data.
Do one of the following:
- To start all RADIUS Gateway instances, run the following command:
```
start radiusGateway
```
- To start a specific RADIUS Gateway instance, run the following command:
```
start Instance_Name
```
  where Instance_Name is the name of the RADIUS Gateway instance as you defined it in the node-name column of the ECE_home/oceceserver/config/eceTopology.conf file.
At startup, RADIUS Gateway connects to the network interface and port on which it listens for RADIUS requests.

To stop RADIUS Gateway, run one of the following commands from ECC:

To stop all RADIUS Gateway instances, run the following command:
```
stop radiusGateway
```
To stop a specific RADIUS Gateway instance, run the following command:
```
stop Instance_Name
```

Starting and Stopping Rated Event Formatter

To start Rated Event Formatter:

Verify that the Oracle NoSQL database is running.
On the driver machine, change to the ECE_home/oceceserver/bin directory.
Start ECC:
```
./ecc
```
Do one of the following:
- To start all Rated Event Formatter instances, run the following command:
```
start ratedEventFormatter
```
- To start a specific Rated Event Formatter instance, run the following command:
```
start Instance_Name
```
  where Instance_Name is the name of the Rated Event Formatter instance as you defined it in the node-name column of the ECE_home/oceceserver/config/eceTopology.conf file.
  
  When a custom plug-in is configured for a non-BRM system, the Rated Event Formatter instance must start at the same time as the custom plug-in instance.
  
  If the BrmCdrPluginDirect Plug-in is already running, stop the Rated Event Formatter instance associated with that plug-in and start both the instances at the same time.
At startup, Rated Event Formatter connects to the Oracle NoSQL database and reads the rated event information from the Oracle NoSQL data store.

To stop Rated Event Formatter, run one of the following commands from ECC:

To stop all Rated Event Formatter instances, run the following command:
```
stop ratedEventFormatter
```
To stop a specific Rated Event Formatter instance, run the following command:
```
stop Instance_Name
```
The instances associated with the BrmCdrPluginDirect Plug-in and custom plug-in can be stopped and restarted at any time. When the instances are restarted, they continue processing the rated events, including any rated events generated during the downtime.

For information about configuring Rated Event Formatter, see the discussion of implementing ECE with BRM in BRM Elastic Charging Engine Implementation Guide.

Starting and Stopping External Manager Gateway

The Elastic Charging Server must be in a usage-processing state before you can start EM Gateway.

ECE uses EM Gateway during the rerating process and to synchronize BRM and ECE customer data. To handle all customer data update requests, EM Gateway must run continuously in a production environment.

EM Gateway and BRM Gateway must be running before rerating is run.

To start EM Gateway:

On the driver machine, change to the ECE_home/oceceserver/bin directory.
Start ECC:
```
./ecc
```
Run the following command:
```
start emGateway
```

To stop EM Gateway, run the following command:

stop emGateway

Starting Charging Server Nodes

To start charging server nodes:

On the driver machine, change to the ECE_home/oceceserver/bin directory.
Start the EEC:
```
./ecc
```
Do one of the following:
- To start all of the charging server nodes in your topology, run the following command:
```
start [server]
```
- To start a specific charging server node, run the following command:
```
start nodeName
```
  where nodeName is the name of the node specified in the node-name column of the ECE_home/oceceserver/config/eceTopology.conf file.

Stopping Charging Server Nodes

Caution:

Stopping all charging server nodes will result in loss of data on a standalone system.

Server redundancy is a minimum requirement of ECE installations to enable built-in high availability and fault tolerance features that prevent all charging server nodes from stopping.

If you stop all charging server nodes, you will lose data in all Coherence caches (such as the data in the Customer cache, Chargeoffering cache, Serviceeventmap cache, and so on). The charging server nodes are not intended to be stopped.

See "Restoring the ECE System" for more information.

To stop a specific charging server node, run the following command:

stop nodeName

where nodeName is the name of the node specified in the node-name column of the ECE_Home/oceceserver/config/eceTopology.conf file.

Stopping Non-Charging Server Nodes

Non-charging server nodes can be stopped and restarted with no loss of cache data. For example, stopping the following ECE nodes disrupts temporarily real-time data updates coming from other applications, but the updates are processed when the ECE system is restarted:

Customer Updater (stop customerUpdater)
Pricing Updater (stop PricingUpdater)
BRM Gateway (stop brmGateway)
Rated Event Formatter (stop ratedEventFormatter)
EM Gateway (stop emGateway)

Starting and Stopping BRM Gateway

Elastic Charging Server must be in a usage-processing state before you can start BRM Gateway.

BRM Gateway, and EM Gateway, must be running before rerating is run.

To start the BRM Gateway:

On the driver machine, change to the ECE_home/oceceserver/bin directory.
Start ECC:
```
./ecc
```
Run the following command:
```
start brmGateway
```

To stop BRM Gateway, run the following command:

stop brmGateway

Starting and Stopping Data-Loading Utility Nodes

The data-loading utility nodes (configLoader, pricingLoader, customerLoader) start as Coherence nodes and then stop when they are done with their processing. You do not need to stop them.

Restoring the ECE System

Restarts of the ECE system are not expected because ECE is built to always be running. ECE has built-in high availability and fault tolerance when server redundancy is employed. Server redundancy is a minimum requirement of ECE installations.

If you stop all charging server nodes on a standalone system, you will lose data in Coherence caches.

Patching ECE to upgrade it to a new version does not require restarting the system when you perform a rolling upgrade. A rolling upgrade applies the patch while the system is running. See the discussion about upgrading to new releases in BRM Elastic Charging Engine Installation Guide for more information about rolling upgrades.

If a restart is required after all ECE nodes are stopped, restore the ECE system by doing the following:

In PDC, publish all the PDC pricing data (the metadata, setup, pricing, and profile data) from the PDC database to ECE by running the following command:
```
ImportExportPricing -publish -metadata -config -pricing -profile -target [ece] 
```
In ECE do the following:
1. On the driver machine, change to the ECE_home/oceceserver/bin directory.
2. Start ECC.
```
./ecc
```
3. Start ECE processes and gateways in the following order:
  
  Note:
  Depending on your installation, you can start Diameter Gateway, RADIUS Gateway, both Diameter Gateway and RADIUS Gateway, or none.
```
start server
start configLoader
start pricingUpdater
start customerUpdater
start emGateway
start brmGateway
start ratedEventFormatter
start diameterGateway
start radiusGateway
```
  All data is now back in the ECE data grid.
  
  Real-time data updates (coming from BRM) that had been temporarily disrupted due to the shutdown are processed upon restart.
  
  Diameter Gateway processing that had been temporarily disrupted due to the shutdown is resumed upon restart. For example, the translation of Diameter requests into ECE Java API requests and the processing of push notifications from Elastic Charging Server (the processing of ECE JMS notifications) is resumed.

Troubleshooting Starting ECE

This section describes troubleshooting tips for starting ECE nodes.

Start Command Fails When Trying to Start Nodes in a Large Cluster

If you try to start nodes in a large cluster, and the start command fails, the problem may be due to the limitation on how many simultaneous SSH connections the machines in your environment can make from another machine or to another machine. The start command, by default, attempts to start 10 nodes simultaneously using 10 different threads. If the maximum number of open sessions permitted per network connection is less than 10 on the driver machine, the start command will fail. Likewise, if the maximum number of simultaneous connections to the SSH daemon on the server machine is less than the number of threads attempting to make connections on the server machine, the start command will fail.

To resolve the issue, change the number of nodes the start command will start simultaneously, or (if your runtime environment allows) change the limit of simultaneous SSH connections your environment can make from or to a machine.

For Linux systems, to set the limit for how many simultaneous SSH connections can be made from the driver machine in your environment:

On the driver machine, stop the SSH daemon.
Open the /etc/ssh/sshd_config file.
Set the MaxSessions property.

MaxSessions specifies the maximum number of open sessions permitted per network connection. The default is 10.
Save and close the file.
Start the SSH daemon.

To set the limit for how many simultaneous SSH connections can be made to the server machine in your environment:

On server machine, stop the SSH daemon.
Open the /etc/ssh/sshd_config file.
Set the MaxStartups property to be equal to or more than the number of nodes the start command starts at one time.

MaxStartups specifies the maximum number of concurrent unauthenticated connections to the SSH daemon on the server machine. Additional connections are dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10.
Save and close the file.
Start the SSH daemon.

To set the number of nodes the start command will start simultaneously:

On the driver machine, open the ECE_home/oceceserver/ece.properties file.
Add the following line to the file:
```
numberOfStartCommandsExecutorThreads=number_of_threads
```
where number_of_threads is the number of threads the start command uses to start the (same) number of nodes simultaneously.

number_of_threads must be equal to or less than the value of the MaxSessions property set for the given environment.
Save and close the file.