6 Voucher and Wallet Server (VWS)

This chapter explains how and why the VWS servers are used.

Voucher and Wallet Server Overview

Any interactions by subscribers that require payment are processed, at some point, by the VWS.

The VWS database contains important subscriber information, and all information relating to their balances, promotions and vouchers. Like the SLC, all the main processing takes place inside the SLEE; the main processes responsible for handling billing interactions are:

  • beVWARS
  • beServer
  • beSync
  • beGroveller

Some NCAs may also be configured to handle other interactions between VWS and SLC (for example, DAP, OSD). Understanding when these interactions are invoked will also aid the troubleshooting process.

Billing pairs

VWSs are configured in billing pairs, rather than the standard N+1 model that the SLCs follow. One machine acts as the primary, the other secondary, whereby the primary processes all traffic, and syncs the information to the secondary.

In the event of an outage of the primary, traffic is handed to the secondary for processing. When the primary is brought back into operation, it requests all the updates it is missing from the secondary and begins to resync. Only once the syncing process is complete, will the primary resume operation and begin to handle all the traffic again.

From a subscriber’s perspective this is a completely transparent process. If an initial request is handled by the primary and it is taken out of service by the time a subsequent reservation comes in, the secondary will step in and handle the reservation. Regardless of the status of the primary, the secondary will continue to handle billing for any calls that "bounce" to it until the calls are complete.

Processes location

Billing processes are located in /IN/service_packages/E2BE/bin. Each process writes to its own log file in E2BE/tmp. Each instance of beVWARS also writes to its own log file.

General Billing Terms

Explanation of a few basic billing terms will aid understanding of how the Convergent Charging Controller VWS works, and the information provided in the rest of the VWS section.

  • Subscriber

The end user

  • Wallet

A container for a subscriber’s buckets.Many-to-many relationship with a subscriber:

  • A subscriber can have one primary and one secondary wallet
  • A wallet may be shared between many subscribers
  • Bucket

A record of a balance for a subscriber inside a wallet.One-to-many relationship with a wallet:

  • A wallet can have many buckets
  • A bucket can only belong to one wallet
  • Periodic Charge

Stored in a regular bucket, but with additional references to the periodic charge.Uses Expiry as the renewal date, that is, when the bucket expires the periodic charge will renew.Uses Value to store the current periodic charge state:

  • 1 - Terminated
  • 2 - Unsubscribed
  • 3,4,5,6 - Active (varying states of active)
  • 7,8,9 - Grace (varying states of Grace Period)
  • ACK/NACK

Standard nomenclature for a successful (ACK) or failed (NACK) response from the VWS.

beServer process

The beServer acts as a central contact point for connecting clients to the billing engine. Essentially any interaction with beVWARS will first go through beServer.

beServer maintains a list of currently connected clients and handles new connections.

Client list

Different clients include:

  • beClient (SLC)
  • PIbeClient (PI on the SMS)
  • ccsBeOrb (Java screens on the SMS)
  • beGroveller (background process for keeping un-used wallets up to date)

Listening port

The VWS server listens on port 1500 for incoming connections, and uses the Oracle Escher protocol for communications. If troubleshooting is required for billing traffic - port 1500 will contain the conversation that happens on the wire between Convergent Charging Controller components.

Client ID

Each connecting client has its own unique client ID. This is derived from a hash of the client name, specified by the configuration of the incoming client process.

Upon connection, beServer logs the client name and client id, which can be a useful reference when trying to determine the client for an EDR, for example:

Nov 23 04:36:11 ube01 beServer: [ID 839465 user.notice]
beServer(21708) NOTICE: Client 'slc01_ccsBeClient' (ClientId
87783972) has connectedNov 23 04:36:13 ube01 beServer: [ID 839465
user.notice] beServer(21708) NOTICE: Client 'PIbeClient' (ClientId
161986004) has connected

beVWARS process

beVWARS is the main process handling the work-load on a VWS. It is responsible for all interactions between the subscriber and their funds.

At a rudimentary level, it holds the cache that represents the most up-to-date information about a subscriber balance information, including uncommitted funds (that is, reservations).

Upon a request for a subscriber’s wallet, beVWARS will load the subscriber information from the database into cache, and periodically flush and write to the database based on configuration (beVWARS.walletCache{} section in eserv.config).

Handlers

beVWARS is reasonably flexible, and will operate using a configured set of Message and Event Handlers.

Message Handlers define how beVWARS will handle message requests from clients (for example, how to handle a voucher recharge request).

Event Handlers (known as plugins) contain a set of instructions to be run on wallets each time an event is triggered (for example, instructions to delete an expired balance).

Plugins example list

Plugins will be run prior to the handlers, so that any maintenance has been run prior to call connection. Plugins include:

  • beVWARSExpiry.so

Processes expired buckets, ensuring that expired funds are removed from the database

  • beVWARSMergeBuckets.so

Manages the number of buckets a wallet is allowed. If the maximum is hit, the new bucket will be merged into an existing one instead

  • ccsVWARSExpiry.so

Maintains CCS Wallet States, for example.

  • Moves Dormant Wallets to Active when they are used
  • Deletes Terminates Wallets after a configurable period of time
  • ccsVWARSActivation.so

Activates Wallets including initial credits

  • ccsVWARSPeriodicCharge.so

Handles all PeriodicCharge interactions and state changes

  • ccsNotification.so

Creates real-time notifications

Handlers example list

Handlers include:

  • ccsVWARSReservationHandler.so

Performs the UBE-side processing of all messages relating to chargeable call processing including calculating tariffs

  • ccsVWARSNamedEventHandler.so

Performs the UBE-side processing of messages relating to named events. This includes:

  • Returning the cost for an event class and event name combination
  • Generating named event EDRs.
  • ccsVWARSRechargeHandler.so

Handles General Wallet Recharges

  • beVWARSCCDRHandler.so

Handles EDR generation in some situations where one would not usually be generated (can be specifically requested by a BE Client)

  • ccsVWARSWalletHandler.so

Performs the UBE side processing of all messages relating directly to wallets. This includes:

  • Wallet Information (WI) - responds with wallet information
  • Wallet Create (WC) - creates new wallets
  • Wallet Update (WU) - updates wallets
  • Wallet Delete (WD) - deletes existing wallets and corresponding buckets
  • Bad PIN updates (BPIN) - updates Bad PIN balance if the wallet has one.

EDRs are produced for all Wallet updates (create/modify/delete/recharge) with the details of the change

  • ccsVWARSVoucherHandler.so

Performs the Billing Engine side processing of messages directly relating to vouchers. This includes voucher reservation/commit, alteration and deletion

beVWARS scalability

beVWARS is a scalable process, and runs multiple instances on the Convergent Charging Controller platform.

As the beVWARS contains the most up to date information about a Wallet, the beServer needs to ensure that not only is the workload even, but subsequent requests for wallet information must always go to the same beVWARS instance.

Workload spreading

The algorithm for this is WalletID MOD Total_beVWARS_Instances.

The Wallet ID is essentially an identity field in the database, and will increment in a way that ensures even workload.

The number of instances created is determined by /IN/service_packages/SLEE/etc/SLEE.cfg. For example: 

INTERFACE=beVWARS0 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS1 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS2 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS3 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS4 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS5 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS6 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS7 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS8 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS9 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS10 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS11 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS12 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS13 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS14 beVWARS.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beVWARS15 beVWARS.sh /IN/service_packages/E2BE/bin
EVENT

However, it is also defined in the BE{} section of eserv.config, which saves time and complexity:

numVWARS = 16

beSync process

beSync is responsible for keeping the Primary and Secondary Billing Engine pair in the same state at any given time.

Each time the beVWARS performs an interaction on the database, a sync file is written to the beSync repository - usually /IN/service_packages/E2BE/sync/<beVWARS Instance #>.

These sync files are processed by beSync and used to write the information to the other Billing Engine in the pair. After they are completed the files are deleted. The number of files waiting to be processed can be indicative of an unsynchronized Billing Engine pair.

When only one node in a Billing Engine pair is running, all the information yet to be sent to the other node will start collecting in the beSync repository and will continue to do so until it comes back up.

beSync will also transmit information about new reservations, ensuring that both Billing Engine nodes are aware of any ongoing uncharged funds. The result is that the Primary can go down mid-call, and the subscriber will not be aware that anything has gone wrong.

Billing Engine startup

Upon startup, beSync will ask each local beVWARS for their last written sequence number.

beSync uses this, and the BE_VWARS_SEQ_NUM table in the database, to track what updates are yet to be synced between the machines in the pair and begin syncing immediately.

Since beServer will not accept any connections until the Billing Engine is completely up to date, it can be prudent to monitor the backlog in E2BE/sync to see how it is progressing.

beSync will also collect reservations from the other node, so it is completely up to date.

beGroveller process

beGroveller is responsible for searching the database for unused wallets, and sending them to the correct beVWARS process when requested.

During normal processing, events are triggered only when a subscriber interacts with the wallet. Some events (such as expiry and periodic charges) should be triggered regardless of whether the wallet has been used by a subscriber recently or not.

Event processing

In order to process these events, beGroveller collects and sends lists of wallets IDs to beVWARS for processing. This processing triggers any events which are due to occur in the same way a normal interaction would, except wallets triggered from beGroveller lists do not trigger any message handlers.

No-processing times

In general it is not imperative that buckets are expired in real-time, and grovelling unused wallets consumes resources that beVWARS would otherwise be using to process regular traffic.

For this reason beGroveller contains some configuration (beGroveller{} section in eserv.config) for running only during certain times of the day: 

noProcessingTimes = [{ startsAt = "06:00", endsAt = "09:30"
}]

During these no-processing times when the beVWARS asks for more wallets to grovel, beGroveller will report that there are none.

beGroveller scalability

Like beVWARS, beGroveller is also a scalable process, and runs multiple instances on the Convergent Charging Controller platform. Although the beGroveller uses the same algorithm for calculating which beGroveller is going to serve a particular wallet, it does not need to run the same number of instances as beVWARS - it will often run much less.

Both beVWARS and beGroveller are able to determine the instances of the other, and will access the appropriate instance accordingly.

Workload spreading

The algorithm for this is WalletID MOD Total_beGroveller_Instances

The number of instances created is determined by /IN/service_packages/SLEE/etc/SLEE.cfg. For example: 

INTERFACE=beGroveller0 beGroveller.sh
/IN/service_packages/E2BE/bin EVENTINTERFACE=beGroveller1
beGroveller.sh /IN/service_packages/E2BE/bin
EVENTINTERFACE=beGroveller2 beGroveller.sh
/IN/service_packages/E2BE/bin EVENTINTERFACE=beGroveller3
beGroveller.sh /IN/service_packages/E2BE/bin EVENT

However, it is also defined in the BE{} section of eserv.config - which saves time and complexity:

numGrovellers = 4

Useful Commands and Scripts

This topic lists a few useful scripts for investigating a billing issue.

Without access to the SMS screens (quite common when dealing with a production environment) it can be difficult to ascertain basic information about a subscriber and their wallets.

showCli.sh

This script displays the basic information about a CLI in the database. At a glance it will tell you:

  • whether the number exists in the database, and
  • if so, some useful information about their entry (including defined wallet IDs)

It can also be used to compare the information on the SMS database with the VWS (or SLC) database; thus allowing a quick check that replication is functioning.

All databases should always have the same information. If there are differences, there could be an issue with replication, and a full resync may be required.

Machine Type: SMS, VWS, SLC

Location: /IN/service_packages/CCS/bin/showCli.sh

Usage: ./showCli.sh CLI/MSISDN

Example Output:

/IN/service_packages/CCS/bin$ ./showCli.sh 12345CLI
ACCT_REFERENCE NAME WALLET ACCOUNT_NUMBER---------- --------------
------------------------------- ------ --------------12345 2143
Primary 2143 1656

showWallet.sh

This script displays the contents of a subscriber’s wallet. At a glance it will give you information on the wallet state, and what buckets are contained within that wallet, including expiry dates and references.

It can also be used to compare the information on the Primary and Secondary Billing Engine databases - thus allowing a quick check that the Billing Engine pair is in sync. Unless one of the nodes has been down for an extended period of time, and is still being brought into sync (in which case it will not be handling traffic), the results on both Billing Engine nodes should have the same information in the database.

Note: beVWARS operates a cache, so the database will not necessarily reflect a subscriber’s available funds. Ongoing reservations are not written to the database until they are confirmed. A subscriber could legitimately appear to have sufficient funds for a call in the database, but have insufficient funds due to an in-progress reservation.

Machine Type: VWS

Location: /IN/service_packages/E2BE/bin/showWallet.sh

Usage: ./showWallet.sh WalletID

Example Output:

/IN/service_packages/E2BE/bin$ ./showWallet.sh 2143Showing
wallet 2143 on e2be_admin ID MAX_CONCURRENT S NEVER_EXPIRES EXPIRY
NEVER_ACTIVATED ACTIVATION_DAT---------- -------------- -
------------- -------------- --------------- -------------- 2143 10
A 1 20100701022600 0 20100726035409 WALLET BALANCE_TYPE L
MINIMUM_CREDIT---------- ------------ - -------------- 2143 22 D 0
2143 27 D 02 rows selected. ID WALLET BALANCE_TYPE NEVER_EXPIRES
EXPIRY VALUE NEVER_USED LAST_USE REFERENCE START_DATE----------
---------- ------------ ------------- -------------- ----------
---------- --------------
---------------------------------------------------------
-------------- 16734 2143 22 0 20111106040500 500 0 20101122043401
19700101000000 16998 2143 27 0 20101223033753 100 0 20101123033753
17078 2143 27 0 20101223041742 600 0 201011230417426 rows
selected.

Tip: The output is wrapped, and generally easier to read within a terminal window