30 Managing Persisted Data in the Oracle Database

Learn about data persistence and the tasks for managing Elastic Charging Engine (ECE) data stored in an Oracle Communications Billing and Revenue Management (BRM) cloud native database.

Topics in this document:

Enabling Persistence in ECE

You can configure ECE to persist its cache data in the Oracle database. Persistence provides a durable copy of the cache data that ECE can use to recover from events such as a node failure or the loss of a cache partition. When ECE needs to recover persisted data, it automatically reloads the data from the persistence database.

When you enable persistence, ECE components such as Customer Updater, Pricing Updater, and ConfigLoader persist the following data:

  • Data that BRM and PDC publish to the ECE cache

  • Mediation specification data that ECE loads into the cache

  • Data that ECE synchronizes with or receives from BRM

  • Other persisted data, including balances, top-up history, recurring bundle history, rated events, and Portal object IDs (POIDs)

Note:

  • If pricingLoadFromPersistence is set to false and the ECE state is not USAGE_PROCESSING, Pricing Updater cleans the existing ECE schema at startup and expects PDC to republish the pricing data.

  • Similarly, if customerLoadFromPersistence is set to false and the ECE state is not USAGE_PROCESSING, Customer Updater attempts to clean the ECE schema at startup before it reloads customer data from BRM.

  • When pricingLoadFromPersistence is set to false for an ECE cold restart, ensure that PDC republishes the pricing data and that the data is available on all ECE nodes before Customer Updater loads customer data. For more information, see "Avoiding Missing Pricing Data During ECE Cold Restarts".

During installation, upgrade, auto-recovery, and pod restart, ECE uses the Kubernetes REST API to:

  • Update the charging-settings-namespace ConfigMap to enable ECE to reload cache data from the persistence database

  • Retrieve metadata from ECE StatefulSets and pods

  • Apply management labels to ecs pods

To configure ECE cloud native for persistence:

  1. Configure ECE to reload cache data, retrieve metadata, and apply management labels during installation, upgrade, auto-recovery, and pod restart. Configure the ece-namespace service account to authenticate with the Kubernetes API server.

    For information about the rules defined in the ece-namespace role-based access control (RBAC) configuration, see the ece-clusterrole-sa.yaml file in oc-cn-ece-helm-chart.

  2. Enable and configure persistence. In the override-values.yaml file for oc-cn-ece-helm-chart, configure the required persistence credentials and connection settings.

    Note:

    Specify all persistence tablespace names in uppercase.

    secretEnv:
  PERSISTENCEDATABASEPASSWORD:
    - schema: 1
      PASSWORD: EncryptedPassword

  PERSISTENCEDBAPASSWORD:
    - schema: 1
      PASSWORD: EncryptedPassword    # SYSDBA user

  PERSISTENCEDATABASEKEYPASS:
    - schema: 1
      PASSWORD: EncryptedPassword

charging:
  persistenceEnabled: "true"

  cachePersistenceConfigurations:
    cachePersistenceConfigurationList:
      - clusterName: "BRM"
        persistenceStoreType: "OracleDB"
        persistenceConnectionName: "oraclePersistence1"
        reloadThreadPoolSize: "10"
        configLoadFromPersistence: "true"
        pricingLoadFromPersistence: "true"
        customerLoadFromPersistence: "true"
        partitionLossRecoverFromPersistence: "true"
        writeBehindThreadPoolSize: "1"

  connectionConfigurations:
    OraclePersistenceConnectionConfigurations:
      - clusterName: "BRM"
        schemaNumber: "1"
        name: "oraclePersistence1"
        dbSysDBAUser: "sys"
        dbSysDBARole: "sysdba"
        userName: "ece"
        hostName: ""
        port: "1521"
        sid: ""
        service: ""
        tablespace: "ECETABLE"
        temptablespace: "ECETEMP"
        cdrstoretablespace: "ECECDRTABLESPACE"
        cdrstoreindexspace: "ECECDRINDEXSPACE"
        jdbcUrl: ""
        retryCount: "3"
        retryInterval: "1"
        maxStmtCacheSize: "100"
        connectionWaitTimeout: "300"
        timeoutConnectionCheckInterval: "300"
        inactiveConnectionTimeout: "300"
        databaseConnectionTimeout: "600"
        persistenceInitialPoolSize: "4"
        persistenceMinPoolSize: "4"
        persistenceMaxPoolSize: "12"
        reloadInitialPoolSize: "0"
        reloadMinPoolSize: "0"
        reloadMaxPoolSize: "20"
        dbSSLEnabled: "true"
        dbSSLType: "twoway"
        sslServerCertDN: "DC=local,DC=oracle,CN=pindb"
        trustStoreLocation: "/home/charging/ext/ece_ssl_db_wallet/schema1/cwallet.sso"
        trustStoreType: "SSO"
        walletLocation: "/home/charging/wallet/ecewallet/"
        cdrStorePartitionCount: "32"
        queryTimeout: "5"

    Note:

    • When configLoadFromPersistence is set to true, ConfigLoader loads configuration data from the persistence database. The configDataDirectory log message appears before ConfigLoader selects the data source and does not indicate that ECE loaded mediation specification files, such as diameter_mediation.spec, from the file system.

    • When ConfigLoader loads mediation specification files from the file system, the logs include file-processing messages such as processing file or ConfigLoader Load Completed From Files.

When you deploy oc-cn-ece-helm-chart with persistence enabled, the Helm chart creates the schema user if it does not already exist, creates the ECE tables and indexes, and runs the required stored procedures.

Rotating Persistence Database Passwords

In an ECE cloud native deployment, you can rotate the Oracle persistence database password without restarting the ECE pods.

When you change a persistence database password, update the corresponding PERSISTENCEDATABASEPASSWORD value for the affected schema in the Kubernetes secret. Apply the updated Helm configuration so that Kubernetes makes the new secret value available to the running ECE pod.

The ECE wallet sidecar monitors the mounted secret data. When it detects a change to a mapped persistence database password, it updates the corresponding credential in the pod-local ECE wallet.

ECE refreshes the password after a connection failure when the affected pool next needs to establish or validate a database connection. The refresh does not occur immediately when the wallet sidecar updates the wallet.

If ECE cannot retrieve a valid password from the wallet, it leaves the existing connection-pool password unchanged. If the connection retry fails, ECE reports the connection failure through the existing error path. ECE logs the initial connection failure, the password-refresh status, and the retry result without logging password values or Kubernetes secret contents.

Note:

Persistence upgrade jobs do not use the runtime password-refresh mechanism. These jobs run as startup or Helm job flows and use the database credentials in the Kubernetes secret for each schema. Before you run an upgrade that includes persistence jobs, ensure that each schema-specific database password matches the corresponding Kubernetes secret.

To view the ECE persistence deployment logs, run the following command:

kubectl logs -f EcePersistenceJobPod -n BrmNameSpace

where:

  • EcePersistenceJobPod is the name of the pod in which ece-persistence-job runs.

  • BrmNameSpace is the namespace that contains the BRM cloud native deployment.

Avoiding Missing Pricing Data During ECE Cold Restarts

Use this procedure in a BRM cloud native deployment when ECE persistence is enabled and you perform an ECE cold restart with pricingLoadFromPersistence set to false.

In this configuration, ECE does not load pricing data from the persistence database. Instead, ECE expects PDC to republish the pricing data. Ensure that PDC republishes the pricing data and that the data is available on all ECE nodes before Customer Updater loads customer data from BRM.

If Customer Updater loads customer data before pricing data is available on all ECE nodes, ECE might skip product data for some service types. Subsequent account, product, discount, or lifecycle updates for affected subscribers can then fail because ECE cannot find valid product or rating-context data.

If this condition occurs, you can use an incremental customer load to restore the missing customer data. To avoid the condition, publish the pricing data and verify that all ECE nodes have the data before Customer Updater begins loading customer data.

Note:

Validate the cold restart procedure in a preproduction environment before you use it in production.

To ensure that all pricing data is available after an ECE cold restart:

  1. In the override-values.yaml file for oc-cn-ece-helm-chart, verify that ECE persistence is enabled.

  2. Set pricingLoadFromPersistence to false.

  3. If Customer Updater must reload all customer data from BRM when ECE starts, set customerLoadFromPersistence to false.

    Note:

    Do not manually delete data from the ECE persistence tables as part of this procedure.

  4. If you use a custom ECE startup script or another startup process, configure it so that Customer Updater loads customer data only after PDC publishes its data to ECE and pricing data is available on all ECE nodes.

    Use a readiness check when one is available. If you use a delay instead, validate the delay in a preproduction environment and set it according to the size of your pricing data and ECE cluster.

  5. Start ECE and verify that Pricing Updater starts and is ready to receive pricing data from PDC.

  6. Use ImportExportPricing to publish PDC data to ECE as separate operations in the following order:

    • metadata

    • config

    • pricing

    If your ECE startup process invokes the utility directly, run:

    ./ImportExportPricing -publish metadata -target ece
./ImportExportPricing -publish config -target ece
./ImportExportPricing -publish pricing -target ece

    For information about running PDC utilities in a BRM cloud native environment, see "About Running the PDC Utilities". For information about the utility options, see "ImportExportPricing" in PDC Creating Product Offerings.

    Note:

    Run separate metadata, config, and pricing publish operations. Do not use a single publish operation with all.

  7. After the publish operations complete and the readiness check or validated delay confirms that pricing data is available, allow Customer Updater to load customer data.

    If your startup process controls Customer Updater, verify that the configured sequence enforces this order.

  8. Confirm that ECE reaches the USAGE_PROCESSING state.

  9. Verify that the ECE customer cache contains the active services and product data from BRM. Review the Pricing Updater and Customer Updater logs and verify that ECE does not skip service types or product data while loading customer data.

Re-Creating the ECE Schema After Deployment

To re-create the ECE schema, a table, or an index after you deploy the ECE Helm chart:

  1. Delete the ECE Helm chart.

  2. Delete the existing ece-persistence-job from your system: 

    kubectl delete job ece-persistence-job -n BrmNameSpace

  3. Install the Helm chart again:

    helm install ece --namespace BrmNameSpace oc-cn-ece-helm-chart [--no-hooks]

    Note:

    Include --no-hooks only when the persistence database already contains everything that ECE requires.

Loading Only Partial Data into ECE Cache

You can optionally configure ECE to load only partial data from the persistence database into the ECE cache. In this case, the initial load of data into the ECE cache includes data only up to a specified minimum amount (back-low-limit). If the data required for processing a usage request is not available in the ECE cache, ECE loads that data into the ECE cache from the persistence database and evicts some other data from the ECE cache. This ensures that the maximum limit (back-high-limit) is not exceeded. Later, when you restart the ECE system, ECE loads the most recently used data into the ECE cache.

For more information, see "Enabling Partial Loading of Data" in BRM System Administrator's Guide.

To load only partial data into the ECE cache, set these parameters for the charging-cache-config-persistence.xml file in the oc-cn-ece-helm-chart/templates/charging-settings.yaml ConfigMap:

  • back-high-limit: The maximum amount of data that can be loaded into the ECE cache.

  • back-low-limit: The minimum amount of data that can be loaded or reloaded into the ECE cache from the persistence database.

Incremental Customer Loading in ECE Cache

By default, the customerupdater pod loads all customer data from the BRM database into the ECE cache at startup, but you can configure the pod to load customer data incrementally.

To incrementally load customer data into the ECE cache:

  1. Configure the customerupdater pod to load only an initial set of customers into the ECE cache and bring ECE to the UsageProcessing state by setting these keys in the override-values.yaml file for oc-cn-ece-helm-chart:

    • job.customerloader.runjob: Set this to false.

    • charging.incrementalCustomerLoad: Set this to true.

    • migration.loader.initialCustomerLoadFilterQuery: Set this to a query such as "and ROWNUM <= 1" to load one customer.

  2. Install the ECE Helm chart.

  3. Load the remaining customers incrementally into the ECE cache by setting these keys in the override-values.yaml file for oc-cn-ece-helm-chart:

    • job.customerloader.runjob: Set this to true.

    • job.customerloader.command: Set this to -incremental customer_updater_schema_name, where customer_updater_schema_name is the schema name specified for the customerupdater pod.

    • migration.loader.incrementalCustomerLoadFilterQuery: Set this to a query such as "and POID_ID0 NOT IN (select POID_ID0 from ACCOUNT_T where POID_ID0 <> 1 and ROWNUM <= 1)" to load remaining customers.

    • migration.loader.customerGroupList: In active-active deployments, set this to a comma-separated list of the customer group names to load, such as "customergroup1,customergroup2". The names must match the customerGroupConfigurations entries in the same file.

    • migration.loader.incrementalCustomerLoadFilterQuery: If this query references ACCOUNT_T, qualify POID_ID0 as account_t.poid_id0. For example, use: 
      and account_t.poid_id0 NOT IN (select account_t.poid_id0 from account_t where account_t.poid_id0 <> 1 and ROWNUM <= 1)"

  4. Perform a Helm upgrade by running this command:

    helm upgrade EceReleaseName oc-cn-ece-helm-chart --values OverrideValuesFile --namespace BrmNameSpace

    where EceReleaseName is the release name for oc-cn-ece-helm-chart and is used to track this installation instance, and OverrideValuesFile is the path to the YAML file that overrides the default configurations in the chart's values.yaml file.