Tuning the Analytical Engine

It is usually necessary to adjust some parameters to configure the Analytical Engine correctly before running it the first time. Other adjustments can be made later to optimize the behavior and performance.

This chapter covers the following topics:

• Editing Engine Parameters
• Creating or Renaming Engine Profiles
• Tuning Analytics
• Tuning Performance
• Reconfiguring the sales_data_engine Table
• Enabling Engine Models Globally
• Configuring the Engine Mode
• Advanced Analytics (Nodal Tuning)

Editing Engine Parameters

To tune the Analytical Engine, you modify values of two types of engine parameters:

• Global parameters that apply to the engine or to most or all of the forecasting models. For convenience, you define engine profiles , which are sets of engine parameters with specific values. Demantra provides some predefined profiles for different purposes, and you can define additional engine profiles, as needed.

• Parameters that apply to specific forecast models.

To edit the global engine parameters

1. Log onto the Business Modeler.

2. Click Parameters > System Parameters. The System Parameters dialog box appears.

3. Click the Engine tab.

   

4. From the Engine Profile drop-down, select the engine profile whose parameter settings you want to adjust.

5. Find the parameter of interest. The dialog box provides find, sort, and filter capabilities to help you with this. See "Engine Parameters".

6. To change the value of the parameter, click the Value field for that parameter. Type the new value or select a value from the drop-down menu.

7. Click Save to save your changes to this profile.

8. Click Close.

To edit specific model parameters

To edit most model-specific parameters, you must work directly within the Demantra database. For information on the parameters and their locations in the database, see "Engine Parameters".

See also

"Creating or Renaming Engine Profiles"

"Tuning Analytics"

"Enabling Engine Models Globally"

Creating or Renaming Engine Profiles

To create or rename an engine profile

1. Log onto the Business Modeler.

2. Click Parameters > System Parameters. The System Parameters dialog box appears.

3. Click the Engine tab.

4. Do one of the following:

   • To rename an existing profile, click the profile from the Engine Profiles list and then click Edit.

   • To create a new profile, click New.

5. Enter a (new) name for the profile.

   See also

   "Editing Engine Parameters"

   "Tuning Analytics"

6. When creating a new Engine Profile, determine whether it is to be a batch or a simulation profile. A simulation profile must be attached to a parent batch forecast, because the simulation is stored in the sim_val column matching the batch parent profile.

   For example, a simulation with the batch parent profile ID of 3 is stored in the sim_val_3 column. The internal profile ID can be found in ENGINE_PROFILES table.

7. If this is a simulation profile, select the Simulation Engine Profile check box.

8. If the new profile is a simulation profile, select the Parent batch Profile from the drop-down menu.

9. Click Ok.

Tuning Analytics

For basic parameters related to the forecast tree, see "Specifying Additional Parameters". For information on all parameters (including default values), see "Engine Parameters".

Analytical Parameters

The following parameters control analytics:

Parameter	Purpose
UseNonNegRegr	Specifies whether to allow negative coefficients. Most of the models use this parameter. In cases with multiple, possibly co-varying causal factors, the Analytical Engine sometimes finds a solution that includes a large positive coefficient for one causal and a large negative coefficient for another causal factor, so that they nearly cancel one another. Mathematically, this solution may be good. But a negative coefficient means that the demand acts in the opposite sense to the causal factor; that is, demand drops when the causal factor increases. And a negative coefficient does not make sense in the vast majority of cases. This means that it is generally good practice to disable negative coefficients.
ShapeSign	Specifies the signs for the shape causal factors when using them in non negative regression.
NumShapes	Specifies the maximum number of allowed shape causal factors for the engine to use for a given node in the forecast tree. Use an integer from 0 to 8, inclusive. This applies to activity shape modeling (rather than to promotional shape modeling).
CannibalizationIgnore	Controls whether the Analytical Engine will calculate switching effects (cannibalization). You can use this parameter to switch off that calculation in order to check that the Analytical Engine is calculating the basic lift appropriately.

Parameters Related to Promotional Causal Factors (PE Mode Only)

The following parameters are related to promotional causal factors:

Parameter	Purpose
PromotionStartDate	Earliest date for which promotion data can be considered reliable.
ShiftDynPromoDate	SQL expression that returns the number of days to add to the sales date for any given promotion; typically this is a negative number. If this expression is null, then the default promotion dates are used. Alternatively, to specify an overall shift in time for all promotions, set the ShiftPromoCausals parameter.
See "Adjusting the Promotion Dates".

Parameters Related to Validation (PE Mode Only)

The Analytical Engine applies different forecasting models to each node of the forecast tree, calculates the uplift for each node, and uses that uplift to check whether the model is appropriate for that node. If not, the model is not used for the node.

The Analytical Engine can discard a model for a given node for either of two reasons:

• The model generated an uplift that was beyond the upper allowed bound, as specified by the UpperUpliftBound parameter.

• The model generated too many exceptional uplifts. An uplift is considered "exceptional" if it exceeds the lower bound specified by the LowerUpliftBound parameter. The AllowableExceptions parameter controls how many exceptional uplifts are permitted.

Parameters Related to Output (PE Mode Only)

The following parameters control the output of Promotion Effectiveness forecast values:

Parameter	Purpose
NormalizeResults	Specifies whether to normalize the historical engine results so that the observed baseline values are preserved. If you normalize the engine results, note that the Analytical Engine writes these results to different fields in promotion_data than it does otherwise. See "Key Tables".
WriteMissingDatesUplift	Specifies whether to write uplifts for dates that are missing from sales_data. If you specify no, then the Analytical Engine writes uplifts only for dates that already have sales. However, the uplifts will not necessarily add up to the total uplift.
UpliftThresholdValue	Specifies a threshold for uplift values. If the Analytical Engine calculates uplift values below this threshold, those values are dropped rather than being written to the database.
UpliftThresholdMethod	Specifies whether the previous threshold is expressed as an absolute value or as a percentage of baseline.

See also

"Editing Engine Parameters"

"Creating or Renaming Engine Profiles"

Tuning Performance

To improve the performance of the Analytical Engine, check the settings of the following parameters. To access these parameters in Business Modeler, click Parameters > System Parameters and then click the Database tab.

Basic Engine Parameters for Performance

The following engine parameters are critical to good performance. Make sure they are set appropriately for your configuration.

Parameter	Purpose
min_fore_level	Minimum forecast level that the engine will forecast. For PE, this must be at or above the lowest promotional level (LPL). Make sure this is defined appropriately for your forecast tree.
start_new_run	Specifies whether to start a new Analytical Engine run or to perform an engine recovery. Use yes or prompt.
node_forecast_details	Specifies whether the Analytical Engine should write forecast data for each node (the NODE_FORECAST table), before splitting to lower levels. Writing to this table slows the engine, so you should switch off this option unless you have tested that the impact is acceptable.
WriteIntermediateResults	Specifies whether to enable the advanced analytics function, which is available only on the desktop. Make sure this option is off unless you have tested that it does not interfere unduly with performance.
BulkLoaderBlockSize	Specifies the minimum number of rows that Analytical Engine loads at one time, when writing to the database. The larger this is, the more quickly the data is loaded, but there is greater risk if the database connection is lost. For a high-volume system, use 20,000.
BulkLoaderEnableRecovery	Specifies whether Oracle Bulk Loader should perform recovery after a lost database connection. For a high-volume system, use 0.

Parameters That Can Speed Performance

The following parameters can help the Analytical Engine run more quickly by omitting processing steps. You should change these only if you are sure that doing so will not cause problems.

Parameter	Purpose
ForecastGenerationHorizon	Specifies what historical fit data the engine will write to the database. If this parameter is 0, the engine writes the forecast only. If this parameter is a positive integer N, the engine writes the last N historical fit values.
ResetForeVals	Controls the method of clearing current forecast values for the forecast version currently being generated. If set to Yes (default), then all combinations with prediction status of 97, 98, or 99, fore = 2, will get null forecast values and active combinations will be overwritten by the new forecast. If set to No, then the existing forecast for inactive combinations will not be cleared. If set to All, then all combinations will have their forecast cleared regardless of prediction status. Note that setting this parameter to 'All' may substantially increase engine run time. If you want to reset the forecast outside the engine date range, set this option to All. The individual engines perform this function during the run. Each engine produces a list of the inactive nodes for the branch/simulation it is processing and adds special rows for the bulk loader. Procedures ProcessTempSaleTable and ProcessTempPromoTable perform the resetting. They update the configured data table and the PROMTION_DATA table. Engine parameter DBHintInitialForeClean applies only to the functionality of Engine Manager parameter DeleteIsSelfRows.
RunInsertUnits	Specifies whether the Analytical Engine calls the INSERT_UNITS procedure at the start of an engine run. This procedure makes sure the engine has rows to write into when generating the forecast. For information on all procedures, see "Database Procedures".
BatchRunMode	Applies to PE mode, and applies to both batch run and simulation run. Specifies the kind of forecasting to do: 0=run the forecast against only the learning (estimation) 1=run the promotion forecast (the normal setting) 2=perform an estimation and promotion forecast run (fast simulation; this option uses previously cached data) For options 0 and 2, the Analytical Engine performs fewer scans. (For details on the engine flow, see "Promotion Effectiveness Engine Phases".)
align_sales_data_levels_in_loading	Specifies whether to adjust the sales_data table for direct use by the engine (instead of the sales_data_engine table). 0=no (do not adjust the sales_data table for direct use by the engine) 1=yes (adjust the sales_data table) For information on this parameter, see "Non-Engine Parameters".
start_date	Beginning of historical data used by the engine. Used together with parameter HistoryLength. If left at default 01/01/1995, may require the engine to find first period of real history in historical demand. For larger environments this can add significant time to the engine run. It is strongly recommended this parameter be reset to beginning actual date where history begins.

The engine divider uses Fast Divider functionality. The engine uses the ENGINE_BRANCH_LIST table to determine the actual branch, not the BRANCH_ID column of the configured combination table. Each time the engine processes a branch, it updates the BRANCH_ID column with the actual allocation.

Database Partitioning for the Engine

You can partition the database so that the Analytical Engine can access data more rapidly. Specifically, you can place different parts of the sales_data, mdp_matrix, and promotion_data tables on different partitions, so that each partition corresponds to a potentially different item and/or location.

The overall process is as follows:

1. Create the partitions and move rows to them as needed. This is beyond the scope of this documentation.

2. To partition only by item, choose a database column that you can use to subdivide the records by item. This column must exist in the sales_data, mdp_matrix, and (in the case of Promotion Effectiveness) promotion_data tables and must have the same name in each of these tables.

   For example, it might be suitable to partition by brand. The brand information is available in mdp_matrix as (for example) the t_ep_p2a_ep_id field. You would have to replicate this column to the sales_data and promotion_data tables as well, perhaps by a database trigger.

   Similarly, to partition only by location, choose a database column that you can use to subdivide the records by location.

   To partition by item and by location, choose a database column that you can use to subdivide the records by item and another column that subdivides them by location.

3. Set the following parameters so that the Analytical Engine can find the partition on which any combination resides:

   Parameter	Purpose
   PartitionColumnItem	Specifies the name of the column that partitions the data by item.
   PartitionColumnLoc	Specifies the name of the column that partitions the data by location.

Other Database Considerations

Pay attention to the indexes of sales_data and mdp_matrix tables.

Also, for Oracle databases, Demantra writes to multiple tablespaces, as specified during installation. The tablespace assignments are controlled by parameters, which you can edit through the Business Modeler. Make sure that these parameters refer to tablespaces within the appropriate database user, and make sure each has enough storage.

Note: Oracle recommends that you use the standard names for these tablespaces, as documented in the Oracle Demantra Installation Guide. Then it is easier for you to share your database with Oracle Support Services in case of problems.

Additional parameters control the default initial sizes and how much storage is added.

Parameter	Description
initial_param	Default initial size of system tablespaces.
next_param	Incremental amount of storage that is added to a tablespace when more space is needed.
tablespace	Tablespace used for the sales table.
indexspace	Database index space that stores the forecast table indexes, as specified during installation.
simulationspace	Tablespace used for simulation data.
simulationindexspace	Tablespace used for simulation index data.
sales_data_engine_index_space	Tablespace used for the index of sales_data_engine.
sales_data_engine_space	Tablespace used for sales_data_engine table.
*For information on these parameters, see "Non-Engine Parameters".

Reconfiguring the sales_data_engine Table

The Analytical Engine creates and uses a table (or view) called sales_data_engine. You can control how the Analytical Engine does this, in order to improve performance.

• You can adjust the sales_data table for direct use by the Analytical Engine, so that the sales_data_engine table is not needed.

• Normally, the Analytical Engine internally creates the sales_data_engine table for its own use, and creating this table can be time-consuming. You can speed up the engine by configuring it to use the sales_data table instead of the sales_data_engine table.

• Normally, when the Analytical Engine runs, it joins sales_data_engine (or its synonym) with the mdp_matrix table. This is not always necessary, and you can prevent this join to speed up the Analytical Engine.

The following table lists the key parameters and some typical settings:

Parameter	Description	Normal batch run	Normal simulation	Faster engine run*	Fast simulation*
align_sales_data_levels_in_loading**	Specifies whether to adjust the sales_data table for direct use by the engine (instead of the sales_data_engine table). 0=no (do not adjust sales_data) 1=yes	0	0	1	1
SdeCreateSwitch	Specifies the type of logic to use in order to create the sales_data_engine table. 0=Use internal engine logic 1=Use external logic, as specified by stored @ procedures, create_process_temp_table, create_object and drop_object @ procedures. These procedures may be modified by consultants.	0	0	1	1
SdeAnalyzeSwitch	Specifies the type of logic to use in order to analyze the sales_data_engine table. 0=no. The engine assumes analyze is already performed and creates indexes as part of external procedures. 1=yes. The engine analyzes and creates indexes. This is the default value.	1	1	0	0
SdeCreateJoin	Specifies whether the Analytical Engine should join sales_data_engine (or its synonym) and mdp_matrix during its run. 0=no (do not join these tables) 1=yes (join these tables; this is the default)	0	0	0	1
*See "Additional Steps". Also note that fast simulation forecasts future uplift only. **For information on this parameter, see "Non-Engine Parameters".

Additional Steps

1. Configure the forecast tree as you normally would. See "Configuring the Forecast Tree".

2. In the database, create a synonym for sales_data. The name of synonym should be sales_data_engine or whatever synonym you plan to use.

3. Rewrite the following database procedures as needed:

   • create_process_temp_table

   • create_object

   • drop_object

4. Consult Demantra for assistance.

5. Test that you have configured the engine correctly.

   1. Add new records to sales_data, in any of the following ways: by loading via the Data Model Wizard, by running integration, or by chaining.

   2. Run the engine.

   3. Check the sales_data table and make sure of the following:

      • This table should have a column for every level in the forecast tree,

      • This table should have a column named do_aggri.

      • This table should have should include non-null data in these columns for at least some of the records.

Enabling Engine Models Globally

Demantra provides a set of theoretical engine models that the Analytical Engine uses when it creates a forecast. Usually you do not make changes, but you can specify which models to use, as well as set basic parameters for each model.

Caution: Only advanced users should make these changes.

When the Analytical Engine runs, it may use a subset of these models on any particular combination. The engine tests each model for applicability; see "The Forecasting Process".

Note: Optimization can only use linear generalized coefficients

To enable models for the Analytical Engine to use

1. Log onto the Business Modeler.

2. Click Engine > Model Library.

   The following dialog box appears.

   

3. Select the batch engine profile to be associated with the model library configurations.

4. For each model, do the following:

   • To enable the Analytical Engine to use this model, make sure the Active check box is checked. For details on these models, see "Theoretical Engine Models". Note that not all models are supported with any given Analytical Engine.

   • The other two settings control the minimum and maximum number of non zero observations that a combination must have in order for the Analytical Engine to consider using this model for this combination. To specify these values, type integers into the Min Len and Max Len fields.

     Note: Min Len must be equal to or greater than the number of causal factors in the forecast, except for the HOLT and FCROST models, which do not use causal factors.

5. Click Save and then click Close.

Configuring the Engine Mode

Oracle provides two different modes of the Analytical Engine:

• The DP mode is for use with Demand Planner or other planning products.

• The PE mode is for use with Promotion Effectiveness.

To specify the engine mode

The RUNMODE parameter specifies the mode of the Analytical Engine to use:

• Use 1 to specify the PE mode.

• Use 0 to specify the DP mode.

If you use this setting, also be sure that you have defined the forecast tree appropriately. In particular, make sure that the LPL (PROMO_AGGR_LEVEL) is the same as the minimum forecast level. To set this, use the forecast tree editor in the Business Modeler.

See also

"Troubleshooting"

Advanced Analytics (Nodal Tuning)

Normally, the Analytical Engine uses the same options for every node in the forecast tree, but you can make certain adjustments for individual nodes, if necessary. This task is recommended only for advanced users in conjunction with Oracle Support.

Of the models you specify for a given node, when the Analytical Engine runs, it may use a subset of these models, as described in "The Forecasting Process". The Analytical Engine indicates (in the models column of mdp_matrix) the models that it used.

To enable advanced analytics

1. Set the usemodelspernode and UseParamsPerNode parameters to yes.

2. Then for each node of the forecast tree, you can specify engine models and engine parameters for different nodes in the forecast tree. To do so, you use the Analytics window.