Basic Concepts

This chapter introduces the basic concepts involved with configuring the analytical engine.

This chapter covers the following topics:

Overview of Forecasting

The Analytical Engine generates a forecast that considers the historical demand and the causal factors.

In this process, the Analytical Engine calculates a set of coefficients that describe how each causal factor affects demand for each item-location combination, over time. The Analytical Engine then uses those coefficients, along with future values for the causal factors, to determine the forecast.

You do not see or work with the coefficients directly, but you may find it helpful to see the general equation to which they apply:

D = constant + A1*CF1 + A2*CF2 + A3*CF3 + ...

Where:

Demantra uses an equation like this for each combination. The Analytical Engine solves all the equations simultaneously and calculates the coefficients, which it then uses to generate the forecast.

After the forecast is generated the following information may be available:

Causal Factors

Causal factors provide information about historical events that are expected to recur in the future. Causal factors cause demand to deviate from a trend. More specifically, a causal factor is a time-varying quantity (such as price, season, or day of the week) that affects demand. Demantra requires historical data for causal factors, as well as future data that describes expected occurrences that will affect demand.

Note: The Analytical Engine uses multiple theoretical models, and not all of them consider causal factors; see "Forecasting Models and the Engine Flow".

Types of Causal Factors

Demantra uses the following general types of causal factors:

Base Causal Factors

Demantra provides the following base causal factors. Except for Price, these are all global causals; Price is local:

For these causal factors (except Price), Demantra provides data (for many years in the future) and the correct configuration. You should not edit or delete these causal factors. In the case of Price, you need to make sure that the sales_data table contains the price information that this causal factor uses.

Data and Configuration Details

Demantra requires the following information for each causal factor:

For reference, the following table summarizes where this information is stored:

Causal factor type Location of data Configuration details
Global factors Column in Inputs table Causal Factors screen of the Forecast Tree Editor
Local causal factors other than activities Column in sales_data table or SQL expression that aggregates data from that table  
Activities Column in sales_data table  
Promotional causal factors (For PE mode only) Aggregation function retrieves data from the promotion_data and promotion tables Promotional Causal Factors screen of the Forecast Tree Editor

Activities and Activity Shape Modeling

The Demantra activity shape modeling feature helps you easily reapply a demand profile that has a distinct shape over time. For any causal factor, Demantra requires past and future data. In the case of causal factors such as price and seasons, it is a simple process to obtain and load the data. Other causal factors are more difficult to describe mathematically. For example, when you run a promotional activity on a product, you may see a demand curve like the following:

the picture is described in the document text

If you plan a future activity that is similar to this historic activity, you would expect it to create similar demand. In general, shape modeling lets you do the following:

Demantra internally represents the shape as a linear combination of as many as eight Oracle proprietary shapes. Then the Analytical Engine automatically uses this demand shape along with all the other data in the system to determine the forecast.

By default, the Analytical Engine averages the most recent data for a given shape with the stored information about that shape, which is an average of all the past observations of this shape. Users can control this, by forcing the Analytical Engine to rescale the generated shape to align with the recent data. Specifically, the user can indicate the number of buckets for which the shape alignment should occur, starting with the beginning of the shape. Typically the user specifies either 0 (the default) or the length of the shape (to realign the entire shape).

Note: Shape modeling capabilities are different in the two engine modes:

See "Engine Modes: DP and PE".

See also

"Configuring Causal Factors"

"Configuring Promotions and Promotional Causal Factors" (PE only)

Promotions (PE Mode Only)

A promotion is an occurrence that starts at a specific date, has a certain duration, and has a certain time-varying affect on sales. Specifically, within Promotion Effectiveness, a promotion is associated with one or more item-location combinations (at any aggregation level) for a given time bucket or buckets. A given combination can have multiple promotions at any given time bucket.

As with sales data, promotion data can be imported. Depending on how your system is configured, Promotion Effectiveness may continue to import new promotions or users might create promotions within the Promotion Effectiveness user interface. Promotion Effectiveness displays promotions in the Activity Browser in the worksheets; here users create, edit, and remove promotions.

Promotion Attributes

The Analytical Engine does not use the promotions directly. Rather it uses the promotion attributes, such as discount amount, display type, and so on, each of which can have a different effect on demand. The Analytical Engine converts the values of the promotion attributes to promotional causal factors.

During implementation, you specify the attributes that promotions can have, and you specify rules for converting attribute values into causal factors. When users create promotions within Promotion Effectiveness, they specify values for these attributes.

Promotion Dates

Promotion Effectiveness assumes that a promotion has an effect between its start and end dates, as provided to Demantra, but typically the promotion has an actual effect in a slightly different span of time, as in the following example:

the picture is described in the document text

There is often a lag between the demand and the promotion associated with that demand. Typically this lag is larger with order data than with point of sale (POS) data, because retailers place orders further in advance. But there is often a lag even with POS data because customers know about an upcoming promotion and often delay normal purchases until the promotion occurs.

Accordingly, Promotion Effectiveness supports a couple of adjustments:

Promotion Hierarchy

For the benefit of users who are creating or managing promotions, you can provide a hierarchy that helps the users group the promotions. Then, within a worksheet, the Activity Browser can display that hierarchy, as in the following example:

the picture is described in the document text

The Analytical Engine ignores the hierarchy itself. For the engine, the important consideration is the promotion attributes, as noted earlier.

Promotions and Promotion Shape Modeling

In addition to performing shape modeling for activities, the Promotion Effectiveness supports shape modeling for promotions. Specifically, you enable shape modeling for individual promotional causal factors, as needed.

As with ordinary activity shape modeling, Demantra internally represents the shape as a linear combination of the shapes. Then the Analytical Engine automatically uses this demand shape along with all the other data in the system to determine the forecast.

Forecasting Models and the Engine Flow

The Analytical Engine uses a set of theoretical models, each of which evaluates some or all of the data. Most, but not all, of these models use causal factors. The models are documented in "Theoretical Engine Models".

The Analytical Engine follows a specific process of examining the data, checking for outliers and so on, evaluating the usefulness of each theoretical model, and generating the forecast. This process is described in detail in The Forecasting Process.

Demantra supports multiple analytical engine profiles. These engine profiles can be configured to generate forecasts for different scenarios such as service parts planning. These engine profiles are described in detail in Engine Profiles.

Demantra provides parameters to control both the theoretical models and the overall engine flow. See Tuning the Analytical Engine; only advanced users should adjust these parameters.

The Forecast Tree

In general, forecasting is most accurate when it can be performed at the lowest possible allowed aggregation level. However, sometimes there is not enough data at that level for all combinations. For those combinations, the Analytical Engine aggregates the data to a higher level and tries to generate a forecast there. The purpose of the forecast tree is to organize data for this process.

Note: The Analytical Engine also considers flags on different combinations; see "Combination-Specific Settings".

A forecast tree is associated with each of the configured batch engine profiles, thereby providing unique forecasting results for different forecast scenarios such as service parts planning.

As noted in "Levels", you define aggregation levels for use in worksheets. You use some of these levels to build the forecast tree. For PE mode, you also use the forecast tree to define the influence relationships.

Basics

Whenever the Analytical Engine generates a forecast at an aggregate level, it automatically splits the forecast for the parent node across the child nodes, again using the structure of the forecast tree. The proport mechanism controls how the aggregated forecast is split. For information on tuning proport, see "Proport Mechanism".

Each node in the forecast tree aggregates both by items and by locations. The following example shows a small part of a forecast tree.

the picture is described in the document text

The bold boxes show the nodes at which the Analytical Engine is forecasting.

Accuracy Metrics for Forecasts

While generating the forecasts, the analytical engine also generates the accuracy metrics for the forecast to provide information regarding the quality of the forecast. The analytical engine generates the quality measures for forecasted combinations based on analysis of past forecasts, as the quality of a forecast is largely indeterminable without performing sample tests on the forecast data. Accuracy metrics generated by the analytical engine are known as in-sample metrics.

In-sample accuracy metrics use the following formulas to judge the quality of the generated forecast for all spares considered in that run:

The observations made by the engine using the above-mentioned formulas are stored in the MDP_MATRIX table.

Additional out-of-sample error calculation can be done using a stand alone error calculation process. For more details refer to Out of Sample Error Calculations.

For a forecast tree, the accuracy metrics allocate down to the lowest level of the tree.

The following figure depicts a forecast tree, where each node represents aggregated data, both by items and by locations.

the picture is described in the document text

Node F has a forecast at the lowest level. Therefore, all accuracy metrics generated at node F would be assigned to the member data for node F.

Node G has a forecast at the lowest level. Therefore, all accuracy metrics generated at node G would be assigned to the member data for node G.

Node H failed at the lowest level and the forecast eventually is generated at node D. The accuracy metrics from node D should be allocated to all nodes that get a forecast from node D. Node G will get the accuracy metrics from node D, whereas node H will not receive the same from node D.

Forecast Tree Example

The following list describes a possible forecast tree.

  1. Highest level: all items and all locations, aggregated together

  2. All items-Division

  3. Brand-Division

  4. Brand-Region

  5. SKU-Region

  6. Lowest level: SKU-Store

Finding the Effects of Promotions (PE Mode Only)

For PE mode, the forecast tree must also be organized to support how the Analytical Engine identifies the effects of promotions. When you set up the forecast tree, you identify levels in the tree that the Analytical Engine can use in specific ways, as follows:

Out of Sample Error Calculations

Service Parts Forecasting makes use of both in-sample MAPE (see Accuracy Metrics for Forecasts) and out-of-sample MAPE, a calculated metric that occurs later, not during the batch engine run. SPF MAPE (out-of-sample) compares the actual sales data as it becomes available in the system to the forecast for a specified time period, and displays the difference as a percentage. This value is based on the forecast that is sent to Service Parts Planning (determined by SPF Final Forecast) and then archived in Demantra. It is a combination of the analytical forecast, the calculated forecast, and any user overrides.

Formula: sum ( absolute value | Actual Demand - Lagged Forecast | / (Actual Demand) / Number of Observations For example, a forecast is generated monthly and the total average quantity for each of the first six months of 2010 is 2000 units, resulting in forecast of January of 2000 units Actual sales for January were lower; 1634. Therefore, this series would display a value of 22.4% (the result of abs(2000-1634)/1634). Additional example: A forecast is generated weekly. The expected demand for the first week in March is 300, but the actual demand was 346. Therefore, this series would display a value of 13.3% (the result of abs(300-346)/346). If either demand or the forecast is zero (0), then the value would be 0%. If both forecast and demand are zero, then the value would be 100%.

The SPF Calc Forecast Accuracy seeded workflow is used to calculate error and variability associated with Service Parts Forecasting. This workflow can be called ad-hoc when accuracy measures should be generated. The seeded workflow is configured to aggregate information at levels Organization and Latest revision for the last four periods of history. The first three series pairs generate an accuracy measure for the final, analytical and calculated forecast streams by comparing the latest archived forecast with actual usage values. The last series pairs compare the last two archived versions of the final forecast with each other to determine forecast variability. If additional error calculation processes are required, it is recommended that additional steps be added to this workflow or separate workflows be created to call the APPROC_CALC_ACCURACY stored procedure.

APPROC_CALC_ACCURACY Stored Procedure

Error calculation is called from the procedure APPROC_CALC_ACCURACY. The procedure has two parameters as inputs defining the aggregation level at which the calculation is done. Each level defines an aggregation dimension. When calculating error for series on sales_data, one level from the item and one level from the location dimension should be specified. If the series being compared reside on a General Level data table, one of the levels specified can reside on the General Level. Levels should be specified by the internal ID of the level being referenced. The calculation start and end period are integers specifying which range periods should be aggregated together when calculating the error. These parameters are relative to the current end of history date specified by the max_sales_date parameter. A value of 0 would match this date, a value of -1 would be one period prior to this date and a value of 5 would be five periods after this date. For example, if the last four periods of history should participate in the calculations, the values -3 and 0 should be assigned to the start and end parameters respectively. The series groups define the groups of three series associated with the calculation. As defined in the previous formula, the difference between series 1 and series 2 are aggregated to the defined levels and time range and then divided by the total of series one for the same aggregation. The resulting value is written to the third series which is the output series. All three series in a group must reside on the same data model dimension, the first two residing on a data table and the third residing on the combination table associated with the two. For example if series 1 and 2 are associated with the Service Part data table, the third series must be associated with the Service Part Matrix table. Up to five series groups may be defined. Each series group can refer to a different data table and dimension.

Inputs are:

Influence and Switching Effects (PE Mode Only)

To describe how the item-location combinations affect each other, you specify the influence ranges, influence groups, competitive item groups, and competitive location groups.

Influence Ranges

When you define the forecast tree, you specify the influence ranges (IR). To do so, you specify the influence range level (IRL); each node within the IRL is an influence range.

Each influence range is a set of combinations that do not interact with combinations within any other IR. The influence ranges control how far the Analytical Engine looks for influence when promotions are run. This determines the breadth of the causality. An influence range is a set of item-location combinations that potentially interact with each other but not with combinations of other IRs. Typically each IR represents a different geographical area.

No information is available above the IRL to calculate effects of promotions. Therefore, if for certain nodes the Analytical Engine generates a forecast above the IRL, the forecast for those nodes includes only the baseline forecast.

Influence Groups

When you define the forecast tree, you also specify the influence groups (IG), which are subdivisions of the influence ranges. To do so, you specify the influence group level (IGL); each node within the IGL is an influence group.

Each influence group consists of an item group and a location group with the following behavior:

Using these definitions, the engine can calculate the following three causal factors for each lowest-level combination:

self Influence caused by promotions on this combination.
own Influence caused by other combinations within the same IG.
other Influence caused by all other combinations within the IR.

The Analytical Engine uses these causal factors internally to calculate the switching effects caused by promotions.

No information is available above the IGL to calculate switching effects. Therefore, if for certain nodes the Analytical Engine generates a forecast above the IGL, the forecast for those nodes includes only the baseline forecast and the direct effects.

Competitive Item Groups (CI) and Competitive Location Groups (CL)

Typically, you also define competitive item groups (CI) and competitive location groups (CL):

You do not define these groups directly in the forecast tree. Instead, you set them via parameters. The Analytical Engine does not aggregate data to the CI and CL, so it is not strictly necessary to make them consistent with the rest of the forecast tree; they must of course, be at a higher aggregation level than the item and location groups, respectively.

Switching Effects

A switching effect occurs when a sale for a given item-location combination affects sales for another item-location combination. Promotion Effectiveness uses the preceding classification system to describe different switching effects. Each effect is associated with relationships between one item-location combination and others.

Effect* CI item group (I) CL location group (L)
Brand switching (or category switching) different different by definition same same or different
Channel switching different different by definition different different by definition
Product switching same same same same
same different same same
same different different different by definition
Store switching same same or different same different
same same different different by definition
*Depending on how you define CI, CL, I, and L, the names of these effects may or may not be appropriate. You can rename these effects by renaming the series that display them.

Notice that if the CI and CL each have only one member, there is no competition, and the only effects that can be seen are product switching and store switching.

For simple example, consider a single store and the following item groups and competitive item groups:

the picture is described in the document text

Combination-Specific Settings

The Analytical Engine also considers specific settings that vary from combination to combination, which are stored in the mdp_matrix table and which are affected by global parameters. This section provides an overview of the key settings, which are provided to support users who want closer control over the forecast. You can create levels or series that use these settings, as needed. Not all settings are meant to be changed directly.

Fictive Status

Demantra sets a flag for each combination to indicate whether that combination is real or not. In mdp_matrix, the is_fictive flag has one of the following values:

Value General meaning
1 Combination is fictive (not real). This combination was created via Member Management.
0 Combination is real and it has non zero sales data.
2 Combination is real but all sales are zero or null.
3 Errors occurred while loading this combination.

The Analytical Engine does not use this flag directly, and users should not edit it.

Age

Each combination is either young, live, or dead, depending on the relative age of its sales data. Demantra uses two cutoff dates to determine the age of a combination:

the picture is described in the document text

The dying date is controlled by the dying_time parameter, and the mature date is controlled by the mature_age parameter. Both parameters are global.

Demantra automatically uses these cutoff dates as follows:

The parameters used to determine if a combination is active, dead or young are:

At times there is a business need to override these global definitions. Typically this would be for specific items and locations for which a more or less reactive status change is desired. For example: The majority of combinations should be deactivated if they have not sold in three months. However, seasonal items require a different setting, and should not be deactivated unless they have not sold for more than a year. These parameters are set globally, but can be overridden individually per combination.

The User-Controlled Do_Fore Flag

Demantra provides a combination-specific flag with which advanced end users can control how the Analytical Engine works on individual combinations. This flag is in the mdp_matrix table and is called do_fore. In order to enable users to set this flag, you generally create an editable series that uses this flag. Users can set this flag to any of the following values:

Value Meaning
0 The Analytical Engine will ignore this combination
1 The Analytical Engine will consider the combination. This is the default value.
2 The Analytical Engine will create a placeholder forecast for this combination that consists of zero values (which is useful if the user wants to create an override). The engine will otherwise ignore this combination. You typically use this setting for combinations created through Member Management.

The sole purpose of the do_fore flag is to give users a way to control the prediction status of the combination, as described next. The do_fore flag is not used directly by the Analytical Engine.

Prediction Status

In mdp_matrix, the prediction_status indicator of a combination instructs the Analytical Engine how to handle this combination. The following values are possible:

Value Affect on the Engine Comments
No Forecast (96) The Analytical Engine ignores this combination. For future use; this setting cannot currently be achieved or used.
Create Zero Forecast (97) The Analytical Engine creates a zero forecast but otherwise ignores the combination.  
Young (98) The Analytical Engine creates a zero forecast but otherwise ignores the combination. Sales for this combination are too new to be used for prediction.
Dead (99) The Analytical Engine creates a zero forecast but otherwise ignores the combination. Sales for this combination are not recent enough to be used for prediction.
Live or Active (1) The Analytical Engine uses this combination for forecasting.  

Demantra automatically sets the prediction_status indicator and users should not change it.

How Prediction Status Is Set for Fictive Combinations

For fictive combinations (is_fictive = 1), Demantra automatically sets the prediction status to 98.

How Prediction Status Is Set for Real Combinations

The proport mechanism considers all real combinations (is_fictive equal to 0 or 2). It automatically sets the prediction status indicator for each combination based on the the age of the combination and the do_fore setting of the combination. Specifically, it sets the prediction_status indicator as follows:

do_fore is 0 do_fore is 1 do_fore is 2
Combination is dead prediction_status is 99 prediction_status is 99 prediction_status is 97
Combination is young prediction_status is 98
Combination is live prediction_status is 1

Aggregation Flags

As noted in "The Forecast Tree", the Analytical Engine sometimes needs to aggregate lowest level data in order to create a more accurate forecast. You can control, per combination, whether this combination should be aggregated to a higher level.

Demantra provides three flags so that you can specify different rules based on the age of the combination. The flags are as follows:

Flag When used
do_aggri If combination is live
aggri_98 If combination is young
aggri_99 If combination is dead

Each of these flags (in mdp_matrix) specifies whether to aggregate data for this combination during forecasting.

Other Combination-Specific Settings

Demantra provides many parameters that control how the Analytical Engine behaves. By default, these parameters affect all the combinations in the forecast tree. Through the user interfaces, an advanced user can set analytical parameters for individual combinations if needed.

The Forecast Data

The Analytical Engine writes the current forecast to one of the following fields in sales_data: Fore_0, Fore_1, Fore_2, and so on.

Note: For PE mode, this is the baseline forecast.

The Analytical Engine cycles through these columns. Each time, it writes the current forecast into one column (overwriting the oldest forecast). The Analytical Engine then adds a row to the forecast_history table that describes this forecast and that indicates which column it is stored in.

The number of saved forecasts is specified by the active_forecasts_versions parameter.

Forecast Decomposition (PE Mode Only)

For PE mode, the Analytical Engine also populates the following database fields in promotion_data to show the effects of the promotions. These fields show the effects of a given promotion on a given combination, at a given date:

Field Purpose
fore_0_uplift Direct lift on a combination during the promotion dates, due to a promotion specifically associated with that combination.
fore_0_pre_effect Direct lift on this combination before the promotion dates, due to a promotion specifically associated with that combination.
fore_0_post_effect Direct lift on this combination after the promotion dates, due to a promotion specifically associated with that combination.
fore_0_brand Effects of brand or category switching as described in "Switching Effects".
fore_0_sw_channel Effects of channel switching on this combination, at this date, due to this promotion.
fore_0_product Effects of product switching on this combination, at this date, due to this promotion.
fore_0_store Effects of store switching on this combination, at this date, due to this promotion.

For the benefit of the users, you create series that use these data fields. Be sure to provide series names that make sense to the users and that are appropriate for the business.

Forecast Versions (for Batch Runs)

As noted earlier, Demantra keeps a number of previous forecasts (as specified by the active_forecasts_versions system parameter). The most recent batch forecast is numbered 0, the previous one is numbered 1, and so on. When the Analytical Engine generates a new forecast, it moves the previous ones to different columns in the database. See "Engine Parameters".

Each series you create is implicitly or explicitly associated with a specific forecast version or multiple forecast versions. Typically, the large majority of series are associated with the most recent forecast, but it is often useful to configure some series to capture information associated with a previous forecast, or to compare multiple forecasts.

Note: If you need to display present and past versions of other data, you can configure and run rolling data sessions, which copy data from one series to another as specified. See "Configuring Rolling Data".

For information on active_forecasts_versions, see "Non-Engine Parameters".