Preventing Gaps in Data

How do the derivation algorithms know what data needs to be derived? A simple answer is any data received since the last time data was derived should be included in the new derivation. This answer sounds good, but it's not as simple as that. Although we know the date and time of the last derivation, we don't know the date range of the data that was derived. A user may ask for data to be derived only through a certain date, even though data exists after that date. The next time data derivation runs, it may need to include the data that was after the cutoff for the previous derivation.

To illustrate this point, let's look at an example. Assume that a derivation algorithm takes a source curve and applies a value from a bill factor to arrive at the new curve. In the illustration below, data for the source curve exists up to 29 th of October 2000 at 4 a.m. When deriving new data, the user enters a cutoff date of the 29 th of October. The time used is the Cutoff Time on the service agreement, which in this case is 2 a.m. Refer to Start and End Times for Billing for more information.

The user runs the data derivation on the 31 st of October at 10:02. That date and time are the Set Date /Time for the newly created data set.

The next time data derivation runs, it is not possible for the algorithm to simply derive data that has been received since the last time data derivation was run. In the above example, only data received after the 31 st of October will be derived. However, the last two intervals for the source data received on the 30 th of October also need to be included.

The following diagram illustrates how the next derivation should behave.

Because users have control over what data may be derived (using the cutoff date), the algorithms cannot rely on data received since the last derivation.

To be safe and ensure that no gaps exist in the derived data, the algorithms should look for each interval from either the start of the SA or the start of the SA/Profile relationship through to the cutoff date/time to determine if new data should be derived. However, for performance reasons (and practical reasons) the algorithms supplied with the base product only look for new data to derive starting 45 days prior to the cutoff date. This is a soft parameter to the algorithms called Number of Days to Process and may be changed. Implementation specific algorithms may choose not to follow this example, but you must be sure to consider the performance implications.