For a suite of test cases to provide any confidence in the correct workings of a rulebase, it is important that the majority of the rules in the project are meaningfully exercised by test cases.
For example, if a rulebase calculates an income threshold, but there is no test case that fails to meet that threshold, then a change to that threshold (for example, by inadvertently bypassing it) may not be detected until the rulebase is put into production. Only by exercising the threshold in both the positive and negative situation (by having test cases that meet the threshold and test cases that do not meet the threshold) can there be any confidence that a change to the rules has not had unintended side effects.
The Test Script Coverage report in Oracle Policy Modeling can be used to find areas of a rulebase that are not well-exercised. The next step is to construct a plausible test case that does exercise those areas.
Generate a Test Script Coverage report by going to Reports | Test Script Coverage.
By default, this report will organize the report by document, which allows you to see how well tested the rules are on a per-document basis. The colored icons represent whether a document contains any rules that do not meet the default coverage threshold of 80%.
Green tick: Indicates the rules meet the threshold.
Red arrow: Used for a rule that does not meet the threshold.
Yellow arrow: Used when a document/folder contains a rule that does not meet the threshold.
Note that the System rules mostly have no coverage at all. This is because these rules are mostly only used during a guided interactive interview, which is not verified by the test cases. The rules in those documents may, of course, be triggered when the test cases run, but their results are never compared with expected values, so the rules have no coverage.
Change Organize By Document to Organize By Goal. This shows all of the top-level goals that are tested in all the test scripts of the project, and shows how well-tested the rules of each goal are. In general, it is most important that top level rules are well covered, as these tend to indicate broad areas of functionality within the rulebase that should absolutely be verified. Lower level rules need coverage also, but exercising all combinations may not be practical for large tables whose only purpose is to reinterpret base-data provided by the user. An example of this can be seen by following the yellow coverage icons from "the applicant may be eligible for child care assistance" down to the rule "the household’s geographical area for the purpose of the Poverty Guidelines" which just maps 50 possible locations into one of three broad categories. Testing all 102 possibilities of this rule would not be as useful as ensuring that all the high level eligibility conditions are tested.
Select "the applicant may be eligible for EITC", which displays the high-level rule for this attribute.
The bold 'true' and 'false' values indicate that most conditions in the rule have been tested in both a true situation and a false situation. The rule meets the coverage threshold by having 9/10 coverage, however the final condition "the household’s annual investment income <= 3150" has only been applied when it was 'true'.
To fix this, open the AllBenefits.tsc test script file and make a copy of Case_50, call it Case_51 and then open it. Change the value of the attribute "the household’s monthly investment income" from 0 to 300. This will change the yearly investment income to $3,600, and the household will no longer eligible for EITC. The expected results in this test case will need to be updated for the test to pass, but this can be done later.
Return to the Test Script Coverage report and click the Regenerate button. The coverage for the rule "the applicant may be eligible for EITC" is now complete.
This example was simple because the change to the test case was immediately significant to the conclusion, therefore the change contributed immediately to the coverage score. Other changes may be more subtle.
For example:
In the Test Script Coverage report, expand "the applicant may be eligible for EITC", then "the monthly gross earned income is below the EITC income limit for the household size" then "the annual EITC income limit for the household size".
This rule is mostly tested, however, there is no test case for an unmarried applicant with 2 EITC qualifying children. The conditions for that situation are tested when they are false (in cases when a later row in the table is triggered instead) but never when they are true. We can also deduce from this that the income limit for that situation is not being tested.
Return to the AllBenefits.tsc file and copy Case_51 to make another new case Case_52. Open this new case. This case already has 2 EITC qualifying children, but the applicant Marge is considered to be married because her spouse Homer is in the household. Remove Homer from the household.
Regenerate the Test Script Coverage report and review the coverage for the rule for "the annual EITC income limit for the household size". Note that the coverage score has not improved, and the conditions for an unmarried applicant with 2 children is still not considered covered. To find out why, the test case must be examined more closely.
Return to Case_52 and locate the attribute "You may not be eligible for EITC" (note the use of second person). Right click and choose Show Decision. Uncheck Relevant Only and expand the decision tree to find "the annual EITC income limit for the household size".
The gray values are those that not relevant to the decision. This explains why the change has not contributed to the coverage score: the change described in the previous example (which made the household not eligible for EITC) has made the income limit irrelevant. Even though the income limit was calculated for an unmarried applicant in a 2 child household, the value was ultimately irrelevant and so those conditions are still considered to have no coverage.
Change the household’s monthly investment income back to 250. The income limit is now relevant again.
Regenerate the Test Script Coverage report and see that the rule for "the annual EITC income limit for the household size" now has a higher coverage score.
The other conditions in this rule cannot be fully covered because they use a defensive style of rule authoring and test for conditions that should never occur, or should always be true (such as testing that there are 0 children when all other numbers of children have been accounted for). For this reason, 100% coverage will rarely be possible in practice.
