8.4 Scenario Performance
Enter a short description hereThe scenario performance metric captures which scenarios offer the most resistance to an intelligent adversarial agent. A high value of the performance metric indicates that the scenario offers significant resistance to the agent by alerting on it as it attempted to move money through your institution.
OFSCA calculates the performance of a scenario by the following:
- Sampling episodes from the trained agent’s policy.
- Estimating the percentage of episodes where the scenario alerted.
If a scenario did not alert in the majority of the simulated episodes, it means that the scenario is unable to resist the agent and has low efficacy.
A value close to 100 means this scenario has high efficacy and offers very high resistance to the agent. A vale close to 0 means the scenario has very low efficacy.
Tuning a low performing scenario can lead to an improvement in the performance of the TMS for the segment in question. 95% confidence intervals are also computed for this metric.