8.1 Segment Strength Score

OFSCA's segment strength score is a metric that measures the transaction monitoring controls' effectiveness in monitoring a customer segment. The score is determined based on the following steps. This metric is calculated by the following:

1. The '% transferred' is estimated, which reflects the percentage of the target amount that an agent can transfer from the source account to the target account before the first alert is triggered.
2. The final metric value is calculated as 100 - '% transferred'.

The metric is computed over multiple episodes sampled from the trained agent. 95% confidence intervals for this metric are also computed from these episodes. The closer the value is to 100%, the better the transaction monitoring system performs for a given customer segment, as the agent's ability to transfer a high% of the target amount is limited. Conversely, a value closer to 0 indicates the system is not performing optimally for this segment, as the agent can transfer a high% of the target amount before the first alert is triggered.

Limitations

This metric is reliable only if the experiment is a success (that is, the agent has been trained successfully), signifying the agent's successful training. However, this metric should not ascertain any decisive conclusions if the agent fails to converge. In such cases, OFSCA generates an error message indicating the unsuccessful experiment.