This section describes information about the OFSCA metrics.
OFSCA's segment strength score is a metric that measures the transaction monitoring controls' effectiveness in monitoring a customer segment. The score is determined based on the following steps.
This metric is calculated by the following:
1. The '% transferred' is estimated, which reflects the percentage of the target amount that an agent can transfer from the source account to the target account before the first alert is triggered.
2. The final metric value is calculated as 100 - '% transferred'.
The metric is computed over multiple episodes sampled from the trained agent. 95% confidence intervals for this metric are also computed from these episodes. The closer the value is to 100%, the better the transaction monitoring system performs for a given customer segment, as the agent's ability to transfer a high% of the target amount is limited. Conversely, a value closer to 0 indicates the system is not performing optimally for this segment, as the agent can transfer a high% of the target amount before the first alert is triggered.
This metric is reliable only if the experiment's success (i.e., the agent has been trained successfully), signifying the agent's successful training. However, this metric should not ascertain any decisive conclusions if the agent fails to converge. In such cases, OFSCA generates an error message indicating the unsuccessful experiment.
The system strength score gives a consolidated view of the performance of the entire transaction monitoring system.
This metric is computed by taking a simple average of the segment strength scores of all of the institution’s customer segments.
The variance for each segment strength score is also aggregated to produce a confidence interval for this metric.
As this metric is an average, it might obscure the poor performance of one or more segments. Even if the system strength score is high, monitoring the individual segment strength scores is important.
The Segment Performance in metric captures the efficacy of the System in detecting various Typologies, for example, Human Trafficking. A high value of the performance metric indicates that the scenarios deployed to combat Typologies (Human Trafficking) offer significant resistance to the agent by alerting it as it attempts to move money through your institution.
OFSCA calculates the segment performance for Red Flag Coverage by the following:
1. Simulating patterns depicting Human Trafficking cases.
2. Estimating the percentage of episodes where the HT Agent scenarios alerted.
If the scenarios did not alert in the majority of the simulated episodes, it means that the System is unable to resist the agent and has low efficacy.
A value close to 100 means the System offers high Coverage in detecting the Human Trafficking pattern. A value close to 0 means the System has very low efficacy. Tuning the existing HT Agent scenarios or deploying more can improve the performance of the TMS for the segment in question.
The scenario performance metric captures which scenarios offer the most resistance to an intelligent adversarial agent. A high value of the performance metric indicates that the scenario offers significant resistance to the agent by alerting on it as it attempted to move money through your institution.
OFSCA calculates the performance of a scenario by the following:
1. Sampling episodes from the trained agent’s policy.
2. Estimating the percentage of episodes where the scenario alerted.
If a scenario did not alert in the majority of the simulated episodes, it means that the scenario is unable to resist the agent and has low efficacy.
A value close to 100 means this scenario has high efficacy and offers very high resistance to the agent. A vale close to 0 means the scenario has very low efficacy. Tuning a low performing scenario can lead to an improvement in the performance of the TMS for the segment in question.
95% confidence intervals are also computed for this metric.
The account vulnerability metric captures which account types are most liable to being abused by an intelligent agent to move money through your financial system. A high value for this metric indicates that this account type was the agent’s preferred account when moving money through your Institution.
OFSCA calculates the vulnerability of an account by the following:
1. Sampling episodes from the trained agent’s policy.
2. Estimating the funds that flowed through each account type. For example, if $100 was credited into an account and debited from the account, the funds that flowed through that account were $100. If only $50 was debited, only $50 flowed through that account.
3. Normalizing this across all account types.
An account type with a high value for this metric is preferred by the agent over an account type with a lower value of this metric. Enhancing controls that monitor a vulnerable account type can improve the performance of the TMS for the segment in question.
1. Currently, any funds that flow through an account are attributed to that account even if those funds did not reach the destination account. This could lead to the vulnerability of an account type being inflated in a given episode.
However, since the metric is computed by averaging across multiple episodes, this should not have a bearing on the final metric.
2. If two are more account types (e.g,. BRK and RBK) are highly vulnerable, then the agent will break ties randomly and will assign a high vulnerability score to one of these account types while assigning a lower vulnerability score to others. If the overall segment score does not improve significantly even after remediating the account type with the highest vulnerability score (e.g,. BRK), this could be because other account types continue to be vulnerable. Once an experiment to address the most vulnerable account type (BRK) has been run and accepted, the segment dashboard will update to now indicate that the second account type (RBK) is most vulnerable. You might have to run an experiment to address monitoring gaps for this second account type (RBK) before overall segment score improves.
The channel vulnerability metric captures which channels are most liable to being abused by an intelligent agent to move money through your financial system. A high value for this metric indicates that this channel was the agent’s preferred instrument for transferring money through your institution.
OFSCA calculates the vulnerability of the channel by the following:
1. Sampling episodes from the trained agent’s policy.
2. Estimating the funds that were transacted using each channel. For example, if A transferred $100 to B using wires and B transferred $50 to C using MI. Funds attributed to wire = 100 and funds attributed to MI = $50.
3. Normalize this across all channel types.
A channel with a high value for this metric is preferred by the agent over a channel with a lower value for this metric. Enhancing controls that monitor a vulnerable channel can improve the performance of the TMS for the segment in question.
1. Currently, any funds that are transferred using a channel are attributed to that channel for computing the vulnerability metric, even if those funds did not reach the destination account. This could lead to the vulnerability of a channel type being inflated in a given episode.
However, since the metric is computed by averaging across multiple episodes, this should not have a bearing on the final metric.
2. If two are more channels (e.g., Wire and MI) are highly vulnerable, then the agent will break ties randomly and will assign a high vulnerability score to one of these channels while assigning a lower vulnerability score to others. If the overall segment score does not improve significantly even after remediating the channel with the highest vulnerability score (e.g., Wire), this could be because other channels continue to be vulnerable. Once an experiment to address the most vulnerable channel has been run and accepted, the segment dashboard will update to now indicate that the second channel (MI) is most vulnerable. You might have to run an experiment to address monitoring gaps for this second channel (MI) before overall segment score improves.