ADI was engaged by a community bank client to help refine its AML transaction monitoring system, with a goal of improving effectiveness and efficiency in generating alerts of suspicious activity. ADI executed a multi-phase analysis plan to develop, test and implement changes to the system that enhanced the performance of the AML monitoring system, satisfying the bank’s regulators.
Assessing the Challenge
Our client was undergoing a significant compliance effort to satisfy the requirements of an agreement with its regulator to improve its BSA/AML compliance program. One of the strategies the bank’s leadership pursued was to optimize the performance of its AML transaction monitoring system.
The bank had been using popular, off-the-shelf BSA/AML compliance software that automatically generated monthly alerts on customer activity for review by bank analysts. Management was concerned that the worklist definitions, which had changed very little from the software’s default definitions, were not as effective as they needed to be. A significant volume of alerts generated each month proved not to be suspicious and a substantial volume of clearly suspicious cases were identified via sources other than the automated monitoring system. Based on these concerns, the Bank engaged ADI to make the system more effective. For ADI, this meant accomplishing two potentially conflicting objectives:
- Improve worklist effectiveness (i.e., capture more cases of suspicious activity); and
- Improve worklist efficiency (i.e., reduce the number of false positive alerts).
Designing the Approach
ADI developed a three-phase plan that incorporated a variety of statistical and other analytical methods to optimize the bank’s AML monitoring system and help the bank achieve its objectives for the project.
In the exploratory phase (Phase 1), ADI consultants met with bank staff to understand the monitoring system and the variables that, from the perspective of bank staff, drove identification of suspicious activities. These meetings were followed by decision tree and regression analyses on a sample of historical worklist data and systematically available customer data elements, to identify the variables that were most highly correlated with suspicious activity. In addition, we created test worklists to evaluate the effect of changing specific worklist settings, such as the length of the historical transaction period (e.g., three months, six months, etc.), in generating customer alerts. The findings from this phase were the basis for identifying a general framework for the new scheme of monthly worklists that ADI would develop.
In the analysis phase (Phase 2), ADI collected from the monitoring system data consisting of monthly customer transaction activity, escalated cases for review, and Suspicious Activity Reports (SARs) for the most recent calendar year. We organized the data according to the developmental scheme mentioned above. Once organized, we ran simulations of randomly-generated settings on each worklist in the developmental scheme. Then, we evaluated the results of the simulations to identify, for each worklist, the combination of optimal settings that balanced the objectives of enhancing effectiveness and efficiency.
In the validation phase (Phase 3), ADI identified the optimal settings from Phase 2 and used them to develop prototype worklists. We implemented these prototype worklists contemporaneously with the worklists already in production within the monitoring system. We conducted a differential analysis comparing the alerts, escalated cases and filed SARs between the production worklists (i.e., the control) and the prototype worklists (i.e., the test). We did additional tuning of the worklist settings to close any material gaps in escalated cases and SARs between the control and test worklists. Finally, we analyzed supplementary factors to gain greater insight into the drivers of the differential outcomes. For example, we evaluated the outcome of alerts assigned to each AML analyst to determine if there were significant differences between analysts’ propensities to escalate an alert.
Implementing the Solution
Findings during the initial exploratory phase supported a new worklist scheme, segmenting customers by risk, using the bank’s new risk scoring system, and type. The analysis showed that this kind of scheme would better target higher-risk customers who were more likely to be identified for suspicious activity than lower-risk customers. Based on this framework, the simulations conducted in the second phase identified optimal settings for each worklist in the developmental scheme. The resulting worklist settings had the effect of capturing more high-risk customers and fewer low-risk customers. The latter had been the largest source of false positive alerts in the production worklists over the historical evaluation period.
The validation phase proved to be a critical component of the overall effort. The use of historical data to generate the initial prototype worklist settings raised the possibility of significant bias. Employee turnover, changes in policies, changes in training programs and many other variables introduced a level of uncertainty in applying results from historical and potentially subjective outcomes to the present environment. The differential analysis ensured that the final worklist settings would satisfy the contemporary dynamics of the monitoring program.
For example, the differential analysis uncovered a material gap between the production and prototype worklists in escalating cases from consumer cash activity. As a result, we tuned the prototype worklist to ensure that appropriate coverage would be retained within the developmental scheme. Insights from this phase of the analysis also underscored opportunities for training and workflow management to help with consistency in escalating alerts as cases for additional review.
After three months of differential analysis, the bank replaced the existing production worklists with the prototype worklists. The results of the final worklist definitions for this evaluation period showed:
- Increases of 21 percent and 14 percent in the count of incremental escalated cases and filed SARs, respectively; and
- Improvements of 10 percent and 4 percent in the conversion rate of alerts to escalated cases and filed SARs, respectively.
The final worklists achieved the objectives of improving the effectiveness and efficiency of the bank’s AML transaction monitoring system, yet our analysis identified more optimization opportunities beyond the scope of the project. ADI discussed additional recommendations with the client, including implementing “whitelisting” procedures and a formal software tuning program, as ways to maintain the monitoring system’s efficiency and effectiveness. In the end, the bank’s regulators received the new worklists as a highly effective improvement of the bank’s overall remediation efforts.