About

1. The shift

AI agents are entering compliance operations.

Financial institutions are starting to use AI agents for compliance work: KYC reviews, transaction monitoring alerts, sanctions screening alerts and fraud cases. This work follows a risk-based approach. Each institution sets its risk appetite, writes it into its policies and expects every decision to follow those policies. Each decision must also stand up to quality assurance review, internal audit and regulatory examination.

Before an institution gives an agent part of this work, it needs evidence that the agent meets this standard. To produce that evidence, AI labs and other teams that build agents must train and evaluate their agents on realistic compliance cases. We started FinCrime Atlas to build these environments, so that these teams can train their agents to this standard and institutions can see how the agents perform against it.

2. The gap

No one can yet show that an agent meets the standard.

Three gaps stop teams from producing that evidence today:

• Realistic case data. Customer confidentiality and the legal restrictions on SAR disclosure keep real case files inside each institution. These teams have no realistic cases to train and evaluate their agents on.
• Tests of the work. Most general evaluations test knowledge of financial crime concepts. Compliance work also requires an agent to investigate a case: retrieve the records, find the gaps, consider the legitimate explanations, apply the policy and document the decision, including the decision to close a case.
• A practitioner standard. Most evaluations grade only the final answer. Quality assurance reviewers check the rationale as closely as the decision. The rationale must trace to the records and follow the applicable policies.

3. Our approach

Realistic case queues, graded to a QA standard.

We build four environments: KYC, transaction monitoring, sanctions screening and fraud prevention. Each environment represents the case queue of one compliance function at an institution, with its own written policies. Our environments address each gap:

• We build each case from the patterns in available public sources. Each case contains dated records, and the relevant activity sits within normal account activity, as it does in a real case file. All names, identifiers and records are synthetic, so these teams can use the cases without the confidentiality restrictions on real case files.
• Each case is a multi-turn agent task. The agent calls tools to search the institution’s systems, requests the information that it needs, closes or escalates the case and submits a decision record.
• We grade the investigation and the decision record to the standard of a quality assurance review. A practitioner reviewer checks each answer key and the grading criteria.
• Each environment includes a held-out evaluation set, to measure the agent on cases that it has not seen.

See the environments · Read the methodology