Methodology

1. Purpose and scope

FinCrime Atlas tests whether an AI agent can investigate a financial crime case and reach a documented decision to the standard of an experienced analyst. The scope is four environments, one for each compliance function: KYC, transaction monitoring, sanctions screening and fraud prevention. Each case is a multi-turn agent task. The agent plans the investigation, calls tools to search the institution’s systems and decides when it has enough evidence. For each case, the agent must:

1. Retrieve the material records and assess them against the customer profile.
2. Identify information gaps and request the information that it needs.
3. Consider each credible legitimate explanation.
4. Follow the applicable policies and reach a decision that the evidence supports.
5. Submit a decision record with the decision, the subjects, the rationale, the recommended next actions and the records that it cites.

We grade the investigation and the decision record.

2. The four environments

Each environment represents the case queue of one compliance function at an institution. Each environment has its own written policies. The agent receives the policies that apply to the case. The entries below describe the cases in each environment, the information that the agent receives, the decisions that it makes and what we grade.

Know Your Customer (KYC)

• The cases include new onboarding files, existing onboarding files with open gaps and periodic reviews, for individuals and businesses.
• The agent receives the customer file and the applicable policies. It identifies the documents that it needs, requests them and verifies them.
• The agent performs the due diligence that the policies require, including enhanced due diligence (EDD) when the risk requires it. It assigns an initial risk rating for onboarding files and a dynamic risk rating for periodic reviews.
• We grade the documents that the agent requested and verified, the gaps that it found, the beneficial owners that it identified, the due diligence, the risk rating and the information request that it writes to the customer.

Transaction Monitoring

• The cases include transaction monitoring alerts on fiat and crypto activity, for individuals and businesses.
• The agent receives the alerts and the applicable policies. It calls tools to retrieve the customer profile, the transactions and the other records that it needs.
• The agent can close the case, request information, escalate the case or draft a suspicious activity report or suspicious transaction report (SAR/STR) for review before filing.
• We grade the records that the agent retrieved, the red flags, the typology, the decision and the SAR/STR draft.

Sanctions Screening

• Each case starts from a screening alert on a customer name, an entity, an ownership structure, a crypto wallet or a real-time payment.
• The agent receives the alert, the sanctions list entry, the identifiers, the ownership records, the payment details and the applicable policies.
• The agent closes the alert as a false positive, releases the payment, blocks or rejects it, requests information or escalates the case.
• We grade the identifiers that the agent compared, the ownership that it checked and the reason for its decision.

Fraud Prevention

• The cases include account takeover, payment fraud and new-account fraud alerts, for individuals and businesses.
• The agent receives the alert, the device and behaviour signals, the payment history, the payee details and the applicable policies.
• The agent can close the case, recommend action to prevent the loss, escalate the case and/or draft a SAR/STR for review before filing.
• We grade the fraud pattern that the agent identified, the decision and the SAR/STR draft.

3. Case design

Institutions cannot share real case files, because of customer confidentiality and the legal restrictions on SAR disclosure. We build each case from available public sources. We use these sources for patterns only. All names, identifiers and records are synthetic.

Each case contains dated records. The relevant activity sits within normal account activity, as it does in a real case. The instructions do not suggest the decision.

In practice, most alerts are false positives. Each environment therefore contains cases with three evidence outcomes:

1. The evidence supports the concern.
2. The evidence supports a legitimate explanation.
3. Material information is missing.

An agent that always escalates or always closes gets a low score.

Each environment has a held-out evaluation set. We do not publish the held-out set. We do not use it to tune the harness or the graders.

4. Execution protocol

• Every model runs in our harness with the same instructions, tools, applicable policies and limit on tool calls. The model card or report states the other settings for each model, such as the output-token limit.
• Each run starts from a clean context. The agent can use only the tools in the harness. It never sees the answer key or the grading criteria.
• In a full benchmark report, each model runs each case more than once, to measure consistency. Every run counts in the score. A model release, such as the Atlas-TM preview, can report one run of each case.
• We keep an audit trail for each run: every tool call, every tool response, the decision record, the model version and the settings.
• We re-run a case only after a documented infrastructure failure, and we report each re-run.

5. Grading

We grade every part of the agent’s work, from the investigation to the final decision. These standards apply in all four environments:

• The evidence and the applicable policies must support the decision.
• Each finding must trace to a record, with correct amounts, parties and dates. Verified facts must stay separate from allegations.
• The conclusions must follow from the evidence.
• The agent must apply the policy requirements that are relevant to the case.
• The rationale must state the recommended next actions and each material gap.

Each case has a private answer key. We write the answer key and the grading criteria before any run, and a practitioner reviewer checks them. A wrong decision, a missing decision record or a critical failure, such as a citation to a record that does not exist, sets the score for the case to zero.

Automated checks and a model judge apply the grading criteria. The model judge and the agent under test come from different model families. Before we publish any score, practitioner reviewers will grade a sample of runs and compare their grades with the grades of the model judge.

6. Reporting

In a full benchmark report, we will report results for each environment and for each type of case, with confidence intervals. Each score will appear with the critical failure rate, the consistency across runs, the cost and the run time. Each score will also appear next to two baselines: an agent that always escalates and an agent that always closes. A model release, such as the Atlas-TM preview, can report fewer of these measures. We will compare scores only within one environment version.

7. Limits

A score measures an agent in our environment. It does not assess performance in a live deployment. The cases and the policies are synthetic, and the policies do not represent the policies of any institution.