Every detection vendor demo runs on data the vendor chose, and data that has been chosen always has fraud in it. The demo tells you what the product looks like when it wins. It tells you nothing about what it does on your transactions, your ledger, your sessions, with the fraud you actually have and the one case you already know about.
Practitioners worked this out before any vendor did. On the compliance forums the standard advice is the same every time someone asks about a shortlist: give every vendor the same test data and see what they caught and what they missed. This post is that advice written down as a procedure, with the things to score and the one result we can share.
The method
Pick a window your team has already investigated, so you know some of the answers in advance. Six months is enough. Export the transactions, the ledger, the sessions and the KYC events for that window, as the files your systems produce, with no cleaning beyond what you would do for an auditor.
Hand the same files to each vendor on the shortlist. Ask for three things back: the accounts flagged, the logic that flagged each one, and a run on a second window you did not announce. Then check every flag by hand. Your team knows these accounts. It will take an afternoon.
What to score
Flags against confirmed cases. How many of the flagged accounts does your team confirm. A long list with a low confirmation rate is a cost, because each flag is review time.
Known cases missed. You chose the window because you investigated it. Every case you found that the vendor did not find is the number that matters most, and the number a demo never shows you.
Time from files to first result. Hours or weeks. The gap is wide, and it predicts what every future change to a rule will cost you.
What the logic is. Ask to see the rule that fired, as it ran, and ask to run it again on a different month. If what comes back is a narrative written after the fact from a score, that narrative is what you will be explaining to an examiner. A rule you can read and run again is evidence. A narrative is an opinion about evidence.
What leaves with you. If you end the contract, what do you keep. If the rules live in the vendor's builder in the vendor's format, the answer is nothing, and that is the real price of the product.
The one result we can share
We ran this on one remittance operator's data with our own tooling. Four exports, about 2.5 million rows, 301,099 accounts, and no labelled cases. The agent wrote and tested rules for five hours, and the rule it kept flagged five accounts. The operator checked them by hand. Four were fraudulent. One case the operator already knew about was missed.
The miss is on our site next to the four, because the test is only honest with both numbers. A vendor that shows you the hits alone has chosen the test for you, which is where this post started.
The trap
The trap is accepting a demo as the test because the test sounds like work. It is one export and one afternoon of checking, and it is the only comparison in which every vendor sees the same fraud. Anything else is a comparison of demos.
If you want to run the test on us, the first run on your export costs nothing. Send the files and you will have the flags, the rules that fired and the misses within a day of our receiving them.
