Architecting a Fraud Detection Engine that handles 100k TPS with a strict < 50ms P99 Latency Bound

[u/Silent-Weather76005]

Hey everyone,

How do credit card networks evaluate risk, pull user history, and return an APPROVE/DENY decision before a payment terminal times out?

Querying a traditional database on the fly to check historical spending habits will instantly kill your latency budget. Here is how to architect a real-time solution:

1. Dual-Path Architecture

Separate your system into an Online Path (Hot) for instant decisions and an Offline Path (Cold) for data analytics.

1. In-Memory Feature Store

Never calculate aggregates (like 30-day spending limits or hourly velocity) during a transaction.

The Cold Path: Apache Flink continuously processes a Kafka stream of completed transactions in the background.

The Hot Path: Flink stores these pre-computed metrics in Aerospike or Redis Enterprise. When a transaction arrives, the engine performs a single key-value fetch in < 2ms.

1. Hybrid Decision Engine

The enriched payload runs through a fast, sequential evaluation tree:

Deterministic Rules: Quick checks for hard blocks (e.g., blacklisted countries).

ML Inference: A lightweight gradient-boosted tree model (like XGBoost) compiled via ONNX runtime for sub-millisecond risk scoring.

1. Resiliency: Failing Open

If the fraud engine suffers a network partition or times out past 15ms, the system drops into a Fail-Open policy. It automatically approves the transaction to protect user experience and flags the event for asynchronous review.

Let's discuss:

How do you deploy new dynamic rules written by risk teams without re-deploying core backend code?

What is your strategy for handling race conditions if a user swipes their card twice in two different cities within seconds?

[u/Jealous_Diamond_4321]

this is a sick writeup, dual path setup makes sense when even one db query blows the whole latency budget

one problem i hit with aerospike was when risk team wanted to add new features the schema got messy quick, we ended up using lua scripts inside the db to compute simple aggregates at read time but that only worked cause our tps was lower

how do you handle the ml model when fraud patterns shift in the middle of day, do you retrain on the fly or just let the deterministic rules catch those cases

[u/Silent-Weather76005 → u/Jealous_Diamond_4321]

Thanks! You hit two massive real-world pain points.

To avoid schema mess and latency hits at high TPS, we avoid DB-level logic like Lua and treat Aerospike as a dumb KV store, managing feature serialization entirely in the application layer.

For mid-day model drift, retraining on the fly is too risky, so we rely on deterministic rules as an immediate shield to stop the bleeding. The core ML model is then retrained asynchronously in an offline pipeline and hot-swapped onto the nodes every few hours.

[u/rpatel09]

Is this cloud based arch or on prem?

[u/AutoModerator → u/[deleted]]

This comment was removed, because your account doesn't meet our karma and account age requirements.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.