Event-Driven, Agentic AI Fraud Detection Pipeline

Building an Event-Driven, Agentic AI Fraud Detection System on Kubernetes

Modern fraud detection systems face a dual challenge: they must operate with low latency to block bad transactions in real-time, and they must provide clear, human-readable explanations for why a transaction was flagged. A black-box machine learning model that simply outputs a 1 or 0 is no longer sufficient for compliance or operational efficiency.

To solve this interesting problem, I built an enterprise-grade, event-driven microservices pipeline. Moving beyond simple classification, this project leverages an autonomous Agentic AI workflow connected to a Vector Database (pgvector) and an Explainable AI (SHAP) pipeline to generate human-readable audit reports on demand.

To ensure fault tolerance and horizontal scalability, the entire stack is orchestrated natively on Kubernetes.

The Architecture: Decoupling for Scale

When designing a high-throughput system, tight coupling is the enemy. If a million transactions spike during a holiday sale, a standard REST API connected directly to a database will buckle. To prevent this, the architecture is broken down into specialized, decoupled layers:

1. Ingestion & Buffering (Apache Kafka): Instead of hitting the API directly, transactions are published to a Kafka topic. Kafka acts as a shock absorber, queuing the events so the compute layer can consume them at a safe, controlled rate

   Deep Dive 1 - Real-Time Streaming & Caching (Kafka + Redis + FastAPI)

2. Real-Time Inference (FastAPI & Redis): A Python consumer pulls from Kafka and hits the inference endpoint. To achieve sub-100ms latency, the API first checks a Redis read-through cache. If the transaction was recently evaluated, it returns immediately. If not, it loads the active XGBoost model from the MLflow Registry and calculates the risk.

   Deep Dive 2 - The ML Pipeline & Explainable AI (XGBoost + MLflow + SHAP)

3. Unified Storage (PostgreSQL + pgvector): Background tasks handle the heavy lifting of generating LLM vector embeddings for the transaction. Both the relational data (timestamps, amounts, ML predictions) and the high-dimensional vector embeddings are stored in a single PostgreSQL database using the pgvector extension. This heavily simplifies the infrastructure compared to running a separate, dedicated vector database.

flowchart TD
    %% Custom styles designed to inherit your site's Neon / Dark Mode theme
    classDef stream fill:#151515,stroke:#D35400,stroke-width:2px,color:#E0F7FA,rx:5px,ry:5px;
    classDef compute fill:#0B0C10,stroke:#45FCDB,stroke-width:2px,color:#E0F7FA,rx:5px,ry:5px;
    classDef storage fill:#151515,stroke:#D35400,stroke-width:2px,stroke-dasharray: 5 5,color:#E0F7FA;
    classDef ai fill:#0B0C10,stroke:#45FCDB,stroke-width:3px,color:#E0F7FA;

    subgraph Streaming Layer
        direction LR
        P[Transaction Producer]:::stream -->|JSON Events| K[(Apache Kafka)]:::storage
        K -->|Subscribes| C[Python Consumer]:::stream
    end

    subgraph Inference & Caching
        C -->|POST /predict| API[FastAPI Inference]:::compute
        API <-->|Read-Through Cache| R[(Redis)]:::storage
        ML[(MLflow Registry)]:::storage -.->|Loads XGBoost Pipeline| API
    end

    subgraph Storage & XAI
        API -->|Async Writes| DB[(PostgreSQL + pgvector)]:::storage
        API -->|Generates| SHAP[SHAP Explainer]:::compute
    end

    subgraph Agentic UI
        UI[Streamlit Dashboard]:::compute -->|Polls Latest 100| DB
        UI -->|User Triggers| Agent((LangGraph AI Agent)):::ai
        Agent -->|Tool 1: Fetch SHAP Values| SHAP
        Agent -->|Tool 2: Vector Similarity| DB
    end

The Agentic Brain: Moving Beyond Static RAG

The crown jewel of this system is the investigation layer. Traditional AI integrations often just wrap an LLM around a static prompt. This system uses LangGraph to create an autonomous ReAct (Reasoning and Acting) agent capable of utilizing external tools.

When an analyst spots a flagged transaction on the Streamlit live dashboard, they can deploy the AI Agent. The agent doesn’t guess; it investigates using two specific tools:

• SHAP Value Analysis: It queries the FastAPI /explain endpoint to extract the exact mathematical features (e.g., specific PCA-transformed behavioral vectors) that pushed the XGBoost model to flag the transaction.

• Vector Similarity Search: It queries pgvector to find historically similar transactions (using cosine distance) to see if this specific behavioral pattern has resulted in confirmed fraud before

The agent synthesizes this raw mathematical and historical data into a concise, professional executive summary, effectively doing the initial legwork of a junior fraud analyst in seconds.

Deep Dive 3 - The Agentic Brain (LangGraph + pgvector)

Cloud-Native Deployment

To bridge the gap between local development and production reality, the entire architecture is built for Kubernetes. Using tools like Minikube or Kind, the system can be spun up locally

The deployment strategy utilizes Kubernetes Jobs for stateful initializations—such as init-model, a containerized training job that automatically trains the baseline model and registers it via MLflow before the core inference pods are allowed to spin up. This ensures the API never boots without a model ready to serve.

Deep Dive 4 - Cloud-Native Orchestration (Kubernetes)

What Was Accomplished

Ultimately, this project bridges the gap between traditional MLOps, real-time streaming, and modern Generative AI. By decoupling the architecture, the system achieves the high-throughput, low-latency performance required of financial systems. At the same time, integrating LangGraph and pgvector solves the “black box” problem, providing analysts with instant, data-backed context without sacrificing operational speed.

Looking Ahead: Future Improvements

While the current architecture is robust, there are a few key areas I plan to expand on in the future:

• Expanding the Agentic AI: Giving the LangGraph agent access to more sophisticated tools, such as the ability to query external threat-intelligence APIs or cross-reference user IP addresses

• Scaling the Dataset: Training the XGBoost model and populating the vector database with a much larger, more complex dataset to further test the limits of pgvector’s search latency

• Authentication & Access Control: Because this system handles highly sensitive financial data, adding a robust authentication and logging interface to the Streamlit dashboard is essential to ensure only authorized personnel can view the transaction streams and trigger investigations