Taking a payment looks like a solved problem: call the SDK, get a success, add credits. In practice, almost every bug in a payment system lives in the gaps. A callback arrives twice. A browser tab closes halfway. Two requests read the same row at the same moment. A server crashes between two writes.

This post walks through the whole flow, step by step, and explains why each decision was made. You can reuse these patterns with any payment gateway.

Step 1: Create the order on the server, and never trust the client's price

When a seller picks a pack, the client sends only a pack ID, never an amount.

packId: z.enum(["trial", "starter", "standard", "pro", "topup", ...]),
topupQuantity: z.number().int().min(1).max(2000).optional(),

The server looks up the price from its own table and creates the Razorpay order.

Nuances:

• Amounts are integers in paise (24900, not 249.00). Floating-point maths on money gives you answers like 0.1 + 0.2 = 0.30000000000000004. Integers in the smallest currency unit avoid that whole class of bugs.

• Inputs are bounded. A custom top-up quantity is clamped to 1–2000. Without an upper bound, someone could request an absurd order. Without a lower bound, 0 or negative amounts become possible.

• We write a Payment row with status: "created" before any money moves. This row is our own record of the order. Every later step looks it up instead of trusting what the client or the webhook claims.

Step 2: Checkout, and why one confirmation isn't enough

The seller pays inside Razorpay Checkout. When it finishes, Razorpay can tell us about it in two independent ways:

You need both. The callback gives a good experience and the webhook guarantees delivery. But now two code paths can try to credit the same order, sometimes at the same millisecond. That race is the main subject of this post.

Step 3: Verify the signature, because anyone can call your endpoint

Both endpoints are public URLs. Before trusting anything, we prove the message really came from Razorpay, using HMAC-SHA256: a hash keyed with a secret only Razorpay and our server know.

Client path: Razorpay hands the browser order_id, payment_id and a signature. We recompute:

HMAC_SHA256(key_secret, `${order_id}|${payment_id}`)

Webhook path: we compute the HMAC over the raw request body with a separate webhook secret.

Nuance: why the raw body? By the time a normal Express handler runs, the JSON has already been parsed. If you JSON.stringify(req.body) to rebuild it, whitespace, key order or number formatting can differ from the bytes Razorpay signed, and a perfectly valid webhook fails verification. We capture the exact bytes as they arrive:

express.json({ verify: (req, _res, buf) => { req.rawBody = buf; } })

If the raw buffer is missing, we reject the request instead of falling back to the parsed body.

Nuance: an IDOR check on the client path. We look up the payment by razorpayOrderId and userId. Otherwise a logged-in user could submit someone else's order ID. (IDOR, "insecure direct object reference", means fetching an object by ID without checking who owns it.)

Step 4: timingSafeEqual, the comparison most people get wrong

After computing the expected signature, the obvious check is:

if (expected === received) { ... }   // ❌

The problem is that === on strings stops at the first character that differs. Comparing "a9f3..." against "b000..." returns slightly faster than comparing it against "a9f0...". That difference is tiny, but over thousands of requests it can be measured. In principle, an attacker can use that timing to discover a valid signature one character at a time. This is called a timing side-channel attack.

crypto.timingSafeEqual always compares every byte, however early a mismatch occurs, so the response time reveals nothing:

const a = Buffer.from(expected, "utf8");
const b = Buffer.from(received || "", "utf8");
if (a.length !== b.length) return false;   // timingSafeEqual throws on unequal lengths
return crypto.timingSafeEqual(a, b);

Nuances:

• Check the length first. timingSafeEqual throws if the buffers differ in length. An unhandled throw becomes a 500, which is a bug and itself a signal to an attacker. Returning early on a length mismatch is safe here, because a SHA-256 hex digest is always 64 characters, so the length is public anyway.

• || "" guards against a missing signature, so an absent header is a clean false rather than a crash.

• We unit-test this exact function with: a valid signature, a tampered signature of the same length (the case === would also catch, but which must still fail), the wrong length, and an empty string.

Step 5: Atomic capture, where the database decides who wins

Now both paths have a verified payment. The naive code looks like this:

const p = await Payment.findOne({ orderId });
if (p.status !== "captured") {        // check…
  p.status = "captured"; await p.save();
  await addCredits(...);              // …then act
}

This is a classic check-then-act race. If the callback and the webhook both run findOne before either saves, both see "created" and both add credits. The seller pays once and gets credited twice.

The fix is to make the check and the write a single atomic operation:

const won = await Payment.findOneAndUpdate(
  { _id: payment._id, status: { $ne: "captured" } },   // the condition…
  { $set: { status: "captured", razorpayPaymentId } }, // …and the write, in one step
  { new: true }
);

if (!won) return currentBalance;   // someone else already captured it: idempotent no-op
await addCredits(...);             // only the winner gets here

MongoDB guarantees that an update to a single document is atomic. Only one caller can move the order from created to captured. The other caller's filter no longer matches, so it gets null back.

Nuances:

• The loser isn't an error. The payment did succeed, so the loser returns success and the current balance. That's what idempotent means: running it twice has the same effect as running it once.

• There's also a fast path. If verifyPayment sees the order is already captured before even checking the signature, it returns immediately. That's an optimisation; the atomic update is the actual guarantee.

• No distributed locks, no Redis mutex. The database row is the lock, which means fewer moving parts.

• A failed browser signature marks the payment failed, but that isn't final. The capture filter is status ≠ captured, so a valid webhook can still capture it. Razorpay remains the source of truth for whether money actually moved.

Step 6: What the webhook sends back matters

Razorpay retries a webhook until it receives a 2xx response. So each response code is a decision:

Step 7: Add credits with $inc, never read-modify-write

User.findOneAndUpdate({ _id }, { $inc: { creditBalance: amount } }, { new: true })

Nuance: user.creditBalance += amount; await user.save() reads a value, changes it in memory, and writes it back. Two concurrent updates can overwrite each other, which is a lost update. $inc makes the database do the arithmetic atomically.

Step 8: An append-only ledger

Every credit movement inserts one immutable row:

{ type: PURCHASE | DEBIT | REFUND | BONUS | TRIAL, amount, balanceAfter, referenceId, referenceType }

Nuances:

• Rows are never updated or deleted. A balance is a cached number; the ledger is the history that explains it. If a support ticket says "I paid but got nothing", you can answer from data.

• balanceAfter comes from the document returned by the atomic $inc, not from a separate read, so it's exact even under concurrency.

• referenceId links each row back to its payment or listing, and it's indexed (userId + referenceId), which makes reconciliation and "was this already refunded?" checks cheap.

Step 9: Side effects must never break the payment

After a purchase, we pay the referrer's bonus if this was the buyer's first purchase:

grantReferralRewardOnFirstPurchase(userId).catch(() => {});

It's fire-and-forget on purpose. A bug in the referral system must never turn a successful payment into a 500, which would cause a webhook retry or show the user an error. The referral claim is itself atomic, so it pays out once even if it's triggered twice.

The same principles on the spending side

Buying credits is half the story. Spending them has the same problems.

Atomic debit, no overdraw:

User.findOneAndUpdate(
  { _id, creditBalance: { $gte: cost } },     // only if they can afford it
  { $inc: { creditBalance: -cost } }
)

Two concurrent generations with a balance of 5 and a cost of 5 each: exactly one succeeds. The other gets null, which becomes a clean "insufficient credits" error.

Compensating transaction: we charge first and then queue the AI job. If queuing fails (for example, Redis is down), we refund immediately and mark the listing failed. The seller is never charged for work that will never run.

Refund exactly once: if the worker exhausts its retries, it first claims the refund atomically:

ListingObject.findOneAndUpdate(
  { _id, creditsRefunded: false, creditsCharged: { $gt: 0 } },
  { $set: { creditsRefunded: true } }
)

Only the caller that flips the flag issues the refund. It's the same pattern as the payment capture, reused.

What's still imperfect, and how I'd fix it

Being honest about the gaps is part of the engineering:

1. Capture and credit are two separate writes. If the process dies after marking the order captured but before $inc, the seller paid and got nothing, and neither path will retry because the order already looks captured. Two possible fixes:

   • Wrap the capture, the credit and the ledger insert in a MongoDB multi-document transaction.

   • Run a reconciliation job that finds payments marked captured with no matching PURCHASE ledger row and repairs them. Real payment systems usually run something like this regardless.

2. The balance update and the ledger insert are also separate writes, so the same transaction would close that gap too.

3. Webhooks aren't de-duplicated by event ID. The atomic capture makes duplicates harmless, but storing the event ID would make replays easy to spot in logs.

Takeaways

• Never trust the client's price. Compute amounts on the server, as integers in paise.

• Expect every message to arrive twice, late, or out of order.

• Verify signatures over raw bytes, with timingSafeEqual.

• Replace check-then-act with one conditional atomic update. Let the database be the referee.

• Use $inc, never read-modify-write.

• Keep an append-only ledger, so every balance can be explained.

• When a later step fails, compensate for the earlier one, and make the compensation itself run exactly once.

Getting the happy path working is easy. The real work is making the code stay correct when things happen twice, out of order, or halfway through.

Code: github.com/Akashjaiswal-IT/ListingLift

I recently made a memory layer for one of my Gen AI project (a lightweight Notebook LM clone ) to apply for Mem0 , After sharing it on X someone suggested me to implement AI evals in the project . At that time I didn’t knew about AI evals so I got curious and started learning about AI evals and how Mem0 incorporates evaluations for their Memory Layer .

So here I am sharing all that I have learned about AI evals and Mem0 evaluation process .

Why We Need AI Evals

Generally most of the systems that we see or build is deterministic in nature meaning if you give the same input the output or expected output is always the same. For example , In a chatting app , if you send a message to someone , the expected output is straightforward , the message is delivered and received correctly at the other end .

Which is not the case with AI systems as they are non deterministic in nature meaning if you give the same input multiple times the output will be slightly different every time , you can’t have a definite expectation of output , which makes testing this systems difficult as compared to the traditional systems . Here AI evals is used to test and quantify how the system is working . AI evals defines the perfect output and how close the response is from the perfect output . You can also define the tone , quality and safety metrics of the response using AI evals .

AI Evals defines benchmarks on quantified criteria to determine the performance of the system . As the software grows more and more complex , it becomes very difficult to improve and correct things without evaluations . Not having evaluation may lead to very severe edge cases that can harm your business and image . For instance , Microsoft’s Tay chatbot had to be taken offline after it learned to produce harmful content within 24 hours. In 2022 Meta’s Galactica model similarly “hallucinated” wildly incorrect scientific content, prompting an immediate shutdown. If the AI evals are designed well it can catch all such failures , loopholes about factual accuracy , tone , safety metrics and all the other metrics in which you want your system to perform well . AI evals provide a feedback loop which can be utilized to improve , fine tune the system to make it more reliable & safe and to work on the system to achieve a specific target .

High-Level Overview of a Basic AI Evaluation System

A basic AI evaluation system starts by defining the tasks that are being performed by the AI system , and then writing the prefect response of various scenarios of input related to the tasks , for a very complex system this scenario’s may exceed a very large number for scenarios , as more the goldens better the evaluations , if needed take help from the LLM’s to cover the edge cases and large a lot of data set of goldens . Then comes the evaluation part , the team decides the criteria in which they want rate the response and then finalize a rating metrics for all and then from the data set created by LLM’s or from the input given by the user rate the response given by the system on the metrics , you can even automate the rating process as well by keeping it check with frequent human feedbacks and knowing the comparison between the reviews given by LLM and the reviews given by human so that the ratings can be trusted accordingly . And then using those metrics and ratings reiterate or improve the system accordingly . This can be summarized as :- define tasks → define goldens → run model → compare output to goldens → compute scores/ratings → analyze results.

Building a Basic AI Eval System

1) Define Goldens

Goldens are the benchmark for the responses given by your system , it is an ideal input and output example for your system keeping in mind the factual correctness , tone , quality of output , safety metrics and all other metrics that you want your system to have . Start by manually defining the perfect and ideal input output examples for all the possible scenarios and edge cases that you can think of , for a complex system it can even be in hundreds or more . Things become easier in longer run if your goldens are sorted in the beginning .
An example of the goldens would be ,

For a customer support agent

Input :-

Hi, my order #45219 was supposed to arrive yesterday, but the tracking link still says “in transit.” Can you check what’s going on? I really needed it by today.

Response :-

Hello! I’m really sorry your order hasn’t arrived yet — I understand how frustrating that must be.  
I’ve checked your order #45219, and it’s currently delayed due to a logistics issue. The package is still on its way and should arrive by tomorrow evening.  

We’ve also refunded your express shipping fee as an apology for the delay.  
You’ll receive a confirmation email shortly.  

Thank you for your patience — please let me know if you’d like me to update the delivery address or time.

Why it is a golden ?
Because it’s the ideal, policy-aligned, and empathetic response that sets the benchmark for correct AI behaviour.

2)Synthetic Data Generation

Using the goldens written generate a larger set of such examples using an LLM increasing the coverage of examples . As the goldens written manually are not enough to test a system at scale . So generating a larger set of examples increases the chance of covering the edge cases that you might have over looked .
It makes testing more thorough resulting in more data to evaluate and improve on .
An example of prompt that can generate more examples from the existing set of goldens

You are an AI assistant that generates synthetic customer support evaluation data using reference golden examples.
Use the given golden as a guide for tone, structure, and policy alignment.
Create new and diverse examples that test reasoning, empathy, and accuracy — including variations and edge cases.

Output must be valid JSON.

Input (Golden Example):
Customer: "Hi, my order #45219 was supposed to arrive yesterday, but the tracking link still says 'in transit.' Can you check what’s going on? I really needed it by today."
Expected Response: "Apologize empathetically, provide tracking update, and offer compensation if policy allows."

Your Task:
Generate 10 new synthetic examples following the same format.

Output Format (JSON):

{
  "examples": [
    {
      "id": 1,
      "customer": "string",
      "expected_response": "string",
      "intent": "string", 
      "category": "string"
    }
  ]
}

Instructions for Generation:

• Include different categories: late delivery, damaged product, refund issue, wrong item, cancellation, subscription, missing refund, rude staff, etc.

• Ensure expected_response models empathy, accuracy, and policy compliance.

• Keep tone natural and professional.

• Avoid duplicates or trivial rewording.

Use prompt written by yourself according to your own requirements or use pre tested open source prompts . Try to avoid generating the prompts from LLMs as it might reduce the quality of your response due to drift and degradation .

3) Unit Tests

Incorporate simple unit tests similar to traditional unit tests to rule out all the basic failures . For example , If you don’t want any specific keywords , a specific format data , if it’s too long or short . For this basic programmatic flaws to be picked before human you can write unit tests .
You can write unit test for either a specific task such as for a feature which fetches perfect product for you can write unit test according to the list of matching queries or a generic unit test which is applicable on all the features . An example will be if you don’t want your response to have any abbreviations , so you can write a regex test to check that .
You must write hundreds of tests to make your system as perfect as possible .
Do not shy away to take help of LLM to brainstorm on the edge cases that you might be over looking .

Talking of LLM , use it to generate the test cases that triggers all the unit tests you have to thoroughly check your system and every edge cases possible .
You can integrate this tests using CI infrastructure ( e.g., GitHub Actions, GitLab Pipelines, etc.) You should also collect metrics along with the results of tests to analyse and track your system .

4) Grading / Rating

Define a clear rating system for the subjective quality dimensions of the response ,the closer the response is to the golden, the higher its rating should be. Rating and grading can be anything a numeric value , binary or bool . The guidelines for every rating should be pre defined strictly to maintain consistency of the rating system . Grading and rating should be done by human evaluation . Use the example data set generated or user’s data to generate response and then rate it on the pre determined rating system .
For example :-

{
  "example_id": 18,
  "customer": "Hi, I received a damaged lamp in my order. Can you send a replacement?",
  "golden_response": "I’m really sorry your lamp arrived damaged. I’ve arranged for a replacement to be shipped immediately, and you’ll receive the confirmation email shortly. You can dispose of the broken lamp safely — no need to return it.",
  "model_response": "Apologies for the damaged lamp. Please send it back so we can review and then send a new one.",
  "rating": {
    "empathy": 4,
    "accuracy": 4,
    "tone_professionalism": 4,
    "policy_alignment": true,
    "closeness_to_golden": 3,
    "overall_score": 3.8
  },
  "graded_by": "human_evaluator_07"
}
A Sample Of Customer Support Agent Evaluation And Rating

5) LLM Auto-Raters

Rating every response manually takes a lot of labour and time , that can be spent on something more productive . So you can implement an LLM auto-rater ( LLM as a judge ) whose job is to rate the response according to the goldens as reference and source of truth specified on rating system specified by you .
LLM auto-rater should be frequently compared to the human evaluator to know the accuracy of LLM’S evaluation and the minimum accuracy should be defined by team (For example , 80%) , above which the LLM auto-rater can be trusted to rate on behalf of humans . Although there should be a human in the loop always to give constant feedbacks and reviews to improve the auto-raters and the goldens . LLM auto-rater should always be seen as to a tool that reduces the time and labour and not as replacement of human . It should be checked by humans frequently to keep it in check .

6) Tracing and Logs

Instrument AI pipelines to show input , response , evaluation results / ratings for better understanding of the response given by the system to improve the system more . LangSmith and other observability tools were being used before to see the logs meaning history of all the user inputs and the model outputs but nowadays OpenAI gives this observability inbuilt reducing the hassle of setting up these tools . This logs can be used to trace back to the exact set of input , output which is causing the problem . Hence making it easier to debug , resolve and improve systems by making changes .

An example of logs of a project like Browser Use (Browser Automation Agent) - made by me

Improve with the Help of Evals

The ultimate goal of evaluation is to inform improvements. Effective evals serve three purposes: analyze system behavior, measure performance quantitatively, and improve by fixing weaknesses. For example, evaluation may highlight that answers often lack factual detail now you can do prompt refinements , pass better contexts with factual data accordingly . You might use evals to label new training data (supervised fine-tuning) or to select better few-shot examples. Evals also help prioritize work , they identify the most common failure modes (e.g. hallucinations or misunderstanding), so engineers know where to focus. By cycling through this Analyze–Measure–Improve loop, teams can steadily raise their AI’s quality and reliability .

How Evaluation Happens in Mem0

Mem0 has a very different technique and strategy for evaluation . It evaluates it’s system using a structured benchmark approach . The core of this is a LOCOMO dataset , a suite of long-term conversations designed to test memory recall . Mem0 compares it’s performance metrics with different tools set a baseline of benchmark to surpass . Let’s see this in detail :-

Overview

1) Established LOCOMO Benchmarks

Mem0 evaluates its performance against five established approaches to see where it stands ,

• LoCoMo

• ReadAgent

• MemoryBank

• MemGPT

• A-Mem

2) Open-Source Memory Solutions

Mem0 also tests on open-source tools like LangMem that handles memory layer in AI models to compare and benchmark results

3 ) RAG Systems

Mem0 implements Retrieval-Augmented Generation with various configuration . It tests different chunk sizes and the retrieval counts to optimize it’s performance .

4) Full-Context Processing

Mem0 also examines the effectiveness of passing the whole conversation history within the context window of LLM as a baseline approach for its evaluation

5) Proprietary Memory Systems

Mem0 evaluates OpenAI’s built in memory feature available in their ChatGPT interface to compare against commercial solutions.

6) Third-Party Memory Providers

Mem0 incorporates Zep, a specialized memory management platform designed for AI agents, to assess the performance of dedicated memory infrastructure.

Dataset

Mem0 tests these techniques on the LOCOMO dataset, which contains conversational data with various question types to evaluate memory recall and understanding. Dataset has 10 multi-session conversations with 26 k tokens in each conversation , having 200 questions per conversation

Download Dataset

This dataset is specially designed to test memory recalls and understanding across various question types and complexity level .

Rating Metrics And Evaluation

Project Flow

• Mem0 is evaluated against to OpenAI , LangMem , Zep to determine a base benchmark of performance for Mem0 .

• RAG techniques are tested to give the best retrieval of context for memory .

• A script for metrics to calculate evaluation metrics and prompt used for the models is available .

• An evaluation script :- Evaluates model responses using BLEU, F1 scores and LLM as a judge, then stores detailed evaluation results in JSON format.

• Run experiment script :- Runs different memory architectures (Mem0, RAG, LangMem, Zep, OpenAI), adds or searches memories, and saves raw responses for evaluation.

• Generate Score script :- Aggregates evaluation results, computes mean BLEU/F1/LLM scores per category, and prints overall performance metrics

Metrics

• BLEU (Bilingual Evaluation Understudy) :- In Mem0’s evaluation pipeline, BLEU measures how textually similar a model’s generated response is to the ground truth (golden) answer.
  It captures n-gram overlap , meaning how many words or short phrases match , giving a sense of linguistic closeness between the system’s output and the ideal answer.
  A higher BLEU score means the memory system’s retrieval and response generation are more faithful to the expected context.

• F1 Score (Harmonic Mean of Precision and Recall) :- Mem0 uses F1 to evaluate how accurate and complete the model’s response is. It checks whether the system retrieves correct factual pieces (precision) and whether it misses any relevant ones (recall) from memory.
  A higher F1 means the memory retrieval and summarization are more balanced and contextually complete.

• LLM Score (Large Language Model Judge Score) :- This is Mem0’s LLM-as-a-judge metric , an additional evaluation layer where an external LLM compares the predicted response with the ground truth. It judges semantic correctness, coherence, tone, and factuality, going beyond simple text overlap.
  A higher LLM score indicates that the system’s memory retrieval and response feel more human-like, context-aware, and semantically aligned — even if the wording differs.

Evaluations

In evaluation Mem0 processes each and every question and answer pair , Calculates BLEU ( Bilingual Evaluation Understudy .i.e. Measures text similarity between the model’s response and the golden response, based on n-gram overlap) and F1 scores ( Combines Precision and Recall) . Then Mem0 uses an LLM judge to evaluate answer correctness and then store the results in an output file .

Generating Scores

Mem0 generates scores by loading evaluation metrics data . Then calculates mean scores for each category (BLEU, F1, LLM) . Also reports the number of question per categories and then calculates the mean scores across all the categories .

📊 Example Output

Overall Mean Scores

Evaluation Metrics

Mem0 uses several metrics to evaluate the performance of different memory techniques:

1. BLEU Score: Measures the similarity between the model's response and the ground truth

2. F1 Score: Measures the harmonic mean of precision and recall

3. LLM Score: A binary score (0 or 1) determined by an LLM judge evaluating the correctness of responses

4. Token Consumption: Number of tokens required to generate final answer.

5. Latency: Time required during search and to generate response.

Conclusion

AI evaluations are the foundation of building reliable, safe, and measurable AI systems. They turn subjective model behaviour into quantifiable metrics that help teams understand, compare, and improve performance. By defining strong goldens, generating diverse synthetic data, setting clear grading standards, and leveraging both human and LLM-based raters, developers can systematically analyze how close their systems are to the ideal output.

Mem0’s structured evaluation framework illustrates how thoughtful benchmarking—using metrics like BLEU, F1, and LLM judge scores—can transform experimentation into evidence-based improvement. Through continuous evaluation, tracing, and iteration, AI systems evolve from unpredictable prototypes into consistent, trustworthy tools.

In short, AI evals are not just about scoring models — they’re about creating a feedback loop that drives progress, reliability, and accountability in every generation of AI.

Reference

https://github.com/mem0ai/mem0

https://mem0.ai/blog/benchmarked-openai-memory-vs-langmem-vs-memgpt-vs-mem0-for-long-term-memory-here-s-how-they-stacked-up

https://hamel.dev/blog/posts/evals/

https://youtu.be/0Ta8ws4pU-8?si=wzGDCTFF0HaFeGw-

If you use the dataset given in blog or code from the evaluation folder of Mem0 repo in your research paper , Please cite Mem0’s paper given below

@article{mem0,
  title={Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory},
  author={Chhikara, Prateek and Khant, Dev and Aryan, Saket and Singh, Taranjeet and Yadav, Deshraj},
  journal={arXiv preprint arXiv:2504.19413},
  year={2025}
}