How Would I Build a Payment System That Doesn't Lose Money

This is the first post in a series I'm calling How Would I Build. I take a real engineering problem, reason through it in plain language first, then attach the actual name for that reasoning at the end. Jargon comes last, not first.

I'll be upfront, I didn't sit down and immediately know all the answers here. A lot of this came from asking "okay but why would that break" until something clicked. That's the energy I'm going for.

Starting with this: how would you build a payment system that handles 10,000 transactions per second without losing a single one?

Two customers, one suya guy

There's a suya spot near you. One guy grilling. Two customers walk up at the same time and both want the last stick.

The suya guy has two hands. He can't serve both customers simultaneously. One of them has to wait.

Now imagine he ignores that and just gives both customers the same stick. That's your payment bug.

Two people hit withdraw on the same wallet at the exact same time. Both requests check the balance. Both see enough money. Both go through. You've just paid out twice what was in the account.

My first instinct was just — lock it. But I had to figure out what "lock it" actually meant before it made sense.

The answer is: whoever gets there first locks the wallet in the database before the second request can even read the balance. The second request waits. When the first one finishes, the second one checks the balance again and finds there's nothing left.

That whole situation — multiple operations running at the same time and stepping on each other, is a concurrency problem. The specific bug where both read stale data before either writes is a race condition.

The suya guy gets popular. He buys a bigger grill.

Business is good now. The queue is long. The suya guy thinks: I need a bigger grill. More charcoal. Better equipment. He upgrades everything.

It helps. Orders come out faster. But the queue is still there. Because no matter how big the grill is, there's still one person running it. Customers still wait behind each other. And there's a limit to how big a grill can get anyway.

That's vertical scaling. You give the same server more power — more CPU, more RAM, faster storage. It buys you time but it doesn't solve the problem. The ceiling just moves higher.

At some point the suya guy realizes the grill isn't the problem. One location is the problem.

So he opens a second spot. Then a third. Now three customers are being served at the same time across three locations. The individual grills are smaller but the total output is way higher.

That's horizontal scaling. More machines, not a bigger machine. The limit is now cost, not hardware.

But not everything needs the main guy

The suya guy notices something. Most people walking up aren't placing new orders. They're just checking if their order is ready. Those people are slowing down the main grill.

So he puts a separate person to handle "is my order ready" questions. That person has a copy of the order list and handles all status checks. The main guy now only deals with new orders.

Read traffic stops choking write performance.

That's read replicas. The main database handles writes. Copies of it handle reads.

But here's where I got confused for a bit. Replicas don't fix everything. If 10,000 new orders are coming in per second, the main grill still has a ceiling on how many it can process. Adding more status-check people doesn't change that.

To scale new orders, you need more main grills. Not copies — actual separate locations taking orders. Customers 1 to 2,000 go to location one. 2,000 to 4,000 go to location two. New orders are now spread across multiple spots.

That's sharding.

What if someone at spot one wants to pay for a friend at spot three

Someone at spot one wants to cover their friend's order at spot three. The suya guy at spot one collects the money. Before he can radio spot three to release the order, his phone dies. Spot three never gets the message. The money is gone and the friend got no suya.

My instinct was to get both spots to agree before anything moves. Spot one holds the money, spot three holds the order, and only when both confirm does anything proceed. That works but it's slow. And if the person coordinating between them disappears mid-process, both spots are frozen waiting for a signal that never comes.

The simpler approach is to not wait for both at the same time. Spot one takes the money and writes down "I took this, spot three owes a release." If spot three never confirms, spot one automatically reverses the charge and refunds the customer. No coordination needed upfront. Each spot handles its own piece and knows exactly what to undo if something breaks.

That pattern is called a SAGA. Failure recovery for distributed operations. Each step has a compensating action ready before anything runs.

Multiple spots, but who tells customers where to go

You've opened five suya spots. A new customer walks up. They see five spots and have no idea which one has the shortest queue. They just stand there confused.

So you put someone at the entrance whose only job is to look at all five spots and point each new customer to the next free one. The customer doesn't think about it. They just get directed.

That's a load balancer. It sits in front of your servers and distributes incoming traffic so no single server carries everything. Nginx handles this well.

But now the person at the entrance is a single point of failure. They go down, everyone is lost.

So you have a backup person standing right next to them. If the first one goes down, the backup steps in automatically. That switch is called failover. Running both is a high availability setup.

Rate limiting sits on top of this. You cap how many orders one person can place per minute. It won't solve legitimate volume but it stops bad actors from making the problem worse.

Something goes wrong and you have no idea which spot caused it

At 10,000 transactions per second, a failed transaction could have broken at the entrance, the order taker, the payment handler, the records person, or the notification sender. Logs scattered across twenty servers.

I genuinely didn't think about this until I asked myself — if something breaks at this scale, where do I even start looking?

The fix is to give every order a unique ticket number the moment it enters the system, then write that ticket number into every record that order touches across every spot.

When something breaks, you search for that ticket number and see the full journey of that order in one place. Every step, every spot, in order.

That ticket number is called a correlation ID. Following a request across services using it is called distributed tracing.

Tools like Datadog or the ELK stack pull logs from all your servers into one searchable place. Elasticsearch stores the logs. Kibana is the dashboard you use to search them.

What the full picture looks like

A request comes in. The load balancer directs it to one of several servers, with a standby ready if the active one dies. Rate limiting filters abuse. The server processes the transaction with pessimistic locking to prevent race conditions. Reads hit replicas, writes hit the primary. Sharding distributes write load across multiple database nodes. Cross-spot transactions use SAGA so failures don't leave money in limbo. Every step logs with a correlation ID that flows into a centralized system where any transaction can be traced end to end in seconds.

That's a payment system that doesn't lose money.

I'm Damola, a backend engineer. Find the rest of this series on GitHub. Follow me on Dev.to for the next one.