How Subscription Billing Systems Actually Work

There’s a footnote in Stripe’s documentation that I think about more than is healthy. It’s explaining how long a card authorisation stays valid before the money goes back, and for Visa it gives the number as five days, with an asterisk. The asterisk says the exact window is four days and eighteen hours, to allow time for clearing.

Four days and eighteen hours. Not a round number, not a number you chose, not a number anyone at your company has ever discussed. It’s a deadline handed down by a card network, measured in hours, and if your system misses it the reserved funds are released and the payment’s status quietly becomes canceled. Nobody clicked cancel. Nothing errored. A clock you don’t own ran out.

That’s the shape of the whole problem. A billing system looks like a request handler with a database behind it, and for about the first fortnight it is one. Then you meet the renewals, and the retries, and the refunds that get reversed, and the users who move country, and you realise you haven’t built an API. You’ve built a very large number of small state machines that run for years, mostly while everyone is asleep.

I worked on billing infrastructure for YouTube, which is where my instincts about this come from, and it taught me mainly that the payment is the least interesting part. Everything I cite below is public, though: vendor documentation, published papers, and other people’s engineering write-ups. Nothing here needs inside knowledge, which is rather the point, because all of it is sitting in documentation that anybody can read and almost nobody does.

Here’s what we’re going to go through. First, why the obvious tool, a transaction, doesn’t reach far enough, and what the literature has been saying about that since 1987. Then the states themselves: what the real ones look like in published APIs, and which of them contradict their own names. Then the thing I think is the single most common design error in this area, which is storing one fact where there are two. Then the machinery that makes any of it survive contact with reality: idempotency, intent records, durable orchestration, and one workflow per kind of transaction. Then the ledger, which is the only part of the system allowed to be certain about anything. And finally, how you’d know whether any of it works, which turns out to be a much better question than how often it fails.

None of this needs a payments background to follow. If you’ve ever had a retry do something twice, you already know the hard part.

The thing you’re building isn’t a transaction

Start with the obvious design, because it’s worth being precise about why it fails.

You want four things to happen together: charge the card, record the payment, grant the entitlement, and tell the user. It would be lovely to wrap those in BEGIN and COMMIT. You can’t, because one of them happens inside a bank you don’t own, over a network, on a timescale of seconds to days, and the bank has never heard of your transaction manager.

Pat Helland wrote the position paper on this in 2007, from inside Amazon, and it’s aged extremely well. His argument in Life beyond Distributed Transactions is that large applications simply don’t get built on distributed transactions, because the performance costs and the fragility make them impractical, and that the honest thing to do is design for it. So you get entities, each of which is a scope inside which you can commit atomically, and between entities you get messages. Messages get retried. Messages arrive out of order. The application has to cope.

The useful move in that paper is that it makes uncertainty a thing you store rather than a thing you avoid. Between “I asked the bank” and “the bank answered”, your system is in a state that is neither charged nor not-charged, and that state needs a name, a row and a timeout. Most billing bugs I’ve seen are a system that had no name for that state and picked one of the two neighbours at random.

The older idea underneath this is the saga, from Hector Garcia-Molina and Kenneth Salem at Princeton in 1987.¹¹Garcia-Molina and Salem, Sagas, ACM SIGMOD 1987. The paper’s running example is booking seats on a flight, which is still the example everyone uses, because it’s still the clearest one. Their problem was long-lived transactions: work that holds database resources for hours or days and blocks everything shorter. Their answer was to break the long transaction into a sequence of smaller ones, and give each of them a compensating transaction that can be run if the sequence has to be abandoned partway.

And there’s the sentence that matters, which is easy to read past. The compensating transaction undoes the effects of its partner from a semantic point of view, but it does not necessarily return the database to the state that existed before. It isn’t a rollback. It’s a second, forward-moving action that makes the books balance.

That distinction is the whole discipline in miniature. You can’t un-charge a card; you refund it, and both the charge and the refund appear on the customer’s statement, and they will email you about it. You can’t un-send the receipt; you send a second email that says sorry. You can’t un-grant access that somebody already used; you revoke it going forward and decide, as a policy question rather than a technical one, whether to claw anything back.

So the design question stops being “how do I make these four things atomic” and becomes “in what order do I do them, and what does each one’s apology look like”. Which means the first thing you need is an honest list of the states.

Draw the states, then count the ones you forgot

Ask an engineer to describe a subscription and you’ll usually get three states: active, cancelled, and something fuzzy in the middle for when the card fails. Ask a published API and you get a rather different answer.

Stripe documents eight subscription statuses: trialing, active, incomplete, incomplete_expired, past_due, unpaid, canceled and paused. Google Play’s real-time developer notifications carry sixteen subscription notification types, and they are numbered one to twenty-two, with six holes in the sequence. That enum is a fossil record: every gap is an event that used to exist, and the list has grown before and will grow again. Apple’s App Store Server Notifications define more than twenty types, many of which change meaning depending on a subtype field, so the real cardinality is higher still.

Three vendors, three state machines, no two of them the same shape. And you have to build a fourth one that consumes all of them.

Now the part I’d put on a poster. Stripe’s own documentation, in the row of the table describing active, says that active doesn’t indicate that all outstanding invoices on the subscription have been paid. You can leave invoices open, mark them uncollectible, or void them, and the subscription sits there saying active throughout.

Read that again if you build these systems. The state named after the thing you care about is not the thing you care about. If your entitlement check is if subscription.status == 'active', you have written a bug, and it’s a bug that only shows up on the accounts where somebody in finance did something reasonable.

There are more of these than you’d like:

• A Stripe subscription that’s waiting on its first payment sits in incomplete, and the customer has twenty-three hours to pay. After that it becomes incomplete_expired and you have to create a new subscription, because the old one is never coming back.
• Whether a failed renewal lands in past_due or unpaid depends on a setting in the Stripe Dashboard. The same event, the same code, a different state, decided by a dropdown in a web console that your service has no visibility into.
• A Google Play subscription that you revoked through the API reports subscriptionState as SUBSCRIPTION_STATE_EXPIRED. Not revoked. Expired. The state machine that produced the event has more states than the one you get to read, so “why did this end” is a question you can only answer from the notification stream, not from the current resource.

That last one generalises into a rule worth keeping: the current state is a lossy summary, and the event log is the actual data. Anywhere you find yourself reconstructing history by comparing two snapshots, you’ve lost something that was in the stream and is now gone.

Which brings up the question of what you should have been storing instead, and the answer is at least one more bit than you think.

Being charged and being entitled are different facts

Here is the design error I’d bet on finding in any billing system that’s under two years old. There’s a column called something like is_active, or subscription_status, and the code uses it for two different questions: is this person paid up? and should this person be able to watch the thing?

Those are independent. Not loosely coupled, not usually-the-same. Independent, with all four combinations occurring in normal operation and all four documented by vendors as things you must handle.

Paying and entitled is the boring one, and it’s the only one most systems model properly.

Not paying but entitled is the grace period. When a renewal is declined, Google Play notifies the user and keeps retrying while the subscription sits in SUBSCRIPTION_STATE_IN_GRACE_PERIOD, and the documentation is explicit that during a grace period the user should retain their benefits. Apple’s equivalent is a DID_FAIL_TO_RENEW notification with the subtype GRACE_PERIOD, and its instruction is to continue providing service. You are giving away the product on purpose, because the alternative, cutting off a good customer whose card expired, is worse.

Paying but not entitled sounds impossible until you meet deferred billing, prepaid plans, and the window between an authorisation and the delivery of whatever was bought. It’s also where every “I paid and nothing happened” support ticket lives.

Neither splits into at least three genuinely different situations that people confuse constantly: expired, paused, and on hold. A paused Play subscription has no entitlement and no charge, but the user is still a subscriber and will come back automatically at the end of the pause. A subscription in account hold has no entitlement and an unresolved payment, and the docs note it isn’t even returned by queryPurchasesAsync(), so a client that checks entitlement by listing purchases can’t distinguish “on hold” from “never subscribed”.

And then there are the cases where the two facts belong to different people. Apple sends a REVOKE notification when an in-app purchase stops being available through Family Sharing, which happens when the purchaser turns sharing off, or when the purchaser or a family member leaves the group. Somebody who never paid, never cancelled and never did anything at all loses access, because of an action taken by a different account. If entitlement is a column on the payer’s subscription row, there is nowhere to put that.

The nastiest one is Apple’s REFUND_REVERSED. The App Store granted a refund, the customer disputed it, and the App Store has now taken the refund back. If you revoked content when the refund arrived, you have to reinstate it. So revocation is not a terminal state, and the money for a single purchase can move three times: out, back, and out again.

Two bits, then, not one, and neither of them is a boolean over the long run. They’re both timelines. Once you’ve accepted that, the next surprise is that the transitions between them are timed by clocks that behave differently depending on where you came from.

The same fix, two different clocks

This is my favourite detail in the whole of the published billing documentation, and both halves of it are bullet points in the same Google page, two sections apart.

A user’s renewal fails. They enter a grace period. They update their card. The subscription renews with its original renewal date, and the clock does not reset.

A user’s renewal fails. The grace period passes. They enter account hold. They update their card. The subscription recovers, and the renewal date resets.

Identical user action. Identical UI. Identical intent. Two different billing dates from here to eternity, decided entirely by which state the subscription was sitting in when the card got fixed. If you’re computing the next charge date yourself, from the previous one plus a month, you are wrong for every user who recovered from hold, and you will not find out until the amounts stop matching.

The neighbourhood is full of clocks like this:

• Before account hold begins, Play makes additional charge attempts for up to 48 hours, and the user keeps their benefits through it.
• Apple keeps retrying billing for 60 days, or until the customer fixes it or cancels, whichever comes first.
• Stripe’s Smart Retries picks the retry times with a model rather than a fixed schedule, and one of its documented inputs is how many different devices have presented that payment method in the last few hours. Your dunning calendar is a prediction somebody else is making about your customer.

And then the one that gets people. You can set your Play grace period to zero days, and Play will still wait a minimum of one. During that silent grace period the subscription stays ACTIVE, you receive no notification, and the retries carry on without you. It is a real state, lasting a full day, that is invisible from outside and that your system will never be told about.

All of which assumes the notifications arrive, and arrive once. They don’t.

Who is allowed to say the money moved

Every message in this system can arrive twice. Webhooks retry. Clients retry. Your own workers retry after a deploy kills them mid-flight. The card network retries. And at least one of those retries will land on the code path that charges people.

The standard answer is an idempotency key, and it’s a good answer, but the details are where it earns its keep. Stripe’s implementation is worth copying almost exactly:

• The server saves the status code and body of the first request for a given key, whether it succeeded or failed. A repeat gets the same answer back, including a replayed 500. That’s deliberate: a client that retries into a cached error is in a defined state, whereas a client that retries into a fresh attempt is gambling.
• If the same key arrives with different parameters, you get an idempotency_error rather than either outcome. The key is a claim about identity, and a changed payload means the claim is a lie.
• Keys are pruned after at least 24 hours, and a key reused after pruning generates a brand new request.

That third one deserves a pause, because it’s where idempotency and billing stop agreeing with each other. A dunning schedule runs for days. Apple’s runs for sixty. If your protection against charging twice is a request-level key with a one-day memory, then a retry on day three is not a retry, it’s a new charge, and everything about it looks legitimate.

So there are two different keys in a well-built billing system, and conflating them is a real bug:

• A request key, per attempt, with a short life, protecting against network-level duplication. That’s the Idempotency-Key header.
• A business key, derived from what the charge is rather than from the call that made it: this subscription, this billing period, this amount. It lives as long as the subscription does. It’s the thing that stops you charging September twice, no matter how many days apart the two attempts are or how many services initiated them.

The AWS Builders’ Library has the cleanest explanation of why the business key has to be supplied rather than inferred. You could hash the request parameters and call identical requests duplicates, and for something like creating a database table that’s probably right. But for launching an EC2 instance it isn’t: two identical launch requests might be somebody who genuinely wants two instances. The system cannot tell duplication from intent, so the caller has to say which it is.

Billing has exactly this problem and people rarely notice. Two identical charges of £9.99, one second apart, on the same card: double-submitted button, or somebody buying two gift subscriptions? Only the caller knows. Deduplication is a statement of intent, and intent can’t be derived from a payload.

Airbnb published the best write-up I’ve found of a general idempotency layer for payments, and its central rule is about shape rather than keys. Their framework splits every API request into three phases: Pre-RPC, RPC and Post-RPC, with two hard constraints. No network calls in the Pre and Post phases. No database calls in the RPC phase.

The reason is unglamorous and completely convincing: the Pre and Post phases are wrapped in database transactions, and holding a database transaction open across a network call is how you exhaust a connection pool during exactly the incident where you need it most. But the rule is worth more than its reason, because it forces the shape that everything else in this post depends on. Write down what you’re about to do. Go and do it. Write down what happened. Never blur the three.

That first step is doing more work than it appears to.

Write down what you’re about to do, before you do it

Consider the smallest possible version of the dual-write problem. You charge the card, the charge succeeds, and your process dies before it writes the row. The money is gone. Your database has never heard of it. Nothing in your system will ever reconcile this, because reconciliation works by comparing your records against the processor’s, and you have no record to compare.

Now invert the order. Write an intent row first, in your own database, in its own transaction: I am about to charge this subscription, for this period, this amount, under this key. Then make the call. Then write the outcome.

If you die after the intent and before the call, a sweeper finds an intent with no outcome and asks the processor what happened, using the key. If you die after the call and before the outcome, same thing, same answer. The intent row turns an unknown into a question with an address on it.

This is the transactional outbox pattern wearing different clothes, and it’s the reason the pattern keeps reappearing: you cannot atomically commit to your database and to something that isn’t your database, so you commit to your database twice and let a separate process carry the message across the gap.

The financial version of the same idea is older and better specified. In double-entry systems you don’t move money in one step, you make a two-phase transfer. TigerBeetle’s documentation describes it precisely: a pending transfer reserves the amount in debits_pending and credits_pending, leaving the posted balances untouched, and is later resolved by being posted, voided, or expired.

Three resolutions, not two. That third one is the one people leave out. A pending transfer created with a timeout that nobody resolves expires on its own and returns the full amount, without anybody making a decision or handling an error.

The timeout is a transition, not a failure. Go back to the card authorisation at the top of this post: four days and eighteen hours, and then the funds are released and the status becomes canceled. That isn’t an error path. It’s a labelled edge in the state machine, with a duration on it, and if your design has no such edge then the expiry happens anyway and your system just doesn’t know.

So every step is: intent, action, outcome, with a deadline attached and a third resolution for when the deadline wins. Now you need something to run thousands of these at once, for years, across process restarts.

The orchestrator has to outlive its own process

A checkout takes seconds. A subscription takes years. Somewhere in between, the abstraction has to change from “a request being served” to “a program that’s still running”.

The name for the modern version of this is durable execution, and Temporal is the clearest implementation to reason about. You write the workflow as ordinary code in an ordinary language. Every decision it makes is recorded to an event history. If the process running it dies, a new worker fetches that history, replays the code against it to rebuild local state, and carries on from where it stopped. A workflow can wait on a durable timer for minutes or months, which is precisely the primitive a grace period needs.

The price is a constraint that sounds trivial and isn’t: the workflow code must be deterministic, because it gets re-executed against its own history. No wall clock, no random numbers, no reading the outside world. Anything that touches anything external belongs in an activity, whose result gets recorded so the replay can use it.

Sit with that for a second in this context. The one kind of program that most needs a calendar is the one kind of program forbidden from asking what day it is. You don’t call now(); you ask the orchestrator for a timer and let the fact that it fired become an event in the history. Which is, I’d argue, the correct discipline anyway. A billing workflow that reads the system clock is one daylight-saving transition away from being interesting.

Uber built Cadence for the same reasons and open-sourced it; by the 1.0 release in 2023 it was being used by over a thousand services internally. Netflix has Conductor. The pattern is thoroughly settled, and if you’re building this today the interesting question isn’t whether to use a durable orchestrator but which of two shapes you want.

Chris Richardson’s summary is the standard one: a saga can be coordinated by choreography, where each service publishes events that trigger the next, or by orchestration, where one component tells the participants what to do. Choreography is lovely for throughput and genuinely awful for billing, because there’s no single place that can answer “what is this subscription doing right now, and why”. In an orchestrated system, the answer is a row. In a choreographed one, the answer is a distributed tracing exercise conducted under pressure at 2am while somebody from finance watches.

Richardson also names the cost honestly, and it’s the one people underestimate: sagas give up the I in ACID. There’s no isolation between concurrently running sagas, so a developer has to design countermeasures by hand. In practice, for billing, that means a lock on the subscription for the duration of any workflow that changes its state, and a hard rule that only one such workflow may be in flight at a time. A refund arriving while an upgrade is halfway through is not a rare event. It’s a Tuesday.

There’s one more shape difference worth naming, because it catches people coming from request-driven systems. Most billing work is not triggered by anybody. A renewal happens because a date passed. The thing that notices is a recurring job on a cluster manager, sweeping for subscriptions whose period has ended, the same sort of workload Borg was built to run and that every scheduler since has copied.²²Verma, Pedrosa, Korupolu, Oppenheimer, Tune and Wilkes, Large-scale cluster management at Google with Borg, EuroSys 2015. The paper is about the cluster manager rather than about billing, but the shape it describes, long-running jobs with declarative specs and automatic restarts, is exactly what a renewal sweeper wants underneath it.

Which means your busiest code path has no user watching it, no session to fail back to, and no obvious place to show an error. That’s worth designing for deliberately, and it’s why the next question is which workflow the sweeper should actually start.

One workflow per kind of transaction

There’s a tempting design where every billing event runs through one big workflow with a lot of branches, because after all they share so many steps. It’s a trap, and the reason is that the steps they share aren’t the same steps.

Take proration. A new purchase has none. An upgrade has it, and the amount depends on how much of the period is left. A downgrade might have it, or might be deferred to the next period, depending on product policy. A refund has a negative one. Same word, four different computations, four different sets of things that can go wrong. Branch on it inside one workflow and you get a function with a five-level conditional at the top that nobody will ever safely change.

Split by the nature of the transaction instead, and each workflow gets to be short and legible:

A new purchase is the only one with a human waiting. It’s latency-sensitive, it can fail fast and tell somebody, and it ends with the step that surprises people. On Play you have three days to acknowledge the purchase, and if you don’t, it is automatically refunded and the entitlement revoked. Read that as what it is: your success path has a deadline on it too, and missing it doesn’t raise an error, it silently undoes the sale.

A renewal has nobody waiting, runs from a sweep, and is allowed to take days because of dunning. It’s the one that needs the business-level idempotency key most, since its retries span far more than any request key’s lifetime.

A plan change is two subscriptions in a trench coat for the duration. Both the old and the new have to be representable at once, with a defined instant where entitlement moves. Apple models the intent separately from the effect: DID_CHANGE_RENEWAL_PREF says what the customer has chosen for the future, and if the subtype is empty it means they’ve changed their preference back to the current plan, effectively cancelling a downgrade they’d previously scheduled. A scheduled future state that can be un-scheduled is itself a state.

A revocation runs backwards through everything the purchase did, and it’s the one workflow that must be able to run against a subscription in any state at all, including states you’d have called terminal. Google’s Voided Purchases API exists specifically so you can build one: it lists orders voided by refunds, cancellations, chargebacks or developer-initiated revokes.

And it has a detail in it that should change how you staff your on-call. The API only returns purchases voided in the past 30 days. Older ones aren’t included, whatever start time you ask for. So the feed that tells you which customers to cut off has a thirty-day memory, and a reconciler that’s been broken for thirty-one days hasn’t fallen behind, it has permanently lost data. That moves the reconciler from “batch job somebody owns” to “tier-one service with an alert on staleness”.

A region change is the one everybody underestimates, and it’s worth walking through because it shows how much of this is policy rather than plumbing. A Play base plan carries a list of regional configurations, each with its own price, so moving a user between regions isn’t an update to an address field. It’s a different price, in a different currency, under different tax treatment, possibly with a different set of available plans, and a live entitlement that mustn’t flicker while you work it out.

Which turns into a price change, and price changes are governed too. Play requires advance notice of at least 30 or 60 days depending on the country, and notes that the notification period may cause additional renewals at the old price. Opt-out increases are capped at one per base plan per country per 365 days, and the maximum is the greater of 50% or 17 US cents per day, converted to local currency.

Seventeen US cents per day. I love that number, because you cannot arrive at it from first principles and no amount of good architecture will let you skip it. Somewhere in your system there has to be a rule engine that knows it, per country, with an effective date, and a history of the last year’s increases to check against.

Every one of these workflows writes to the same place, and that place has to be stricter than any of them.

The ledger is the only thing that’s true

Everything described so far can be wrong. Notifications get lost, workflows have bugs, someone runs a backfill with a bad filter, a vendor changes a state’s meaning in a minor version. If the only record of what happened is the state machine’s own opinion of itself, then when it’s wrong it’s confidently wrong, and there’s nothing in the system capable of noticing.

So the money gets recorded separately, in a structure with different rules. Double-entry bookkeeping long predates anything with a CPU in it, and it earns its keep here for one reason: every movement touches two accounts in opposite directions, so the sum across all accounts is always zero, and any single-sided mistake shows up as a non-zero total rather than as a plausible-looking balance.

TigerBeetle’s design notes are a good short read on what that looks like as an actual database rather than as an accounting convention: transfers between accounts, enforced financial consistency, strict serializability of events, and balances that are derived rather than stored as an editable number. The important architectural claim is the split. Your general-purpose database holds entities, metadata and workflow state. The ledger holds movements, and nothing mutates a movement after the fact, ever. A correction is a new entry.

Then reconciliation, which is the boring job that justifies all of it. Every day, pull the processor’s record of what moved, compare it against the ledger, and produce a list of disagreements. Not a metric. A list, with subscription identifiers on it, that a human can work through. The value isn’t in the count being zero, it’s in the list being short enough that somebody looks at it.

Everything up to here is true of a billing system with a thousand subscribers on it. Same shape, same states, same two keys. What changes when there are hundreds of millions is not the shape, and it isn’t throughput either.

Three problems you only meet at scale

Throughput is the easy part of scale, and it’s the part everyone prepares for. You shard by subscription, you add workers, the graph goes flat and you move on.

These three are different. In each one the standard distributed-systems answer is either unavailable or actively makes things worse, which is what makes them worth writing down. They’re also the ones I didn’t see coming, and between them they’re most of why I think very large scale billing is a distinct discipline rather than ordinary systems work with money in it.

The renewal calendar is a comb, and you can’t jitter it

Subscriptions renew on their anniversary. That sounds like it spreads load out, and at small numbers it does. At large numbers it does the opposite, because the distribution of anniversaries isn’t something you chose. It’s the shape of your acquisition history, played back at you once a month, forever.

Every launch, every campaign, every free-trial cohort that converted together leaves a tooth in that comb. A promotion that signed up a few million people over one weekend three years ago is still a load spike today, on that weekend’s day-of-month, and it will be one until those subscriptions churn.

Then the calendar makes it worse in a way that’s easy to miss. Months aren’t the same length, so anchors have to be clamped. Stripe documents the rule plainly: a monthly subscription anchored to January 31 bills on the last day of the closest month, so February 28, then March 31, then April 30. Follow that through. Everyone who signed up on the 29th, the 30th and the 31st collapses onto a single day every February. Three days of acquisition, landing at once, annually, for the life of those accounts.

The first of the month is worse still, because it collects trial conversions and anything anyone ever nudged onto a tidy date.

Here’s what makes it a distributed systems problem rather than a capacity one. The universal answer to a thundering herd is jitter: spread the work over a window and the spike disappears. You cannot jitter a billing date. Moving a charge by a day changes the period it covers, changes the proration, changes when entitlement lapses, and in some jurisdictions changes the notice you owed before charging. The one lever every other system reaches for is bolted down, because in this system the timestamp isn’t telemetry. It’s the product.

So you jitter everything except the instant of record. Resolve the price, the tax and the invoice hours before the boundary and park them. Reserve the business idempotency key in advance, so the boundary is doing an authorisation and nothing else. Spread the attempt across the day while keeping the period exact to the second. And decouple the entitlement decision from the charge entirely, so that a renewal still grinding through a queue at 00:04 doesn’t cut anybody off.

The line I’d keep: your peak load is a property of your sales history rather than your traffic. It’s the only load curve you can forecast years ahead and the only one you’re not allowed to flatten.

A load spike at least has the decency to show up on a graph. The next problem is one where the system that’s failing you uses the same word for “no” and for “not from you, not right now”.

Dunning is congestion control, and the congestion signal is the error

Retrying a declined payment is free at small scale. At large scale you are pointing a firehose of authorisation attempts at a fairly small number of issuing banks, and they notice.

Stripe says the consequence out loud in its own documentation: card issuers might see additional retries as potential fraud, which can result in increased declines. Read that as a control loop. The mechanism you built to recover failed payments degrades the success rate of the payments it’s recovering. Push harder, get less. That’s the classic congestion-collapse shape, and the classic answer is backoff.

Except the feedback you need for backoff isn’t there, and this is the part I find genuinely hard. In an ordinary congestion problem you can tell a dropped packet from an application error. Here you can’t. Everything comes back as a decline code, and the most common one means, in effect, no. Insufficient funds, an issuer’s fraud model, and your own retry volume all arrive wearing the same clothes. The capacity signal and the payload error are the same symbol.

What you get instead is advice, in band, from the counterparty. Mastercard’s merchant advice codes are a backoff ladder with the numbers already filled in: 24 means retry in an hour, 25 in 24 hours, 26 in two days, 27 in four, 28 in six, 29 in eight, 30 in ten. Codes 03 and 99 mean never come back. Visa’s categories are coarser and say whether the issuer never approves this account, can’t approve right now, or wants you to fix your data before asking again.

Sit with that for a second, because it inverts something. Your exponential backoff has an exponent, and the exponent is chosen by the other side and delivered inside the failure.

Two consequences follow, and the second one is the awkward one.

First, the limits are enforced rather than advisory. The networks cap how many times you may reattempt a single charge, and Stripe’s own guidance is a maximum of eight on a charge that permits retries at all. Some permit none: Visa’s first decline category means the issuer will never approve this account, and Mastercard’s 03 and 99 say the same thing in fewer characters. Going past any of it costs real money, which you can infer from the fact that Stripe ships a feature whose stated job is helping you avoid excessive retry penalties. When a payments company builds a product to stop you doing something, the something has a price list.

Second, and this is the bit that breaks a naive design: your retry scheduler cannot be per-subscription. The contended resource is a particular bank’s tolerance for you, and that tolerance is shared across every subscriber whose card that bank issued. Two subscriptions with nothing to do with each other, different countries, different plans, different everything, are contending for the same budget because of six digits at the front of a card number. So the scheduler has to be global, keyed on the issuer, with a rate budget per issuer, and it has to hold that budget across a fleet of workers that would much rather each make their own decision.

You are doing congestion control on a channel that won’t tell you it’s congested, against a counterparty that fines you for probing.

Both of those are about money arriving. The last one is about money leaving, which turns out to be the harder direction.

You have to pay out a share of money that can still be taken back

The first two problems exist for anyone with enough subscribers. This one is specific to platforms that split the money, and it’s the one I’d call genuinely novel.

On YouTube, a Premium membership isn’t only revenue. Google shares the membership fee with creators, and how much each creator gets depends on how much of their work that member actually watched. So one inbound payment fans out into a large number of outbound ones, and the shape of the fan-out is not known when the payment settles. Payments to creators go out at the start of each month.

Two facts make that hard, and neither is the one people expect.

The allocation key arrives late, from devices you don’t control. Premium members can download videos and watch them offline. YouTube’s documentation is explicit about what happens next: watch time for downloaded videos is recorded and incorporated the next time the member signs in online. Which might be tomorrow. It might be after a fortnight somewhere with no signal. So the denominator you’re dividing the money by isn’t final when the period closes. It converges afterwards. There is a phone in a bag somewhere holding evidence about how to divide money you have already collected, and it will hand that evidence over when it feels like it.

The inbound payment is reversible for far longer than the outbound one. Card networks typically let a cardholder dispute a payment within 120 days, sometimes longer, and a chargeback reverses the payment immediately and pulls the money back. The payout it was split into went out months earlier, to hundreds of people, and it is not coming back on its own.

So you have an irreversible transfer sitting downstream of a reversible one, with a join key that hasn’t finished arriving. Every obvious fix is bad. Hold the money until every window has closed and you’re sitting on people’s earnings for a third of a year. Pay immediately and reverse on dispute and you’re sending creators invoices for money you already gave them. Ignore it and you have quietly decided to absorb the difference, which works right up to the scale where it doesn’t.

The answer isn’t a clever protocol, it’s a change to what the words mean. The ledger has to treat attribution and payment as separate, revisable facts: an amount attributed to a creator for a period, an amount actually paid against that attribution, and the difference between them as a balance that can go negative and be recovered from future earnings rather than clawed back. Which is a distributed transaction whose rollback is deduct it from the next one, running across counterparties who can close their accounts and walk away mid-transaction.

That’s a long way from BEGIN and COMMIT. It’s also, I think, the clearest example of the thing this whole post is about: the hard part was never moving the money. It was agreeing, across systems and over months, on what the money meant.

That reframing, what a number actually means, is the right note to end on, because there’s one more number worth being careful about: the one everybody quotes.

What a failure rate even means here

The instinct to put a number on reliability is right, and the usual number is availability. Google’s SRE book frames it as nines, and makes a point most people skip: the target is a minimum and a maximum. Being more reliable than you need to be means you spent budget you could have spent on features. The book even notes that a user on a 99% reliable phone can’t distinguish 99.99% from 99.999% service, so the extra nine bought nothing anybody can perceive.

That framing is excellent for a serving system and it doesn’t transfer cleanly to billing, for one specific reason. An availability failure is transient by construction and a billing failure isn’t. A dropped request costs the user a retry. A charge without an entitlement, or an entitlement without a charge, is a divergence that will still be there tomorrow, and next month, and at the end of the financial year, because nothing in the system is trying to converge it. It doesn’t decay. It accrues.

Which means a failure rate is the wrong instrument, and not because it’s too crude. Because it measures the wrong noun. It counts events, and the thing that hurts you is states.

Three measurements I’d rather have than a percentage:

Divergences currently open, and their age. How many subscriptions are right now in a combination your model says is impossible: charged and not entitled, entitled with no successful payment and no grace period, two active subscriptions for one account, a pending transfer older than its timeout. This is a gauge, not a counter, and it should be near zero with an alert on the age of the oldest, because an old divergence is one that nothing is fixing.

Time to detect. Given a deliberately injected divergence, how long until something automated notices? If the answer is longer than your Voided Purchases window, you have an upper bound on your own correctness that no amount of uptime will improve.

Contacts per million subscription-days. How many humans had to get in touch because the billing was wrong. It is lagging, noisy, impossible to game and the only one that measures the actual harm.

That last one is the honest reframe. A rate like 0.001% sounds like a statement about engineering quality, but it’s really a statement about volume: at ten million transactions a month it means a hundred people a month had a bad experience you caused, which is a support load, a churn number and a regulatory exposure rather than a triumph. The interesting claim was never the rate. It’s that the divergences are found, and found by the system rather than by the customer, and that the queue of open ones stays short.

Say that instead. It’s a stronger claim and a much harder one to make, and it’s the reframing I’d argue for on any billing team I’ve sat on.

Which leaves the question of how you find the impossible states before they’re in production.

Checking it before it ships

You can enumerate a state machine. That’s the nice thing about having built one deliberately: it’s finite, and small enough to check exhaustively, which is not true of the code that implements it.

The cheapest version costs an afternoon. Make a table with your states down one side and your events across the top, and fill in every cell. Not the plausible ones. Every cell. What happens when a refund arrives for a subscription that’s already expired? When a renewal succeeds for one you cancelled ninety seconds ago? When a REFUND_REVERSED lands on an account you already revoked and whose entitlement you already deleted?

The blank cells are your incident list for next year, roughly in order. Every one of them is a case where the code will do something, and what it does is currently an accident.

The rigorous version is model checking, and the case for it in industry was made convincingly by the AWS team who wrote How Amazon Web Services Uses Formal Methods in 2015. Their finding, after applying TLA+ to DynamoDB, S3 and others, is that design reviews, code reviews, static analysis, stress testing and fault injection are all necessary and none of them are sufficient, because testing the code can’t explore the design’s state space: the number of reachable states is astronomical.

The line I keep coming back to is their diagnosis of why. Human intuition is poor at estimating the true probability of supposedly rare combinations of events in a system running at millions of requests per second. That’s not a comment about carelessness. It’s a comment about arithmetic. At a million requests a second, the thing you correctly judged to be a one-in-a-billion interleaving comes round about every seventeen minutes, and you were not wrong about the odds. You were wrong about how many times the dice get rolled.

I want to be honest about the limits here, because formal methods get oversold. Model checking checks the model. It says nothing about whether your implementation matches it, whether the vendor’s state machine is what their documentation claims, or whether the invariants you asserted are the ones that matter. It’s a very good way to discover that two of your workflows can interleave into a state you never named. It is not a proof that you’ll never charge anyone twice.

But the invariants themselves are worth writing down whether or not you ever check them mechanically, because they double as the reconciler’s queries. Mine would start:

• No entitlement without either a settled payment, an explicit grant with an expiry, or an active grace period.
• No two settled payments sharing a business key.
• Every pending transfer resolves: posted, voided or expired. None older than its timeout.
• Every state transition has a recorded cause, and that cause is an event, not a deploy.
• Entitlement revocation is reversible, because REFUND_REVERSED exists.

Those five sentences are simultaneously a specification, a test suite and a monitoring configuration, which is a decent return for something you can write on an index card. They’re also about as much as anyone can usefully say in the abstract. The rest is the order you do it in.

Where I’d start

If you’re building one of these from scratch, the order I’d go in is roughly the order of this post, which is not the order anyone actually goes in.

Draw the state machine first, on paper, including entitlement and payment as two separate rows rather than one. Get the transitions from the vendors’ own documentation rather than from your product spec, because the vendors’ states are the ones that will actually arrive. Fill in the grid and argue about the blank cells.

Then build the ledger, before the workflows. It’s the thing you can’t retrofit honestly, because a ledger that starts in year two can only tell you about year two.

Then pick a durable orchestrator and accept the determinism constraint rather than fighting it. Then write one workflow per transaction type, short, with a lock on the subscription so only one can run at a time.

Then the two idempotency keys, and be clear in the code which is which, because the day somebody uses the request key where the business key belongs is the day you charge a month twice.

Then the reconciler, and treat it as a tier-one service with an alert on its own staleness, because your revocation feed may well forget things after thirty days.

And then the part that isn’t engineering at all. Go and find out what the rules are in each country you sell in, because there’s a cap somewhere measured in cents per day, and a notice period measured in days, and a clearing window measured as four days and eighteen hours, and none of them will be discovered by writing better code. They get discovered by reading, and then they get encoded, and the quality of the system is mostly a function of how many of them you found before your customers did.

Sources

Documentation

• Apple, notificationType, App Store Server Notifications, 2025. The notification types and their subtypes, the sixty-day billing retry window, REFUND_REVERSED and Family Sharing REVOKE.
• Google, Subscription lifecycle, Google Play Billing, 2025. Grace period, account hold, the silent grace period, the two recovery clocks, pause, defer and revocation.
• Google, Real-time developer notifications reference, 2025. The numbered subscription notification types.
• Google, Integrate the Google Play Billing Library into your app, 2025. The three-day acknowledgement deadline and what happens if you miss it.
• Google, Understanding subscriptions, Play Console Help, 2025. Regional configurations, price increase notice periods and the opt-out increase caps.
• Google, Voided Purchases API, 2025. The revocation feed and its thirty-day window.
• Stripe, Idempotent requests, API reference, 2025. Cached responses including errors, parameter matching, and the twenty-four-hour key lifetime.
• Stripe, How subscriptions work, 2025. The eight statuses, the twenty-three-hour incomplete window, and what active does and doesn’t mean.
• Stripe, Place a hold on a payment method, 2025. Authorisation validity windows by network, including the four days and eighteen hours.
• Stripe, Automate payment retries, 2025. Retry scheduling signals and non-retryable declines.
• Stripe, Card declines, 2025. Why retries can raise your decline rate, and the recommended cap of eight.
• Stripe, Declines, 2025. That excessive retry penalties exist, and that avoiding them is a product.
• Stripe, Set the subscription billing renewal date, 2025. Billing cycle anchors and how short months clamp them.
• Stripe, How disputes work, 2025. The 120-day dispute window and what a chargeback does to a settled payment.
• Visa, Association response category and decline codes, 2025. Visa’s decline categories and Mastercard’s merchant advice codes, including the retry-after ladder.
• YouTube, How YouTube Premium supports creators and Your content & YouTube Premium, 2025. That membership fees are shared by watch time, that payouts go monthly, and that offline watch time is counted when the member next signs in.
• Stripe, Designing robust and predictable APIs with idempotency, Brandur Leach, 2017.
• Amazon Web Services, Making retries safe with idempotent APIs, Malcolm Featonby, Builders’ Library. Why client-supplied tokens beat inferred ones.
• Temporal, Understanding Temporal, 2025. Durable execution, event history, replay and the determinism constraint.
• Uber, Announcing Cadence 1.0, 2023.
• TigerBeetle, Two-phase transfers and TigerBeetle in your system architecture, 2025.
• Chris Richardson, Pattern: Saga and Pattern: Transactional outbox, microservices.io.
• Jon Chew, Avoiding double payments in a distributed payments system, Airbnb Engineering, 2019.
• Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), Embracing Risk, Site Reliability Engineering, O’Reilly, 2016.

Research

• Hector Garcia-Molina and Kenneth Salem, Sagas, ACM SIGMOD Record, 1987.
• Pat Helland, Life beyond Distributed Transactions: an Apostate’s Opinion, CIDR, 2007.
• Chris Newcombe, Tim Rath, Fan Zhang, Bogdan Munteanu, Marc Brooker and Michael Deardeuff, How Amazon Web Services Uses Formal Methods, Communications of the ACM, 2015.
• Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune and John Wilkes, Large-scale cluster management at Google with Borg, EuroSys, 2015.

@article{ghosh2025checkout,
  title = {How Subscription Billing Systems Actually Work},
  author = {Ghosh, Krish},
  journal = {krishghosh.com},
  year = {2025},
  month = {December},
  url = "https://krishghosh.com/writing/checkout-is-the-easy-part"
}