The software around the signature

The mechanism in the last two chapters is a few paragraphs long. Almost all of the engineering is in the system built around it, and none of it is cryptographic.

The reason is that the mechanism is synchronous and the chain is not. A user signs, the gateway answers in milliseconds, and the chain confirms seconds later. Every piece of the system exists to keep those two clocks consistent, and to make sure a crash, a duplicate request, or a dropped connection does not leave money and the record of money disagreeing.

One gateway, many sellers

Fermah Pay runs as its own service, and sellers consume it over gRPC, with the .proto files as the single source of truth for the wire contract: buyers, deposits, charges, balances, withdrawals, mandates and recurring charges. A seller creates a charge without touching Soroban transaction construction, fee-bump wrapping, nonces, batching, retries or chain observation. For agents and APIs paid over HTTP, the same gateway serves the x402 facilitator interface: /verify, /settle and /supported.

Many sellers share one deployment of the system, one fee account and one Postgres instance, which makes isolation a correctness property rather than a nicety. A product owns seller deployments, each pinned to one Stellar network, and an API key belongs to exactly one deployment. The key resolves server-side to its product, deployment and network, never from the request body, and every query that reads buyer data carries all three in its WHERE clause. Reaching for another seller’s buyer returns the same uniform NotFound an unknown identifier would, because a PermissionDenied would itself confirm the buyer exists. The database stores only a SHA-256 digest of each key.

Retries are part of the contract, not an afterthought. Every deposit, charge and withdrawal takes an idempotency key. Repeating a request with the same key returns the original, whatever happened in between: a repeated charge is never debited again, even if the balance is now too low for a new one. Reusing a key with a different amount is refused with idempotency_conflict. So a retry can never become a second charge.

Every seller needs somewhere to develop without spending real dollars, and the usual way to provide it is a flag on the request or a column in the database. Both put live and test money one bug away from each other.

Here the separation is structural. A seller deployment is pinned to one network, stellar:testnet or stellar:pubnet, and its API keys carry the network in their prefix, fps_test_ or fps_live_, derived from the deployment at issuance rather than from anyone’s input. A gateway process serves exactly one network, and at startup it checks that its RPC endpoint is on the network it is configured for, by passphrase, and refuses to start otherwise. Nothing in a request crosses that boundary, because the network is a property of which process received it.

Settlement happens after the answer

A charge returns to the caller before it reaches the chain, which is what makes this feel like a payments API rather than a blockchain application. Inside one Postgres transaction the gateway locks the buyer’s row, checks the available balance, debits it, and records the charge with its idempotency key. A worker picks it up and batches it into a single charge_batch call, with up to 98 charges from any buyers amortising one transaction’s overhead.

So for a window, Postgres says a charge happened and the chain has not heard of it. That window is where the failure modes live, and they are all the same shape: something interrupts the sequence between the answer and the confirmation.

The answer to all of them is a submission engine that never has to guess:

Install before send. The complete signed fee-bump envelope, its hashes and its source sequence are written in one row, in the same transaction that marks the charges it carries as submitted, before any broadcast. The worker’s own database role cannot change those columns afterwards. A crash at any point leaves either nothing, because nothing was sent, or the exact bytes to send again.
Outcomes come only from the hash. A reply to sendTransaction, even a refusal, never settles anything. A submission is final only when the network returns the transaction for its hash, or when a node that has ingested ledgers past the envelope’s time bound proves it was never included and never can be. The local clock is never evidence.
The contract decides the rest. When a transaction’s fate is unknown, the worker waits for the operator’s authorisation in it to lapse, then reads the contract’s own record of each charge, which holds its outcome. Past the record’s lifetime, the contract’s charges events over the exact ledger range decide. A charge that still cannot be decided is quarantined for an operator rather than guessed at.
Final means final. A refused charge’s amount returns to the available balance in the same statement that records the refusal, and a database trigger refuses any change to an outcome once it is final, so each balance change happens exactly once.

The worker sends from several source accounts at once, each holding no XLM, so one slow envelope does not hold back the rest. On testnet, 1,000 charges admitted through the API at about 84 per second all settled on-chain in 11 batches sent from five sources. Inclusion fees follow the market reported by the RPC node, within a cap, and double only for a source whose last envelope provably expired.

Watching the chain

The system learns what happened on-chain by reading the contract’s events, which on Stellar is simpler than it sounds.

Stellar’s consensus is aBFT. There are no reorgs. A ledger that has been closed is final. This eliminates the hardest part of chain observation on EVM: the confirmation-depth gate, the parent-hash ring for detecting reorganisations, the buffer that holds unconfirmed events, and the dedup logic across the historical-scan and live-subscription handoff.

What remains: the observer reads the contract’s events with getEvents from the Soroban RPC, filtered by contract address, from a persisted position per seller deployment. Each page is stored in one database transaction that also moves the position forward, so a crash never skips or repeats an event. If the observer falls further behind than the node’s retention, about seven days on public endpoints, the ledgers it can no longer read become a durable event_gap finding rather than an absence nobody notices.

Only one observer, and one worker, acts at a time, chosen by a lease in Postgres. The others run on standby ready to take over, and the lease each holds is exported as a metric, so a deployment where none holds it raises an alert.

Comparing beliefs against the chain, continuously

None of the above is trusted to be sufficient. The observer is a separate process with no key: it can read, and it can append findings, and it cannot change a balance. That separation is the point. A compromised gateway or worker cannot silence the process that checks them.

It checks two things. Every event is matched against the gateway’s records as it arrives:


    finding                      means                                    severity
    ──────────────────────────────────────────────────────────────────────────────
    unknown_charge               the operator key charged a buyer the     critical
                                 gateway never admitted
    charge_amount_mismatch       a settled amount differs from the row    critical
    unknown_withdrawal           USDC left the treasury outside the       warning
                                 worker
    mandate_changed_elsewhere    a mandate changed through another        warning
                                 client
    code_changed                 the contract runs code nobody expected   critical

And at a regular interval it reconciles three views that must agree: the treasury’s USDC on the SAC against the contract’s liabilities and revenue, the contract’s totals against the sum of its events, and both against the database. A discrepancy is recorded only once it has persisted through several checks, so a deposit still in flight does not page anyone.

Findings are append-only, readable by the operator, and wired to Prometheus alerts and a Grafana dashboard with a row per threat in the threat model . The monitor also watches itself: a stalled RPC node, failing observer rounds or an observer that stops advancing each raise their own alert, so the monitoring cannot go quietly blind. Each threat is drilled on testnet , by provoking the condition for real and recording the alert that fired.

What the product is told

Because answers come early, results arrive later, and a seller reads them through the same API that started them. Every deposit, charge, withdrawal, mandate and recurring charge has a Get call returning its current state, from ADMITTED or SIGNED to its final outcome and the transaction that settled it. Over x402, GET /settlements/{commitment} returns the charge a commitment became and its on-chain transaction once it has landed.

Fermah Pay also delivers results as signed webhooks, through a transactional outbox with retries. The Stellar implementation answers by query instead, and the reason is who runs it. On EVM, Fermah operates one service for its own product, whose endpoint it knows and trusts. On Stellar, each seller or operator runs Fermah Pay for itself, and a gateway that calls out to arbitrary seller URLs is a gateway that must defend against being turned into a way to reach private infrastructure. Reading state by query keeps the gateway’s network surface inbound-only, keeps the seller in control of when it learns a result, and loses nothing: a state, once final, cannot change.