I designed, built and operate single-handedly a multi-tenant SaaS ecosystem in production, with real paying customers. This page shows the measured numbers, how a release happens, and — the part that matters — the incidents that took the system down and what changed after each one.
This table exists so you do not have to read the whole page. The column that matters most is the last one: measuring is what separates operating from assuming. And where I only have training and not practice, it says so — ticking every box informs nobody.
This is not a lab. There are paying customers, and when it goes down somebody stops selling. Everything else on this page exists because of that sentence.
ARM64 on public cloud, 8 vCPU and 30 GB each, behind a Cloudflare tunnel. Both serve traffic: there is no idle node waiting for a disaster.
Row-Level Security in PostgreSQL, fail-closed: without the customer scope in the session, the query returns zero rows. The runtime role does not have BYPASSRLS.
A minimum coverage that fails the build: 95% on two services, 90.3% on another. It is not a poster target — it is a condition for releasing.
A number without provenance is an opinion wearing the costume of data. Each line below says what was counted, how, when — and what the measurement does NOT prove. The last column is the one that interests me: an honest measurement states its own limit.
docker ps on both nodes: 112 each. Of those 112, 59 are application processes (web, worker, publisher, consumer, scheduler); the other 53 are bus, data, observability, edge and vault
it is not a measure of load or utilisation — it is a count of processes“Show me the diagram and I am sceptical. Show me what happens when it fails and I believe you.”
what this page tries to answerThe full picture: where traffic comes in, where each piece runs, how the data is reached — and what happens when each part fails.
Two views that come before the deployment drawing. The first answers “who uses this and what outside world does it talk to”. The second answers “which applications exist and how do they talk to each other” — with no infrastructure at all, on purpose: mixing the two in a single diagram is the mistake that makes most of them unreadable.
The 224 containers are not 224 copies of the same thing. Counted by role, they draw the platform — and every layer exists because some pain asked for it.
Neither node exposes a public port. Traffic comes in through a Cloudflare tunnel (outbound connector) and lands on Traefik, which does host-based routing, automatic TLS and acts as the balancing point. Draining a node from the tunnel is what makes canary releases possible: that is how you pull a node out of the balancer without taking anything down.
PostgreSQL on a dedicated host, shared by the verticals, with HAProxy as the single access point and pgbouncer provisioned but still outside the path (measured: zero active clients; the applications go straight to HAProxy) — it enters when the connection count calls for it. There are two instances: primary and hot replica. Customer isolation is Row-Level Security, not WHERE tenant_id scattered through the code: the runtime role has no BYPASSRLS, so forgetting the scope returns zero rows instead of leaking.
Honesty about the target: there is no automatic database failover. Promotion is manual and documented — and I would rather say that than draw a Patroni that does not exist.
Services do not call each other in a chain: they publish facts. The path is a transactional outbox (the fact commits together with the data, or neither does) → relay → Redis Streams → inbox with dedup per consumer group → DLQ for whatever failed. There are 14 Redis instances: the bus is per vertical and per node, and that is exactly why convergence is settled in Postgres, not in a distributed lock that would not cross the nodes.
.env fileCredentials live in a vault (OpenBao) with a per-consumer service token. There is a dedicated alert for a vault token nearing expiry — because a secret that expires silently brings everything down at once, at the worst hour.
The rule is to publish a fact, not to call a service. The default path is asynchronous: transactional outbox (the fact only exists if the data exists), relay, queue, inbox with deduplication per consumer group and a DLQ for whatever failed. That means a service that is down delays the other one — it does not take it down with it.
When the answer is needed now — “is this customer entitled to this feature?”, “is this company a child of that one?” — the call is synchronous, but it is never loose HTTP scattered through the code: it goes through a typed client from the single SDK, with named operations and machine-to-machine authentication by a key of its own for each service. Authority over plan, hierarchy and entitlement has a single owner; the others ask, they do not infer.
When a sale happens, the sales service notifies nobody. It publishes the fact — and twelve independent consumer groups read it at their own pace. Inventory draws down the item, finance creates the receivable, tax prepares the document, accounting posts the entry. None of them knows the others exist, and one of them being down delays its own queue, not the sale.
Per node: 13 workers, 11 publishers, 10 consumers and 10 schedulers. Each runs with concurrency 1, declared in the composition file. The tool's default is 8 — and that is how the fleet went down once. A limit that is not written does not exist.
AI work, analytics work and routine work have queues of their own. A model call that takes thirty seconds cannot sit in front of the queue that confirms a sale. Separating is what stops the slow from delaying the urgent.
The bus writes to disk. If a node goes down with queued work, that work does not evaporate — it waits for the node to come back and resumes. The difference between “the customer waited” and “the order vanished” is this configuration line.
Prometheus scraping 9 targets (both nodes, cAdvisor, the PostgreSQL exporter, Traefik, HTTP probes), Grafana, Alertmanager, cAdvisor, node and postgres exporter, blackbox probing routes from the outside in. 12 alerting rules live.
What I consider the mature part is not the quantity — it is what they watch. Three of them do not ask “did something break?”, they ask “did something stop happening?”: off-site backup empty, backup late, backup missing the database inside. A backup vault that exists but is empty passes as healthy in any presence check — and you only find out on restore day. The others cover PostgreSQL going down, disk filling up on each node, a node off the air, a WAL slot accumulating and a container in a restart loop.
“The best tool is planning.”
why each piece is where it isEvery choice here has a gain, a cost I pay and a commercial consequence. Anyone listing only the gain is selling, not designing.
The entire compute budget is two nodes of 8 vCPU and 30 GB, and they already run 224 containers. A Kubernetes control plane — API server, etcd, scheduler, controllers, overlay network, ingress controller — consumes CPU and memory from that same budget, continuously. And sustained CPU load is exactly what has already cost me two infrastructure suspensions.
As a business decision: Kubernetes solves a scale problem I do not have yet, charging today in operational complexity and in consumption of the scarcest resource I have. While the load fits in two nodes, adopting it would mean paying up front for insurance against a risk that does not exist — and creating a new, real one: a surface I cannot sustain alone while on call. This is not rejecting the tool; it is not switching tools out of fashion. The day the maths changes — more than one operator, or load that does not fit in two nodes — the migration is already designed, and it is cheap precisely because everything is already container images and declarative configuration.
As a business decision: a managed balancer costs per hour and per volume, and still hands me back a public IP to defend. The tunnel costs nothing on the plan in use, removes the exposed surface and delivers application filtering along with it. I traded a risk I would have to operate (exposure) for a risk I monitor (third-party availability) — and that second one has an external probe and an alert.
As a business decision: automatic failover without a consensus layer and fencing of the faulty node trades a rare and recoverable outage for silent and permanent corruption. For anyone holding other people's financial data, that trade is bad in every scenario. The mitigation is not automating the promotion: it is keeping the replica hot and current, having the procedure written, and point-in-time recovery in place — all measured.
As a business decision: idle standby is a cost that only pays off on the worst day — and that usually fails precisely then, because nobody exercises it. With both active, the same money buys capacity and resilience, and the proof it works is yesterday's traffic. Honesty about the limit: today both sit in the same availability domain, so a correlated domain failure still takes me down. It is marked in the diagram because it is the next step, not because I cannot see it.
As a business decision: the value of the gate is not where it runs, it is what it rejects. A hosted runner executing the suite against a database with no customer isolation gives a feeling of coverage and lets through the class of defect that hurts most in a multi-tenant SaaS: leakage between customers. To offset the cost I took on: a test that watches the gate itself and fails loudly.
The twelve verticals have no public address of their own. They are all mounted by path under a single origin, and the identity server — OAuth2 and key publication — lives on that same origin. Authorization does not sit only at a gate out front: it travels with the service, through a single SDK that every vertical carries and that denies the call when the scope is not present.
As a business decision: the product is multi-tenant and sells to companies that need to inherit customer hierarchy, contracted plan and separation of duties. Outsourced identity charges per active user — the cost grows with the base at the exact moment the margin should improve — and pushes the business rule outside the system that owns it. Keeping it in-house makes the cost engineering (once) instead of subscription (always), and the who-sees-what rule stays where it is true. The cost is mitigated operationally: there is a dedicated alert for an expiring key and token, and the release gate validates that the key issuer responds before returning the node to traffic.
Java with Spring Boot in identity, accounting and the workflow engine. Go in the new services — a single binary, no runtime, starts fast and uses little memory, which is the scarce resource here. Rust in two precise places: the agent that observes the operating system core, and the tax calculation engine. Python with Django in the business verticals, where speed of change is worth more than microseconds.
As a business decision: rewriting what works is the most expensive way to deliver nothing. The productive legacy keeps earning in Python while what is born is born in Go, and the stable contract between them is what allows swapping piece by piece without stopping the operation — without the risk of a total rewrite, which historically takes companies down.
Card acquirer, credit bureau, government APIs, open banking, e-signature, marketplace, messaging — everything comes in and goes out through a single service. It translates each vendor into a normalized capability and returns the result to the ecosystem as a fact on the bus. No business vertical knows the vendor's name; the business rule stays with the consumer, not with the adapter.
As a business decision: third-party integration is where contractual risk lives — a leaked credential, an improper charge, customer data leaving the premises. Concentrating means that auditing “who talks to the outside world, and with which credential?” is looking at one service, not twelve. And this is verifiable, not a promise: I measured the six business verticals and none has a third-party SDK installed — they only exist in the integration service. A rule that depends on human discipline fails silently; this one can be measured.
Operating alone with real customers imposes a constraint a large company does not have: every managed service I subscribe to is money every month and a part I do not control. That is why the design avoids the three classic sources of a bill that grows without warning: billed egress traffic, a balancer charged by the hour, and identity priced per active user.
As a business decision: in a SaaS, infrastructure cost is what separates margin from loss as the base grows. The question I ask before adopting any service is “does this cost grow with the number of customers?” — if it grows linearly with revenue, it eats the economy of scale the model is supposed to provide. Right-sizing, declared CPU and concurrency limits and the release ledger exist as much for cost as for reliability: a resource nobody measures is a resource nobody cuts.
A recovery plan that has never been timed is an intention, not a plan. These three numbers were genuinely measured, not estimated.
“Nothing is released by hand — and nothing compiles on the production server.”
a rule born from two suspensionsReleasing is the moment a stable system becomes dangerous. The path below exists so that changing stops being an act of courage.
Nothing is released by hand and nothing compiles on the production server. The constraint was born out of real pain: the previous infrastructure was suspended twice for sustained CPU load — compiling on the host was one of the triggers. The architecture changed because of it.
The image is built on the workstation/CI and published to the registry with the commit tag. The production server only runs pull — it never compiles.
The tunnel connector is stopped on node A. 100% of traffic goes to node B. The release happens on a node that no user is accessing.
No crashloop, key issuer responding, and zero pending migrations. That last check was born from an incident: a process standing up does not mean the database kept up.
It goes back to the previous digest on that same node and aborts — the stable node was never touched. And the tunnel connector is restored with verification: if that fails, it shouts on stderr with the manual fix command.
The validated node returns to the tunnel and only then does the second node receive the same image, one service at a time. A global lock serializes the whole fleet: two simultaneous deploys once took the infrastructure down.
Every recreated service is recorded in a ledger on the node with the date, the image digest and the commit. That is how you answer “what exactly is running?” without guessing.
The pipeline is in-house: there is no hosted runner checking the pull request. GitHub stores the code and serves as the image registry; the quality gate runs before that, on the workstation. This has an uncomfortable consequence I would rather say out loud: there is no remote safety net — if I do not run the gate, nobody runs it.
Formatting and lint with a ratchet (the floor only goes up), a project integrity check and the suite running on PostgreSQL with RLS armed — not on a toy database. Coverage below the floor fails. On the Java service, the same gate adds architecture tests, integration against real containers and static analysis.
Work happens on a branch, publishing goes through a pull request, and the main line is protected. push --force, reset --hard and deleting main are blocked by policy. Before a large refactor, a return marker.
The gate target called the system interpreter and died right after the lint printed “all good”. Anyone reading the screen saw green on a gate that had not executed a single test. Today a test watches the gate itself — and it fails loudly.
“A system that never broke is a system nobody uses.”
three real incidents, and the scar each one leftThe part that normally does not get published. Cause, what failed in detection, what I made worse while trying to fix it, and the guard that prevents it from happening again.
A system that never broke is a system nobody uses. What sets things apart is not the absence of incidents — it is what is left after one. These three are real, from the same day, and each one left a guard in the code.
The guard was born with the very vice it went hunting: it counted only the pending ones, and grep -c on empty output prints zero — identical to “none pending”. Today it counts the total too: an empty list is not a schema up to date, it is a lack of evidence.
except wrapping the entire customer: every discrepancy was detected and discarded. A phantom transaction at the acquirer, or a payment without confirmation — money — became a log line.Absence of measurement is not absence of the fact. And an alert that always screams blinds the watchman — it would have been worse than the original defect.
When measuring an integration, do not stop at “the event is published”. Ask who listens — and whether what it carries means anything on the other side.
Every incident above became a test that fails the build if the class of defect returns. A convention written in a document ages; an executable guard does not. And every guard is tested with a mutant: if I reintroduce the defect and it stays green, it is worthless.
“Does this warehouse count as sellable stock?” was written, literally, in five places. With a second warehouse type, every forgotten copy becomes a silent hole. The rule now has a single owner — and a test fails anyone who writes it by hand.
An AST sweep: every event type published in the code must be declared. It found two live cases — one of them was swallowing card discrepancies.
A structural guard, not a behavioural one: in tests the two database roles collapse, so the defect only exists in production. A behavioural test would stay green with the system on the floor.
Two absolute links pointing at the machine of whoever created them ended up inside the production image — and they were self-referential: infinite recursion for any sweep that follows them.
SRE, here, is not a title: it is a set of rules that already cost me dearly to learn and today are written into scripts, tests and policy. Six of them:
The report of the first incident on this page has a field called “I made it worse”: my attempt at a manual fix stacked three stuck sessions before I understood the problem. A post-mortem that omits the author's own mistake does not help anyone learn — it protects the author.
With production on the floor, the move is to go back to the previous digest and only then investigate. The rollback is automatic and by digest — not by “rebuilding the earlier image”, which is not the same thing and has fooled me before.
An alert fired on a cumulative total stays lit forever and ends up ignored — so the new death arrives with nobody watching. An alert has to measure the change, and go quiet when the change stops.
The previous infrastructure was suspended twice for sustained CPU load, and compiling on the server was one of the triggers; a third suspension would have been permanent. Today “production never compiles” is not a style preference — it is a rule with a review that blocks the pull request.
Two simultaneous deploys once took the fleet down. There is a global lock: one service at a time, one node at a time, and an orphaned lock is reclaimed instead of jamming the fleet forever.
Every service recreation goes into a ledger on the node: date, digest and commit. The question “what exactly is running?” is answered by query, not by the memory of whoever deployed. That is where the measured average of 92 releases per day came from.
I once reported “it is operator configuration” about a field that was impossible to configure — the type could not hold the other service's identifier. Zero out of twelve records filled in across the whole fleet was not negligence: it was a defect. Today I measure before concluding.
A validation that could not measure cannot report “all good”. That confusion has shown up three times: in the deploy, in the card reconciliation and in a guard of my own — always wearing green.
A suite with thousands of green tests coexisted with the main screen broken. A test with a stand-in does not measure the rule that only exists at the real boundary — and visual verification catches what no test catches.
“What I have not tested yet is worth as much as what I have — as long as I say which is which.”
the honest state of what is missingWhat is done, what is partial and what does not exist yet. Separating the three is what makes the first two columns trustworthy.
I can demonstrate a canary release end to end — including the automatic rollback when validation fails.