ADR-001AcceptedReliabilityEvidence published –
Build high availability into the hosting platform from the start
A floating IP in front of HAProxy, a three-node Patroni cluster on etcd, PgBouncer in between: every layer redundant, every failover automatic.
Context
Makdos was a commercial hosting platform that Umut built and ran between 2020 and 2023. Customers ran their businesses on it, so a few minutes of database downtime could mean lost transactions, broken customer dashboards and support tickets that took days to resolve. The target was 99.9 percent uptime, which the post translates to roughly 8.7 hours of downtime a year, planned maintenance included.
Where the decision changes
Two constraints against each other, the rest held where the dials are. Each square shows the option that wins there; pick one to set both dials.
Arrow keys move across the map, and each square you reach sets both dials.
| Spiky | |||||
|---|---|---|---|---|---|
| High | |||||
| Steady | |||||
| Modest | |||||
| Tiny | |||||
| None | By hand | Alerts | On-call | Platform |
- One database server with backups
- A standby promoted by hand
- Redundant at every layer
- Current setting
- Recorded context
Change the constraints
Each dial starts at the context the record describes. The dot marks that level: filled when the source states it, ringed when the level is this page's reading of the source's words, hollow for a neutral placeholder where no rule is in force. The letter under each level is the option that would win there, with the other dials held where they are.
Options on the table
A · One database server with backups
One PostgreSQL server behind one proxy, with regular backups and a restore procedure.
3 points- 3Starting scoreThe fewest moving parts, but every failure or maintenance window becomes downtime that customers see, which is what the post set out to avoid.from the record
5 rules not in force right now
- +3Scale: TinyWith only a handful of users, a short outage and a restore from backup cost little.model assumption
- +1Operations maturity ≤ Ad hoc, by handThe post counts every extra component as another potential failure point; without regular failover drills, fewer components are safer.from the record
- +2Time to deliver: DaysNothing to build before it runs.model assumption
- +2Budget: Almost noneOne machine instead of a cluster.model assumption
- −1Compliance and data control ≥ Regulated dataRegulated data raises the cost of losing the latest transactions.model assumption
B · A standby promoted by hand
One primary streaming to an asynchronous standby; when the primary fails, someone promotes the standby.
4 points- 4Starting scoreRecovery no longer waits for a restore, but it waits for a person, and an asynchronous standby can be missing the last transactions the primary had already confirmed.from a reference
3 rules not in force right now
- +2Operations maturity ≤ Ad hoc, by handOne well-understood promotion step suits a team without on-call drills.model assumption
- +1Time to deliver ≤ A few weeksStreaming replication is quicker to set up than a consensus cluster.model assumption
- +1Budget ≤ LeanTwo database machines instead of a cluster.model assumption
C · Redundant at every layer
keepalived moves a virtual IP between two HAProxy nodes; Patroni runs PostgreSQL as one primary and two synchronous standbys, with a three-node etcd cluster for leader election; PgBouncer pools connections behind its own virtual IP.
6 points- 6Starting scoreSurvives node failures and maintenance without downtime that customers see: the requirement the post calls non-negotiable for a platform with paying customers.from the record
6 rules not in force right now
- −4Operations maturity ≤ Ad hoc, by handThe post calls regular failover testing non-negotiable: its monthly drills kept finding missing callbacks, drifted configuration and monitoring gaps. Without that practice the extra components are extra failure points.from the record
- −4Time to deliver: DaysSetting up Patroni was not straightforward; the post describes considerable time spent testing failure scenarios.from the record
- −3Budget: Almost noneThree database nodes, three etcd nodes, and pairs of proxies and poolers.model assumption
- −2Scale: TinyFor a handful of users, five components to run outweigh the outages they prevent.model assumption
- −1Team size: SoloOne person carries every failover drill and every night-time incident.model assumption
- +1Compliance and data control ≥ Regulated dataWith synchronous standbys, Patroni will not promote a standby that may be missing committed transactions.from a reference
Decision
Design redundancy into every layer from the start: an active-passive HAProxy pair behind a keepalived virtual IP, PostgreSQL under Patroni with synchronous standbys and etcd for leader election, PgBouncer in transaction pooling mode behind its own virtual IP, and Prometheus, Grafana and Alertmanager watching all of it.
Consequences
What it bought
- The post reports that the platform reached its 99.9 percent uptime target.
- Failover is automatic at every layer: keepalived moves the virtual IP within seconds, and the post says Patroni typically promoted a standby in under 10 seconds.
- PgBouncer's transaction pooling took the platform, in the post's figures, from needing 300+ database connections to about 50.
What it cost
- Five components to understand, monitor and test together: HAProxy, keepalived, Patroni, etcd and PgBouncer. The post counts every extra component as another potential failure point.
- Patroni's edge cases are numerous; finding them meant deliberately killing the primary and simulating network partitions.
- Synchronous replication makes every commit wait for a standby's confirmation, which PostgreSQL's documentation says increases response time.
What happened next
- Monthly failure drills that deliberately killed nodes; each one turned up a missing callback, drifted configuration or a monitoring gap.
- A runbook for every recovery procedure, so an incident in the middle of the night meant following a checklist rather than debugging from scratch.
- The post's closing lesson: high availability is an architecture designed in from the start, not a feature added later.
Evidence
- Building a Hosting Platform from Scratch: The Makdos Story/blog/building-hosting-platform-from-scratch ·
- High-Availability Infrastructure: HAProxy, Patroni, and 99.9% Uptime/blog/high-availability-infrastructure-haproxy ·
- Experience: ran the high-availability infrastructure at Makdos
- Projects: Makdos, on HAProxy, Patroni, PgBouncer, etcd and keepalived
Technical reference
Patroni's synchronous mode will not promote a standby unless it is certain the standby has every transaction that may have been reported as committed. Its documentation adds that synchronous replication does not guarantee zero loss in every case, for example when the primary and the synchronous standby fail together. PostgreSQL notes that waiting for the standby's confirmation increases each commit's response time.