For a while, my per­sonal in­fra­struc­ture lived on a small Hetzner VPS. It ran a few web apps, a re­mote browser called Surf, Caddy, and the usual sup­port­ing cast. Nothing par­tic­u­larly se­ri­ous. It worked, I just did­n’t like pay­ing for it.

One of the apps I run, Surf, made the com­pro­mise dif­fi­cult to ig­nore. The cheap­est shared ma­chines were fine un­til Chrome had real work to do, at which point they felt starved. Dedicated CPU ma­chines fix that, but cost enough each month to make a per­sonal browser feel like a ques­tion­able fi­nan­cial com­mit­ment.

Buying an­other ma­chine was­n’t an ap­peal­ing es­cape hatch ei­ther. DRAM prices have gone com­pletely stu­pid, so putting to­gether a new box with a com­fort­able amount of mem­ory felt es­pe­cially ill timed. I looked at used mini PCs and briefly con­sid­ered turn­ing my desk­top into a server when­ever I was­n’t us­ing it. Then I re­mem­bered the CMF Phone 1 I al­ready own.

Eight ARM cores, 8 GB of RAM, 128 GB of flash, Wi-Fi 6, a 5G mo­dem, and a built in bat­tery backup, that is all at­tached to an SoC that I feel is overqual­i­fied for sit­ting in a drawer. And I had al­ready paid for it. After dust­ing it off and play­ing with it for a while, I de­cided to turn the phone into the server.

Today it runs Surf and its man­aged Chrome in­stance, my per­sonal fi­nance tracker, a screen shar­ing ser­vice, and a hand­ful of smaller web apps. They sur­vive re­boots, de­ploy from Git, and re­main reach­able when the phone moves be­tween net­works, so this is now the ma­chine that ac­tu­ally re­placed the VPS.

the first bad idea: re­plac­ing an­droid

The clean­est ver­sion of this idea seemed to be flash­ing a nor­mal Linux dis­tri­b­u­tion. The CMF Phone 1 has a post­mar­ke­tOS de­vice port, it boots, and the de­vice page has enough green boxes to make a reck­less per­son op­ti­mistic. I ended up be­ing that per­son.

What I paid less at­ten­tion to was every­thing marked bro­ken: Wi-Fi, Bluetooth, hard­ware ac­cel­er­a­tion, and most of the other things that make the phone use­ful as a small server. I got as far as the post­mar­ke­tOS splash screen and a black dis­play. At that point I had nei­ther a server nor a phone.

Recovering stock Nothing OS turned into its own side quest. The flash­ing util­ity needed Windows, so I in­stalled Windows in QEMU, fought USB passthrough and MediaTek dri­vers, watched the flash­ing tool hang, then even­tu­ally moved the process to an ac­tual Windows in­stal­la­tion and re­stored the fac­tory im­ages.

There was a mo­ment in the mid­dle of this where the phone was soft bir­cked and only showed a black screen and I gen­uinely thought I had con­verted a per­fectly good de­vice into a pa­per­weight.

It came back, and les­son learned: Android al­ready has work­ing dri­vers for every piece of this hard­ware. Wi-Fi, power man­age­ment, the bat­tery, the GPU, the mo­dem, and every weird ven­dor de­tail al­ready work. Throwing all of that away in pur­suit of a more con­ven­tional user­space was the wrong trade.

I did­n’t ac­tu­ally need the phone to be­come a nor­mal Linux ma­chine. I needed it to run Linux ap­pli­ca­tions re­li­ably while Android con­tin­ued do­ing the hard­ware spe­cific work it is good at.

ter­mux is the host op­er­at­ing sys­tem

The sec­ond at­tempt kept stock Android and treated Termux as the host en­vi­ron­ment.

Termux gives me OpenSSH, runit, Caddy, Cloudflared, pack­age man­age­ment, and nor­mal enough Unix tool­ing. Termux:Boot starts the su­per­vi­sor and SSH af­ter a re­boot. Tailscale gives the phone a sta­ble pri­vate ad­dress, so from any ma­chine on my tail­net I can just run:

ssh cmf

Termux is not a vir­tual ma­chine. Its processes still ex­e­cute against Android’s Linux ker­nel, but its Bionic based user­space is dif­fer­ent enough from an or­di­nary Debian in­stal­la­tion that ex­ist­ing Linux ap­pli­ca­tion im­ages can­not sim­ply be dropped into it. That split ended up be­ing use­ful, though: Termux could re­main the small host con­trol plane while each ap­pli­ca­tion brought the Linux filesys­tem it ex­pected.

The ac­tual ser­vices are su­per­vised by runit. Android’s bat­tery man­age­ment is very good at its nor­mal job and very bad for a de­vice pre­tend­ing to be a server, so I had my Ansible build also ap­ply an Android host pro­file: it in­stalls a per­sis­tent wake lock, dis­ables light and deep idle, ex­empts Termux, Termux:Boot, and Tailscale from back­ground re­stric­tions, dis­ables the child process lim­iter, pre­vents Wi-Fi sus­pen­sion, and con­fig­ures Tailscale as the al­ways on VPN.

The re­cov­ery chain mat­ters more than any in­di­vid­ual set­ting. Android boots, al­ways on VPN brings Tailscale back, Termux:Boot starts runit, runit starts every res­i­dent ser­vice, and health checks ver­ify the lo­cal and pub­lic paths. The phone can re­boot with­out wait­ing for me to no­tice.

Android boot -> Tailscale al­ways-on VPN -> Termux:Boot -> runit -> res­i­dent ser­vices -> lo­cal and pub­lic health checks

This is­n’t a con­ven­tional Linux server. There is no sys­temd, no nor­mal Docker dae­mon, and no rea­son to pre­tend oth­er­wise. But it is a Linux ker­nel with a very ca­pa­ble user­land sit­ting on top of it, and that turns out to be enough.

the sec­ond bad idea: proot

Most of my ap­pli­ca­tions al­ready shipped as Linux ARM64 OCI im­ages. proot-dis­tro made those sur­pris­ingly easy to run un­der Debian with­out chang­ing the ap­pli­ca­tions them­selves.

PRoot in­ter­cepts filesys­tem and process op­er­a­tions in user­space and makes a reg­u­lar Termux process be­lieve it lives in­side a Debian root filesys­tem. It is­n’t a con­tainer bound­ary. Everything still shares Android’s ker­nel, net­work name­space, and Termux UID. But as an ap­pli­ca­tion com­pat­i­bil­ity layer, it is ex­tremely use­ful be­cause it needs nei­ther root nor a spe­cial ker­nel.

The or­di­nary web ser­vices ini­tially ran fine this way. Each of my ap­pli­ca­tions got a ver­i­fied root filesys­tem, a loop­back port, and a runit ser­vice. Caddy ran di­rectly in Termux and routed host­names to those ports.

The per­for­mance/​la­tency sen­si­tive Surf browser work­load was the ex­cep­tion. Starting processes, open­ing li­braries, walk­ing paths, read­ing browser pro­files, and shuf­fling cap­ture data all crossed PRoot’s user­space trans­la­tion layer. There was CPU avail­able, but Chrome could not reach it ef­fi­ciently. So I rooted the phone, not to re­place Android, but to mount the same Debian filesys­tem prop­erly and en­ter it with a real ch­root.

Runit still owned the life­cy­cle from Termux, con­fig­u­ra­tion still came from the same place, and ap­pli­ca­tion data still lived in Termux stor­age. The work­load sim­ply reached Android’s ker­nel through na­tive syscalls in­stead of PRoot. The im­prove­ment was not sub­tle!

Once that path was solid, leav­ing the smaller res­i­dents un­der PRoot stopped mak­ing much sense. They now run the same way. My work­sta­tion re­solves each ARM64 im­age to an ex­act di­gest and ex­ports its filesys­tem; Ansible ver­i­fies and in­stalls it on the phone. A small root helper cre­ates a pri­vate mount name­space, binds the re­quired paths, en­ters the filesys­tem with ch­root, drops priv­i­leges, and starts the orig­i­nal im­age en­try­point.

Neither Docker nor a com­piler needs to ex­ist on the phone. These are still com­pat­i­bil­ity en­vi­ron­ments rather than se­cu­rity bound­aries: the res­i­dents share Android’s ker­nel and net­work stack, while the pri­vate mount name­spaces mainly keep mounts and cleanup pre­dictable.

I also spent far too long try­ing to bridge Debian’s graph­ics stack to the phone’s Mali GPU through VirGL and Android Vulkan. I got hard­ware com­posit­ing check­marks along­side cor­rupt pages and worse per­for­mance… The bor­ing soft­ware ren­dered path turned out to per­form bet­ter.

in­fra­struc­ture! not a pile of shell his­tory

By this point the phone could run every­thing, but I did­n’t want a pet server as­sem­bled from com­mands I would for­get in a week. I moved the en­tire host into an Ansible man­aged state: ver­sions, ser­vice de­f­i­n­i­tions, routes, power set­tings, se­crets, and health checks all live in one pri­vate repos­i­tory.

The de­ploy­ment flow is roughly:

re­lease or OCI im­age -> check­sum/​di­gest pinned in Git -> Ansible over SSH -> ver­sioned files on the phone -> atomic cur­rent sym­link -> runit ser­vice -> lo­cal health check -> pub­lic edge check

Releases are pinned by di­gest or check­sum and in­stalled into ver­sioned di­rec­to­ries be­hind an atomic cur­rent sym­link. A failed check­sum or health check stops the de­ploy­ment, roll­back means re­vert­ing the pin and ap­ply­ing again. Application data lives sep­a­rately from re­leases.

After the small man­ual boot­strap: in­stall the three Android apps, root the phone, grant Termux su­pe­ruser ac­cess, and au­tho­rize SSH, the same repos­i­tory owns the rest. From my work­sta­tion, bring­ing the host to the de­clared state is de­lib­er­ately sim­ple:

make phone make phone-sta­tus make phone-edge-check

Applying it again does not re­place un­changed run­time files or restart healthy res­i­dents. More im­por­tantly, the con­fig­u­ra­tion is use­ful if this phone dies: an­other rootable ARM64 phone can be brought to­ward the same state with­out re­con­struct­ing a shell his­tory.

Secrets are not stored on the phone’s Git check­out, be­cause there is no Git check­out. Ansible Vault val­ues live en­crypted in the in­fra­struc­ture repos­i­tory. The vault pass­word is de­rived by ask­ing my 1Password SSH agent to sign a fixed chal­lenge, so the pri­vate key stays in 1Password and the phone never needs ac­cess to it. During de­ploy­ment, Ansible ren­ders only the run­time val­ues each ser­vice needs into Termux’s pri­vate stor­age.

get­ting traf­fic to a phone be­hind home in­ter­net

The next prob­lem to tackle was ingress. My home con­nec­tion does not come with the kind of sta­tic server setup a VPS gives you, and I don’t want to ex­pose SSH or a col­lec­tion of ran­dom ap­pli­ca­tion ports through the router. I also wanted the phone to re­main a phone in one im­por­tant sense: I should be able to un­plug it, take it some­where else, con­nect it to the in­ter­net, and still have my server.

The HTTP ap­pli­ca­tions use a Cloudflare Tunnel. Cloudflared makes one out­bound con­nec­tion from the phone, Cloudflare sends each host­name through it, and Caddy routes the re­quest to the cor­rect loop­back ser­vice.

Internet -> Cloudflare Tunnel -> Caddy on 127.0.0.1 -> ap­pli­ca­tion on 127.0.0.1

There is no in­bound router rule for those ser­vices. Cloudflared only needs an out­bound con­nec­tion, so if I move the phone to an­other net­work, the tun­nel re­con­nects and the host­names fol­low it. Tailscale does the same for ad­min­is­tra­tion. The phone’s bat­tery can bridge the move, and the pub­lic ser­vices do not care which Wi-Fi hap­pens to be un­der­neath them.

The Surf re­mote browser back­end needed a dif­fer­ent path though. Its di­rect con­nec­tion is la­tency sen­si­tive, ter­mi­nates its own TLS, and the old iPad that con­nects to it pins the server iden­tity. At home, Cloudflare DDNS keeps a DNS only record pointed at the cur­rent pub­lic ad­dress and the router for­wards one port to Surf. On the LAN, the iPad con­nects di­rectly to the phone.

That still left roam­ing. A nor­mal Cloudflare Tunnel ter­mi­nates TLS at Cloudflare, which is ex­actly what Surf’s pinned con­nec­tion does not want. The so­lu­tion was to wrap the com­plete Surf TLS stream in­side an or­di­nary WebSocket. Cloudflare sees and for­wards the WebSocket, but the ac­tual au­then­ti­cated Surf con­nec­tion re­mains en­crypted end to end in­side it.

This ob­vi­ously adds la­tency, from out­side home I usu­ally see roughly an­other net­work round trip, and the iPad con­nec­tion was around 60 ms when I first tested it. But it works through a tun­nel that only needs out­bound con­nec­tiv­ity, on an op­er­at­ing sys­tem from 2012, with­out in­stalling Tailscale on the iPad.

That was the mo­ment the phone setup be­came slightly ridicu­lous in the best way. I left it plugged in at home, went to the of­fice, SSHed from my MacBook through Tailscale into my work­sta­tion and the phone, and used the orig­i­nal iPad through the phone hosted Surf in­stance. The VPS was no longer, it felt lib­er­at­ing.

what is ac­tu­ally run­ning

The ma­chine has two lay­ers. Android and Termux own the hard­ware, net­work­ing, ingress, and su­per­vi­sion. The Linux res­i­dents get the filesys­tem they ex­pect and oth­er­wise stay out of the host’s way.

The de­mand­ing work­load is Surf, which brings a mod­ern Chromium browser to old iPhones and iPads. The phone runs desk­top Chrome (recent ar­m64 re­lease of Chrome) and the Surf back­end in­side its Debian run­time, an iPad mini re­ceives H.264 video and au­dio while send­ing touch, key­board, tab, and nav­i­ga­tion com­mands back. Surf is why I cared so much about syscall over­head and la­tency in the first place.

Alongside it, the phone hosts my per­sonal fi­nance tracker. It records re­cur­ring in­come and ex­penses, one time trans­ac­tions, monthly spend­ing lim­its, and pro­jects my bal­ance for­ward day by day. Unlike the re­place­able ap­pli­ca­tion ar­ti­facts, its SQLite data­base con­tains state I care about, so it has au­to­mated off de­vice back­ups and a tested re­store path.

None of those names are baked into some grand “phone server frame­work.” A new res­i­dent is an­other im­mutable ar­ti­fact, a runit de­f­i­n­i­tion, an ex­plicit data con­tract, a health check, and op­tion­ally a Caddy route.

The lat­est ad­di­tion is ob­serv­abil­ity. I can still SSH in, in­spect runit, tail logs, check pub­lic routes, and query Android di­rectly, but I no longer have to do that merely to see what’s go­ing on with the ma­chine at a glance: One na­tive ser­vice col­lects CPU us­age for all eight cores, mem­ory, stor­age, up­time, bat­tery, ther­mals, lo­cal and pub­lic reach­a­bil­ity, and every dis­cov­ered runit res­i­dent. It keeps bounded his­tory and serves an em­bed­ded Vue in­ter­face at https://​dash.cmf, reach­able only over the LAN or tail­net.

The log view dis­cov­ers the same ser­vice di­rec­to­ries at run­time, so adding an­other res­i­dent does­n’t re­quire teach­ing the UI its name. It is much nicer than SSHing in just to re­mem­ber which thing was noisy.

should you do this?

If you al­ready have a rea­son­ably mod­ern, rootable ARM64 phone sit­ting un­used, this is much less ridicu­lous than it sounds. You get quiet hard­ware, low power con­sump­tion, flash stor­age, Wi-Fi, a built in dis­play for re­cov­ery, and a bat­tery that be­haves like a tiny UPS. Stock Android al­ready sup­ports the hard­ware, while Termux and a rooted ch­root are enough to run a sur­pris­ing amount of nor­mal Linux soft­ware.

I would not put ir­re­place­able data on one with­out au­to­mated off de­vice back­ups, and I would not treat the ch­roots as hos­tile work­load iso­la­tion. Rooting ex­pands the trust bound­ary, Android re­mains an un­usual server host, and soft­ware ren­dered desk­top Chrome is not go­ing to beat a ded­i­cated work­sta­tion with a GPU.

But for a hand­ful of per­sonal ser­vices, es­pe­cially when the al­ter­na­tive is pay­ing in­def­i­nitely for a VPS that is ei­ther slow or an­noy­ingly ex­pen­sive, it is a gen­uinely use­ful op­tion rather than only a stunt.

The phone is still a weird server. It shares one ker­nel, Android oc­ca­sion­ally needs to be re­minded not to “optimize” it, and a fu­ture Android up­date could al­ways cre­ate a new sur­prise.

But it is quiet, bat­tery backed, fast enough, reach­able from any­where, re­pro­ducible from Git, and al­ready sit­ting in my house. I started this try­ing to save money on a VPS. I ended up with a rooted phone run­ning my per­sonal in­fra­struc­ture, and some­how that is much more sat­is­fy­ing. :)

Postscript: This post was dis­cussed on Hacker News. In true HN fash­ion, a sur­pris­ing amount of the con­ver­sa­tion ended up be­ing about the ti­tle. I en­joyed the ac­ci­den­tal lin­guis­tics thread. :)

Last week I launched a pro­ject called Dark Hours, which was a web­site util­ity to give you an idea of what could be seen in the sky that night.

A de­vel­oper who cre­ated an­other web app called DarkHours.app replied to my com­ment on Bluesky yes­ter­day to show me how sim­i­lar the pro­ject was to his, in­clud­ing the name. The thread is here.

I told him I’d sig­nif­i­cantly dif­fer­en­ti­ate the fea­ture­set, change the name, and write up a blog post to show peo­ple his pro­ject.

About an hour later, once it be­came clear to me that the web app I had launched us­ing Claude was strik­ingly sim­i­lar to his open source pro­ject, even re­pro­duc­ing a bug he had later fixed, it was clear that there was­n’t any­thing to do ex­cept to redi­rect the do­main di­rectly to him and kill any plans I had to launch an iOS app for the pro­ject.

I’d like to give credit where credit is due. I think he’s made a won­der­ful open source app and if you liked the fea­tures that were in­cluded in what I launched, please use the ac­tual ver­sion of the pro­ject. That’s what I’ll be do­ing.

I’d also like to apol­o­gize for my ir­re­spon­si­ble use of AI to build such a thing. While I had gen­uinely never seen DarkHours.app be­fore yes­ter­day, I was care­less in re­ly­ing on AI to gen­er­ate the pro­ject with­out do­ing the work to un­der­stand whether it closely re­sem­bled an ex­ist­ing pro­ject. That’s on me, and I am re­spon­si­ble for what I pub­lished.

Going for­ward, I won’t be us­ing AI in this way to cre­ate any more web stuff, and I do not use it to any­thing close to this ex­tent on iOS soft­ware. I do ask ques­tions, de­bug is­sues, things like that, but I do not cre­ate apps with Claude for iOS.

Making your own dithered QR codes

QR codes: the ba­sics

A QR code is re­ally just a way of en­cod­ing a few bytes of data in a way that can be eas­ily read froma photo by a smart­phone or sim­i­lar de­vice. They’re de­signed to be very ro­bust to bad fo­cus, bad print­ing, funny an­gles, and miss­ing ar­eas. Exactly how they work is­n’t ter­ri­bly im­por­tant, but for the pur­poses of this page, a QR code is a grid of squares that di­vide into two parts — the func­tion pat­terns (which are the bold shapes in the above ex­am­ple) and the data mod­ules (everything else). The func­tion mod­ules are mostly used to let the scan­ner find the QR code eas­ily, so they have to be very clear and dis­tinct. The data mod­ules are where the ac­tual data is stored, along with some ex­tra header data etc, and since those are only read af­ter the scan­ner has used the func­tion mod­ules to work out ex­actly where the QR code is, they can be mod­i­fied a bit more. This is of­ten used by brands to cre­ate QR codes that look a bit dis­tinc­tive, and while this kind of mod­i­fi­ca­tion makes them a bit less ro­bust when scan­ning, but you can gen­er­ally get away with a fair bit be­fore the codes be­come un­scannable.

Putting pic­tures in QR codes

This is a QR code I saw on Mastodon. One of the mod­i­fi­ca­tions you can do with the data mod­ules is to shrink them — the scan­ner will look where the finder pat­terns say the cen­tres of the pix­els should be, and so as long as the code is­n’t dis­torted when you try to scan it. That means the rest of the space is yours to play with. Dave di­vides each pixel into a three-by-three grid, and uses the mid­dle one to store the data, and the oth­ers for a photo. The re­sult is a low-res, one-bit photo with some salt-and-pep­per noise on it.

Dithering

Obviously when we crush an im­age into a low colour depth, we don’t nor­mally just thresh­old it — that is, make the dark pix­els black and the light pix­els white. We nor­mally use a kind of che­quer­board ef­fect to cre­ate mid­tones as well. A Bayer fil­ter can be used to ap­ply this idea across a whole im­age with­out hav­ing to sac­ri­fice too much fine de­tail.

Error dif­fu­sion

An at­tempt to im­prove on this method was Floyd-Steinberg dither­ing.

In this method, you start in the top left and thresh­old the pixel nor­mally — if it’s more than 50% bright­ness, you make it white, and oth­er­wise you make it black. Say it was 70% bright­ness — we’re go­ing to make it white, which is 100% bright­ness, which means we’ve added 30% of a pixel too much bright­ness. To coun­ter­act that, we’re go­ing to “diffuse” that er­ror to other pix­els — all the nearby pix­els that we haven’t thresh­olded yet are made a few percend darker, to a to­tal of 30% of a pixel. When we come to thresh­old those pix­els, we’ll take that into ac­count. The idea is that once we’ve thresh­olded the en­tire pic­ture, every part of the im­age will be, on av­er­age, closer to the cor­rect bright­ness than more reg­u­lar dither­ing. The ir­reg­u­lar pat­terns are also a bit less dis­tract­ing.

QR codes with er­ror-dif­fu­sion dither­ing

The other ad­van­tage of this ir­reg­u­lar dither­ing is that the salt-and-pep­per noise caused by the QR code data mod­ules is much harder to spot. But it’s still there, and it’s why the im­age looks so noisy. I mean, it would look fairly noisy any­way be­cause it’s a 147×147 pixel one-bit im­age, but some of that noise is be­cause one in nine of the pix­els are ef­fec­tively ran­dom colours.

But we can solve that us­ing more er­ror dif­fu­sion.

Error-diffusing the data mod­ules

In this case, we’re do­ing two passes on the er­ror dif­fu­sion. The sec­ond is the same as nor­mal, but the first is specif­i­cally to mask the data mod­ules. We al­ready know what colour all of those pix­els have to be, so we can make them those colours and dif­fuse that er­ror to the sur­round­ing eight pix­els. The er­ror will po­ten­tially be quite big — nor­mally we get to choose the colour so the er­ror can never be more than 50%, but here we might have to make a pixel in a dark area pure white and end up cre­at­ing 95% er­ror. That sounds bad, but that’s ex­actly why it’s so im­por­tant to dif­fuse this er­ror in­stead of just ac­cept­ing it.

You can see that this makes the im­age much cleaner.

Getting fancy with it

The gen­er­a­tor tool al­lows you to do some ex­tra tricks. You can ro­tate the QR code be­fore adding the im­age, and tin­ker with the QR code set­tings to see if any of the al­ter­na­tive en­cod­ings hap­pen to look nicer.

You can also al­low the gen­er­a­tor to change a few of those 95%-error data mod­ules — QR codes have enough er­ror cor­rec­tion in them that you can af­ford to change a few pix­els and they’ll still scan. (That’s how QR codes with lo­gos in the mid­dle work.) But in prac­tice that does­n’t re­ally af­fect im­age qual­ity much if you dif­fuse the er­ror from the data mod­ules and it se­ri­ously af­fects how well the code scans.

Using these in real life

This all works pretty well if you don’t take it too far and the code is go­ing to be on a big screen or a poster or some­thing. If it’s on, say, a pa­per flyer that could get crum­pled then you’re go­ing to need more of that re­dun­dancy, ro­bust­ness and er­ror cor­rec­tion that we gave up to make the QR code pretty. Ultimately it’s a trade-off be­tween aes­thet­ics and scannabil­ity — and re­mem­ber that just be­cause a code scans on your phone, from a lap­top screen, does­n’t mean it will scan on a ran­dom stranger’s potato phone from a print­out in bad light­ing.

Also bear in mind that the gen­er­a­tor tool makes tiny im­ages with no mar­gin — some mar­gin is needed to make a QR code scan re­li­ably, and browsers will nor­mally blur im­ages when up­scal­ing them if you don’t dis­able that with CSS. The mar­gin has to be the op­po­site colour to the mid­dle of the big three “finder” pat­tern squares — nor­mally that’s white but you can gen­er­ate in­veted QR codes that need a black back­ground in­stead.

Table of Contents

1. A*’s use of the heuris­tic

2. Perfect heuris­tic

3. Reusing a per­fect heuris­tic

4. Multiple land­marks

5. Placement of land­marks

6. Automated place­ment

7. Implementation

8. Demos

8.1. Dragon Age, The Circle Tower 8.2. Cogmind, Factory 5 8.3. Maze 8.4. Dragon Age, Lothering 8.5. Cogmind, Research 2 8.6. Cogmind, Factory 4

8.1. Dragon Age, The Circle Tower

8.2. Cogmind, Factory 5

8.3. Maze

8.4. Dragon Age, Lothering

8.5. Cogmind, Research 2

8.6. Cogmind, Factory 4

9. More read­ing

On my Introduction to A* page I cover the ba­sics of the A* al­go­rithm. To op­ti­mize it, we usu­ally look at the pri­or­ity queue im­ple­men­ta­tion or re­duc­ing the size of the graph. A third way to op­ti­mize is im­prov­ing the heuris­tic func­tion. Here’s an ex­am­ple from the town of Denerim in Dragon Age Origins. Try mov­ing the start B and goal to see A* in ac­tion:

On this page I’ll show a way to im­prove the heuris­tic by adding “landmark nodes”. Try mov­ing the land­mark L to be near the goal . As the heuris­tic value gets closer to the true dis­tance , the num­ber of nodes A* has to ex­plore changes from to . The blue area is the sav­ings.

At the end of the page are demos of how this tech­nique helps with maps from real games. Although the demos on this page use grids, this op­ti­miza­tion works for any type of graph, not only grids.

1  A*’s use of the heuris­tic#

A* uses a heuris­tic to guide it to­wards the goal. We can think of it like wind push­ing us in the right di­rec­tion. Here, the heuris­tic pushes us east, and the short­est path goes east:

But some­times it pushes us in the wrong di­rec­tion. Here, the short­est path leads west of the B but the heuris­tic pushes us east:

A* runs faster when the heuris­tic guides us in the right di­rec­tion. It wastes time when the heuris­tic guides us in the wrong di­rec­tion. But why is it in the wrong di­rec­tion? It’s be­cause the com­monly used dis­tance-based heuris­tic does­n’t know about the walls.

2  Perfect heuris­tic#

Ideally, we’d find a heuris­tic that knows about walls and never points in the wrong di­rec­tion:

Can we cal­cu­late this “perfect” heuris­tic? Yes!

But the per­fect heuris­tic is dif­fer­ent for every goal and wall con­fig­u­ra­tion. Move the goal and you’ll see the heuris­tic changes. Move the start B and you’ll see it does­n’t.

If the goal and walls stay the same, then we can use flow field pathfind­ing. But usu­ally the goal is­n’t the same, so we need to con­struct a brand new per­fect heuris­tic for each goal. That is im­prac­ti­cally slow to cal­cu­late every time we run A*, and it’s also too large to store if we want to com­pute it ahead of time.

It’d be nice if we could cal­cu­late the heuris­tic once and then reuse it for mul­ti­ple A* runs with dif­fer­ent goals.

3  Reusing a per­fect heuris­tic#

Let’s cal­cu­late a per­fect heuris­tic to the green L, which we call a “landmark”. Can we reuse it for an­other goal ? Yes, some­times! Move the start point B and the land­mark L around to see which pur­ple goals are helped:

The idea is that if we al­ready have the path from B→L, we also get the short­est path to any along the way:

Think of the land­mark as some­thing far in the dis­tance. Your friend tells you “from your house B, walk to­wards the Eiffel Tower L un­til you get to Daniel’s house ”. The goal is not to reach the land­mark. The land­mark tells us a di­rec­tion to go in. The goal, Daniel’s house, is on the way.

Most goals aren’t on the path B→L but some­times they are close to that path:

But what does it mean to be “close”? We can use the path length, cost(B, L). When the paths are al­most the same, cost(B, L) is close to cost(B, X) + cost(X, L).

In A*, we use the heuris­tic func­tion as a lower bound for the path length cost(B, X). The tri­an­gle in­equal­ity[1] says that the sum of two sides of a tri­an­gle is at least as long as the third side. Adapted for di­rected graphs, we can say cost(B, X) + cost(X, L) ≥ cost(B, L). To cal­cu­late a lower bound, we rewrite this in­equal­ity as cost(B, X) ≥ cost(B, L) - cost(X, L).

That’s the key idea here. It’s im­prac­ti­cal to pre­cal­cu­late all costs to all lo­ca­tions, but if we’ve pre­cal­cu­lated the costs to a spe­cific lo­ca­tion L, we can use that to es­ti­mate the cost to a dif­fer­ent lo­ca­tion .

Some of the aca­d­e­mic re­search pa­pers re­fer to this as a heuris­tic based on the tri­an­gle in­equal­ity. Other pa­pers call this the “differential heuris­tic” be­cause it takes the dif­fer­ence be­tween al­ready com­puted dis­tances.

4  Multiple land­marks#

How of­ten is this tri­an­gle in­equal­ity use­ful?

It de­pends on where L is rel­a­tive to the path B→:

Move the start B and goal around to see where a land­mark would help:

Try mov­ing the land­mark L out­side the green shaded re­gion, and see that the heuris­tic and path don’t al­ways match.

Since a land­mark needs to be “after” the goal , a sin­gle land­mark won’t be use­ful for all paths. We need mul­ti­ple land­marks L₁, L₂, L₃, etc. Each one gives us some lower bound for the heuris­tic:

cost(B, X) ≥ cost(B, L₁) - cost(X, L₁) cost(B, X) ≥ cost(B, L₂) - cost(X, L₂) cost(B, X) ≥ cost(B, L₃) - cost(X, L₃) … cost(B, X) ≥ cost(B, Lₙ) - cost(X, Lₙ)

We can take the max() of these to pick the high­est bound. In this di­a­gram, try mov­ing the goal to one of the pur­ple shaded ar­eas to see how those ar­eas are im­proved by the land­marks. Then try mov­ing it to one of the un­shaded ar­eas to see how A* is­n’t any faster there. Try mov­ing the start point B to see how the shaded area also de­pends on where the start is.

5  Placement of land­marks#

The best land­mark po­si­tion de­pends on the start point B and goal . We want the land­mark to be “after” the goal , but what’s “after” de­pends on where the start point B and goal are.

We want to use land­marks to im­prove as many (start, goal) pairs as pos­si­ble.

Let’s start with a sin­gle land­mark. Try mov­ing the start B, goal , and land­mark L on this map:

The pur­ple shaded ar­eas show the goal po­si­tions that the land­mark helps. It looks like the land­mark can cover the main cor­ri­dors but not the side rooms. We need many more land­marks:

Much more of the map is cov­ered in pur­ple now. Picking the num­ber and place­ment of land­marks is pro­ject-spe­cific. Consider:

Are all paths equally likely? For ex­am­ple in a colony builder game like Dwarf Fortress, we may care a lot more about paths to/​from the main base, and not paths be­tween a for­est and a mine.

Are all paths equally valu­able to op­ti­mize? For ex­am­ple if pathfind­ing is lim­it­ing the frame rate, we might want to fo­cus on long paths that are slower to com­pute and not on short paths.

Is the map sta­tic or does it change over time? If sta­tic, we might want to spend a lot of time in the map de­signer tool to pre­cal­cu­late op­ti­mal land­marks. But if dy­namic, we might want to use the last few goal lo­ca­tions to de­cide new land­mark po­si­tions.

If the change re­duces an edge cost, the heuris­tic will over­es­ti­mate some­times, and A* will re­turn a non-short­est path un­til we up­date the cost table. Pathfinding is op­ti­mized but nonop­ti­mal. Example: the player broke a wall but the unit won’t look for the shorter path right away. If the change in­creases an edge cost, the heuris­tic will be lower than de­sired, and A* will take a lit­tle longer to run un­til we up­date the cost table. Pathfinding is op­ti­mal but not op­ti­mized. Example: the player added a wall so the unit might think that area’s safe to walk through but will have to find a path around it.

If the change re­duces an edge cost, the heuris­tic will over­es­ti­mate some­times, and A* will re­turn a non-short­est path un­til we up­date the cost table. Pathfinding is op­ti­mized but nonop­ti­mal. Example: the player broke a wall but the unit won’t look for the shorter path right away.

If the change in­creases an edge cost, the heuris­tic will be lower than de­sired, and A* will take a lit­tle longer to run un­til we up­date the cost table. Pathfinding is op­ti­mal but not op­ti­mized. Example: the player added a wall so the unit might think that area’s safe to walk through but will have to find a path around it.

If many units find paths to com­mon ar­eas (such as the Dwarf Fortress din­ing room), con­sider drop­ping the least used land­mark and adding one near the com­mon area.

Are the maps open-world or con­strained? A real-time strat­egy game may have dif­fer­ent needs than a room+cor­ri­dor dun­geon crawler.

Thomas Nobes has a video ex­pla­na­tion[2] in­clud­ing more tips on where to place the land­mark points.

Fortunately, even if a land­mark is­n’t op­ti­mal, it might still help some­what, and it’s still no worse than if we use the reg­u­lar A* heuris­tic.

6  Automated place­ment#

Although the best land­mark po­si­tions will be pro­ject spe­cific, one al­go­rithm to place land­marks in a pro­ject-ag­nos­tic way is to keep track of which lo­ca­tions are good for many ran­domly cho­sen paths. Try it here to find a land­mark po­si­tion:

It usu­ally but not al­ways picks a spot in the up­per left. It matches our in­tu­ition that land­marks should go on the outer edges of the map.

The sec­ond land­mark should be away from the first land­mark. The third land­mark should be away from the first and sec­ond land­mark. Each sub­se­quent land­mark should be eval­u­ated based on what it adds. This is what it looks like with two ex­ist­ing land­marks:

It picks a third away from the first two, but not al­ways in the same place.

7  Implementation#

The change de­scribed on this page is to the heuris­tic func­tion given to A*. We don’t need to change A* it­self.

We need to pick land­marks. If the maps are known ahead of time, land­marks can be placed in a map de­signer tool. If the maps are pro­ce­du­rally gen­er­ated, try the ran­dom­ized map analy­sis ear­lier on this page. Some of the pa­pers linked at the end have more so­phis­ti­cated place­ment al­go­rithms.

Then we need to an­a­lyze the map. Allocate a 2D ar­ray of num­bers, cost[nodeId][land­markId].

For each land­mark, we run Dijkstra’s Algorithm. It’s a “single source short­est path” al­go­rithm but we want a sin­gle goal in­stead of a sin­gle source. In a di­rected graph, we need to re­verse all the edges. In an undi­rected graph, we can use the edges as is. We set cost[nodeId][land­markId] to the cost of the short­est path from node nodeId to node land­markId. If the weights are all 1, we can use Breadth First Search in­stead of Dijkstra’s Algorithm.

This is ap­prox­i­mately what I’m run­ning for the demos on this page (undirected graphs):

const L = [ /* ar­ray of land­mark lo­ca­tions */ ]; let L_cost = [ /* ar­ray[nodeId] of ar­rays[land­markId] */ ]; for (let land­markId = 0; land­markId < L.length; land­markId++) { let out­put = dijk­straSearch(L[land­markId]); for (let nodeId = 0; nodeId < graph.num_n­odes; nodeId++) { L_cost[nodeId][landmarkId] = out­put.cost_­so_­far[nodeId]; } }

Note that it’s not much code. It’s run­ning our ex­ist­ing al­go­rithm (Dijkstra’s, A*, or BFS) and stor­ing the re­sults in an ar­ray. It could run in a back­ground thread.

Then we need to mod­ify the heuris­tic func­tion. Previously the heuris­tic was dis­tance(B, X). For ex­am­ple:

func­tion heuris­tic­Man­hat­tan(a, z) { re­turn Math.abs(a.x - z.x) + Math.abs(a.y - z.y); }

Each land­mark Li gives us a lower bound cost(Lᵢ, X) - cost(Lᵢ, B). We want to take the high­est of these:

func­tion heuris­ti­cLand­mark(B, X) { let h = heuris­tic­Man­hat­tan(B, X); // or any base heuris­tic for (let i = 0; i < L.length; i++) { let lower­Bound = L_cost[B][i] - L_cost[X][i]; lower­Bound = Math.abs(lowerBound); // if undi­rected if (lowerBound > h) { h = lower­Bound; } } re­turn h; }

Note that it’s not much code. It’s run­ning the ex­ist­ing heuris­tic (typically Manhattan, Chebyshev, or Euclidean dis­tance) and some­times in­creas­ing it if the land­marks form a good tri­an­gle.

What changes with the A* code? Nothing.

There are lots of tech­niques for mak­ing A* run faster. I like this one be­cause it’s very lit­tle code.

8  Demos#

I tried the dif­fer­en­tial heuris­tic on some maps from Dragon Age (provided by movin­gai.com[3]), a maze (also pro­vided by movin­gai.com), and Cogmind[4] (maps pro­vided by Josh Ge). All of these maps are undi­rected graphs (edges are bidi­rec­tional) so I’ve used that ver­sion of the dif­fer­en­tial heuris­tic.

The blue area is what we no longer have to search by us­ing the dif­fer­en­tial heuris­tic. The or­ange area is what we search even with the im­proved heuris­tic.

Try mov­ing the start B and goal to see the per­for­mance on dif­fer­ent paths.

8.1 Dragon Age, The Circle Tower

The land­mark is badly placed for the ini­tial B→ path. Try mov­ing it.

8.2 Cogmind, Factory 5

In the next demo the land­marks L are in places that don’t help. Move them around to im­prove search.

The blue area are the nodes we no longer have to search. More blue is bet­ter.

The land­mark L points help more when closer to the start point B than the goal . They help more when they’re “past” the goal point. Move the start B and goal around and see that there’s a big im­prove­ment no mat­ter which path you want to find:

However, it took a lot of land­marks to get that im­prove­ment. We can do bet­ter by us­ing the ran­dom path map analy­sis to pick fewer land­marks but in smarter lo­ca­tions:

8.3 Maze

A* with a dis­tance heuris­tic be­haves par­tic­u­larly badly with mazes, but in this one, just four land­marks make a big dif­fer­ence! Then try click­ing Random path re­peat­edly. The blue ar­eas are the ar­eas we did­n’t have to search by us­ing the land­marks. Also tog­gle the bidi­rec­tional flag to see how much of a dif­fer­ence that makes.

8.4 Dragon Age, Lothering

This map has large open ar­eas, and it seems to work well with the land­marks.

8.5 Cogmind, Research 2

This is a room-and-cor­ri­dor map from Cogmind.

During check­out, when a buyer clicks “Complete pur­chase,” we need to guar­an­tee the items they’re buy­ing are still avail­able. If we get this wrong in one di­rec­tion, two buy­ers pur­chase the same last unit: the mer­chant has to can­cel an or­der, send an apol­ogy email, and eat the sup­port cost. If we get it wrong in the other di­rec­tion, we tell a buyer some­thing is sold out when it is­n’t, and the mer­chant loses a sale they should have made.

At Shopify’s scale, ei­ther fail­ure com­pounds fast. On Black Friday 2025, mer­chants on our plat­form hit a record $5.1 mil­lion in sales per minute at peak. Every one of those trans­ac­tions touches in­ven­tory.

Our over­sell pro­tec­tion sys­tem han­dles this by re­serv­ing in­ven­tory dur­ing pay­ment pro­cess­ing—a short hold that pre­vents two con­cur­rent check­outs from claim­ing the same unit. For years, this ran on Redis. When we moved to­ward a uni­fied data­base strat­egy, we had to an­swer a hard ques­tion: could MySQL han­dle the same scale?

Earlier at­tempts had failed. A sin­gle row with a quan­tity col­umn could­n’t han­dle the con­tention. MySQL 8′s SKIP LOCKED fea­ture in­tro­duced a dif­fer­ent de­sign: one row per in­ven­tory unit in­stead of one row per item. Inspired by 37signals’ ap­proach to data­base-backed load dis­tri­b­u­tion, we re­built reser­va­tions on MySQL and hit our high-through­put tar­gets dur­ing peak 2025 traf­fic.

But the hard­est les­son was­n’t about data­base de­sign. It was dis­cov­er­ing that the real bot­tle­neck was­n’t what we were ob­serv­ing and mea­sur­ing. This post walks through the so­lu­tion and what we found along the way.

The chal­lenge

What is over­sell pro­tec­tion?

Oversell pro­tec­tion has two main op­er­a­tions:

Reserve: When pay­ment starts, we mark items as re­served (a short hold, e.g. sev­eral min­utes).

Claim: When pay­ment suc­ceeds, we per­ma­nently deduct quan­tity from the in­ven­tory ledger (source of truth).

Checkout com­ple­tion de­pends on this be­ing fast and cor­rect. Slow reser­va­tions trig­ger throt­tling and a worse buyer ex­pe­ri­ence. Mistakes mean over­selling (angry cus­tomers) or un­der­selling (lost rev­enue).

Scale and cor­rect­ness re­quire­ments

Scale here is not ab­stract: Shopify pow­ers over 14% of U.S. ecom­merce, and on Black Friday 2025 we saw an 11% in­crease in sales per minute at peak over the prior year. Reservations run on every check­out that touches in­ven­tory, so the sys­tem must han­dle that burst with­out drop­ping re­quests or break­ing con­sis­tency.

We needed to:

Support the plat­for­m’s high-per­for­mance through­put tar­gets dur­ing peak traf­fic

Respect multi-lo­ca­tion in­ven­tory (only re­serve from lo­ca­tions that can ful­fill)

Keep ACID guar­an­tees be­tween reser­va­tions and the in­ven­tory ledger

Prioritize cor­rect­ness: no over­selling and no lost reser­va­tions

The Redis model and its lim­its

The pre­vi­ous sys­tem stored reser­va­tions in Redis. Each item had a quan­tity key, and re­serv­ing meant DECR, re­leas­ing meant INCR. Redis han­dled con­cur­rency fine, but reser­va­tions and the in­ven­tory ledger lived in two dif­fer­ent sys­tems.

The claim step (payment processed, per­ma­nently deduct in­ven­tory) re­quired up­dat­ing MySQL and clean­ing up Redis, and those two op­er­a­tions could­n’t be wrapped in a sin­gle atomic step. Depending on the or­der, this could cause over­selling (item sold but was never de­ducted from the ledger) or un­der­selling (item de­ducted and still marked re­served).

On top of that, the Redis model had no multi-lo­ca­tion aware­ness and added the op­er­a­tional cost of a sep­a­rate clus­ter to main­tain. Moving reser­va­tions into the same MySQL data­base as the ledger meant we could wrap every­thing in ACID trans­ac­tions and elim­i­nate these fail­ure modes en­tirely.

The so­lu­tion: SKIP LOCKED

Core idea: one row per unit, bounded by de­sign

Instead of one row per item with a quan­tity col­umn, we use one row per sell­able unit. An item with 10 units has 10 rows. Reserving three units means se­lect­ing and mov­ing three rows in a sin­gle trans­ac­tion. By keep­ing reser­va­tions and the in­ven­tory ledger in the same data­base, we get ACID across re­serve and claim—fix­ing classes of bugs that were pos­si­ble with Redis (e.g. pay­ment suc­ceeds but in­ven­tory is­n’t claimed, or the re­verse).

A sim­pli­fied re­serve flow looks like this:

SKIP LOCKED is what makes this scal­able: if an­other trans­ac­tion has locked some rows, MySQL skips them and re­turns other avail­able rows. No wait­ing on the same row, less con­tention.

But one row per unit for all in­ven­tory would break down at scale—an item with 50,000 units across 10 lo­ca­tions would mean 500,000 rows, and the re­serve query would slow as it scans through them. Instead, we main­tain a bounded pool of avail­able rows, capped at 1,000 per item/​lo­ca­tion com­bi­na­tion. Reservations con­sume rows from this pool; a re­plen­ish­ment process re­fills it from the in­ven­tory ledger.

Why 1,000? The cap needs to be large enough to ab­sorb bursts with­out run­ning dry, but small enough to keep the table com­pact and the SKIP LOCKED scan fast. We sized it based on ob­served peak reser­va­tion rates per item/​lo­ca­tion dur­ing flash sales: 1,000 gives us enough head­room that re­plen­ish­ment can keep up un­der sus­tained load with­out the table grow­ing to a point where query per­for­mance de­grades.

What hap­pens if the pool emp­ties? During an ex­treme flash sale, the pool for a hot item can be tem­porar­ily ex­hausted. When that hap­pens, the re­serve path trig­gers re­plen­ish­ment in­line. A lock en­sures only one trans­ac­tion re­plen­ishes at a time; other con­cur­rent re­serves for the same item wait for it to fin­ish rather than all rac­ing to in­sert rows, avoid­ing a thun­der­ing herd. Once re­plen­ish­ment com­pletes, the wait­ing trans­ac­tions pro­ceed with a full pool. The buyer never sees the item as un­avail­able (unless it truly is). This adds la­tency to that spe­cific reser­va­tion, but it pre­serves cor­rect­ness: a buyer with avail­able in­ven­tory is never turned away.

Key tech­ni­cal de­ci­sions

1. Composite pri­mary key: fewer locks per row

Our first pro­to­type used an auto-in­cre­ment ID as the pri­mary key. When we ob­served lock be­hav­ior (e.g. with SHOW ENGINE INNODB STATUS), we saw two row locks per reser­va­tion in­stead of one.

With an auto-in­cre­ment pri­mary key, InnoDB was lock­ing both the sec­ondary in­dex used in the WHERE clause and the clus­tered in­dex (primary key). We switched to a com­pos­ite pri­mary key (shop_id, in­ven­to­ry_item_id, in­ven­to­ry_­group_id, id) so the columns we fil­ter on are part of the pri­mary key. That re­duced to one lock per row, which was im­por­tant when run­ning many reser­va­tions per sec­ond.

Takeaway: at this scale, in­dex and pri­mary key de­sign di­rectly af­fect lock count and through­put.

2. READ COMMITTED: avoid­ing gap (supremum) locks

When we ran SELECT … FOR UPDATE SKIP LOCKED on an empty table that needed re­plen­ish­ment, we saw gap locks (including on the “supremum” pseudo-record). Those locks blocked the re­plen­ish­ment trans­ac­tion from in­sert­ing new rows and could lead to dead­locks.

We changed the trans­ac­tion iso­la­tion level from REPEATABLE READ (MySQL de­fault) to READ COMMITTED for these trans­ac­tions. Under READ COMMITTED, InnoDB does­n’t take gap locks in the same way, so re­plen­ish­ment could pro­ceed. Jahfer Husain’s guide to InnoDB lock­ing was very help­ful for un­der­stand­ing this. This was our first use of a non-de­fault iso­la­tion level in this code­base; it re­quired small frame­work sup­port for set­ting iso­la­tion per trans­ac­tion.

3. Consistent lock or­der­ing: avoid­ing dead­locks

We hit dead­locks when re­serve and claim touched two ta­bles in dif­fer­ent or­ders. Reserve was do­ing INSERT into re­served_quan­ti­ties then DELETE from reser­va­tion_u­nits; claim was do­ing DELETE from re­served_quan­ti­ties. Different trans­ac­tions could lock the two ta­bles in dif­fer­ent or­ders and form a cy­cle.

The fix was to stan­dard­ize the or­der: re­serve al­ways DELETEs from the units table first, then INSERTs into re­served_quan­ti­ties. Claim only touches re­served_quan­ti­ties. Because both paths now ac­quire locks in the same or­der, nei­ther can hold a lock the other is wait­ing for—no more cir­cu­lar waits.

4. Batching with UNION ALL

Each round trip to the data­base has a cost. For carts with mul­ti­ple line items, we batch reser­va­tion queries us­ing UNION ALL so we fetch all needed units in one round trip:

That cut to­tal round trips and helped la­tency un­der load.

The real bot­tle­neck: con­nec­tions, not CPU

In pro­duc­tion we hit a through­put ceil­ing well be­low our tar­get. Reservation la­tency (e.g. P90) was ac­cept­able, CPU was­n’t maxed out, and the queries were al­ready op­ti­mized. So we looked else­where.

We tried batch­ing reser­va­tions across mul­ti­ple check­outs in a sin­gle SKIP LOCKED query to use fewer con­nec­tions. That helped in load tests but added com­plex­ity. We also moved some read load to repli­cas. Still, some­thing did­n’t add up.

Following the symp­toms

During load tests we saw:

Threads queu­ing in MySQL

CPU spik­ing when queued work ran

Connection ex­haus­tion to MySQL back­ends on ProxySQL layer

So we added vis­i­bil­ity into which busi­ness processes were hold­ing data­base con­nec­tions and for how long. Knowing that con­nec­tions are ex­hausted does­n’t tell you who is hold­ing them. We needed per-caller at­tri­bu­tion.

On the ap­pli­ca­tion side, we an­no­tated every SQL state­ment with a com­ment tag iden­ti­fy­ing the busi­ness process, like /* con­n_­tag:check­out_­com­ple­tion */. On the ProxySQL layer, we added track­ing that parses the tag and mea­sures how long each caller holds a con­nec­tion. The re­sult: to­tal con­nec­tion hold time, bro­ken down by busi­ness process.

This im­me­di­ately showed which callers were con­sum­ing the most con­nec­tion time. Not which queries were slow, but which processes were hold­ing con­nec­tions across long trans­ac­tions. If you’re hit­ting con­nec­tion lim­its and can’t tell why, this pat­tern (tag at the app layer, ag­gre­gate at the proxy) is straight­for­ward to im­ple­ment and im­me­di­ately ac­tion­able.

What we found

Once we could see con­nec­tion us­age, we learned that reser­va­tions weren’t the only heavy user. Other parts of the check­out path were hold­ing con­nec­tions longer than nec­es­sary. They had­n’t been op­ti­mized be­cause they had­n’t been the first to hit the limit. Connections are fi­nite: at high through­put we need many short trans­ac­tions per sec­ond. When other code held con­nec­tions longer, reser­va­tions were the straw that broke the camel’s back—not be­cause reser­va­tions were slow, but be­cause the pool was al­ready near de­ple­tion.

The cleanup of the check­out path re­moved 50% of reads and 33% of trans­ac­tions on the pri­mary data­base. We also re­vis­ited MySQL con­fig­u­ra­tion. InnoDB thread con­cur­rency had been set con­ser­v­a­tively years ear­lier and never re-eval­u­ated. Our work­load had changed. After in­creas­ing thread con­cur­rency where we had head­room, we re­moved a bot­tle­neck we had­n’t seen un­til we had con­nec­tion and CPU met­rics side by side.

Combined, the cleanup and con­fig­u­ra­tion changes re­moved the ceil­ing. We could scale be­yond our pre­vi­ous limit and meet our tar­gets. During high-vol­ume flash sales, writer CPU stayed un­der 50% and reader CPU un­der 16%, with head­room to spare.

The cu­tover

We did­n’t flip a switch from Redis to MySQL. We ran both sys­tems in par­al­lel in what we called “shadow mode”: every reser­va­tion was writ­ten to both Redis and MySQL, with Redis re­main­ing the source of truth. This let us com­pare the two sys­tems side by side, val­i­dat­ing that MySQL pro­duced the cor­rect busi­ness out­comes and met our per­for­mance re­quire­ments on real pro­duc­tion traf­fic. Because both sys­tems were live, there were no in-flight reser­va­tions to mi­grate. Redis reser­va­tions con­tin­ued to be hon­ored while MySQL built up its own state.

Once we were sat­is­fied with cor­rect­ness and per­for­mance, we switched the source of truth to MySQL. If any­thing went wrong, we could re­vert to Redis with a kill switch; the dual-write path was still ac­tive, so Redis had a com­plete view of reser­va­tions at all times. The roll­out was grad­ual, pod by pod, start­ing with low-traf­fic pods and work­ing up to our high­est-vol­ume mer­chants.

What we learned

We learned so much from this pro­ject but the two main take­aways:

1. Revisit old de­ci­sions

What was­n’t pos­si­ble five years ago (e.g. MySQL for this work­load) can be pos­si­ble to­day with new fea­tures like SKIP LOCKED. The same goes for con­fig­u­ra­tion: thread lim­its and other “rule of thumb” set­tings are worth re-check­ing when work­load and hard­ware evolve. If the num­bers don’t add up (e.g. low CPU but queu­ing), dig in.

2. Start small and ob­serve

We got a lot of value from a min­i­mal pro­to­type: a small ruby script and MySQL, no full frame­work like Rails. Observing the data­base (e.g. lock be­hav­ior in a sec­ond ter­mi­nal) taught us more than the­ory alone. Simple tool­ing and a tight feed­back loop beat big, opaque sys­tems when ex­plor­ing.

MySQL can now han­dle work­loads we used to as­sume re­quired spe­cial­ized in­fra­struc­ture. If you’re reach­ing for Redis, Kafka, or a cus­tom co­or­di­na­tion layer for high-through­put mu­tual ex­clu­sion, your ex­ist­ing data­base might al­ready be enough.

The bot­tle­neck was­n’t where we ex­pected. We op­ti­mized queries and locks for weeks; the real limit was con­nec­tion us­age in code we weren’t even look­ing at. If the num­bers don’t add up—low CPU but high queu­ing—in­stru­ment the full path. The an­swer is of­ten in the plumb­ing, not the en­gine.

Crucially, this was­n’t about mak­ing reser­va­tions fast. It was about mak­ing them safe neigh­bors. Reservations share a data­base with cart up­dates, pay­ment pro­cess­ing, and or­der cre­ation. A sys­tem that sat­u­rates con­nec­tions or holds locks too long jeop­ar­dizes all of them. The real bar was sus­tain­ing through­put with­out de­grad­ing data­base health for every­thing else.

For us, the pay­off is con­crete: more re­li­able reser­va­tions mean no over­sells and more suc­cess­ful pur­chases from our mer­chants.

Yesterday I pub­lished an ar­ti­cle ti­tled “App Store Rejection of the Week: Dark Hours”. I have re­tracted it. Its premise was so fun­da­men­tally wrong that there’s no point merely cor­rect­ing or edit­ing it. Even the ti­tle, as I ex­plain be­low, was in­ac­cu­rate. Although the orig­i­nal is now re­tracted, I’m not mem­ory-hol­ing it. The text of the orig­i­nal story is avail­able, for trans­parency and ac­count­abil­ity, in plain text (Markdown, natch) and PDF (preserving orig­i­nal pre­sen­ta­tion). Both of those ver­sions in­clude a pref­ace at the top link­ing to this re­trac­tion. The URL for the orig­i­nal story now redi­rects to this one that you are cur­rently read­ing.

To the best of my rec­ol­lec­tion, this is the first post I’ve re­tracted in the 24 years I’ve been writ­ing Daring Fireball. I hope it’s the last. I was mis­led, both overtly and through omis­sions, in sev­eral ways, but what I pub­lish is my re­spon­si­bil­ity, and I apol­o­gize for the er­ror.

Terry Godier first came to my at­ten­tion in February, when I linked to his ex­cel­lent in­ter­ac­tive es­say on RSS feed reader de­sign, “Phantom Obligation”, which es­say in­tro­duced Current, Godier’s new RSS reader that he made to ex­em­plify the ideas from his es­say. Current is, de­servedly, a bit of a break­out hit. (E.g., David Pierce, at The Verge, put it on a very short list of iOS-ex­clu­sive in­die apps that keep him from switch­ing to Android.) In March I linked to an­other in­ter­ac­tive es­say from Godier, “The Last Quiet Thing”, and again in April to a post re­gard­ing the App Store’s 5-star re­view sys­tem. I struck up an iMes­sage cor­re­spon­dence with Godier around when I first linked to his work.

Yesterday Godier posted “Browsers Have Standards, the App Store Has Judgment”. As orig­i­nally pub­lished, that post con­tained these two para­graphs:

A while ago I tried to sub­mit an iOS app for Dark Hours, my as­tron­omy web­site for nor­mal peo­ple. It was re­jected on the grounds that it was as­trol­ogy.

It has no tarot func­tion, no horo­scopes, and noth­ing that I, or any­one else I’ve asked, would as­so­ci­ate with as­trol­ogy.

A while ago I tried to sub­mit an iOS app for Dark Hours, my as­tron­omy web­site for nor­mal peo­ple. It was re­jected on the grounds that it was as­trol­ogy.

It has no tarot func­tion, no horo­scopes, and noth­ing that I, or any­one else I’ve asked, would as­so­ci­ate with as­trol­ogy.

Those para­graphs, at this writ­ing (August 8, 11:00 pm ET), have been deleted and re­placed by this:

Note: the orig­i­nal ver­sion of this post had a sec­tion here about an as­tron­omy app I am work­ing on that be­gan as an as­trol­ogy app and was re­jected af­ter hav­ing the as­trol­ogy con­tent re­moved. App store [sic] re­view reached out to me and let me know that they had ap­par­ently never been given the up­dated build and that the app should be fine to sub­mit now.

Note: the orig­i­nal ver­sion of this post had a sec­tion here about an as­tron­omy app I am work­ing on that be­gan as an as­trol­ogy app and was re­jected af­ter hav­ing the as­trol­ogy con­tent re­moved. App store [sic] re­view reached out to me and let me know that they had ap­par­ently never been given the up­dated build and that the app should be fine to sub­mit now.

You can now see the prob­lem that has led me to fully re­tract my orig­i­nal post, given that my post was en­tirely pred­i­cated on the premise that Godier’s orig­i­nal de­scrip­tion of the re­jected app was true — that the app “has no tarot func­tion, no horo­scopes, and noth­ing that I, or any­one else I’ve asked, would as­so­ci­ate with as­trol­ogy.” The truth is, the app, as orig­i­nally sub­mit­ted by Godier to the App Store (under the name “Asterly”, not “Dark Hours”), was en­tirely ded­i­cated to as­trol­ogy, not as­tron­omy, and did in fact in­clude a “Tarot card of the day” fea­ture amongst other oc­cultist horse­shit.

The grounds of Apple’s orig­i­nal App Store re­jec­tion of the app, and the re­jec­tion’s up­hold­ing by the App Review Board, were cor­rect.1 I wrongly took Godier at his word, both in his pub­lic blog post and in pri­vate iMes­sage cor­re­spon­dence yes­ter­day, that the re­jec­tion was­n’t just merely de­bat­able, but com­pletely and rather pre­pos­ter­ously un­grounded. Whether Godier ever sub­mit­ted a build of “Asterly” to the App Store that con­tained no oc­cult horse­shit and only the hard-sci­ence as­tron­omy fea­tures that were pre­sent in his “Dark Hours” web­site that was avail­able for the last week, I don’t know. But I have no rea­son to be­lieve that he did.

My dis­dain for as­trol­ogy is so ut­ter, and my es­teem for Godier’s pre­vi­ous work so high, that it sim­ply never oc­curred to me that he might have ac­tu­ally made and sub­mit­ted to the App Store an as­trol­ogy app, let alone that he’d then feign sur­prise and frus­tra­tion that an as­trol­ogy app was re­jected for be­ing an as­trol­ogy app. I showed him a draft of my post be­fore pub­li­ca­tion, to make sure I had the story straight, and he of­fered not a word of cau­tion, only grat­i­tude for my draw­ing at­ten­tion to the mat­ter.

It gets messier. Godier’s as­trol­ogy app that he sub­mit­ted to the App Store back in January was named “Asterly”. That was still the name when the App Review Board up­held its re­jec­tion in April. According to Godier, frus­trated by the App Store’s re­jec­tion, he ported the as­tron­omy ver­sion to the web, launch­ing it last week un­der the name “Dark Hours” at the do­main dark­hours.io. (This is why is was in­cor­rect for me, in the very ti­tle of my post, to claim that Apple had re­jected an app named “Dark Hours”. They re­jected an app named “Asterly” and had never seen an app from Godier named “Dark Hours”.) Yesterday, af­ter I linked to Godier’s post and Hacker News then linked to my post, Godier’s “Dark Hours” (with a space) came to the at­ten­tion (and jus­ti­fi­able sur­prise) of Miguel Beher, cre­ator of an open-source “astrophotography and dark-sky plan­ner” pro­ject named DarkHours (no space). Beher’s GitHub pro­ject con­tains the source code, and the ac­tual web app is freely avail­able at the do­main dark­hours.app. In an un­com­fort­able ex­change be­tween Beher and Godier on Bluesky, Beher pointed out that Godier’s Dark Hours had the same bug as Beher’s that routed peo­ple to “random fields in Mexico”. Earlier to­day, Godier took his web app down and redi­rected his dark­hours.io do­main to Beher’s dark­hours.app.

At the end of my now-re­tracted post yes­ter­day, as­sum­ing I had right­eously and rightly skew­ered Apple for an egre­giously er­ro­neous App Store re­view re­jec­tion, I wrote:

Mistakes hap­pen. But in a func­tion­ing sys­tem mis­takes get cor­rected, and mis­takes as ob­vi­ous as this one get cor­rected al­most in­stantly and in­clude a quick apol­ogy for the con­fla­tion.

Mistakes hap­pen. But in a func­tion­ing sys­tem mis­takes get cor­rected, and mis­takes as ob­vi­ous as this one get cor­rected al­most in­stantly and in­clude a quick apol­ogy for the con­fla­tion.

Obviously it was I who was mis­taken. This ar­ti­cle is my cor­rec­tion, and I apol­o­gize to all who read and be­lieved my now-re­tracted post, and to the re­view­ers at the App Store whose com­pe­tence (if not lit­er­acy) I be­smirched. I am deeply sorry about that. Won’t hap­pen again.

Regardless of one’s opin­ion re­gard­ing pseu­do­science and oc­cultist horse­shit — pro, con, or in­dif­fer­ent — one might rea­son­ably think it wrong for Apple to dis­al­low or dis­cour­age such apps from the App Store. In fact, there ex­ist plenty of such apps in the App Store, and Apple has even run “Best Astrology Apps” ed­i­to­r­ial fea­tures. Apple’s stance is ba­si­cally that the App Store has enough of these apps (and I sus­pect they’re a com­mon source of scams, given that by their very na­ture they tar­get the gullible). Guideline 4.3(b) states (emphasis added):

Certain kinds of apps, such as dat­ing, flash­light, sound ef­fects, wall­pa­per, sim­ple timers, and for­tune telling, are well es­tab­lished on the App Store and we will not ac­cept new sub­mis­sions un­less they of­fer a mean­ing­fully dif­fer­ent or im­proved ex­pe­ri­ence. We may re­move these apps from the App Store go­ing for­ward if they are not up­dated, im­proved, or do not at­tract cus­tomers. Other kinds of apps, such as drink­ing games, Kama Sutra, fart, and burp apps, are mediocre, low-qual­ity, or low-ef­fort and do not add value to the App Store.

That se­r­ial comma, ever use­ful, gives hope to any­one hard at work on a “fart and burp” app. ↩︎

Regardless of one’s opin­ion re­gard­ing pseu­do­science and oc­cultist horse­shit — pro, con, or in­dif­fer­ent — one might rea­son­ably think it wrong for Apple to dis­al­low or dis­cour­age such apps from the App Store. In fact, there ex­ist plenty of such apps in the App Store, and Apple has even run “Best Astrology Apps” ed­i­to­r­ial fea­tures. Apple’s stance is ba­si­cally that the App Store has enough of these apps (and I sus­pect they’re a com­mon source of scams, given that by their very na­ture they tar­get the gullible). Guideline 4.3(b) states (emphasis added):

Certain kinds of apps, such as dat­ing, flash­light, sound ef­fects, wall­pa­per, sim­ple timers, and for­tune telling, are well es­tab­lished on the App Store and we will not ac­cept new sub­mis­sions un­less they of­fer a mean­ing­fully dif­fer­ent or im­proved ex­pe­ri­ence. We may re­move these apps from the App Store go­ing for­ward if they are not up­dated, im­proved, or do not at­tract cus­tomers. Other kinds of apps, such as drink­ing games, Kama Sutra, fart, and burp apps, are mediocre, low-qual­ity, or low-ef­fort and do not add value to the App Store.

Certain kinds of apps, such as dat­ing, flash­light, sound ef­fects, wall­pa­per, sim­ple timers, and for­tune telling, are well es­tab­lished on the App Store and we will not ac­cept new sub­mis­sions un­less they of­fer a mean­ing­fully dif­fer­ent or im­proved ex­pe­ri­ence. We may re­move these apps from the App Store go­ing for­ward if they are not up­dated, im­proved, or do not at­tract cus­tomers. Other kinds of apps, such as drink­ing games, Kama Sutra, fart, and burp apps, are mediocre, low-qual­ity, or low-ef­fort and do not add value to the App Store.

That se­r­ial comma, ever use­ful, gives hope to any­one hard at work on a “fart and burp” app. ↩︎

Five pro­grams run­ning at once. A Note Pad loaded off the soft­ware floppy, a tick­ing Clock, a bounc­ing ball, the Control Panel and the Task Manager, each a sep­a­rate in­stance with its own dock tile. The Task Manager re­ports the ma­chine at 70% busy and 147K of RAM in use.

The desk­top at rest, a few sec­onds af­ter boot. The 512-byte boot sec­tor has loaded the ker­nel and jumped to it. There is a floppy icon for each drive the ma­chine re­ports through the BIOS, and the dock is empty be­cause os­8088 launches noth­ing at startup.

A menu is open only while the but­ton is held. Press in the menu bar, drag through the items, re­lease on the one you want. There is no spare copy of the screen to re­store from, so the pix­els be­hind the pull-down are copied into a block claimed off the heap for ex­actly as long as the menu is up.

A ti­tle-bar drag, caught halfway through. The win­dow has not moved yet. A hol­low one-pixel out­line tracks the pointer in XOR mode, so eras­ing it is the iden­ti­cal op­er­a­tion to draw­ing it — you can watch it in­vert the Disk win­dow’s own ti­tle text where it crosses.

Several in­stances of the same pro­gram. A Disk win­dow, two Clocks and two Bounces, each a sep­a­rate record in the in­stance table, each cas­caded six­teen pix­els down and right from the pre­vi­ous copy of its kind. Every in­stance gets a dock tile, and the tile’s po­si­tion is its table slot.

Keystrokes reach the front win­dow only. Two lines typed on the em­u­lated key­board, wrapped in the win­dow’s 258-pixel con­tent area, with the caret af­ter the last char­ac­ter. Each Note Pad in­stance gets its own 512-character buffer, and as many can be open at once as the heap has room for.

The About box reads live ker­nel state. The word pre-emp­tive on the third line is ren­dered from the ker­nel’s sched­uler byte every time the box paints, so the About box, the Control Panel and the Task Manager can­not dis­agree about which sched­uler is run­ning.

The Task Manager, with its his­tory graph filled. One sam­ple every nine ticks into a 160-column ring, so a full sweep across the win­dow is about eighty sec­onds of his­tory. Apps with no task of their own get their win­dow call­backs timed at the dis­patch site and billed to them, so the rows still add up to one to­tal.

The sched­uler set­ting, read straight from the ker­nel. The set­tings pane keeps no copy of the sched­uler mode: it reads the ker­nel’s mode byte every time it draws. Click Cooperative and the very next timer tick de­clines to switch tasks, with no restart and no hand­shake.

Two of these five pro­grams are min­i­mized. Clock and Bounce were sent to the dock with the min­i­mize box in their ti­tle bars, and their tiles draw in­verted so you can see at a glance what is hid­den. Minimized is not stopped — the Clock is still count­ing, it only skips draw­ing.

The file man­ager, read­ing the soft­ware floppy. The soft­ware disk mounted on drive B:, with each file’s name, size and its own icon. The icons come off the disk it­self — the browser peeks each file’s first sec­tor, where a pack­age car­ries its own 16x16 bitmap — so the list shows a real icon with­out load­ing it.

Two loaded pro­grams, each in its own seg­ment. MINES and HELLO came off the same floppy into sep­a­rate re­gions claimed off the heap — 2,048 bytes and 512 — and each runs at off­set zero in­side its own, so noth­ing had to be patched on the way in.

Minesweeper: 1,510 bytes, loaded from the floppy. Double-clicking the MINES row read it into a seg­ment of its own and opened its win­dow; from then on its paint, key and click procs are plain near off­sets the ker­nel calls ex­actly like a built-in’s. This board is the only place in the sys­tem that uses more than two col­ors.

ArtfulType, a Markdown writer. ActionRetro’s writ­ing app for clas­sic 68k Macintoshes, re­built for the 8086. Full-screen Writer mode with its own menu bar, styling as you type, and plain Markdown files that open any­where else.

Tracker, a four-chan­nel MOD player. Load an Amiga .MOD off the floppy and play it: the pat­tern scrolls un­der the play­ing row, four chan­nel me­ters move with the mu­sic, and the desk­top stays us­able while it plays.

Paint, a bitmap ed­i­tor. Pencil, eraser, fill, shapes, se­lec­tion, eye­drop­per and text, with a re­siz­able can­vas, undo, a clip­board and six­teen col­ors. Reads and writes BMP and GIF.

Fractal, a Mandelbrot ex­plorer. Zoom in, pick a color scheme, and watch the pic­ture sharpen through three passes. It draws in the back­ground, so the rest of the desk­top keeps work­ing while it ren­ders.

Piano, a two-oc­tave key­board. Play it with the mouse or the com­puter keys. Notes ap­pear on the staff above as you play, and the staff plays back. Three built-in songs, and FM syn­the­sis on a ma­chine with a sound card.

Recorder, for sam­pled sound. Record, stop and play, with a wave­form dis­play and a built-in demo tone. It uses a Sound Blaster where the ma­chine has one and the PC speaker where it does not; the sta­tus line says which.

Solitaire: Klondike. Deal from the stock, drag cards and runs be­tween the seven columns, build the four foun­da­tions up from the aces. Cards drag with an out­line, the same way win­dows do.

Arkanoid, a brick-breaker. Move the pad­dle with the ar­row keys, clear six rows of bricks, keep the score across lev­els. The ball runs on its own timer, so it keeps a steady speed what­ever else is hap­pen­ing.

Minesweeper, on a 9x9 board with ten mines. Click a cell to open it, F for flag mode, N for a new game. Empty re­gions open in one click, and the first click is al­ways safe — the mines are laid af­ter you have cho­sen where to start.

Note Pad, a plain text ed­i­tor. Word wrap, a caret, and Open, Save and Save As through the sys­tem file di­a­log. It writes or­di­nary files onto the FAT12 floppy, so a note writ­ten here opens on a mod­ern com­puter.

Hello: a win­dow with two lines of text. It does noth­ing else. It ships as the small­est work­ing ex­am­ple for any­one writ­ing a pro­gram of their own, and as the only pack­age with no icon, which is why the Disk win­dow draws it with the generic one.

In her up­com­ing book “The Rise and Fall of the Artificial State,” Jill Lepore warns that tech com­pa­nies are in­creas­ingly re­plac­ing the func­tions of de­mo­c­ra­tic gov­ern­ment. This shift, she said, marks “a re­turn to tyranny and mys­ti­fi­ca­tion in the form of rule by al­go­rithms, cor­po­ra­tions, ma­chines.”

On the lat­est episode of TechCrunch’s Equity pod­cast, I spoke to Lepore — a Harvard his­to­rian and New Yorker staff writer who re­cently won a Pulitzer Prize for her his­tory of the U.S. Constitution — about the evo­lu­tion of what she de­scribed as “the idea that we should live un­der an ar­ti­fi­cial state or gov­ern­ment by ma­chines.”

“I’m not an anti-tech­nol­o­gist,” Lepore in­sisted. Instead, she said, “My beef is the ways in which pri­vate cor­po­ra­tions have in­creas­ingly taken on the func­tions of the state.”

While Lepore’s book ex­am­ines tech­no­cratic philoso­phies that go back cen­turies, she ar­gued that many of Silicon Valley’s “charismatic or not-so-charis­matic lead­ers” — es­pe­cially Elon Musk — seem to be ush­er­ing in a fu­ture pulled from mis­read pulp sci­ence fic­tion and comic books.

“But what’s funny about Musk is, the stuff he likes ac­tu­ally com­pletely de­feats and de­fies all of his po­lit­i­cal be­liefs,” she said.

Our con­ver­sa­tion also cov­ered Apple’s fa­mous “1984” Macintosh ad, why it’s “bananas” to call Twitter a dig­i­tal town hall, and the cur­rent data cen­ter back­lash. Keep read­ing for high­lights, edited for length and clar­ity.

So you’ve prob­a­bly had to do this a lot al­ready, but can you ex­plain what you mean by the “artificial state”?

By the ar­ti­fi­cial state, I mean a kind of state that is re­plac­ing the lib­eral de­mo­c­ra­tic na­tion-state in the United States and around the world. It’s both a real thing, a con­struct, but it’s also an idea.

And so, in this book “The Rise and Fall of the Artificial State,” I trace the rise of the idea that we should live un­der an ar­ti­fi­cial state or gov­ern­ment by ma­chines. I also trace the no­tion that this is an in­evitable fail­ure, that the ar­ti­fi­cial state can­not sur­vive, and I trace that idea through sci­ence fic­tion.

At one point, you say the rise of the ar­ti­fi­cial state marks the end of cen­turies of democ­racy and equal rights, and it’s “a re­turn to tyranny and mys­ti­fi­ca­tion in the form of rule by al­go­rithms, cor­po­ra­tions, ma­chines.” Can you just say a lit­tle bit more about why you see it in such stark terms?

Yeah, I do have a pretty neg­a­tive view of it, and I think it’s im­por­tant to dis­tin­guish the ar­ti­fi­cial state from tech­nol­ogy it­self or modes of tech­nol­ogy. I’m not an anti-tech­nol­o­gist. I’m mar­ried to a com­puter sci­en­tist. I’m re­ally ex­cited about all kinds of in­tel­lec­tual rev­o­lu­tions that we’re in the midst of right now.

That’s not my beef, right? My beef is the ways in which pri­vate cor­po­ra­tions have in­creas­ingly taken on the func­tions of the state. No one con­sented to that. This has been a kind of grad­ual, largely ac­ci­den­tal trans­for­ma­tion of how many na­tion-states around the world work — it’s hap­pened first in the United States.

I think of­ten these in­no­va­tions in bring­ing new tech­nolo­gies to the op­er­a­tions of gov­ern­ment have been ex­tremely well in­ten­tioned; they orig­i­nate with an in­ter­est in ef­fi­ciency and speed and cheap­ness. And then, I think, only in the last 20, 25 years or so have these de­ci­sions been pur­pose­ful and de­lib­er­ate as a kind of usurpa­tion of the role of the na­tion-state.

And that’s not my spec­u­la­tion. You hear a lot of a lot of very promi­nent tech en­tre­pre­neurs talk about want­ing to move be­yond the era of the na­tion-state. […] A lot of fu­tur­ists in the ’90s were lib­er­tar­i­ans, and they had a spe­cific in­ter­est in us­ing the ad­vance of the in­ter­net and the suc­ces­sive in­no­va­tions that fol­lowed as a means to erad­i­cate the na­tion-state.

You talk about, on the one hand, the tech­nolo­gies them­selves, and then also the philoso­phies be­hind them, the role the cor­po­ra­tion has in­creas­ingly played. I’m cu­ri­ous to what ex­tent we can sep­a­rate them. Can we ac­tu­ally have a ver­sion of the in­ter­net and of so­cial me­dia that does­n’t nec­es­sar­ily lead to this fu­ture that it seems like we’re [currently] hurtling to­wards?

Absolutely. I’m a his­to­rian. I’m not a tech writer. I’m not a tech jour­nal­ist. I’m not a com­puter sci­en­tist. I’m a his­to­rian, and I’m chiefly a po­lit­i­cal his­to­rian, though I’m also a lit­er­ary his­to­rian. And so, one of the things that I’m re­ally in­ter­ested in un­rav­el­ing for read­ers in this book is all the what-ifs, all the al­ter­na­tives, the paths along the road that were not taken and why.

There was, of course, a re­ally avid dis­cus­sion in the 1990s about what the in­ter­net should look like when it was opened up, and what we ended up with, the 1996 Telecommunications Act — I think, a lot of peo­ple would say [that] just was a mis­take, not an act of sin­is­ter in­tent, right?

But it was a prod­uct of a par­tic­u­lar po­lit­i­cal mo­ment, re­ally was deeply in­flu­enced by Newt Gingrich and his Contract with America, and it’s been very dif­fi­cult to re­visit. I think it’s worth think­ing about what were the al­ter­na­tives that were in play at the time.

And you could say the same thing about the per­sonal com­puter. So, to the de­gree that we can lo­cate an ori­gin point for the promise that bet­ter com­puter tech­nol­ogy would make for bet­ter democ­ra­cies, I think the mo­ment you would first look to would be January 1984, that Super Bowl ad that Apple ran for the re­lease of the Macintosh, with the the sort of George Orwell, 1984 [theme]. Apple was re­ally big on the idea that main­frame com­put­ers rep­re­sented to­tal­i­tar­i­an­ism. They were try­ing to dis­man­tle the gi­ant gray IBM ma­chines, as in rep­re­sent­ing them in that ad as a to­tal­i­tar­ian state. And the lithe, beau­ti­ful, quick, adorable, per­sonal Macintosh would be the ax that would de­stroy that ma­chine and would usher in a new era in which “1984 would not be ‘1984.’”

That was clever ad­ver­tis­ing. I doubt that any­body at Apple re­ally be­lieved the per­sonal com­puter was go­ing to be an in­stru­ment of per­sonal lib­er­a­tion. I mean, it was go­ing to make pos­si­ble a lot of cool things. I re­mem­ber when I got my first Macintosh — it cer­tainly was­n’t 1984, but it was re­ally cool, it was re­ally fun, it was re­ally ex­cit­ing, I did a lot of things on it. It would never oc­cur to me that it was im­prov­ing my ca­pac­ity for cit­i­zen­ship or my abil­ity to func­tion bet­ter in civil so­ci­ety. It was a cool tool.

But if you wind the reel for­ward in time, down to 2026 — stops along the way in­clude the 2016 elec­tion when Facebook News, in re­sponse to its crit­ics, es­tab­lishes a Supreme Court. You get to last year, when Anthropic hired a moral philoso­pher to write a con­sti­tu­tion. You get to re­cently, when Sam Altman was on Joe Rogan and said [in re­sponse to a ques­tion from Rogan], “Oh, an AI pres­i­dent would be a great idea.”

In some ways, they’re silly ex­am­ples. But you see the ways in which these cor­po­ra­tions, these tech com­pa­nies from Silicon Valley, and es­pe­cially their charis­matic or not-so-charis­matic lead­ers, are just tak­ing on the trap­pings of the na­tion-state and the func­tions of democ­racy.

They’re not peo­ple with a so­phis­ti­cated po­lit­i­cal phi­los­o­phy, but it’s like a car­toon ver­sion of that 1984 Macintosh ad, ex­cept that it takes it­self so se­ri­ously. And these com­pa­nies have so much power.

But that said, the book does­n’t be­gin in 1984. I just think that’s a good ex­am­ple of our mod­ern era and the way a fun ad­ver­tis­ing cam­paign turns into a kind of delu­sional fan­tasy on the part of peo­ple like Sam Altman.

You [also] talk about the promise of the quote-un­quote “Twitter rev­o­lu­tion,” and this idea that it would bring democ­racy every­where. I can’t help but let that color the way I [react] when Sam Altman or some other AI CEO now says that AI is go­ing to bring all these in­cred­i­ble gifts — and there­fore, if you stand in the way, you’re stand­ing in the way of progress, in the way of his­tory.

To what ex­tent should we just dis­miss all these claims out-of-hand, or are there ways that it might come true?

I mean, Twitter is ac­tu­ally a good ex­am­ple, right? When it was launched, when Jack Dorsey started it, it did­n’t an­nounce it­self as, “We’re go­ing to save hu­man­ity, we’re go­ing to res­cue hu­man civ­i­liza­tion from ex­tinc­tion.” It was kind of a goof, and I think peo­ple that used Twitter re­ally early on were like, “You know what? It was ac­tu­ally re­ally fun.” It was like, “I made a tuna fish sand­wich to­day. What did you have for lunch?” Twitter as a com­pany did not launch it­self on a stage say­ing, “We’re here to save democ­racy.”

And re­ally, what hap­pened was that politi­cians, elected of­fi­cials be­gan us­ing Twitter in ways that en­hanced their po­lit­i­cal power, in ways that am­pli­fied their mes­sages, in ways that al­lowed them to reach a younger au­di­ence, in ways that al­lowed them to have a con­stant con­nec­tion with an au­di­ence. Politicians and po­lit­i­cal cam­paigns re­ally kind of con­vinced Twitter — at least in­so­far as I see them, I don’t have an in­side ac­count of the com­pany — but some­what be­grudg­ingly, Twitter came around to like, “Twitter’s got­ten so big, and peo­ple post about pol­i­tics so of­ten that it’s al­most like Twitter is a town hall.”

By the time you get to, I think it’s 2012 — many years into Twitter’s fairly short his­tory — they pub­lish this thing called the Twitter Politics and Elections Handbook, which is re­ally a guide for po­lit­i­cal can­di­dates and elected of­fi­cials and how to most ef­fec­tively use Twitter. And then they be­gin the roll­out of, “It’s a town hall in your pocket, and it’s im­prov­ing our democ­ra­cies be­cause we’re restor­ing the de­funct New England town meet­ing,” and that’s all just ba­nanas.

Objectively, noth­ing could be fur­ther from the truth. At that time, one in five Americans had a Twitter ac­count. Most peo­ple who had Twitter ac­counts had never used them, and above 90% of all tweets about pol­i­tics were posted by fewer than 10% of the peo­ple that did use Twitter all the time. There was no way in which Twitter was a rep­re­sen­ta­tion of the elec­torate. Twitter was a rep­re­sen­ta­tion of the most ex­treme, po­lit­i­cally ac­tive, hy­per-par­ti­san among Americans, who were fol­low­ing pol­i­tics re­ally avidly. Looking at it now, we can see, “Well, that’s re­ally just a dis­tor­tion ma­chine. And if politi­cians are us­ing it to gauge the elec­torate, they’re get­ting re­ally bad in­for­ma­tion.”

Again, you can say Twitter was not try­ing to par­tic­i­pate in the ar­ti­fi­cial state or un­der­mine democ­racy. Twitter is try­ing to do busi­ness and get more users and sell more what­ever. But it had these un­in­tended con­se­quences that then it sort of set­tles into and be­comes com­fort­able with.

I want to talk a lit­tle bit more about the struc­ture of the book. Like you said, it starts with this his­tory of tech­nol­ogy, his­tory of ideas, and the sec­ond half is about sci­ence fic­tion. Can you say more about how that struc­ture came to you and why you wanted to ad­dress things that way?

I be­came re­ally in­ter­ested, on the one hand, in how of­ten sci­ence fic­tion sto­ries pre­dict the ar­rival of what I then came to call the ar­ti­fi­cial state, and so I re­ally wanted to iden­tify a lit­er­ary tra­di­tion that I think of as the para­ble of the ar­ti­fi­cial state, in which ma­chines get more and more so­phis­ti­cated, they take over more and more of the func­tions of hu­mans, in­clud­ing the func­tions of gov­ern­ment, and even­tu­ally they come to rule the hu­mans, and then maybe they de­stroy all the hu­mans be­cause they don’t re­ally need them any­more.

Maybe they just en­slave them, it kind of de­pends. Are we in “The Terminator” or are we in “Battlestar Galactica”? There’s dif­fer­ent ver­sions, and these sto­ries go way back. They go back to the 1850s and the early decades of ru­mi­na­tion about the con­se­quences of in­dus­tri­al­ism.

I think a lot of peo­ple — this is cer­tainly true of my stu­dents, my un­der­grad­u­ates — re­ally be­lieve that tech­no­log­i­cal change equals progress. And not only that, but the only kind of progress is tech­no­log­i­cal change. That’s a nov­elty in hu­man his­tory. That’s an in­tel­lec­tual in­ven­tion of the 19th cen­tury, and it is partly be­cause tech­no­log­i­cal change was ac­cel­er­at­ing right at the time that Charles Darwin was de­vis­ing and then pub­lish­ing his the­ory of evo­lu­tion.

So there’s kind of a weird mar­riage be­tween evo­lu­tion as progress and tech­no­log­i­cal change as progress, and what drops out of that are all other, ear­lier no­tions of progress, which chiefly in­volve moral progress — like, things are get­ting bet­ter be­cause peo­ple are be­com­ing bet­ter, or things are get­ting bet­ter be­cause peo­ple are more free.

There are a lot of other ways we might think about progress, but what dom­i­nates to­day is this 19th-century no­tion of tech­no­log­i­cal progress as the only kind of progress, and there­fore all tech­no­log­i­cal change is progress, as op­posed to — ob­jec­tively, it’s only progress if things are get­ting bet­ter.

But in any event, that con­flu­ence in the 19th cen­tury of the idea of tech­no­log­i­cal progress and the idea of evo­lu­tion meant that peo­ple who were think­ing clearly were like, “Well, if the ma­chines keep get­ting bet­ter and faster and able to do more things — not just la­bor, but maybe talk or think or move around — what if they evolve to be­come bet­ter at every­thing than we are? Not just bet­ter at run­ning a loom, not just faster at mov­ing through time and space like a rail­road car?” And with that grew an in­cred­i­ble anx­i­ety that found form in sci­ence fic­tion again and again and again and again and again.

My fa­vorite one of these sto­ries was pub­lished, I think, in 1909 by E. M. Forster, right around when he was writ­ing “A Room with a View.” He wrote this story called “The Machine Stops,” which could be sub­ti­tled, “The Room Without a View.” He imag­ines a near fu­ture in which every­body just lives in these rooms, these lit­tle cells. You never see other peo­ple be­cause every­thing you need comes right to your room. It’s like DoorDash, your food is de­liv­ered, you have a screen where you can com­mu­ni­cate with other peo­ple. All your needs are met.

The thing that peo­ple fear most is the nat­ural world. No one wants to ever see the sun, it’s a lit­tle “Matrix”-y, and they all wor­ship the ma­chine that or­ga­nizes their lives and brings to them in their cubby-like rooms all the things that they need. The story is about, “Humans have be­come es­sen­tially slaves of the ma­chine, which is stronger, more pow­er­ful, and has more ca­pac­ity than hu­mans do, and hu­mans have lost what ca­pac­ity they had.” And then the cli­max of the story is when the ma­chine stops.

If read­ers were to go look at that story, it feels like it could be writ­ten to­day, ex­cept that it’s less sci­ence fic­tion-y to­day than it is the di­ary of a very un­happy YouTuber.

You con­nect that thread to some of the folks run­ning com­pa­nies and ar­guably run­ning as­pects of our gov­ern­ment to­day, like Elon Musk. Essentially, you sug­gest that they’re very bad sci­ence fic­tion read­ers. They read a lot of warn­ing sto­ries, or at least am­biva­lent sto­ries, as if they were man­u­als for the fu­ture.

This is some­thing I wres­tle with a reader of sci­ence fic­tion — some­one who loves Isaac Asimov, for ex­am­ple. I think it’s true that when Musk or Altman is just un­am­bigu­ously be­ing like, “Yes, this story is a tem­plate for what I should do with my com­pany,” that’s bonkers. But there is [also] this tech­no­cratic lib­er­tar­ian thread in sci­ence fic­tion that they are pick­ing up on. It’s not some­thing that they’re mak­ing up out of whole cloth, right?

Although weirdly, that’s Heinlein. That’s not Asimov, that’s not Douglas Adams.

Sure, there is that thread in sci­ence fic­tion. I don’t know, I guess [Jeff] Bezos is a big Robert Heinlein fan. You could say, “Okay, that lines up well. They’re read­ing it lit­er­ally, but at least they’re get­ting the po­lit­i­cal mes­sage that any ra­tio­nal per­son could find within that lit­er­ary work.”

But what’s funny about Musk is, the stuff he likes ac­tu­ally com­pletely de­feats and de­fies all of his po­lit­i­cal be­liefs.

You also say, re­peat­edly, that the ar­ti­fi­cial state in its cur­rent form is in­com­plete and doomed to fail­ure. Why is it doomed to fail­ure?

This is some­thing that’s fore­seen in all the sci­ence fic­tion that I dis­cuss.

It’s not an Asimov story, but it’s one of Asimov’s [favorite] sto­ries from his boy­hood [“The Man Who Awoke” by Laurence Manning] about a fu­ture in which the foresters have de­feated the wasters. […] The war that the fu­ture hu­mans had was be­tween the wasters, who just fig­ured you could just use every­thing up and waste it, and the foresters, who re­ally be­lieved in — we would call re­for­esta­tion and rewil­d­ing.

That’s gen­er­ally the ten­sion in these sto­ries. It’s be­tween the ar­ti­fi­cial state and the nat­ural world. To erect an ar­ti­fi­cial state and rule hu­mans within it, you must alien­ate them from the nat­ural world be­cause you are de­stroy­ing it. The ar­ti­fi­cial state will de­stroy the nat­ural world, and yet it needs the re­sources of the nat­ural world to run.

So, it is doomed in the sense that there is not a pos­si­bil­ity that the nat­ural world, a hab­it­able planet — hab­it­able for hu­mans — can sur­vive the full con­struc­tion and re­liance on the de­vices of the ar­ti­fi­cial state. That’s how the sci­ence fic­tion works, in any event.

Like you said, you’re a his­to­rian, not a politi­cian or a fu­tur­ist. But what do you think the de­feat of the ar­ti­fi­cial state looks like? Is it ba­si­cally just dis­man­tling all these com­pa­nies, tear­ing down the data cen­ters? Or is there a fu­ture that’s more about bring­ing it un­der con­trol?

I mean, I don’t have a play­book here, ex­cept for the rec­om­men­da­tion that we live in a democ­racy where de­ci­sions have to be made in con­sul­ta­tion with the gov­erned, and these de­ci­sions are not pop­u­lar.

You see this in all the lit­tle data cen­ter crises, town to town, county to county, state to state —  which are partly a con­se­quence of the de­cline of lo­cal news­pa­pers and the de­struc­tion of jour­nal­ism that has been one of the many con­se­quences of so­cial me­dia, and in the case of [Mark] Zuckerberg, I think a some­what in­ten­tional con­se­quence.

What you see is a lot of peo­ple show up at these town meet­ings and say, “We don’t even have hous­ing. We don’t have health­care. We don’t have jobs. Who said we’re build­ing this data cen­ter? I need to know a lot more about it. I need to know what its en­ergy costs are go­ing to be. Tell me about the wa­ter con­sump­tion. Are there go­ing to be jobs? Are the jobs go­ing to be long last­ing? Are they just go­ing to be for six months? What’s go­ing to hap­pen to the egrets that live in this area?” Whatever it is that peo­ple want to know.

More and more, you see peo­ple are — like in the Salt Lake ex­am­ple, where well over 70% of the peo­ple re­ally were op­posed to this data cen­ter, and their rep­re­sen­ta­tives sup­ported it. That’s not rep­re­sent­ing the peo­ple. I think there are po­lit­i­cal costs, and we’ll be­gin to see those at elec­tions.

Or maybe we won’t. Enough of de­mo­c­ra­tic func­tion­ing has to be in­tact for peo­ple to ac­tu­ally be able to re­spond to malfea­sance on the part of their rep­re­sen­ta­tives.

Part of your think­ing about [the ar­ti­fi­cial state] started with this great piece you wrote more than a decade ago for The New Yorker, about Clayton Christensen, cri­tiquing his idea of the in­no­va­tor’s dilemma and dis­rup­tive in­no­va­tion — which is very closely as­so­ci­ated with TechCrunch, be­cause we have a big con­fer­ence called Disrupt.

Ten years on, how do you feel about that idea of dis­rup­tive in­no­va­tion?

I stand by every­thing in that piece. [At the time, Lepore wrote, “Disruptive in­no­va­tion is a the­ory about why busi­nesses fail. It’s not more than that. It does­n’t ex­plain change. It’s not a law of na­ture.” Christensen re­sponded that Lepore broke “all the rules of schol­ar­ship that she ac­cused me of break­ing.”]

I reread it last sum­mer when I was work­ing on this book. What I would say here is, try­ing to be a peace­able hu­man be­ing, I think it re­ally is a prob­lem that his­to­ri­ans have not en­gaged with these ideas. One of the rea­sons I wrote that ar­ti­cle about dis­rup­tive in­no­va­tion — which was not an idea of mine, it was an as­sign­ment […] — was be­cause I just felt like, “Disruptive in­no­va­tion is a the­ory of his­tory. It’s a the­ory of his­tor­i­cal change, and it’s based on ev­i­dence from the archives.” And I just thought, as a his­to­rian, it makes no sense. His use of ev­i­dence is com­pletely un­ac­cept­able by any proper un­der­stand­ing of his­tor­i­cal method. Its ar­gu­ment is in con­ver­sa­tion with no mean­ing­ful un­der­stand­ing of how change hap­pens.

So I went and re­did the re­search, and it just did not stand up at all. I felt like I had to write it. And I wish that I felt like there were more en­gage­ment, in the years since, of aca­d­e­mic his­to­ri­ans think­ing through the na­ture of change — which are ques­tions that gen­uinely and au­then­ti­cally in­ter­est peo­ple who are in­volved in de­vel­op­ing new tech­nolo­gies.

People re­ally want to think [about], “What is this? What am I do­ing? What are go­ing to be the con­se­quences? Is there any­thing I could learn from his­tory? What hap­pened when the au­to­mo­bile re­placed the horse? What hap­pened to the law? How did we end up with dri­ver’s li­censes? How did we end up with traf­fic law? We did­n’t have traf­fic rules be­fore the au­to­mo­bile. We did­n’t have cer­tain kinds of in­sur­ance sys­tems. We did­n’t have dri­ver’s tests. How did those things emerge? How did [we de­velop] those guardrails on a tech­nol­ogy that was tremen­dously ex­cit­ing, im­proved peo­ple’s lives in many many ways, ut­terly changed the land­scape, rev­o­lu­tion­ized tort law? Maybe I should think about that.”

I just wish that his­to­ri­ans were more in con­ver­sa­tion with tech­nol­o­gists over these years, and with en­tre­pre­neurs. Not just be­cause we can stand around and say, “You know, I have a lec­ture to of­fer you on his­tory,” but I think there’s a real con­ver­sa­tion to be had.

All of which is just to say, thanks for hav­ing me on.

When you pur­chase through links in our ar­ti­cles, we may earn a small com­mis­sion. This does­n’t af­fect our ed­i­to­r­ial in­de­pen­dence.

ⓘ Microsoft

A new re­port shows Windows 11′s built-in Weather app can con­sume more than 1 GB of RAM. By com­par­i­son, Apple’s na­tive Weather app on ma­cOS uses roughly five times less mem­ory un­der sim­i­lar con­di­tions.

Microsoft has been work­ing to make Windows 11 more ef­fi­cient on PCs with lim­ited RAM, but one of its own built-in ap­pli­ca­tions ap­pears to be work­ing against that goal. According to tests pub­lished by Windows Latest, the op­er­at­ing sys­tem’s Weather app can con­sume more than 1 GB of mem­ory de­spite per­form­ing a rel­a­tively sim­ple task.

Windows Latest re­ports that the app ex­ceeded 1.2 GB of RAM while dis­play­ing a weather fore­cast, with no in­ten­sive in­ter­ac­tion from the user. Wccftech ob­served sim­i­lar be­hav­ior, not­ing that mem­ory us­age typ­i­cally starts at around 1 GB, drops to roughly 500 – 600 MB when idle, and can climb to 1.5 – 1.6 GB dur­ing ba­sic ac­tions such as zoom­ing or nav­i­gat­ing the in­ter­face. On a PC equipped with 8 GB of RAM, that means the ap­pli­ca­tion alone may oc­cupy nearly 20% of the sys­tem’s mem­ory.

By com­par­i­son, Apple’s na­tive Weather app on ma­cOS re­port­edly uses less than 250 MB of RAM un­der sim­i­lar con­di­tions, giv­ing Microsoft’s im­ple­men­ta­tion a mem­ory foot­print roughly five times larger.

According to Windows Latest, the high mem­ory con­sump­tion is due to the fact that Weather is not a fully na­tive Windows ap­pli­ca­tion. Instead, it is es­sen­tially an MSN Weather web app built on Microsoft’s WebView2 frame­work. Task Manager shows mul­ti­ple Chromium-based sub­processes run­ning si­mul­ta­ne­ously, which con­tributes to the un­usu­ally high RAM us­age.

The is­sue is un­likely to af­fect high-end PCs with 32 GB or more of RAM, but it could have a no­tice­able im­pact on en­try-level sys­tems. On com­put­ers with 8 GB or even 16 GB of mem­ory, launch­ing the Weather app may in­crease mem­ory pres­sure enough for Windows to rely more heav­ily on the page file, po­ten­tially mak­ing the sys­tem feel less re­spon­sive.

The ap­pli­ca­tion also in­cludes ad­ver­tis­ing within its in­ter­face. According to Windows Latest, spon­sored con­tent is em­bed­ded di­rectly into the fore­cast feed, ap­pear­ing along­side weather cards in a sim­i­lar vi­sual style. While Microsoft li­censes weather data from mul­ti­ple providers — in­clud­ing Foreca, the European Centre for Medium-Range Weather Forecasts (ECMWF), and other re­gional me­te­o­ro­log­i­cal ser­vices — the pres­ence of ads in a built-in Windows ap­pli­ca­tion has at­tracted crit­i­cism.

The find­ings also ap­pear to con­tra­dict Microsoft’s re­cent ef­forts to im­prove Windows 11′s ef­fi­ciency. The com­pany has been up­dat­ing sev­eral built-in ap­pli­ca­tions and has re­peat­edly said it wants the op­er­at­ing sys­tem to per­form bet­ter on lower-end hard­ware. Microsoft ex­ec­u­tive Rudy Huyn has also stated that the com­pany in­tends to de­velop more fully na­tive Windows ap­pli­ca­tions in the fu­ture, al­though it re­mains un­clear whether MSN-branded ap­pli­ca­tions such as Weather will even­tu­ally be re­built us­ing WinUI.

Andrew Sozinow - Tech Writer - 71 ar­ti­cles pub­lished on Notebookcheck since 2024

I’ve been fas­ci­nated by com­put­ers, elec­tron­ics and mod­ern tech­nol­ogy since child­hood. I started writ­ing IT-news at high school and have been do­ing it con­tin­u­ously for more than 10 years. During this time, I have worked for many me­dia out­lets, and now I am a news ed­i­tor at 3DNews. Sometimes I also write smart­phone re­views. In July 2024, I de­cided to try my hand at writ­ing news on Notebookcheck. When I’m not work­ing, I like to play videogames, do puz­zles, and travel.

Andrew Sozinov, 2026 – 08- 9 (Update: 2026 – 08- 9)

Bet 601

Duration 11 years (02011 – 02022)

STAKES $1,000

will go to Bletchly Park Trust if Keith wins, or The Internet Archive if Haughey wins.

Keith’s Argument

“Cool URIs don’t change” wrote Tim Berners-Lee in 01999, but link rot is the en­tropy of the web. The prob­a­bil­ity of a web doc­u­ment sur­viv­ing in its orig­i­nal lo­ca­tion de­creases greatly over time. I sus­pect that even a rel­a­tively short time pe­riod (eleven years) is too long for a re­source to sur­vive.

I would love to be proven wrong.

Haughey’s Argument

Though much of the web is ephemeral in na­ture, now that we have sur­passed the 20 year mark since the web was cre­ated and gone through sev­eral booms and busts, tech­nol­ogy and strate­gies have ma­tured to the point where keep­ing a site go­ing with a sta­ble URI sys­tem is within reach of any­one with mod­er­ate tech­no­log­i­cal knowl­edge. My old­est sites are go­ing on 13 years old at the time of this bet and the orig­i­nal URL scheme still func­tions via 301 redi­rects to a fi­nal for­mat we se­lected about six years ago.

Detailed Terms

On February 22nd, 2022 from 00:01 UTC un­til 23:59 UTC,entering the char­ac­ters http://​www.long­bets.org/​601 into the ad­dress bar of a web browser or com­mand line tool (like curl)ORus­ing a web browser to fol­low a hy­per­link that points to http://​www.long­bets.org/​601­MUS­Tre­turn an HTML doc­u­ment that still con­tains the fol­low­ing text: “The orig­i­nal URL for this pre­dic­tion (www.long­bets.org/​601) will no longer be avail­able in eleven years.”

A 301 redi­rect from www.long­bets.org/​601 to a dif­fer­ent URL con­tain­ing that text would also ful­fill those con­di­tions.

If those con­di­tions are met, Matt wins.

If those con­di­tions aren’t met, Jeremy wins.