The soft­ware de­vel­op­ment pro­fes­sion is in the midst of up­heaval. Nobody knows how the AI rev­o­lu­tion will play out in the end, but it is clear many as­pects of work and life will be trans­formed—in­clud­ing pro­gram­ming.

One of the com­ments I hear of­ten lately boils down to “LLMs may be good at cod­ing, but soft­ware was never the hard part” and “coding is easy, it’s fig­ur­ing out what to code that’s hard”.

I be­lieve that’s a gross in­sult to all pro­gram­mers every­where.

If cod­ing is easy…

If cod­ing is easy, how come pro­gram­mers were in high de­mand, and have de­manded large salaries for years (even be­fore ZIRP)? Why was there so much stress, over­work and burnout even be­fore AI started churn­ing out 5000-line PRs? Why did com­pa­nies seek 10x ninja rock­star coders and sub­ject them to leet­code in­ter­views—surely, a ju­nior fresh out of col­lege could churn out some­thing if it’s so easy?

If cod­ing is easy, why do we have doorstop­pers like Clean Code and The Pragmatic Programmer? Is The Art of Computer Programming a light sum­mer read? Is SICP a cof­fee-table book? Why do we have boot­camps or even whole col­lege de­grees ded­i­cated to it?

If cod­ing is easy, was Carmack just at the right place at the right time? Why do we con­sider Fabrice Bellard a ge­nius?

If cod­ing is easy, why are peo­ple an­gry at AI (or any­one else) copy­ing their code? Why do they act like they’ve poured their sweat, soul, and co­pi­ous amounts of time into some­thing so triv­ial?

If cod­ing is easy, why do many now feel like their iden­tity and pro­fes­sional pur­pose are be­ing stripped away from them?

If cod­ing is easy, why is soft­ware so damn buggy?

If fig­ur­ing out what to build is the hard part…

If de­cid­ing what to build is the hard part, why do so many prod­uct man­agers seem clue­less? Why aren’t there rig­or­ous 10-step in­ter­views for them? Why aren’t they get­ting paid more than the de­vel­op­ers?

If de­cid­ing what to build is the hard part, why aren’t mar­ket re­searchers, us­abil­ity ex­perts and—hell, cus­tomer suc­cess—con­sid­ered rock­stars in a soft­ware com­pany? If “understanding the cus­tomer” is harder, why are busi­ness an­a­lysts looked down on as pen­cil push­ers?

If im­ple­men­ta­tion is easy and find­ing de­mand is harder, why are pro­gram­mers up­set when the sales­peo­ple promise a new fea­ture to a cus­tomer to close the sale? They’ve found a gen­uine de­mand, some­thing peo­ple will pay for!

If cod­ing is easy, why does­n’t every­one just build ten vari­a­tions of a thing and see which pans out?

Another cliché com­ment is “most work in soft­ware de­vel­op­ment is talk­ing to stake­hold­ers, un­der­stand­ing the cus­tomer’s needs, and hav­ing clar­ity on the pri­or­i­ties”.

I have met many pro­gram­mers through­out my ca­reer, and very few of them want to talk to stake­hold­ers, much less cus­tomers (exceptions are free­lancers and founders, es­pe­cially of soft­ware de­vel­op­ment shops). And, “having clar­ity on the pri­or­i­ties” boils down to “just tell me what to do and don’t switch it up every two days”.

Some soft­ware de­vel­op­ers do say “I don’t write code, I solve cus­tomer’s prob­lems”. But then they turn around and start to opine on mon­ads, mem­ory safety, and DRY prin­ci­ples, while their un­der­stand­ing of the cus­tomer is a made-up “user per­sona”, and they think “affordance” is the money your par­ents used to give you on week­ends so you could go out and have a good time.

Yet oth­ers will say “Software de­vel­op­ment is the­ory build­ing”. Programs are ac­tu­ally proofs (as in, math­e­mat­i­cal proofs). Every com­mit should tell a story. And solv­ing a cus­tomer’s prob­lem by FTPing a PHP file is a car­di­nal sin.

I don’t mean to im­ply there are no de­vel­op­ers that si­mul­ta­ne­ously care deeply about the craft of soft­ware de­vel­op­ment and re­ally em­pathize with the cus­tomer. I do be­lieve they might want to see a pro­fes­sional about a split per­son­al­ity dis­or­der, tho.

What is im­por­tant?

I do be­lieve that talk­ing to users, un­der­stand­ing their ex­pe­ri­ence, em­pathiz­ing with them, solv­ing cus­tomers’ prob­lems and hav­ing all the stake­hold­ers on the same page is crit­i­cal to the suc­cess of a soft­ware pro­ject.

I also be­lieve that cre­at­ing good code is a craft that re­quires skill, pa­tience, at­ten­tion to de­tail, ex­pe­ri­ence and wis­dom, and that it will con­tinue to be rel­e­vant in the times ahead.

¿Por qué no los dos?

To the ex­tent that we can pull it off, I think we should aim for both. A deep un­der­stand­ing of the sys­tem we’re build­ing, to­gether with a deep un­der­stand­ing of why we’re build­ing it.

Loudly pro­claim­ing that “code is easy” or, at the op­po­site end, “code is art, a cre­ative hu­man ex­pres­sion that can­not be au­to­mated”, is just bury­ing our heads in the sand.

It’s cope. And you don’t want cope, you want to thrive.

By this, I don’t mean “jump on the LLM band­wagon.” I don’t mean “become a man­ager of fleets of AI agents.” I also don’t mean “AI-generated code is stolen slop garbage, fight it with tooth and nail, the bub­ble will pop soon enough any­ways.”

But do rec­og­nize we’re in the mid­dle of an in­dus­try-wide tec­tonic change. We need to fig­ure out how to adapt. We need to un­der­stand what is likely to change and what never changes.

What does­n’t change?

Software will be get­ting more com­plex. Software will al­ways need main­te­nance: bit-rot is a fact of life. So is en­tropy. Technology (hardware and soft­ware) will move for­ward, for bet­ter or worse. The tower (skyscraper?) of ab­strac­tions grows ever higher.

Users will al­ways want more and be pre­pared to spend less. They still won’t know how to re­lay their needs and wants. Worse, they still won’t know ex­actly what they want. The dis­con­nect be­tween the cus­tomers (who ac­tu­ally pay for the soft­ware) and users (who use it) will still be here, as will the ten­sion be­tween the needs of the busi­ness and the needs of its cus­tomers.

Also: there will never be a short­age of snake oil sales­men. Tech du jour comes and goes (I’m still wait­ing for the new VR re­nais­sance!)

What changes?

Programmers have been in the busi­ness of dis­rupt­ing our own in­dus­try since the be­gin­ning. Nobody uses punch-cards any more. Very few peo­ple need to code in as­sem­bly, or COBOL. Those decades spent fight­ing mem­ory bugs in C or C++, with the scars to prove it, are worth­less in the age of Rust, Go, Python and JavaScript.

I’m old enough to ap­pre­ci­ate val­grind or re­mem­ber mysql_re­al_es­cape_string() from the PHP4 era—stuff I’ll never again need in my life. And that was­n’t even so long ago! I nar­rowly missed the dBase, Clipper, HyperCard and Access era, tech­nolo­gies which I can still spot op­er­at­ing in shops, cafes, or a dusty, once beige and now golden-brown, midi-tower still hap­pily run­ning some be­spoke biz so­lu­tion (backups? what back­ups?)

How do we thrive?

Accept that change hap­pens. Be equal parts cu­ri­ous and crit­i­cal about the new stuff.

Understand there’s a lot of hype and try to dis­crim­i­nate be­tween hot air and what re­ally works (and to what ex­tent). Also be aware of ever-shift­ing goal­posts: stand back and look at the past year, or five, and as­sess the ve­loc­ity of change (technical, eco­nomic, so­ci­etal).

Your role and your re­spon­si­bil­i­ties will be chang­ing. Be will­ing to in­vest time and en­ergy into bet­ter un­der­stand­ing fields or roles ad­ja­cent to yours.

If you’re a se­nior de­vel­oper, don’t just find so­lace in deep­en­ing your ex­per­tise. Learn about user ex­pe­ri­ence, cus­tomer in­ter­views, or busi­ness strate­gies for the com­pa­nies in your do­main. It will help you gain a bet­ter ap­pre­ci­a­tion of all the work done to put a piece of soft­ware into users’ hands, whether or not you’ll ac­tu­ally ever have to do any of those other bits.

If you’re just start­ing or are ju­nior in your role: in­vest in deep­en­ing your un­der­stand­ing of how soft­ware works. Understanding point­ers, re­cur­sion, or mem­ory hi­er­ar­chy will help you even if you’re a JavaScript de­vel­oper. Understanding net­work pro­to­cols and how HTTP works will be use­ful even if you’re build­ing WordPress plu­g­ins. Do leet­code and learn about al­go­rithms and data struc­tures even if you don’t need to. Don’t be afraid to ask why and how ex­actly.

For in­spi­ra­tion, here are a few books and other re­sources that might be help­ful:

Structure and Interpretation of Computer Programs (PDF)

Cracking the Coding Interview

The Mythical Man-Month

Working Backwards

Team Topologies

7 Powers

The Soul of a New Machine

Obviously Awesome

The Design of Everyday Things

Don’t Make Me Think

Continuous Discovery Habits

The Mom Test

One more thing

Whoever you are, don’t out­source your un­der­stand­ing, judge­ment, em­pa­thy and taste to AI. Don’t ab­di­cate your re­spon­si­bil­ity. Don’t be a meat proxy.

PS. Lots of in­ter­est­ing and in­sight­ful com­ments over at Hacker News—the post re­ally hit a nerve. Fascinating how many dif­fer­ent ex­pe­ri­ences peo­ple have and the vary­ing de­f­i­n­i­tions of cod­ing, pro­gram­ming, de­vel­op­ment and en­gi­neer­ing they use.

Students at Dueholmskolen are pic­tured in their class­room in Nykøbing Mors, Jutland, Denmark, on March 1, 2021. Bo Amstrup/Ritzau Scanpix/AFP/Getty Images.

The pol­icy takes ef­fect im­me­di­ately, but ed­u­ca­tors say the first re­sponse must evolve as AI tools keep ad­vanc­ing.

Danish high school stu­dents will be re­quired to de­fend their writ­ten as­sign­ments orally un­der new gov­ern­ment mea­sures aimed at com­bat­ing cheat­ing with ar­ti­fi­cial in­tel­li­gence.

The Ministry of Education said that an oral de­fense will be­come manda­tory for all writ­ten as­sign­ments com­pleted at home and that it will work closely with schools to de­velop the best frame­work for im­ple­ment­ing these changes.

The reg­u­la­tion takes ef­fect im­me­di­ately and ap­plies to up­per-sec­ondary stu­dents, who are typ­i­cally around 16 years old. The mea­sure con­cerns ap­prox­i­mately 9,000 stu­dents en­rolled in the two-year HF (Higher Preparatory Examination) pro­gram, who are re­quired to sub­mit ma­jor writ­ten as­sign­ments each year, the min­istry said.

The min­istry is also urg­ing up­per-sec­ondary schools to use screen-mon­i­tor­ing tools dur­ing ex­ams and in­tro­duce fire­walls to re­strict the con­tent stu­dents can ac­cess dur­ing classes and fi­nal as­sign­ments.

In ad­di­tion, schools are ad­vised to have more as­sign­ments com­pleted on cam­pus un­der con­trolled con­di­tions so that su­per­vi­sion can be more ef­fec­tive.

Reaction from the ed­u­ca­tion com­mu­nity and next steps

Three or­ga­ni­za­tions rep­re­sent­ing school lead­ers, teach­ers, and up­per-sec­ondary stu­dents wel­comed the mea­sures but called for more last­ing so­lu­tions in re­sponse to the “rapid pace of tech­no­log­i­cal de­vel­op­ment,” the as­so­ci­a­tion Danske Gymnasier re­ported on Thursday.

“Unfortunately, we have a prob­lem with stu­dents us­ing AI to cheat in up­per-sec­ondary schools. Action is needed now, and we are start­ing with these three ini­tia­tives. In the com­ing pe­riod, I will in­volve schools, teach­ers, and stu­dents in dis­cus­sions about what can be done both in the short and long term to en­sure that AI does not un­der­mine stu­dents’ skills, their aca­d­e­mic abil­i­ties, or, not least, their ca­pac­ity for in­de­pen­dent thought,” — Magnus Heunicke

“Unfortunately, we have a prob­lem with stu­dents us­ing AI to cheat in up­per-sec­ondary schools. Action is needed now, and we are start­ing with these three ini­tia­tives. In the com­ing pe­riod, I will in­volve schools, teach­ers, and stu­dents in dis­cus­sions about what can be done both in the short and long term to en­sure that AI does not un­der­mine stu­dents’ skills, their aca­d­e­mic abil­i­ties, or, not least, their ca­pac­ity for in­de­pen­dent thought,”

– Magnus Heunicke

“These re­quire­ments in­clude hav­ing stu­dents clearly state when AI has been used in ma­jor writ­ten as­sign­ments and en­sur­ing that prepa­ra­tion for oral ex­ams takes place with­out ac­cess to AI,” — Anders Frikke

“These re­quire­ments in­clude hav­ing stu­dents clearly state when AI has been used in ma­jor writ­ten as­sign­ments and en­sur­ing that prepa­ra­tion for oral ex­ams takes place with­out ac­cess to AI,”

– Anders Frikke

According to Oscar Tønsberg Hoffmann, chair of the Danish Association of Upper-Secondary Students (DGS), it is im­por­tant that stu­dents have the op­por­tu­nity to help de­velop long-term so­lu­tions and par­tic­i­pate in shap­ing fu­ture poli­cies.

“It is im­por­tant that stu­dents have the op­por­tu­nity to help de­velop long-term so­lu­tions.” — Oscar Tønsberg Hoffmann

“It is im­por­tant that stu­dents have the op­por­tu­nity to help de­velop long-term so­lu­tions.”

– Oscar Tønsberg Hoffmann

The Danish Association of Upper-Secondary Schools also em­pha­sizes the need for a swift and con­sid­ered re­sponse to tech­no­log­i­cal de­vel­op­ment, while the Ministry of Education says that the three ini­tia­tives are only the be­gin­ning and that con­sul­ta­tions with ed­u­ca­tional in­sti­tu­tions, teach­ers, and stu­dents will con­tinue in both the short and long term.

The gov­ern­ment as­sured that ef­forts to pre­vent AI-assisted cheat­ing and de­velop crit­i­cal think­ing and in­de­pen­dent learn­ing skills will re­main a pri­or­ity in re­form­ing Denmark’s ed­u­ca­tion sys­tem.

US or EU? Your choice

Many of you have been telling us that where your email is stored mat­ters to you. There are var­i­ous rea­sons for this, like lo­cal law, keep­ing your data close to home, or com­pli­ance.

You can now make the European Union the pri­mary home for your Fastmail data, on our own se­cure servers in Amsterdam. Previously, all ac­counts were stored en­tirely in the US. Now you have more choice.

Built by us, not rented from some­one else

We’ve in­stalled our own servers, co-lo­cated in a se­cure fa­cil­ity in Amsterdam, set up by our own en­gi­neers. This new lo­ca­tion is built to the same high stan­dards as our ex­ist­ing in­fra­struc­ture in Philadelphia and St Louis, with our own hard­ware and our own soft­ware — spec­i­fied right down to the ex­act model of disks in each ma­chine.

In all our lo­ca­tions, data is stored en­crypted at rest in­side locked racks, and man­aged by our in-house team. We don’t rent com­put­ing or man­age­ment ser­vices from a big cloud provider and pass on their as­sur­ances. That’s how we’ve ap­proached pri­vacy, re­li­a­bil­ity, and per­for­mance for more than 25 years.

For many years, we have kept at least two copies of every­body’s email on sep­a­rate servers in their pri­mary lo­ca­tion, and at least one more in a ge­o­graph­i­cally sep­a­rate lo­ca­tion to en­sure data safety.

Here’s the de­tail about where your data will flow based on your re­gional choice.

If your ac­count is in the EU re­gion:

Your pri­mary copy of data will live in the EU. The main, live copy of your mail and files will sit on our own servers in Amsterdam.

Incoming mail will go to EU servers by pref­er­ence if you are us­ing your own do­main and Fastmail’s name­servers, or an ad­dress at one of Fastmail’s EU-region do­mains.

Our desk­top, web, and mo­bile apps will con­nect to EU servers.  Day to day, our apps will talk di­rectly to our Amsterdam in­fra­struc­ture. If those servers are ever un­avail­able, con­nec­tions fall back to one of our US lo­ca­tions so you can still reach your mail.

Resilient repli­cas of your data will live in the US (for now). As we only have one lo­ca­tion in Europe so far, the ge­o­graph­i­cally sep­a­rate copy will re­main on servers in one of our US lo­ca­tions.

If your ac­count is in the US re­gion:

Most of your data lives in the US. Both the pri­mary and replica lo­ca­tions of your mail and files will sit on our own servers in Philadelphia or St Louis.

Incoming mail goes to US servers by pref­er­ence if you are us­ing your own do­main and Fastmail’s name­servers, or an ad­dress at one of Fastmail’s US-region do­mains.

Our desk­top, web, and mo­bile apps will con­nect to US servers.  Day to day, our apps will talk di­rectly to whichever US lo­ca­tion con­tains your pri­mary data copy. If those servers are ever un­avail­able, con­nec­tions fall back to our other US lo­ca­tion so you can still reach your mail.

What ap­plies to every­one:

We favour avail­abil­ity, so if your home lo­ca­tion is down, you will tem­porar­ily con­nect to an­other of our lo­ca­tions so you can con­tinue to ac­cess your data.

If you are us­ing one of Fastmail’s generic non-re­gional do­mains then your mail may go in or out via ei­ther lo­ca­tion.

Emergency back­ups for every­body are stored in our Philadelphia lo­ca­tion. As well as the live repli­cas of your data, we also keep a sep­a­rate set of en­crypted back­ups taken every few hours for every ac­count. These are in Philadelphia for all users at the mo­ment.

Some data is repli­cated to all sites, so parts of every­one’s data are in both Europe and America. This in­cludes email ad­dresses and other user/​cus­tomer meta­data, stor­age for web­sites and the stand­alone Files fea­ture, and the de­tails of any third party ser­vices you have linked.

Logs are in the US. All sys­tem logs are con­sol­i­dated into a sin­gle place for mon­i­tor­ing sys­tem health and to as­sist with cus­tomer sup­port.

Third party ser­vices are shared. The third par­ties we use for de­bug­ging, billing, and sup­port are linked to your ac­count in the same way re­gard­less of your re­gion.

Your email client con­fig (IMAP/POP3) does not have to change. The generic host­names will proxy in­ter­nally to your ac­tive server. We’re work­ing on mak­ing those ter­mi­nate at the clos­est net­work lo­ca­tion. However, we also of­fer re­gional server names for each pro­to­col to give you more con­trol over where you con­nect.

We’re an Australian com­pany, sub­ject to Australian law in­clud­ing le­gal-co­op­er­a­tion treaties be­tween Australia and other coun­tries. Wherever your data is stored, we will re­spond the same way to law­ful re­quests from rel­e­vant au­thor­i­ties (see our trans­parency re­port).

If what you need is a guar­an­tee that your data re­mains only in the EU, we don’t have that, and we’d rather tell you di­rectly than let you as­sume oth­er­wise.

We made a pre­dic­tion, but you can change it

We pre-se­lected all the users with billing ad­dresses in or near Europe for the EU re­gion. If you are one of these, an en­crypted copy of your data was trans­ferred to Europe in ad­vance of this an­nounce­ment, and will shortly be­come your pri­mary copy. You can change re­gion to the US and your data will mi­grate back.

Conversely, if we did­n’t iden­tify you for our ini­tial group, you can put your­self in the queue to be mi­grated. Moves this way will be a lit­tle slower be­cause there’s no copy of the data al­ready in the re­gion, so we have to sync every email across the ocean! For those mov­ing back, there’s al­ready a copy in the US, so only a small amount of data needs to be syn­chro­nised to fully rec­on­cile your mail­box.

This blog post is all about giv­ing you the facts and the tools to make the trade-off that’s right for you. Most providers de­cide for you and tell you as lit­tle as they can get away with. We’d rather show our work — what’s repli­cated, what is­n’t, who can com­pel what — and let you choose with your eyes open.

How it works

When you sign up, you choose your re­gion, and the pri­mary copy of your data is placed on our se­cure servers in that re­gion, with a copy also repli­cated to a ge­o­graph­i­cally sep­a­rate lo­ca­tion. Regardless of your choice, all copies of your data are en­crypted at rest. Choosing a re­gion changes where your pri­mary copy lives, not how well it’s pro­tected.

If you were with us when we se­lected the users to trans­fer, we’ve pre-set your re­gion based on your billing ad­dress. If you signed up more re­cently, you’ll have been al­lo­cated to the US. Either way, if you’d pre­fer a dif­fer­ent re­gion, you can switch it in your set­tings.

Switching re­gion

Go to Set­tings → Users & Sharing → Team Settings, then look be­low GDPR for the Data res­i­dency sec­tion. Pick your lo­ca­tion and we’ll move your pri­mary copy for you. You’ll see it go from queued, to trans­fer­ring, to done, and your ac­count keeps work­ing the whole time. You can change your mind and move back, though we may place rea­son­able lim­its on how fre­quently you can change re­gion!

Your data, your call

We’re re­ally ex­cited to have an­other lo­ca­tion, and to be able to of­fer this op­tion to you. Setting this up was a sig­nif­i­cant in­vest­ment. We con­sid­ered a sur­charge for choos­ing the EU, but we don’t be­lieve that’s right. We proudly charge a fair rate for an ex­cep­tional ser­vice, and this choice should be yours. Select the re­gion that is right for you.

What it is

_for-sale is a re­served DNS leaf node name, de­fined by RFC 10023 (Informational, July 2026) and reg­is­tered with IANA. A TXT record pub­lished at _for-sale.example.com sig­nals that ex­am­ple.com, al­though reg­is­tered and re­solv­ing nor­mally, is avail­able for pur­chase.

_for-sale IN TXT “v=FORSALE1;furi=https://​ex­am­ple.com/​for-sale”

The record car­ries a manda­tory ver­sion tag fol­lowed by at most one tag=value pair:

The wrong be­lief to clear first is that this is a way of park­ing a do­main. It is close to the op­po­site. Parking re­places the site with a sales page, which costs you every vis­i­tor the do­main still has. _for-sale sits be­side a live site in DNS and says noth­ing to a browser: the home­page keeps serv­ing, the mail keeps flow­ing, and the record can be added and re­moved at will. RFC 10023 makes the point ex­plic­itly — the con­ven­tion is de­signed to work while the do­main is still in ac­tive use.

It is also not the same thing as reg­is­tra­tion data. WHOIS and RDAP an­swer “is this name reg­is­tered?”; a reg­is­tered name may still be pur­chasable, and an un­reg­is­tered one may not be worth hav­ing. That gap is the whole rea­son the con­ven­tion ex­ists, and it is why bro­kers and au­to­mated avail­abil­ity ser­vices are the in­tended au­di­ence rather than peo­ple.

Why it mat­ters

The sig­nal a do­main owner most wants to send is the one there has never been a chan­nel for. If you are will­ing to sell, the in­ter­ested buyer has no way to learn that short of a cold email to a WHOIS con­tact that pri­vacy redac­tion has prob­a­bly re­moved. Enquiries that would have been wel­come never ar­rive, and the ones that do ar­rive are in­dis­tin­guish­able from spam.

Putting the sig­nal in DNS rather than on the page is what makes it use­ful to the par­ties who can act on it. A bro­ker or an avail­abil­ity ser­vice check­ing a name re­solves it any­way; one ex­tra lookup tells them what a ren­dered page could not, be­cause noth­ing on a work­ing home­page says “the do­main un­der this is ne­go­tiable”. It is ex­ter­nally check­able, costs one record, and car­ries no risk to the site it­self — a browser never sees it.

How to im­ple­ment

Publish a sin­gle TXT record at the _for-sale leaf of the zone you are sell­ing, and only while you mean it.

; Free text _for-sale IN TXT “v=FORSALE1;ftxt=Serious of­fers only”

; A URI to ne­go­ti­ate through — https, mailto and tel are the us­able schemes _for-sale IN TXT “v=FORSALE1;furi=https://​ex­am­ple.com/​fs?d=eHl6”

; An ask­ing price: up­per­case cur­rency code, then the amount _for-sale IN TXT “v=FORSALE1;fval=USD12500”

Rules worth get­ting right the first time:

The ver­sion tag is manda­tory and case-sen­si­tive: every record starts v=FOR­SALE1;. It ex­ists so a proces­sor can tell a real _for-sale record from an un­re­lated TXT record that a DNS wild­card hap­pened to ex­pand into that name.

One tag-value pair per record. To pub­lish a price and a con­tact URI, pub­lish two records in the same RRset and let the proces­sor pick what it un­der­stands. This is not SPF; the pairs do not con­cate­nate.

One char­ac­ter-string per record, 255 octets max­i­mum, so noth­ing has to be re­assem­bled dur­ing pars­ing.

Keep the TTL at 3600 sec­onds or less. A stale record ad­ver­tis­ing a price you have with­drawn, or a do­main you al­ready sold, is worse than no record.

Place it at a leaf. _for-sale.example.com is valid at any level of the tree, but xyz._for-sale.ex­am­ple.com is not, and records un­der .arpa must be ig­nored — an of­fer to sell ad­dress space is out of scope.

Remove it when the do­main is no longer for sale. The con­ven­tion has no “not for sale” value; ab­sence is the only way to say no.

Sign the zone with DNSSEC if you can. An un­signed TXT record as­sert­ing your do­main is for sale, at a price, with a con­tact URI, is a com­fort­able thing for some­one else to forge.

This site does not ship a _for-sale record: spec­i­fi­ca­tion.web­site is not for sale.

Common mis­takes

Cramming sev­eral pairs into one record. “v=FORSALE1;fval=EUR2500;furi=https://…” looks rea­son­able and is not what the for­mat de­fines. Use one pair per record, mul­ti­ple records per RRset.

Publishing it as­pi­ra­tionally. The in­di­ca­tor is only for do­mains ac­tu­ally avail­able. It is not a mar­ket­ing ban­ner, and a record that ex­ists to lure en­quiries is an abuse the RFC calls out by name.

Assuming it obliges any­one. Publishing the record does not com­mit the holder to sell, and an ad­ver­tised fval= price is in­dica­tive — the RFC tells proces­sors to dis­play a dis­claimer and never to treat it as a pur­chase com­mit­ment.

Expecting a wild­card to cover a whole zone. _for-sale.*.example.com is not a valid wild­card. There is no way to put every do­main un­der a TLD up for sale with one record.

Trusting the con­tent. If you are on the read­ing side, ftxt= is at­tacker-con­trolled text and furi= is an at­tacker-con­trolled URI. Sanitise be­fore dis­play — the RFC’s own ex­am­ple con­tent is <script>…</script> — and never auto-nav­i­gate a user to a furi= tar­get with­out an ex­plicit con­fir­ma­tion step.

Verification

dig +short TXT _for-sale.example.com

The an­swer be­gins with v=FOR­SALE1; and con­tains at most one tag=value pair per string.

The TTL is 3600 or lower: dig TXT _for-sale.example.com | grep _for-sale.

If the zone is signed, dig +dnssec TXT _for-sale.example.com re­turns a val­i­dat­ing RRSIG.

The record re­solves at all. During a re­demp­tion or pend­ingDelete pe­riod, or when DNSSEC val­i­da­tion is bo­gus, the name will not re­solve and the sig­nal silently dis­ap­pears.

Related top­ics

Sources & fur­ther read­ing

For a while, my per­sonal in­fra­struc­ture lived on a small Hetzner VPS. It ran a few web apps, a re­mote browser called Surf, Caddy, and the usual sup­port­ing cast. Nothing par­tic­u­larly se­ri­ous. It worked, I just did­n’t like pay­ing for it.

One of the apps I run, Surf, made the com­pro­mise dif­fi­cult to ig­nore. The cheap­est shared ma­chines were fine un­til Chrome had real work to do, at which point they felt starved. Dedicated CPU ma­chines fix that, but cost enough each month to make a per­sonal browser feel like a ques­tion­able fi­nan­cial com­mit­ment.

Buying an­other ma­chine was­n’t an ap­peal­ing es­cape hatch ei­ther. DRAM prices have gone com­pletely stu­pid, so putting to­gether a new box with a com­fort­able amount of mem­ory felt es­pe­cially ill timed. I looked at used mini PCs and briefly con­sid­ered turn­ing my desk­top into a server when­ever I was­n’t us­ing it. Then I re­mem­bered the CMF Phone 1 I al­ready own.

Eight ARM cores, 8 GB of RAM, 128 GB of flash, Wi-Fi 6, a 5G mo­dem, and a built in bat­tery backup, that is all at­tached to an SoC that I feel is overqual­i­fied for sit­ting in a drawer. And I had al­ready paid for it. After dust­ing it off and play­ing with it for a while, I de­cided to turn the phone into the server.

Today it runs Surf and its man­aged Chrome in­stance, my per­sonal fi­nance tracker, a screen shar­ing ser­vice, and a hand­ful of smaller web apps. They sur­vive re­boots, de­ploy from Git, and re­main reach­able when the phone moves be­tween net­works, so this is now the ma­chine that ac­tu­ally re­placed the VPS.

the first bad idea: re­plac­ing an­droid

The clean­est ver­sion of this idea seemed to be flash­ing a nor­mal Linux dis­tri­b­u­tion. The CMF Phone 1 has a post­mar­ke­tOS de­vice port, it boots, and the de­vice page has enough green boxes to make a reck­less per­son op­ti­mistic. I ended up be­ing that per­son.

What I paid less at­ten­tion to was every­thing marked bro­ken: Wi-Fi, Bluetooth, hard­ware ac­cel­er­a­tion, and most of the other things that make the phone use­ful as a small server. I got as far as the post­mar­ke­tOS splash screen and a black dis­play. At that point I had nei­ther a server nor a phone.

Recovering stock Nothing OS turned into its own side quest. The flash­ing util­ity needed Windows, so I in­stalled Windows in QEMU, fought USB passthrough and MediaTek dri­vers, watched the flash­ing tool hang, then even­tu­ally moved the process to an ac­tual Windows in­stal­la­tion and re­stored the fac­tory im­ages.

There was a mo­ment in the mid­dle of this where the phone was soft bir­cked and only showed a black screen and I gen­uinely thought I had con­verted a per­fectly good de­vice into a pa­per­weight.

It came back, and les­son learned: Android al­ready has work­ing dri­vers for every piece of this hard­ware. Wi-Fi, power man­age­ment, the bat­tery, the GPU, the mo­dem, and every weird ven­dor de­tail al­ready work. Throwing all of that away in pur­suit of a more con­ven­tional user­space was the wrong trade.

I did­n’t ac­tu­ally need the phone to be­come a nor­mal Linux ma­chine. I needed it to run Linux ap­pli­ca­tions re­li­ably while Android con­tin­ued do­ing the hard­ware spe­cific work it is good at.

ter­mux is the host op­er­at­ing sys­tem

The sec­ond at­tempt kept stock Android and treated Termux as the host en­vi­ron­ment.

Termux gives me OpenSSH, runit, Caddy, Cloudflared, pack­age man­age­ment, and nor­mal enough Unix tool­ing. Termux:Boot starts the su­per­vi­sor and SSH af­ter a re­boot. Tailscale gives the phone a sta­ble pri­vate ad­dress, so from any ma­chine on my tail­net I can just run:

ssh cmf

Termux is not a vir­tual ma­chine. Its processes still ex­e­cute against Android’s Linux ker­nel, but its Bionic based user­space is dif­fer­ent enough from an or­di­nary Debian in­stal­la­tion that ex­ist­ing Linux ap­pli­ca­tion im­ages can­not sim­ply be dropped into it. That split ended up be­ing use­ful, though: Termux could re­main the small host con­trol plane while each ap­pli­ca­tion brought the Linux filesys­tem it ex­pected.

The ac­tual ser­vices are su­per­vised by runit. Android’s bat­tery man­age­ment is very good at its nor­mal job and very bad for a de­vice pre­tend­ing to be a server, so I had my Ansible build also ap­ply an Android host pro­file: it in­stalls a per­sis­tent wake lock, dis­ables light and deep idle, ex­empts Termux, Termux:Boot, and Tailscale from back­ground re­stric­tions, dis­ables the child process lim­iter, pre­vents Wi-Fi sus­pen­sion, and con­fig­ures Tailscale as the al­ways on VPN.

The re­cov­ery chain mat­ters more than any in­di­vid­ual set­ting. Android boots, al­ways on VPN brings Tailscale back, Termux:Boot starts runit, runit starts every res­i­dent ser­vice, and health checks ver­ify the lo­cal and pub­lic paths. The phone can re­boot with­out wait­ing for me to no­tice.

Android boot -> Tailscale al­ways-on VPN -> Termux:Boot -> runit -> res­i­dent ser­vices -> lo­cal and pub­lic health checks

This is­n’t a con­ven­tional Linux server. There is no sys­temd, no nor­mal Docker dae­mon, and no rea­son to pre­tend oth­er­wise. But it is a Linux ker­nel with a very ca­pa­ble user­land sit­ting on top of it, and that turns out to be enough.

the sec­ond bad idea: proot

Most of my ap­pli­ca­tions al­ready shipped as Linux ARM64 OCI im­ages. proot-dis­tro made those sur­pris­ingly easy to run un­der Debian with­out chang­ing the ap­pli­ca­tions them­selves.

PRoot in­ter­cepts filesys­tem and process op­er­a­tions in user­space and makes a reg­u­lar Termux process be­lieve it lives in­side a Debian root filesys­tem. It is­n’t a con­tainer bound­ary. Everything still shares Android’s ker­nel, net­work name­space, and Termux UID. But as an ap­pli­ca­tion com­pat­i­bil­ity layer, it is ex­tremely use­ful be­cause it needs nei­ther root nor a spe­cial ker­nel.

The or­di­nary web ser­vices ini­tially ran fine this way. Each of my ap­pli­ca­tions got a ver­i­fied root filesys­tem, a loop­back port, and a runit ser­vice. Caddy ran di­rectly in Termux and routed host­names to those ports.

The per­for­mance/​la­tency sen­si­tive Surf browser work­load was the ex­cep­tion. Starting processes, open­ing li­braries, walk­ing paths, read­ing browser pro­files, and shuf­fling cap­ture data all crossed PRoot’s user­space trans­la­tion layer. There was CPU avail­able, but Chrome could not reach it ef­fi­ciently. So I rooted the phone, not to re­place Android, but to mount the same Debian filesys­tem prop­erly and en­ter it with a real ch­root.

Runit still owned the life­cy­cle from Termux, con­fig­u­ra­tion still came from the same place, and ap­pli­ca­tion data still lived in Termux stor­age. The work­load sim­ply reached Android’s ker­nel through na­tive syscalls in­stead of PRoot. The im­prove­ment was not sub­tle!

Once that path was solid, leav­ing the smaller res­i­dents un­der PRoot stopped mak­ing much sense. They now run the same way. My work­sta­tion re­solves each ARM64 im­age to an ex­act di­gest and ex­ports its filesys­tem; Ansible ver­i­fies and in­stalls it on the phone. A small root helper cre­ates a pri­vate mount name­space, binds the re­quired paths, en­ters the filesys­tem with ch­root, drops priv­i­leges, and starts the orig­i­nal im­age en­try­point.

Neither Docker nor a com­piler needs to ex­ist on the phone. These are still com­pat­i­bil­ity en­vi­ron­ments rather than se­cu­rity bound­aries: the res­i­dents share Android’s ker­nel and net­work stack, while the pri­vate mount name­spaces mainly keep mounts and cleanup pre­dictable.

I also spent far too long try­ing to bridge Debian’s graph­ics stack to the phone’s Mali GPU through VirGL and Android Vulkan. I got hard­ware com­posit­ing check­marks along­side cor­rupt pages and worse per­for­mance… The bor­ing soft­ware ren­dered path turned out to per­form bet­ter.

in­fra­struc­ture! not a pile of shell his­tory

By this point the phone could run every­thing, but I did­n’t want a pet server as­sem­bled from com­mands I would for­get in a week. I moved the en­tire host into an Ansible man­aged state: ver­sions, ser­vice de­f­i­n­i­tions, routes, power set­tings, se­crets, and health checks all live in one pri­vate repos­i­tory.

The de­ploy­ment flow is roughly:

re­lease or OCI im­age -> check­sum/​di­gest pinned in Git -> Ansible over SSH -> ver­sioned files on the phone -> atomic cur­rent sym­link -> runit ser­vice -> lo­cal health check -> pub­lic edge check

Releases are pinned by di­gest or check­sum and in­stalled into ver­sioned di­rec­to­ries be­hind an atomic cur­rent sym­link. A failed check­sum or health check stops the de­ploy­ment, roll­back means re­vert­ing the pin and ap­ply­ing again. Application data lives sep­a­rately from re­leases.

After the small man­ual boot­strap: in­stall the three Android apps, root the phone, grant Termux su­pe­ruser ac­cess, and au­tho­rize SSH, the same repos­i­tory owns the rest. From my work­sta­tion, bring­ing the host to the de­clared state is de­lib­er­ately sim­ple:

make phone make phone-sta­tus make phone-edge-check

Applying it again does not re­place un­changed run­time files or restart healthy res­i­dents. More im­por­tantly, the con­fig­u­ra­tion is use­ful if this phone dies: an­other rootable ARM64 phone can be brought to­ward the same state with­out re­con­struct­ing a shell his­tory.

Secrets are not stored on the phone’s Git check­out, be­cause there is no Git check­out. Ansible Vault val­ues live en­crypted in the in­fra­struc­ture repos­i­tory. The vault pass­word is de­rived by ask­ing my 1Password SSH agent to sign a fixed chal­lenge, so the pri­vate key stays in 1Password and the phone never needs ac­cess to it. During de­ploy­ment, Ansible ren­ders only the run­time val­ues each ser­vice needs into Termux’s pri­vate stor­age.

get­ting traf­fic to a phone be­hind home in­ter­net

The next prob­lem to tackle was ingress. My home con­nec­tion does not come with the kind of sta­tic server setup a VPS gives you, and I don’t want to ex­pose SSH or a col­lec­tion of ran­dom ap­pli­ca­tion ports through the router. I also wanted the phone to re­main a phone in one im­por­tant sense: I should be able to un­plug it, take it some­where else, con­nect it to the in­ter­net, and still have my server.

The HTTP ap­pli­ca­tions use a Cloudflare Tunnel. Cloudflared makes one out­bound con­nec­tion from the phone, Cloudflare sends each host­name through it, and Caddy routes the re­quest to the cor­rect loop­back ser­vice.

Internet -> Cloudflare Tunnel -> Caddy on 127.0.0.1 -> ap­pli­ca­tion on 127.0.0.1

There is no in­bound router rule for those ser­vices. Cloudflared only needs an out­bound con­nec­tion, so if I move the phone to an­other net­work, the tun­nel re­con­nects and the host­names fol­low it. Tailscale does the same for ad­min­is­tra­tion. The phone’s bat­tery can bridge the move, and the pub­lic ser­vices do not care which Wi-Fi hap­pens to be un­der­neath them.

The Surf re­mote browser back­end needed a dif­fer­ent path though. Its di­rect con­nec­tion is la­tency sen­si­tive, ter­mi­nates its own TLS, and the old iPad that con­nects to it pins the server iden­tity. At home, Cloudflare DDNS keeps a DNS only record pointed at the cur­rent pub­lic ad­dress and the router for­wards one port to Surf. On the LAN, the iPad con­nects di­rectly to the phone.

That still left roam­ing. A nor­mal Cloudflare Tunnel ter­mi­nates TLS at Cloudflare, which is ex­actly what Surf’s pinned con­nec­tion does not want. The so­lu­tion was to wrap the com­plete Surf TLS stream in­side an or­di­nary WebSocket. Cloudflare sees and for­wards the WebSocket, but the ac­tual au­then­ti­cated Surf con­nec­tion re­mains en­crypted end to end in­side it.

This ob­vi­ously adds la­tency, from out­side home I usu­ally see roughly an­other net­work round trip, and the iPad con­nec­tion was around 60 ms when I first tested it. But it works through a tun­nel that only needs out­bound con­nec­tiv­ity, on an op­er­at­ing sys­tem from 2012, with­out in­stalling Tailscale on the iPad.

That was the mo­ment the phone setup be­came slightly ridicu­lous in the best way. I left it plugged in at home, went to the of­fice, SSHed from my MacBook through Tailscale into my work­sta­tion and the phone, and used the orig­i­nal iPad through the phone hosted Surf in­stance. The VPS was no longer, it felt lib­er­at­ing.

what is ac­tu­ally run­ning

The ma­chine has two lay­ers. Android and Termux own the hard­ware, net­work­ing, ingress, and su­per­vi­sion. The Linux res­i­dents get the filesys­tem they ex­pect and oth­er­wise stay out of the host’s way.

The de­mand­ing work­load is Surf, which brings a mod­ern Chromium browser to old iPhones and iPads. The phone runs desk­top Chrome (recent ar­m64 re­lease of Chrome) and the Surf back­end in­side its Debian run­time, an iPad mini re­ceives H.264 video and au­dio while send­ing touch, key­board, tab, and nav­i­ga­tion com­mands back. Surf is why I cared so much about syscall over­head and la­tency in the first place.

Alongside it, the phone hosts my per­sonal fi­nance tracker. It records re­cur­ring in­come and ex­penses, one time trans­ac­tions, monthly spend­ing lim­its, and pro­jects my bal­ance for­ward day by day. Unlike the re­place­able ap­pli­ca­tion ar­ti­facts, its SQLite data­base con­tains state I care about, so it has au­to­mated off de­vice back­ups and a tested re­store path.

None of those names are baked into some grand “phone server frame­work.” A new res­i­dent is an­other im­mutable ar­ti­fact, a runit de­f­i­n­i­tion, an ex­plicit data con­tract, a health check, and op­tion­ally a Caddy route.

The lat­est ad­di­tion is ob­serv­abil­ity. I can still SSH in, in­spect runit, tail logs, check pub­lic routes, and query Android di­rectly, but I no longer have to do that merely to see what’s go­ing on with the ma­chine at a glance: One na­tive ser­vice col­lects CPU us­age for all eight cores, mem­ory, stor­age, up­time, bat­tery, ther­mals, lo­cal and pub­lic reach­a­bil­ity, and every dis­cov­ered runit res­i­dent. It keeps bounded his­tory and serves an em­bed­ded Vue in­ter­face at https://​dash.cmf, reach­able only over the LAN or tail­net.

The log view dis­cov­ers the same ser­vice di­rec­to­ries at run­time, so adding an­other res­i­dent does­n’t re­quire teach­ing the UI its name. It is much nicer than SSHing in just to re­mem­ber which thing was noisy.

should you do this?

If you al­ready have a rea­son­ably mod­ern, rootable ARM64 phone sit­ting un­used, this is much less ridicu­lous than it sounds. You get quiet hard­ware, low power con­sump­tion, flash stor­age, Wi-Fi, a built in dis­play for re­cov­ery, and a bat­tery that be­haves like a tiny UPS. Stock Android al­ready sup­ports the hard­ware, while Termux and a rooted ch­root are enough to run a sur­pris­ing amount of nor­mal Linux soft­ware.

I would not put ir­re­place­able data on one with­out au­to­mated off de­vice back­ups, and I would not treat the ch­roots as hos­tile work­load iso­la­tion. Rooting ex­pands the trust bound­ary, Android re­mains an un­usual server host, and soft­ware ren­dered desk­top Chrome is not go­ing to beat a ded­i­cated work­sta­tion with a GPU.

But for a hand­ful of per­sonal ser­vices, es­pe­cially when the al­ter­na­tive is pay­ing in­def­i­nitely for a VPS that is ei­ther slow or an­noy­ingly ex­pen­sive, it is a gen­uinely use­ful op­tion rather than only a stunt.

The phone is still a weird server. It shares one ker­nel, Android oc­ca­sion­ally needs to be re­minded not to “optimize” it, and a fu­ture Android up­date could al­ways cre­ate a new sur­prise.

But it is quiet, bat­tery backed, fast enough, reach­able from any­where, re­pro­ducible from Git, and al­ready sit­ting in my house. I started this try­ing to save money on a VPS. I ended up with a rooted phone run­ning my per­sonal in­fra­struc­ture, and some­how that is much more sat­is­fy­ing. :)

Postscript: This post was dis­cussed on Hacker News. In true HN fash­ion, a sur­pris­ing amount of the con­ver­sa­tion ended up be­ing about the ti­tle. I en­joyed the ac­ci­den­tal lin­guis­tics thread. :)

Table of Contents

1. A*’s use of the heuris­tic

2. Perfect heuris­tic

3. Reusing a per­fect heuris­tic

4. Multiple land­marks

5. Placement of land­marks

6. Automated place­ment

7. Implementation

8. Demos

8.1. Dragon Age, The Circle Tower 8.2. Cogmind, Factory 5 8.3. Maze 8.4. Dragon Age, Lothering 8.5. Cogmind, Research 2 8.6. Cogmind, Factory 4

8.1. Dragon Age, The Circle Tower

8.2. Cogmind, Factory 5

8.3. Maze

8.4. Dragon Age, Lothering

8.5. Cogmind, Research 2

8.6. Cogmind, Factory 4

9. More read­ing

For op­ti­miz­ing A* we usu­ally look at the pri­or­ity queue or the map rep­re­sen­ta­tion. Often over­looked is im­prov­ing the heuris­tic func­tion. Here’s an ex­am­ple from the town of Denerim in Dragon Age Origins. Try mov­ing the start B and goal to see A* in ac­tion:

Now try mov­ing the green L to be near the pur­ple . As the heuris­tic value gets closer to the true dis­tance , the num­ber of nodes A* has to ex­plore de­creases from to . The blue area is the sav­ings.

On this page I’ll show a way to im­prove the heuris­tic to speed up A*. At the end of the page I show this tech­nique with maps from real games.

1  A*’s use of the heuris­tic#

A* uses a heuris­tic to guide it to­wards the goal. We can think of it like wind push­ing us in the right di­rec­tion. Here, the heuris­tic pushes us east, and the short­est path goes east:

But some­times it pushes us in the wrong di­rec­tion. Here, the short­est path is to the west but the heuris­tic pushes us east:

A* runs faster when the heuris­tic guides us in the right di­rec­tion. It wastes time when the heuris­tic guides us in the wrong di­rec­tion. But why is it in the wrong di­rec­tion? It’s be­cause the usual dis­tance-based heuris­tic does­n’t know about the walls.

2  Perfect heuris­tic#

Ideally, we’d find a heuris­tic that knows about walls and never points in the wrong di­rec­tion:

Can we cal­cu­late this “perfect” heuris­tic? Yes!

But the per­fect heuris­tic is dif­fer­ent for every goal and wall con­fig­u­ra­tion. Move the goal and you’ll see the heuris­tic changes. Move the start B and you’ll see it does­n’t.

If the goal and walls stay the same, then we can use flow field pathfind­ing. But usu­ally the goal is­n’t the same, so we need to con­struct a brand new per­fect heuris­tic for each goal. That is im­prac­ti­cally slow to cal­cu­late every time we run A*, and it’s also im­prac­ti­cally too large to store if we want to com­pute it ahead of time.

It’d be nice if we could cal­cu­late the heuris­tic once and then reuse it for mul­ti­ple A* runs with dif­fer­ent goals.

3  Reusing a per­fect heuris­tic#

Let’s cal­cu­late a per­fect heuris­tic to the green L, which we call a “landmark”. Can we reuse it for an­other goal ? Yes, some­times! Move the start point B and the land­mark L around to see which pur­ple goals are helped:

The idea is that if we al­ready have the path from B→L, we also get the short­est path to any along the way:

Think of the land­mark as some­thing far in the dis­tance. Your friend tells you “from your house B, walk to­wards the Eiffel Tower L un­til you get to Daniel’s house ”. The goal is not to reach the land­mark. The land­mark tells us a di­rec­tion to go in. The goal, Daniel’s house, is on the way.

Most goals aren’t on the path B→L but some­times they are close to that path:

But what does it mean to be “close”? We can use the path length, cost(B, L). When the paths are al­most the same, cost(B, L) is close to cost(B, X) + cost(X, L).

In A*, we use the heuris­tic func­tion as a lower bound for the path length cost(B, X). The tri­an­gle in­equal­ity[1] says that the sum of two sides of a tri­an­gle is at least as long as the third side. Adapted for di­rected graphs, we can say cost(B, X) + cost(X, L) ≥ cost(B, L). To cal­cu­late a lower bound, we rewrite this in­equal­ity as cost(B, X) ≥ cost(B, L) - cost(X, L).

That’s the key idea here. It’s im­prac­ti­cal to pre­cal­cu­late all costs to all lo­ca­tions, but if we’ve pre­cal­cu­lated the costs to a spe­cific lo­ca­tion L, we can use that to es­ti­mate the cost to a dif­fer­ent lo­ca­tion .

Some of the aca­d­e­mic re­search pa­pers re­fer to this as a heuris­tic based on the tri­an­gle in­equal­ity. Other pa­pers call this the “differential heuris­tic” be­cause it takes the dif­fer­ence be­tween al­ready com­puted dis­tances.

4  Multiple land­marks#

How of­ten is this tri­an­gle in­equal­ity use­ful?

It de­pends on where L is rel­a­tive to the path B→:

Move the start B and goal around to see where a land­mark would help:

Try mov­ing the land­mark L out­side the green shaded re­gion, and see that the heuris­tic and path don’t al­ways match.

Since a land­mark needs to be “after” the goal , a sin­gle land­mark won’t be use­ful for all paths. We need mul­ti­ple land­marks L₁, L₂, L₃, etc. Each one gives us some lower bound for the heuris­tic:

cost(B, X) ≥ cost(B, L₁) - cost(X, L₁) cost(B, X) ≥ cost(B, L₂) - cost(X, L₂) cost(B, X) ≥ cost(B, L₃) - cost(X, L₃) … cost(B, X) ≥ cost(B, Lₙ) - cost(X, Lₙ)

We can take the max() of these to pick the high­est bound. In this di­a­gram, try mov­ing the goal to one of the pur­ple shaded ar­eas to see how those ar­eas are im­proved by the land­marks. Then try mov­ing it to one of the un­shaded ar­eas to see how A* is­n’t any faster there. Also try mov­ing the start point B to see how the shaded area also de­pends on where the start is.

5  Placement of land­marks#

The best land­mark po­si­tion de­pends on the start point B and goal . We want the land­mark to be “after” the goal , but what’s “after” de­pends on where the start point B and goal are.

We want to use land­marks to im­prove as many (start, goal) pairs as pos­si­ble.

Let’s start with a sin­gle land­mark. Try mov­ing the start B, goal , and land­mark L on this map:

The pur­ple shaded ar­eas show the goal po­si­tions that the land­mark helsp. It looks like the land­mark can cover the main cor­ri­dors but not the side rooms. We need many more land­marks:

Much more of the map is cov­ered in pur­ple now. Picking the num­ber and place­ment of land­marks is pro­ject spe­cific. Consider:

Are all paths equally likely? For ex­am­ple in a colony builder game like Dwarf Fortress, we may care a lot more about paths to/​from the main base, and not paths be­tween a for­est and a mine.

All all paths equally valu­able to op­ti­mize? For ex­am­ple if pathfind­ing is lim­it­ing the frame rate, we might want to fo­cus on long paths that are slower to com­pute and not on short paths.

Is the map sta­tic or does it change over time? If sta­tic, we might want to spend a lot of time in the map de­signer tool to pre­cal­cu­late op­ti­mal land­marks. But if dy­namic, we might want to use the last few goal lo­ca­tions to de­cide new land­mark po­si­tions.

If the change re­duces an edge cost, the heuris­tic will over­es­ti­mate some­times, and A* will re­turn a non-short­est path un­til we up­date the cost table. Pathfinding is op­ti­mized but nonop­ti­mal. Example: the player broke a wall but the unit won’t look for the shorter path right away. If the change in­creases an edge cost, the heuris­tic will be lower than de­sired, and A* will take a lit­tle longer to run un­til we up­date the cost table. Pathfinding is op­ti­mal but not op­ti­mized. Example: the player added a wall so the unit might think that area’s safe to walk through but will have to find a path around it.

If the change re­duces an edge cost, the heuris­tic will over­es­ti­mate some­times, and A* will re­turn a non-short­est path un­til we up­date the cost table. Pathfinding is op­ti­mized but nonop­ti­mal. Example: the player broke a wall but the unit won’t look for the shorter path right away.

If the change in­creases an edge cost, the heuris­tic will be lower than de­sired, and A* will take a lit­tle longer to run un­til we up­date the cost table. Pathfinding is op­ti­mal but not op­ti­mized. Example: the player added a wall so the unit might think that area’s safe to walk through but will have to find a path around it.

If many units find paths to com­mon ar­eas (such as the Dwarf Fortress din­ing room), con­sider drop­ping the least used land­mark and adding one near the com­mon area.

Are the maps open world or con­strained? A real time strat­egy game may have dif­fer­ent needs than a room+cor­ri­dor dun­geon crawler.

Thomas Nobes has a video ex­pla­na­tion[2] in­clud­ing more tips on where to place the land­mark points.

Fortunately, even if a land­mark is­n’t op­ti­mal, it might still help some­what, and it’s still no worse than if we use the reg­u­lar A* heuris­tic.

6  Automated place­ment#

Although the best land­mark po­si­tions will be pro­ject spe­cific, one al­go­rithm to place land­marks in a pro­ject-ag­nos­tic way is to keep track of which lo­ca­tions are good for many ran­domly cho­sen paths. Try it here to find a land­mark po­si­tion:

It usu­ally but not al­ways picks a spot in the up­per left. It matches our in­tu­ition that land­marks should go on the outer edges of the map.

The sec­ond land­mark should be away from the first land­mark. The third land­mark should be away from the first and sec­ond land­mark. Each sub­se­quent land­mark should be eval­u­ated based on what it adds. This is what it looks like with two ex­ist­ing land­marks:

It picks a third away from the first two, but not al­ways in the same place.

7  Implementation#

The change de­scribed on this page is to the heuris­tic func­tion given to A*. We don’t need to change A* it­self.

We need to pick land­marks. If the maps are known ahead of time, land­marks can be placed in a map de­signer tool. If the maps are pro­ce­du­rally gen­er­ated, try the ran­dom­ized map analy­sis ear­lier on this page. Some of the pa­pers linked at the end have more so­phis­ti­cated place­ment al­go­rithms.

Then we need to an­a­lyze the map. Allocate a 2D ar­ray of num­bers, cost[nodeId][land­markId].

For each land­mark, we run Dijkstra’s Algorithm. It’s a “single source short­est path” al­go­rithm but we want a sin­gle goal in­stead of a sin­gle source. In a di­rected graph, we need to re­verse all the edges. In an undi­rected graph, we can use the edges as is. We set cost[nodeId][land­markId] to the cost of the short­est path from node nodeId to node land­markId. If the weights are all 1, we can use Breadth First Search in­stead of Dijkstra’s Algorithm.

This is ap­prox­i­mately what I’m run­ning for the demos on this page (undirected graphs):

const L = [ /* ar­ray of land­mark lo­ca­tions */ ]; let L_cost = [ /* ar­ray[nodeId] of ar­rays[land­markId] */ ]; for (let land­markId = 0; land­markId < L.length; land­markId++) { let out­put = dijk­straSearch(L[land­markId]); for (let nodeId = 0; nodeId < graph.num_n­odes; nodeId++) { L_cost[nodeId][landmarkId] = out­put.cost_­so_­far[nodeId]; } }

Note that it’s not much code. It’s run­ning our ex­ist­ing al­go­rithm (Dijkstra’s, A*, or BFS) and stor­ing the re­sults in an ar­ray. It could run in a back­ground thread.

Then we need to mod­ify the heuris­tic func­tion. Previously the heuris­tic was dis­tance(B, X). For ex­am­ple:

func­tion heuris­tic­Man­hat­tan(a, z) { re­turn Math.abs(a.x - z.x) + Math.abs(a.y - z.y); }

Each land­mark Li gives us a lower bound cost(Lᵢ, X) - cost(Lᵢ, B). We want to take the high­est of these:

func­tion heuris­ti­cLand­mark(B, X) { let h = heuris­tic­Man­hat­tan(B, X); // or any base heuris­tic for (let i = 0; i < L.length; i++) { let lower­Bound = L_cost[B][i] - L_cost[X][i]; lower­Bound = Math.abs(lowerBound); // if undi­rected if (lowerBound > h) { h = lower­Bound; } } re­turn h; }

Note that it’s not much code. It’s run­ning the ex­ist­ing heuris­tic (typically Manhattan, Chebyshev, or Euclidean dis­tance) and some­times in­creas­ing it if the land­marks form a good tri­an­gle.

What changes with the A* code? Nothing.

There are lots of tech­niques for mak­ing A* run faster. I like this one be­cause it’s very lit­tle code.

8  Demos#

I tried the dif­fer­en­tial heuris­tic on some maps from Dragon Age (provided by movin­gai.com[3]), a maze (also pro­vided by movin­gai.com), and Cogmind[4] (maps pro­vided by Josh Ge). All of these maps are undi­rected graphs (edges are bidi­rec­tional) so I’ve used that ver­sion of the dif­fer­en­tial heuris­tic.

Blue ar­eas are what we no longer have to search by us­ing the dif­fer­en­tial heuris­tic. Orange ar­eas are what we search even with the im­proved heuris­tic.

Try mov­ing the start B and goal to see the per­for­mance on dif­fer­ent paths.

8.1 Dragon Age, The Circle Tower

The land­mark is badly placed for the ini­tial B→ path. Try mov­ing it.

8.2 Cogmind, Factory 5

In the next demo the land­marks L are in places that don’t help. Move them around to im­prove search.

The blue area are the nodes we no longer have to search. More blue is bet­ter.

The land­mark L points help more when closer to the start point B than the goal . They help more when they’re “past” the goal point. Move the start B and goal around and see that there’s a big im­prove­ment no mat­ter which path you want to find:

However, it took a lot of land­marks to get that im­prove­ment. We can do bet­ter by us­ing the ran­dom path map analy­sis to pick fewer land­marks but in smarter lo­ca­tions:

8.3 Maze

A* with a dis­tance heuris­tic be­haves par­tic­u­larly badly with mazes, but in this one, just four land­marks make a big dif­fer­ence! Then try click­ing Random path re­peat­edly. The blue ar­eas are the ar­eas we did­n’t have to search by us­ing the land­marks. Also tog­gle the bidi­rec­tional flag to see how much of a dif­fer­ence that makes.

8.4 Dragon Age, Lothering

This map has large open ar­eas, and it seems to work well with the land­marks.

8.5 Cogmind, Research 2

This is a room-and-cor­ri­dor map from Cogmind.

Making your own dithered QR codes

QR codes: the ba­sics

A QR code is re­ally just a way of en­cod­ing a few bytes of data in a way that can be eas­ily read froma photo by a smart­phone or sim­i­lar de­vice. They’re de­signed to be very ro­bust to bad fo­cus, bad print­ing, funny an­gles, and miss­ing ar­eas. Exactly how they work is­n’t ter­ri­bly im­por­tant, but for the pur­poses of this page, a QR code is a grid of squares that di­vide into two parts — the func­tion pat­terns (which are the bold shapes in the above ex­am­ple) and the data mod­ules (everything else). The func­tion mod­ules are mostly used to let the scan­ner find the QR code eas­ily, so they have to be very clear and dis­tinct. The data mod­ules are where the ac­tual data is stored, along with some ex­tra header data etc, and since those are only read af­ter the scan­ner has used the func­tion mod­ules to work out ex­actly where the QR code is, they can be mod­i­fied a bit more. This is of­ten used by brands to cre­ate QR codes that look a bit dis­tinc­tive, and while this kind of mod­i­fi­ca­tion makes them a bit less ro­bust when scan­ning, but you can gen­er­ally get away with a fair bit be­fore the codes be­come un­scannable.

Putting pic­tures in QR codes

This is a QR code I saw on Mastodon. One of the mod­i­fi­ca­tions you can do with the data mod­ules is to shrink them — the scan­ner will look where the finder pat­terns say the cen­tres of the pix­els should be, and so as long as the code is­n’t dis­torted when you try to scan it. That means the rest of the space is yours to play with. Dave di­vides each pixel into a three-by-three grid, and uses the mid­dle one to store the data, and the oth­ers for a photo. The re­sult is a low-res, one-bit photo with some salt-and-pep­per noise on it.

Dithering

Obviously when we crush an im­age into a low colour depth, we don’t nor­mally just thresh­old it — that is, make the dark pix­els black and the light pix­els white. We nor­mally use a kind of che­quer­board ef­fect to cre­ate mid­tones as well. A Bayer fil­ter can be used to ap­ply this idea across a whole im­age with­out hav­ing to sac­ri­fice too much fine de­tail.

Error dif­fu­sion

An at­tempt to im­prove on this method was Floyd-Steinberg dither­ing.

In this method, you start in the top left and thresh­old the pixel nor­mally — if it’s more than 50% bright­ness, you make it white, and oth­er­wise you make it black. Say it was 70% bright­ness — we’re go­ing to make it white, which is 100% bright­ness, which means we’ve added 30% of a pixel too much bright­ness. To coun­ter­act that, we’re go­ing to “diffuse” that er­ror to other pix­els — all the nearby pix­els that we haven’t thresh­olded yet are made a few percend darker, to a to­tal of 30% of a pixel. When we come to thresh­old those pix­els, we’ll take that into ac­count. The idea is that once we’ve thresh­olded the en­tire pic­ture, every part of the im­age will be, on av­er­age, closer to the cor­rect bright­ness than more reg­u­lar dither­ing. The ir­reg­u­lar pat­terns are also a bit less dis­tract­ing.

QR codes with er­ror-dif­fu­sion dither­ing

The other ad­van­tage of this ir­reg­u­lar dither­ing is that the salt-and-pep­per noise caused by the QR code data mod­ules is much harder to spot. But it’s still there, and it’s why the im­age looks so noisy. I mean, it would look fairly noisy any­way be­cause it’s a 147×147 pixel one-bit im­age, but some of that noise is be­cause one in nine of the pix­els are ef­fec­tively ran­dom colours.

But we can solve that us­ing more er­ror dif­fu­sion.

Error-diffusing the data mod­ules

In this case, we’re do­ing two passes on the er­ror dif­fu­sion. The sec­ond is the same as nor­mal, but the first is specif­i­cally to mask the data mod­ules. We al­ready know what colour all of those pix­els have to be, so we can make them those colours and dif­fuse that er­ror to the sur­round­ing eight pix­els. The er­ror will po­ten­tially be quite big — nor­mally we get to choose the colour so the er­ror can never be more than 50%, but here we might have to make a pixel in a dark area pure white and end up cre­at­ing 95% er­ror. That sounds bad, but that’s ex­actly why it’s so im­por­tant to dif­fuse this er­ror in­stead of just ac­cept­ing it.

You can see that this makes the im­age much cleaner.

Getting fancy with it

The gen­er­a­tor tool al­lows you to do some ex­tra tricks. You can ro­tate the QR code be­fore adding the im­age, and tin­ker with the QR code set­tings to see if any of the al­ter­na­tive en­cod­ings hap­pen to look nicer.

You can also al­low the gen­er­a­tor to change a few of those 95%-error data mod­ules — QR codes have enough er­ror cor­rec­tion in them that you can af­ford to change a few pix­els and they’ll still scan. (That’s how QR codes with lo­gos in the mid­dle work.) But in prac­tice that does­n’t re­ally af­fect im­age qual­ity much if you dif­fuse the er­ror from the data mod­ules and it se­ri­ously af­fects how well the code scans.

Using these in real life

This all works pretty well if you don’t take it too far and the code is go­ing to be on a big screen or a poster or some­thing. If it’s on, say, a pa­per flyer that could get crum­pled then you’re go­ing to need more of that re­dun­dancy, ro­bust­ness and er­ror cor­rec­tion that we gave up to make the QR code pretty. Ultimately it’s a trade-off be­tween aes­thet­ics and scannabil­ity — and re­mem­ber that just be­cause a code scans on your phone, from a lap­top screen, does­n’t mean it will scan on a ran­dom stranger’s potato phone from a print­out in bad light­ing.

Also bear in mind that the gen­er­a­tor tool makes tiny im­ages with no mar­gin — some mar­gin is needed to make a QR code scan re­li­ably, and browsers will nor­mally blur im­ages when up­scal­ing them if you don’t dis­able that with CSS. The mar­gin has to be the op­po­site colour to the mid­dle of the big three “finder” pat­tern squares — nor­mally that’s white but you can gen­er­ate in­veted QR codes that need a black back­ground in­stead.

During check­out, when a buyer clicks “Complete pur­chase,” we need to guar­an­tee the items they’re buy­ing are still avail­able. If we get this wrong in one di­rec­tion, two buy­ers pur­chase the same last unit: the mer­chant has to can­cel an or­der, send an apol­ogy email, and eat the sup­port cost. If we get it wrong in the other di­rec­tion, we tell a buyer some­thing is sold out when it is­n’t, and the mer­chant loses a sale they should have made.

At Shopify’s scale, ei­ther fail­ure com­pounds fast. On Black Friday 2025, mer­chants on our plat­form hit a record $5.1 mil­lion in sales per minute at peak. Every one of those trans­ac­tions touches in­ven­tory.

Our over­sell pro­tec­tion sys­tem han­dles this by re­serv­ing in­ven­tory dur­ing pay­ment pro­cess­ing—a short hold that pre­vents two con­cur­rent check­outs from claim­ing the same unit. For years, this ran on Redis. When we moved to­ward a uni­fied data­base strat­egy, we had to an­swer a hard ques­tion: could MySQL han­dle the same scale?

Earlier at­tempts had failed. A sin­gle row with a quan­tity col­umn could­n’t han­dle the con­tention. MySQL 8′s SKIP LOCKED fea­ture in­tro­duced a dif­fer­ent de­sign: one row per in­ven­tory unit in­stead of one row per item. Inspired by 37signals’ ap­proach to data­base-backed load dis­tri­b­u­tion, we re­built reser­va­tions on MySQL and hit our high-through­put tar­gets dur­ing peak 2025 traf­fic.

But the hard­est les­son was­n’t about data­base de­sign. It was dis­cov­er­ing that the real bot­tle­neck was­n’t what we were ob­serv­ing and mea­sur­ing. This post walks through the so­lu­tion and what we found along the way.

The chal­lenge

What is over­sell pro­tec­tion?

Oversell pro­tec­tion has two main op­er­a­tions:

Reserve: When pay­ment starts, we mark items as re­served (a short hold, e.g. sev­eral min­utes).

Claim: When pay­ment suc­ceeds, we per­ma­nently deduct quan­tity from the in­ven­tory ledger (source of truth).

Checkout com­ple­tion de­pends on this be­ing fast and cor­rect. Slow reser­va­tions trig­ger throt­tling and a worse buyer ex­pe­ri­ence. Mistakes mean over­selling (angry cus­tomers) or un­der­selling (lost rev­enue).

Scale and cor­rect­ness re­quire­ments

Scale here is not ab­stract: Shopify pow­ers over 14% of U.S. ecom­merce, and on Black Friday 2025 we saw an 11% in­crease in sales per minute at peak over the prior year. Reservations run on every check­out that touches in­ven­tory, so the sys­tem must han­dle that burst with­out drop­ping re­quests or break­ing con­sis­tency.

We needed to:

Support the plat­for­m’s high-per­for­mance through­put tar­gets dur­ing peak traf­fic

Respect multi-lo­ca­tion in­ven­tory (only re­serve from lo­ca­tions that can ful­fill)

Keep ACID guar­an­tees be­tween reser­va­tions and the in­ven­tory ledger

Prioritize cor­rect­ness: no over­selling and no lost reser­va­tions

The Redis model and its lim­its

The pre­vi­ous sys­tem stored reser­va­tions in Redis. Each item had a quan­tity key, and re­serv­ing meant DECR, re­leas­ing meant INCR. Redis han­dled con­cur­rency fine, but reser­va­tions and the in­ven­tory ledger lived in two dif­fer­ent sys­tems.

The claim step (payment processed, per­ma­nently deduct in­ven­tory) re­quired up­dat­ing MySQL and clean­ing up Redis, and those two op­er­a­tions could­n’t be wrapped in a sin­gle atomic step. Depending on the or­der, this could cause over­selling (item sold but was never de­ducted from the ledger) or un­der­selling (item de­ducted and still marked re­served).

On top of that, the Redis model had no multi-lo­ca­tion aware­ness and added the op­er­a­tional cost of a sep­a­rate clus­ter to main­tain. Moving reser­va­tions into the same MySQL data­base as the ledger meant we could wrap every­thing in ACID trans­ac­tions and elim­i­nate these fail­ure modes en­tirely.

The so­lu­tion: SKIP LOCKED

Core idea: one row per unit, bounded by de­sign

Instead of one row per item with a quan­tity col­umn, we use one row per sell­able unit. An item with 10 units has 10 rows. Reserving three units means se­lect­ing and mov­ing three rows in a sin­gle trans­ac­tion. By keep­ing reser­va­tions and the in­ven­tory ledger in the same data­base, we get ACID across re­serve and claim—fix­ing classes of bugs that were pos­si­ble with Redis (e.g. pay­ment suc­ceeds but in­ven­tory is­n’t claimed, or the re­verse).

A sim­pli­fied re­serve flow looks like this:

SKIP LOCKED is what makes this scal­able: if an­other trans­ac­tion has locked some rows, MySQL skips them and re­turns other avail­able rows. No wait­ing on the same row, less con­tention.

But one row per unit for all in­ven­tory would break down at scale—an item with 50,000 units across 10 lo­ca­tions would mean 500,000 rows, and the re­serve query would slow as it scans through them. Instead, we main­tain a bounded pool of avail­able rows, capped at 1,000 per item/​lo­ca­tion com­bi­na­tion. Reservations con­sume rows from this pool; a re­plen­ish­ment process re­fills it from the in­ven­tory ledger.

Why 1,000? The cap needs to be large enough to ab­sorb bursts with­out run­ning dry, but small enough to keep the table com­pact and the SKIP LOCKED scan fast. We sized it based on ob­served peak reser­va­tion rates per item/​lo­ca­tion dur­ing flash sales: 1,000 gives us enough head­room that re­plen­ish­ment can keep up un­der sus­tained load with­out the table grow­ing to a point where query per­for­mance de­grades.

What hap­pens if the pool emp­ties? During an ex­treme flash sale, the pool for a hot item can be tem­porar­ily ex­hausted. When that hap­pens, the re­serve path trig­gers re­plen­ish­ment in­line. A lock en­sures only one trans­ac­tion re­plen­ishes at a time; other con­cur­rent re­serves for the same item wait for it to fin­ish rather than all rac­ing to in­sert rows, avoid­ing a thun­der­ing herd. Once re­plen­ish­ment com­pletes, the wait­ing trans­ac­tions pro­ceed with a full pool. The buyer never sees the item as un­avail­able (unless it truly is). This adds la­tency to that spe­cific reser­va­tion, but it pre­serves cor­rect­ness: a buyer with avail­able in­ven­tory is never turned away.

Key tech­ni­cal de­ci­sions

1. Composite pri­mary key: fewer locks per row

Our first pro­to­type used an auto-in­cre­ment ID as the pri­mary key. When we ob­served lock be­hav­ior (e.g. with SHOW ENGINE INNODB STATUS), we saw two row locks per reser­va­tion in­stead of one.

With an auto-in­cre­ment pri­mary key, InnoDB was lock­ing both the sec­ondary in­dex used in the WHERE clause and the clus­tered in­dex (primary key). We switched to a com­pos­ite pri­mary key (shop_id, in­ven­to­ry_item_id, in­ven­to­ry_­group_id, id) so the columns we fil­ter on are part of the pri­mary key. That re­duced to one lock per row, which was im­por­tant when run­ning many reser­va­tions per sec­ond.

Takeaway: at this scale, in­dex and pri­mary key de­sign di­rectly af­fect lock count and through­put.

2. READ COMMITTED: avoid­ing gap (supremum) locks

When we ran SELECT … FOR UPDATE SKIP LOCKED on an empty table that needed re­plen­ish­ment, we saw gap locks (including on the “supremum” pseudo-record). Those locks blocked the re­plen­ish­ment trans­ac­tion from in­sert­ing new rows and could lead to dead­locks.

We changed the trans­ac­tion iso­la­tion level from REPEATABLE READ (MySQL de­fault) to READ COMMITTED for these trans­ac­tions. Under READ COMMITTED, InnoDB does­n’t take gap locks in the same way, so re­plen­ish­ment could pro­ceed. Jahfer Husain’s guide to InnoDB lock­ing was very help­ful for un­der­stand­ing this. This was our first use of a non-de­fault iso­la­tion level in this code­base; it re­quired small frame­work sup­port for set­ting iso­la­tion per trans­ac­tion.

3. Consistent lock or­der­ing: avoid­ing dead­locks

We hit dead­locks when re­serve and claim touched two ta­bles in dif­fer­ent or­ders. Reserve was do­ing INSERT into re­served_quan­ti­ties then DELETE from reser­va­tion_u­nits; claim was do­ing DELETE from re­served_quan­ti­ties. Different trans­ac­tions could lock the two ta­bles in dif­fer­ent or­ders and form a cy­cle.

The fix was to stan­dard­ize the or­der: re­serve al­ways DELETEs from the units table first, then INSERTs into re­served_quan­ti­ties. Claim only touches re­served_quan­ti­ties. Because both paths now ac­quire locks in the same or­der, nei­ther can hold a lock the other is wait­ing for—no more cir­cu­lar waits.

4. Batching with UNION ALL

Each round trip to the data­base has a cost. For carts with mul­ti­ple line items, we batch reser­va­tion queries us­ing UNION ALL so we fetch all needed units in one round trip:

That cut to­tal round trips and helped la­tency un­der load.

The real bot­tle­neck: con­nec­tions, not CPU

In pro­duc­tion we hit a through­put ceil­ing well be­low our tar­get. Reservation la­tency (e.g. P90) was ac­cept­able, CPU was­n’t maxed out, and the queries were al­ready op­ti­mized. So we looked else­where.

We tried batch­ing reser­va­tions across mul­ti­ple check­outs in a sin­gle SKIP LOCKED query to use fewer con­nec­tions. That helped in load tests but added com­plex­ity. We also moved some read load to repli­cas. Still, some­thing did­n’t add up.

Following the symp­toms

During load tests we saw:

Threads queu­ing in MySQL

CPU spik­ing when queued work ran

Connection ex­haus­tion to MySQL back­ends on ProxySQL layer

So we added vis­i­bil­ity into which busi­ness processes were hold­ing data­base con­nec­tions and for how long. Knowing that con­nec­tions are ex­hausted does­n’t tell you who is hold­ing them. We needed per-caller at­tri­bu­tion.

On the ap­pli­ca­tion side, we an­no­tated every SQL state­ment with a com­ment tag iden­ti­fy­ing the busi­ness process, like /* con­n_­tag:check­out_­com­ple­tion */. On the ProxySQL layer, we added track­ing that parses the tag and mea­sures how long each caller holds a con­nec­tion. The re­sult: to­tal con­nec­tion hold time, bro­ken down by busi­ness process.

This im­me­di­ately showed which callers were con­sum­ing the most con­nec­tion time. Not which queries were slow, but which processes were hold­ing con­nec­tions across long trans­ac­tions. If you’re hit­ting con­nec­tion lim­its and can’t tell why, this pat­tern (tag at the app layer, ag­gre­gate at the proxy) is straight­for­ward to im­ple­ment and im­me­di­ately ac­tion­able.

What we found

Once we could see con­nec­tion us­age, we learned that reser­va­tions weren’t the only heavy user. Other parts of the check­out path were hold­ing con­nec­tions longer than nec­es­sary. They had­n’t been op­ti­mized be­cause they had­n’t been the first to hit the limit. Connections are fi­nite: at high through­put we need many short trans­ac­tions per sec­ond. When other code held con­nec­tions longer, reser­va­tions were the straw that broke the camel’s back—not be­cause reser­va­tions were slow, but be­cause the pool was al­ready near de­ple­tion.

The cleanup of the check­out path re­moved 50% of reads and 33% of trans­ac­tions on the pri­mary data­base. We also re­vis­ited MySQL con­fig­u­ra­tion. InnoDB thread con­cur­rency had been set con­ser­v­a­tively years ear­lier and never re-eval­u­ated. Our work­load had changed. After in­creas­ing thread con­cur­rency where we had head­room, we re­moved a bot­tle­neck we had­n’t seen un­til we had con­nec­tion and CPU met­rics side by side.

Combined, the cleanup and con­fig­u­ra­tion changes re­moved the ceil­ing. We could scale be­yond our pre­vi­ous limit and meet our tar­gets. During high-vol­ume flash sales, writer CPU stayed un­der 50% and reader CPU un­der 16%, with head­room to spare.

The cu­tover

We did­n’t flip a switch from Redis to MySQL. We ran both sys­tems in par­al­lel in what we called “shadow mode”: every reser­va­tion was writ­ten to both Redis and MySQL, with Redis re­main­ing the source of truth. This let us com­pare the two sys­tems side by side, val­i­dat­ing that MySQL pro­duced the cor­rect busi­ness out­comes and met our per­for­mance re­quire­ments on real pro­duc­tion traf­fic. Because both sys­tems were live, there were no in-flight reser­va­tions to mi­grate. Redis reser­va­tions con­tin­ued to be hon­ored while MySQL built up its own state.

Once we were sat­is­fied with cor­rect­ness and per­for­mance, we switched the source of truth to MySQL. If any­thing went wrong, we could re­vert to Redis with a kill switch; the dual-write path was still ac­tive, so Redis had a com­plete view of reser­va­tions at all times. The roll­out was grad­ual, pod by pod, start­ing with low-traf­fic pods and work­ing up to our high­est-vol­ume mer­chants.

What we learned

We learned so much from this pro­ject but the two main take­aways:

1. Revisit old de­ci­sions

What was­n’t pos­si­ble five years ago (e.g. MySQL for this work­load) can be pos­si­ble to­day with new fea­tures like SKIP LOCKED. The same goes for con­fig­u­ra­tion: thread lim­its and other “rule of thumb” set­tings are worth re-check­ing when work­load and hard­ware evolve. If the num­bers don’t add up (e.g. low CPU but queu­ing), dig in.

2. Start small and ob­serve

We got a lot of value from a min­i­mal pro­to­type: a small ruby script and MySQL, no full frame­work like Rails. Observing the data­base (e.g. lock be­hav­ior in a sec­ond ter­mi­nal) taught us more than the­ory alone. Simple tool­ing and a tight feed­back loop beat big, opaque sys­tems when ex­plor­ing.

MySQL can now han­dle work­loads we used to as­sume re­quired spe­cial­ized in­fra­struc­ture. If you’re reach­ing for Redis, Kafka, or a cus­tom co­or­di­na­tion layer for high-through­put mu­tual ex­clu­sion, your ex­ist­ing data­base might al­ready be enough.

The bot­tle­neck was­n’t where we ex­pected. We op­ti­mized queries and locks for weeks; the real limit was con­nec­tion us­age in code we weren’t even look­ing at. If the num­bers don’t add up—low CPU but high queu­ing—in­stru­ment the full path. The an­swer is of­ten in the plumb­ing, not the en­gine.

Crucially, this was­n’t about mak­ing reser­va­tions fast. It was about mak­ing them safe neigh­bors. Reservations share a data­base with cart up­dates, pay­ment pro­cess­ing, and or­der cre­ation. A sys­tem that sat­u­rates con­nec­tions or holds locks too long jeop­ar­dizes all of them. The real bar was sus­tain­ing through­put with­out de­grad­ing data­base health for every­thing else.

For us, the pay­off is con­crete: more re­li­able reser­va­tions mean no over­sells and more suc­cess­ful pur­chases from our mer­chants.

Yesterday I pub­lished an ar­ti­cle ti­tled “App Store Rejection of the Week: Dark Hours”. I have re­tracted it. Its premise was so fun­da­men­tally wrong that there’s no point merely cor­rect­ing or edit­ing it. Even the ti­tle, as I ex­plain be­low, was in­ac­cu­rate. Although the orig­i­nal is now re­tracted, I’m not mem­ory-hol­ing it. The text of the orig­i­nal story is avail­able, for trans­parency and ac­count­abil­ity, in plain text (Markdown, natch) and PDF (preserving orig­i­nal pre­sen­ta­tion). Both of those ver­sions in­clude a pref­ace at the top link­ing to this re­trac­tion. The URL for the orig­i­nal story now redi­rects to this one that you are cur­rently read­ing.

To the best of my rec­ol­lec­tion, this is the first post I’ve re­tracted in the 24 years I’ve been writ­ing Daring Fireball. I hope it’s the last. I was mis­led, both overtly and through omis­sions, in sev­eral ways, but what I pub­lish is my re­spon­si­bil­ity, and I apol­o­gize for the er­ror.

Terry Godier first came to my at­ten­tion in February, when I linked to his ex­cel­lent in­ter­ac­tive es­say on RSS feed reader de­sign, “Phantom Obligation”, which es­say in­tro­duced Current, Godier’s new RSS reader that he made to ex­em­plify the ideas from his es­say. Current is, de­servedly, a bit of a break­out hit. (E.g., David Pierce, at The Verge, put it on a very short list of iOS-ex­clu­sive in­die apps that keep him from switch­ing to Android.) In March I linked to an­other in­ter­ac­tive es­say from Godier, “The Last Quiet Thing”, and again in April to a post re­gard­ing the App Store’s 5-star re­view sys­tem. I struck up an iMes­sage cor­re­spon­dence with Godier around when I first linked to his work.

Yesterday Godier posted “Browsers Have Standards, the App Store Has Judgment”. As orig­i­nally pub­lished, that post con­tained these two para­graphs:

A while ago I tried to sub­mit an iOS app for Dark Hours, my as­tron­omy web­site for nor­mal peo­ple. It was re­jected on the grounds that it was as­trol­ogy.

It has no tarot func­tion, no horo­scopes, and noth­ing that I, or any­one else I’ve asked, would as­so­ci­ate with as­trol­ogy.

A while ago I tried to sub­mit an iOS app for Dark Hours, my as­tron­omy web­site for nor­mal peo­ple. It was re­jected on the grounds that it was as­trol­ogy.

It has no tarot func­tion, no horo­scopes, and noth­ing that I, or any­one else I’ve asked, would as­so­ci­ate with as­trol­ogy.

Those para­graphs, at this writ­ing (August 8, 11:00 pm ET), have been deleted and re­placed by this:

Note: the orig­i­nal ver­sion of this post had a sec­tion here about an as­tron­omy app I am work­ing on that be­gan as an as­trol­ogy app and was re­jected af­ter hav­ing the as­trol­ogy con­tent re­moved. App store [sic] re­view reached out to me and let me know that they had ap­par­ently never been given the up­dated build and that the app should be fine to sub­mit now.

Note: the orig­i­nal ver­sion of this post had a sec­tion here about an as­tron­omy app I am work­ing on that be­gan as an as­trol­ogy app and was re­jected af­ter hav­ing the as­trol­ogy con­tent re­moved. App store [sic] re­view reached out to me and let me know that they had ap­par­ently never been given the up­dated build and that the app should be fine to sub­mit now.

You can now see the prob­lem that has led me to fully re­tract my orig­i­nal post, given that my post was en­tirely pred­i­cated on the premise that Godier’s orig­i­nal de­scrip­tion of the re­jected app was true — that the app “has no tarot func­tion, no horo­scopes, and noth­ing that I, or any­one else I’ve asked, would as­so­ci­ate with as­trol­ogy.” The truth is, the app, as orig­i­nally sub­mit­ted by Godier to the App Store (under the name “Asterly”, not “Dark Hours”), was en­tirely ded­i­cated to as­trol­ogy, not as­tron­omy, and did in fact in­clude a “Tarot card of the day” fea­ture amongst other oc­cultist horse­shit.

The grounds of Apple’s orig­i­nal App Store re­jec­tion of the app, and the re­jec­tion’s up­hold­ing by the App Review Board, were cor­rect.1 I wrongly took Godier at his word, both in his pub­lic blog post and in pri­vate iMes­sage cor­re­spon­dence yes­ter­day, that the re­jec­tion was­n’t just merely de­bat­able, but com­pletely and rather pre­pos­ter­ously un­grounded. Whether Godier ever sub­mit­ted a build of “Asterly” to the App Store that con­tained no oc­cult horse­shit and only the hard-sci­ence as­tron­omy fea­tures that were pre­sent in his “Dark Hours” web­site that was avail­able for the last week, I don’t know. But I have no rea­son to be­lieve that he did.

My dis­dain for as­trol­ogy is so ut­ter, and my es­teem for Godier’s pre­vi­ous work so high, that it sim­ply never oc­curred to me that he might have ac­tu­ally made and sub­mit­ted to the App Store an as­trol­ogy app, let alone that he’d then feign sur­prise and frus­tra­tion that an as­trol­ogy app was re­jected for be­ing an as­trol­ogy app. I showed him a draft of my post be­fore pub­li­ca­tion, to make sure I had the story straight, and he of­fered not a word of cau­tion, only grat­i­tude for my draw­ing at­ten­tion to the mat­ter.

It gets messier. Godier’s as­trol­ogy app that he sub­mit­ted to the App Store back in January was named “Asterly”. That was still the name when the App Review Board up­held its re­jec­tion in April. According to Godier, frus­trated by the App Store’s re­jec­tion, he ported the as­tron­omy ver­sion to the web, launch­ing it last week un­der the name “Dark Hours” at the do­main dark­hours.io. (This is why it was in­cor­rect for me, in the very ti­tle of my post, to claim that Apple had re­jected an app named “Dark Hours”. They re­jected an app named “Asterly” and had never seen an app from Godier named “Dark Hours”.) Yesterday, af­ter I linked to Godier’s post and Hacker News then linked to my post, Godier’s “Dark Hours” (with a space) came to the at­ten­tion (and jus­ti­fi­able sur­prise) of Miguel Beher, cre­ator of an open-source “astrophotography and dark-sky plan­ner” pro­ject named DarkHours (no space). Beher’s GitHub pro­ject con­tains the source code, and the ac­tual web app is freely avail­able at the do­main dark­hours.app. In an un­com­fort­able ex­change be­tween Beher and Godier on Bluesky, Beher pointed out that Godier’s Dark Hours had the same bug as Beher’s that routed peo­ple to “random fields in Mexico”. Earlier to­day, Godier took his web app down and redi­rected his dark­hours.io do­main to Beher’s dark­hours.app.

At the end of my now-re­tracted post yes­ter­day, as­sum­ing I had right­eously and rightly skew­ered Apple for an egre­giously er­ro­neous App Store re­view re­jec­tion, I wrote:

Mistakes hap­pen. But in a func­tion­ing sys­tem mis­takes get cor­rected, and mis­takes as ob­vi­ous as this one get cor­rected al­most in­stantly and in­clude a quick apol­ogy for the con­fla­tion.

Mistakes hap­pen. But in a func­tion­ing sys­tem mis­takes get cor­rected, and mis­takes as ob­vi­ous as this one get cor­rected al­most in­stantly and in­clude a quick apol­ogy for the con­fla­tion.

Obviously it was I who was mis­taken. This ar­ti­cle is my cor­rec­tion, and I apol­o­gize to all who read and be­lieved my now-re­tracted post, and to the re­view­ers at the App Store whose com­pe­tence (if not lit­er­acy) I be­smirched. Sorry about that. Won’t hap­pen again.

Regardless of one’s opin­ion re­gard­ing pseu­do­science and oc­cultist horse­shit — pro, con, or in­dif­fer­ent — one might rea­son­ably think it wrong for Apple to dis­al­low or dis­cour­age such apps from the App Store. In fact, there ex­ist plenty of such apps in the App Store, and Apple has even run “Best Astrology Apps” ed­i­to­r­ial fea­tures. Apple’s stance is ba­si­cally that the App Store has enough of these apps (and I sus­pect they’re a com­mon source of scams, given that by their very na­ture they tar­get the gullible). Guideline 4.3(b) states (emphasis added):

Certain kinds of apps, such as dat­ing, flash­light, sound ef­fects, wall­pa­per, sim­ple timers, and for­tune telling, are well es­tab­lished on the App Store and we will not ac­cept new sub­mis­sions un­less they of­fer a mean­ing­fully dif­fer­ent or im­proved ex­pe­ri­ence. We may re­move these apps from the App Store go­ing for­ward if they are not up­dated, im­proved, or do not at­tract cus­tomers. Other kinds of apps, such as drink­ing games, Kama Sutra, fart, and burp apps, are mediocre, low-qual­ity, or low-ef­fort and do not add value to the App Store.

That se­r­ial comma, ever use­ful, gives hope to any­one hard at work on a “fart and burp” app. ↩︎

Regardless of one’s opin­ion re­gard­ing pseu­do­science and oc­cultist horse­shit — pro, con, or in­dif­fer­ent — one might rea­son­ably think it wrong for Apple to dis­al­low or dis­cour­age such apps from the App Store. In fact, there ex­ist plenty of such apps in the App Store, and Apple has even run “Best Astrology Apps” ed­i­to­r­ial fea­tures. Apple’s stance is ba­si­cally that the App Store has enough of these apps (and I sus­pect they’re a com­mon source of scams, given that by their very na­ture they tar­get the gullible). Guideline 4.3(b) states (emphasis added):

Certain kinds of apps, such as dat­ing, flash­light, sound ef­fects, wall­pa­per, sim­ple timers, and for­tune telling, are well es­tab­lished on the App Store and we will not ac­cept new sub­mis­sions un­less they of­fer a mean­ing­fully dif­fer­ent or im­proved ex­pe­ri­ence. We may re­move these apps from the App Store go­ing for­ward if they are not up­dated, im­proved, or do not at­tract cus­tomers. Other kinds of apps, such as drink­ing games, Kama Sutra, fart, and burp apps, are mediocre, low-qual­ity, or low-ef­fort and do not add value to the App Store.

Certain kinds of apps, such as dat­ing, flash­light, sound ef­fects, wall­pa­per, sim­ple timers, and for­tune telling, are well es­tab­lished on the App Store and we will not ac­cept new sub­mis­sions un­less they of­fer a mean­ing­fully dif­fer­ent or im­proved ex­pe­ri­ence. We may re­move these apps from the App Store go­ing for­ward if they are not up­dated, im­proved, or do not at­tract cus­tomers. Other kinds of apps, such as drink­ing games, Kama Sutra, fart, and burp apps, are mediocre, low-qual­ity, or low-ef­fort and do not add value to the App Store.

That se­r­ial comma, ever use­ful, gives hope to any­one hard at work on a “fart and burp” app. ↩︎

Amazon is qui­etly try­ing to build the biggest gas power plant in the coun­try.

The Distilled newslet­ter re­ported Friday that the mega­cor­po­ra­tion has bought land and ac­quired per­mits in Pecos County, Texas, for an AI data cen­ter pow­ered by a 7.65 gi­gawatt gas power plant. The plant will be com­pletely sep­a­rate from Texas’s power grid, at least in the be­gin­ning, the per­mits show.

The site, known as GW Ranch, got a state per­mit al­low­ing the pro­posed power plant to emit 33 mil­lion tons of car­bon diox­ide, which would make it the biggest pol­lu­tion site in the U.S., emit­ting more than the coun­try’s biggest coal power plant, ac­cord­ing to Distilled. That’s in sharp con­trast to Amazon’s com­mit­ment to reach net-zero emis­sions by 2040 as part of The Climate Pledge.

Amazon filed three con­struc­tion per­mits with the state of Texas this week to build three data cen­ter build­ings im­me­di­ately upon ap­proval. Land clear­ing has al­ready be­gun, ac­cord­ing to satel­lite im­agery. Amazon would join Microsoft, Google, and Meta in hav­ing its own off-grid gas power.

Amazon says that it has 10 gi­gawatts of car­bon-free en­ergy across 40 pro­jects to power its ex­ist­ing data cen­ter op­er­a­tions in Texas, and that the GW Ranch will use brack­ish ground­wa­ter that is­n’t potable and thus can’t be used for ir­ri­ga­tion or drink­ing.

But that’s not likely to quell pub­lic op­po­si­tion. Data cen­ters are hugely un­pop­u­lar across the coun­try among Republicans and Democrats, both in rural and sub­ur­ban ar­eas. The cen­ters don’t cre­ate many jobs or boost lo­cal economies. If con­nected to lo­cal power grids, they can drive util­ity rates up and cause black- and brownouts. Amid cli­mate change and droughts, the claim that data cen­ters will take ad­van­tage of un­us­able wa­ter will likely in­vite skep­ti­cism.

The fact that Amazon is build­ing its own power plant for the pro­ject will be a small com­fort for res­i­dents wor­ried about util­ity rates, but lo­cals still will have con­cerns about pol­lu­tion from a gas plant big­ger than any other in the U.S. Rural Texas is deeply Republican, and de­spite President Donald Trump’s delu­sions about data cen­ters’ pop­u­lar­ity, op­po­si­tion to them on the right is grow­ing. Now that this pro­ject is pub­lic, a big back­lash could soon fol­low.

Read more about data cen­ters:

Representative Max Miller blamed a day­care af­ter his daugh­ter was rushed to the hos­pi­tal last year with bruises.

Miller sued a Westlake, Ohio, day­care provider in February 2025, af­ter the daugh­ter he shares with his ex-wife Emily Moreno (the daugh­ter of Ohio Senator Bernie Moreno) was taken to an emer­gency room with in­juries “on her thighs, face, groin, and vagi­nal area of her body and she was caused to in­cur psy­cho­log­i­cal trauma.” Her mother took the girl, then un­der two years old, to a hos­pi­tal on January 27 af­ter a day­care staff mem­ber alerted po­lice as well as the Department of Children and Family Services.

The law­suit does­n’t clearly state where her in­juries came from, and Miller has a long list of abuse al­le­ga­tions against him re­gard­ing his ex-wife and daugh­ter, even al­legedly frac­tur­ing his daugh­ter’s col­lar­bone. His law­suit, filed four months be­fore his di­vorce was fi­nal­ized, ac­cuses the school and four named em­ploy­ees of “failing to prop­erly train and su­per­vise their staff in the proper treat­ment of chil­dren in their care. Defendants abused and ne­glected a child de­pen­dent upon them for pro­tec­tion.”

Miller is the only per­son named in the com­plaint, and Moreno is­n’t men­tioned. Miller claims in the law­suit that an in­ves­ti­ga­tion from county au­thor­i­ties backs up his claims, along with video and au­dio ev­i­dence doc­u­ment­ing al­leged abuse and ne­glect. The school de­nied the al­le­ga­tions, stat­ing that the tod­dler was not “injured, bru­tal­ized, or harmed in any man­ner while un­der the de­fen­dants’ al­leged care and su­per­vi­sion.”

The case was dis­missed four months ago, and if there was a set­tle­ment, it has­n’t been dis­closed. In light of newly re­vealed al­le­ga­tions that Miller has a habit of hold­ing his daugh­ter’s beloved blue bunny hostage and his ad­mis­sion that he shared an in­ap­pro­pri­ate photo of her on­line, this law­suit raises even more ques­tions.

Read more about Miller:

Women ac­counted for 100 per­cent of the de­cline in the la­bor force in July, ac­cord­ing to a National Women’s Law Center analy­sis of the Bureau of Labor Statistics’s lat­est dis­mal jobs re­port.

Overall, the econ­omy lost 23,000 jobs in July: Women lost 32,000 jobs, while men gained 9,000. Many of the jobs lost were in lo­cal gov­ern­ment and leisure and hos­pi­tal­ity—real jobs that real women, who are al­ready deal­ing with slowed wages and ris­ing in­fla­tion, work.

Meanwhile, a whop­ping 165,000 women ex­ited the work force last month, mean­ing they were nei­ther work­ing nor look­ing for a new job. The num­ber of men in the la­bor force re­mained steady.

Jasmine Tucker, vice pres­i­dent of re­search at NWLC, warned that this lat­est jobs re­port was “not a sign of a healthy econ­omy.”

“Women are leav­ing the la­bor force in alarm­ing num­bers, and many who re­main em­ployed are un­der­em­ployed. This ad­min­is­tra­tion can­not cel­e­brate an econ­omy that is fail­ing so many women,” Tucker said.

In fact, more than dou­ble the num­ber of women have left the la­bor force than men since the start of the year. A to­tal of 845,000 women have left the work­force since January, while 406,000 men have left.

It’s hard not to see these star­tling sta­tis­tics as part of the Trump ad­min­is­tra­tion’s broader cam­paign to un­der­mine wom­en’s rights and au­ton­omy.

Whether it’s mak­ing it harder for women to vote, sidelin­ing women pro­fes­sion­ally, or en­dan­ger­ing emer­gency preg­nancy care—tar­get­ing wom­en’s liveli­hoods just feels like part of the sys­tem that’s work­ing as it was de­signed.

Read more about the jobs re­port:

Marco Rubio is slowly chok­ing Cuba to death.

The Secretary of State told Axios his plan to take over the is­land—long plagued by U.S. in­tel­li­gence agency med­dling and ex­ten­sive fed­eral sanc­tions—for good.

“What we’re try­ing to teach them is there are no es­cape valves … Every time they cre­ate a new mech­a­nism in which they try to get out of the noose, we just close it off,” Rubio said. “They cer­tainly can’t wait us out. Certainly this is­n’t go­ing to go away for the next [two and a half] years.”

The Trump ad­min­is­tra­tion has kept a con­stant thumb on Cuba, start­ing with a black­out-in­duc­ing oil block­ade in January that has di­rectly con­tributed to wide­spread food in­se­cu­rity, eco­nomic col­lapse, and death, as in­fant mor­tal­ity dou­bled on the is­land. The U.S. has levied 24 dif­fer­ent at­tacks against Cuba, from ac­cus­ing its doc­tor ex­port pro­gram of hu­man traf­fick­ing, claim­ing that it’s aid­ing in­ter­na­tional ter­ror­ism, and run­ning an “irregular and covert” cam­paign against west­ern ide­olo­gies. And mil­i­tary ac­tion is still a pos­si­bil­ity.

“The Cuban Communist regime is a state spon­sor of ter­ror­ism that spies on America, arms vi­o­lent left-wing rad­i­cals, spreads poi­so­nous Marxist ide­ol­ogy, and serves as a stag­ing ground for Russia, China & Iran just 90 miles from our shores,” Rubio an­nounced on Thursday. “Today I sanc­tioned five Cuban en­ti­ties and eight in­di­vid­u­als as­so­ci­ated with procur­ing arms for the regime. As President Trump has said: The United States will not tol­er­ate a rogue state har­bor­ing hos­tile mil­i­tary, in­tel­li­gence & ter­ror op­er­a­tions on our doorstep. Anyone sup­port­ing, spon­sor­ing, or pro­vid­ing ser­vices to these sanc­tioned ac­tors is at risk of be­ing sanc­tioned them­selves.”

The Trump ad­min­is­tra­tion (Rubio in par­tic­u­lar) is lean­ing on Cold War, Red Scare rhetoric to jus­tify per­pet­u­at­ing a full-scale hu­man­i­tar­ian cri­sis in Cuba. On Thursday—the same day as Rubio’s an­nounce­ment and four days af­ter the most re­cent coun­try­wide black­out—a panel of in­de­pen­dent ex­perts ap­pointed by the Human Rights Council de­clared Cuba at risk of be­com­ing a “silent Gaza.”

“The hu­man­i­tar­ian con­se­quences are al­ready un­fold­ing into a full-blown cri­sis, threat­en­ing the rights to health, to life, to food and to de­vel­op­ment.… Measures that know­ingly de­prive a pop­u­la­tion of the means to sur­vive strike at the most ba­sic guar­an­tees of the rights to life and di­min­ish the core of hu­man dig­nity,” the re­port reads. “The U.S. Government must cease all threats and hos­tile acts against Cuba’s sov­er­eignty and re­voke all mea­sures it has im­posed on the coun­try that stand con­trary to in­ter­na­tional law.”

Nationwide black­outs have left mil­lions across Cuba with­out power. Some out­ages can last up to 20 hours, dis­rupt­ing every­thing from food stor­age, to cook­ing, to sleep.Black­outs have be­come an on­go­ing cri­sis since the U.S. blocked all oil ship­ments to the is­land in January. pic.twit­ter.com/​6T­P1Wv6SG2— AJ+ (@ajplus) August 7, 2026

Nationwide black­outs have left mil­lions across Cuba with­out power. Some out­ages can last up to 20 hours, dis­rupt­ing every­thing from food stor­age, to cook­ing, to sleep.

Blackouts have be­come an on­go­ing cri­sis since the U.S. blocked all oil ship­ments to the is­land in January. pic.twit­ter.com/​6T­P1Wv6SG2

Editor’s Pick

President Donald Trump is se­ri­ously crash­ing out af­ter a fed­eral ap­peals court or­dered him to stop il­le­gal con­struc­tion on his White House ball­room.

In a fu­ri­ous screed on Truth Social Friday, Trump an­nounced that he would im­me­di­ately ap­peal the de­ci­sion to the Supreme Court.

“The Military and Secret Service are view­ing this hor­ren­dous, po­lit­i­cally mo­ti­vated, and un­law­ful rul­ing as a National Security threat to our Nation in that the en­tire Complex is be­ing built for the pro­tec­tion of our Country and, ad­di­tion­ally, all fu­ture Presidents,” the pres­i­dent wrote.

Trump de­scribed his plans to build “one big, ex­pen­sive, and very com­plex unit” that in­cludes a state of the art hos­pi­tal, bomb shel­ter, and a top se­cret mil­i­tary fa­cil­ity—well, not so se­cret any­more, I’d gather.

In a 2 – 1 rul­ing ear­lier Friday, a  D.C. Circuit Court panel de­ter­mined the Trump ad­min­is­tra­tion must seek ap­proval for con­struc­tion from Congress, which has the “exclusive au­thor­ity to reg­u­late the con­struc­tion and de­mo­li­tion of White House struc­tures.”

“At this pre­lim­i­nary stage, the National Trust has shown, com­pellingly, that Congress has not ceded un­fet­tered au­thor­ity to the Executive Branch to dra­mat­i­cally re­design, re­shape, and re­con­struct the White House—the People’s House—to fit a par­tic­u­lar President’s de­sires,” the rul­ing stated.

Trump-appointed Judge Neomi Rao dis­sented, agree­ing with the ad­min­is­tra­tion that the National Trust for Historic Preservation did not have the stand­ing to chal­lenge the con­struc­tion. The non­prof­it’s case was cen­tered around Professor Alison Hoagland, a mem­ber of the National Trust, who for­mally claimed that the ball­room was an aes­thetic in­jury to the White House.

Rao ar­gued that the de­ci­sion would “permit ad­ju­di­ca­tion of any gov­ern­ment ac­tion that a plain­tiff finds un­sightly.” But by the judge’s logic, the Trump ad­min­is­tra­tion could make uni­lat­eral sweep­ing changes to any na­tional land­mark—even, as the DOJ tried to claim, the Statue of Liberty.

For months, Trump has treated the White House—which be­longs to all Americans, not just the pres­i­dent—like one of his gaudy re­sort prop­er­ties.

Meanwhile, the price tag on Trump’s ball­room has ex­ploded. The pres­i­dent orig­i­nally claimed that his ball­room pro­ject would only cost $200 mil­lion, but that num­ber later bal­looned to $300 mil­lion, and then $400 mil­lion af­ter he de­cided to tack on ex­tra con­struc­tion. In June, a bomb­shell re­port re­vealed that tax­pay­ers would be ex­pected to foot the bill for half of a $600 mil­lion to­tal cost.

This story has been up­dated.

Read more about the ball­room: