Your com­puter mo­dem screeches as it tries to con­nect to some­thing called the in­ter­net. Maybe it works. Maybe you try again.

For the first time in his­tory, you can ex­change let­ters with some­one across the world in sec­onds. Only 2000-something web­sites ex­ist, so you could the­o­ret­i­cally visit them all over a week­end. Most web­sites are just text on gray back­grounds with the oc­ca­sional pix­e­lated im­age. Loading times are bru­tal. A sin­gle im­age takes a minute, a 1-minute video could take hours. Most peo­ple do not trust putting their credit cards on­line. The ad­vice every­one gives: don’t trust strangers on the in­ter­net.

People split into two camps very soon.

Optimists pre­dict grand trans­for­ma­tions. Some be­lieve dig­i­tal com­merce will over­take phys­i­cal re­tail within years. Others in­sist we’ll wan­der around in vir­tual re­al­ity worlds.

“I ex­pect that within the next five years more than one in ten peo­ple will wear head-mounted com­puter dis­plays while trav­el­ing in buses, trains, and planes.” - Nicholas Negroponte, MIT Professor, 1993

If you told the av­er­age per­son in 1995 that within 25 years, we’d con­sume news from strangers on so­cial me­dia over news­pa­pers, watch shows on-de­mand in place of ca­ble TV, find ro­man­tic part­ners through apps more than through friends, and flip “don’t trust strangers on the in­ter­net” so com­pletely that we’d let in­ter­net strangers pick us up in their per­sonal ve­hi­cles and sleep in their spare bed­rooms, most peo­ple would find that hard to be­lieve.

We’re in 1995 again. This time with Artificial Intelligence.

And both sides of to­day’s de­bate are mak­ing sim­i­lar mis­takes.

One side warns that AI will elim­i­nate en­tire pro­fes­sions and cause mass un­em­ploy­ment within a cou­ple of years. The other claims that AI will cre­ate more jobs than it de­stroys. One camp dis­misses AI as over­hyped va­por­ware des­tined for a bub­ble burst, while the other pre­dicts it will au­to­mate every knowl­edge task and re­shape civ­i­liza­tion within the decade.

Both are part right and part wrong.

Geoffrey Hinton, who some call the Father of AI, warned in 2016 that AI would trig­ger mass un­em­ploy­ment. “People should stop train­ing ra­di­ol­o­gists now,” he de­clared, cer­tain that AI would re­place them within years.

Yet as Deena Mousa, a re­searcher, shows in “The Algorithm Will See You Now,” AI has­n’t re­placed ra­di­ol­o­gists, de­spite pre­dic­tions. It is thriv­ing.

In 2025, American di­ag­nos­tic ra­di­ol­ogy res­i­dency pro­grams of­fered a record 1,208 po­si­tions across all ra­di­ol­ogy spe­cial­ties, a four per­cent in­crease from 2024, and the field’s va­cancy rates are at all-time highs. In 2025, ra­di­ol­ogy was the sec­ond-high­est-paid med­ical spe­cialty in the coun­try, with an av­er­age in­come of $520,000, over 48 per­cent higher than the av­er­age salary in 2015.

Mousa iden­ti­fies a few fac­tors for why the pre­dic­tion failed - real-world com­plex­ity, the job in­volves more than im­age recog­ni­tion, and reg­u­la­tory/​in­sur­ance hur­dles. Most crit­i­cal she points is Jevons Paradox, which is the eco­nomic prin­ci­ple that a tech­no­log­i­cal im­prove­ment in re­source ef­fi­ciency leads to an in­crease in the to­tal con­sump­tion of that re­source, rather than a de­crease. Her ar­gu­ment is that as AI makes ra­di­ol­o­gists more pro­duc­tive, bet­ter di­ag­nos­tics and faster turn­around at lower costs mean more peo­ple get scans. So em­ploy­ment does­n’t de­crease. It in­creases.

This is also the Tech world’s con­sen­sus. Microsoft CEO Satya Nadella agrees, as does Box CEO Aaron Levie, who sug­gests:

“The least un­der­stood yet most im­por­tant con­cept in the world is Jevons Paradox. When we make a tech­nol­ogy more ef­fi­cient, de­mand goes well be­yond the orig­i­nal level. AI is the per­fect ex­am­ple of this—al­most any­thing that AI is ap­plied to will see more de­mand, not less.”

First, as Andrej Karpathy, the com­puter sci­en­tist who coined the term vibe cod­ing, points out, ra­di­ol­ogy is not the right job to look for ini­tial job dis­place­ments.

“Radiology is too multi-faceted, too high risk, too reg­u­lated. When look­ing for jobs that will change a lot due to AI on shorter time scales, I’d look in other places - jobs that look like rep­e­ti­tion of one rote task, each task be­ing rel­a­tively in­de­pen­dent, closed (not re­quir­ing too much con­text), short (in time), for­giv­ing (the cost of mis­take is low), and of course au­tomat­able giv­ing cur­rent (and dig­i­tal) ca­pa­bil­ity. Even then, I’d ex­pect to see AI adopted as a tool at first, where jobs change and refac­tor (e.g. more mon­i­tor­ing or su­per­vis­ing than man­ual do­ing, etc).”

Second, the tech con­sen­sus that we will see in­creased em­ploy­ment ac­tu­ally de­pends on the in­dus­try. Specifically, how much un­ful­filled de­mand can be un­locked in that in­dus­try, and whether this un­ful­filled de­mand growth out­paces con­tin­ued au­toma­tion and pro­duc­tiv­ity im­prove­ment.

To un­der­stand this bet­ter, look at what ac­tu­ally hap­pened in three in­dus­tries over a 200-year pe­riod from 1800 to 2000. In the pa­per Automation and jobs: when tech­nol­ogy boosts em­ploy­ment, James Bessen, an econ­o­mist, shows the em­ploy­ment, pro­duc­tiv­ity, and de­mand data for tex­tile, iron & steel, and mo­tor ve­hi­cle in­dus­tries.

After au­toma­tion, both tex­tile and iron/​steel work­ers saw em­ploy­ment in­crease for nearly a cen­tury be­fore ex­pe­ri­enc­ing a steep de­cline. Vehicle man­u­fac­tur­ing, by con­trast, holds steady and has­n’t seen the same steep de­cline yet.

To an­swer why those two in­dus­tries saw sharp de­clines but mo­tor ve­hi­cle man­u­fac­tur­ing did not, first look at the pro­duc­tiv­ity of work­ers in all three in­dus­tries:

Then look at the de­mand across those three in­dus­tries:

What the graphs show is a con­sis­tent pat­tern (note: the pro­duc­tiv­ity and de­mand graphs are log­a­rith­mic, mean­ing pro­duc­tiv­ity and de­mand grew ex­po­nen­tially). Early on, a ser­vice or prod­uct is ex­pen­sive be­cause many work­ers are needed to pro­duce it. Most peo­ple can’t af­ford it or use them spar­ingly. For ex­am­ple, in the early 1800s, most peo­ple could only af­ford a pair of pants or shirt. Then au­toma­tion makes work­ers dra­mat­i­cally more pro­duc­tive. A tex­tile worker in 1900 could pro­duce fifty times more than one in 1800. This pro­duc­tiv­ity ex­plo­sion crashes prices, which cre­ates mas­sive new de­mand. Suddenly every­one can af­ford mul­ti­ple out­fits in­stead of just one or two. Employment and pro­duc­tiv­ity both surge (note: em­ploy­ment growth masks in­ter­nal seg­ment dis­place­ment and wage changes. See foot­note)

Once de­mand sat­u­rates, em­ploy­ment does­n’t fur­ther in­crease but holds steady at peak de­mand. But as au­toma­tion con­tin­ues and work­ers keep get­ting more pro­duc­tive, em­ploy­ment starts to de­cline. In tex­tiles, mech­a­niza­tion en­abled mas­sive out­put growth but ul­ti­mately dis­placed work­ers once con­sump­tion plateaued while au­toma­tion and pro­duc­tiv­ity con­tin­ued climb­ing. We prob­a­bly don’t need in­fi­nite cloth­ing. Similarly, pa­tients will likely never need a mil­lion ra­di­ol­ogy re­ports, no mat­ter how cheap they be­come and so ra­di­ol­o­gists will even­tu­ally hit a ceil­ing. We don’t need in­fi­nite food, cloth­ing, tax re­turns, and so on.

Motor ve­hi­cles, in Bessen’s graphs, tell a dif­fer­ent story be­cause de­mand re­mains far from sat­u­rated. Most peo­ple glob­ally still don’t own cars. Automation has­n’t com­pletely con­quered man­u­fac­tur­ing ei­ther (Tesla’s re­treat from full man­u­fac­tur­ing au­toma­tion proves the cur­rent tech­ni­cal lim­its). When both de­mand and au­toma­tion po­ten­tial re­main high, em­ploy­ment can sus­tain or even grow de­spite pro­duc­tiv­ity gains.

Software pre­sents an even more in­ter­est­ing ques­tion. How many apps do you need? What about soft­ware that gen­er­ates ap­pli­ca­tions on de­mand, that cre­ates en­tire soft­ware ecosys­tems au­tonomously? Until now, hand­crafted soft­ware was the con­straint. Expensive soft­ware en­gi­neers and their la­bor costs lim­ited what com­pa­nies could af­ford to build. Automation changes this equa­tion by mak­ing those en­gi­neers far more pro­duc­tive. Both con­sumer and en­ter­prise soft­ware mar­kets sug­gest sig­nif­i­cant un­met de­mand be­cause busi­nesses have con­sis­tently left pro­jects un­built. They could­n’t jus­tify the de­vel­op­ment costs or had to al­lo­cate lim­ited re­sources to their top pri­or­ity pro­jects. I saw this first­hand at Amazon. Thousands of ideas went un­funded not be­cause they lacked busi­ness value, but be­cause of the lack of en­gi­neer­ing re­sources to build them. If AI can pro­duce soft­ware at a frac­tion of the cost, that un­leashes enor­mous la­tent de­mand. The key ques­tion then is if and when that de­mand will sat­u­rate.

So to gen­er­al­ize, for each in­dus­try, em­ploy­ment hinges on a race be­tween two forces:

The mag­ni­tude and growth of un­met mar­ket de­mand, and­Whether that de­mand growth out­paces pro­duc­tiv­ity im­prove­ments from au­toma­tion.

Different in­dus­tries will ex­pe­ri­ence dif­fer­ent out­comes de­pend­ing on who’s win­ning that de­mand and pro­duc­tiv­ity race.

The sec­ond de­bate cen­ters on whether this AI boom is a bub­ble wait­ing to burst.

The dot­com boom of the 1990s saw a wave of com­pa­nies adding “.com” to their name to ride the ma­nia and watch their val­u­a­tions soar. Infra com­pa­nies poured bil­lions into fiber op­tics and un­der­sea ca­bles - ex­pen­sive pro­jects only pos­si­ble be­cause peo­ple be­lieved the hype. All of this even­tu­ally burst in spec­tac­u­lar fash­ion in the dot­com crash in 2000-2001. Infrastructure com­pa­nies like Cisco briefly be­came the most valu­able in the world only to come tum­bling down. Pets.com served as the poster child of this ex­u­ber­ance rais­ing $82.5 mil­lion in its IPO, spend­ing mil­lions on a Super Bowl ad only to col­lapse nine months later.

But the dot­com bub­ble also got sev­eral things right. More im­por­tantly, it even­tu­ally bought us the phys­i­cal in­fra­struc­ture that made YouTube, Netflix, and Facebook pos­si­ble. Sure, com­pa­nies like Worldcom, NorthPoint, and Global Crossing mak­ing these in­vest­ments went bank­rupt, but they also laid the foun­da­tion for the fu­ture. Although the crash proved the skep­tics right in the short term, it proved the op­ti­mists were di­rec­tion­ally cor­rect in the long term.

Today’s AI boom shows sim­i­lar ex­u­ber­ance. Consider the AI startup founded by for­mer OpenAI ex­ec­u­tive Mira Murati, which raised $2 bil­lion at a $10 bil­lion val­u­a­tion, the largest seed round in his­tory. This de­spite hav­ing no prod­uct and de­clin­ing to re­veal what it’s build­ing or how it will gen­er­ate re­turns. Several AI wrap­pers have raised mil­lions in seed fund­ing with lit­tle to no moat.

Yet some in­vest­ments will out­last the hype and will likely help fu­ture AI com­pa­nies even if this is a bub­ble. For ex­am­ple, the an­nual cap­i­tal ex­pen­di­tures of Hyperscalers that have more than dou­bled since ChatGPT’s re­lease - Microsoft, Google, Meta, and Amazon are col­lec­tively spend­ing al­most half a tril­lion dol­lars on data cen­ters, chips, and com­pute in­fra­struc­ture. Regardless of which spe­cific com­pa­nies sur­vive, this in­fra­struc­ture be­ing built now will cre­ate the foun­da­tion for our AI fu­ture - from in­fer­ence ca­pac­ity to the power gen­er­a­tion needed to sup­port it.

The in­fra­struc­ture in­vest­ments may have long-term value, but are we al­ready in bub­ble ter­ri­tory? Azeem Azhar, a tech an­a­lyst and in­vestor, pro­vides an ex­cel­lent prac­ti­cal frame­work to an­swer the AI bub­ble ques­tion. He bench­marks to­day’s AI boom us­ing five gauges: eco­nomic strain (investment as a share of GDP), in­dus­try strain (capex to rev­enue ra­tios), rev­enue growth tra­jec­to­ries (doubling time), val­u­a­tion heat (price-to-earnings mul­ti­ples), and fund­ing qual­ity (the re­silience of cap­i­tal sources). His analy­sis shows that AI re­mains in a de­mand-led boom rather than a bub­ble, but if two of the five gauges head into red, we will be in bub­ble ter­ri­tory.

The de­mand is real. After all OpenAI is one of the fastest-grow­ing com­pa­nies in his­tory. But that alone does­n’t pre­vent bub­bles. OpenAI will likely be fine given its prod­uct-mar­ket fit, but many other AI com­pa­nies face the same unit eco­nom­ics ques­tions that plagued dot­com com­pa­nies in the 1990s. Pets.com had mil­lions of users too (a then large por­tion of in­ter­net users), but as the tech ax­iom goes, you can ac­quire in­fi­nite cus­tomers and gen­er­ate in­fi­nite rev­enue if you sell dol­lars for 85 cents. So de­spite the de­mand, the pat­tern may rhyme with the 1990s. Expect over­build­ing. Expect some spec­tac­u­lar fail­ures. But also ex­pect the in­fra­struc­ture to out­last the hype cy­cle and en­able things we can’t yet imag­ine.

So where does this leave us?

We’re early in the AI rev­o­lu­tion. We’re at that metaphor­i­cal screech­ing mo­dem phase of the in­ter­net era. Just as in­fra­struc­ture com­pa­nies poured bil­lions into fiber op­tics, hy­per­scalers now pour bil­lions into com­pute. Startups add “.ai” to their names like com­pa­nies once added “.com” as they seek higher val­u­a­tions. The hype will cy­cle through both eu­pho­ria and de­spair. Some pre­dic­tions will look laugh­ably wrong. Some that seem crazy will prove con­ser­v­a­tive.

Different in­dus­tries will ex­pe­ri­ence dif­fer­ent out­comes. Unlike what the Jevons op­ti­mists sug­gest, de­mand for many things plateaus once hu­man needs are met. Employment out­comes in any in­dus­try de­pend on the mag­ni­tude and growth of un­met mar­ket de­mand and whether that de­mand growth out­paces pro­duc­tiv­ity im­prove­ments from au­toma­tion.

Cost re­duc­tion will un­lock mar­ket seg­ments. Aswath Damodaran, a fi­nance pro­fes­sor, (in)famously un­der­val­ued Uber as­sum­ing it would only cap­ture a por­tion of the ex­ist­ing taxi mar­ket. He missed that mak­ing rides dra­mat­i­cally cheaper would ex­pand the mar­ket it­self as peo­ple took Ubers to des­ti­na­tions they’d never have paid taxi prices to reach. AI will sim­i­larly en­able prod­ucts and ser­vices cur­rently too ex­pen­sive to build with hu­man in­tel­li­gence. A restau­rant owner might use AI to cre­ate cus­tom sup­ply chain soft­ware that say at $100,000 with hu­man de­vel­op­ers would never have been built. A non-profit might de­ploy AI to con­test a le­gal bat­tle that was pre­vi­ously un­af­ford­able.

We can pre­dict change, but we can’t pre­dict the de­tails. No one in 1995 pre­dicted we’d date strangers from the in­ter­net, ride in their ubers, or sleep in their airbnbs. Or that a job called in­flu­encers would be­come the most sought-af­ter ca­reer among young peo­ple. Human cre­ativ­ity gen­er­ates out­comes we can’t fore­cast with our cur­rent men­tal mod­els. Expect new do­mains and in­dus­tries to emerge. AI has al­ready helped us de­code more an­i­mal com­mu­ni­ca­tion in the last five years than in the last fifty. Can we pre­dict what jobs a tech­nol­ogy that al­lows us to have full-blown con­ver­sa­tions with them will un­lock? A job that does­n’t ex­ist to­day will likely be the most sought-af­ter job in 2050. We can’t name it be­cause it has­n’t been in­vented yet.

Job cat­e­gories will trans­form. Even as the in­ter­net made some jobs ob­so­lete, it also trans­formed oth­ers and cre­ated new cat­e­gories. Expect the same with AI. Karpathy ends with a ques­tion:

About 6 months ago, I was also asked to vote if we will have less or more soft­ware en­gi­neers in 5 years. Exercise left for the reader.

To an­swer this ques­tion, go back to 1995 and ask the same ques­tion but with jour­nal­ists. You might have pre­dicted more jour­nal­ists be­cause the in­ter­net would cre­ate more de­mand by en­abling you to reach the whole world. You’d be right for 10 or so years as em­ploy­ment in jour­nal­ism grew un­til the early 2000s. But 30 years later, the num­ber of news­pa­pers and the num­ber of jour­nal­ists both have de­clined, even though more “journalism” hap­pens than ever. Just not by peo­ple we call jour­nal­ists. Bloggers, in­flu­encers, YouTubers, and newslet­ter writ­ers do the work that tra­di­tional jour­nal­ists used to do.

The same pat­tern will play out with soft­ware en­gi­neers. We’ll see more peo­ple do­ing soft­ware en­gi­neer­ing work and in a decade or so, what “software en­gi­neer” means will have trans­formed. Consider the restau­rant owner from ear­lier who uses AI to cre­ate cus­tom in­ven­tory soft­ware that is use­ful only for them. They won’t call them­selves a soft­ware en­gi­neer.

So just like in 1995, if the AI op­ti­mists to­day say that within 25 years, we’d pre­fer news from AI over so­cial me­dia in­flu­encers, watch AI-generated char­ac­ters in place of hu­man ac­tors, find ro­man­tic part­ners through AI match­mak­ers more than through dat­ing apps (or per­haps use AI ro­man­tic part­ners it­self), and flip “don’t trust AI” so com­pletely that we’d rely on AI for life-or-death de­ci­sions and trust it to raise our chil­dren, most peo­ple would find that hard to be­lieve. Even with all the in­tel­li­gence, both nat­ural and ar­ti­fi­cial, no one can pre­dict with cer­tainty what our AI fu­ture will look like. Not the tech CEOs, not the AI re­searchers, and cer­tainly not some ran­dom guy pon­tif­i­cat­ing on the in­ter­net. But whether we get the de­tails right or not, our AI fu­ture is load­ing.

An er­ror fare to Iceland pops up. It’s cheap enough to feel like a typo and most likely will be gone in min­utes. I’m mod­er­ately ob­sessed with ways to travel on bud­get, so I keep an eye on these.

Before I click Buy, I need to know (fast!) if it ac­tu­ally works for me: would I need a visa, are there any odd pass­port re­quire­ments, can I quickly sort out the dri­ving per­mit, would it af­fect my Schengen 90/180 win­dow, break UK pres­ence tests or ac­ci­den­tally pre­vent a tax res­i­dency I am chas­ing.

It is­n’t one check, it’s a stack of small un­friendly ones, and takes around 20 min­utes to process. Some bits are fun, like hunt­ing for a seat up­grade, but mostly it’s count­ing mid­nights and ex­piry dates so a cheap week­end does­n’t be­come an ex­pen­sive les­son.

I’ve been do­ing this dance for a decade now. In 2015 I made a spread­sheet for a US visa ap­pli­ca­tion that wanted ten years of travel his­tory, down to the day. The spread­sheet grew: UK work visa, Indefinite Leave to Remain and cit­i­zen­ship ap­pli­ca­tions, Canadian work per­mits. Any gov­ern­ment form that asked “where have you been?” got its an­swer from the same bat­tered CSV. It worked well enough, in the sense that I was never de­tained.

It also made me think that this was a solv­able prob­lem I was solv­ing badly. I built a ledger to an­swer “where was I on 15 March 2023?” Instead, I ran sim­u­la­tions to check, “if I book this, what breaks later?”

The only ques­tion is whether the com­puter can an­swer all of faster than I do, and leave December, the next bor­der con­trol, and the end of the tax year bliss­fully un­event­ful.

That twenty-minute panic be­fore buy­ing a flight comes from one ba­sic prob­lem: none of the sys­tems that judge you will tell you your state.

Schengen is run­ning one check. The UK is run­ning an­other. Tax res­i­dency is run­ning a third. Your pass­port is run­ning its own quiet clock in the back­ground. None of them ex­plain them­selves, and none of them agree on what “a day” even is.

Schengen cares about pres­ence across rolling win­dows. The UK counts how many mid­nights you were phys­i­cally in the coun­try in a tax year that, for his­tor­i­cal rea­sons , starts on 6 April out of all op­tions. Some coun­tries track how many days you’ve spent in cer­tain places and change the med­ical pa­per­work they ex­pect from you once you cross a thresh­old. Meanwhile your pass­port might fail you with its ex­piry date, va­lid­ity rules that may ap­ply on ar­rival or de­par­ture de­pend­ing on rout­ing, and a fi­nite num­ber of blank fac­ing pages that some coun­tries re­quire.

None of that is eas­ily ex­posed. The of­fi­cer at the desk can see it but you can’t. That’s pars­ing the State—both kinds. The gov­ern­men­t’s view of you, and the state ma­chine that tracks it.

The rules aren’t just com­plex—they’re oc­ca­sion­ally spe­cific in ways that make you re­gret leav­ing the house in the first place.

To ap­ply for British cit­i­zen­ship, you need to prove you were phys­i­cally in the UK on your ap­pli­ca­tion date but five years ago. Not ap­prox­i­mately five years, not that week—that ex­act day when you press “submit” on the form mi­nus five years. Miss it by 24 hours and your ap­pli­ca­tion is re­ject af­ter months of wait­ing, and you have to pay a hefty fee to re-ap­ply.

Transiting through a UK air­port? Leaving the ter­mi­nal does­n’t count as pres­ence un­less you do some­thing “unrelated to your travel”—buy a sausage roll at Greggs, see a play in West End, meet a friend. The guid­ance does­n’t even spec­ify a min­i­mum spend.

Morocco runs on UTC+1 most of the year but switches to UTC dur­ing Ramadan to shorten the fast­ing day. Which means “days spent in Morocco” de­pends on your time­zone data­base ver­sion and whether you re­mem­bered to up­date it.

It would be al­right with a sin­gle source of truth, but all these facts are scat­tered across (semi)official web­sites and PDFs, and you’re sup­posed to fig­ure it out your­self.

So the job is­n’t “log trips” (I al­ready did that for ten years in a spread­sheet). The job is: given what I’ve al­ready done and what I’m about to do, does this plan qui­etly break any­thing, and if so, where, and by how much.

“You’re at 56 days be­cause Amsterdam con­tributed 12, Prague 3, Barcelona 10, Iceland would add a month, and February does­n’t count any­more” is some­thing I can trust, ar­gue with, or fix. “You’re fine” is­n’t.

That’s where this stops be­ing a spread­sheet and starts be­ing a lin­ter. Apparently I want the com­piler warn­ing be­fore I press Buy.

If I want a com­piler warn­ing I trust, the com­piler has to agree with the of­fi­cer about what a day is.

For the past five years I’ve been work­ing on an app for peo­ple with epilepsy, man­ag­ing time­zones, med­ica­tion re­minders, and edge cases. We jug­gled mul­ti­ple sources of truth and mul­ti­ple stor­age styles (some records in UTC, some in lo­cal time with time­zones stored sep­a­rately—his­tor­i­cal rea­sons, ob­vi­ously).

This time, I tried to learn from that: facts are stored as in­stants, rea­son­ing hap­pens in lo­cal days of the ju­ris­dic­tion that cares.

Take this rout­ing: de­part Dublin morn­ing of November the 17th, brief Newark lay­over, a longer one in Mexico City, 23-hour Heathrow stop, then Tenerife. Ask five im­mi­gra­tion sys­tems “how many tax res­i­dency days?” and you get five an­swers:

* US: zero (foreign-to-foreign tran­sit un­der 24 hours).

* Mexico: two (you cross mid­night twice).

* UK: zero (even though you cross mid­night once), un­less you went land­side for non-travel rea­sons, then one.

* Schengen: one (entry day counts, exit day will count too, even if both are only for 15 min­utes).

Each stop has same or sim­i­lar con­di­tions, but dif­fer­ent state ma­chines are ask­ing dif­fer­ent ques­tions. I pin the time­zone data­base ver­sion that pro­duced each re­sult, and when rules or clocks shift, I re­com­pute so I could show both an­swers if needed. Yesterday should stay re­pro­ducible even when to­mor­row dis­agrees.

In other words, the lin­ter is meant to an­swer the same ques­tion in var­i­ous dis­guises: “what hap­pens if I do this?”

Can I book Christmas in the Alps with three sum­mer week­ends planned in Europe? Does it mat­ter if I leave UK be­fore the tax year ends? What pass­port should I travel on? Does any­thing ex­pire be­tween book­ing and board­ing?

Every ques­tion has the same shape: sim­u­late for­ward, find what breaks, de­cide if you care. The goal is­n’t to con­vince bor­der of­fi­cers—it’s to not make mis­takes they’d catch. Trips get as­sem­bled from what­ever I can ver­ify later: ge­o­t­agged pho­tos, back­ground lo­ca­tion, man­ual en­tries. A re­solver turns that into “present on this lo­cal day” and keeps track of why.

I don’t hard­code rules, I ship in­ter­pre­ta­tions in­stead: each ju­ris­dic­tion has a small ver­sioned blob that says what counts, how the win­dow is mea­sured, where that read­ing came from.

The pa­per­work gets the same treat­ment be­cause doc­u­ments are state ma­chines too. A pass­port is­n’t just means of iden­tity; it has con­straints and timers. Some re­quire­ments, like six months va­lid­ity are legacy and usu­ally ex­ist to keep de­por­ta­tions pos­si­ble with­out is­su­ing emer­gency doc­u­ments, but still need to be checked.

Before I buy any­thing the lin­ter should tell me that I don’t have the cor­rect flavour of IDP (and man, get­ting those in Scotland since they del­e­gated it from Post Offices to cor­ner shops is tough), that a Dubai con­nec­tion flips a “valid on ar­rival” buffer into “invalid on de­par­ture”. Quiet warn­ings, early enough to change dates or re­new the right book­let, and clear enough that I won’t have to im­pro­vise at a counter.

If the world changes—new ex­am­ples, re­vised guid­ance, a de­layed sys­tem fi­nally launches—I don’t rewrite his­tory. I ver­sion the as­sump­tion, keep both an­swers recorded, and move on.

Keeping all those rules up-to-date is hard, so rather than main­tain­ing rules for every coun­try, I parse a few data­bases, then let users con­fig­ure their own track­ing goals. A user emailed about Cyprus’s fast-tracked tax res­i­dency scheme; an­other pointed out I’d missed a few coun­tries en­tirely. The app gets bet­ter as peo­ple use it, which feels more hon­est than pre­tend­ing to be a global au­thor­ity on 195 coun­tries’ im­mi­gra­tion rules.

The app is lo­cal by de­fault. Calculations hap­pen on-de­vice, and if you’re in air­plane mode, it still works. Network is al­ways the bot­tle­neck, and I’m the per­son who spent a week­end re­verse-en­gi­neer­ing gym en­try to save 44 sec­onds. I’m not adding server round-trips when you’re stand­ing at a gate.

Being lo­cal also means no li­a­bil­ity. Personal im­mi­gra­tion his­tory is ex­actly the kind of data gov­ern­ments might want. Keeping it off my servers means no­body can de­mand I hand it over. Some friends asked about cloud sync: I keep say­ing no. Not be­cause sync is hard—it is, though surely Claude Code can do that for me —but be­cause the mo­ment you add a server you add re­ten­tion poli­cies, ju­ris­dic­tion ques­tions, and a mag­net for le­gal de­mands. If you want it on an­other de­vice, ex­port a file and move it your­self like the an­ces­tors did it.

The first ver­sion just counted Schengen days, then I added the UK’s mid­night arith­metic be­cause I needed it for my own cal­cu­la­tions. Then doc­u­ments with their ex­piry rules be­cause I was tired of man­u­ally look­ing them up. Then the what-if layer be­cause adding and delet­ing trips to see im­pact felt like man­u­ally diff­ing state. Then visa re­quire­ments and IDP rules: none of this was planned, it ac­cu­mu­lated from use, the same way my fer­men­ta­tion tracker grew from “can I eat this?” to HACCP com­pli­ance doc­u­ments.

I shipped it be­cause keep­ing it pri­vate felt un­fin­ished, and be­cause I’d like fewer peo­ple spend­ing twenty min­utes re­search­ing whether a £62 re­turn flight will cause prob­lems six months later.

That Iceland er­ror fare? I bought it. The app told me I would­n’t need an IDP, that the trip would­n’t push me over any Schengen thresh­old, that I’d leave with 34 days of mar­gin in my 90/180 win­dow, and—im­por­tantly—that book­ing it would mean I’d stop be­ing a UK tax res­i­dent given my up­com­ing Canada move. Useful things to know be­fore click­ing pur­chase. The of­fi­cer at Keflavík looked at his screen, agreed with his sys­tems, and waved me through.

I called the app Residency and you can get it here. No sub­scrip­tions, costs less than an air­port mar­tini, and you’ll likely re­gret it less a few hours later.

You can’t cURL a bor­der. But you can track your own state care­fully enough that when the gov­ern­ments know the an­swer, so do you.

Working on prob­lems where rules are com­plex and of­fi­cial doc­u­men­ta­tion is con­tra­dic­tory? I build sys­tems that han­dle state when the state won’t tell you your state. work@drobinin.com

OK, I said there would never be a ver­sion three of htmx.

But, tech­ni­cally, I never said any­thing about a ver­sion four…

In The Future of htmx I said the fol­low­ing:

We are go­ing to work to en­sure that htmx is ex­tremely sta­ble in both API & im­ple­men­ta­tion. This means ac­cept­ing and doc­u­ment­ing the quirks of the cur­rent im­ple­men­ta­tion.

Earlier this year, on a whim, I cre­ated fixi.js, a hy­per­min­i­mal­ist im­ple­men­ta­tion of the ideas in htmx. That work gave me a chance to get a lot more fa­mil­iar with the fetch() and, es­pe­cially, the

async in­fra­struc­ture avail­able in JavaScript.

In do­ing that work I be­gan to won­der if that, while the htmx API

is (at least rea­son­ably) cor­rect, maybe there was room for a more dra­matic change of the im­ple­men­ta­tion that took ad­van­tage of these fea­tures in or­der to sim­plify the li­brary.

Further, chang­ing from ye olde XMLHttpRequest

(a legacy of htmx 1.0 IE sup­port) to fetch() would be a pretty vi­o­lent change, guar­an­teed to break at least some stuff.

So I be­gan think­ing: if we are go­ing to con­sider mov­ing to fetch, then maybe we should also use this up­date as a chance ad­dress at least some of the quirks & cruft that htmx has ac­quired over its life­time.

So, even­tu­ally & re­luc­tantly, I have changed my mind: there will be an­other ma­jor ver­sion of htmx.

However, in or­der to keep my word that there will not be a htmx 3.0, the next re­lease will in­stead be htmx 4.0.

With htmx 4.0 we are re­build­ing the in­ter­nals of htmx, based on the lessons learned from fixi.js and five+ years of sup­port­ing htmx.

There are three ma­jor sim­pli­fy­ing changes:

The biggest in­ter­nal change is that fetch() will re­place XMLHttpRequest as the core ajax in­fra­struc­ture. This won’t ac­tu­ally have a huge ef­fect on most us­ages of htmx ex­cept that the events model will nec­es­sar­ily change due to the dif­fer­ences be­tween fetch() and XMLHttpRequest.

I feel that my biggest mis­take in htmx 1.0 & 2.0 was mak­ing at­tribute in­her­i­tance im­plicit. I was in­spired by CSS in do­ing this, and the re­sults have been roughly the same as CSS: pow­er­ful & mad­den­ing.

In htmx 4.0, at­tribute in­her­i­tance will be ex­plicit rather than im­plicit, via the :inherited mod­i­fier:

Here the hx-tar­get at­tribute is ex­plic­itly de­clared as in­her­ited on the en­clos­ing div and, if it was­n’t, the

but­ton el­e­ments would not in­herit the tar­get from it.

This will be the most sig­nif­i­cant up­grade change to deal with for most htmx users.

Another con­stant source of pain for both us and for htmx users is his­tory sup­port. htmx 2.0 stores his­tory in lo­cal cache to make nav­i­ga­tion faster. Unfortunately, snap­shot­ting the DOM is of­ten brit­tle be­cause of third-party mod­i­fi­ca­tions, hid­den state, etc. There is a ter­ri­ble sim­plic­ity to the web 1.0 model of blow­ing every­thing away and start­ing over. There are also se­cu­rity con­cerns stor­ing his­tory in­for­ma­tion in ses­sion stor­age.

In htmx 2.0, we of­ten end up rec­om­mend­ing that peo­ple fac­ing his­tory-re­lated is­sues sim­ply dis­able the cache en­tirely, and that usu­ally fixes the prob­lems.

In htmx 4.0, his­tory sup­port will no longer snap­shot the DOM and keep it lo­cally. It will, rather, is­sue a net­work re­quest for the re­stored con­tent. This is the be­hav­ior of 2.0 on a his­tory cache-miss, and it works re­li­ably with lit­tle ef­fort on be­half of htmx users.

We will of­fer an ex­ten­sion that en­ables his­tory caching like in htmx 2.0, but it will be opt-in, rather than the de­fault.

This tremen­dously sim­pli­fies the htmx code­base and should make the out-of-the-box be­hav­ior much more plug-and-play.

What Stays The Same?

The core func­tion­al­ity of htmx will re­main the same, hx-get, hx-post,

hx-tar­get, hx-boost, hx-swap, hx-trig­ger, etc.

Except for adding an :inherited mod­i­fier on a few at­trib­utes, many htmx pro­jects will “just work” with htmx 4.

These changes will make the long term main­te­nance & sus­tain­abil­ity of the pro­ject much stronger. It will also take pres­sure off the 2.0 re­leases, which can now fo­cus on sta­bil­ity rather than con­tem­plat­ing new fea­tures.

That said, htmx 2.0 users will face an up­grade pro­ject when mov­ing to 4.0 in a way that they did not have to in mov­ing from 1.0 to 2.0.

I am sorry about that, and want to of­fer three things to ad­dress it:

htmx 2.0 (like htmx 1.0 & in­ter­cooler.js 1.0) will be sup­ported in per­pe­tu­ity, so there is ab­solutely no pres­sure to

up­grade your ap­pli­ca­tion: if htmx 2.0 is sat­is­fy­ing your hy­per­me­dia needs, you can stick with it.

We will cre­ate ex­ten­sions that re­vert htmx 4 to htmx 2 be­hav­iors as much as is fea­si­ble (e.g. Supporting the old im­plicit

at­tribute in­her­i­tance model, at least)

We will roll htmx 4.0 out slowly, over a multi-year pe­riod. As with the htmx 1.0 -> 2.0 up­grade, there will be a long

pe­riod where htmx 2.x is lat­est and htmx 4.x is next

Of course, it is­n’t all bad. Beyond sim­pli­fy­ing the im­ple­men­ta­tion of htmx sig­nif­i­cantly, switch­ing to fetch also gives us the op­por­tu­nity to add some nice new fea­tures to htmx

By switch­ing to fetch(), we can take ad­van­tage of its sup­port for

read­able streams, which al­low for a stream of con­tent to be swapped into the DOM, rather than a sin­gle re­sponse.

htmx 1.0 had Server Sent Event sup­port in­te­grated into the li­brary. In htmx 2.0 we pulled this func­tion­al­ity out as an ex­ten­sion. It turns out that SSE is just a spe­cial­ized ver­sion of a stream­ing re­sponse, so in adding stream­ing sup­port, it’s an al­most-free free two-fer to add that back into core as well.

This will make in­cre­men­tal re­sponse swap­ping much cleaner and well-sup­ported in htmx.

Three years ago I had an idea for a DOM mor­ph­ing al­go­rithm that im­proved on the ini­tial al­go­rithm pi­o­neered by mor­phdom.

The idea was to use “id sets” to make smarter de­ci­sions re­gard­ing which nodes to pre­serve and which nodes to delete when merg­ing changes into the DOM, and I called this idea “idiomorph”. Idiomorph has gone on to be adopted by many other web pro­ject such as Hotwire.

We strongly con­sid­ered in­clud­ing it in htmx 2.0, but I de­cided not too be­cause it worked well as an ex­ten­sion and htmx 2.0 had al­ready grown larger than I wanted.

In 4.0, with the com­plex­ity sav­ings we achieved by mov­ing to fetch(), we can now com­fort­ably fit a mor­phIn­ner and

mor­phOuter swap into core, thanks to the ex­cel­lent work of Michael West.

htmx has, since very early on, sup­ported a con­cept of “Out-of-band” swaps: con­tent that is re­moved from the main HTML re­sponse and swapped into the DOM else­where. I have al­ways been a bit am­biva­lent about them, be­cause they move away from Locality of Behavior, but there is no doubt that they are use­ful and of­ten cru­cial for achiev­ing cer­tain UI pat­terns.

Out-of-band swaps started off very sim­ply: if you marked an el­e­ment as hx-swap-oob=‘true’, htmx would swap the el­e­ment as the outer HTML of any ex­ist­ing el­e­ment al­ready in the DOM with that id. Easy-peasy.

However, over time, peo­ple started ask­ing for dif­fer­ent func­tion­al­ity around Out-of-band swaps: prepend­ing, ap­pend­ing, etc. and the fea­ture be­gan ac­quir­ing some fairly baroque syn­tax to han­dle all these needs.

We have come to the con­clu­sion that the prob­lem is that there are re­ally two use cases, both cur­rently try­ing to be filled by Out-of-band swaps:

Therefore, we are in­tro­duc­ing the no­tion of s in htmx 4.0

A par­tial el­e­ment is, un­der the cov­ers, a tem­plate el­e­ment and, thus, can con­tain any sort of con­tent you like. It spec­i­fies on it­self all the stan­dard htmx op­tions re­gard­ing swap­ping, hx-tar­get and hx-swap in par­tic­u­lar, al­low­ing you full ac­cess to all the stan­dard swap­ping be­hav­ior of htmx with­out us­ing a spe­cial­ized syn­tax. This tremen­dously sim­pli­fies the men­tal model for these sorts of needs, and dove­tails well with the stream­ing sup­port we in­tend to of­fer.

Out-of-band swaps will be re­tained in htmx 4.0, but will go back to their ini­tial, sim­ple fo­cus of sim­ply re­plac­ing an ex­ist­ing el­e­ment by id.

htmx 2.0 has had View Transition sup­port since

April of 2023. In the in­ter­ced­ing two years, sup­port for the fea­ture has grown across browsers (c’mon, sa­fari, you can do it) and we’ve gained ex­pe­ri­ence with the fea­ture.

One thing that has be­come ap­par­ent to us while us­ing them is that, to use them in a sta­ble man­ner, it is im­por­tant to es­tab­lish a queue of tran­si­tions, so each can com­plete be­fore the other be­gins. If you don’t do this, you can get vi­su­ally ugly tran­si­tion can­cel­la­tions.

So, in htmx 4.0 we have added this queue which will en­sure that all view tran­si­tions com­plete smoothly.

CSS tran­si­tions will con­tinue to work as be­fore as well, al­though the swap­ping model is again made much sim­pler by the async run­time.

We may en­able View Transitions by de­fault, the jury is still out on that.

A won­der­ful thing about fetch() and the async sup­port in gen­eral is that it is much eas­ier to guar­an­tee a sta­ble or­der of events. By lin­eariz­ing asyn­chro­nous code and al­low­ing us to use stan­dard lan­guage fea­tures like try/​catch, the event model of htmx should be much more pre­dictable and com­pre­hen­si­ble.

We are go­ing to adopt a new stan­dard for event nam­ing to make things even clearer:

So, for ex­am­ple, htmx:be­fore:re­quest will be trig­gered be­fore a re­quest is made.

Another op­por­tu­nity we have is to take ad­van­tage of the async be­hav­ior of fetch() for much bet­ter per­for­mance in our pre­load ex­ten­sion (where we is­sue a spec­u­la­tive (GET only!) re­quest in an­tic­i­pa­tion of an ac­tual trig­ger). We have also added an op­ti­mistic up­date ex­ten­sion to the core ex­ten­sions, again made easy by the new async fea­tures.

In gen­eral, we have opened up the in­ter­nals of the htmx re­quest/​re­sponse/​swap cy­cle much more fully to ex­ten­sion de­vel­op­ers, up to and in­clud­ing al­low­ing them to re­place the fetch() im­ple­men­ta­tion used by htmx for a par­tic­u­lar re­quest. There should not be a need for any hacks to get the be­hav­ior you want out of htmx now: the events and the open “context” ob­ject should pro­vide the abil­ity to do al­most any­thing.

In htmx 2.0, I some­what re­luc­tantly added the hx-on at­trib­utes to sup­port light script­ing in­line on el­e­ments. I added this be­cause HTML does not al­low you to lis­ten for ar­bi­trary events via on

at­trib­utes: only stan­dard DOM events like onclick can be re­sponded to.

We hemmed and hawed about the syn­tax and so, un­for­tu­nately, there are a few dif­fer­ent ways to do it.

In htmx 4.0 we will adopt a sin­gle stan­dard for the hx-on at­trib­utes: hx-on:. Additionally, we are work­ing to im­prove the htmx JavaScript API (especially around async op­er­a­tion sup­port) and will make those fea­tures avail­able in hx-on:

Get A Response Then Remove It 3 Seconds Later

htmx will never sup­port a fully fea­tured script­ing mech­a­nism in core, we rec­om­mend some­thing like

Alpine.js for that, but our hope is that we can pro­vide a rel­a­tively min­i­mal­ist API that al­lows for easy, light async script­ing of the DOM.

I should note that htmx 4.0 will con­tinue to work with eval() dis­abled, but you will need to forego a few fea­tures like

hx-on if you choose to do so.

All in all, our hope is that htmx 4.0 will feel an aw­ful lot like 2.0, but with bet­ter fea­tures and, we hope, with fewer bugs.

As al­ways, soft­ware takes as long as it takes.

However, our cur­rent planned time­line is:

An al­pha re­lease is avail­able to­day: htmx@4.0.0-al­pha1

A 4.0.0 re­lease should be avail­able in early-to-mid 2026

4.0 will be marked lat­est in early-2027ish

You can track our progress (and see quite a bit of dust fly­ing around) in the four branch on

github and at:

Thank you for your pa­tience and par­don our dust!

“Well, when events change, I change my mind. What do you do?” –Paul Samuelson or John Maynard Keynes

Arthur Whitney is an es­teemed com­puter sci­en­tist who led the de­sign on a few well-known pieces of soft­ware:

* kdb, a high-per­for­mance data­base built on K used in fin­tech

* Shakti, which is like kdb but faster, de­signed for tril­lion-row datasets.

I’ve never even seen a tril­lion num­bers, much less cal­cu­lated them, but kdb is ap­par­ently a stan­dard tool on Wall Street. They prob­a­bly care about money, so I’ll as­sume kdb does its job well. His lan­guages take sig­nif­i­cantly af­ter APL, which was a very pop­u­lar lan­guage for sim­i­lar ap­pli­ca­tions be­fore the in­ven­tion of (qwerty) key­boards.

But I’m not here to talk about bor­ing things like “using soft­ware to make in­com­pre­hen­si­ble amounts of money in fi­nance” or “human be­ings and their ca­reers”, I’m here to talk about how a guy writes C code weird. For a very sim­ple ver­sion of the pro­gram­ming lan­guage K, there’s a pub­licly avail­able in­ter­preter he wrote in a few days us­ing about 50 lines of C to show the ba­sics of in­ter­preter writ­ing. This is the C (specifically the January 16, 2024 ver­sion #2):

type­def char*s,c;s Q=(s)128;

#define _(e…) ({e;})

#define x(a,e…) _(s x=a;e)

#define $(a,b) if(a)b;else

#define i(n,e) {int $n=n;int i=0;for(;i#in­clude”a.h”//​fF[+-!#,@] atom/​vec­tor 1byte(int/token) clang-13 -Os -oa a.c -w

#define r(n,e) _(s r=m(n);i(n,r[i]=e)r)

f(w,write(1,ax?&x:x,ax?1:strlen(x));x)F(err,w(a);w((s)58);w(x);w((s)10);Q)

_i(wi,s b[5];sprintf(b,“%d ”,x);w(b);0)

f(W,Q(x)$(ax,wi(ix))i(nx,wi(xi))w(10);x)

f(srt,Qz(1)0)f(uni,Qz(1)0)F(Cut,Qz(1)0)F(Drp,Qz(1)0)_i(m,s a=mal­loc(1+x);*a++=x;a)

#define A(c) ((s)memchr(a,c,na)?:a+na)-a

#define g(a,v) ax?255&a:r(nx,v)

f(not,g(!ix,!xi))f(sub,g(-ix,-xi))F(At,Qr(aa)g(a[ix],a[xi]))F(_A,Qr(aa)g(A(ix),A(xi)))

f(ind,Qr(!ax)0>ix?r(-ix,-ix-1-i):r(ix,i))F(Ind,Qr(!aa)Qd(1>ia)g(ix%ia,xi%ia))

#define G(f,o) F(f,ax?aa?255&ia o ix:Ltn==f?f(sub(x),sub(a)):f(x,a):r(nx,(aa?ia:a[i])o xi))

G(Ltn,

This is the en­tire in­ter­preter, and this is ap­par­ently how he nor­mally writes code. Opinions on his cod­ing style are di­vided, though gen­eral con­sen­sus seems to be that it’s in­com­pre­hen­si­ble. As daunt­ing as it is, I fig­ured I should give it a chance for a few rea­sons.

* As I work on larger and larger code­bases, scrolling up and down to track in­for­ma­tion has be­come a more com­mon an­noy­ance. Whitney’s talked about cod­ing the way he does to avoid ex­actly that: he wants to keep his logic on one screen.

Perhaps learn­ing to read code like this could give me ideas on writ­ing my own code more com­pactly.

* In a Hacker News com­ments sec­tion, some­body asked “would you rather spend 10 days read­ing 100,000 lines of code, or 4 days read­ing 1000?”, and that raises a good point.

The com­plex­ity of the code is be­cause even a sim­ple in­ter­preter is pretty com­plex. Writing it in 500 lines would­n’t make the com­plex­ity go away, it just spreads it out.

Does writ­ing in this more com­pact for­mat feel more daunt­ing be­cause you’re ex­posed to more of the com­plex­ity at once? I think so.

Does show­ing it all at once ac­tu­ally help you un­der­stand the whole thing faster? I don’t know.

* Reading code has be­come a more im­por­tant part of my job than writ­ing it, so I should chal­lenge my read­ing skills, re­gard­less.

* It con­fuses peo­ple, and that’s ba­si­cally the same as be­ing smart.

So I’m go­ing to go line by line and ex­plain my un­der­stand­ing. I tried to use the notes pro­vided in the repo only when I was stuck, which was a few times early on, but by the end I could un­der­stand it pretty well.

type­def char*s,c;

This al­ready shows some funky C. It de­fines s as char *, and c as char, be­cause the * at­taches to the name, not the type. It’s an odd­ity of C syn­tax that I’ve never been a fan of. Otherwise this is pretty straight for­ward: s is for string, and c is for char­ac­ter.

s Q=(s)128;

Fuck. Shit. He as­signed 128 to a string named Q. What does it mean? s is char *. Why is Q a pointer to the ad­dress 128? I thought I must have mis­un­der­stood, and s was ac­tu­ally a char­ac­ter or some­thing, but it’s clearly spec­i­fied as char *. s is for string! I could­n’t fig­ure out the mean­ing, so I soon gave up and looked at the an­no­tated code. The char * is un­signed long long in other ver­sions, and they ex­plain that the type is used for both in­te­gers and point­ers. The code op­er­ates on vec­tors of 8-bit in­te­gers, ei­ther as ASCII or num­bers, so it makes some sense to use char * from a mem­ory lay­out per­spec­tive, but I don’t use point­ers as in­te­gers very of­ten.

#define _(e…) ({e;})

#define x(a,e…) _(s x=a;e)

#define $(a,b) if(a)b;else

#define i(n,e) {int $n=n;int i=0;for(;i

These are all pretty straight for­ward, with one sub­tle caveat I only re­al­ized from the an­no­tated code. They’re all macros to make com­mon op­er­a­tions more com­pact: wrap­ping an ex­pres­sion in a block, defin­ing a vari­able x and us­ing it, con­di­tional state­ments, and run­ning an ex­pres­sion n times.

The sub­tle thing the an­no­ta­tions point out is the first macro, ({e;}). The paren­the­ses around curly brack­ets make this a state­ment ex­pres­sion, a non-stan­dard C ex­ten­sion that al­lows you to treat a block of state­ments as a sin­gle ex­pres­sion, if the last state­ment is an ex­pres­sion that pro­vides a value. In other words, int x = ({int a = func1(); int b = func2(a); a+b;}); sets x to what­ever a+b is. This is used every­where in the code af­ter this.

#define Q(e) if(Q==(e))re­turn Q;

#define Qs(e,s) if(e)re­turn err(__­func__,s);

#define Qr(e) Qs(e,“rank”)

#define Qd(e) Qs(e,“domain”)

#define Qz(e) Qs(e,“nyi”)

These are er­ror macros us­ing that mys­te­ri­ous Q de­fined ear­lier. Q seems to have been used to rep­re­sent er­rors, pos­si­bly short for “Quit”. The Qr/d/z func­tions seem to be types of er­rors. I have no idea what “nyi” means (I fig­ure it out later).

#define _s(f,e,x…) s f(x){re­turn _(e);}

#define _i(f,e) _s(f,e,c x)

#define f(f,e) _s(f,e,s x)

#define F(f,e) _s(f,e,s a,s x)

These re­place func­tion de­c­la­ra­tions, and we can see that _ macro be­ing used to add an im­plicit re­turn.

_s could be used like

_s(my_function, puts(“I rock!!!“); x*5+e, s x, int e)

, which would cre­ate ba­si­cally this stan­dard C:

char *my_function(char *x, int e) {

puts(“I rock!!!“);

re­turn x*5+e;

All the macros ex­cept the base _s also add im­plicit ar­gu­ments like a and x and you bet it’s hard to tell them apart.

#define ax (256>x)

This was an­other one that baf­fled me un­til I looked at the an­no­ta­tions. Remember how I said s val­ues were ei­ther in­te­gers or point­ers? 256 is the cut­off value for these in­te­gers, which the an­no­ta­tions call atoms, so ax means “is x an atom?”

#define ix (c)x

#define nx x[-1]

#define xi x[i]

These aren’t too con­fus­ing. ix casts x to a char. nx im­plies x is some sort of fat pointer, mean­ing there’s prob­a­bly a length at x[-1], but we’ll see. xi just in­dexes x as a nor­mal pointer, us­ing our im­plic­itly de­fined i from the i(…) macro.

#define aa x(a,ax)

#define ia x(a,ix)

#define na x(a,nx)

These copy ax, ix, and nx re­spec­tively to work on the a vari­able, which is an im­plicit ar­gu­ment in func­tions de­fined us­ing the F(f,e) macro. You re­mem­bered the x(name, ex­pres­sion) macro for as­sign­ing to a lo­cally-scoped x, right?

#define oo w(“oo\n”)

It prints oo. It’s not used any­where.

I wound up not need­ing to re­fer to the an­no­tated code at all to un­der­stand this. The C code is mostly us­ing every­thing in the head­ers to build the in­ter­preter.

#define r(n,e) _(s r=m(n);i(n,r[i]=e)r)

We cre­ate a vec­tor r from m(n) (which is de­fined later (it’s mal­loc)), fill r with the re­sults of e, and re­turn it out of the state­ment ex­pres­sion.

f(w,write(1,ax?&x:x,ax?1:strlen(x));x)

This de­fines s w(s x), which is our print func­tion. If x is an atom (ax?), we print it as a sin­gle char­ac­ter by get­ting its ad­dress (&x) and pro­vid­ing a length of 1. If it’s a vec­tor, we print it as a string us­ing strlen to cal­cu­late how long it is, so now we also know vec­tors must be null-ter­mi­nated here.

write and strlen are stan­dard func­tions that we call with­out in­clud­ing the head­ers, be­cause fuck head­ers. Let the linker fig­ure it out.

F(err,w(a);w((s)58);w(x);w((s)10);Q)

Our fancy shmancy er­ror print­ing func­tion, s err(s a, s x). The con­fus­ing thing is that : and the new­line are rep­re­sented by their ASCII num­bers 58 and 10, re­spec­tively. This just prints a mes­sage in the for­mat {a}:{x}\n and re­turns our spe­cial er­ror value Q.

_i(wi,s b[5];sprintf(b,“%d ”,x);w(b);0)

Defines s wi(c x), which takes x as a char, for­mats it as an in­te­ger in up to 5*sizeof(char*)/sizeof(char) char­ac­ters (40 on 64-bit ma­chines), and writes that.

f(W,Q(x)$(ax,wi(ix))i(nx,wi(xi))w(10);x)

Another print func­tion, s W(s x) ei­ther writes x as an in­te­ger or a list of in­te­gers. It also re­fuses to print the Q vec­tor.

f(srt,Qz(1)0) f(uni,Qz(1)0) F(Cut,Qz(1)0) F(Drp,Qz(1)0)

I fig­ured out what nyi means! It means “Not yet im­ple­mented”, as we can see from these func­tion de­f­i­n­i­tions.

_i(m,s a=mal­loc(1+x);*a++=x;a)

And we find our pre­vi­ously-used func­tion s m(c x), which al­lo­cates our buffer and re­turns a fat pointer (with the size at x[-1], hence the 1+x and a++). x is the length we’re al­lo­cat­ing here, which means our vec­tors are lim­ited to 255 bytes. The repo sug­gests up­grad­ing ca­pac­ity as an ex­er­cise to the reader, which could be fun.

#define A(c) ((s)memchr(a,c,na)?:a+na)-a

This macro finds the first oc­curence of the char­ac­ter c in our vec­tor a as an in­dex into the string (hence the -a, since mem­chr re­turns a pointer). If the re­sult is null, it just re­turns the length of the string (a+na - a). We see an­other fun bit of non-stan­dard syn­tax, ?:, which I had to look up. a ?: b is equiv­a­lent to a ? a : b with­out eval­u­at­ing a twice. Pretty snazzy!

#define g(a,v) ax?255&a:r(nx,v)

Strange lit­tle op­er­a­tion, I’ll have to see it in ac­tion. If x is an atom, it clamps a to be an atom with a sim­ple mask (255&a), oth­er­wise it cre­ates a new vec­tor the same size as x filled with the re­sult from v.

f(not,g(!ix,!xi)) f(sub,g(-ix,-xi)) F(At,Qr(aa)g(a[ix],a[xi])) F(_A,Qr(aa)g(A(ix),A(xi)))

Ah, I see now. g(a, v) lets us de­fine func­tions that work on both atoms and vec­tors. If x is an atom, it re­turns the atom re­sult clamped within the cor­rect bounds. Otherwise it al­lo­cates a new vec­tor and com­putes the other ex­pres­sion. All the above func­tions work ei­ther on x as an in­te­ger (ix), or on every el­e­ment of x (xi).

* s At(s a, s x) in­dexes into a, ei­ther to get one value or to shuf­fle them into a new vec­tor. a has to be a vec­tor.

* s _A(s a, s x) searches a vec­tor a for the value of x and gives us the in­dex, ei­ther one value or every value in the vec­tor.

This is a lot of func­tion­al­ity in such a small bit of code.

f(ind,Qr(!ax)0>ix?r(-ix,-ix-1-i):r(ix,i))

F(Ind,Qr(!aa)Qd(1>ia)g(ix%ia,xi%ia))

These are some atom-only func­tions.

* s ind(s x) cre­ates a vec­tor of length |x| con­tain­ing 0, 1… x-1 if x is pos­i­tive, oth­er­wise it con­tains -x-1, -x-2… 0.

The diode might be the most ne­glected com­po­nent in the elec­tron­ics cur­ricu­lum to­day. Pages upon pages have been writ­ten about the me­chan­ics of re­sis­tors, ca­pac­i­tors, and in­duc­tors; on this blog alone, we cov­ered the com­po­nents’ fun­da­men­tal equa­tions, the model of com­plex im­ped­ance, the be­hav­ior of R-C fil­ters, and more. As for semi­con­duc­tors, the diode’s more use­ful sib­ling — the tran­sis­tor — hogs the lime­light.

The diode is given nei­ther the math­e­mat­i­cal rigor of lin­ear cir­cuits nor the red-car­pet treat­ment of the tran­sis­tor. To the ex­tent that the com­po­nent is mar­veled at all, it’s usu­ally in the con­text of ex­otic in­ven­tions such as the Gunn diode or the tun­nel diode — both of which are al­most never en­coun­tered in real life.

If you’re rusty on con­cepts such as volt­age, cur­rent, or im­ped­ance, I sug­gest re­view­ing this ar­ti­cle first.

In an ear­lier post about the physics of semi­con­duc­tors, I noted that pure sil­i­con is a poor con­duc­tor of elec­tric­ity. This is be­cause the ma­te­r­ial lacks long-lived, mo­bile charge car­ri­ers. Some con­duc­tion is still pos­si­ble due to the short-lived ther­mal ex­ci­ta­tion of va­lence elec­trons that briefly pop into a higher-en­ergy state, but these elec­trons can’t travel far be­fore re­turn­ing to a lower-en­ergy level, so the ef­fect is more or less neg­li­gi­ble.

The con­duc­tiv­ity of the ma­te­r­ial is im­proved by the ad­di­tion of dopants. Some dopants con­tribute long-lived elec­trons that oc­cupy higher en­ergy lev­els with no lower-en­ergy va­can­cies to re­turn to; this is what’s called an n-type semi­con­duc­tor. Other ad­di­tives cre­ate eas­ily-ac­ces­si­ble va­lence band va­can­cies (“holes”); in such a p-type ma­te­r­ial, lower-en­ergy elec­trons can slither around from atom to atom with­out need­ing to be knocked into a higher-en­ergy state.

When an n-type ma­te­r­ial is brought into con­tact with a p-type semi­con­duc­tor, higher-en­ergy elec­trons from the n-side ran­domly dif­fuse to the p-side and then promptly fall into the abun­dant lower-en­ergy holes. This pro­duces a ther­mo­dy­namic equi­lib­rium with an in­ter­nal elec­tric field across the junc­tion. There is a pos­i­tive charge on the n-side and a neg­a­tive charge on the p-side:

The field pushes any mo­bile elec­trons that wan­der into the de­ple­tion re­gion back to the n-side. The re­sult is a thin, poorly con­duc­tive de­ple­tion re­gion at the bound­ary.

The field of a p-n junc­tion can be off­set by an ex­ter­nally-ap­plied volt­age; if we make the p-side suf­fi­ciently more pos­i­tive than the n-side, some cur­rent will flow. In the case of sil­i­con, the junc­tion be­comes markedly con­duc­tive when the for­ward volt­age reaches about 600 mV, al­though mi­croamp-range cur­rents will be flow­ing sooner than that.

A con­ven­tional diode is just a p-n junc­tion. The com­po­nent can be thought of as a rudi­men­tary volt­age-con­trolled gate: when the volt­age is be­low a cer­tain thresh­old, it pre­sents it­self as a very high re­sis­tance — typ­i­cally in ex­cess of 100 kΩ — so vir­tu­ally no cur­rent can flow. When the thresh­old is cleared, the diode ad­mits cur­rent that’s roughly pro­por­tional to the “excess” ap­plied volt­age, an ohmic be­hav­ior that’s a con­se­quence of the re­sis­tance of the ma­te­r­ial it­self:

So far, we dis­cussed a con­di­tion known as for­ward bias. If the diode is re­verse-bi­ased — that is, if the p-side is more neg­a­tive than the n-side — the com­po­nent no­tion­ally re­mains non­con­duc­tive. Well… up to a point.

One of the pos­si­ble re­verse-bias sce­nar­ios is that if the re­verse volt­age be­comes high enough, the elec­tric field can ac­cel­er­ate the odd charge car­rier in the de­ple­tion re­gion to a point where the par­ti­cle knocks other elec­trons into con­duc­tion band, pro­duc­ing an avalanche-like ef­fect. The abun­dance of these ki­net­i­cally-gen­er­ated charge car­ri­ers makes the junc­tion un­ex­pect­edly con­duc­tive again.

Most diodes are de­signed to keep this re­verse break­down volt­age well out­side the de­vice’s in­tended op­er­at­ing range. A spe­cial cat­e­gory of com­po­nents — known as Zener diodes — is en­gi­neered to ex­hibit re­verse break­down at lower, care­fully cal­i­brated volt­ages. Either way, once the thresh­old is ex­ceeded, a re­verse-bi­ased diode starts con­duct­ing just fine:

Now that we have the ba­sic the­ory laid out, we can have a look at some of the com­mon uses of diodes.

About the sim­plest ap­pli­ca­tion of diodes is cir­cuit pro­tec­tion. Let’s start with the arrange­ment shown on the left:

In the first cir­cuit, a Zener-type diode is placed in re­verse bias be­tween a pro­tected line and the ground. The diode re­mains non-con­duc­tive un­der nor­mal op­er­at­ing con­di­tions, but ex­pe­ri­ences elec­tri­cal break­down — and thus tem­porar­ily starts con­duct­ing — when the in­put volt­age ex­ceeds a safe limit. In ef­fect, the diode acts as a crow­bar that dis­si­pates en­ergy and pro­tects more del­i­cate com­po­nents down­stream. Diodes de­signed for this pur­pose are of­ten mar­keted as tran­sient volt­age sup­pres­sors (TVS) and are par­tic­u­larly im­por­tant for pro­tect­ing semi­con­duc­tors against sta­tic dis­charge. Another ap­pli­ca­tion is the sup­pres­sion of volt­age spikes that hap­pen when we abruptly cut the cur­rent sup­plied to mo­tors and other in­duc­tive loads.

A sin­gle diode can only be used to pro­tect in­puts where the in­put sig­nal has a de­fined po­lar­ity — that is, where one rail is al­ways more pos­i­tive than the other. To pro­vide over­volt­age pro­tec­tion for AC sig­nals with al­ter­nat­ing po­lar­ity, we in­stead rely a two-diode arrange­ment shown on the right in the il­lus­tra­tion above. This arrange­ment of com­po­nents is also avail­able in a sin­gle pack­age known as a bidi­rec­tional TVS.

In the lat­ter cir­cuit, re­gard­less of the po­lar­ity of the ap­plied volt­age, the “crowbar” path al­ways con­sists of a for­ward-bi­ased diode in se­ries with a re­verse-bi­ased one. It fol­lows that pos­i­tive and neg­a­tive con­duc­tion thresh­olds are the same: they’re equal to the diode’s re­verse-bias break­down volt­age, plus roughly 600 mV.

Yet an­other pro­tec­tion tech­nique is shown in the bot­tom panel. A reg­u­lar for­ward-bi­ased diode with a high re­verse break­down volt­age is placed in se­ries with the sup­ply; this arrange­ment pre­vents dam­age to sen­si­tive com­po­nents if the po­lar­ity of the sup­ply is ac­ci­den­tally re­versed, for ex­am­ple if the bat­ter­ies are in­serted the wrong way. The trade-off is the in­evitable volt­age drop across the p-n junc­tion, along with re­sis­tive heat­ing if the cur­rent is high, so a tran­sis­tor-based so­lu­tion is typ­i­cally pre­ferred, es­pe­cially in low-volt­age cir­cuits.

As men­tioned ear­lier, most diodes are de­signed to with­stand very high re­verse-bias volt­ages, of­ten in ex­cess of -100 V; that said, a spe­cial cat­e­gory of prod­ucts — Zener diodes — is de­signed to start con­duct­ing ear­lier than that.

When for­ward-bi­ased, such a diode be­haves the same as its con­ven­tional coun­ter­part, be­com­ing markedly con­duc­tive around 600 mV. When re­verse-bi­ased, it starts con­duct­ing at a low volt­age of the man­u­fac­tur­er’s choice, with com­mon low-volt­age op­tions rang­ing from 1.8 V to 30 V.

The V-I plots shown ear­lier in the ar­ti­cle tell us that once re­verse break­down be­gins, a small change in the ap­plied volt­age can pro­duce a com­par­a­tively large swing in the cur­rent flow­ing through. We can also look at it an­other way: if the cur­rent through the diode is lim­ited in some way, fluc­tu­a­tions in that cur­rent will have a com­par­a­tively mi­nor ef­fect on the volt­age that de­vel­ops across the diode’s ter­mi­nals.

This ob­ser­va­tion lends it­self to the use of diodes as volt­age ref­er­ences. We take an un­reg­u­lated power sup­ply — for ex­am­ple, a bat­tery — and in the sim­plest vari­ant, use a re­sis­tor to roughly limit the cur­rent flow­ing through the diode. Depending on the volt­age you want, you can use one or more for­ward-bi­ased diodes, a Zener diode, or some com­bi­na­tion thereof.

From Ohm’s law, the cur­rent ad­mit­ted by the re­sis­tor will change lin­early with the sup­ply volt­age (I = V/R), but these cur­rent fluc­tu­a­tions af­fect the diode volt­age to a much lower ex­tent. Here’s an ex­per­i­men­tal plot for the cir­cuit con­structed with a 1N4733 diode and a 100 Ω re­sis­tor:

The swing of the out­put volt­age is un­der 5% of the fluc­tu­a­tions of the in­put sig­nal: 45 mV ver­sus 1 V. This fig­ure might not sound par­tic­u­larly im­pres­sive, but the cir­cuit can be cas­caded: the out­put of one re­sis­tor-diode volt­age ref­er­ence can be used as the sup­ply volt­age for a sec­ond one. The ef­fects stack up: 5% of 5% is equal to 0.25%.

Of course, the Zener volt­age of the first diode should be higher than the sec­ond one. Further, for the cas­caded lay­out to be­have cor­rectly, we need to make sure that the cur­rent si­phoned off by the sec­ond stage is much lower than the cur­rent flow­ing through the in­tended re­sis­tor-diode path. One way to en­sure this prop­erty is to choose R1 ≪ R2. Another so­lu­tion is to sep­a­rate the stages with a tran­sis­tor cir­cuit that mir­rors the out­put volt­age of stage 1 for use by stage 2; such volt­age fol­lower cir­cuits are dis­cussed here.

Nowadays, more com­plex tran­sis­tor-based cir­cuits with tem­per­a­ture com­pen­sa­tion are more fa­vored for pre­ci­sion ap­pli­ca­tions. Nevertheless, a Zener diode still of­fers a vi­able a so­lu­tion in a pinch.

Let’s con­sider the cir­cuit shown be­low:

This cir­cuit is called a half-wave rec­ti­fier. The analy­sis is the eas­i­est if we ref­er­ence all cir­cuit volt­ages to the bot­tom out­put leg marked as B, and then eval­u­ate the pos­i­tive half-cy­cle of the in­put AC wave­form sep­a­rately from the neg­a­tive half-cy­cle:

During the pos­i­tive half-cy­cle of a sine wave — when the up­per ter­mi­nal of the sup­ply is more pos­i­tive — the diode is ini­tially for­ward-bi­ased. Assuming a low-im­ped­ance sig­nal source, this al­lows the ca­pac­i­tor to charge to a volt­age equal to the peak am­pli­tude of the in­put sig­nal, mi­nus some diode volt­age drop.

When the po­lar­ity of the sig­nal is re­versed — mak­ing the up­per sup­ply ter­mi­nal more neg­a­tive — the diode be­comes re­verse-bi­ased and does­n’t con­duct, so the ca­pac­i­tor holds charge. The fol­low­ing is a dis­crete-time sim­u­la­tion of the process; I added a mod­est re­sis­tive load across the out­put ter­mi­nals to dis­charge the ca­pac­i­tor slightly in be­tween each pos­i­tive peak:

If the load re­sis­tor is made a per­ma­nent part of the cir­cuit and if it’s cho­sen so that the ca­pac­i­tor dis­charges quickly enough to keep up with the am­pli­tude mod­u­la­tion of a car­rier wave, we get a cir­cuit known as the en­ve­lope fol­lower. The cir­cuit is a sim­ple way to ex­tract the ap­prox­i­mate mod­u­lat­ing wave­form from the car­rier sig­nal in AM ra­dio cir­cuits:

The same prin­ci­ple can be used if we want to build a cir­cuit that mea­sures the ap­prox­i­mate loud­ness of an au­dio sig­nal, or more gen­er­ally, hones in on the slow-chang­ing com­po­nent of any com­pos­ite wave­form.

The draw­back of the half-wave rec­ti­fier is that the ca­pac­i­tor is only charged by the pos­i­tive half-cy­cle of the sine wave; this is waste­ful if the goal is to max­i­mize the power de­liv­ered to a load. This de­fi­ciency can be ad­dressed by con­struct­ing a full-wave rec­ti­fier, shown be­low:

Once again, the analy­sis of the cir­cuit is the sim­plest if we con­sider out­put B to be the ref­er­ence point:

During the pos­i­tive half-cy­cle, diodes D1 and D2 are ini­tially for­ward-bi­ased; this al­lows the ca­pac­i­tor to charge to the AC peak volt­age (minus the summed volt­age drop of two diodes). During the neg­a­tive half-cy­cle, diodes D3 and D4 be­come for­ward-bi­ased, which con­nects the out­put B to the up­per sup­ply ter­mi­nal. Meanwhile, the bot­tom sup­ply ter­mi­nal, which is cur­rently at a higher volt­age, is con­nected to point A. This al­lows the ca­pac­i­tor to con­tinue charg­ing in the same po­lar­ity as be­fore.

A sim­u­la­tion of the process is shown be­low:

The rec­ti­fier cir­cuits out­lined in the pre­ced­ing sec­tion use diodes as volt­age-con­trolled switches. Another ap­pli­ca­tion of the same prin­ci­ple is a cir­cuit known as the volt­age dou­bler.

There are many fla­vors of volt­age dou­blers, but one par­tic­u­larly clean de­sign is shown be­low. The cir­cuit out­puts a DC volt­age that is equal to twice the peak am­pli­tude of the zero-cen­tered in­put wave­form, mi­nus the usual diode volt­age drops. This in con­trast to the rec­ti­fier cir­cuits dis­cussed ear­lier on, which only pro­duce DC volt­ages in the vicin­ity of the peak am­pli­tude:

This time, let’s use use the mid­point be­tween the two ca­pac­i­tors as the ref­er­ence point for volt­age mea­sure­ments. During the pos­i­tive half-cy­cle of the AC sig­nal (panel on the left), the up­per diode (D1) can con­duct to charge the top ca­pac­i­tor. This puts the out­put ter­mi­nal A at a pos­i­tive volt­age in re­la­tion to the mid­point.

Next, let’s have a look at the neg­a­tive half-cy­cle (right). In this sce­nario, the top diode is al­ways re­verse-bi­ased, so it does­n’t con­duct; C1 re­tains charge. At the same time, con­duc­tion is pos­si­ble for the lower diode D2, which charges C2 so that the out­put ter­mi­nal B ends up at a neg­a­tive volt­age in re­la­tion to the mid­point. In ef­fect, we store the pos­i­tive peak volt­age of the in­put wave­form in C1 and the neg­a­tive max­i­mum in C2. The to­tal volt­age be­tween B and A is V· 2 (once again, mi­nus the ex­pected diode volt­age drops).

This method of mul­ti­ply­ing volt­ages with switched ca­pac­i­tors lives on, al­though mod­ern cir­cuits typ­i­cally use dig­i­tally-con­trolled tran­sis­tors in­stead of diodes; this avoids volt­age drops and elim­i­nates the need for an AC volt­age sup­ply. If you’re in­ter­ested in such cir­cuits, I have a sep­a­rate ar­ti­cle here.

Most of the time, al­ter­nat­ing wave­forms that are cen­tered around zero volts are in­con­ve­nient to work with; in par­tic­u­lar, it’s more chal­leng­ing to build cir­cuits that run off a sin­gle volt­age sup­ply but that are able to dis­cern, am­plify, or gen­er­ate sig­nals that ex­tend be­low the neg­a­tive sup­ply rail.

This brings us to a some­what mind-bend­ing al­ter­na­tive ap­proach. The cir­cuit is known as a clam­per — or, less cryp­ti­cally, as a DC re­storer. It takes an AC wave­form and shifts it so that the neg­a­tive peaks are at roughly zero volts, with no ap­pre­cia­ble im­pact on sig­nal am­pli­tude:

For now, let’s ig­nore the op­tional re­sis­tor. We start with the first pos­i­tive half-cy­cle of the in­put sine wave (top left):

Initially, the ca­pac­i­tor is not charged (V = 0 V); fur­ther, there’s no cur­rent path via the diode, so charg­ing is not pos­si­ble. To elab­o­rate: for en­ergy to be stored in a ca­pac­i­tor, there must be a sym­me­try in the mo­tion of charges flow­ing onto and off the plates. This way, the re­sult­ing elec­tro­sta­tic fields — in­creas­ingly pos­i­tive on one plate and in­creas­ingly neg­a­tive on an­other — largely can­cel out, al­low­ing a non-triv­ial charge to be moved with the help of a mod­est volt­age.

If one of the ca­pac­i­tor’s is left float­ing, the de­vice can’t be charged or dis­charged; in­stead, its pre­vi­ous charge state per­sists as a volt­age across its ter­mi­nals. If we take a ca­pac­i­tor charged to 1 V and con­nect one of its legs to a 10 V sup­ply, the other leg will read 11 V in ref­er­ence to the ground. If we con­nect it to a -5 V sup­ply, we’ll get a read­ing of -4 V. It’s no dif­fer­ent from putting a bat­tery in se­ries with an­other volt­age source. In our cir­cuit, the ini­tial V = 0 V, so in the pos­i­tive half-cy­cle, the ca­pac­i­tor adds noth­ing, and the volt­age on the out­put leg A sim­ply fol­lows the in­put wave­form.

In the fol­low­ing neg­a­tive half-cy­cle, once the volt­age at point A reaches about -600 mV, the diode starts to con­duct (top right). This clamps the volt­age at point A while si­mul­ta­ne­ously al­low­ing the ca­pac­i­tor to charge so that its left ter­mi­nal be­comes neg­a­tive in re­la­tion to the right ter­mi­nal. If we mea­sure V left to right, the re­sult­ing volt­age is pos­i­tive and is equal to the peak am­pli­tude of the sup­ply sig­nal (minus the diode volt­age drop).

In the sub­se­quent pos­i­tive cy­cle, the diode be­comes re­verse-bi­ased again, so the ca­pac­i­tor must hold its pre­vi­ous charge; this means that V re­mains un­changed and that the volt­age at point A is nec­es­sar­ily off­set from the in­put wave­form by that amount. If the in­put wave­form runs -V to +V, the out­put will be now mov­ing from roughly -600 mV to V· 2 — 600 mV.

The role of the op­tional load re­sis­tor is that it con­trols the cir­cuit’s abil­ity to re­spond to grad­ual shifts in sig­nal am­pli­tude. Without it, the cir­cuit would the­o­ret­i­cally be stuck for­ever at a volt­age off­set dic­tated by the largest en­coun­tered sin­gle-cy­cle swing in the in­put wave­form. With a re­sis­tor, the ca­pac­i­tor can dis­charge over time, so the off­set can change if the en­ve­lope of the wave­form changes in some way.

In prac­tice, the diode’s leak­age cur­rent and the ca­pac­i­tor’s self-dis­charge rate are typ­i­cally enough to make the cir­cuit be­have rea­son­ably even with­out an out­put re­sis­tance. If you wish to play around with this lay­out, I rec­om­mend us­ing a 10-100 µF ca­pac­i­tor and ei­ther skip­ping the re­sis­tor or us­ing a large one (1 MΩ).

An OR gate is a cir­cuit with two or more in­puts that pro­duces a pos­i­tive volt­age when any of the in­puts is pos­i­tive. An AND gate, in con­trast, out­puts a pos­i­tive sig­nal only if all the in­puts are pos­i­tive. Many read­ers are prob­a­bly fa­mil­iar with the ap­pli­ca­tion of this con­cept in com­put­ing, but more sim­ply: a prac­ti­cal ap­pli­ca­tion of an OR gate might be a cir­cuit that raises alarm if any of the door or win­dow sen­sors are trig­gered. An ap­pli­ca­tion of an AND gate might be a sys­tem that il­lu­mi­nates a “PARKING FULL” sign when all the spots are oc­cu­pied.

There are sim­ple ways to im­ple­ment this logic with diodes:

In the case of an OR cir­cuit (left), if the pos­i­tive sup­ply rail is con­nected to any of the in­put ter­mi­nals, this for­ward-bi­ases the cor­re­spond­ing diode and causes a cur­rent to flow through a re­sis­tor. Because the im­ped­ance of a for­ward-bi­ased diode is much lower than that of a 10 kΩ re­sis­tor, the out­put volt­age shoots up very close to the up­per sup­ply rail.

The AND cir­cuit (right) es­sen­tially works in re­verse: if any of the in­puts is con­nected to the ground, this pulls the out­put volt­age close to 0 V, so the out­put is pos­i­tive only if both in­puts are high (or are left float­ing).

The rea­son I put “gate” in scare quotes in the il­lus­tra­tion is that the cir­cuits are not read­ily com­pos­able to im­ple­ment more com­plex dig­i­tal logic; each of the gates re­quires cur­rent to flow through the in­put ter­mi­nals, but it can’t nec­es­sar­ily de­liver such cur­rents on its out­put leg. A par­tic­u­larly trou­ble­some sit­u­a­tion is shown be­low:

With the in­put sig­nals as in­di­cated in the draw­ing, three out of four diodes are re­verse-bi­ased; the only cur­rent path is the se­ries re­sis­tor-diode-re­sis­tor con­nec­tion be­tween the pos­i­tive sup­ply rail and the ground. The cir­cuit is es­sen­tially a re­sis­tor-based volt­age di­vider; in­stead of pro­vid­ing a bi­nary out­put, it pro­duces an am­bigu­ous in­ter­me­di­ate volt­age.

In other words, the so­lu­tion works for sin­gle-step logic; to build real com­put­ers, we’d need gates that can de­liver out­put cur­rents higher than what they de­mand as in­put of the gates up­stream.

👉 For a write-up on pho­to­di­odes, click here. For more ar­ti­cles about elec­tron­ics, math, and geek life, check out this page. And you en­joy the con­tent, please sub­scribe. I don’t sell any­thing; it’s just a good way to stay in touch with the au­thors you like.

The morn­ing went well. The morn­ings al­ways go well. Everybody knows what they’re do­ing. We’re pro­fes­sion­als, equals. Same pay. Same ben­e­fits. All work­ing to­gether to­ward re­tire­ment. We look out for each other. Whoever has the hard­est task in this crew to­day could be the fore­man to­mor­row, and vice versa. Nobody wants to be the boss, so our bosses are the best kind.

See, the truck no­body else wanted had been my of­fice. I’d built a portable desk in­side it. My truck desk, I called it. A cou­ple of planks screwed to­gether, our union sticker slapped on, the whole deal sealed with shel­lac. I’d built the desk so it slid into the bot­tom of the steer­ing wheel and sat across the arm­rests. I used to hang back at the job and sneak in some cre­ative work while the rest of the crew went to break. My desk—which I’d taken far too long to build and per­fect through many pro­to­types—had been stowed be­hind the dri­ver’s seat when the truck was hauled off by the wrecker.

Back at the break trailer, I took my old seat and joined in on the jokes, in­sults, tall tales. That trailer was, to me, the best place for sto­ry­telling in the world—but, as al­ways, it was too loud, too rau­cous, too fun to do any writ­ing or read­ing, which is all I ever want to do on break. At lunch, I re­treated into the rel­a­tive quiet of the ma­chine shop. I sat down by the drill press and took out my cell phone and started writ­ing. Just like I used to do.

For nearly two decades I’ve worked off and on at this petro­chem­i­cal plant as a me­chanic and welder. The union dis­patched me here: When it gets slow, I get laid off; when work picks up, I boomerang back. And the whole time, I’ve writ­ten sto­ries and parts of my nov­els dur­ing breaks—fif­teen min­utes for cof­fee and then half an hour for lunch. I’ve also made use of the heaven-sent de­lays brought on by light­ning, se­vere rain­storms, evac­u­a­tions, per­mit­ting prob­lems, equip­ment is­sues, and so on. I’m thank­ful for each and every de­lay that hap­pens on this con­struc­tion site, and, be­lieve me, there are many.

One day I walked into the pay­roll trailer where the sec­re­taries and site man­ager sat. There was­n’t an ex­plicit sign that said , but it was an un­spo­ken rule. The trailer had a few un­used old cu­bi­cles tucked to the side. I sat down in one and hap­pily pecked away with my thumbs. Every break for a week I went in and worked on my writ­ing. After a few days I started to feel like I should hang pic­tures of my mom and dad and my wife in­side it. But I did­n’t dare.

Then things re­ally heated up. I brought in a Bluetooth key­board and wrote a whole story that day on my breaks. There was no go­ing back. My heart soared. I thought I should adopt a brown dog with a ban­danna around his neck just so I could thumb­tack his pic­ture to the cu­bi­cle wall. I had­n’t in­ter­acted with any of the of­fice staff, but they’d seen me. They’d fol­lowed my oily boot­prints down the hall­way and be­gun to leer. Who is this diesel-stink­ing con­trac­tor? He’s prob­a­bly the one who’s been eat­ing Janelle’s Oreos. He raided the mango-kiwi yo­gurt from the fridge. He glommed all the sporks. I knew my cu­bi­cle dreams were over the morn­ing I found the site man­ager wait­ing in “my” cu­bi­cle.

In all my years work­ing at that place, I’d never seen the site man­ager out on the site. I’m not sure he knew what it was or where it was. You went to him to or­der tools; he was the one who said no. I’d only ever seen him at a uri­nal or buy­ing ba­con and eggs off the lunch truck. But if I had ever seen him out on the site, it would have never oc­curred to me to ask him what he was do­ing there. He was wear­ing a blue polo shirt and khakis, and I was in his world—and he was ask­ing.

I started writ­ing in the ma­chine shop again. It was­n’t the same. Once I’d been in­fected by the cu­bi­cle virus, there was no go­ing back. Out of scrap lum­ber I gath­ered from var­i­ous dump­sters, I built a proper desk for my­self in the north­east cor­ner of the shop. That desk was a huge leap for­ward in pos­si­bil­ity and pro­duc­tiv­ity. In the evenings, if I wrote some­thing by hand or on my type­writer at home, I could now use my time at work to re­type it at my shop desk.

It was made of three boards cut at twenty-four inches. Light and com­pact. Sealed with shel­lac. It slid into the bot­tom of the steer­ing wheel, one side sup­ported by a curved re­bar I welded into a nut that fit ex­actly in a re­cess on the dri­ver’s door. The cen­ter con­sole sup­ported the other side of the desk. I kept it stored be­hind the seat. Whenever break time came and the crew drove back to the trailer com­pound, I stayed parked on the unit and got at least ten ex­tra min­utes to write.

Every morn­ing, when I find out what crew I’m in, I bring that plank with me. I stick it on the dash­board and climb into the dri­ver’s seat. I drive us all out to the job and at break time I take them to the trailer. I clean my hands with pumice wipes and sit alone in who­ev­er’s truck it is that day, pulling the plank off the dash­board and set­ting it across the arm­rests. Within a minute or so, I’ve got the lap­top out and I’m work­ing. If some­body from the crew is still in the back seat, ban­danna over their eyes, snooz­ing, I do my best to keep ex­tra quiet. And if they be­gin to snore, I don’t let that bother me at all.

Bud Smith is the au­thor of the novel Teenager and the story col­lec­tion Double Bird, among other books. Mighty, a novel, is forth­com­ing from Knopf in spring 2027. His story “Skyhawks” ap­pears in the new Fall is­sue of The Paris Review.

Welcome! This is an­other Sunday edi­tion of the Animation Obsessive newslet­ter, and here’s our slate:

A quick note be­fore we start. The newslet­ter re­cently passed 60,000 sub­scribers — a num­ber we can’t be­lieve. Huge thanks to every­one who’s cho­sen to come along as we look into an­i­ma­tion from all over the world.

With that, let’s go!

Before YouTube, TikTok or so­cial me­dia as a whole, there was Flash.

It changed the in­ter­net. The Shockwave Flash (.SWF) file for­mat al­lowed an­i­ma­tion and even games to run on dial-up con­nec­tions. It’s all been re­tired now — but, in its hey­day, web de­sign­ers every­where used Flash to make sites feel slick.

There was an­other group that took to Flash, too: am­a­teur an­i­ma­tors. The pro­gram was easy to use, and cre­ations with it were easy to share. This caused the first on­line an­i­ma­tion boom. A solo artist with only a home com­puter could reach mil­lions.

That was a big deal in the United States — but it was, in some ways, a big­ger deal in China. Flash ar­rived there in the late ’90s, and it be­came era-defin­ing. The so-called “Flashers” (闪客) spoke with the bold, new voice of the younger gen­er­a­tion. Their work fas­ci­nated the press and pub­lic alike.

“Chinese un­der­ground films and rock ‘n’ roll mu­sic were once the ex­pres­sion of re­bel­lious pas­sions in the pur­suit of … new cul­tural di­men­sions,” scholar Weihua Wu once wrote, “but to­day these pas­sions are ex­pressed through the chan­nels of Flash.”

China’s in­ter­net cafes were packed with Flash-watchers by the early 2000s. And one of their fa­vorites was Xiao Xiao (2000–2002), a vi­o­lent ac­tion se­ries about stick fig­ures. It was sim­ple, ut­terly of its time and copied to in­fin­ity. Quickly, it out­grew China and be­came a phe­nom­e­non around the world.

For peo­ple of a cer­tain age, who grew up on­line, Xiao Xiao and its clones were part of life. The se­ries is­n’t deep and has lit­tle to say: it’s just kung fu, fire­fights, blood and chaos. But it was the height of cool to its (mostly young) au­di­ence. It was also a gate­way to Flash an­i­ma­tion for many.

That was true in­side and out­side China. Xiao Xiao was a hit on Western sites like Newgrounds and Albino Blacksheep, which hosted Flash files. It did num­bers across Taiwan, Korea, Japan and be­yond. These bru­tal lit­tle stick fig­ures — and their moves rem­i­nis­cent of Hong Kong ac­tion movies and The Matrix — con­nected world­wide.

Still, the se­ries is Chinese. Its cre­ator, Zhu Zhiqiang, was a man in his mid-20s. And he was an un­likely star.

Zhu was­n’t trained in an­i­ma­tion. Before he dis­cov­ered Flash, he eked out a liv­ing as a graphic de­signer. He was­n’t tech-y, ei­ther. Zhu first en­coun­tered com­put­ers only in 1997, af­ter his move from Jilin City (in the Northeast) to Beijing.

Those were hard years. Zhu had dropped out af­ter ju­nior high, and his first Beijing job paid 800 yuan per month — some­thing like $181 af­ter in­fla­tion. He was clock­ing end­less over­time to make up the dif­fer­ence. In fact, for him, get­ting work at all could mean telling lies. He landed one job by claim­ing he’d grad­u­ated from a “technical sec­ondary school” but had­n’t brought his diploma.

Still, Zhu had a pas­sion. He loved stick fig­ures. He’d started an­i­mat­ing them in 1989, at age 14 — draw­ing in the cor­ners of book af­ter book, flip­ping the pages to make his char­ac­ters move. His in­spi­ra­tions were partly Jackie Chan films, partly his child­hood love of Dragon Ball.

Around the turn of the cen­tury, Zhu re­vis­ited that pas­sion through soft­ware.

A lot was hap­pen­ing when he made his first vir­tual stick-fig­ure fight, Dugu Qiubai, in April 2000. The Flash scene was just tak­ing off in China through sites like Flash Empire. A few months ear­lier, a vi­o­lent Flash called Gangster’s Paradise got pop­u­lar, launch­ing a new era of am­bi­tion for Flashers. Its cre­ator be­came Zhu’s “idol.”

Zhu an­i­mated Dugu Qiubai with Corel Painter, but he quickly switched to Flash. He ba­si­cally started from zero. Not long be­fore, he said, “I had no com­puter skills, my English was not good, I could­n’t pro­gram and my draw­ing [ability] was av­er­age.” He learned Flash mostly alone, us­ing a mouse to cre­ate each frame. That be­came his process. In mid-2000, Zhu made a se­quel to Dugu Qiubai called Xiao Xiao No. 2.

These early pro­jects were pop­u­lar, and they got Zhu a web de­sign job at Sohu, the Beijing com­pany that hosted his work. As 2000 went on, the press no­ticed China’s Flash scene, and he was one of the artists they pro­filed. But all of this was only a tee-up for Xiao Xiao No. 3 — the pro­ject that made Zhu’s stick fig­ures world-fa­mous.

Zhu put seven months into Xiao Xiao No. 3. It was a hobby pro­ject, made in his spare time. The film is sim­ply drawn, and there’s no plot or act­ing here: just a gi­ant brawl, as a lone stick­man takes down a small army. The cool fac­tor was un­prece­dented for a Flash an­i­ma­tion back then, though. It counted for a lot. There’s some­thing a lit­tle mes­mer­iz­ing even to­day about the move­ment of Zhu’s fight­ers.

When Xiao Xiao No. 3 ap­peared in April 2001, the re­ac­tion was im­me­di­ate, even in the States. Forum users were call­ing it “mind blow­ing,” and get­ting their threads locked be­cause so many peo­ple had al­ready posted about it. A Nintendo fan­site went off-script to show this “l33t an­i­ma­tion” to its read­ers. Later in the year, it hit the Detroit Free Press:

Just when you thought you’d seen the most Flash an­i­ma­tions could do on­line, along comes Xiao Xiao No. 3. Put sim­ply, it’s a Jackie Chan-style gang-against-one mar­tial arts fight among stick fig­ures. If that sounds silly, the chore­og­ra­phy is truly Chan-worthy, with fig­ures run­ning up walls, do­ing 360-degree kicks and gen­er­ally show­ing off the best that mar­tial arts of­fer. … Even the cin­e­matog­ra­phy is in­spir­ing: 3D shots, slow-mo­tion and a Matrix-style cam­era wrap­around that’ll have you ask­ing for more.

Zhu could tell that he’d hit upon some­thing. He’d burned his email ad­dress into the video, and view­ers used it to con­tact him. “The part that made the deep­est im­pres­sion,” Zhu re­called, “was on the third day, I re­ceived 1,200 emails in one day. There was no junk in them, and 80% of them were not writ­ten by Chinese peo­ple, but by peo­ple from all over the world.”

Even though the in­ter­net was small at the time, Xiao Xiao No. 3’s num­bers re­main im­pres­sive. By late 2001, it had 800,000-plus views on Flash Empire and more than half a mil­lion on Newgrounds, not count­ing the (many) other sites that hosted it. Over the years, the Newgrounds up­load alone would climb up­ward of 5 mil­lion. A writer for Flash Empire noted, “I don’t know how the au­thor made such a good work.”

Without warn­ing, Zhu had turned into an early e-celebrity. The Xiao Xiao se­ries blew up just as Flash was tak­ing over the main­land. Some called 2001 “the year of Chinese Flash,” and one mag­a­zine named Zhu the “Internet Person of the Year.”

Hearing all the buzz, Zhu’s fa­ther went to an in­ter­net cafe to see Xiao Xiao for him­self. He was shocked. It was the same stuff, he re­al­ized, that Zhu had drawn in the cor­ners of those books.

Zhu soon be­came a free­lance an­i­ma­tor — and he started to make money. Major brands (like Cityplaza in Hong Kong) wanted his stick fig­ures in their ads. Just three months af­ter Xiao Xiao No. 3, he made a deal with the com­pany Barunson to host the se­ries in Korea.

By the end of 2001, Zhu was on TV with his fel­low Flashers, and the press was vis­it­ing his home. Again, Zhu was an un­likely star. Reporters found that he slept on a couch, rarely went out­side and woke up close to the af­ter­noon. His limit as a cook was a sim­ple potato-and-egg dish, so he ate out a lot — but only cheaply. Asked how long he worked each day, he said, “More than 8 hours.” Besides that, he mainly watched TV.

… is a quiet young man, easy­go­ing and a lit­tle shy; he spoke very slowly and it takes him longer than oth­ers to put his ideas into words. But when it comes to de­sign and Flash he res­olutely sticks to his own ideas. … He stays at home most of the time and sel­dom goes out, ex­cept for some Flashmaker get-to­geth­ers and ac­tiv­i­ties. All of his time is de­voted to an­i­ma­tion.

Zhu did­n’t in­vent vi­o­lent stick­man an­i­ma­tions. In the ’90s, the Western site Stick Figure Death Theatre hosted ex­actly what its name im­plied. But Xiao Xiao, and its mix of Jackie Chan with Jet Li with The Matrix, per­fected the idea. (Also, Zhu de­nied hav­ing seen the for­eign stick­man work be­fore­hand — af­ter all, he’d done these him­self since age 14.)

Either way, it was Xiao Xiao that made “stick fights” mas­sive on­line. Clones were ram­pant — even Stick Figure Death Theatre had them. As one pa­per re­ported in 2002:

The Web’s le­gions of part-time Flash an­i­ma­tors have be­gun pro­duc­ing their own copies of Xiao Xiao — so many, in fact, that there’s a whole por­tal ded­i­cated to them. Stick Figure Death Theatre … has so many stick man knock­offs, you have to won­der why Zhu does­n’t just give up.

The com­pe­ti­tion did­n’t shrink Xiao Xiao, though. As Zhu made new episodes through 2001 and 2002 (a shoot­ing game here, a kung fu film there), his work still tow­ered. And he was get­ting more am­bi­tious. His code was bet­ter, and episodes seven and eight are full of 3D cam­er­a­work — done with the help of 3ds Max. Xiao Xiao re­mained to­tally style-over-sub­stance, but that style con­tin­ued to ex­cite the in­ter­net.

A ques­tion hung over the Flashers, though. Where was all of this headed?

The Flash scene in China was de­fined by self-ex­pres­sion. As Zhu said, “TV is made for every­one to watch; Flash is made for one­self to watch.” That ethos made the scene big­ger and big­ger — by 2003, Flash Empire had its own show on tele­vi­sion. Yet these artists needed to make money, too. Otherwise, it could only be a hobby.

Money was the hard prob­lem. Getting hired to do Flash an­i­ma­tion was­n’t a guar­an­tee, and the rates for this work weren’t al­ways the best. Zhu noted that “most Flashers have very dif­fi­cult lives.” There was the prob­lem of own­er­ship, too. A lot of Flashers freely swiped copy­righted sound, mu­sic and char­ac­ters. How could they ever charge?

Zhu tried to an­swer those ques­tions. His “matchstick man,” the pitch-black stick fig­ure that leads Xiao Xiao, was a rec­og­niz­able char­ac­ter. Brands wanted to as­so­ci­ate with it, and Zhu wanted to profit from it. He kept his work as orig­i­nal as pos­si­ble, copy­righted his films and, in 2003, sub­mit­ted a trade­mark ap­pli­ca­tion for the stick­man.

By then, though, the Nike ads were al­ready air­ing abroad.

In early 2003, Nike rolled out the “Stickman” cam­paign. It stars an an­i­mated stick fig­ure much like the one in Xiao Xiao, and its flashy moves are, at times, sus­pi­ciously sim­i­lar to Zhu’s ideas. When this cam­paign hit China, he con­tacted Nike about it. That went nowhere, so he sued.

Nike was de­fi­ant when the story hit the pa­pers. “It is ob­vi­ous that the plain­tiff in­tended to pro­mote him­self and his Flash works by ac­cus­ing a fa­mous multi-na­tional com­pany,” said one Nike lawyer.

Meanwhile, Zhu said that the Stickman char­ac­ter was so sim­i­lar to his own, peo­ple as­sumed he’d done the ads him­self. Plus, he called stick fig­ures like these a “perfect” fit for sports ad­ver­tis­ing — which meant trou­ble for him. “Once Nike uses [these char­ac­ters], how is any­one go­ing to come to me to make them?” he asked. Already, a pub­lisher had backed out of his book pro­ject thanks to Nike’s ads, Zhu claimed.

Although Zhu kept re­leas­ing a lit­tle Xiao Xiao-branded com­mer­cial work, there were no new episodes in 2003. “I’ll re­sume af­ter the law­suit is over,” he said. The case turned into a years­long or­deal. He saw its out­come as crit­i­cal to his fu­ture.

Zhu Zhiqiang told re­porters that he has al­ready drawn up a bright “career blue­print” for “Matchstick Man,” but the most im­por­tant thing right now is to win the law­suit. After all, these bright prospects are all based on the con­fir­ma­tion of the copy­right to “Matchstick Man.”

Zhu ini­tially won in court. Then the ap­peals process ran un­til 2006, when he fi­nally lost. The stick fig­ures were too dif­fer­ent, ac­cord­ing to the judges, and the im­agery was too sim­ple to copy­right. Nike was in the clear. Zhu faced thou­sands of dol­lars in court fees.

After this, he did­n’t an­i­mate much. Motorola hired him to do an­other Xiao Xiao-inspired ad in 2007, but the se­ries never re­turned. By 2008, Zhu worked as a coder. He and James Gao, the cre­ator of Flash Empire, joined the cre­ative agency VML and got in­volved in mo­bile games. Xiao Xiao was done.

In many ways, Zhu’s art ca­reer was a ca­su­alty of the early in­ter­net. If the roadmap from vi­ral suc­cess to solid in­come is faint to­day, it did­n’t ex­ist at all in 2003. Getting a Xiao Xiao-sized hit now would, most likely, give some­one a shot at start­ing a busi­ness. Zhu had none of the tools nec­es­sary to do so.

The Stickman cam­paign was just an­other layer of the prob­lem. It was the era when a ma­jor com­pany could brush off the bad PR that comes with copy­ing a ma­jor on­line artist. Is it be­liev­able that no one in­volved in the Nike ads had seen Xiao Xiao? Not re­ally — it was pop­u­lar with young peo­ple world­wide. Yet Zhu was new me­dia at a time when old me­dia ruled. What could he do?

All that said, the Xiao Xiao story is­n’t one of fail­ure, first and fore­most. It has to be em­pha­sized: this se­ries was huge. It in­spired peo­ple around the world. That hap­pened in China (Bu Hua, the cre­ator of Cat, was a big Xiao Xiao fan) and abroad.

Xiao Xiao is def­i­nitely shal­low, and its look is­n’t ad­ven­tur­ous. But that sim­plic­ity made it ap­proach­able. Any young per­son could watch it, get it and copy it. Other defin­ing hits from China’s Flash scene — Big Fish & Chinese Flowering Crab Apple, The Black Bird and so on — could be too com­plex or spe­cific to China for the world to im­i­tate. Xiao Xiao was­n’t. Back then, any­one could un­der­stand a stick-fig­ure fight.

Zhu may not have reaped the ben­e­fits of this se­ries him­self, but he ab­solutely paved the way for the Flash scene — which paved the way for what came next.

Chinese Flash, which Xiao Xiao helped to pi­o­neer, trained a new gen­er­a­tion of an­i­ma­tors and film­mak­ers. Lots of key peo­ple started there, in­clud­ing Zhang Ping, whose Legend of Hei II made a se­ri­ous dent in the box of­fice this year. Flash artists abroad were no less im­pacted by the suc­cess of Zhu’s work.

A cou­ple of years ago, an archivist named Ben Latimore put out an ebook. Since Adobe be­gan the re­tire­ment of Flash in 2017, he’s been pre­serv­ing .SWF files and the his­tory around them. His book is a chron­i­cle of the Flash era, which he sees as a lost golden age. On the fi­nal page, he wrote this about that time:

… in­tense cre­ativ­ity, easy-to-ac­cess soft­ware, no­table but not crip­pling lim­i­ta­tions, al­most uni­ver­sal com­pat­i­bil­ity across the en­tire tech­no­log­i­cal space of its time, wide­spread adop­tion by en­cour­ag­ing free con­sump­tion and shar­ing in an age where “going vi­ral” ac­tu­ally meant some­thing, all com­bin­ing to in­flu­ence the en­tire en­ter­tain­ment in­dus­try with one strike af­ter an­other? That’s some­thing that we’ll never be able to recre­ate, only re­mem­ber fondly. All dri­ven by a bunch of guys sit­ting in their bed­rooms who watched too much Xiao Xiao.

Police in Israel have ar­rested and de­tained the mil­i­tary’s top le­gal of­fi­cer af­ter she ad­mit­ted leak­ing footage of sol­diers al­legedly at­tack­ing a Palestinian de­tainee and then in ef­fect ly­ing about her ac­tions to Israel’s high court.

The mil­i­tary ad­vo­cate gen­eral, Yifat Tomer-Yerushalmi, said in a res­ig­na­tion let­ter last week that she had au­tho­rised pub­li­ca­tion of the video to defuse at­tacks on mil­i­tary in­ves­ti­ga­tors and pros­e­cu­tors work­ing on the case.

Rightwing politi­cians and pun­dits cham­pi­oned sol­diers de­tained over the case as “heroes”, at­tacked mil­i­tary in­ves­ti­ga­tors as trai­tors, and called for the case against the sol­diers to be dropped.

Tomer-Yerushalmi has now been ar­rested on sus­pi­cion of fraud and breach of trust, abuse of of­fice, ob­struc­tion of jus­tice, and dis­clo­sure of of­fi­cial in­for­ma­tion by a pub­lic ser­vant, Israeli me­dia re­ported.

Her ar­rest and de­ten­tion raises se­ri­ous ques­tions about the rule of law in Israel, ac­count­abil­ity for abuse and killing of Palestinians dur­ing what a UN com­mis­sion has called a geno­ci­dal war, and the coun­try’s abil­ity to de­fend it­self in in­ter­na­tional courts.

In July 2024 pros­e­cu­tors raided the Sde Teiman mil­i­tary de­ten­tion cen­tre, which has be­come no­to­ri­ous for tor­ture, and de­tained 11 sol­diers for in­ter­ro­ga­tion.

They were sus­pects in a vi­o­lent as­sault on a Palestinian from Gaza, in­clud­ing anal rape. The vic­tim was hos­pi­talised with in­juries in­clud­ing bro­ken ribs, a punc­tured lung and rec­tal dam­age, ac­cord­ing to the in­dict­ment, and Tomer-Yerushalmi launched an in­ves­ti­ga­tion.

The gov­ern­ment and far-right politi­cians and pun­dits have ac­cused her of dam­ag­ing Israel’s global stand­ing by purs­ing the case and re­leas­ing the video, in ef­fect cast­ing her ef­forts to pros­e­cute ex­treme vi­o­lence as a pro­ject to un­der­mine the state.

“The in­ci­dent in Sde Teiman caused im­mense dam­age to the im­age of the state of Israel and the IDF [Israel Defense Forces],” the Israeli prime min­is­ter, Benjamin Netanyahu, said in a state­ment on Sunday. “This is per­haps the most se­vere pub­lic re­la­tions at­tack that the state of Israel has ex­pe­ri­enced since its es­tab­lish­ment.”

After the first de­ten­tions of sol­diers in the case in sum­mer 2024, a far-right mob gath­ered out­side Sde Teiman call­ing for the in­ves­ti­ga­tion to be dropped. Some of the pro­test­ers — in­clud­ing a min­is­ter and two mem­bers of the Knesset — broke into the base.

Tomer-Yerushalmi leaked the video in August 2024 af­ter the protests, say­ing in her res­ig­na­tion let­ter that it was “an at­tempt to de­bunk false pro­pa­ganda against army law en­force­ment bod­ies”.

Days later, five sol­diers were charged with ag­gra­vated abuse and caus­ing se­ri­ous bod­ily harm. They have not been named and are cur­rently not in cus­tody or un­der any le­gal re­stric­tions, Israeli me­dia re­ported.

Tomer-Yerushalemi sub­se­quently re­fused to open or ad­vance in­ves­ti­ga­tions into other cases of pos­si­ble war crimes by the Israeli mil­i­tary, be­cause of the pres­sure of pub­lic at­tacks over the case, Haaretz re­ported.

There has been only one con­vic­tion of an Israeli sol­dier for as­sault­ing Palestinians in de­ten­tion dur­ing the war, al­though wide­spread tor­ture and abuse have been doc­u­mented in Israel’s jail sys­tem, and dozens of Palestinians have died in cap­tiv­ity.

No sol­diers have been charged for killing civil­ians in Gaza, even af­ter high-pro­file at­tacks that prompted in­ter­na­tional out­rage, in­clud­ing the killing of para­medics and strikes on a team from the World Central Kitchen char­ity. Tens of thou­sands of Palestinian civil­ians in Gaza have been killed in at­tacks and airstrikes over two years.

Attacks on Tomer-Yerushalemi over the Sde Teiman af­fair in­ten­si­fied in re­cent days amid re­ports that she was re­spon­si­ble for leak­ing the video. There were of­fi­cial de­mands for her to step down and per­sonal threats on­line, even af­ter she an­nounced her res­ig­na­tion.

The cam­paign briefly halted on Sunday af­ter­noon amid fears for her life, af­ter her part­ner re­ported her miss­ing to the po­lice and her car was found empty at a beach in the Tel Aviv area with a note in­side, Israeli me­dia re­ported.

Then she was found, and within min­utes the at­tacks re­sumed. The far-right com­men­ta­tor Yinon Magal posted on X, “we can pro­ceed with the lynch­ing”, adding a wink­ing emoji.

Soon af­ter, pro­test­ers had gath­ered out­side her house, Israeli me­dia re­ported, shout­ing slo­gans in­clud­ing “we will give you no peace”. The de­fence min­is­ter, Israel Katz, later ac­cused her of “spreading blood li­bels”.

Traditionally Israel’s gov­ern­ment and mil­i­tary have con­sid­ered the ex­is­tence of an in­de­pen­dent ju­di­ciary a cru­cial bar­rier to in­ter­na­tional le­gal tri­bunals in­ves­ti­gat­ing Israel for al­leged abuses against Palestinians.

Where there is a ro­bust na­tional le­gal sys­tem will­ing and able to in­ves­ti­gate and pros­e­cute crimes, in­ter­na­tional courts are less likely to have ju­ris­dic­tion to in­ter­vene.

“Don’t they un­der­stand we had no choice? That the only way to ad­dress the wave of in­ter­na­tional le­gal pro­ceed­ings is by prov­ing we can in­ves­ti­gate our­selves?” the in­ves­tiga­tive re­porter Ronen Bergman quoted the ad­vo­cate gen­eral telling col­leagues six weeks ago, in a re­port for Yedioth Ahronoth news­pa­per.

In re­cent decades many Israelis have seen the role of the mil­i­tary ad­vo­cate gen­eral “as pro­tect­ing sol­diers from pros­e­cu­tion abroad”, said Prof Yagil Levy, head of the Institute for the Study of Civil-Military Relations at Israel’s Open University.

“In other words, the law is not up­held as a value in it­self, but as a de­fence against in­ter­na­tional tri­bunals.”

Now even such le­gal prag­ma­tism is un­der at­tack by the po­lit­i­cal right, whose in­flu­ence can be seen in the lack of le­gal ac­count­abil­ity for sol­diers’ con­duct in Gaza over the past two years, Levy added.

“During the war, the ad­vo­cate gen­eral gave the army a free hand in Gaza, for ex­am­ple, re­gard­ing the un­prece­dented col­lat­eral dam­age from airstrikes,” he said.

“This re­flects a far weaker com­mit­ment to in­ter­na­tional law, with some on the right claim­ing that Israel is ex­empt from re­spect­ing it, and even pro­vid­ing re­li­gious jus­ti­fi­ca­tions for this view.”

Skip to main con­tent­The Case That A. I. Is ThinkingChatGPT does not have an in­ner life. Yet it seems to know what it’s talk­ing about.How con­vinc­ing does the il­lu­sion of un­der­stand­ing have to be be­fore you stop call­ing it an il­lu­sion?Dario Amodei, the C.E.O. of the ar­ti­fi­cial-in­tel­li­gence com­pany Anthropic, has been pre­dict­ing that an A.I. “smarter than a Nobel Prize win­ner” in such fields as bi­ol­ogy, math, en­gi­neer­ing, and writ­ing might come on­line by 2027. He en­vi­sions mil­lions of copies of a model whirring away, each con­duct­ing its own re­search: a “country of ge­niuses in a dat­a­cen­ter.” In June, Sam Altman, of OpenAI, wrote that the in­dus­try was on the cusp of build­ing “digital su­per­in­tel­li­gence.” “The 2030s are likely go­ing to be wildly dif­fer­ent from any time that has come be­fore,” he as­serted. Meanwhile, the A.I. tools that most peo­ple cur­rently in­ter­act with on a day-to-day ba­sis are rem­i­nis­cent of Clippy, the one­time Microsoft Office “assistant” that was ac­tu­ally more of a gad­fly. A Zoom A.I. tool sug­gests that you ask it “What are some meet­ing ice­break­ers?” or in­struct it to “Write a short mes­sage to share grat­i­tude.” Siri is good at set­ting re­minders but not much else. A friend of mine saw a but­ton in Gmail that said “Thank and tell anec­dote.” When he clicked it, Google’s A.I. in­vented a funny story about a trip to Turkey that he never took.The rushed and un­even roll­out of A.I. has cre­ated a fog in which it is tempt­ing to con­clude that there is noth­ing to see here—that it’s all hype. There is, to be sure, plenty of hype: Amodei’s time­line is sci­ence-fic­tional. (A.I. mod­els aren’t im­prov­ing that fast.) But it is an­other kind of wish­ful think­ing to sup­pose that large lan­guage mod­els are just shuf­fling words around. I used to be sym­pa­thetic to that view. I sought com­fort in the idea that A.I. had lit­tle to do with real in­tel­li­gence or un­der­stand­ing. I even cel­e­brated its short­com­ings—root­ing for the home team. Then I be­gan us­ing A.I. in my work as a pro­gram­mer, fear­ing that if I did­n’t I would fall be­hind. (My em­ployer, a trad­ing firm, has sev­eral in­vest­ments in and part­ner­ships with A.I. com­pa­nies, in­clud­ing Anthropic.) Writing code is, by many ac­counts, the thing that A.I. is best at; code has more struc­ture than prose does, and it’s of­ten pos­si­ble to au­to­mat­i­cally val­i­date that a given pro­gram works. My con­ver­sion was swift. At first, I con­sulted A.I. mod­els in lieu of look­ing some­thing up. Then I gave them small, self-con­tained prob­lems. Eventually, I gave them real work—the kind I’d trained my whole ca­reer to do. I saw these mod­els di­gest, in sec­onds, the in­tri­cate de­tails of thou­sands of lines of code. They could spot sub­tle bugs and or­ches­trate com­plex new fea­tures. Finally, I was trans­ferred to a fast-grow­ing team that aims to make bet­ter use of A.I. tools, and to cre­ate our own.The sci­ence-fic­tion au­thor William Gibson is said to have ob­served that the fu­ture is al­ready here, just not evenly dis­trib­uted—which might ex­plain why A.I. seems to have minted two cul­tures, one dis­mis­sive and the other en­thralled. In our daily lives, A.I. “agents” that can book va­ca­tions or file taxes are a flop, but I have col­leagues who com­pose much of their code us­ing A.I. and some­times run mul­ti­ple cod­ing agents at a time. Models some­times make am­a­teur mis­takes or get caught in inane loops, but, as I’ve learned to use them ef­fec­tively, they have al­lowed me to ac­com­plish in an evening what used to take a month. Not too long ago, I made two iOS apps with­out know­ing how to make an iOS app.“O.K., we’re good on bread crumbs. Now we’re look­ing for a pound of ground beef and a pound of veal.”I once had a boss who said that a job in­ter­view should probe for strengths, not for the ab­sence of weak­nesses. Large lan­guage mod­els have many weak­nesses: they fa­mously hal­lu­ci­nate rea­son­able-sound­ing false­hoods; they can be servile even when you’re wrong; they are fooled by sim­ple puz­zles. But I re­mem­ber a time when the ob­vi­ous strengths of to­day’s A.I. mod­els—flu­ency, flu­id­ity, an abil­ity to “get” what some­one is talk­ing about—were con­sid­ered holy grails. When you ex­pe­ri­ence these strengths first­hand, you won­der: How con­vinc­ing does the il­lu­sion of un­der­stand­ing have to be be­fore you stop call­ing it an il­lu­sion?On a bru­tally hot day this sum­mer, my friend Max met up with his fam­ily at a play­ground. For some rea­son, a sprin­kler for kids was switched off, and Max’s wife had promised every­one that her hus­band would fix it. Confronted by red-faced six- and seven-year-olds, Max en­tered a util­ity shed hop­ing to find a big, fat “On” switch. Instead, he found a maze of an­cient pipes and valves. He was about to give up when, on a whim, he pulled out his phone and fed a photo into ChatGPT-4o, along with a de­scrip­tion of his prob­lem. The A.I. thought for a sec­ond, or maybe did­n’t think, but all the same it said that he was look­ing at a back­flow-pre­ven­ter sys­tem typ­i­cal of ir­ri­ga­tion se­tups. Did he see that yel­low ball valve to­ward the bot­tom? That prob­a­bly con­trolled the flow. Max went for it, and cheers rang out across the play­ground as the wa­ter turned on.Was ChatGPT mind­lessly string­ing words to­gether, or did it un­der­stand the prob­lem? The an­swer could teach us some­thing im­por­tant about un­der­stand­ing it­self. “Neuroscientists have to con­front this hum­bling truth,” Doris Tsao, a neu­ro­science pro­fes­sor at the University of California, Berkeley, told me. “The ad­vances in ma­chine learn­ing have taught us more about the essence of in­tel­li­gence than any­thing that neu­ro­science has dis­cov­ered in the past hun­dred years.” Tsao is best known for de­cod­ing how macaque mon­keys per­ceive faces. Her team learned to pre­dict which neu­rons would fire when a mon­key saw a spe­cific face; even more strik­ingly, given a pat­tern of neu­rons fir­ing, Tsao’s team could ren­der the face. Their work built on re­search into how faces are rep­re­sented in­side A.I. mod­els. These days, her fa­vorite ques­tion to ask peo­ple is “What is the deep­est in­sight you have gained from ChatGPT?” “My own an­swer,” she said, “is that I think it rad­i­cally de­mys­ti­fies think­ing.”The most ba­sic ac­count of how we got here goes some­thing like this. In the nine­teen-eight­ies, a small team of cog­ni­tive psy­chol­o­gists and com­puter sci­en­tists tried to sim­u­late think­ing in a ma­chine. Among the more fa­mous of them were David Rumelhart, Geoffrey Hinton, and James McClelland, who went on to form a re­search group at U.C. San Diego. They saw the brain as a vast net­work in which neu­rons fire in pat­terns, caus­ing other sets of neu­rons to fire, and so on; this dance of pat­terns is think­ing. The brain learns by chang­ing the strength of the con­nec­tions be­tween neu­rons. Crucially, the sci­en­tists mim­ic­ked this process by cre­at­ing an ar­ti­fi­cial neural net­work, and by ap­ply­ing a sim­ple al­go­rithm called gra­di­ent de­scent to in­crease the ac­cu­racy of its pre­dic­tions. (The al­go­rithm could be com­pared to a hiker nav­i­gat­ing from a moun­tain­top to a val­ley; a sim­ple strat­egy for even­tu­ally find­ing one’s way is to in­sure that every step moves down­hill.) The use of such al­go­rithms in large net­works is known as deep learn­ing.Other peo­ple in A.I. were skep­ti­cal that neural net­works were so­phis­ti­cated enough for real-world tasks, but, as the net­works got big­ger, they be­gan to solve pre­vi­ously un­solv­able prob­lems. People would de­vote en­tire dis­ser­ta­tions to de­vel­op­ing tech­niques for dis­tin­guish­ing hand­writ­ten dig­its or for rec­og­niz­ing faces in im­ages; then a deep-learn­ing al­go­rithm would di­gest the un­der­ly­ing data, dis­cover the sub­tleties of the prob­lem, and make those pro­jects seem ob­so­lete. Deep learn­ing soon con­quered speech recog­ni­tion, trans­la­tion, im­age cap­tion­ing, board games, and even the prob­lem of pre­dict­ing how pro­teins will fold.To­day’s lead­ing A.I. mod­els are trained on a large por­tion of the in­ter­net, us­ing a tech­nique called next-to­ken pre­dic­tion. A model learns by mak­ing guesses about what it will read next, then com­par­ing those guesses to what­ever ac­tu­ally ap­pears. Wrong guesses in­spire changes in the con­nec­tion strength be­tween the neu­rons; this is gra­di­ent de­scent. Eventually, the model be­comes so good at pre­dict­ing text that it ap­pears to know things and make sense. So that is some­thing to think about. A group of peo­ple sought the se­cret of how the brain works. As their model grew to­ward a brain-like size, it started do­ing things that were thought to re­quire brain-like in­tel­li­gence. Is it pos­si­ble that they found what they were look­ing for?There is un­der­stand­able re­sis­tance to such a sim­plis­tic and tri­umphant ac­count of A.I. The case against it was well ar­gued by Ted Chiang, who wrote an ar­ti­cle for this mag­a­zine in early 2023 ti­tled “ChatGPT Is a Blurry JPEG of the Web.” He meant it in a more or less de­fla­tion­ary way: that’s all ChatGPT is. You feed the whole in­ter­net to a pro­gram and it re­gur­gi­tates it back to you im­per­fectly, like a copy of a copy of a pho­to­graph—but with just enough fa­cil­ity to fool you into be­liev­ing that the pro­gram is in­tel­li­gent. This spring, a sim­i­lar ar­gu­ment was made in a book, “The AI Con,” by Emily M. Bender, a lin­guist, and Alex Hanna, a so­ci­ol­o­gist. Bender is per­haps best known for de­scrib­ing L.L.M.s as “stochastic par­rots.” “Large lan­guage mod­els do not, can­not, and will not ‘understand’ any­thing at all,” the writer Tyler Austin Harper de­clared in a book re­view in The Atlantic. Models “produce writ­ing not by think­ing but by mak­ing sta­tis­ti­cally in­formed guesses about which lex­i­cal item is likely to fol­low an­other.” Harper but­tressed these tech­ni­cal ar­gu­ments with moral ones. A.I. en­riches the pow­er­ful, con­sumes enough en­ergy to ac­cel­er­ate cli­mate change, and mar­gin­al­izes work­ers. He con­cluded that “the foun­da­tion of the AI in­dus­try is a scam.”But the moral case against A.I. may ul­ti­mately be stronger than the tech­ni­cal one. “The ‘stochastic par­rot’ thing has to be dead at some point,” Samuel J. Gershman, a Harvard cog­ni­tive sci­en­tist who is no A.I. hype man, told me. “Only the most hard­core skep­tics can deny these sys­tems are do­ing things many of us did­n’t think were go­ing to be achieved.” Jonathan Cohen, a cog­ni­tive neu­ro­sci­en­tist at Princeton, em­pha­sized the lim­i­ta­tions of A.I., but ar­gued that, in some cases, L.L.M.s seem to mir­ror one of the largest and most im­por­tant parts of the hu­man brain. “To a first ap­prox­i­ma­tion, your neo­cor­tex is your deep-learn­ing mech­a­nism,” Cohen said. Humans have a much larger neo­cor­tex than other an­i­mals, rel­a­tive to body size, and the species with the largest neo­cor­tices—ele­phants, dol­phins, go­ril­las, chim­panzees, dogs—are among the most in­tel­li­gent.In 2003, the ma­chine-learn­ing re­searcher Eric B. Baum pub­lished a book called “What Is Thought?” (I stum­bled upon it in my col­lege’s li­brary stacks, drawn by the ti­tle.) The gist of Baum’s ar­gu­ment is that un­der­stand­ing is com­pres­sion, and com­pres­sion is un­der­stand­ing. In sta­tis­tics, when you want to make sense of points on a graph, you can use a tech­nique called lin­ear re­gres­sion to draw a “line of best fit” through them. If there’s an un­der­ly­ing reg­u­lar­ity in the data—maybe you’re plot­ting shoe size against height—the line of best fit will ef­fi­ciently ex­press it, pre­dict­ing where new points could fall. The neo­cor­tex can be un­der­stood as dis­till­ing a sea of raw ex­pe­ri­ence—sounds, sights, and other sen­sa­tions—into “lines of best fit,” which it can use to make pre­dic­tions. A baby ex­plor­ing the world tries to guess how a toy will taste or where food will go when it hits the floor. When a pre­dic­tion is wrong, the con­nec­tions be­tween neu­rons are ad­justed. Over time, those con­nec­tions be­gin to cap­ture reg­u­lar­i­ties in the data. They form a com­pressed model of the world.Ar­ti­fi­cial neural net­works com­press ex­pe­ri­ence just like real neural net­works do. One of the best open-source A.I. mod­els, DeepSeek, is ca­pa­ble of writ­ing nov­els, sug­gest­ing med­ical di­ag­noses, and sound­ing like a na­tive speaker in dozens of lan­guages. It was trained us­ing next-to­ken pre­dic­tion on many ter­abytes of data. But when you down­load the model it is one six-hun­dredth of that. A dis­til­la­tion of the in­ter­net, com­pressed to fit on your lap­top. Ted Chiang was right to call an early ver­sion of ChatGPT a blurry JPEG of the web—but, in my view, this is the very rea­son these mod­els have be­come in­creas­ingly in­tel­li­gent. Chiang noted in his piece that, to com­press a text file filled with mil­lions of ex­am­ples of arith­metic, you would­n’t cre­ate a zip file. You’d write a cal­cu­la­tor pro­gram. “The great­est de­gree of com­pres­sion can be achieved by un­der­stand­ing the text,” he wrote. Perhaps L.L.M.s are start­ing to do that.It can seem un­nat­ural, even re­pul­sive, to imag­ine that a com­puter pro­gram ac­tu­ally un­der­stands, ac­tu­ally thinks. We usu­ally con­cep­tu­al­ize think­ing as some­thing con­scious, like a Joycean in­ner mono­logue or the flow of sense mem­o­ries in a Proustian day­dream. Or we might mean rea­son­ing: work­ing through a prob­lem step by step. In our con­ver­sa­tions about A.I., we of­ten con­flate these dif­fer­ent kinds of think­ing, and it makes our judg­ments pat. ChatGPT is ob­vi­ously not think­ing, goes one ar­gu­ment, be­cause it is ob­vi­ously not hav­ing a Proustian reverie; ChatGPT clearly is think­ing, goes an­other, be­cause it can work through logic puz­zles bet­ter than you can.Some­thing more sub­tle is go­ing on. I do not be­lieve that ChatGPT has an in­ner life, and yet it seems to know what it’s talk­ing about. Understanding—having a grasp of what’s go­ing on—is an un­der­ap­pre­ci­ated kind of think­ing, be­cause it’s mostly un­con­scious. Douglas Hofstadter, a pro­fes­sor of cog­ni­tive sci­ence and com­par­a­tive lit­er­a­ture at Indiana University, likes to say that cog­ni­tion is recog­ni­tion. Hofstadter be­came fa­mous for a book about the mind and con­scious­ness called “Gödel, Escher, Bach: An Eternal Golden Braid,” which won a Pulitzer Prize in 1980. Hofstadter’s the­ory, de­vel­oped through decades of re­search, is that “seeing as” is the essence of think­ing. You see one patch of color as a car and an­other as a key chain; you rec­og­nize the let­ter “A” no mat­ter what font it is writ­ten in or how bad the hand­writ­ing might be. Hofstadter ar­gued that the same process un­der­lies more ab­stract kinds of per­cep­tion. When a grand mas­ter ex­am­ines a chess board, years of prac­tice are chan­nelled into a way of see­ing: white’s bishop is weak; that endgame is prob­a­bly a draw. You see an eddy in a river as a sign that it’s dan­ger­ous to cross. You see a meet­ing you’re in as an em­peror-has-no-clothes sit­u­a­tion. My nearly two-year-old son rec­og­nizes that late-morn­ing stroller walks might be an op­por­tu­nity for a crois­sant and makes de­mands ac­cord­ingly. For Hofstadter, that’s in­tel­li­gence in a nut­shell.Hof­s­tadter was one of the orig­i­nal A.I. de­fla­tion­ists, and my own skep­ti­cism was rooted in his. He wrote that most A.I. re­search had lit­tle to do with real think­ing, and when I was in col­lege, in the two-thou­sands, I agreed with him. There were ex­cep­tions. He found the U.C.S.D. group in­ter­est­ing. And he ad­mired the work of a lesser-known Finnish American cog­ni­tive sci­en­tist, Pentti Kanerva, who no­ticed some un­usual prop­er­ties in the math­e­mat­ics of high-di­men­sional spaces. In a high-di­men­sional space, any two ran­dom points may be ex­tremely far apart. But, coun­ter­in­tu­itively, each point also has a large cloud of neigh­bors around it, so you can eas­ily find your way to it if you get “close enough.” That re­minded Kanerva of the way that mem­ory works. In a 1988 book called “Sparse Distributed Memory,” Kanerva ar­gued that thoughts, sen­sa­tions, and rec­ol­lec­tions could be rep­re­sented as coör­di­nates in high-di­men­sional space. The brain seemed like the per­fect piece of hard­ware for stor­ing such things. Every mem­ory has a sort of ad­dress, de­fined by the neu­rons that are ac­tive when you re­call it. New ex­pe­ri­ences cause new sets of neu­rons to fire, rep­re­sent­ing new ad­dresses. Two ad­dresses can be dif­fer­ent in many ways but sim­i­lar in oth­ers; one per­cep­tion or mem­ory trig­gers other mem­o­ries nearby. The scent of hay re­calls a mem­ory of sum­mer camp. The first three notes of Beethoven’s Fifth beget the fourth. A chess po­si­tion that you’ve never seen re­minds you of old games—not all of them, just the ones in the right neigh­bor­hood.“Bye, sweetie—have a day filled with so­cial drama, dras­ti­cally shift­ing friend­ships, and aca­d­e­mic mile­stones, which you’ll de­scribe to me later as ‘fine.’ ”Hofstadter re­al­ized that Kanerva was de­scrib­ing some­thing like a “seeing as” ma­chine. “Pentti Kanerva’s mem­ory model was a rev­e­la­tion for me,” he wrote in a fore­word to Kanerva’s book. “It was the very first piece of re­search I had ever run across that made me feel I could glimpse the dis­tant goal of un­der­stand­ing how the brain works as a whole.” Every kind of think­ing—whether Joycean, Proustian, or log­i­cal—de­pends on the rel­e­vant thing com­ing to mind at the right time. It’s how we fig­ure out what sit­u­a­tion we’re in.Kan­er­va’s book re­ceded from view, and Hofstadter’s own star faded—ex­cept when he oc­ca­sion­ally poked up his head to crit­i­cize a new A.I. sys­tem. In 2018, he wrote of Google Translate and sim­i­lar tech­nolo­gies: “There is still some­thing deeply lack­ing in the ap­proach, which is con­veyed by a sin­gle word: un­der­stand­ing.” But GPT-4, which was re­leased in 2023, pro­duced Hofstadter’s con­ver­sion mo­ment. “I’m mind-bog­gled by some of the things that the sys­tems do,” he told me re­cently. “It would have been in­con­ceiv­able even only ten years ago.” The staunchest de­fla­tion­ist could de­flate no longer. Here was a pro­gram that could trans­late as well as an ex­pert, make analo­gies, ex­tem­po­rize, gen­er­al­ize. Who were we to say that it did­n’t un­der­stand? “They do things that are very much like think­ing,” he said. “You could say they are think­ing, just in a some­what alien way.”L.L.M.s ap­pear to have a “seeing as” ma­chine at their core. They rep­re­sent each word with a se­ries of num­bers de­not­ing its coör­di­nates—its vec­tor—in a high-di­men­sional space. In GPT-4, a word vec­tor has thou­sands of di­men­sions, which de­scribe its shades of sim­i­lar­ity to and dif­fer­ence from every other word. During train­ing, a large lan­guage model tweaks a word’s coör­di­nates when­ever it makes a pre­dic­tion er­ror; words that ap­pear in texts to­gether are nudged closer in space. This pro­duces an in­cred­i­bly dense rep­re­sen­ta­tion of us­ages and mean­ings, in which anal­ogy be­comes a mat­ter of geom­e­try. In a clas­sic ex­am­ple, if you take the word vec­tor for “Paris,” sub­tract “France,” and then add “Italy,” the near­est other vec­tor will be “Rome.” L.L.M.s can “vectorize” an im­age by en­cod­ing what’s in it, its mood, even the ex­pres­sions on peo­ple’s faces, with enough de­tail to re­draw it in a par­tic­u­lar style or to write a para­graph about it. When Max asked ChatGPT to help him out with the sprin­kler at the park, the model was­n’t just spew­ing text. The pho­to­graph of the plumb­ing was com­pressed, along with Max’s prompt, into a vec­tor that cap­tured its most im­por­tant fea­tures. That vec­tor served as an ad­dress for call­ing up nearby words and con­cepts. Those ideas, in turn, called up oth­ers as the model built up a sense of the sit­u­a­tion. It com­posed its re­sponse with those ideas “in mind.”A few months ago, I was read­ing an in­ter­view with an Anthropic re­searcher, Trenton Bricken, who has worked with col­leagues to probe the in­sides of Claude, the com­pa­ny’s se­ries of A.I. mod­els. (Their re­search has not been peer-re­viewed or pub­lished in a sci­en­tific jour­nal.) His team has iden­ti­fied en­sem­bles of ar­ti­fi­cial neu­rons, or “features,” that ac­ti­vate when Claude is about to say one thing or an­other. Features turn out to be like vol­ume knobs for con­cepts; turn them up and the model will talk about lit­tle else. (In a sort of thought-con­trol ex­per­i­ment, the fea­ture rep­re­sent­ing the Golden Gate Bridge was turned up; when one user asked Claude for a choco­late-cake recipe, its sug­gested in­gre­di­ents in­cluded “1/4 cup dry fog” and “1 cup warm sea­wa­ter.”) In the in­ter­view, Bricken men­tioned Google’s Transformer ar­chi­tec­ture, a recipe for con­struct­ing neural net­works that un­der­lies lead­ing A.I. mod­els. (The “T” in ChatGPT stands for “Transformer.”) He ar­gued that the math­e­mat­ics at the heart of the Transformer ar­chi­tec­ture closely ap­prox­i­mated a model pro­posed decades ear­lier—by Pentti Kanerva, in “Sparse Distributed Memory.”Should we be sur­prised by the cor­re­spon­dence be­tween A.I. and our own brains? L.L.M.s are, af­ter all, ar­ti­fi­cial neural net­works that psy­chol­o­gists and neu­ro­sci­en­tists helped de­velop. What’s more sur­pris­ing is that when mod­els prac­ticed some­thing rote—pre­dict­ing words—they be­gan to be­have in such a brain-like way. These days, the fields of neu­ro­science and ar­ti­fi­cial in­tel­li­gence are be­com­ing en­tan­gled; brain ex­perts are us­ing A.I. as a kind of model or­gan­ism. Evelina Fedorenko, a neu­ro­sci­en­tist at M.I.T., has used L.L.M.s to study how brains process lan­guage. “I never thought I would be able to think about these kinds of things in my life­time,” she told me. “I never thought we’d have mod­els that are good enough.”It has be­come com­mon­place to say that A.I. is a black box, but the op­po­site is ar­guably true: a sci­en­tist can probe the ac­tiv­ity of in­di­vid­ual ar­ti­fi­cial neu­rons and even al­ter them. “Having a work­ing sys­tem that in­stan­ti­ates a the­ory of hu­man in­tel­li­gence—it’s the dream of cog­ni­tive neu­ro­science,” Kenneth Norman, a Princeton neu­ro­sci­en­tist, told me. Norman has cre­ated com­puter mod­els of the hip­pocam­pus, the brain re­gion where episodic mem­o­ries are stored, but in the past they were so sim­ple that he could only feed them crude ap­prox­i­ma­tions of what might en­ter a hu­man mind. “Now you can give mem­ory mod­els the ex­act stim­uli you give to a per­son,” he said.The Wright broth­ers stud­ied birds dur­ing their early ef­forts to build an air­plane. They noted that birds take off into the wind, even though a rea­son­able per­son might have as­sumed they’d want the wind at their backs, and that they warp the tips of their wings for bal­ance. These find­ings in­flu­enced their rudi­men­tary glider de­signs. Then they built a six-foot-long wind tun­nel, which al­lowed them to test a set of ar­ti­fi­cial wings un­der pre­cisely con­trolled con­di­tions. Their next round of glider flights was far more suc­cess­ful. Strangely, it was only well af­ter they’d made a work­ing fly­ing ma­chine that it be­came pos­si­ble to un­der­stand ex­actly how the birds do it.A.I. en­ables sci­en­tists to place think­ing it­self in a wind tun­nel. For a pa­per provoca­tively ti­tled “On the Biology of a Large Language Model,” Anthropic re­searchers ob­served Claude re­spond­ing to queries and de­scribed “circuits”—cascades of fea­tures that, to­gether, per­form com­plex com­pu­ta­tions. (Calling up the right mem­o­ries is one step to­ward think­ing; com­bin­ing and ma­nip­u­lat­ing them in cir­cuits is ar­guably an­other.) One long­stand­ing crit­i­cism of L.L.M.s has been that, be­cause they must gen­er­ate one to­ken of their re­sponse at a time, they can’t plan or rea­son. But, when you ask Claude to fin­ish a rhyming cou­plet in a poem, a cir­cuit be­gins con­sid­er­ing the last word of the new line, to in­sure that it will rhyme. It then works back­ward to com­pose the line as a whole. Anthropic re­searchers counted this as ev­i­dence that their mod­els do en­gage in plan­ning. Squint a lit­tle and you might feel, for the first time, that the in­ner work­ings of a mind are in view.You re­ally do have to squint, though. “The worry I have is that peo­ple flipped the bit from ‘I’m re­ally skep­ti­cal of this’ to to­tally drop­ping their shields,” Norman, the Princeton neu­ro­sci­en­tist, told me. “Many things still have to get fig­ured out.” I’m one of the peo­ple that Norman is talk­ing about. (Perhaps I am too eas­ily moved by the seem­ing con­ver­gence of “Sparse Distributed Memory” and an Anthropic model.) In the past year or two, I started to be­lieve what Geoffrey Hinton, who re­cently won a Nobel Prize for his A.I. re­search, told the jour­nal­ist Karen Hao in 2020: “Deep learn­ing is go­ing to be able to do every­thing.” But we have also seen that larger mod­els aren’t al­ways bet­ter mod­els. Curves plot­ting model per­for­mance against size have be­gun flat­ten­ing out. It’s be­com­ing dif­fi­cult to find high-qual­ity data that the mod­els haven’t al­ready di­gested, and com­put­ing power is in­creas­ingly ex­pen­sive. When GPT-5 came out, in August, it was a merely in­cre­men­tal im­prove­ment—and so pro­found a dis­ap­point­ment that it threat­ened to pop the A.I. in­vest­ment bub­ble. The mo­ment de­mands a mid­dle kind of skep­ti­cism: one that takes to­day’s A.I. mod­els se­ri­ously with­out be­liev­ing that there are no hard prob­lems left.Per­haps the most con­se­quen­tial of these prob­lems is how to de­sign a model that learns as ef­fi­ciently as hu­mans do. It is es­ti­mated that GPT-4 was ex­posed to tril­lions of words in train­ing; chil­dren need only a few mil­lion to be­come flu­ent. Cognitive sci­en­tists tell us that a new­born’s brain has cer­tain “inductive bi­ases” that ac­cel­er­ate learn­ing. (Of course, the brain is the re­sult of mil­lions of years of evo­lu­tion—it­self a sort of train­ing data.) For in­stance, hu­man ba­bies have the ex­pec­ta­tion that the world is made of ob­jects, and that other be­ings have be­liefs and in­ten­tions. When Mama says “banana,” an in­fant con­nects that word to the en­tire yel­low ob­ject she’s look­ing at—not just its tip or its peel. Infants per­form lit­tle ex­per­i­ments: Can I eat this? How far can I throw that? They are mo­ti­vated by emo­tions such as de­sire, cu­rios­ity, and frus­tra­tion. Children are al­ways try­ing to do some­thing just be­yond their abil­ity. Their learn­ing is ef­fi­cient be­cause it’s em­bod­ied, adap­tive, de­lib­er­ate, and con­tin­u­ous. Maybe truly un­der­stand­ing the world re­quires par­tic­i­pat­ing in it.An A.I.’s ex­pe­ri­ence, in com­par­i­son, is so im­pov­er­ished that it can’t re­ally be called “experience.” Large lan­guage mod­els are trained on data that is al­ready ex­tra­or­di­nar­ily re­fined. “I think the rea­son they work is that they’re pig­gy­back­ing on lan­guage,” Tsao, the U.C. Berkeley neu­ro­sci­en­tist, told me. Language is like ex­pe­ri­ence pre-chewed; other kinds of data are less dense with mean­ing. “Why is it that we haven’t had a com­pa­ra­ble rev­o­lu­tion in terms of rea­son­ing about video data?” Gershman, the Harvard cog­ni­tive sci­en­tist, asked. “The kinds of vi­sion mod­els that we have still strug­gle with com­mon-sense rea­son­ing about physics.” A re­cent model from DeepMind can gen­er­ate videos in which paints are mixed cor­rectly and mazes are solved—but they also de­pict a glass bounc­ing, in­stead of shat­ter­ing, and ropes de­fy­ing physics by be­ing smooshed into a knot. Ida Momennejad, a cog­ni­tive neu­ro­sci­en­tist who now works for Microsoft Research, has done ex­per­i­ments in which an L.L.M. is given a vir­tual walk-through of a build­ing and then asked ques­tions about routes and short­cuts—spa­tial in­fer­ences that come eas­ily to hu­mans. With all but the most ba­sic se­tups, the A.I.s tend to fail or hal­lu­ci­nate nonex­is­tent paths. “Do they re­ally do plan­ning?” she said. “Not re­ally.”In my con­ver­sa­tions with neu­ro­sci­en­tists, I sensed a con­cern that the A.I. in­dus­try is rac­ing ahead some­what thought­lessly. If the goal is to make ar­ti­fi­cial minds as ca­pa­ble as hu­man minds are, then “we’re not train­ing the sys­tems in the right way,” Brenden M. Lake, a cog­ni­tive sci­en­tist at Princeton, told me. When an A.I. is done train­ing, the neural net­work’s “brain” is frozen. If you tell the model some facts about your­self, it does­n’t rewire its neu­rons. Instead, it uses a crude sub­sti­tute: it writes down a bit of text—“The user has a tod­dler and is study­ing French”—and con­sid­ers that be­fore other in­struc­tions you give. The hu­man brain up­dates it­self con­tin­u­ously, and there’s a beau­ti­ful the­ory about one of its ways of do­ing so: when you sleep, se­lected snap­shots from your episodic mem­ory are re­played for your neo­cor­tex in or­der to train it. Your high-di­men­sional thought space gets dim­pled by the re­played mem­o­ries; you wake up with a slightly new way of see­ing.The A.I. com­mu­nity has be­come so ad­dicted to—and so fi­nan­cially in­vested in—break­neck progress that it some­times pre­tends that ad­vance­ment is in­evitable and there’s no sci­ence left to do. Science has the in­con­ve­nient prop­erty of some­times stalling out. Silicon Valley may call A.I. com­pa­nies “labs,” and some em­ploy­ees there “researchers,” but fun­da­men­tally it has an en­gi­neer­ing cul­ture that does what­ever works. “It’s just so re­mark­able how lit­tle the ma­chine-learn­ing com­mu­nity both­ers look­ing at, let alone re­spects, the his­tory and cog­ni­tive sci­ence that pre­cedes it,” Cohen said.To­day’s A.I. mod­els owe their suc­cess to decades-old dis­cov­er­ies about the brain, but they are still deeply un­like brains. Which dif­fer­ences are in­ci­den­tal and which are fun­da­men­tal? Every group of neu­ro­sci­en­tists has its pet the­ory. These the­o­ries can be put to the test in a way that was­n’t pos­si­ble be­fore. Still, no one ex­pects easy an­swers. The prob­lems that con­tinue to plague A.I. mod­els “are solved by care­fully iden­ti­fy­ing ways in which the mod­els don’t be­have as in­tel­li­gently as we want them to and then ad­dress­ing them,” Norman said. “That is still a hu­man-sci­en­tist-in-the-loop process.”In the nineties, bil­lions of dol­lars poured into the Human Genome Project on the as­sump­tion that se­quenc­ing DNA might solve med­i­cine’s most vex­ing prob­lems: can­cer, hered­i­tary con­di­tions, even ag­ing. It was a time of blus­ter and con­fi­dence—the era of Dolly the cloned sheep and “Jurassic Park”—when biotech was as­cen­dant and the com­men­tariat reck­oned with whether hu­mans should be play­ing God. Biologists soon found that the re­al­ity was more com­pli­cated. We did­n’t cure can­cer or dis­cover the causes of Alzheimer’s or autism. We learned that DNA tells just one part of the story of life. In fact, one could ar­gue that bi­ol­ogy got swept up in a kind of gene fever, fix­at­ing on DNA be­cause we had the means to study and un­der­stand it.Still, no­body would claim that Francis Crick was wrong when, on the day in 1953 that he helped con­firm the struc­ture of DNA, he walked into a Cambridge pub talk­ing about hav­ing dis­cov­ered the se­cret of life. He and his col­leagues did more to de­mys­tify life than al­most any­one, ever. The decades fol­low­ing their dis­cov­ery were among the most pro­duc­tive and ex­cit­ing in the his­tory of sci­ence. DNA be­came a house­hold term; every high schooler learns about the dou­ble he­lix.With A.I., we once again find our­selves in a mo­ment of blus­ter and con­fi­dence. Sam Altman talks about rais­ing half a tril­lion dol­lars to build Stargate, a new clus­ter of A.I. data cen­ters, in the U.S. People dis­cuss the race for su­per­in­tel­li­gence with a grav­i­tas and an ur­gency that can seem un­grounded, even silly. But I sus­pect the rea­son that the Amodeis and Altmans of the world are mak­ing mes­sianic pro­nounce­ments is that they be­lieve that the ba­sic pic­ture of in­tel­li­gence has been worked out; the rest is just de­tails.Even some neu­ro­sci­en­tists be­lieve that a cru­cial thresh­old has been crossed. “I re­ally think it could be the right model for cog­ni­tion,” Uri Hasson, a col­league of Cohen’s, Norman’s, and Lake’s at Princeton, said of neural net­works. This up­sets him as much as it ex­cites him. “I have the op­po­site worry of most peo­ple,” he said. “My worry is not that these mod­els are sim­i­lar to us. It’s that we are sim­i­lar to these mod­els.” If sim­ple train­ing tech­niques can en­able a pro­gram to be­have like a hu­man, maybe hu­mans aren’t as spe­cial as we thought. Could it also mean that A.I. will sur­pass us not only in knowl­edge but also in judg­ment, in­ge­nu­ity, cun­ning—and, as a re­sult, power? To my sur­prise, Hasson told me that he is “worried these days that we might suc­ceed in un­der­stand­ing how the brain works. Pursuing this ques­tion may have been a colos­sal mis­take for hu­man­ity.” He likened A.I. re­searchers to nu­clear sci­en­tists in the nine­teen-thir­ties: “This is the most in­ter­est­ing time in the life of these peo­ple. And, at the same time, they know that what they are work­ing on has grave im­pli­ca­tions for hu­man­ity. But they can­not stop be­cause of the cu­rios­ity to learn.”One of my fa­vorite books by Hofstadter is a nerdy vol­ume called “Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought.” When I was in col­lege, it elec­tri­fied me. The premise was that a ques­tion such as “What is think­ing?” was not merely philo­soph­i­cal but, rather, had a real an­swer. In 1995, when the book was pub­lished, Hofstadter and his re­search group could only ges­ture at what the an­swer might be. Thinking back on the book, I won­dered whether Hofstadter would feel ex­cited that A.I. re­searchers may have at­tained what he had yearned for: a me­chan­i­cal ac­count of the rudi­ments of think­ing. When we spoke, how­ever, he sounded pro­foundly dis­ap­pointed—and fright­ened. Current A.I. re­search “confirms a lot of my ideas, but it also takes away from the beauty of what hu­man­ity is,” he told me. “When I was younger, much younger, I wanted to know what un­der­lay cre­ativ­ity, the mech­a­nisms of cre­ativ­ity. That was a holy grail for me. But now I want it to re­main a mys­tery.” Perhaps the se­crets of think­ing are sim­pler than any­one ex­pected—the kind of thing that a high schooler, or even a ma­chine, could un­der­stand. ♦

We just did some­thing crazy: we com­pletely rewrote our back­end from Python to Node just one week af­ter our launch.

We did this so we can scale. Yes, scale. A week in.

In some ways, it’s a good time right? The code­base is still small and we don’t have too many users.

But on the other hand, it goes com­pletely against the ad­vice given to early-stage star­tups which is to just ship and sell, and worry about scale once you’ve hit prod­uct-mar­ket-fit. “Do things that don’t scale”, as PG put it.

You see, we did­n’t have a mag­i­cal launch week that flooded us with users and force us to scale. And gen­er­ally you can ex­pect that any stack you pick should be able to scale rea­son­ably well for a long time un­til you ac­tu­ally get to the point where you should con­sider chang­ing frame­works or rewrit­ing your back­end in a dif­fer­ent lan­guage (read: Rust).

So why do it?

I’m a big fan of Django. I was in­tro­duced to it at PostHog and it’s be­come my go-to back­end for most pro­jects since. It gets you off the ground re­ally fast, has great tool­ing and ab­strac­tions, and is still flex­i­ble enough to tweak to your needs.

So nat­u­rally, when I started writ­ing our back­end at Skald, I started us off with Django too.

Now, we make a lot of calls to LLM and em­bed­ding APIs at Skald, so we’re gen­er­ally do­ing a lot of net­work I/O that we’d like to be async. Not only that, we of­ten want to fire a lot of re­quests con­cur­rently, such as when need to gen­er­ate vec­tor em­bed­dings for the var­i­ous chunks of a doc­u­ment.

And things quickly got re­ally messy in Django.

I’ll pref­ace this by say­ing that nei­ther of us has a lot of ex­pe­ri­ence writ­ing Python async code (I’ve mostly worked on async-heavy ser­vices in Node) but I think this is partly the point here: it’s re­ally hard and un­in­tu­itive to write solid and per­for­mant Python async code. You need to go deep into the foun­da­tions of every­thing in or­der to be able to do so.

I’m ac­tu­ally re­ally in­ter­ested in spend­ing proper time in be­com­ing more knowl­edge­able with Python async, but in our con­text you a) lose pre­cious time that you need to use to ship as an early-stage startup and b) can shoot your­self in the foot very eas­ily in the process.

Nevertheless, I thought I was to blame. “Bad pro­gram­mer! Bad pro­gram­mer!” was what I was hear­ing in my head as I tried to grasp every­thing. But while more knowl­edge­able folks would cer­tainly have a bet­ter time, we dis­cov­ered that the foun­da­tions of Python async are ac­tu­ally a bit shaky too.

Unlike JavaScript, which had the event loop from the be­gin­ning, and Go, that cre­ated the con­cept of gor­ou­tines (both con­cur­rency mod­els that I quite like and have used in pro­duc­tion), Python async sup­port was patched on later, and that’s where the dif­fi­culty lies.

Two blog posts that cover this re­ally well are “Python has had async for 10 years — why is­n’t it more pop­u­lar?” and “Python con­cur­rency: gevent had it right”, both con­ve­niently pub­lished not long be­fore I started dig­ging into all this.

As for us, we learned a few things:

* Django still does­n’t have full async sup­port. Async in the ORM is not done yet and the col­ored func­tions prob­lem re­ally shines here. You can tech­ni­cally use Django with async, but their docs on this have so many caveats that it should scare any­one.

* You gotta write sync_­to_a­sync and async_­to_­sync every­where.

* All sorts of mod­els have emerged to bring bet­ter async sup­port to dif­fer­ent parts of the Python ecosys­tem, but as they’re not na­tive they have their own caveats. For in­stance, aiofiles brings async API-compatible file op­er­a­tions but uses a thread pool un­der the hood, and Gevent with its green­lets is pretty cool but it lit­er­ally patches the stdlib in or­der to work.

* Due to a lot of async sup­port in Python re­ly­ing on lay­ers that sit on top of the lan­guage rather than be­ing na­tive, you need to be care­ful about the async code you write as it will have dif­fer­ent im­pli­ca­tions de­pend­ing on e.g. the Gunicorn worker type you run (good luck learn­ing much about those from the Gunicorn docs, btw).

Overall, just get­ting an equiv­a­lent of Promise.all to work, while un­der­stand­ing all of its gotchas was not sim­ple at all.

Faced with this, I went into the PostHog code­base.

I worked at PostHog for three years and we had no async in the Django code­base back then but they’re a mas­sive com­pany and they have AI fea­tures now so they must have fig­ured this out!

And what I re­al­ized was that they’re still run­ning WSGI (not ASGI) with Gunicorn Gthread work­ers (where the max con­cur­rent re­quests you’re able to han­dle is usu­ally max 4x CPU cores), thus not get­ting much ben­e­fit from run­ning things async. The code­base also has a lot of utils to make async work prop­erly, like their own im­ple­men­ta­tion of async_­to_­sync. So I guess way they’re han­dling a lot of load is prob­a­bly just hor­i­zon­tal scal­ing.

There’s sim­ply no great way to run async in Django.

We es­sen­tially con­cluded that Django was go­ing to hurt us re­ally soon, not just when we started to have a lot of load.

Without too many users we’d al­ready need to start run­ning mul­ti­ple ma­chines in or­der to not have ter­ri­ble la­tency, plus we’d be writ­ing clunky code that would be hard to main­tain.

We could of course just “do things that don’t scale” for now and just solve the prob­lem with money (or AWS cred­its), but it did­n’t feel right. And be­ing so early would make the mi­gra­tion to an­other frame­work much eas­ier.

At this point, some peo­ple are prob­a­bly scream­ing at their screens go­ing: “just use FastAPI!” — and we did in­deed con­sider it.

FastAPI does have proper async sup­port and is quite a loved frame­work said to be per­for­mant. And if you want an ORM with it you could use SQLAlchemy which also sup­ports async.

Migrating to FastAPI would have prob­a­bly saved us a day or two (our mi­gra­tion took 3 days) due to be­ing able to reuse a lot of code with­out trans­lat­ing it, but at this point we weren’t feel­ing great about the Python async ecosys­tem over­all, and we had ac­tu­ally al­ready writ­ten our back­ground worker ser­vice in Node, so we thought it would be a good op­por­tu­nity to go all-in on one ecosys­tem.

And so mi­grate to Node we did. We took a lit­tle time pick­ing the frame­work + ORM combo but set­tled on Express + MikroORM.

Yeah sure Express is old but it’s bat­tle-tested and feels fa­mil­iar. Coming over to the JS event loop was the main point of all this any­way.

Our ini­tial bench­marks show we’ve gained ~3x through­put out of the box and that’s just with us run­ning what is mostly se­quen­tial code in an async con­text. Being over on Node now, we’re plan­ning on do­ing a lot con­cur­rent pro­cess­ing when chunk­ing, em­bed­ding, rerank­ing, and so on. This means this change should have an even greater pay­off over time.

Losing Django hurts, and we’ve al­ready found our­selves build­ing a lot more mid­dle­ware and util­i­ties our­selves on the Express side. Adonis ex­ists, which is a more fully-fea­tured Node frame­work, but mov­ing to a whole new ecosys­tem felt like more work to us than just us­ing some­thing min­i­mal.

What I’m miss­ing the most is the ORM, which in my opin­ion is re­ally er­gonomic. And while you al­ways have to be care­ful with ORMs when look­ing to ex­tract the best pos­si­ble per­for­mance, the Django ORM does do some nice things un­der the hood in or­der to make it per­for­mant enough to write queries in Python, and I learned a bit more about this when mi­grat­ing our Django mod­els over to MikroORM en­ti­ties.

MikroORM was a con­so­la­tion prize in this whole mi­gra­tion. I still much pre­fer the Django ORM but at the same time dif­fer­ent ecosys­tems call for dif­fer­ent tool­ing.

I’d never used it be­fore and was pos­i­tively sur­prised to find Django-like lazy load­ing, a mi­gra­tions setup that felt much bet­ter than Prisma’s, as well as a rea­son­ably er­gonomic API (once you man­u­ally set up the foun­da­tions right).

Overall, we’re early into this change, but cur­rently happy to have picked MikroORM over the in­cum­bent Prisma.

I think this is pretty self-ex­plana­tory. While most tools for build­ing RAGs and agents have Python and TypeScript SDKs, Python still takes pri­or­ity, and we’re just talk­ing about API wrap­pers here.

Once you want to ac­tu­ally get into ML stuff your­self, there’s just no com­pe­ti­tion. I sus­pect that as we get more so­phis­ti­cated we’ll end up hav­ing a Python ser­vice, but for now we’re ok.

We’d al­ways re­al­ized that mi­grat­ing to Node would mean we’d have two Node ser­vices in­stead of a Python one and a Node one, but it did­n’t oc­cur to us un­til a day in that we could ac­tu­ally merge the code­bases and that that would be ex­tremely help­ful.

There was a lot of du­pli­cate logic across the Node worker and the Django server, and now we’ve uni­fied the Express server and back­ground worker into one code­base, which feels so much bet­ter. They can both use the ORM now (previously the worker was run­ning raw SQL) and share a bunch of utils.

This is not a pytest vs jest thing, it’s just that in or­der to make sure every­thing was work­ing as ex­pected af­ter mi­grat­ing, we just wrote a ton more tests. This and some refac­tor­ing were wel­come side ben­e­fits.

I think it’s about time to wrap this post up, but here are some quick notes about the ac­tual mi­gra­tion process.

* It took us three days.

* We barely used AI code gen­er­a­tion at all un­til the fi­nal bits — it felt im­por­tant to us to un­der­stand the foun­da­tions of our new setup re­ally well, par­tic­u­larly the in­ner work­ings of the new ORM. Once we had the foun­da­tions of every­thing down Claude Code was quite help­ful in gen­er­at­ing code for some less im­por­tant end­points, and also helped us in scan­ning the code­base for is­sues.

* We al­most quit mul­ti­ple times. We were get­ting cus­tomer re­quests for new fea­tures and had some bugs in the Django code and it felt like we were wast­ing time mi­grat­ing in­stead of serv­ing cus­tomers.

Honestly, we’re quite happy with our de­ci­sion and would 100% do it again. Not only will this pay off in the long term but it’s al­ready pay­ing off to­day.

We learned a lot of stuff in the process too, and if the whole point of this whole post is that some­one comes to tell me that we’re dumb and we should just have done X or Y, or comes to teach me about how Python async works, then that will hon­estly be great. For my part, I gladly rec­og­nize my in­ex­pe­ri­ence with Python async and if I can learn more about it, that’s a win.

And if you’re in­ter­ested in ac­tu­ally see­ing the code, check out the fol­low­ing PRs:

Skald is an MIT-licensed RAG API plat­form, so if you have any thoughts or con­cerns, you can come yell at us on GitHub, or open a PR to rewrite the back­end to your frame­work of choice :D