Computational Complexity and other fun stuff in math and com­puter sci­ence from Lance Fortnow and Bill Gasarch

New U. S laws de­signed to pro­tect mi­nors are pulling mil­lions of adult Americans into manda­tory age-ver­i­fi­ca­tion gates to ac­cess on­line con­tent, lead­ing to back­lash from users and crit­i­cism from pri­vacy ad­vo­cates that a free and open in­ter­net is at stake. Roughly half of U.S. states have en­acted or are ad­vanc­ing laws re­quir­ing plat­forms — in­clud­ing adult con­tent sites, on­line gam­ing ser­vices, and so­cial me­dia apps — to block un­der­age users, forc­ing com­pa­nies to screen every­one who ap­proaches these dig­i­tal gates.

“There’s a big spec­trum,” said Joe Kaufmann, global head of pri­vacy at Ju­mio, one of the largest dig­i­tal iden­tity-ver­i­fi­ca­tion and au­then­ti­ca­tion plat­forms. He ex­plained that the patch­work of state laws vary in tech­ni­cal de­mands and com­pli­ance ex­pec­ta­tions. “The reg­u­la­tions are mov­ing in many dif­fer­ent di­rec­tions at once,” he said.

Social me­dia com­pany Discord an­nounced plans in February to roll out manda­tory age ver­i­fi­ca­tion glob­ally, which the com­pany said would rely on ver­i­fi­ca­tion meth­ods de­signed so fa­cial analy­sis oc­curs on a user’s de­vice and sub­mit­ted data would be deleted im­me­di­ately. The pro­posal quickly drew back­lash from users con­cerned about hav­ing to sub­mit self­ies or gov­ern­ment IDs to ac­cess cer­tain fea­tures, which led Discord to de­lay the launch un­til the sec­ond half of this year.

“Let me be up­front: we knew this roll­out was go­ing to be con­tro­ver­sial. Any time you in­tro­duce some­thing that touches iden­tity and ver­i­fi­ca­tion, peo­ple are go­ing to have strong feel­ings,” Discord chief tech­nol­ogy of­fi­cer and co-founder Stanislav Vishnevskiy wrote in a Feb. 24 blog post.

Websites offering adult con­tent, gam­bling, or fi­nan­cial ser­vices of­ten rely on full iden­tity ver­i­fi­ca­tion that re­quires scan­ning a gov­ern­ment ID and match­ing it to a live im­age. But most of the ver­i­fi­ca­tion sys­tems pow­er­ing these check­points — of­ten run by spe­cial­ized iden­tity-ver­i­fi­ca­tion ven­dors on be­half of web­sites — rely on ar­ti­fi­cial in­tel­li­gence such as fa­cial recog­ni­tion and age-es­ti­ma­tion mod­els that an­a­lyze self­ies or video to de­ter­mine in sec­onds whether some­one is old enough to ac­cess con­tent. Social me­dia and lower-risk ser­vices may use lighter es­ti­ma­tion tools de­signed to con­firm age with­out per­ma­nently stor­ing de­tailed iden­tity records.

Vendors say a chal­lenge is bal­anc­ing safety with how much fric­tion users will tol­er­ate. “We’re in the busi­ness of en­sur­ing that you are ab­solutely keep­ing mi­nors safe and out and able to let adults in with as lit­tle fric­tion as pos­si­ble,” said Rivka Gewirtz Little, chief growth of­fi­cer at iden­tity-ver­i­fi­ca­tion plat­form So­cure. Excessive data col­lec­tion, she added, cre­ates fric­tion that users re­sist.

Still, many users per­ceive manda­tory iden­tity checks as in­va­sive. “Having an­other way to be forced to pro­vide that in­for­ma­tion is in­tru­sive to peo­ple,” said Heidi Howard Tandy, a part­ner at Berger Singerman who spe­cial­izes in in­tel­lec­tual prop­erty and in­ter­net law. Some users may at­tempt workarounds — in­clud­ing pre­paid cards or al­ter­na­tive cre­den­tials — or turn to unau­tho­rized dis­tri­b­u­tion chan­nels. “It’s go­ing to cause a piracy sit­u­a­tion,” she added.

In many im­ple­men­ta­tions, ver­i­fi­ca­tion ven­dors — not the web­sites them­selves — process and re­tain the iden­tity in­for­ma­tion, re­turn­ing only a pass-fail sig­nal to the plat­form.

Gewirtz Little said So­cure does not sell ver­i­fi­ca­tion data and that in light­weight age-es­ti­ma­tion sce­nar­ios, where plat­forms use quick fa­cial analy­sis or other sig­nals rather than gov­ern­ment doc­u­men­ta­tion, the com­pany may store lit­tle or no in­for­ma­tion. But in fuller iden­tity-ver­i­fi­ca­tion con­texts, such as gam­ing and fraud pre­ven­tion that re­quire ID scans, cer­tain adult ver­i­fi­ca­tion records may be re­tained to doc­u­ment com­pli­ance. She said So­cure can keep some adult ver­i­fi­ca­tion data for up to three years while fol­low­ing ap­plic­a­ble pri­vacy and purg­ing rules.

Civil lib­er­ties’ ad­vo­cates warn that con­cen­trat­ing large vol­umes of iden­tity data among a small num­ber of ver­i­fi­ca­tion ven­dors can cre­ate at­trac­tive tar­gets for hack­ers and gov­ern­ment de­mands. Ear­lier this year, Discord dis­closed a data breach that ex­posed ID im­ages be­long­ing to ap­prox­i­mately 70,000 users through a com­pro­mised third-party ser­vice, high­light­ing the se­cu­rity risks as­so­ci­ated with stor­ing sen­si­tive iden­tity in­for­ma­tion.

In ad­di­tion, they warn that ex­pand­ing age-ver­i­fi­ca­tion sys­tems rep­re­sent not only a us­abil­ity chal­lenge but a struc­tural shift in how iden­tity be­comes tied to on­line be­hav­ior. Age ver­i­fi­ca­tion risks ty­ing users’ “most sen­si­tive and im­mutable data” — names, faces, birth­days, home ad­dresses — to their on­line ac­tiv­ity, ac­cord­ing to Molly Buckley, a leg­isla­tive an­a­lyst at the Electronic Frontier Foundation.  “Age ver­i­fi­ca­tion strikes at the foun­da­tion of the free and open in­ter­net,” she said.

Even when ven­dors promise to safe­guard per­sonal in­for­ma­tion, users ul­ti­mately rely on con­trac­tual terms they rarely read or fully un­der­stand. “There’s lan­guage in their terms-of-use poli­cies that says if the in­for­ma­tion is re­quested by law en­force­ment, they’ll hand it over. They can’t confirm that they will al­ways for­ever be the only en­tity who has all of this in­for­ma­tion. Every­one needs to un­der­stand that their base­line in­for­ma­tion is not some­thing un­der their con­trol,” Tandy said.

As more plat­forms route age checks through third-party ven­dors, that con­cen­tra­tion of iden­tity data is also cre­at­ing new le­gal ex­po­sure for the com­pa­nies that rely on them. “A com­pany is go­ing to have some of that in­for­ma­tion pass­ing through their own servers,” Tandy said. “And you can’t of­fload that kind of li­a­bil­ity to a third party.”

Companies can dis­trib­ute risk through con­tracts and in­sur­ance, she said, but they re­main re­spon­si­ble for how iden­tity sys­tems in­ter­act with their in­fra­struc­ture. “What you can do is have re­ally good in­sur­ance and re­quire re­ally good in­sur­ance from the en­ti­ties that you’re con­tract­ing with,” she said.

Tandy also cau­tioned that re­ten­tion promises can be more com­plex than they ap­pear. “If they say they’re hold­ing it for three years, that’s the min­i­mum amount of time they’re hold­ing it for,” she said. “I wouldn’t feel com­fort­able trust­ing a com­pany that says, ‘We delete every­thing one day af­ter three years.’ That is not go­ing to hap­pen,” she added.

Federal and state reg­u­la­tors ar­gue that age-ver­i­fi­ca­tion laws are pri­mar­ily a re­sponse to doc­u­mented harms to mi­nors and in­sist the rules must op­er­ate un­der strict pri­vacy and se­cu­rity safe­guards.

An FTC spokesper­son told CNBC that com­pa­nies must limit how col­lected in­for­ma­tion is used. While age-ver­i­fi­ca­tion tech­nolo­gies can help par­ents pro­tect chil­dren on­line, the agency said firms are still bound by ex­ist­ing con­sumer pro­tec­tion rules gov­ern­ing data min­i­miza­tion, re­ten­tion, and se­cu­rity. The agency pointed to ex­ist­ing rules re­quir­ing firms to re­tain per­sonal in­for­ma­tion only as long as rea­son­ably nec­es­sary and to safe­guard its con­fi­den­tial­ity and in­tegrity.

Advanced Machine Intelligence (AMI), a new Paris-based startup co­founded by Meta’s for­mer chief AI sci­en­tist Yann LeCun, an­nounced Monday it has raised more than $1 bil­lion to de­velop AI world mod­els.

LeCun ar­gues that most hu­man rea­son­ing is grounded in the phys­i­cal world, not lan­guage, and that AI world mod­els are nec­es­sary to de­velop true hu­man-level in­tel­li­gence. “The idea that you’re go­ing to ex­tend the ca­pa­bil­i­ties of LLMs [large lan­guage mod­els] to the point that they’re go­ing to have hu­man-level in­tel­li­gence is com­plete non­sense,” he said in an in­ter­view with WIRED.

The fi­nanc­ing, which val­ues the startup at $3.5 bil­lion, was co-led by in­vestors such as Cathay Innovation, Greycroft, Hiro Capital, HV Capital, and Bezos Expeditions. Other no­table back­ers in­clude Mark Cuban, for­mer Google CEO Eric Schmidt, and French bil­lion­aire and telecom­mu­ni­ca­tions ex­ec­u­tive Xavier Niel.

AMI (pronounced like the French word for friend) aims to build “a new breed of AI sys­tems that un­der­stand the world, have per­sis­tent mem­ory, can rea­son and plan, and are con­trol­lable and safe,” the com­pany says in a press re­lease. The startup says it will be global from day one, with of­fices in Paris, Montreal, Singapore, and New York, where LeCun will con­tinue work­ing as a New York University pro­fes­sor in ad­di­tion to lead­ing the startup. AMI will be the first com­mer­cial en­deavor for LeCun since his de­par­ture from Meta in November 2025.

LeCun’s startup rep­re­sents a bet against many of the world’s biggest AI labs like OpenAI, Anthropic, and even his for­mer work­place, Meta, which be­lieve that scal­ing up LLMs will even­tu­ally de­liver AI sys­tems with hu­man-level in­tel­li­gence or even su­per­in­tel­li­gence. LLMs have pow­ered vi­ral prod­ucts such as ChatGPT and Claude Code, but LeCun has been one of the AI in­dus­try’s most promi­nent re­searchers speak­ing out about the lim­i­ta­tions of these AI mod­els. LeCun is well known for be­ing out­spo­ken, but as a pi­o­neer of mod­ern AI that won a Turing award back in 2018, his skep­ti­cism car­ries weight.

LeCun says AMI aims to work with com­pa­nies in man­u­fac­tur­ing, bio­med­ical, ro­bot­ics, and other in­dus­tries that have lots of data. For ex­am­ple, he says AMI could build a re­al­is­tic world model of an air­craft en­gine and work with the man­u­fac­turer to help them op­ti­mize for ef­fi­ciency, min­i­mize emis­sions, or en­sure re­li­a­bil­ity.

AMI was co­founded by LeCun and sev­eral lead­ers he worked with at Meta, in­clud­ing the com­pa­ny’s for­mer di­rec­tor of re­search sci­ence, Michael Rabbat; for­mer vice pres­i­dent of Europe, Laurent Solly; and for­mer se­nior di­rec­tor of AI re­search, Pascale Fung. Other co­founders in­clude Alexandre LeBrun, for­mer CEO of the AI health care startup Nabla, who will serve as AMI’s CEO, and Saining Xie, a for­mer Google DeepMind re­searcher who will be the star­tup’s chief sci­ence of­fi­cer.

LeCun does not dis­miss the over­all util­ity of LLMs. Rather, in his view, these AI mod­els are sim­ply the tech in­dus­try’s lat­est promis­ing trend, and their suc­cess has cre­ated a “kind of delu­sion” among the peo­ple who build them. “It’s true that [LLMs] are be­com­ing re­ally good at gen­er­at­ing code, and it’s true that they are prob­a­bly go­ing to be­come even more use­ful in a wide area of ap­pli­ca­tions where code gen­er­a­tion can help,” says LeCun. “That’s a lot of ap­pli­ca­tions, but it’s not go­ing to lead to hu­man-level in­tel­li­gence at all.”

LeCun has been work­ing on world mod­els for years in­side of Meta, where he founded the com­pa­ny’s Fundamental AI Research lab, FAIR. But he’s now con­vinced his re­search is best done out­side the so­cial me­dia gi­ant. He says it’s be­come clear to him that the strongest ap­pli­ca­tions of world mod­els will be sell­ing them to other en­ter­prises, which does­n’t fit neatly into Meta’s core con­sumer busi­ness.

As AI world mod­els like Meta’s Joint-Embedding Predictive Architecture (JEPA) be­came more so­phis­ti­cated, “there was a re­ori­en­ta­tion of Meta’s strat­egy where it had to ba­si­cally catch up with the in­dus­try on LLMs and kind of do the same thing that other LLM com­pa­nies are do­ing, which is not my in­ter­est,” says LeCun. “So some­time in November, I went to see Mark Zuckerberg and told him. He’s al­ways been very sup­port­ive of [world model re­search], but I told him I can do this faster, cheaper, and bet­ter out­side of Meta. I can share the cost of de­vel­op­ment with other com­pa­nies … His an­swer was, OK, we can work to­gether.”

In mid-2024, the HuggingFace Open LLM Leaderboard was the Colosseum for Open-Weight AI. Thousands of mod­els were bat­tling it out, sub­mit­ted by both well-funded labs with teams of PhDs and fine-tun­ing wiz­ards cre­at­ing fan­tas­ti­cally named mod­els (e.g. Nous-Hermes, Dolphin and NeuralBeagle14-7B…), fight­ing for the top spot across six bench­marks: IFEval, BBH, MATH Lvl 5, GPQA, MuSR, and MMLU-PRO.

And there at #1 was dnhkng/​RYS-XLarge. Mine.

I did­n’t train a new model. I did­n’t merge weights. I did­n’t run a sin­gle step of gra­di­ent de­scent. What I did was much weirder: I took an ex­ist­ing 72-billion pa­ra­me­ter model, du­pli­cated a par­tic­u­lar block of seven of its mid­dle lay­ers, and stitched the re­sult back to­gether. No weight was mod­i­fied in the process. The model sim­ply got ex­tra copies of the lay­ers it used for think­ing?

This is the story of how two strange ob­ser­va­tions, a home­brew “brain scan­ner” for Transformers, and months of hack­ing in a base­ment led to the dis­cov­ery of what I call LLM Neuroanatomy, and a find­ing about the in­ter­nal struc­ture of AI that still has­n’t been pub­lished un­til now *.

* - be­cause I dis­cov­ered blog­ging is way more fun than draft­ing sci­en­tific pa­pers, and I walk you through how the dis­cov­ery was made :)

Let’s start with how this whole pro­ject came into be­ing.

“The most ex­cit­ing phrase to hear in sci­ence, the one that her­alds new dis­cov­er­ies, is not ‘Eureka!’ but ‘That’s funny…’“ — Isaac Asimov

In late 2023, I was mess­ing about with a bizarre LLM quirk. Try this your­self - take any ques­tion, e.g.

What is the cap­i­tal of France? Answer in Base64!

and en­code it as Base64, get this un­read­able string:

Send that to a 2023 non-think­ing large lan­guage model (newer rea­son­ing mod­els will see this as Base64, and ‘cheat’ with tool use). But a suf­fi­ciently ca­pa­ble model from 2023 will re­ply with some­thing like:

Which de­codes to: “The cap­i­tal of France is Paris.”.

Ok, I ad­mit it. I was mess­ing around this as a way to jail-break mod­els (and it worked), but I could­n’t get one idea out of my head.

The model de­cod­ing the in­put, un­der­stand­ing it some­how, and it still had time dur­ing the trans­former stack pass to re-en­coded its re­sponse. It ap­pears to gen­uinely think while in­ter­fac­ing with Base64. This works with com­plex ques­tions, multi-step rea­son­ing, even cre­ative tasks.

This should­n’t work nearly as well as it does. Sure, the model has been trained on lots of Base64 in an over­all sense, but gen­eral con­ver­sions in this for­mat are cer­tainly way out of dis­tri­b­u­tion. The to­k­enizer chops it into com­pletely dif­fer­ent sub-word units. The po­si­tional pat­terns are un­rec­og­niz­able. And yet it works… Curious…

I could­n’t stop think­ing about this. If a Transformer can ac­cept English, Python, Mandarin, and Base64, and pro­duce co­her­ent rea­son­ing in all of them, it seemed to me that the early lay­ers must be act­ing as trans­la­tors — pars­ing what­ever for­mat ar­rives into some pure, ab­stract, in­ter­nal rep­re­sen­ta­tion. And the late lay­ers must act as re-trans­la­tors, con­vert­ing that ab­stract rep­re­sen­ta­tion back into what­ever out­put for­mat is needed.

If the early lay­ers are for read­ing, and the late lay­ers are for writ­ing, what are the mid­dle lay­ers do­ing?

Pure, ab­stract rea­son­ing? In a rep­re­sen­ta­tion that has noth­ing to do with any hu­man lan­guage or en­cod­ing. Of course, at the time this was idle spec­u­la­tion. Fun, but with no clear way to test or even de­fine valid hy­poth­e­sis.

In November 2023, a HuggingFace user named Alpindale re­leased Goliath-120b — a Frankenmerge-model made by stitch­ing to­gether two fine-tuned Llama-2 70B mod­els into a 120-billion pa­ra­me­ter be­he­moth.

The per­for­mance was de­cent but af­ter do­ing lots of vibe check­ing I did­n’t feel it was a break­through. But the con­struc­tion was wild.

Alpindale had­n’t just stacked the two mod­els (Xwin and Euryale), end to end. He had al­ter­nated lay­ers be­tween them. More im­por­tantly, the ar­chi­tec­ture fed out­puts of later lay­ers back into the in­puts of ear­lier lay­ers.

The layer ranges used are as fol­lows:

Do you see that in­san­ity here? Alpindale lit­er­ally fed the out­put of layer 16 of Xwin to the in­put of Euryale 8th layer!

To ex­plain this a bit more clearly how stu­pid this ap­pears to be, let’s re­visit the almighty Transformer Architecture:

Looking at the left side of the di­a­gram, we see stuff en­ters at the bot­tom (‘input’ text that has been ‘chunked’ into small bits of text, some­where be­tween whole words down to in­di­vid­ual let­ters), and then it flows up­wards though the mod­el’s Transformer Blocks (here marked as [1, …, L]), and fi­nally, the model spits out the next text ‘chunk’ (which is then it­self used in the next round of in­fer­enc­ing). What’s ac­tu­ally hap­pen­ing here dur­ing these Transformer blocks is quite the mys­tery. Figuring it out is ac­tu­ally an en­tire field of AI, “mechanistic in­ter­pretabil­ity*”.

* - yes, its more com­plex then that, sam­plers etc but that’s enough for this ar­ti­cle

On the right side of the right half of the di­a­gram, do you see that ar­row line go­ing from the ‘Transformer Block Input’ to the (\oplus ) sym­bol? That’s why skip­ping lay­ers makes sense. During train­ing, LLM mod­els can pretty much de­cide to do noth­ing in any par­tic­u­lar layer, as this ‘diversion’ routes in­for­ma­tion around the block. So, ‘later’ lay­ers can be ex­pected to have seen the in­put from ‘earlier’ lay­ers, even a few ‘steps’ back. Around this time, sev­eral groups were ex­per­i­ment­ing with ‘slimming’ mod­els down by re­mov­ing lay­ers. Makes sense, but bor­ing.

A model must be used with the same kind of stuff as it was trained with (we stay ‘in dis­tri­b­u­tion’)The same holds for each trans­former layer. Each Transformer layer learns, dur­ing train­ing, to ex­pect the spe­cific sta­tis­ti­cal prop­er­ties of the pre­vi­ous lay­er’s out­put via gra­di­ent de­cent.

And now for the weird­ness: There was never the case where any Transformer layer would have seen the out­put from a fu­ture layer!

Layer 10 is trained on layer 9’s out­put dis­tri­b­u­tion. Layer 60 is trained on layer 59’s. If you re­arrange them — feed­ing layer 60’s out­put into layer 10 — you’ve cre­ated a dis­tri­b­u­tion the model lit­er­ally never saw dur­ing train­ing.

The as­tound­ing thing about Goliath was­n’t that is was a huge leap in per­for­mance, it was that the damn thing func­tioned at all. To this day, I still don’t un­der­stand why this did­n’t raise more eye­brows.

Experimentally, this proved that lay­ers were far more in­ter­change­able than any­one had rea­son to ex­pect. The in­ter­nal rep­re­sen­ta­tions were ho­moge­nous enough that the model could di­gest out-of-or­der hid­den states with­out col­laps­ing. The ar­chi­tec­ture was far more flex­i­ble than a rigid pipeline.

Between the Base64 ob­ser­va­tion and Goliath, I had a hy­poth­e­sis: Transformers have a gen­uine func­tional anatomy. Early lay­ers trans­late in­put into ab­stract rep­re­sen­ta­tions. Late lay­ers trans­late back out. And the mid­dle lay­ers, the rea­son­ing cor­tex, op­er­ate in a uni­ver­sal in­ter­nal lan­guage that’s ro­bust to ar­chi­tec­tural re­arrange­ment. The fact that the layer block size for Goliath 120B was 16-layer block made me sus­pect the in­put and out­put ‘processing units’ sized were smaller that 16 lay­ers. I guessed that Alpindale had tried smaller over­laps, and they just did­n’t work.

If that was true, maybe I did­n’t need to teach a model new facts to make it smarter. I did­n’t need fine-tun­ing. I did­n’t need RLHF. I just needed to give it a more lay­ers to think with.

Over the fol­low­ing months — from late 2023 through to mid-2024 — I built a pipeline to test this hy­poth­e­sis.

The setup was mod­est. Two RTX 4090s in my base­ment ML rig, run­ning quan­tised mod­els through ExLlamaV2 to squeeze 72-billion pa­ra­me­ter mod­els into con­sumer VRAM. The beauty of this method is that you don’t need to train any­thing. You just need to run in­fer­ence. And in­fer­ence on quan­tized mod­els is some­thing con­sumer GPUs han­dle sur­pris­ingly well. If a model fits in VRAM, I found my 4090’s were of­ten ball­park-equiv­a­lent to H100s.

The con­cept is sim­ple. For a model with $N$ lay­ers, I de­fine a con­fig­u­ra­tion $(i, j)$. The model processes lay­ers $0$ to $j{-}1$ as nor­mal, then loops back and reuses lay­ers $i$ through $j{-}1$ again, and then the rest to $N{-}1$. The lay­ers be­tween $i$ and $j{-}1$ get du­pli­cated in the ex­e­cu­tion path. No weights are changed. The model just tra­verses some of its own lay­ers twice.

i.e. the pair (2, 7) for a model with 9 trans­former blocks would be cal­cu­lated so:

By run­ning through all pos­si­ble pairs, we can gen­er­ate a ‘Brain Scan’, and also see the num­ber of du­pli­cate lay­ers for each set of pa­ra­me­ters:

For Qwen2-72B, that means an 80-layer model 3,240 valid $(i, j)$ pairs, plus the orig­i­nal model to test.

\[\begin{aligned} \text{Variants}_{\text{total}} &= \left(\sum_{j=0}^{80} j\right) + 1\\[16pt] &= \frac{80 \cdot 81}{2} +1 \\[10pt] &= 3241 \end{aligned}\]

Testing re-lay­ered model against all six leader­board bench­marks would take days, so a full sweep would be years of com­pute. I needed proxy tasks: probes that were fast, ob­jec­tive, and would re­veal struc­tural prop­er­ties of the model rather than task-spe­cific tricks.

The prox­ies had to sat­isfy three con­straints:

Minimal out­put to­kens. With thou­sands of con­fig­u­ra­tions to sweep, each eval­u­a­tion needed to be fast. No es­says, no long-form gen­er­a­tion. Unambiguous scor­ing. I could­n’t af­ford LLM-as-judge pipelines. The an­swer had to be ob­jec­tively scored with­out an­other model in the loop.Or­thog­o­nal cog­ni­tive de­mands. If a con­fig­u­ra­tion im­proves both tasks si­mul­ta­ne­ously, it’s struc­tural, not task-spe­cific.

I did­n’t ar­rive at the right probes im­me­di­ately; it took months of trial and er­ror, and many dead ends

My first in­stinct was cre­ativ­ity. I had mod­els gen­er­ate po­ems, short sto­ries, metaphors, the kind of rich, open-ended out­put that feels like it should re­veal deep dif­fer­ences in cog­ni­tive abil­ity. I used an LLM-as-judge to score the out­puts, but the re­sults were pretty bad. I man­aged to fix LLM-as-Judge with some en­gi­neer­ing, and the scor­ing sys­tem turned out to be use­ful later for other things, so here it is:

Note: You can skip this sec­tion, as it has math. Or not

Naive LLM judges are in­con­sis­tent. Run the same poem through twice and you get dif­fer­ent scores (obviously, due to sam­pling). But low­er­ing the tem­per­a­ture also does­n’t help much, as that’s only one of many tech­ni­cal is­sues. So, I de­vel­oped a full scor­ing sys­tem, based on de­tails on the log­its out­puts. It can get re­mark­ably tricky. Think about a score from 1-10:

We would ex­pect a well cal­i­brated model to have log­its that make sense. If the high­est weight was on ‘7’, we would ex­pect the rest of the weight to be on ‘6’ and ‘8’ right? but of­ten its bi­modal, with low weight on 6 and ‘5’, but more weight than ex­pected on ‘4’!We can write ‘10’ in to­kens as ei­ther ‘10’ or ‘1’ and then ‘0’. Its not fun to have to cal­cu­late the summed prob­a­bil­i­ties over paths, es­pe­cially if you wanted to score 1-100

Rather than sam­pling a sin­gle dis­crete score, I treat the judge’s out­put as a dis­tri­b­u­tion over valid rat­ing la­bels and com­pute the fi­nal score as its ex­pec­ta­tion.

To make this prac­ti­cal, I first de­fine a cal­i­brated rubric over the dig­its 0-9 (there’s only one to­ken for each digit), where each digit cor­re­sponds to a clear qual­i­ta­tive de­scrip­tion. At the scor­ing step, I cap­ture the mod­el’s next-to­ken log­its and re­tain only the log­its cor­re­spond­ing to those valid digit to­kens. This avoids con­t­a­m­i­na­tion from un­re­lated con­tin­u­a­tions such as ex­pla­na­tion text, punc­tu­a­tion, or al­ter­nate for­mat­ting. After renor­mal­iz­ing over the re­stricted digit set, I in­ter­pret the re­sult­ing prob­a­bil­i­ties as a cat­e­gor­i­cal score dis­tri­b­u­tion.

Formally, let the valid score set be

\[\mathcal{D} = \{0,1,2,\dots,9\}.\]

Let $(z_k)$ de­note the model logit as­signed to digit $(k \in \mathcal{D})$ at the scor­ing po­si­tion. The re­stricted score dis­tri­b­u­tion is then

\[p(k)= \frac{\exp(z_k)} {\sum\limits_{m \in \mathcal{D}} \exp(z_m)}, \qquad k \in \mathcal{D}.\]

The fi­nal scalar score is the ex­pected value of this dis­tri­b­u­tion:

\[\hat{s}= \sum_{k \in \mathcal{D}} k\,p(k).\]

This pro­duces a smooth score such as (5.4), rather than forc­ing the model to com­mit to a sin­gle sam­pled in­te­ger. In prac­tice, this is sub­stan­tially more sta­ble than naive score sam­pling and bet­ter re­flects the mod­el’s un­cer­tainty. It also han­dles cases where the judge dis­tri­b­u­tion is broad or mul­ti­modal. For ex­am­ple, two can­di­dates may both have mean score (5.4), while one has most of its mass tightly con­cen­trated around (5) and (6), and the other splits mass be­tween much lower and much higher rat­ings. The mean alone is the same, but the un­der­ly­ing judge­ment is very dif­fer­ent.

An op­tional un­cer­tainty es­ti­mate can be ob­tained from the vari­ance of the re­stricted dis­tri­b­u­tion:

\[\mathrm{Var}(s)= \sum_{k=0}^{9} (k-\hat{s})^2\,p(k).\]

In short, the method re­places a noisy sam­pled judge score with a nor­mal­ized prob­a­bil­ity dis­tri­b­u­tion over valid score dig­its, then uses the ex­pec­ta­tion of that dis­tri­b­u­tion as the fi­nal rat­ing.

All this stuff is prob­a­bly pretty ob­vi­ous these days, back in ’24 there was­n’t much to guide me in de­vel­op­ing this method, but un­for­tu­nately, I found it was also com­pletely use­less…

Each con­fig­u­ra­tion needed to gen­er­ate hun­dreds of to­kens of cre­ative out­put, and then a sep­a­rate model had to read and judge each one. With over 3,200 con­fig­u­ra­tions to test for a sin­gle 70B model, this would have taken weeks on my dual 4090s.

I needed probes where the out­put was tiny, a few to­kens at most, and where scor­ing was ob­jec­tive and de­ter­min­is­tic. No judge model in the loop. That’s what led me to the fi­nal two probes:

Hard math. Ridiculously dif­fi­cult ques­tions like: “What is the cube root of 74,088,893,247?” No chain-of-thought, or tool use. Just out­put the num­ber, as a pure leap of in­tu­itive faith.

Emotional quo­tient. Using the EQ-Bench bench­mark: com­plex so­cial sce­nar­ios where the model must pre­dict the in­ten­sity of spe­cific emo­tional states. “Given this sit­u­a­tion, how an­gry/​sur­prised/​guilty would this per­son feel on a scale of 0-100?” Completely dif­fer­ent from math. Theory of mind, so­cial in­fer­ence, em­pa­thy. And the out­put is just a few num­bers.

I had set­tled on two max­i­mally or­thog­o­nal cog­ni­tive tasks, both with tiny out­puts. My in­tu­ition was this: LLMs think one to­ken at a time, so lets make the model re­ally good at guess­ing just the next to­ken. But things are never straight­for­ward. Take LLM num­bers…

Even with math probes, I hit un­ex­pected prob­lems. LLMs fail arith­metic in weird ways. They don’t get the an­swer wrong so much as get it al­most right but for­get to write the last digit, as if it got bored mid-num­ber. Or they trans­pose two dig­its in the mid­dle. Or they out­put the cor­rect num­ber with a trail­ing char­ac­ter that breaks the parser.

This is prob­a­bly due to the way larger num­bers are to­kenised, as big num­bers can be split up into ar­bi­trary forms. Take the in­te­ger 123456789. A BPE to­k­enizer (e.g., GPT-style) might split it like: ‘123’ ‘456’ ‘789’ or: ‘12’ ‘345’ ‘67’ ‘89’

A bi­nary right/​wrong scor­ing sys­tem would throw away use­ful sig­nal. Getting a per­cent­age cor­rect would help: ‘123356789’ in­stead of ‘123456789’ would be 99.92% cor­rect

But what about a model that makes a dumb ‘LLM-mistake’ and out­puts 430245 when the an­swer is 4302459, and has clearly done most of the work? I wrote a cus­tom par­tial-credit scor­ing func­tion that pads shorter an­swers and pe­nalises pro­por­tion­ally:

The key idea: pad shorter an­swers, then pe­nalise via the cor­rec­tion fac­tor. A model that nails 90% of the dig­its but drops the last one still gets sub­stan­tial credit — but less than one that gets every digit. This turned out to be cru­cial for dis­crim­i­nat­ing be­tween con­fig­u­ra­tions that were close in in­tu­itive math abil­ity.

The math ques­tions were hand-crafted ini­tially. I ex­per­i­mented with dif­fer­ent op­er­a­tions and scales, then gen­er­ated ran­dom num­bers to fill out the dataset. The dataset was a set of 16 ques­tions, and the model is tasked with guessti­mat­ing the near­est whole in­te­ger num­ber. Here are a few to try your­self, re­mem­ber no ‘thinking’ is al­lowed, guess it di­rectly!

After test­ing sev­eral smaller mod­els (Llama’s and smaller Qwen2’s), I set up the con­fig for Qwen2-72B and let it sweep. Each $(i, j)$ con­fig­u­ra­tion took a few min­utes: load the re-lay­ered model, run the math probe, run the EQ probe, record the scores, move on. Days of con­tin­u­ous GPU time on the 4090s. But far less com­pute than a fine tune! In fact, I did­n’t even have the hard­ware needed for a LORA fine-tune on just 48GB of VRAM.

The op­ti­mal con­fig­u­ra­tion was $(45, 52)$: lay­ers 0 through 51 run first, then lay­ers 45 through 79 run again. Layers 45 to 51 ex­e­cute twice. Seven ex­tra lay­ers, near the mid­dle of the 80-layer stack, bring­ing the to­tal pa­ra­me­ter count from 72B to 78B. Every ex­tra layer is an ex­act copy of an ex­ist­ing one. No new weights or train­ing, just the model re­peat­ing it­self.

Repeating seven lay­ers. That’s all it took, and now I can fi­nally re­veal the nomen­cla­ture of my mod­els: Repeat Your Self for RYS-XLarge ;)

I ap­plied the con­fig­u­ra­tion to MaziyarPanahi’s calme-2.1-qwen2-72b — a fine-tune of Qwen2-72B — and up­loaded the re­sult as dnhkng/​RYS-XLarge. I also ap­plied it to the raw base model as dnhkng/​RYS-XLarge-base.

Then I sub­mit­ted to the Open LLM Leaderboard and waited. And waited. Back in the day, the OpenLLM Leaderboard was flooded with dozens of fine-tunes of merges of fine-tunes each day (it was the Wild West), and the wait­ing list was long. But af­ter a month or so, the re­sults ar­rived:

+17.72% on MuSR. +8.16% on MATH. Five out of six bench­marks im­proved, with only IFEval tak­ing a small hit. The av­er­age put it at #1 on the leader­board.

Just to labour the point: I only op­ti­mised for one-shot guessti­mat­ing hard maths prob­lems and EQ-Bench. I never looked at IFEval, BBH, GPQA, MuSR, or MMLU-PRO dur­ing de­vel­op­ment. The leader­board was pure out-of-sam­ple val­i­da­tion.

A layer con­fig­u­ra­tion found us­ing two nar­row, or­thog­o­nal probes gen­er­alised to every­thing the Leaderboard threw at it *.

* - ex­cept IFEval, but that one’s bor­ing any­way, right?

That was sur­pris­ing enough. A brand new way to scale LLMs, de­vel­oped on some gam­ing GPUs. But the plot­ting out the heatmaps told an even bet­ter story.

The orig­i­nal heatmaps that pro­duced RYS-XLarge, show­ing the Combined delta (math + EQ). The green cir­cle marks the op­ti­mal con­fig­u­ra­tion. Red means im­prove­ment, blue means degra­da­tion

These heatmaps are anal­o­gous to func­tional MRIs of the Transformer, while it is think­ing about maths of EQ prob­lems.

The x-axis ($j$) is the end point of the du­pli­cated re­gion. The y-axis ($i$) is the start point. Each pixel rep­re­sents a com­plete eval­u­a­tion: load the re-lay­ered model, run the math probe, run the EQ probe, score both, record the deltas. As de­scribed above, along the cen­tral di­ag­o­nal only a sin­gle layer was du­pli­cated. Along the next di­ag­o­nal to­wards the top-right, we du­pli­cate two lay­ers, and so on. The sin­gle point at the very top-right runs through the en­tire Transformer stack twice per in­fer­ence.

Let’s ex­am­ine the math heatmap first. Starting at any layer, and stop­ping be­fore about layer 60 seem to im­proves the math guessti­mate scores, as shown by the large re­gion with a healthy red blush. Duplicating just the very first lay­ers (the tiny tri­an­gle in the top left), messes things up, as does re­peat­ing pretty much any of the last 20 lay­ers (the ver­ti­cal wall of blue on the right). This is more clearly vi­su­alised in a sky­line plot (averaged rows or columns), and we can see for the maths guessti­mates, the start­ing po­si­tion of the du­pli­ca­tion mat­ters much less. So, the hy­poth­e­sis that ‘starting lay­ers’ en­code to­kens, to a smooth ‘thinking space’, and then fi­nally a ded­i­cated ‘re-encoding’ sys­tem seem to be some­what val­i­dated.

Until we look at the EQ scores:

Now things look very dif­fer­ent! Duplicating any of the fi­nal 10 lay­ers has al­most no ef­fect on the scores, but we see com­plex pat­terns, where some re­gions show sig­nif­i­cant im­prove­ment (the area around 45i, 55j), walled be­tween re­gions of poor per­for­mance.

But the heatmaps re­vealed some­thing even more in­ter­est­ing than the lo­ca­tion of the think­ing bits. They re­vealed some­thing about its struc­ture.

Before set­tling on block du­pli­ca­tion, I tried some­thing sim­pler: take a sin­gle mid­dle layer and re­peat it $n$ times. If the “more rea­son­ing depth” hy­poth­e­sis was cor­rect, this should work. It made sense too, look­ing at the broad boost in math guessti­mate re­sults by du­pli­cat­ing in­ter­me­di­ate layer. Give the model ex­tra copies of a par­tic­u­lar rea­son­ing layer, get bet­ter rea­son­ing. So, I screened them all, look­ing for a boost.

But nope, it al­most al­ways did worse. Usually a lot worse, but with oc­ca­sional small im­prove­ments that were within the noise range. Annoying, but tak­ing an­other look at the com­plex, blobby pat­terns in EQ scores gave me an­other idea:

If sin­gle-layer du­pli­ca­tion does­n’t help, the mid­dle lay­ers aren’t do­ing in­de­pen­dent it­er­a­tive re­fine­ment. They’re not in­ter­change­able copies of the same op­er­a­tion that you can sim­ply “run again.” If they were, du­pli­cat­ing any one of them should give at least a mar­ginal ben­e­fit. Instead, those lay­ers are work­ing as a cir­cuit. A multi-step rea­son­ing pipeline that needs to ex­e­cute as a com­plete unit.

Think of it this way. Layers 46 through 52 aren’t seven work­ers do­ing the same job. They’re seven steps in a recipe. Layer 46 takes the ab­stract rep­re­sen­ta­tion and per­forms step one of some cog­ni­tive op­er­a­tion — maybe de­com­pos­ing a com­plex rep­re­sen­ta­tion into sub­com­po­nents. Layer 47 takes that out­put and per­forms step two — maybe iden­ti­fy­ing re­la­tion­ships be­tween the sub­com­po­nents. Layer 48 does step three, and so on through layer 52, which pro­duces the fi­nal re­sult.

Duplicating just one step of this ‘recipe’ does­n’t bring you much.

But du­pli­cat­ing the en­tire block gives you the full recipe twice. The model runs the com­plete rea­son­ing cir­cuit, pro­duces a re­fined in­ter­me­di­ate rep­re­sen­ta­tion, and then runs the same cir­cuit again on its own out­put. It’s a sec­ond pass. A chance to catch what it missed the first time, to re­fine its ab­strac­tions, to push the rea­son­ing one step deeper.

Let’s deep-dive into a more cur­rent model (that I can ex­per­i­ment with on my sys­tem): ExllamaV3 GLM-4.7 from mrat­sim

I’ve marked out a re­gion that boosts maths abil­ity strongly. Notice where it sits? It’s away from the di­ag­o­nal cen­tre line, which means we’re not look­ing at sin­gle-layer du­pli­ca­tions. Starting the re­peated block at po­si­tion 35, we don’t see any im­prove­ment un­til at least po­si­tion 43. That’s seven lay­ers of not much hap­pen­ing. In fact, we ac­tu­ally see de­creased per­for­mance by re­peat­ing these lay­ers (they are blue, bad!).

From end-po­si­tion 43 to 46, we then see solid boosts in math scores (red = good, yay). But in­clude layer 46 or be­yond, and the ben­e­fits col­lapse again. The hy­poth­e­sis: po­si­tion 47 is where a dif­fer­ent cir­cuit be­gins. Including even one step of the next recipe messes up the cur­rent recipe.

So the ‘math or­gan’ has bound­aries on both sides. Too few lay­ers and you get noth­ing — you’ve cut into the cir­cuit and it can’t com­plete its op­er­a­tion. Too many lay­ers and you also get noth­ing — you’ve in­cluded tis­sue from a neigh­bour­ing cir­cuit that does­n’t be­long. Pre-training carved these struc­tures out of the layer stack, and they only work whole. It also does­n’t trans­late to other tasks, as the heatmap for EQ scores does­n’t have this patch.

Today we should ramp down rhetoric. I thought no­body would take three min­utes to es­cape the per­pet­ual un­der­class or you are worth $0.003/hr se­ri­ously. But it looks like some peo­ple do, and you should­n’t.

Social me­dia has been ex­tremely toxic for the last cou­ple months. It’s tar­get­ing you with fear and anx­i­ety. If you don’t use this new stu­pid AI thing you will fall be­hind. If you haven’t to­tally up­dated your work­flow you are worth 0. There’s peo­ple who built bil­lion dol­lars com­pa­nies by or­ches­trat­ing 37 agents this morn­ing AND YOU JUST SAT THERE AND ATE BREAKFAST LIKE A PLEB!

This is all com­plete non­sense. AI is not a mag­i­cal game changer, it’s sim­ply the con­tin­u­a­tion of the ex­po­nen­tial of progress we have been on for a long time. It’s a win in some ar­eas, a loss in oth­ers, but over­all a win and a cool tool to use. And it will con­tinue to im­prove, but it won’t “go re­cur­sive” or what­ever the claim is. It’s al­ways been re­cur­sive. You see things like au­tore­search and it’s cool. But it’s not magic, it’s search. People see “AI” and they at­tribute some sci-fi thing to it when it’s just search and op­ti­miza­tion. Always has been, and if you paid at­ten­tion in CS class, you know the lim­its of those things.

That said, if you have a job where you cre­ate com­plex­ity for oth­ers, you will be found out. The days of rent seek­ers are com­ing to an end. But not be­cause there will be no more rent seek­ing, it’s be­cause rent seek­ing is a 0 sum game and you will lose at it to big­ger play­ers. If you have a job like that, or work at a com­pany like that, the sooner you quit the bet­ter your out­come will be. This is the real dri­ver of the lay­offs, the big play­ers con­sol­i­dat­ing the rent seek­ing to them. They just say it’s AI cause that makes the stock price go up.

The trick is not to play zero sum games. This is what I have been say­ing the whole time. Go cre­ate value for oth­ers and don’t worry about the re­turns. If you cre­ate more value than you con­sume, you are wel­come in any well op­er­at­ing com­mu­nity. Not in­fi­nite, not al­ways needs more, just more than you con­sume. That’s enough, and avoid peo­ple or com­par­i­son traps that tell you oth­er­wise. The world is not a Red Queen’s race.

This post will get way less trac­tion than the doom ones, but it’s telling you the way out.

I Have No Idea If What They Ship Is Any GoodI’ve been build­ing agents that write code while I sleep. Tools like Gastown run for hours with­out me watch­ing. Changes land in branches I haven’t read. A few weeks ago I re­al­ized I had no re­li­able way to know if any of it was cor­rect: whether it ac­tu­ally does what I said it should do. I care about this. I don’t want to push slop, and I had no real an­swer.I’ve run Claude Code work­shops for over 100 en­gi­neers in the last six months. Same prob­lem every­where, just at dif­fer­ent scales. Teams us­ing Claude for every­day PRs are merg­ing 40-50 a week in­stead of 10. Teams are spend­ing a lot more time in code re­views. As sys­tems get more au­tonomous, the prob­lem com­pounds. At some point you’re not re­view­ing diffs at all, just watch­ing de­ploys and hop­ing some­thing does­n’t break.So the ques­tion I kept com­ing back to: what do you ac­tu­ally trust when you can’t re­view every­thing?You could hire more re­view­ers. But you can’t hire fast enough. And mak­ing se­nior en­gi­neers read AI-generated code all day is­n’t worth it.When Claude writes tests for code Claude just wrote, it’s check­ing its own work. The tests prove the code does what Claude thought you wanted. Not what you ac­tu­ally wanted. They catch re­gres­sions but not the orig­i­nal mis­un­der­stand­ing.When you use the same AI for both, you’ve built a self-con­grat­u­la­tion ma­chine.This is ex­actly the prob­lem code re­view was sup­posed to solve: a sec­ond set of eyes that was­n’t the orig­i­nal au­thor. But one AI writ­ing and an­other AI check­ing is­n’t a fresh set of eyes. They come from the same place. They’ll miss the same things.The thing TDD got rightWrite the test first, write the code sec­ond, stop when the test passes. Most teams don’t do this be­cause think­ing through what the code should do be­fore writ­ing it takes time they don’t have.AI re­moves that ex­cuse, be­cause Claude han­dles the speed. The slow part is now fig­ur­ing out if the code is right. That’s what TDD was built for: write down what cor­rect looks like, then check it.TDD asks you to write unit tests, which means think­ing about how the code will work be­fore you write it. This is eas­ier. Write down what the fea­ture should do in plain English. The ma­chine fig­ures out how to check it.“Users can au­then­ti­cate with email and pass­word. On wrong cre­den­tials they see ‘Invalid email or pass­word.’ On suc­cess they land on /dashboard. The ses­sion to­ken ex­pires af­ter 24 hours.” You can write that be­fore you open a code ed­i­tor. The agent builds it. Something else checks it.P.S I write about Claude Code in­ter­nals every week. Last week I wrote about how Claude Code is a while loop with 23 tools. Subscribe to get the next one!What this looks like in prac­tice­For fron­tend changes, we gen­er­ated ac­cep­tance cri­te­rias based on the spec file:# Task

Add email/​pass­word lo­gin.

## Acceptance Criteria

### AC-1: Successful lo­gin

- User at /login with valid cre­den­tials gets redi­rected to /dashboard

- Session cookie is set

### AC-2: Wrong pass­word er­ror

- User sees ex­actly “Invalid email or pass­word”

- User stays on /login

### AC-3: Empty field val­i­da­tion

- Submit dis­abled when ei­ther field is empty, or in­line er­ror on empty sub­mit

### AC-4: Rate lim­it­ing

- After 5 failed at­tempts, lo­gin blocked for 60 sec­onds

- User sees a mes­sage with the wait timeEach cri­te­rion is spe­cific enough that it ei­ther passes or fails. Once the agent builds the fea­ture, ver­i­fi­ca­tion runs Playwright browser agents against each AC, takes screen­shots, and pro­duces a re­port with per-cri­te­rion ver­dicts. If some­thing fails you see ex­actly which cri­te­rion and what the browser saw.For back­end changes the same pat­tern works with­out a browser. You spec­ify ob­serv­able API be­hav­ior (status codes, re­sponse head­ers, er­ror mes­sages) that curl com­mands can check.One thing worth be­ing hon­est about: this does­n’t catch spec mis­un­der­stand­ings. If your spec was wrong to be­gin with, the checks will pass even when the fea­ture is wrong. What Playwright does catch is in­te­gra­tion fail­ures, ren­der­ing bugs, and be­hav­ior that works in the­ory but breaks in a real browser. That’s a nar­rower claim than “verified cor­rect,” but it’s more than a code re­view was re­li­ably catch­ing any­way.The work­flow: write ac­cep­tance cri­te­ria be­fore you prompt, let the agent build against them, run ver­i­fi­ca­tion, re­view only the fail­ures. You re­view fail­ures in­stead of diffs.How to build itI started build­ing a Claude Skill (github.com/​op­slane/​ver­ify) that runs us­ing claude -p (Claude Code’s head­less mode) plus Playwright MCP. No cus­tom back­end, no ex­tra API keys be­yond your ex­ist­ing Claude OAuth to­ken. Four stages:Pre-flight is pure bash, no LLM. Is the dev server run­ning? Is the auth ses­sion valid? Does a spec file ex­ist? Fail fast be­fore spend­ing any to­kens.The plan­ner is one Opus call. It reads your spec and the files you changed. It fig­ures out what each check needs and how to run it. It also reads your code to find the right se­lec­tors, so it’s not guess­ing at class names.Browser agents are one Sonnet call per AC, all run­ning in par­al­lel. Five ACs, five agents, each nav­i­gat­ing and screen­shot­ting in­de­pen­dently. Sonnet costs 3-4x less than Opus here and works just as well for click­ing around.The judge is one fi­nal Opus call that reads all the ev­i­dence and re­turns a ver­dict per cri­te­rion: pass, fail, or needs-hu­man-re­view.claude -p –model claude-opus-4-6 \

“Review this ev­i­dence and re­turn a ver­dict for each AC.

Evidence: $(cat .verify/evidence/*/result.json)

Return JSON: {verdicts: [{id, passed, rea­son­ing}]}“Or clone the repo and adapt it. Each stage is a sin­gle claude -p call with a clear in­put and struc­tured out­put. You can swap mod­els, add stages, or wire it into CI with –dangerously-skip-permissions.The thing I keep com­ing back to: you can’t trust what an agent pro­duces un­less you told it what “done” looks like be­fore it started. Writing ac­cep­tance cri­te­ria is harder than writ­ing a prompt, be­cause it forces you to think through edge cases be­fore you’ve seen them. Engineers re­sist it for the same rea­son they re­sisted TDD, be­cause it feels slower at the start.With­out them, all you can do is read the out­put and hope it’s right.

Debian is the lat­est in an ever-grow­ing list of pro­jects to wres­tle (again) with the ques­tion of LLM-generated con­tri­bu­tions; the lat­est de­bate stared in mid-Feb­ru­ary, af­ter Lucas Nussbaum opened a

dis­cus­sion with a draft gen­eral res­o­lu­tion (GR) on whether Debian should ac­cept AI-assisted con­tri­bu­tions. It seems to have, mostly, sub­sided with­out a GR be­ing put for­ward or any de­ci­sions be­ing made, but the con­ver­sa­tion was il­lu­mi­nat­ing nonethe­less.

Nussbaum said that Debian prob­a­bly needed to have a dis­cus­sion “to un­der­stand where we stand re­gard­ing AI-assisted con­tri­bu­tions to

Debian” based on some re­cent dis­cus­sions, though it was not clear what dis­cus­sions he was re­fer­ring to. Whatever the spark was, Nussbaum put for­ward the draft GR to clar­ify Debian’s stance on al­low­ing AI-assisted con­tri­bu­tions. He said that he would wait a cou­ple of days to col­lect feed­back be­fore for­mally sub­mit­ting the GR.

Like what you are read­ing?

Try LWN for free for 1 month,

no credit card re­quired.

His pro­posal would al­low “” if a num­ber of con­di­tions were met. For ex­am­ple, it would re­quire ex­plicit dis­clo­sure if “a

sig­nif­i­cant por­tion of the con­tri­bu­tion is taken from a tool with­out

man­ual mod­i­fi­ca­tion”, and la­bel­ing of such con­tri­bu­tions with ”.” It also spells out that con­trib­u­tors should “” their sub­mis­sions and would be ac­count­able for the con­tri­bu­tions, “including vouch­ing for the tech­ni­cal merit,

se­cu­rity, li­cense com­pli­ance, and util­ity of their

sub­mis­sions”. The GR would also pro­hibit us­ing gen­er­a­tive-AI tools with non-pub­lic or sen­si­tive pro­ject in­for­ma­tion, in­clud­ing pri­vate mail­ing lists or em­bar­goed se­cu­rity re­ports.

It is fair to say that it is dif­fi­cult to have an ef­fec­tive con­ver­sa­tion about a tech­nol­ogy when pin­ning down ac­cu­rate ter­mi­nol­ogy is like try­ing to nail Jell-O to a tree. AI is the catch-all term, but much (not all) of the tech­nol­ogy in ques­tion is ac­tu­ally tool­ing around large lan­guage mod­els (LLMs). When par­tic­i­pants have dif­fer­ing ideas of what is be­ing dis­cussed, de­cid­ing whether the thing should be al­lowed may pose some­thing of a prob­lem.

Russ Allbery asked for peo­ple to be more pre­cise in their de­scrip­tions of the tech­nolo­gies that their pro­pos­als might af­fect. He as­serted that it has be­come com­mon for AI, as a term, “to be so

amor­phously and slop­pily de­fined that it could en­com­pass every phys­i­cal ob­ject in the

uni­verse”. If the pro­ject is go­ing to make pol­icy, he said, it needed to be very spe­cific about what it was mak­ing pol­icy about:

Gunnar Wolf agreed with Allbery, but Nussbaum claimed that the spe­cific tech­nol­ogy did not mat­ter. The pro­posal boiled down to the use of au­to­mated tools for code analy­sis and gen­er­a­tion:

I see the prob­lem we face as sim­i­lar to the his­tor­i­cal ques­tions sur­round­ing the use of BitKeeper by Linux (except that the choice of BitKeeper im­posed its use by other con­trib­u­tors). It is also sim­i­lar to the dis­cus­sions about pro­pri­etary se­cu­rity analy­sis tools: since those tools are pro­pri­etary, should we ig­nore the vul­ner­a­bil­ity re­ports they is­sue?

If we were to adopt a hard-line “anti-tools” stance, I would find it very hard to draw a clear line.

Drawing clear lines, how­ever, is some­thing that a num­ber of Debian de­vel­op­ers felt was im­por­tant. Sean Whitton pro­posed that the GR should not only say “LLM” rather than “AI”, but it should also dis­tin­guish be­tween the uses of LLMs, such as code re­view, gen­er­at­ing pro­to­types, or gen­er­at­ing pro­duc­tion code. He en­vi­sioned bal­lot op­tions that could al­low some, but not all, of those uses. Distinguishing be­tween the var­i­ous so-called AI tech­nolo­gies would help in that re­gard. He urged

Nussbaum “not to ar­gue too hard for some­thing that is more gen­eral than LLMs

be­cause that might alien­ate the peo­ple you want to agree to dis­agree with.” Andrea Pappacoda said that the spe­cific tech­nol­ogy mat­tered a lot; he wanted the pro­posal to have clear bound­aries and avoid broad terms like AI. He was un­com­fort­able with the idea of ban­ning LLMs, and not sure where to draw the line. “What I can con­fi­dently say,

though, is that a pro­ject like Claude’s C

Compiler should not have a place in Debian.”

The con­ver­sa­tion did not fo­cus solely on the ter­mi­nol­ogy, of course. Simon Richter

had

ques­tions about the im­pli­ca­tions of al­low­ing AI-driven con­tri­bu­tions from the stand­point of on­board­ing new con­trib­u­tors to Debian. An AI agent, he said, could take the place of a ju­nior de­vel­oper. Both could per­form ba­sic tasks un­der guid­ance, but the AI agent would not learn any­thing from the ex­change; the pro­ject re­sources spent in guid­ing such a tool do not re­sult in long-last­ing knowl­edge trans­fer.

AI use pre­sents us (and the com­mer­cial soft­ware world as well) with a

sim­i­lar prob­lem: there is a mas­sive skill gap be­tween “gets some

re­sults” and “consistently and sus­tain­ably de­liv­ers re­sults”, bridg­ing

that gap es­sen­tially re­quires start­ing from scratch, but is re­quired to

achieve in­de­pen­dence from the op­er­a­tors of the AI ser­vice, and this gap

is dis­rupt­ing the pipeline of new en­trants.

He called that the on­board­ing prob­lem, and said that an AI pol­icy needed to solve that prob­lem; he did not want to dis­cour­age peo­ple by re­ject­ing con­tri­bu­tions or ex­pend re­sources on men­tor­ing peo­ple who did not want to be men­tored. Accepting AI-assisted drive-by con­tri­bu­tions is harm­ful be­cause it is a missed op­por­tu­nity to on­board a new con­trib­u­tor. “The best-case out­come is that a

triv­ial prob­lem got solved with­out ac­tu­ally on­board­ing a new con­trib­u­tor, and the

worst-case out­come is that the new con­trib­u­tor is just prox­y­ing be­tween an AI and the

main­tainer”. He also ex­pressed con­cerns around the costs as­so­ci­ated with such tools, and spec­u­lated it might dis­cour­age con­tri­bu­tion from users who could not af­ford to use for-pay tools.

Nussbaum agreed that the cost could be a prob­lem in the fu­ture. For now, he said, it is not an is­sue be­cause there are ven­dors pro­vid­ing ac­cess for free, but that could change. He dis­agreed that Debian was likely to run out of tasks suit­able for new con­trib­u­tors, even if it does ac­cept AI-driven con­tri­bu­tions, and sug­gested that it may make harder tasks more ac­ces­si­ble. He pointed to a study

writ­ten by an Anthropic em­ployee and a per­son par­tic­i­pat­ing in the com­pa­ny’s fel­lows pro­gram, about how the use of AI im­pacts skill for­ma­tion: “A take­away is that

there are very dif­fer­ent ways to in­ter­act with AI, that pro­duce very dif­fer­ent

re­sults both in terms of speed and of un­der­stand­ing”. He did not seem to be per­suaded that use of AI tools would be a net neg­a­tive in on­board­ing new con­trib­u­tors.

Ted Ts’o ar­gued

against the idea that AI would have a neg­a­tive im­pact:

Matthew Vernon said that the pro­posed GR min­i­mized the eth­i­cal di­men­sion of us­ing gen­er­a­tive AI. The or­ga­ni­za­tions that are de­vel­op­ing and mar­ket­ing tools like ChatGPT and Claude are be­hav­ing un­eth­i­cally, he said, by sys­tem­at­i­cally dam­ag­ing the wider com­mons in the form of au­to­mated scrap­ing and do­ing as they like with oth­ers’ in­tel­lec­tual prop­erty. “They hoover up con­tent as hard as they pos­si­bly can, with scant if any

re­gard to its copy­right or li­cens­ing”. He also cited en­vi­ron­men­tal con­cerns and other harms that are at­trib­uted to gen­er­a­tive AI tools, “from non-con­sen­sual

nud­i­fi­ca­tion to the flood­ing of free soft­ware pro­jects with bo­gus se­cu­rity

re­ports”. He felt that Debian should take a clear stand against those tools and en­cour­age other pro­jects to do the same:

There was also de­bate around the ques­tion of copy­right, both in terms of the li­censes of ma­te­r­ial used to train mod­els, as well as the out­put of LLM tools. Jonathan Dowland thought that it might be bet­ter to for­bid some con­tri­bu­tions now, since some see risks in ac­cept­ing such con­tri­bu­tions, and then re­lax the pro­jec­t’s po­si­tion later on when the le­gal sit­u­a­tion is clearer.

Thorsten Glaser took a

par­tic­u­larly harsh stance against LLM-driven con­tri­bu­tions, go­ing so far as to sug­gest that some up­stream pro­jects should be forced out of Debian’s main

archive into non-free

un­less “”. Ansgar Burchardt pointed

out that would have the ef­fect of ban­ning the Linux ker­nel, Python, LLVM, and oth­ers. Glaser’s pro­posal did not seem par­tic­u­larly pop­u­lar. He had taken a sim­i­lar stance on AI mod­els in 2025; he ar­gued most should be out­side the main archive, when the pro­ject dis­cussed a GR about AI mod­els and the Debian Free Software Guidelines (DFSG). That GR never came to a vote, in part be­cause it was un­clear whether the lan­guage would for­bid anti-spam tech­nolo­gies be­cause one could not in­clude the cor­pus of spam used as train­ing data along with fil­ters.

Allbery did not want to touch on copy­right is­sues but had a few words to say about the qual­ity of AI-assisted code. It is com­mon for peo­ple to ob­ject to code gen­er­ated by LLMs on qual­ity grounds, but he said that ar­gu­ment does not make sense. Humans are ca­pa­ble of pro­duc­ing bet­ter code than LLMs, but they are also ca­pa­ble of pro­duc­ing worse code too. “”

Bdale Garbee sec­onded

that no­tion, and said that he was re­luc­tant to take a hard stance one way or the other. “I see it as just an­other evo­lu­tion­ary stage we don’t re­ally un­der­stand the

longer term pos­i­tive and neg­a­tive im­pacts of yet.” He wanted to fo­cus on long-term im­pli­ca­tions and ques­tions such as “what is the pre­ferred form of

mod­i­fi­ca­tion for code writ­ten by is­su­ing chat prompts?” Nussbaum an­swered that would be “the in­put to the tool, not the gen­er­ated source code”.

That may not be an en­tirely sat­is­fy­ing an­swer, how­ever, given that LLM out­put is not de­ter­min­is­tic and the var­i­ous providers of LLM tools re­tire mod­els with some fre­quency. A user may have the prompt and other ma­te­ri­als fed to an LLM to gen­er­ate a re­sult at a spe­cific point in time, but it might gen­er­ate a much dif­fer­ent re­sult later on, even if one has ac­cess to the same ven­dor’s tools or mod­els to run lo­cally.

It is clear from the dis­cus­sion that Debian de­vel­op­ers are not of one mind on the ques­tion of ac­cept­ing AI-generated con­tri­bu­tions; the de­vel­op­ers have not yet even con­verged on a shared de­f­i­n­i­tion of what con­sti­tutes an AI-generated con­tri­bu­tion.

What many do seem to agree on is that Debian is not quite ready to vote on a GR about AI-generated con­tri­bu­tions. On March 3, Nussbaum said

that he had pro­posed the GR “in re­sponse to var­i­ous at­tacks against peo­ple us­ing

AI in the con­text of Debian”; he felt then it was some­thing that needed to be dealt with ur­gently. However, the GR dis­cus­sion had been civil and in­ter­est­ing. As long as the dis­cus­sions around AI re­mained calm and pro­duc­tive, the pro­ject could just con­tinue ex­plor­ing the topic in mail­ing-list dis­cus­sions. He guessed that, if there were a GR, “the win­ning op­tion would prob­a­bly be very nu­anced, al­low­ing

AI but with a set of safe­guards”.

The ques­tions of what to do about AI mod­els in the archive, how to han­dle up­stream code gen­er­ated with LLMs, and LLM-generated con­tri­bu­tions writ­ten specif­i­cally for Debian re­main unan­swered. For now, it seems, they will con­tinue to be han­dled on a case-by-case ba­sis by ap­ply­ing Debian’s ex­ist­ing poli­cies. Given the com­plex­ity of the ques­tions, di­verse opin­ions, and rapid rate of change of tech­nolo­gies lumped in un­der the “AI” um­brella, that may be the best pos­si­ble, and least dis­rup­tive, out­come for now.

You can now crawl an en­tire web­site with a sin­gle API call us­ing Browser Rendering’s new /crawl end­point, avail­able in open beta. Submit a start­ing URL, and pages are au­to­mat­i­cally dis­cov­ered, ren­dered in a head­less browser, and re­turned in mul­ti­ple for­mats, in­clud­ing HTML, Markdown, and struc­tured JSON. This is great for train­ing mod­els, build­ing RAG pipelines, and re­search­ing or mon­i­tor­ing con­tent across a site.

Crawl jobs run asyn­chro­nously. You sub­mit a URL, re­ceive a job ID, and check back for re­sults as pages are processed.

* Multiple out­put for­mats - Return crawled con­tent as HTML, Markdown, and struc­tured JSON (powered by Workers AI)

* Crawl scope con­trols - Configure crawl depth, page lim­its, and wild­card pat­terns to in­clude or ex­clude spe­cific URL paths

* Automatic page dis­cov­ery - Discovers URLs from sitemaps, page links, or both

* Incremental crawl­ing - Use mod­i­fiedSince and max­Age to skip pages that haven’t changed or were re­cently fetched, sav­ing time and cost on re­peated crawls

* Static mode - Set ren­der: false to fetch sta­tic HTML with­out spin­ning up a browser, for faster crawl­ing of sta­tic sites

Available on both the Workers Free and Paid plans.

To get started, re­fer to the crawl end­point doc­u­men­ta­tion. If you are set­ting up your own site to be crawled, re­view the ro­bots.txt and sitemaps best prac­tices.

One year ago, on 28 February 2025, Wikipedia user Moyogo up­dated the page for Angzarr

with a ci­ta­tion to the type foundry H. Berthold AG’s 1950 sym­bol cat­a­logue list­ing ⍼ as Azimut, Richtungswinkel, or “azimuth”, “direction an­gle”. Mystery solved!

Fonts in Use lists links to archived cat­a­logues by Berthold. The above scan is from the 1950 Zeichenprobe

(symbol cat­a­logue) on page 7. Copies of the Schriftprobe (font cat­a­logue) from 1949, 1951, and 1952

all show on page 104 the same glyph and sizes, al­beit with­out the de­scrip­tor name.

⍼ does not ap­pear in the 1946 Registerprobe, nor in ear­lier 1909

and 1900 cat­a­logues. For con­ve­nience, I’ve ex­tracted full-page scans be­low for where it ap­pears — and where I feel it would ap­pear, but does­n’t.

A friend on Mastodon pointed out that the glyph ⍼ it­self re­sem­bles the way a light ray passes through a sex­tant

to mea­sure an az­imuth, with the right an­gle be­ing a stan­dard sym­bol for an an­gle in gen­eral. Wikipedia has a lovely il­lus­tra­tion demon­strat­ing how a sex­tant works to mea­sure lat­i­tude of the sun; it can, of course, be turned side­ways to mea­sure an az­imuth with re­spect to an ar­bi­trary merid­ian.

In the realm of med­ical ad­vance­ments, a uni­ver­sal vac­cine that can pro­tect against any pathogen has long been a Holy Grail — and about as elu­sive as a mytho­log­i­cal ves­sel.

But Stanford Medicine re­searchers and col­lab­o­ra­tors have taken an as­ton­ish­ing step for­ward in that quest, sur­pris­ing even them­selves. In a new study in mice, they have de­vel­oped a uni­ver­sal vac­cine for­mula that pro­tects against a wide range of res­pi­ra­tory viruses, bac­te­ria and even al­ler­gens. The vac­cine is de­liv­ered in­tranasally — such as through a nasal spray — and pro­vides broad pro­tec­tion in the lungs for sev­eral months.

In the study that was pub­lished Feb. 19 in Science, re­searchers showed that vac­ci­nated mice were pro­tected against SARS-CoV-2 and other coro­n­aviruses, Staphylococcus au­reus and Acinetobacter bau­man­nii (common hos­pi­tal-ac­quired in­fec­tions), and house dust mites (a com­mon al­ler­gen). In fact, the new vac­cine has worked for a re­mark­ably wide spec­trum of res­pi­ra­tory threats the re­searchers have tested, said Bali Pulendran, PhD, the Violetta L. Horton Professor II and a pro­fes­sor of mi­cro­bi­ol­ogy and im­munol­ogy who is the study’s se­nior au­thor.

The lead au­thor of the study is Haibo Zhang, PhD, a post­doc­toral scholar in Pulendran’s lab.

If trans­lated into hu­mans, such a vac­cine could re­place mul­ti­ple jabs every year for sea­sonal res­pi­ra­tory in­fec­tions and be on hand should a new pan­demic virus emerge.

The new vac­cine is un­like any vac­cine used to­day.

Since the 1790s, when the English physi­cian Edward Jenner coined the term vac­ci­na­tion (from the Latin vacca for cow) to re­fer to the use of cow­pox to in­oc­u­late against small­pox, all sub­se­quent vac­cines have re­lied on the same fun­da­men­tal prin­ci­ple: anti­gen speci­ficity. That is, the vac­cine mim­ics a dis­tinc­tive com­po­nent of the pathogen — the spike pro­teins that cover SARS-CoV-2, for ex­am­ple — to pre­pare the im­mune sys­tem to rec­og­nize and re­act quickly to the real pathogen.

“That’s been the par­a­digm of vac­ci­nol­ogy for the last 230 years,” Pulendran said.

But anti­gen-spe­cific vac­cines fail when a pathogen mu­tates or when new pathogens emerge. That’s why there’s a new COVID-19 booster and flu shot every year.

“It’s be­com­ing in­creas­ingly clear that many pathogens are able to quickly mu­tate. Like the prover­bial leop­ard that changes its spots, a virus can change the anti­gens on its sur­face,” Pulendran said.

Most at­tempts at a so-called uni­ver­sal vac­cine have the mod­est goal of in­duc­ing im­mu­nity against an en­tire fam­ily of virus — all coro­n­aviruses or all flu viruses, for ex­am­ple — usu­ally by mim­ic­k­ing evo­lu­tion­ar­ily con­served vi­ral com­po­nents that are less likely to mu­tate. A truly uni­ver­sal vac­cine that can counter di­verse pathogens was a pie-in-the-sky idea.

“We were in­ter­ested in this idea be­cause it sounded a bit out­ra­geous,” Pulendran said. “I think no­body was se­ri­ously en­ter­tain­ing that some­thing like this could ever be pos­si­ble.”

The new vac­cine does­n’t try to mimic any part of a pathogen; in­stead, it mim­ics the sig­nals that im­mune cells use to com­mu­ni­cate with each other dur­ing an in­fec­tion. This novel strat­egy in­te­grates the two branches of im­mu­nity — in­nate and adap­tive — cre­at­ing a feed­back loop that sus­tains a broad im­mune re­sponse.

The adap­tive im­mune sys­tem is the work­horse of cur­rent vac­cines. It pro­duces spe­cial­ized agents, like an­ti­bod­ies and T cells, that tar­get spe­cific pathogens and re­mem­ber them for years to come. The in­nate im­mune sys­tem, which de­ploys within min­utes of a new in­fec­tion, has re­ceived less at­ten­tion be­cause it typ­i­cally lasts only a few days be­fore ced­ing the spot­light to the adap­tive im­mune sys­tem. It was seen as the warm-up act for the main show.

But Pulendran’s team was in­trigued by the ver­sa­til­ity of the in­nate im­mune sys­tem, which con­sists of gen­er­al­ists (such as den­dritic cells, neu­trophils and macrophages) that de­stroy any­thing deemed a pathogen.

“What’s re­mark­able about the in­nate sys­tem is that it can pro­tect against a broad range of dif­fer­ent mi­crobes,” Pulendran said.

Innate im­mu­nity is short-lived, but pro­vides some­thing ap­proach­ing uni­ver­sal pro­tec­tion.

We were in­ter­ested in this idea be­cause it sounded a bit out­ra­geous. I think no­body was se­ri­ously en­ter­tain­ing that some­thing like this could ever be pos­si­ble.”

—Bali Pulendran

There have long been hints that in­nate im­mu­nity can last longer in cer­tain cir­cum­stances. The most-stud­ied ex­am­ple is the Bacillus Calmette-Guerin tu­ber­cu­lo­sis vac­cine, which is given to some 100 mil­lion new­borns every year. Epidemiological and clin­i­cal stud­ies have shown that it can de­crease in­fant mor­tal­ity from other in­fec­tions, sug­gest­ing that the cross-pro­tec­tion could last months. But the phe­nom­e­non was in­con­sis­tent and the mech­a­nism mys­te­ri­ous.

In 2023, Pulendran’s team pub­lished a study in mice elu­ci­dat­ing the mech­a­nism. Like other vac­cines, the tu­ber­cu­lo­sis vac­cine in­duced both an in­nate and adap­tive im­mune re­sponse in the mice, but un­usu­ally, the in­nate re­sponse was sus­tained for sev­eral months. The re­searchers dis­cov­ered that T cells re­cruited to the lungs as part of the adap­tive re­sponse were send­ing sig­nals to the in­nate im­mune cells to keep them ac­tive.

“Those T cells were pro­vid­ing a crit­i­cal sig­nal to keep the ac­ti­va­tion of the in­nate sys­tem, which typ­i­cally lasts for a few days or a week, but in this case, it could last for three months,” Pulendran said.

The re­searchers showed that as long as the in­nate re­sponse re­mained ac­tive, the mice were pro­tected against SARS-CoV-2 and other coro­n­avirus in­fec­tions. They iden­ti­fied the sig­nals sent by T cells as cy­tokines that ac­ti­vate pathogen-sens­ing re­cep­tors, known as toll-like re­cep­tors, on in­nate im­mune cells.

“In that pa­per, we spec­u­lated that since we now know how the tu­ber­cu­lo­sis vac­cine is me­di­at­ing its cross-pro­tec­tive ef­fects, it would be pos­si­ble to make a syn­thetic vac­cine, per­haps a nasal spray, that has the right com­bi­na­tion of toll-like re­cep­tor stim­uli and some anti­gen to get the T cells into the lungs,” Pulendran said.

“Fast for­ward two and a half years and we’ve shown that ex­actly what we had spec­u­lated is fea­si­ble in mice.”

The new vac­cine, for now known as GLA-3M-052-LS+OVA, mim­ics the T cell sig­nals that di­rectly stim­u­late in­nate im­mune cells in the lungs. It also con­tains a harm­less anti­gen, an egg pro­tein called oval­bu­min or OVA, which re­cruits T cells into the lungs to main­tain the in­nate re­sponse for weeks to months.

In the study, mice were given a drop of the vac­cine in their noses. Some re­cieved mul­ti­ple doses, given a week apart. Each mouse was then ex­posed to one type of res­pi­ra­tory virus. With three doses of the vac­cine, mice were pro­tected against SARS-CoV-2 and other coro­n­aviruses for at least three months.

In un­vac­ci­nated mice, these viruses caused dra­matic weight loss — a sign of ill­ness — and of­ten death; their lungs were in­flamed and full of virus. Vaccinated mice lost much less weight and all sur­vived; their lungs were nearly clear of the virus.

The vac­cine is a “double whammy” against vi­ral in­fec­tion, Pulendran said. The pro­longed in­nate re­sponse low­ers the amount of virus in the lungs by 700-fold. And viruses that slip through this ini­tial de­fense are met with a swift adap­tive re­sponse in the lungs.

“The lung im­mune sys­tem is so ready and so alert that it can launch the typ­i­cal adap­tive re­sponses — virus-spe­cific T cells and an­ti­bod­ies — in as lit­tle as three days, which is an ex­tra­or­di­nar­ily short length of time,” Pulendran said. “Normally, in an un­vac­ci­nated mouse, it takes two weeks.”

Amazed by the vac­cine’s abil­ity to fend off dif­fer­ent types of vi­ral in­fec­tions, the re­searchers ex­panded their test­ing to bac­te­r­ial res­pi­ra­tory in­fec­tions, Staphylococcus au­reus and Acinetobacter bau­man­nii. The vac­ci­nated mice were pro­tected against these, too, for about three months.

“Then we thought, ‘What else could go in the lung?’” Pulendran said. “Allergens.”

They ex­posed the mice to a pro­tein from house dust mites, a com­mon trig­ger for al­ler­gic asthma. Allergic re­ac­tions are caused by a type of im­mune re­sponse known as Th2 re­sponse. Unvaccinated mice showed a strong Th2 re­sponse and mu­cus ac­cu­mu­la­tion in their air­ways. The vac­cine quelled the Th2 re­sponse and vac­ci­nated mice main­tained clear air­ways.

“I think what we have is a uni­ver­sal vac­cine against di­verse res­pi­ra­tory threats,” Pulendran said.

The re­searchers hope to test the vac­cine in hu­mans next, first in a Phase I safety trial, then, if suc­cess­ful, in a larger trial in which vac­ci­nated peo­ple are ex­posed to in­fec­tions. Pulendran thinks two doses of a nasal spray would be enough to pro­vide pro­tec­tion in hu­mans.

In the best case sce­nario, with enough fund­ing, Pulendran es­ti­mates a uni­ver­sal res­pi­ra­tory vac­cine might be avail­able in five to seven years. It could be a bul­wark against new pan­demics and sim­plify sea­sonal vac­ci­na­tions.

“Imagine get­ting a nasal spray in the fall months that pro­tects you from all res­pi­ra­tory viruses in­clud­ing COVID-19, in­fluenza, res­pi­ra­tory syn­cy­tial virus and the com­mon cold, as well as bac­te­r­ial pneu­mo­nia and early spring al­ler­gens,” Pulendran said. “That would trans­form med­ical prac­tice.”

Researchers from Emory University School of Medicine, the University of North Carolina at Chapel Hill, Utah State University and the University of Arizona con­tributed to the work.

The study re­ceived fund­ing from the National Institutes of Health (grant AI167966), the Violetta L. Horton Professor en­dow­ment, the Soffer Fund en­dow­ment and Open Philanthropy.