Writing this makes me ir­ra­tionally sad, but Ghostty will be leav­ing GitHub1.

I’m GitHub user 1299, joined Feb 2008.

Since then, I’ve opened GitHub every sin­gle day. Every day, mul­ti­ple times per

day, for over 18 years. Over half my life. A hand­ful of ex­cep­tions in there

(I’d love to see the data), but I can’t imag­ine more than a week per year.

GitHub is the place that has made me the most happy. I al­ways made time for

it. When I went through tough breakups? I lost my­self in open source… on

GitHub. During col­lege at 4 AM when every­one is passed out? Let me get one

com­mit in. During my hon­ey­moon while my wife is still asleep? Yeah, GitHub.

It’s where I’ve his­tor­i­cally been hap­pi­est and wanted to be.

Even the an­noy­ing stuff! Some peo­ple doom scroll so­cial me­dia. I’ve been doom

scrolling GitHub is­sues since be­fore that was a word. On va­ca­tions I’d have

book­marks of dif­fer­ent pro­jects on GitHub I wanted to study. Not just source

code, but OSS processes, how other main­tain­ers re­act to dif­fi­cult sit­u­a­tions.

Etc. Believe it or not, I like this.

Some might call this sick, but my hobby and work and pas­sion all align and for

most of my life they got to also live in one place on the in­ter­net: GitHub.

Did you know I started Vagrant (my first suc­cess­ful open source pro­ject) in

large part be­cause I hoped it would get me a job at GitHub? It’s no se­cret,

I’ve said this re­peat­edly, and in my first pub­lic talk about Vagrant, when I

was a mere 20 years old, I joked “maybe GitHub will hire me if it’s good!”

GitHub was my dream job. I did­n’t ever get to work there (not their fault).

But it was the per­fect place I wanted to be. The en­gi­neers were in­cred­i­ble,

the prod­uct was in­cred­i­ble, and it was some­thing I lived and breathed every

day. I still do and con­sis­tently have… for these 18 years. Enough time for

an en­tire hu­man to be­come an adult, all on GitHub.

Lately, I’ve been very pub­licly crit­i­cal of GitHub. I’ve been mean about it.

I’ve been an­gry about it. I’ve hurt peo­ple’s feel­ings. I’ve been lash­ing out.

Because GitHub is fail­ing me, every sin­gle day, and it is per­sonal. It is

ir­ra­tionally per­sonal. I love GitHub more than a per­son should love a thing,

and I’m mad at it. I’m sorry about the hurt feel­ings to the peo­ple work­ing on

it.

I’ve felt this way for a long time, but for the past month I’ve kept a jour­nal

where I put an “X” next to every date where a GitHub out­age has neg­a­tively

im­pacted my abil­ity to work2. Almost every day has an X. On the day I am

writ­ing this post, I’ve been un­able to do any PR re­view for ~2 hours be­cause

there is a GitHub Actions out­age3. This is no longer a place for se­ri­ous

work if it just blocks you out for hours per day, every day.

It’s not a fun place for me to be any­more. I want to be there but it does­n’t

want me to be there. I want to get work done and it does­n’t want me to get

work done. I want to ship soft­ware and it does­n’t want me to ship soft­ware.

I want it to be bet­ter, but I also want to code. And I can’t code with GitHub

any­more. I’m sorry. After 18 years, I’ve got to go. I’d love to come back one

day, but this will have to be pred­i­cated on real re­sults and im­prove­ments,

not words and promises.

I’ll share more de­tails about where the Ghostty pro­ject will be mov­ing to in

the com­ing months. We have a plan but I’m also very much still in dis­cus­sions

with mul­ti­ple providers (both com­mer­cial and FOSS).

It’ll take us time to re­move all of our de­pen­den­cies on GitHub and we have a

plan in place to do it as in­cre­men­tally as pos­si­ble. We plan on keep­ing a

read-only mir­ror avail­able on GitHub at the cur­rent URL.

My per­sonal pro­jects and other work will re­main on GitHub for now.

Ghostty is where I, our main­tain­ers, and our open source com­mu­nity are

most im­pacted so that is the fo­cus of this change. We’ll see where it

goes af­ter that.

Footnotes

The tim­ing of this is co­in­ci­den­tal with the large out­age on April 27, 2026.

We’ve been dis­cussing and putting to­gether a plan to leave GitHub

for months, and this blog post was writ­ten over a week ago. We only

made the fi­nal de­ci­sion this week. ↩

The tim­ing of this is co­in­ci­den­tal with the large out­age on April 27, 2026.

We’ve been dis­cussing and putting to­gether a plan to leave GitHub

for months, and this blog post was writ­ten over a week ago. We only

made the fi­nal de­ci­sion this week. ↩

To the “Git is dis­trib­uted!” crowd: the is­sue is­n’t Git, it’s the

in­fra­struc­ture we rely on around it: is­sues, PRs, Actions, etc. ↩

To the “Git is dis­trib­uted!” crowd: the is­sue is­n’t Git, it’s the

in­fra­struc­ture we rely on around it: is­sues, PRs, Actions, etc. ↩

This is not the large Elasticsearch out­age they had on April 27, 2026.

This blog post was writ­ten a week be­fore that, so this was a dif­fer­ent

out­age. ↩

This is not the large Elasticsearch out­age they had on April 27, 2026.

This blog post was writ­ten a week be­fore that, so this was a dif­fer­ent

out­age. ↩

Your phone is about to stop be­ing yours.

126 days un­til lock­down

Starting September 2026, a silent up­date, non­con­sen­su­ally pushed by Google, will block every Android app whose de­vel­oper has­n’t reg­is­tered with Google, signed their con­tract, paid up, and handed over gov­ern­ment ID.

Every app and every de­vice, world­wide, with no opt-out.

Post on X Post on Mastodon Post on Bluesky LinkedIn Facebook

What Google is do­ing

In August 2025, Google an­nounced a new re­quire­ment: start­ing September 2026, every Android app de­vel­oper must reg­is­ter cen­trally with Google be­fore their soft­ware can be in­stalled on any de­vice. Not just Play Store apps: all apps. This in­cludes apps shared be­tween friends, dis­trib­uted through F-Droid, built by hob­by­ists for per­sonal use. Independent de­vel­op­ers, church and com­mu­nity groups, and hob­by­ists alike will all be frozen out of be­ing able to de­velop and dis­trib­ute their soft­ware.

Registration re­quires:

Paying a fee to Google

Agreeing to Google’s Terms and Conditions

Surrendering your gov­ern­ment-is­sued iden­ti­fi­ca­tion

Providing ev­i­dence of your pri­vate sign­ing key

Listing all cur­rent and all fu­ture ap­pli­ca­tion iden­ti­fiers

If a de­vel­oper does not com­ply, their apps get silently blocked on every Android de­vice world­wide.

Who this hurts

You

You bought an Android phone be­cause Google told you it was open. You could in­stall what you wanted, and that was the deal.

Google is now rewrit­ing that deal, retroac­tively, on hard­ware you al­ready own. After the up­date lands, you can only run soft­ware that Google has pre-ap­proved. On your phone: your prop­erty, that you paid for.

Independent de­vel­op­ers

A teenager’s first app, a vol­un­teer’s pri­vacy tool, or a com­pa­ny’s con­fi­den­tial in­ter­nal beta. It does­n’t mat­ter. After September 2026, none of these can be in­stalled with­out Google’s bless­ing.

F-Droid, home to thou­sands of free and open-source Android apps, has called this an “existential” threat. Cory Doctorow calls it “Darth Android”.

Governments & civil so­ci­ety

Google has a doc­u­mented track record of com­ply­ing when au­thor­i­tar­ian regimes de­mand app re­movals. With this pro­gram, the soft­ware that runs your coun­try’s in­sti­tu­tions will ex­ist at the plea­sure of a sin­gle un­ac­count­able for­eign cor­po­ra­tion.

The EFF calls app gate­keep­ing “an ever-ex­pand­ing path­way to in­ter­net cen­sor­ship.”

Google’s “escape hatch” is a trap door

Google says “power users” can “still in­stall” un­ver­i­fied apps. Here’s what that ac­tu­ally looks like:

Delve into System Settings, find Developer Options

Tap the build num­ber seven times to en­able Developer Mode

Dismiss scare screens about co­er­cion

Enter your PIN

Restart the de­vice

Wait 24 hours

Come back, dis­miss more scare screens

Pick “allow tem­porar­ily” (7 days) or “allow in­def­i­nitely”

Confirm, again, that you un­der­stand “the risks”

Nine steps. A manda­tory 24-hour cool­ing-off pe­riod. For in­stalling soft­ware on a de­vice you own.

Worse: this flow runs en­tirely through Google Play Services, not the Android OS. Google can change it, tighten it, or kill it at any time, with no OS up­date re­quired and no con­sent needed. And as of to­day, it has­n’t shipped in any beta, pre­view, or ca­nary build. It ex­ists only as a blog post and some mock­ups.

This is big­ger than Android

If Google can retroac­tively lock down bil­lions of de­vices that were sold as open plat­forms, every hard­ware man­u­fac­turer on the planet is watch­ing.

The prin­ci­ple be­ing es­tab­lished: the com­pany that made your de­vice gets to de­cide, af­ter you’ve bought it, what soft­ware you’re al­lowed to run. In soft­ware, this is called a “rug pull”; but at least you could al­ways in­stall com­pet­ing soft­ware. In hard­ware, it is a fait ac­com­pli that strips you of your agency and ren­ders you pow­er­less to the whims of a sin­gle un­ac­count­able gate­keeper and con­victed mo­nop­o­list.

Android’s open­ness was never just a fea­ture. It was the promise that dis­tin­guished it from iPhone. Millions chose Android for ex­actly that rea­son. Google is now re­vok­ing that promise uni­lat­er­ally, on de­vices al­ready in peo­ple’s pock­ets, be­cause they’ve de­cided they have enough mar­ket dom­i­nance and reg­u­la­tory cap­ture to get away with it.

Ars Technica: “Google’s Apple envy threat­ens to dis­man­tle Android’s open legacy.”

But wait, is­n’t this…

″…just about se­cu­rity?”

The se­cu­rity ra­tio­nale is a smoke­screen. Google Play Protect al­ready scans for mal­ware in­de­pen­dent of de­vel­oper iden­tity. Requiring a gov­ern­ment ID does­n’t make code safer. It makes de­vel­op­ers iden­ti­fi­able and con­trol­lable. Malware au­thors can reg­is­ter. Indie de­vel­op­ers and dis­si­dents of­ten can’t. The EFF is blunt: iden­tity-based gate­keep­ing is a cen­sor­ship tool, not a se­cu­rity one.

″…still side­load­ing if you use the ad­vanced flow?”

Nine steps, 24-hour wait, buried in Developer Options, de­liv­ered through a pro­pri­etary ser­vice that Google can re­voke when­ever they want. That’s not side­load­ing. That’s a de­ter­rence mech­a­nism built to en­sure al­most no­body com­pletes it. And since it runs through Play Services rather than the OS, Google can tighten or kill it silently.

″…only a prob­lem if you have some­thing to hide?”

Whistleblowers, jour­nal­ists, and ac­tivists un­der au­thor­i­tar­ian gov­ern­ments will be the first vic­tims. People in do­mes­tic abuse sit­u­a­tions are next. All these groups have le­git­i­mate rea­sons to dis­trib­ute or use soft­ware with­out putting their le­gal iden­tity in a Google data­base. Anonymous open-source con­tri­bu­tion is a tra­di­tion older than Google it­self. This pol­icy ends it on Android.

″…the same thing Apple does?”

Apple has been a walled gar­den from day one. People chose Android be­cause it was dif­fer­ent. “Apple does it too” is a race to the bot­tom and a weak tu quoque ar­gu­ment. And un­der reg­u­la­tory pres­sure (the EU’s Digital Markets Act), even Apple is be­ing forced to open up. Google is mov­ing in the op­po­site di­rec­tion: at­tempt­ing to fur­ther en­trench its gate­keep­ing sta­tus.

″…just $25 and some pa­per­work?”

Maybe, if you’re a de­vel­oper in the US with a credit card and a dri­ver’s li­cense. Try be­ing a stu­dent in sub-Sa­ha­ran Africa, or a dis­si­dent in Myanmar, or a vol­un­teer main­tain­ing a com­mu­nity health app. The cost is­n’t only fi­nan­cial: you’re sur­ren­der­ing gov­ern­ment ID and ev­i­dence of your sign­ing keys to a com­pany that rou­tinely com­plies with gov­ern­ment de­mands to re­move apps and ex­pose de­vel­op­ers.

Fight back

Everyone

Install F-Droid on every Android de­vice you own. Alternative stores only sur­vive if peo­ple ac­tu­ally use them.

Contact your reg­u­la­tors. Regulators world­wide are gen­uinely con­cerned about mo­nop­o­lies and the cen­tral­iza­tion of power in the tech sec­tor, and want to hear di­rectly from in­di­vid­u­als who are af­fected and con­cerned.

Share this page. Link to keepan­droidopen.org every­where.

Push back on as­tro­turfers. The “well, ac­tu­ally…” crowd is out in force. Don’t let them set the nar­ra­tive.

Sign the change.org pe­ti­tion and join the over 100,000 sig­na­to­ries who have made their voices heard.

Read and share our open let­ter

Tell Google what you think of this through their own de­vel­oper ver­i­fi­ca­tion sur­vey (for all the good that will do).

Developers

Do not sign up. Don’t join the pro­gram by sign­ing up for the Android Developer Console and agree­ing to their ir­rev­o­ca­ble Terms and Conditions. Don’t ver­ify your iden­tity. Don’t play ball.

Google’s plan only works if de­vel­op­ers com­ply. Don’t.

Talk other de­vel­op­ers and or­ga­ni­za­tions out of sign­ing up.

Add the FreeDroidWarn li­brary to your apps to warn users.

Run a web­site? Add the count­down ban­ner.

Google em­ploy­ees

If you know some­thing about the pro­gram’s tech­ni­cal im­ple­men­ta­tion or in­ter­nal ra­tio­nale, con­tact tips@keepan­droidopen.org from a non-work ma­chine and a non-Gmail ac­count. Strict con­fi­dence guar­an­teed.

All those op­posed…

69 or­ga­ni­za­tions from 21 coun­tries have signed the open let­ter

Read the full open let­ter and thank the sig­na­to­ries →

What they’re say­ing

Tech press

“Google’s Android de­vel­oper ver­i­fi­ca­tion pro­gram draws push­back” InfoWorld

“Google’s Android de­vel­oper ver­i­fi­ca­tion pro­gram draws push­back”

“Google’s New Developer ID Rule Could Harm F-Droid” Reclaim The Net

“Google’s New Developer ID Rule Could Harm F-Droid”

“Google will make you wait 24 hours to side­load Android apps” How-To Geek

“Google will make you wait 24 hours to side­load Android apps”

“An ‘existential’ threat to al­ter­na­tive app stores” The New Stack

“An ‘existential’ threat to al­ter­na­tive app stores”

“I’ve been an Android user for al­most 15 years — and Google’s side­load­ing changes are push­ing me back to iPhone” Tom’s Guide

“I’ve been an Android user for al­most 15 years — and Google’s side­load­ing changes are push­ing me back to iPhone”

“F-Droid Says Google Is Lying About the Future of Sideloading on Android” How-To Geek

“F-Droid Says Google Is Lying About the Future of Sideloading on Android”

“Android Security or Vendor Lock-In? Google’s New Sideloading Rules Smell Fishy” It’s FOSS News

“Android Security or Vendor Lock-In? Google’s New Sideloading Rules Smell Fishy”

“Keep Android Open” Linux Magazine

“Keep Android Open”

“F-Droid says Google’s new side­load­ing re­stric­tions will kill the pro­ject” Ars Technica

“F-Droid says Google’s new side­load­ing re­stric­tions will kill the pro­ject”

“Sideloading is dead for all in­tents and pur­poses. The Android you know and love is slowly dis­ap­pear­ing.” Android Police

“Sideloading is dead for all in­tents and pur­poses. The Android you know and love is slowly dis­ap­pear­ing.”

“F-Droid pro­ject threat­ened by Google’s new dev reg­is­tra­tion rules” Bleeping Computer

“F-Droid pro­ject threat­ened by Google’s new dev reg­is­tra­tion rules”

“Google’s new de­vel­oper rules could threaten side­load­ing and F-Droid’s fu­ture” Gizmochina

“Google’s new de­vel­oper rules could threaten side­load­ing and F-Droid’s fu­ture”

“Open let­ter warns manda­tory reg­is­tra­tion ‘threatens in­no­va­tion, com­pe­ti­tion, pri­vacy and user free­dom’” Infosecurity Magazine

File read

Shell

Shell

Upload

Download

Command

Shell

File read

Shell

Shell

Shell

File read

File read

Inherit

Shell

Command

File write

File read

Inherit

Shell

Inherit

Shell

Command

File write

File read

Inherit

Shell

Inherit

Shell

Command

File write

File read

Inherit

Inherit

Shell

Command

File write

File read

Inherit

File read

Shell

Command

File read

Download

File write

File read

File read

File read

File read

File read

Shell

File write

File read

Shell

Shell

Command

File read

Shell

Shell

Shell

Shell

File write

File read

File read

Inherit

Shell

Command

File write

File read

Inherit

File read

File read

File read

File read

File read

Shell

Reverse shell

File write

File read

Upload

Download

Library load

Inherit

Shell

File write

File read

Inherit

Shell

Command

File write

File read

Inherit

File read

File read

Shell

File read

Inherit

Shell

Command

Reverse shell

The phones we carry around in our pock­ets have two mil­lion times more mem­ory and are thou­sands of times faster than the room-sized com­put­ers that guided the Apollo mis­sion to the Moon. This in­cred­i­ble shrink­ing act has been dri­ven by our abil­ity to make tran­sis­tors smaller and smaller.

Each tran­sis­tor is a mi­cro­scopic switch that can al­ter­nate be­tween a one and a zero, the ba­sic lan­guage of all com­put­ing. Billions are packed onto tiny sil­i­con chips called semi­con­duc­tors. The more tran­sis­tors that fit onto a chip, the more logic and mem­ory cir­cuits it holds, and the more it can do.

Get the print mag­a­zine

Subscribe for $100 to re­ceive six beau­ti­ful is­sues per year.

Subscribe

Advanced semi­con­duc­tors are, ar­guably, the most im­por­tant tech­nol­ogy in the world. Over the last five years, they have even emerged as a geopo­lit­i­cal flash­point be­tween the US and China. But for all this ri­valry, any coun­try or com­pany that hopes to man­u­fac­ture semi­con­duc­tors is de­pen­dent on a sin­gle firm: ASML. Dubbed ‘a rel­a­tively ob­scure Dutch com­pa­ny’ by the BBC in 2020, ASML makes the only ma­chines in the world ca­pa­ble of sten­cil­ing the tran­sis­tors onto chips with the pre­ci­sion nec­es­sary to fit bil­lions on a 30-centimeter wafer.

These ma­chines are roughly the size of dou­ble-decker buses. To ship one re­quires 40 freight con­tain­ers, three cargo planes, and 20 trucks. They are the world’s most com­plex ob­jects. Each con­tains over one hun­dred thou­sand com­po­nents, all of which have to be per­fectly cal­i­brated for the ma­chine to pro­duce light con­sis­tently at the right wave­length.

While ASML is now the sole sup­plier of these ma­chines, and will be for some time to come, it started out as a lag­gard in the chip­mak­ing in­dus­try. Overtaking its com­pe­ti­tion re­quired many things rarely as­so­ci­ated with European com­pa­nies: close col­lab­o­ra­tion with the American gov­ern­ment, sell­ing large stakes to for­eign com­peti­tors, and a huge gam­ble on an un­proven tech­nol­ogy.

Let there be light

The key to ASML’s suc­cess is a tech­nol­ogy called pho­tolith­o­g­ra­phy (sometimes just called lith­o­g­ra­phy). The tech­nique in­volves trans­fer­ring a pat­tern onto a semi­con­duc­tor wafer by ex­pos­ing it to light. In the 1950s, the first chip­mak­ers had tried to draw these pat­terns by hand, but any­thing that phys­i­cally touches the wafer scratches it, dirt­ies it, or warps the pat­tern. Scientists work­ing in­de­pen­dently for Bell Labs and the US mil­i­tary re­al­ized that they could use light to print iden­ti­cal pat­terns with­out mak­ing phys­i­cal con­tact with the wafer.

To make chips, en­gi­neers start with a thin wafer of semi­con­duc­tor ma­te­r­ial, usu­ally sil­i­con. This wafer is coated with a chem­i­cal called pho­tore­sist, which re­acts when ex­posed to light. In pho­tolith­o­g­ra­phy, light is pro­jected through a de­tailed pat­tern onto the pho­tore­sist-coated wafer, soft­en­ing the ex­posed ar­eas. The wafer is washed to re­move any soft­ened ar­eas, re­veal­ing the sil­i­con un­der­neath. It is then moved to an etch­ing ma­chine that blasts it with charged chlo­rine or bromine gas, carv­ing the de­sired pat­tern into the ex­posed sil­i­con. These fea­tures are later filled with metal, such as tung­sten and cop­per, to con­nect the tran­sis­tor to power. These etched lay­ers then com­bine into an in­tri­cate net­work of tran­sis­tors.

Over time, the semi­con­duc­tor man­u­fac­tur­ing ecosys­tem has de­vel­oped in­creas­ingly so­phis­ti­cated etch­ing us­ing ever smaller wave­lengths of light. Smaller wave­lengths dif­fract less, al­low­ing the light to travel in straighter lines and print sharper, tinier de­tails with­out blur­ring. These al­low for more pre­cise pat­tern pro­jec­tions that, in turn, al­low smaller and more densely packed tran­sis­tors.

Early lith­o­g­ra­phy re­lied on mer­cury va­por lamps that were sim­i­lar to street­lights, while more mod­ern ma­chines rely on lasers cre­ated us­ing ar­gon and flu­o­rine gases. By 2010, such lasers made it pos­si­ble to cre­ate a 22-nanometer fea­ture through mul­ti­ple ex­po­sures us­ing a 193-nanometer wave­length.

The most ad­vanced ver­sion of this tech­nol­ogy, ex­treme ul­tra­vi­o­let lith­o­g­ra­phy, is used to make the very small­est chips. The small­est in 2025 were mar­keted as three nanome­ters, roughly 25,000 times thin­ner than a hu­man hair.

To make them, a droplet of liq­uid tin is re­leased into a cham­ber and hit with a sin­gle pulse of light, which melts and flat­tens it. As the droplet con­tin­ues to fall, a sec­ond, more pow­er­ful pulse va­por­izes the tin, cre­at­ing an ex­tremely hot plasma that emits light at the nar­row wave­lengths needed for ex­treme ul­tra­vi­o­let lith­o­g­ra­phy. The light beam is then con­cen­trated by re­flect­ing it across a se­ries of slightly con­cave mir­rors so flaw­less that, if scaled to the size of Germany, their im­per­fec­tions would be mea­sured in mil­lime­ters. Engineers need to use mir­rors, rather than the glass lenses used in stan­dard lith­o­g­ra­phy, as al­most all solid ma­te­ri­als ab­sorb light at such short wave­lengths.

The light even­tu­ally hits the mask, which con­tains the pat­tern to be printed on the chip. As the pat­tern on the mask is usu­ally sev­eral times larger than what is wanted on the chip, the light is then re­flected by a sec­ond sys­tem of mir­rors.

Path of light through an ex­treme ul­tra­vi­o­let lith­o­g­ra­phy scan­ner.

Image

ASML.

After the light re­flects from the mask, it car­ries the pat­tern as a bun­dle of rays spread­ing out from each point. The next mir­rors tip these rays in­ward so that, in­stead of spread­ing widely, they re­unite over a shorter dis­tance. When the rays from each point come to­gether sooner, the pic­ture they form is phys­i­cally smaller. By re­peat­ing this with sev­eral care­fully shaped mir­rors, en­gi­neers shrink the pat­tern by a fixed amount while keep­ing it in fo­cus. After be­ing shrunk four times, it hits the wafer.

The great shrink­ing act

Longer wave­lengths act like a blunt chisel, suit­able for rough shap­ing, but they strug­gle to cap­ture finer de­tails. The longer light waves are larger rel­a­tive to the tiny fea­tures on the ret­i­cle that they must re­flect from. When a wave meets some­thing smaller than it­self, it nat­u­rally spreads and bends around its edges in­stead of cast­ing a sharp shadow. To cre­ate the same de­tails, the blunt chisel needs to go over the same spot a num­ber of times (creating blur­rier edges). Lithography had to take wave­lengths all the way to the ex­treme ul­tra­vi­o­let range to achieve the high res­o­lu­tion pat­tern­ing needed for cut­ting-edge process nodes.

Wavelengths as low as 13.5 nanome­ters can achieve more pre­cise pat­terns in a sin­gle ex­po­sure. In fact, ex­treme ul­tra­vi­o­let lith­o­g­ra­phy can com­bine three or four pho­tolith­o­g­ra­phy pat­tern­ing cy­cles into a sin­gle one on a seven-nanome­ter node. Without EUV, pro­duc­ing five-nanome­ter nodes might re­quire as many as one hun­dred dif­fer­ent steps.

Extreme ul­tra­vi­o­let lith­o­g­ra­phy was able to pro­duce more ac­cu­rate pat­terns on wafers than older tech­niques even if they were used mul­ti­ple times.

Today, ASML dom­i­nates the over­all mar­ket for lith­o­g­ra­phy and has an ef­fec­tive mo­nop­oly in ex­treme ul­tra­vi­o­let lith­o­g­ra­phy. Its EUV ma­chines sell for more than $120 mil­lion. With a mar­ket cap­i­tal­iza­tion of over $400 bil­lion, ASML is one of Europe’s most valu­able com­pa­nies. But it was­n’t al­ways like this.

Origins

ASML started off life within Philips, the Dutch con­sumer elec­tron­ics gi­ant. During the 1970s, Philips had roughly 20 per­cent of the global elec­tron­ics mar­ket and was a ma­jor chip­maker. In this era, lith­o­g­ra­phy ma­chines used wave­lengths of over 400 nanome­ters to pat­tern 1,000-nanometer fea­tures. The in­dus­try strug­gled to shrink fea­tures with­out los­ing ac­cu­racy or let­ting dust and flaws creep in. Philips be­gan to work on its own pro­to­type, draw­ing on its ex­per­tise in op­tics and pre­ci­sion me­chan­ics. By the early 1980s, the pro­ject was run­ning into trou­ble. The com­pany was look­ing to cut costs and en­gi­neers es­ti­mated that they would need over $280 mil­lion in to­day’s money to fin­ish the ma­chine’s de­vel­op­ment and pro­duc­tion.

In 1984, Philips spun out Advanced Semiconductor Materials Lithography (which later dropped the full name in fa­vor of its acronym) as a joint ven­ture with ASM International, a Dutch con­glom­er­ate that sold equip­ment to the semi­con­duc­tor in­dus­try. The busi­ness orig­i­nally strug­gled. It had no mar­ket share and no brand recog­ni­tion. Its first prod­uct, the PAS 2000, was a com­mer­cial fail­ure. The ma­chine used oil pres­sure, like that in power steer­ing, to move the table that held the wafer dur­ing ex­po­sure, rather than elec­tric mo­tors. This made it smooth and pre­cise, but it was prone to leak­ing. At the first con­fer­ence ASML at­tended, one in­dus­try ex­ec­u­tive told them: ‘The race has al­ready been run. There’s no room for you here.’ ASML switched back to elec­tric mo­tors.

The com­pany took an un­usual ap­proach from the out­set. While Japanese gi­ants Nikon and Canon were ver­ti­cally in­te­grated, ASML out­sourced key com­po­nents like op­tics and mo­tors so that it could fo­cus on as­sem­bling and op­ti­miz­ing the fi­nal ma­chine. Given this out­sourc­ing, it made sense for ASML to em­brace a mod­u­lar de­sign with clearly de­fined sub­sys­tems. This ap­proach was mocked in European man­u­fac­tur­ing cir­cles. German en­gi­neers warned ASML’s lead­er­ship that they were ‘asking for trou­ble’ and would ‘lose all con­trol’ if they did­n’t make crit­i­cal com­po­nents them­selves. But ASML had no choice: it lacked the cap­i­tal, ex­per­tise, and time to build these sub­sys­tems from scratch.

By 1988, ASML was on the verge of col­lapse. ASM International had al­ready pulled out, and Philips con­sid­ered shut­ting it down. It was saved by a sin­gle Philips board mem­ber, Gerd Lorenz, who was par­tic­u­larly wor­ried about Europe’s grow­ing de­pen­dence on Asia for strate­gic tech­nol­ogy. Lorenz ar­gued that Europe needed a stake in chip man­u­fac­tur­ing. This was enough to con­vince Philips to give ASML more time, but did­n’t fix its fun­da­men­tal prob­lem: it was still an in­fe­rior sup­plier with no com­pet­i­tive edge.

ASML used the time it was given to de­velop the PAS 5500, re­leased in 1991 and the com­pa­ny’s first com­mer­cial break­out. While Nikon’s con­tem­po­rary pho­tolith­o­g­ra­phy sys­tem was more pre­cise, ASML’s mod­u­lar de­sign meant that ma­chines could be fixed quickly on site. This re­duced down­time and, by mak­ing it easy to re­place parts when they broke, it was pos­si­ble to ex­tend the ma­chine’s life. This was a key fac­tor that led John Kelly, IBM’s di­rec­tor of semi­con­duc­tor R&D, to push IBM to or­der the PAS 5500 over the Japanese ma­chines. ASML had gone global.

The first break­throughs

ASML’s suc­cess de­pended on two pro­jects in the late 1990s and 2000s that gave it a huge ad­van­tage in re­search and de­vel­op­ment. The first was a pub­lic-pri­vate part­ner­ship, started in 1997, called the Extreme Ultraviolet Limited Liability Company. The Extreme Ultraviolet Limited Liability Company be­gan life as a res­cue mis­sion. Before 1997, ba­sic semi­con­duc­tor re­search was car­ried out in a small hand­ful of re­search labs, all de­pen­dent on gov­ern­ment grants.

The orig­i­nal pro­gram for EUV re­search was a ‘virtual na­tional lab’ that com­bined Lawrence Livermore National Laboratory, Sandia National Laboratories, and the Lawrence Berkeley National Laboratory. Each cov­ered a dif­fer­ent com­po­nent: Livermore fo­cused on mir­rors and op­tics, Sandia on the light source and sys­tems en­gi­neer­ing, and Berkeley on ad­vanced equip­ment for test­ing. But in 1996, Department of Energy bud­get cuts had placed the vir­tual na­tional lab pro­gram on the chop­ping block.

Intel, then the undis­puted world leader in mi­cro­proces­sors, was keen to pre­serve the work and spear­headed the cre­ation of the Extreme Ultraviolet Limited Liability Company, the largest pub­lic-pri­vate part­ner­ship of its kind in the his­tory of the US Department of Energy. During its six-year life, the com­pany in­vested over $270 mil­lion into ex­treme ul­tra­vi­o­let lith­o­g­ra­phy de­vel­op­ment, funded by the sale of shares to mem­ber com­pa­nies, giv­ing them a right of first re­fusal to pur­chase the pho­tolith­o­g­ra­phy tools be­ing pro­duced.

The com­pany ini­tially re­stricted mem­ber­ship to American firms. ASML, along with its main Japanese ri­vals, Canon and Nikon, was ini­tially barred from mem­ber­ship.

The only es­tab­lished semi­con­duc­tor equip­ment man­u­fac­turer to join the part­ner­ship from the be­gin­ning was Silicon Valley Group, which had a mar­ket share of just 5 per­cent to ASML’s 20 per­cent. Fearing the dan­ger of be­ing re­liant on such a small man­u­fac­turer, the rest of the com­pa­nies in­volved con­cluded that it would be bet­ter to open up to for­eign firms, rather than risk ced­ing the en­tire mar­ket.

ASML was al­lowed to par­tic­i­pate so long as it com­mit­ted to es­tab­lish a re­search cen­ter in the US and source 55 per­cent of com­po­nents for the sys­tems sold in the US from American sup­pli­ers. In prac­tice, this com­mit­ment was never en­forced. Its Japanese com­peti­tors were never al­lowed to join, due to wide­spread fear in the US of Japanese com­pe­ti­tion.

The pro­gram built up a vast base of in­tel­lec­tual prop­erty and process knowl­edge. These types of pub­lic-pri­vate part­ner­ships typ­i­cally grant the par­tic­i­pat­ing com­pa­nies a non-ex­clu­sive li­cense to use the in­tel­lec­tual prop­erty gen­er­ated, but in this case the com­pa­nies in part­ner­ship got com­plete own­er­ship.

In 2001, ASML ac­quired Silicon Valley Group af­ter it ran into cash flow dif­fi­cul­ties, mak­ing ASML the sole sur­viv­ing equip­ment man­u­fac­turer in the part­ner­ship. When the con­sor­tium pro­duced the first full-scale ex­treme ul­tra­vi­o­let lith­o­g­ra­phy pro­to­type — the Engineering Test Stand — ASML stood alone at the van­guard of lith­o­g­ra­phy. This was the first demon­stra­tion that 13.5-nanometer light could print dense fea­tures on a chip.

By the time the Engineering Test Stand was built, the pro­gram had al­ready proved that it was pos­si­ble to gen­er­ate ex­treme ul­tra­vi­o­let light re­li­ably, which let en­gi­neers start build­ing mir­rors and lenses that could be used in real pro­duc­tion tools. To solve out­stand­ing ques­tions, such as how to boost the through­put of their ma­chines or in­crease the power of their light sources in pro­duc­tion set­tings, ASML needed to test its ma­chines in en­vi­ron­ments close to the real world. But no chip­mak­ers were will­ing to shoul­der a pro­ject so large and risky at such an early stage.

The sec­ond pro­ject es­sen­tial to ASML’s suc­cess was the Belgium-based Interuniversity Microelectronics Centre (IMEC), a re­search or­ga­ni­za­tion that col­lects ma­chines from dif­fer­ent com­pa­nies and al­lows re­searchers to test them in semi-real en­vi­ron­ments while pro­tect­ing the com­pa­nies’ in­tel­lec­tual prop­erty.

As po­ten­tial cus­tomers be­gan to con­sider dif­fer­ent op­tions for next gen­er­a­tion lith­o­g­ra­phy tech­nolo­gies, ASML used IMEC to pro­mote its ex­treme ul­tra­vi­o­let lith­o­g­ra­phy pro­to­type. Topping ASML’s tar­get list was TSMC, which to­day is the world’s largest semi­con­duc­tor foundry. Founded in 1987, TSMC’s his­tory had been in­ter­twined with ASML’s since its birth: Philips, ASML’s for­mer par­ent, owned a 27.5 per­cent stake in it. Seeing ASML’s ma­chin­ery ex­hib­ited at IMEC was what led TSMC to part­ner with ASML in EUV de­vel­op­ment.

By con­trast, Canon and Nikon were tight-lipped about their re­search and made lit­tle ef­fort to co­op­er­ate with out­side com­pa­nies. While this the­o­ret­i­cally al­lowed them to main­tain greater con­trol over their work, and cap­ture more of the value chain, it also made them solely re­spon­si­ble for si­mul­ta­ne­ously solv­ing a be­wil­der­ing ar­ray of fun­da­men­tal physics prob­lems, while as­sum­ing all the fi­nan­cial risk of do­ing so.

ASML’s pro­to­type ex­treme ul­tra­vi­o­let lith­o­g­ra­phy sys­tem.

Image

ASML.

Since al­most all of the parts in ASML’s ma­chines are made by other com­pa­nies, it has be­come mas­ter of a sprawl­ing sup­ply chain of over five thou­sand com­pa­nies. It has di­ver­si­fied its sup­pli­ers over the years in a very de­lib­er­ate way: 80 per­cent of its spend­ing goes to com­pa­nies across Europe and the Middle East (notably not the US, de­spite prior agree­ments), which re­duces the risk of po­ten­tial ex­port re­stric­tions, tar­iffs, and other geopo­lit­i­cal risks that may face crit­i­cal sup­pli­ers based in the US or Asia. It also aims for its sup­pli­ers to make no more than 25 per­cent of their rev­enue from ASML, to force them not to be­come over­re­liant on the volatile semi­con­duc­tor mar­ket.

While most of its com­po­nents come from a large num­ber of small sup­pli­ers, ASML has formed deep bonds with its biggest sup­pli­ers. It ac­quired a 24.9 per­cent stake in op­tics man­u­fac­turer Zeiss. Peter Leibinger, vice chair­man of laser man­u­fac­turer Trumpf, has said that ASML and Trumpf are a ‘virtually merged com­pa­ny’.

Winning the war

Extreme ul­tra­vi­o­let lith­o­g­ra­phy would not be­come a suc­cess­ful com­mer­cial tech­nol­ogy un­til 2018, over 20 years af­ter the cre­ation of the Extreme Ultraviolet Limited Liability Company and 34 years af­ter IMEC was founded. In the mean­time, it was con­sum­ing more and more re­sources. By 2015, ASML was spend­ing more than $1 bil­lion a year on R&D, more than dou­ble its 2010 to­tal. According to some es­ti­mates, by 2014, the in­dus­try had col­lec­tively in­vested over $20 bil­lion in ex­treme ul­tra­vi­o­let lith­o­g­ra­phy, with no guar­an­tee of any re­turn.

ASML was able to con­tinue pour­ing money into this black hole partly be­cause it had al­ready beaten its com­peti­tors. By 2010, it had two thirds of the over­all lith­o­g­ra­phy mar­ket and was the dom­i­nant sup­plier for the rapidly grow­ing smart­phone mar­ket, with deep ties to Intel, Samsung, and TSMC. It had se­cured this po­si­tion by win­ning the de­ci­sive tech­ni­cal bat­tle of the 2000s.

At the start of the mil­len­nium, the en­tire semi­con­duc­tor in­dus­try hit a phys­i­cal wall. Circuits had been get­ting steadily smaller for decades by sim­ply switch­ing to shorter wave­lengths, but the stan­dard 193-nanometer light (roughly one five-hun­dredth of the thick­ness of a hu­man hair) was too blunt to draw smaller cir­cuits.

Nikon tried to solve this by de­vel­op­ing a new light source with a smaller wave­length of 157 nanome­ters. But this shorter wave­length light was ab­sorbed and dis­torted by stan­dard glass, forc­ing Nikon to build lenses out of cal­cium flu­o­ride, a rare, brit­tle crys­tal that was ex­pen­sive to pol­ish and prone to crack­ing un­der heat. The in­dus­try poured hun­dreds of mil­lions of dol­lars into this ‘dry’ lith­o­g­ra­phy path, only to find the man­u­fac­tur­ing chal­lenges in­sur­mount­able.

ASML’s part­ner­ships helped it avoid this dead end. TSMC re­searcher Burn Lin had ad­vised them to switch to a tech­nol­ogy called im­mer­sion lith­o­g­ra­phy. ASML con­tin­ued to use 193-nanometer light but placed a layer of wa­ter be­tween the lens and the sil­i­con wafer. Just as a straw ap­pears bent and mag­ni­fied when placed in a glass of wa­ter, the wa­ter in the ma­chine bent the light waves, sharp­en­ing the fo­cus and al­low­ing smaller cir­cuits to be printed with­out need­ing new lenses.

ASML com­pounded this ad­van­tage by in­tro­duc­ing a rev­o­lu­tion­ary ma­chine ar­chi­tec­ture called TWINSCAN. In older ma­chines, the light source would sit idle while the ma­chine stopped to mea­sure the sur­face of the sil­i­con wafer to en­sure it was flat. ASML re­placed this with a dual-stage sys­tem: a mas­sive ma­chine with two ta­bles would mea­sure one wafer in the back­ground while an­other was be­ing printed si­mul­ta­ne­ously. This elim­i­nated the dead time in the man­u­fac­tur­ing process, al­low­ing chip­mak­ers to pro­duce sig­nif­i­cantly more chips per hour. By the time Nikon aban­doned its 157-nanometer pro­ject in 2005, ASML had be­come the in­dus­try stan­dard, with 53.2 per­cent of the mar­ket.

ASML’s ma­chines were so much bet­ter than the com­pe­ti­tion that it could charge nearly twice as much for them: $55 mil­lion ver­sus $30 mil­lion for the com­pa­ra­ble Nikon de­vice.

But even this was not enough. While ASML was be­gin­ning to ship pro­to­type EUV ma­chines to IMEC from 2006 on­wards, they were so slow and prone to break­ing down that they were com­mer­cially use­less. In 2012, ASML, still reel­ing from the global fi­nan­cial cri­sis, was strug­gling to con­tinue fi­nanc­ing its EUV ef­forts.

In a dras­tic move — part des­per­ate at­tempt to keep the com­pa­ny’s re­search ef­forts afloat and part strate­gic bet to win the EUV mar­ket once and for all — the ASML lead­er­ship launched a co-in­vest­ment pro­gram that sold 23 per­cent of the com­pany to its three largest cus­tomers: Intel, TSMC and Samsung.

The fund­ing also al­lowed ASML to com­plete a $2.5 bil­lion ac­qui­si­tion of one of its sup­pli­ers, Cymer, which pro­duces lith­o­g­ra­phy light sources. The ac­qui­si­tion al­lowed ASML to in­vest in Cymer’s R&D ef­forts to per­fect its soft X-ray light source, which in­volved hit­ting fast-mov­ing droplets of tin with such force that they lost elec­trons, but pre­cisely enough that this did not shed so much de­bris that it coated the mir­rors. They ac­com­plished this by mov­ing from a sin­gle pulse to two sep­a­rate laser pulses: the pre-pulse would shape the droplet and the main pulse would gen­er­ate the plasma. This im­proved ef­fi­ciency and sta­bil­ity.

ASML’s close part­ner­ship with TSMC proved es­pe­cially crit­i­cal. In 2014, TSMC launched its first chip for Apple, which was now its largest cus­tomer and was putting pres­sure on the chip­maker to pro­duce higher per­for­mance chips than its ex­ist­ing ma­chin­ery was ca­pa­ble of. It had be­come ur­gent for ASML to com­plete a com­mer­cial EUV ma­chine.

The two com­pa­nies worked so closely to­gether that Anthony Yen, the Division Director at TSMC re­spon­si­ble for over­see­ing EUV de­vel­op­ment, de­scribed them as ‘one team’. ASML and TSMC en­gi­neers on the ground worked tire­lessly, trou­bleshoot­ing and it­er­at­ing un­til they had reached the nec­es­sary through­put: 500 wafers a day for a month.

During this pe­riod, the joint team re­designed both the tin-droplet gen­er­a­tor and the way the laser hit each droplet. The new setup pro­duced droplets that were about half the orig­i­nal size while still yield­ing the same ul­tra­vi­o­let en­ergy. Smaller droplets throw off far less de­bris when va­por­ized, which slows the rate at which tin builds up on the col­lec­tor mir­ror. Because the mir­ror de­grades more slowly, it needs fewer re­place­ments, keep­ing the ma­chine up and run­ning for longer stretches.

The part­ner­ship was a win for ASML, as it was able to work through some of its key en­gi­neer­ing and com­mer­cial­iza­tion chal­lenges. It also helped TSMC be­come an early adopter of the most cut­ting-edge tech­nol­ogy. By 2019, TSMC was ramp­ing up mass pro­duc­tion of its seven-nanome­ter process and the first phones with EUV chips were be­ing sold by the end of the year.

ASML’s most ad­vanced ex­treme ul­tra­vi­o­let lith­o­g­ra­phy scan­ner, the TWINSCAN EXE:5000.

Image

ASML.

Meanwhile, com­peti­tor firms like Nikon, which had never be­lieved as strongly in ex­treme ul­tra­vi­o­let lith­o­g­ra­phy, ef­fec­tively gave up. In its 2013 an­nual re­port, Nikon noted that its own EUV progress had not pro­ceeded as planned, and it was not men­tioned in an an­nual re­port again. With ASML pulling ahead on R&D and lock­ing up key cus­tomer de­mand, and with com­peti­tors strug­gling to jus­tify their own R&D spend­ing in the wake of the fi­nan­cial cri­sis, ASML be­came the last com­pany stand­ing in the race to com­mer­cial­ize the tech­nol­ogy.

The im­por­tance of tacit knowl­edge

Early on, ASML cul­ti­vated a cul­ture that was more risk tol­er­ant than other play­ers in the in­dus­try. It pro­moted high-po­ten­tial tal­ent early and had a track record of re­tain­ing key em­ploy­ees for decades. Much of this is a prod­uct of its chal­leng­ing early years. ASML needed the tal­ent of its younger gen­er­a­tion to save the com­pany, so it was more will­ing to pro­mote and em­power them quickly.

For ex­am­ple, Martin Van Den Brink joined ASML in 1984. Within 18 months, aged 29, he be­came one of two peo­ple pro­moted to lead the de­vel­op­ment of one of the com­pa­ny’s early flag­ship pro­jects. He car­ried on work­ing at ASML for his en­tire ca­reer, serv­ing as pres­i­dent and chief tech­ni­cal of­fi­cer un­til his re­tire­ment in 2024. This prac­tice was far less com­mon among ASML’s Japanese ri­vals, who were more hi­er­ar­chi­cal and tended to re­ward se­nior­ity over per­for­mance.

Retaining the best work­ers is es­pe­cially cru­cial in an area like pho­tolith­o­g­ra­phy, where a huge amount of tacit knowl­edge is used to as­sem­ble its ma­chines. An ASML en­gi­neer once told He Rongming, the founder of Shanghai Micro Electronics Equipment, one of China’s top ASML com­peti­tors, that the com­pany would­n’t be able to repli­cate ASML’s prod­ucts even if it had the blue­prints. He sug­gested that ASML’s prod­ucts re­flected ‘decades, if not cen­turies’ of knowl­edge and ex­pe­ri­ence. ASML’s Chinese com­peti­tors have sys­tem­at­i­cally at­tempted to hire for­mer ASML en­gi­neers, and there is at least one doc­u­mented case of a for­mer ASML em­ployee un­law­fully hand­ing over pro­pri­etary in­for­ma­tion. But none of this ap­pears to have nar­rowed the gap.

A European gi­ant

ASML is a rare ex­am­ple of a European tech gi­ant. Its suc­cess was the re­sult of transat­lantic co­op­er­a­tion, not con­ti­nen­tal parochial­ism. Had the com­pany not joined a pro­gram funded by US chip­mak­ers, Canon and Nikon would likely still dom­i­nate a less ad­vanced lith­o­g­ra­phy in­dus­try.

Cooperation with other com­pa­nies was just as im­por­tant. While ver­ti­cal in­te­gra­tion gave Nikon and Canon to­tal con­trol, it capped their in­no­va­tion at the lim­its of their in­ter­nal re­sources. In a sys­tem ex­ceed­ing one hun­dred thou­sand com­po­nents, that ceil­ing proved fa­tal. ASML’s mod­u­lar ap­proach al­lowed it to im­port cut­ting-edge physics by ac­quir­ing Cymer and in­vest­ing in Zeiss, while dis­trib­ut­ing the risk to cus­tomers like Intel and TSMC. This strat­egy cre­ated a col­lec­tive en­gine that out­spent and out­paced every ri­val at­tempt­ing to shoul­der the bur­den alone.

This took a great deal of courage. ASML sank bil­lions of dol­lars into the de­vel­op­ment and com­mer­cial­iza­tion of EUV tech­nol­ogy, with no guar­an­tee that it would ever work. As late as the 2010s, many semi­con­duc­tor ex­perts doubted that the tech­nol­ogy could be suc­cess­fully com­mer­cial­ized. Now it is the most im­por­tant tech­nol­ogy in the world.

But ASML, and by ex­ten­sion the con­ti­nent, can­not stand still. As ASML en­joys its place as an in­dis­pens­able pil­lar in one of the world’s most im­por­tant in­dus­tries, oth­ers are work­ing to cre­ate a new par­a­digm in chip tech­nol­ogy. Moore’s Law prob­a­bly does­n’t end here, and in a mat­ter of years, five nanome­ters won’t be small enough.

Enjoyed this? There’s more.

Get new ar­ti­cles from Works in Progress de­liv­ered to your in­box.

A few years back, I had a run­ning joke with the guy I was see­ing about adding him to my pe­riod tracker. Being a wom­en’s health ex­pert, I en­joy weav­ing nerdy anec­dotes about cy­cles and at­trac­tion and de­sires into my flir­ta­tions and mar­veling at my own wit and woo-woo mas­tery of my cycli­cal body. This ruse seemed like a harm­less jab at my dig­i­tally tracked self-aware­ness — a very late mil­len­nial fem­i­nist liv­ing in the Bay Area ver­sion of co­quetry.

It maybe was­n’t all that harm­less, af­ter all.

Turns out, the mat­ter of shar­ing the data around my cy­cle, and po­ten­tially the even more pri­vate in­for­ma­tion about my in­ti­mate ex­pe­ri­ences, was­n’t as much of a mat­ter of choice as I might have ex­pected. Worse, it might have been used to sell me stretch­mark creme or den­tal dams.

Caught bloody handed

That pe­riod track­ing app, Flo, has been found li­able in con­nec­tion with sell­ing user data to Meta all the while promis­ing their users they were pro­tect­ing their pri­vacy. The class ac­tion suit had 13 mil­lion Flo users in­cluded as plain­tiffs, which is a size­able chunk of pissed off users amongst their re­ported 75 mil­lion-strong user base.

Those law­suits against Meta and Flo, first filed in 2021 with more in the US and Canada, re­veal a big­ger is­sue in non-med­ical health track­ing soft­ware — there’s too much gray area around con­sent when it comes to sell­ing your health in­for­ma­tion to ad­ver­tis­ers.

What’s im­por­tant about the le­gal prece­dent be­ing set is in high­light­ing how the cur­rent guide­lines around health data pri­vacy (like HIPAA) are woe­fully lag­ging be­hind the health track­ing tech al­ready avail­able di­rectly to users. It raises a num­ber of crit­i­cal ques­tions:

What does this le­gal vague­ness mean for how we choose to self mon­i­tor our bi­o­log­i­cal mark­ers?

What does this le­gal vague­ness mean for how we choose to self mon­i­tor our bi­o­log­i­cal mark­ers?

In a post-Dobbs en­vi­ron­ment, how do con­cerns around dig­i­tal pri­vacy im­pact our con­sumer choices in sex­ual health and pe­riod track­ing apps?

In a post-Dobbs en­vi­ron­ment, how do con­cerns around dig­i­tal pri­vacy im­pact our con­sumer choices in sex­ual health and pe­riod track­ing apps?

Why is it still up to the con­sumer to run safety checks when it should be the role of prod­uct teams and healthtech brands to build less creepy tech?

Why is it still up to the con­sumer to run safety checks when it should be the role of prod­uct teams and healthtech brands to build less creepy tech?

Do we re­ally need to be track­ing every pos­si­ble symp­tom and mood and cramp and let­ting pri­vate tech com­pa­nies de­cide what to do with that data?

Do we re­ally need to be track­ing every pos­si­ble symp­tom and mood and cramp and let­ting pri­vate tech com­pa­nies de­cide what to do with that data?

Feeling “creamy” to­day? Great, we’ll let Mark Zuckerberg know.

Joking about the con­sis­tency of my ovu­la­tion was al­ready a bridge too far and a line I opted not to ven­ture to cross with said beau. I cer­tainly would­n’t have will­ingly an­nounced to any­one pars­ing through data at Meta if I had mas­tur­bated or had un­pro­tected sex on any given day. The Flo app might have made that de­ci­sion for me, though.

For all my men­tal back and forths about whether or not to ac­tu­ally send a part­ner my cy­cle cal­en­dar, Flo might have been send­ing the in­ti­mate de­tails of our sex­ual en­coun­ters to a bunch of tech bros be­hind my back. Turns out, Flo had em­bed­ded a se­cret “eavesdropping” tool which passed along in­for­ma­tion like men­stru­a­tion cy­cle, ovu­la­tion, and if a user was try­ing to get preg­nant to Meta, even while ex­plic­itly claim­ing not to in their pri­vacy pol­icy.

As slip­pery as an ovu­la­tion flow, Flo was telling us our pri­vate data was safely hid­den from pry­ing eyes. The guilty ver­dict in the August 2025 Frasco v. Flo law­suit proved oth­er­wise:

“Flo, through the Flo App, un­law­fully shared users’ sen­si­tive health data — in­clud­ing men­strual cy­cle, ovu­la­tion, and preg­nancy-re­lated in­for­ma­tion — with third par­ties such as Meta, Google, and Flurry for their own com­mer­cial use (Burr & Forman, 2025).”

“Flo, through the Flo App, un­law­fully shared users’ sen­si­tive health data — in­clud­ing men­strual cy­cle, ovu­la­tion, and preg­nancy-re­lated in­for­ma­tion — with third par­ties such as Meta, Google, and Flurry for their own com­mer­cial use (Burr & Forman, 2025).”

The jury found Meta li­able for col­lect­ing sen­si­tive re­pro­duc­tive health data and us­ing it for its own gain. The other par­ties listed set­tled out of court, which means their in­volve­ment in the breach gets to stay more pri­vate than the health data of Flo users be­tween 2016 and 2019.

Nothing fem­i­nism needs more nowa­days than a bit more irony, right?

This was­n’t a hack. It was a de­sign de­ci­sion.

It’s im­por­tant to call out that these third-party plat­forms did­n’t hack into the Flo app. The folks in charge of mak­ing pri­vacy de­ci­sions at Flo handed them our sen­si­tive data on a sil­ver plat­ter. It was sim­ple track-and-sell data shar­ing and we maybe should have seen it com­ing.

I’ve writ­ten be­fore about how ‘pinkwashing’ femtech can dis­guise a whole host of un­eth­i­cal prod­uct de­ci­sions. Prior to head­ing for greener and more pri­vate pas­tures with my pe­riod track­ing app se­lec­tion, Flo was al­ready start­ing to give me the ick. The UX de­sign was get­ting more con­vo­luted, more clut­tered, more car­toon­ish with every up­date.

Quickly, the Flo home screen was be­com­ing more bloated than a late-luteal phase tummy. Opening the app to log whether I had spot­ted a bit that morn­ing or had in­som­nia or ten­der breasts was like nav­i­gat­ing a mine­field of tired femme de­signs and re­dun­dant re­minders to med­i­tate.

With each up­date, the home dis­play pre­sented me with the op­tion for ever grow­ing op­por­tu­ni­ties for neg­a­tive symp­tom re­port­ing. Without any dif­fer­en­ti­a­tion in hi­er­ar­chy, every­thing seemed flatly patho­log­i­cal. The symp­toms were pushed more and more to the front and ad­vice popped out at every turn, es­sen­tially bury­ing the ac­tual cy­cle tracker.

In the con­text of the Flo-Meta fil­ings, this makes sense — fo­cus­ing on the “problems” of pe­ri­ods can help drive sales of items pur­port­ing to al­le­vi­ate symp­toms. There is­n’t much to mon­e­tize from a sim­ple pe­riod cal­en­dar, is there? It’s dystopian to re­al­ize the em­pha­sis on symp­to­mol­ogy was help­ing to drive ad­ver­tis­ing on sites even more re­cently found li­able for per­sonal harm on par with to­bacco com­pa­nies.

At the end of the day, no amount of pinkwashed ‘empowerment’ or ‘evolved’ men­tions of sex toys and self plea­sure can cover up who ben­e­fit­ted* from these de­sign choices.

The gap be­tween HIPAA and ‘wellness’ is where con­sent goes to die

Flo changed its pri­vacy pol­icy a whop­ping 13 times in the three years rel­e­vant to the le­gal claims (2016 – 2019). These law­suits show that all those ed­its did noth­ing to make the con­sent users might have thought they were giv­ing real in any mean­ing­ful way.

Lawsuits like the Flo-Meta law­suits are no­table in that they are help­ing to build a foun­da­tion of le­gal prece­dent within the gray zone of non-HIPAA com­pli­ant well­ness tech. Much of health tech, which in­cludes a lot of re­pro­duc­tive health tech cur­rently on the mar­ket, is­n’t ex­plic­itly clin­i­cal or di­rectly tied to com­mu­ni­ca­tions with a health­care provider.

Which means, you can be log­ging some deep in­for­ma­tion about the func­tions of your body and given au­to­mated ad­vice on mak­ing ad­just­ments to po­ten­tially im­prove these bod­ily func­tions, and in all like­li­hood, it man­ages to not fall un­der the pro­tec­tion of cur­rent health and pri­vacy laws. This means that it is at the dis­cre­tion of the apps them­selves to cre­ate the poli­cies around what data to share or sell or re­port to gov­ern­ment agen­cies them­selves.

They also have pretty broad dis­cre­tion in the de­signs around con­sent they are will­ing and able to of­fer users. The de­sign de­ci­sions and con­sent frame­works in-prod­uct can be guided by best-prac­tices, but those choices are still largely dri­ven by the opin­ions within prod­uct teams. This is how sloppy con­sent pat­terns con­tinue to get shipped out to users, even when the prod­uct might deal in in­cred­i­bly sen­si­tive data col­lec­tion.

It was­n’t like some cy­ber crim­i­nal was hold­ing Flo ran­som, these were em­bed­ded le­gal, de­sign, en­gi­neer­ing, and sales po­si­tions that got through a chain of em­ploy­ees that ul­ti­mately threw users un­der the bus for profit.

It’s hard to track down ex­act in­for­ma­tion on the num­ber of staff em­ployed by Flo from 2016 – 2019 and who was di­rectly re­spon­si­ble for these choices. By most ac­counts, it was a lean op­er­a­tion — prob­a­bly around 350 em­ploy­ees at any given time in those years. That’s a pretty small group of folks mak­ing po­ten­tially mon­u­men­tal de­ci­sions about how highly sen­si­tive health data got col­lected, stored, and shared in ad­di­tion to how those processes and poli­cies were com­mu­ni­cated to their mil­lions of users world­wide.

If we’re left to our own de­vices, who will pro­tect us?

It seems like we can’t just nec­es­sar­ily leave it up to com­pa­nies — or their rag­tag teams of crack­pot lawyers rewrit­ing pri­vacy poli­cies every few months — to keep our pri­vate data pri­vate. I guess we’re left need­ing to hurt Mark Zuckerberg’s feel­ings every now and again in or­der to just use our vi­bra­tors in peace.

The law is slow to catch up, even more so when it comes to reg­u­lat­ing tech. This makes me ner­vous when con­sid­er­ing the rush to in­crease the col­lec­tion of data around wom­en’s health in an ef­fort to close the data gap. This is a wor­thy aim, but how much trust can we re­ally place in pri­vate com­pa­nies op­er­at­ing out­side of clin­i­cally guided struc­tures?

This is even be­fore we fac­tor in the in­creased use of gen­er­a­tive AI in pop­u­lat­ing health ad­vice within apps which seem to in­ten­tion­ally cir­cum­vent the health­care space and thus not have to be com­pli­ant with the user pro­tec­tions un­der that cat­e­gor­i­cal um­brella. There is such a thing as too much data, though try telling that to a PM try­ing to make his KPIs. If the data comes from un­man­aged flows, the col­lec­tion meth­ods pri­or­i­tized for third-party ad sales, and done with­out the di­rect con­sent of users, how much can we even rely on the de­riv­a­tive gen­er­a­tive out­puts? Is this the stan­dard we want to set for col­lect­ing wom­en’s health data? Is it worth all the costs?

Personally, this reeks of mov­ing fast and break­ing things to me. Flo def­i­nitely broke my trust, along with at least 13 mil­lion for­mer Flo users. With (reportedly) over a third of US women uti­liz­ing pe­riod track­ing apps and a sim­i­lar rate of use amongst women in the EU, there’s a sig­nif­i­cant mar­ket to cap­ture here. Unlike in 2016 when Flo was one of few play­ers on the field, there are hun­dreds of cy­cle track­ing apps for savvy users to se­lect from to­day, not to men­tion the in­creas­ing avail­abil­ity of built-in cy­cle track­ers within other health apps and wear­ables.

Though Flo re­mains one of the top down­loaded of the bunch, for many of us, it’s a mat­ter of once burned, twice shy. Personally, I’m a big fan of WildAI, which does­n’t bother to ask me if I’ve rubbed one out and there­fore has no in­ter­est in telling a tech be­he­moth a whole lot more than if I both­ered to note if I was thirsty and horny and hun­gry on the same day. You and Mark can guess how much space those notes take up on my cy­cle cal­en­dar all on your own. I pre­fer it that way, and Flo should too.

*Let’s just take a mo­ment, by the way, to re­flect on how the dev dudes set­ting up per­son­al­ized ad gat­ing at Google might have been track­ing the sex toy use and preva­lence of anal sex amongst Flo users so they might drive up pay per click (PPC) rates across your apps. Obviously, this is fem­i­nism at its finest.

**It might be worth ar­gu­ing if in a post-Dobbs world and in coun­tries with wishy-washy dig­i­tal pri­vacy stan­dards that maybe metic­u­lously log­ging sexy self-play might not have the po­ten­tial health ben­e­fits worth the risk hav­ing it wind up in the hands of such loose-lipped data bro­kers. It’s bad enough we have to worry about the pri­vacy vi­o­la­tions of the vi­bra­tors them­selves. Maybe “dumb” dil­dos are the bet­ter op­tion these days, ac­tu­ally. We’ll have to get to that in an­other post.

Share The Femtech Design Desk

Leave a com­ment

For help please visit help.ft.com. We apologise for any in­con­ve­nience.

The fol­low­ing in­for­ma­tion can help our sup­port team to re­solve this is­sue.

📰 News

2026 – 03-06: 🚀 VibeVoice ASR is now part of a Transformers re­lease! You can now use our speech recog­ni­tion model di­rectly through the Hugging Face Transformers li­brary for seam­less in­te­gra­tion into your pro­jects.

2026 – 01-21: 📣 We open-sourced VibeVoice-ASR, a uni­fied speech-to-text model de­signed to han­dle 60-minute long-form au­dio in a sin­gle pass, gen­er­at­ing struc­tured tran­scrip­tions con­tain­ing Who (Speaker), When (Timestamps), and What (Content), with sup­port for User-Customized Context. Try it in Playground.

⭐️ VibeVoice-ASR is na­tively mul­ti­lin­gual, sup­port­ing over 50 lan­guages — check the sup­ported lan­guages for de­tails.

🔥 The VibeVoice-ASR fine­tun­ing code is now avail­able!

⚡️ vLLM in­fer­ence is now sup­ported for faster in­fer­ence; see vllm-asr for more de­tails.

📑 VibeVoice-ASR Technique Report is avail­able.

2025 – 12-16: 📣 We added ex­per­i­men­tal speak­ers to VibeVoice‑Realtime‑0.5B for ex­plo­ration, in­clud­ing mul­ti­lin­gual voices in nine lan­guages (DE, FR, IT, JP, KR, NL, PL, PT, ES) and 11 dis­tinct English style voices. Try it. More speaker types will be added over time.

2025 – 12-03: 📣 We open-sourced VibeVoice‑Realtime‑0.5B, a real‑time text‑to‑speech model that sup­ports stream­ing text in­put and ro­bust long-form speech gen­er­a­tion. Try it on Colab.

2025 – 09-05: VibeVoice is an open-source re­search frame­work in­tended to ad­vance col­lab­o­ra­tion in the speech syn­the­sis com­mu­nity. After re­lease, we dis­cov­ered in­stances where the tool was used in ways in­con­sis­tent with the stated in­tent. Since re­spon­si­ble use of AI is one of Microsoft’s guid­ing prin­ci­ples, we have re­moved the VibeVoice-TTS code from this repos­i­tory.

2025 – 08-25: 📣 We open-sourced VibeVoice-TTS, a long-form multi-speaker text-to-speech model that can syn­the­size speech up to 90 min­utes long with up to 4 dis­tinct speak­ers. — ac­cepted as an Oral at ICLR 2026! 🔥

Overview

VibeVoice is a fam­ily of open-source fron­tier voice AI mod­els that in­cludes both Text-to-Speech (TTS) and Automatic Speech Recognition (ASR) mod­els.

A core in­no­va­tion of VibeVoice is its use of con­tin­u­ous speech to­k­eniz­ers (Acoustic and Semantic) op­er­at­ing at an ul­tra-low frame rate of 7.5 Hz. These to­k­eniz­ers ef­fi­ciently pre­serve au­dio fi­delity while sig­nif­i­cantly boost­ing com­pu­ta­tional ef­fi­ciency for pro­cess­ing long se­quences. VibeVoice em­ploys a next-to­ken dif­fu­sion frame­work, lever­ag­ing a Large Language Model (LLM) to un­der­stand tex­tual con­text and di­a­logue flow, and a dif­fu­sion head to gen­er­ate high-fi­delity acoustic de­tails.

For more in­for­ma­tion, demos, and ex­am­ples, please visit our Project Page.

Models

1. 📖 VibeVoice-ASR - Long-form Speech Recognition

VibeVoice-ASR is a uni­fied speech-to-text model de­signed to han­dle 60-minute long-form au­dio in a sin­gle pass, gen­er­at­ing struc­tured tran­scrip­tions con­tain­ing Who (Speaker), When (Timestamps), and What (Content), with sup­port for Customized Hotwords.

🕒 60-minute Single-Pass Processing:

Unlike con­ven­tional ASR mod­els that slice au­dio into short chunks (often los­ing global con­text), VibeVoice ASR ac­cepts up to 60 min­utes of con­tin­u­ous au­dio in­put within 64K to­ken length. This en­sures con­sis­tent speaker track­ing and se­man­tic co­her­ence across the en­tire hour.

🕒 60-minute Single-Pass Processing:

Unlike con­ven­tional ASR mod­els that slice au­dio into short chunks (often los­ing global con­text), VibeVoice ASR ac­cepts up to 60 min­utes of con­tin­u­ous au­dio in­put within 64K to­ken length. This en­sures con­sis­tent speaker track­ing and se­man­tic co­her­ence across the en­tire hour.

👤 Customized Hotwords:

Users can pro­vide cus­tomized hot­words (e.g., spe­cific names, tech­ni­cal terms, or back­ground info) to guide the recog­ni­tion process, sig­nif­i­cantly im­prov­ing ac­cu­racy on do­main-spe­cific con­tent.

👤 Customized Hotwords:

Users can pro­vide cus­tomized hot­words (e.g., spe­cific names, tech­ni­cal terms, or back­ground info) to guide the recog­ni­tion process, sig­nif­i­cantly im­prov­ing ac­cu­racy on do­main-spe­cific con­tent.

📝 Rich Transcription (Who, When, What):

The model jointly per­forms ASR, di­ariza­tion, and time­stamp­ing, pro­duc­ing a struc­tured out­put that in­di­cates who said what and when.

📝 Rich Transcription (Who, When, What):

The model jointly per­forms ASR, di­ariza­tion, and time­stamp­ing, pro­duc­ing a struc­tured out­put that in­di­cates who said what and when.

📖 Documentation | 🤗 Hugging Face | 🎮 Playground | 🛠️ Finetuning | 📊 Paper

2. 🎙️ VibeVoice-TTS - Long-form Multi-speaker TTS

Best for: Long-form con­ver­sa­tional au­dio, pod­casts, multi-speaker di­a­logues

⏱️ 90-minute Long-form Generation:

Synthesizes con­ver­sa­tional/​sin­gle-speaker speech up to 90 min­utes in a sin­gle pass, main­tain­ing speaker con­sis­tency and se­man­tic co­her­ence through­out.

⏱️ 90-minute Long-form Generation:

Synthesizes con­ver­sa­tional/​sin­gle-speaker speech up to 90 min­utes in a sin­gle pass, main­tain­ing speaker con­sis­tency and se­man­tic co­her­ence through­out.

👥 Multi-speaker Support:

Supports up to 4 dis­tinct speak­ers in a sin­gle con­ver­sa­tion, with nat­ural turn-tak­ing and speaker con­sis­tency across long di­a­logues.

👥 Multi-speaker Support:

Supports up to 4 dis­tinct speak­ers in a sin­gle con­ver­sa­tion, with nat­ural turn-tak­ing and speaker con­sis­tency across long di­a­logues.

🎭 Expressive Speech:

Generates ex­pres­sive, nat­ural-sound­ing speech that cap­tures con­ver­sa­tional dy­nam­ics and emo­tional nu­ances.

🎭 Expressive Speech:

Generates ex­pres­sive, nat­ural-sound­ing speech that cap­tures con­ver­sa­tional dy­nam­ics and emo­tional nu­ances.

🌐 Multi-lingual Support:

Supports English, Chinese and other lan­guages.

🌐 Multi-lingual Support:

Supports English, Chinese and other lan­guages.

📖 Documentation | 🤗 Hugging Face | 📊 Paper

English

Chinese

Cross-Lingual

Spontaneous Singing

Long Conversation with 4 peo­ple

3. ⚡ VibeVoice-Streaming - Real-time Streaming TTS

VibeVoice-Realtime is a light­weight real‑time text-to-speech model sup­port­ing stream­ing text in­put and ro­bust long-form speech gen­er­a­tion.

Parameter size: 0.5B (deployment-friendly)

Real-time TTS (~300 mil­lisec­onds first au­di­ble la­tency)

Streaming text in­put

Robust long-form speech gen­er­a­tion (~10 min­utes)

📖 Documentation | 🤗 Hugging Face | 🚀 Colab

Contributing

Please see CONTRIBUTING.md for de­tailed con­tri­bu­tion guide­lines.

⚠️ Risks and Limitations

While ef­forts have been made to op­ti­mize it through var­i­ous tech­niques, it may still pro­duce out­puts that are un­ex­pected, bi­ased, or in­ac­cu­rate. VibeVoice in­her­its any bi­ases, er­rors, or omis­sions pro­duced by its base model (specifically, Qwen2.5 1.5b in this re­lease).

Potential for Deepfakes and Disinformation: High-quality syn­thetic speech can be mis­used to cre­ate con­vinc­ing fake au­dio con­tent for im­per­son­ation, fraud, or spread­ing dis­in­for­ma­tion. Users must en­sure tran­scripts are re­li­able, check con­tent ac­cu­racy, and avoid us­ing gen­er­ated con­tent in mis­lead­ing ways. Users are ex­pected to use the gen­er­ated con­tent and to de­ploy the mod­els in a law­ful man­ner, in full com­pli­ance with all ap­plic­a­ble laws and reg­u­la­tions in the rel­e­vant ju­ris­dic­tions. It is best prac­tice to dis­close the use of AI when shar­ing AI-generated con­tent.

We do not rec­om­mend us­ing VibeVoice in com­mer­cial or real-world ap­pli­ca­tions with­out fur­ther test­ing and de­vel­op­ment. This model is in­tended for re­search and de­vel­op­ment pur­poses only. Please use re­spon­si­bly.

Star History

Please en­able JS and dis­able any ad blocker

I wanted to give an up­date on GitHub’s avail­abil­ity in light of two re­cent in­ci­dents. Both of those in­ci­dents are not ac­cept­able, and we are sorry for the im­pact they had on you. I wanted to share some de­tails on them, as well as ex­plain what we’ve done and what we’re do­ing to im­prove our re­li­a­bil­ity.

We started ex­e­cut­ing our plan to in­crease GitHub’s ca­pac­ity by 10X in October 2025 with a goal of sub­stan­tially im­prov­ing re­li­a­bil­ity and failover. By February 2026, it was clear that we needed to de­sign for a fu­ture that re­quires 30X to­day’s scale.

The main dri­ver is a rapid change in how soft­ware is be­ing built. Since the sec­ond half of December 2025, agen­tic de­vel­op­ment work­flows have ac­cel­er­ated sharply. By nearly every mea­sure, the di­rec­tion is al­ready clear: repos­i­tory cre­ation, pull re­quest ac­tiv­ity, API us­age, au­toma­tion, and large-repos­i­tory work­loads are all grow­ing quickly.

This ex­po­nen­tial growth does not stress one sys­tem at a time. A pull re­quest can touch Git stor­age, merge­abil­ity checks, branch pro­tec­tion, GitHub Actions, search, no­ti­fi­ca­tions, per­mis­sions, web­hooks, APIs, back­ground jobs, caches, and data­bases. At high scale, small in­ef­fi­cien­cies com­pound: queues deepen, cache misses be­come data­base load, in­dexes fall be­hind, re­tries am­plify traf­fic, and one slow de­pen­dency can af­fect sev­eral prod­uct ex­pe­ri­ences.

Our pri­or­i­ties are clear: avail­abil­ity first, then ca­pac­ity, then new fea­tures. We are re­duc­ing un­nec­es­sary work, im­prov­ing caching, iso­lat­ing crit­i­cal ser­vices, re­mov­ing sin­gle points of fail­ure, and mov­ing per­for­mance-sen­si­tive paths into sys­tems de­signed for these work­loads. This is dis­trib­uted sys­tems work: re­duc­ing hid­den cou­pling, lim­it­ing blast ra­dius, and mak­ing GitHub de­grade grace­fully when one sub­sys­tem is un­der pres­sure. We’re mak­ing progress quickly, but these in­ci­dents are ex­am­ples of where there’s still work to do.

What we’re do­ing

Short term, we had to re­solve a va­ri­ety of bot­tle­necks that ap­peared faster than ex­pected from mov­ing web­hooks to a dif­fer­ent back­end (out of MySQL), re­design­ing user ses­sion cache to re­do­ing au­then­ti­ca­tion and au­tho­riza­tion flows to sub­stan­tially re­duce data­base load. We also lever­aged our mi­gra­tion to Azure to stand up a lot more com­pute.

Next we fo­cused on iso­lat­ing crit­i­cal ser­vices like git and GitHub Actions from other work­loads and min­i­miz­ing the blast ra­dius by min­i­miz­ing sin­gle points of fail­ure. This work started with care­ful analy­sis of de­pen­den­cies and dif­fer­ent tiers of traf­fic to un­der­stand what needs to be pulled apart and how we can min­i­mize im­pact on le­git­i­mate traf­fic from var­i­ous at­tacks. Then we ad­dressed those in or­der of risk. Similarly, we ac­cel­er­ated parts of mi­grat­ing per­for­mance or scale sen­si­tive code out of Ruby mono­lith into Go.

While we were al­ready in progress of mi­grat­ing out of our smaller cus­tom data cen­ters into pub­lic cloud, we started work­ing on path to multi cloud. This longer-term mea­sure is nec­es­sary to achieve the level of re­silience, low la­tency, and flex­i­bil­ity that will be needed in the fu­ture.

The num­ber of repos­i­to­ries on GitHub is grow­ing faster than ever, but a much harder scal­ing chal­lenge is the rise of large monore­pos. For the last three months, we’ve been in­vest­ing heav­ily in re­sponse to this trend both within git sys­tem and in the pull re­quest ex­pe­ri­ence.

We will have a sep­a­rate blog post soon de­scrib­ing ex­ten­sive work we’ve done and the new up­com­ing API de­sign for greater ef­fi­ciency and scale. As part of this work, we have in­vested in op­ti­miz­ing merge queue op­er­a­tions, since that is key for re­pos that have many thou­sands of pull re­quests a day.

Recent in­ci­dents

The two re­cent in­ci­dents were dif­fer­ent in cause and im­pact, but both re­flect why we are in­creas­ing our fo­cus on avail­abil­ity, iso­la­tion, and blast-ra­dius re­duc­tion.

April 23 merge queue in­ci­dent

On April 23, pull re­quests ex­pe­ri­enced a re­gres­sion af­fect­ing merge queue op­er­a­tions.

Pull re­quests merged through merge queue us­ing the squash merge method pro­duced in­cor­rect merge com­mits when a merge group con­tained more than one pull re­quest. In af­fected cases, changes from pre­vi­ously merged pull re­quests and prior com­mits were in­ad­ver­tently re­verted by sub­se­quent merges.

During the im­pact win­dow, 658 repos­i­to­ries and 2,092 pull re­quests were af­fected. We ini­tially shared slightly higher num­bers be­cause our first as­sess­ment was in­ten­tion­ally con­ser­v­a­tive. The is­sue did not af­fect pull re­quests merged out­side merge queue, nor did it af­fect merge queue groups us­ing merge or re­base meth­ods.

There was no data loss: all com­mits re­mained stored in Git. However, the state of af­fected de­fault branches was in­cor­rect, and we could not safely re­pair every repos­i­tory au­to­mat­i­cally. More de­tails are avail­able in the in­ci­dent root cause analy­sis.

This in­ci­dent ex­posed mul­ti­ple process fail­ures, and we are chang­ing those processes to pre­vent this class of is­sue from re­cur­ring.

On April 27, an in­ci­dent af­fected our Elasticsearch sub­sys­tem, which pow­ers sev­eral search-backed ex­pe­ri­ences across GitHub, in­clud­ing parts of pull re­quests, is­sues, and pro­jects.

We are still com­plet­ing the root cause analy­sis and will pub­lish it shortly. What we know now is that the clus­ter be­came over­loaded (likely due to a bot­net at­tack) and stopped re­turn­ing search re­sults. There was no data loss, and Git op­er­a­tions and APIs were not im­pacted. However, parts of the UI that de­pended on search showed no re­sults, which caused a sig­nif­i­cant dis­rup­tion.

This is one of the sys­tems we had not yet fully iso­lated to elim­i­nate as a sin­gle point of fail­ure, be­cause other ar­eas had been higher in our risk-pri­or­i­tized re­li­a­bil­ity work. That im­pact is un­ac­cept­able, and we are us­ing the same de­pen­dency and blast-ra­dius analy­sis de­scribed above to re­duce the like­li­hood and im­pact of this type of fail­ure in the fu­ture.

Increasing trans­parency

We have also heard clear feed­back that cus­tomers need greater trans­parency dur­ing in­ci­dents.

We re­cently up­dated the GitHub sta­tus page to in­clude avail­abil­ity num­bers. We have also com­mit­ted to sta­tus­ing in­ci­dents both large and small, so you do not have to guess whether an is­sue is on your side or ours.

We are con­tin­u­ing to im­prove how we cat­e­go­rize in­ci­dents so that the scale and scope are eas­ier to un­der­stand. We are also work­ing on bet­ter ways for cus­tomers to re­port in­ci­dents and share sig­nals with us dur­ing dis­rup­tions.

Our com­mit­ment

GitHub’s role has al­ways been to sup­port de­vel­op­ers on an open and ex­ten­si­ble plat­form.

The team at GitHub is in­cred­i­bly pas­sion­ate about our work. We hear the pain you’re ex­pe­ri­enc­ing. We read every email, so­cial post, sup­port ticket, and we take it all to heart. We’re sorry.

We are com­mit­ted to im­prov­ing avail­abil­ity, in­creas­ing re­silience, scal­ing for the fu­ture of soft­ware de­vel­op­ment, and com­mu­ni­cat­ing more trans­par­ently along the way.

Editor’s note: This post was up­dated on April 28, 2026, to up­date the num­ber of re­pos af­fected dur­ing the April 23 in­ci­dent.

Written by

Vladimir Fedorov is GitHub’s Chief Technology Officer, bring­ing decades of ex­pe­ri­ence in en­gi­neer­ing lead­er­ship and in­no­va­tion. A pas­sion­ate ad­vo­cate for de­vel­oper pro­duc­tiv­ity, Vlad is lead­ing GitHub’s en­gi­neer­ing team to shape the fu­ture of de­vel­oper tools and in­no­va­tion with a de­vel­oper-first mind­set.

Before join­ing GitHub, Vlad co-founded UserClouds, a startup spe­cial­iz­ing in data gov­er­nance and pri­vacy. He spent 12 years at Facebook, now Meta, as Senior Vice President, lead­ing en­gi­neer­ing teams of over 2,000 across Privacy, Ads, and Platform. Earlier in his ca­reer, Vlad worked at Microsoft and earned both his BS and MS in Computer Science from Caltech. He cur­rently serves on the board of Codepath.org, an or­ga­ni­za­tion ded­i­cated to re­pro­gram­ming higher ed­u­ca­tion to cre­ate the first AI-native gen­er­a­tion of en­gi­neers, CTOs, and founders.

Vlad lives in the Bay Area and when not work­ing en­joys spend­ing time out­side and on the wa­ter with his fam­ily.

Explore more from GitHub

Docs

Everything you need to mas­ter GitHub, all in one place.

Go to Docs

GitHub

Build what’s next on GitHub, the place for any­one from any­where to build any­thing.

Start build­ing

Customer sto­ries

Meet the com­pa­nies and en­gi­neer­ing teams that build with GitHub.

Learn more

The GitHub Podcast

Catch up on the GitHub pod­cast, a show ded­i­cated to the top­ics, trends, sto­ries and cul­ture in and around the open source de­vel­oper com­mu­nity on GitHub.

Listen now

On April 17, 2026, Sam Altman’s other AI com­pany, Tools For Humanity, an­nounced a part­ner­ship with Bruno Mars as he em­barks on his Romantic Tour. The an­nounce­ment co­in­cided with the com­pa­ny’s Concert Kit tool, which al­legedly al­lows “verified hu­mans” to ac­cess VIP tick­ets and con­cert ex­pe­ri­ences.

However, Bruno Mars’ man­age­ment and Live Nation re­leased a joint state­ment on April 22, claim­ing that the part­ner­ship did­n’t ex­ist. “To be clear, we were never even ap­proached by [Tools For Humanity], nor were we in any dis­cus­sions re­gard­ing a part­ner­ship or tour ac­cess,” the state­ment read. “We first learned that our tour was be­ing used to pro­mote their pro­ject af­ter their keynote made those ini­tial claims.”

Those claims orig­i­nated from TFH’s chief prod­uct of­fi­cer, Tiago Sada, dur­ing a com­pany event. The com­pany then pub­lished a post on its web­site in­clud­ing Sada’s quote about Bruno Mars’ Romantic Tour. Eventually, word got back to Mars’ team.

AI Company Executive Gets His Marses Confused, Is Actually Partnering with Jared Leto’s Band

The ini­tial post on Tools For Humanity’s web­site has since been edited to cor­rect the false in­for­ma­tion. A spokesper­son also con­firmed the com­pany “does not have any agree­ment with Bruno Mars to test or fea­ture Concert Kit.” Additionally, “there is no as­so­ci­a­tion or af­fil­i­a­tion with the artist or his tour.”

Tools For Humanity is ac­tu­ally part­ner­ing with Thirty Seconds to Mars on their 2027 European tour. While TFH has not dis­closed the ac­tual rea­son for the false Bruno Mars an­nounce­ment, it looks a bit like a case of mis­taken iden­tity. Pretty ironic, since the com­pa­ny’s whole shtick is sup­pos­edly ver­i­fy­ing hu­man iden­ti­ties.

The com­pany launched in 2019 ini­tially as a way to ver­ify hu­man iden­ti­ties in on­line spaces to pre­vent fraud. This in­cluded live mu­sic mo­nop­oly Live Nation-Ticketmaster, which is of­ten plagued by bots and scam­mers. In 2023, TFH launched a phys­i­cal iden­tity ver­i­fi­ca­tion de­vice in the form of an orb that scans hu­man irises.

Unfortunately, the orb does not also tell for­tunes, which is clearly a ma­jor de­sign flaw. If it did, they’d prob­a­bly be able to pre­vent this Mars mix-up be­fore it hap­pened.