10 interesting stories served every morning and every evening.

FUTO Swipe

swipe.futo.tech

Fast, ac­cu­rate swipe typ­ing sys­tem. Use it to­day in FUTO Keyboard, our fully of­fline Android key­board app. Or down­load the mod­els and build with it.

This is a server­side demo to keep this web­page small. In pro­duc­tion, it runs on-de­vice, with much lower la­tency.

For a long time, good mo­bile swipe typ­ing was locked be­hind pri­vacy-in­va­sive key­board apps or un­li­censed pri­vate li­braries.

FUTO Swipe is our fam­ily of open mod­els and al­go­rithms that aims to solve this prob­lem. We de­vel­oped this pri­mar­ily for FUTO Keyboard, but we also wel­come the broader com­mu­nity to make use of the FUTO Swipe mod­els. As this has been a long-term in­vest­ment for us, we ask that an at­tri­bu­tion is made vis­i­ble to end-users. Read li­cense

Dataset

In August 2024, we launched a dataset col­lec­tion ef­fort on the swipe.futo.org do­main to col­lect QWERTY English swipes. Users would vol­un­tar­ily visit the web­page on their mo­bile phone and be given in­struc­tions and in­for­ma­tion about the dataset. After con­sent­ing, they would be given sen­tences, pri­mar­ily from Wikipedia, and would be asked to swipe them word-by-word.

In the end, this pro­duced over 1 mil­lion swipes. We fil­tered out a small set of low-qual­ity swipes. In March 2025, we re­leased a dataset of 1 mil­lion swipes un­der the MIT li­cense, and it is avail­able to­day on HuggingFace.

We made heavy use of this data to train our mod­els and to eval­u­ate dif­fer­ent swipe typ­ing sys­tems.

Models

Our ar­chi­tec­ture in­cludes three model types.

The Encoder model is a uni­ver­sal lay­out-ag­nos­tic and lan­guage-ag­nos­tic, and is used for mak­ing swipe typ­ing pre­dic­tions in the gen­eral case. However, it does not of­fer cut­ting-edge ac­cu­racy.

The ContextLM model is a very small lan­guage model that is trained for a sin­gle lan­guage. It’s used to im­prove the qual­ity of pre­dic­tions by elim­i­nat­ing non­sen­si­cal words given the pre­ced­ing words in the sen­tence. It only re­quires text data for train­ing.

Finally, the de­coder is a lan­guage-spe­cific and lay­out-spe­cific model that learns lay­out’s pe­cu­liar­i­ties and achieves lead­ing ac­cu­racy. As it re­quires swipe typ­ing data for a spe­cific lay­out and lan­guage for train­ing, we only have a QWERTY English de­coder for now.

With all 3 mod­els and with a beam width of 300, we achieve a top-4 fail rate of only ~4% on our test set. Ignoring out-of-vo­cab­u­lary cases, the er­ror rate is be­low 1%.

Note: These num­bers heav­ily de­pend on the bench­mark, so real-world use may vary, but we be­lieve we match big tech’s key­boards.

Footprint

The en­coder model is just 635,140 pa­ra­me­ters, and the de­coder is 304,155 ex­tra. The biggest one is the ContextLM at 1.5 mil­lion, but 1.1 mil­lion of that is just em­bed­dings. This brings us to 1,364,271 ac­tive pa­ra­me­ters, or 2,494,767 to­tal pa­ra­me­ters.

This means the foot­print of the mod­els are very small, and the model can run on low-end de­vices in mil­lisec­onds. In ad­di­tion, the en­vi­ron­men­tal costs in­volved in train­ing the mod­els were also very low, be­cause we never needed more than 1 work­sta­tion GPU!

C++ Library

The mod­els them­selves are only half of the story when go­ing from a swipe to word pre­dic­tions. The model pre­dic­tions are not very use­ful on their own and it’s nec­es­sary to per­form a dic­tio­nary-con­strained beam search to score a set of words and find the most likely can­di­dates.

For this, we re­lease swipe-li­brary, a li­brary writ­ten in C++ that han­dles the en­tire in­fer­ence, de­cod­ing, and beam search part so you can eas­ily go from swipe paths to word pre­dic­tions.

Make some­thing cool!

…or on a lap­top track­pad

Want to build with FUTO Swipe?

The FUTO Swipe mod­els are avail­able un­der the FUTO Model License, and the in­fer­ence li­brary is un­der GPL. We are work­ing on a pa­per that will de­tail more on the train­ing and ar­chi­tec­ture.

We’re making Bunny DNS free

bunny.net

At bunny.net, our mis­sion has al­ways been am­bi­tious but fo­cused: help make the in­ter­net hop faster.

To do that, we’ve built a mas­sive global net­work span­ning 119 lo­ca­tions and count­ing. Today, this net­work pow­ers over 1.5 mil­lion web­sites and con­sis­tently de­liv­ers some of the fastest con­tent de­liv­ery around the globe. But while de­ploy­ing thou­sands of servers glob­ally is an im­pres­sive feat on its own, the hard­ware it­self does not ex­plain how bunny.net is able to de­liver such an im­pres­sive level of per­for­mance.

The real se­cret hides un­der the hood, em­bed­ded in the rout­ing en­gine that di­rects every re­quest, every user, and sends traf­fic ex­actly where it needs to go. That en­gine is Bunny DNS.

From in­ter­nal en­gine to 200 bil­lion cus­tomer queries per month

Originally, Bunny DNS was built with one sim­ple goal: to build the most ad­vanced rout­ing en­gine pos­si­ble, ca­pa­ble of an­a­lyz­ing every DNS query and di­rect­ing it to the op­ti­mal des­ti­na­tion for serv­ing your con­tent. Even to this day, it’s what makes Bunny CDN achieve it’s ex­cep­tional per­for­mance.

Four years ago, we took every­thing we had learned from de­sign­ing and run­ning this sys­tem and turned it into a prod­uct our users could use them­selves. With Bunny DNS, we’ve up­graded DNS from be­ing a ba­sic record lookup table into a glob­ally dis­trib­uted, smart rout­ing en­gine. Instead of just re­turn­ing sta­tic records, it al­lows de­vel­op­ers to use la­tency data, health checks, and even JavaScript to dy­nam­i­cally de­ter­mine ex­actly where re­quests should go.

We ap­plied our tra­di­tional mantra. We made it af­ford­able, scal­able, and added a gen­er­ous free tier. The re­sponse was in­cred­i­ble, and to­day, Bunny DNS pow­ers over 300,000 do­mains and han­dles nearly 200 bil­lion queries every sin­gle month.

But as we looked closer at our mis­sion, we re­al­ized some­thing needed to change.

Dealing with in­fra­struc­ture costs is al­ready no­to­ri­ously com­plex. You should­n’t have to stress about pric­ing tiers or whether a sud­den spike of a mil­lion queries is go­ing to re­sult in an un­pre­dictable bill.

If we truly be­lieve in our mis­sion to help make the in­ter­net hop faster, then the fun­da­men­tal sys­tem that sits be­tween your users and your ser­vices should­n’t be a pre­mium add-on. It should be ac­ces­si­ble to every­one.

So, we’ve elim­i­nated DNS query fees en­tirely.

Bunny DNS no longer charges for DNS queries and in­cludes free DNS host­ing for up to 500 do­mains per ac­count. There are no query lim­its, no per-re­quest billing, and no crit­i­cal fea­tures hid­den be­hind en­ter­prise plans. (Yes, that in­cludes smart records and health mon­i­tor­ing too.)

As with all bunny.net ser­vices, ac­counts us­ing the plat­form are sub­ject to our stan­dard $1/month min­i­mum spend, but DNS it­self no longer in­curs any us­age-based charges.

The en­try point for every­thing else

Making Bunny DNS free does­n’t mean we’re los­ing in­ter­est in it. Quite the op­po­site. More than ever, we view DNS as the core prod­uct that glues our en­tire plat­form to­gether. It’s the start­ing line for every­thing else your ap­pli­ca­tion does.

Getting that start­ing line set up is now eas­ier than ever. If you’re mi­grat­ing from some­where else, our new au­to­matic zone scan­ning checks your do­main’s most com­mon record names and types, re­con­struct­ing your zone so you only have to make a few tweaks in­stead of start­ing from scratch. (You can also just up­load a BIND file if you pre­fer.)

Once your records are in place, the real magic hap­pens. With 1-Click Acceleration, you can en­able the CDN di­rectly from your DNS records. We’ll spin up a Pull Zone be­hind the scenes and in­stantly start rout­ing re­quests through our edge net­work. Once traf­fic is flow­ing, 1-Click Security lets you en­able Bunny Shield in­stantly to fil­ter traf­fic at the edge, block­ing com­mon ex­ploits and ab­sorb­ing DDoS at­tacks be­fore they ever touch your ori­gin server.

Performance, se­cu­rity, and rout­ing are now uni­fied in one place, rather than stitched to­gether af­ter the fact. Our goal is to keep evolv­ing this with even more ad­vanced record types.

Beyond mak­ing it free, we’re also mak­ing it bet­ter

Saving on costs is great, but ul­ti­mately, what dri­ves us at bunny.net is build­ing in­cred­i­ble prod­ucts. While all of that was hap­pen­ing, we’ve also been mak­ing steady changes to DNS it­self. There haven’t been many ma­jor re­leases, but rather a lot of smaller im­prove­ments over time.

IPv6 is no longer op­tional

More and more net­works de­fault to IPv6 now, es­pe­cially on mo­bile. So we made sure every­thing on our side just works in a dual-stack world.

If you’re us­ing Bunny DNS, your name­server records al­ready re­solve over both IPv4 and IPv6. There’s noth­ing to con­fig­ure, noth­ing to mi­grate. It just works the way it prob­a­bly should have a while ago.

We’ve added DNSSEC, with­out the usual trade-offs

DNSSEC is one of those things peo­ple want in the­ory but hes­i­tate to adopt in prac­tice.

Part of that is the com­plex­ity, but part of it is also that tra­di­tional DNSSEC can ex­pose in­for­ma­tion about your zone that you might not want to share.

We im­ple­mented DNSSEC with NSEC Black Lies to get around that. You still get the val­i­da­tion and pro­tec­tion against tam­per­ing, but with­out mak­ing it easy for some­one to walk your en­tire do­main struc­ture.

It’s one of those de­tails most peo­ple won’t no­tice di­rectly, but it does change how com­fort­able you can be with turn­ing DNSSEC on.

We’ve mod­ern­ized record types

DNS has moved far be­yond sim­ple ad­dress records. Modern ap­pli­ca­tions in­creas­ingly rely on DNS not just to point a name at an IP ad­dress, but to de­scribe how clients should con­nect, how cer­tifi­cates should be val­i­dated, and how se­cu­rity set­tings should be man­aged over time.

That’s why we’ve been ex­pand­ing Bunny DNS with sup­port for more ad­vanced record types.

We’ve added sup­port for HTTPS and SVCB records, which let you hint how clients should con­nect to your ser­vices. TLSA records are there if you’re us­ing DANE and want tighter con­trol over cer­tifi­cate val­i­da­tion. And CDS and CDNSKEY help au­to­mate DNSSEC key man­age­ment so you’re not ro­tat­ing things by hand.

None of this is par­tic­u­larly flashy, but it’s the kind of stuff you end up need­ing once you move past a ba­sic setup.

Helping you build faster

We take our mis­sion se­ri­ously, and the help” part of it is per­haps the most im­por­tant. By drop­ping us­age charges for DNS and in­te­grat­ing it deeply with the rest of our stack, start­ing with CDN and Shield, we want to help you build faster, safer, and more re­silient ap­pli­ca­tions with­out wor­ry­ing about ar­bi­trary lim­its.

In a world where every­one sim­ply wants to ship as many fea­tures as fast as pos­si­ble, we’re fo­cused on some­thing else: mak­ing Bunny DNS in­cred­i­ble to use and seam­lessly in­te­grated into every­thing else we do, so you can build faster, sleep eas­ier, and ul­ti­mately cre­ate build bet­ter user ex­pe­ri­ences for every­one.

If you haven’t tried Bunny DNS in a while, now’s a good time to take an­other look. You can add your zones, point your do­mains, and leave it at that, or start lay­er­ing on CDN and Shield when you need them.

It’s free now, so you can log in or sign up and start us­ing it straight away.

The Map — Jerry's Map

www.jerrysmap.com

Landing

The Map

Exhibitions

Videos

Sales

About/Contact

.

What is it?

In the sum­mer of 1963 Jerry be­gan draw­ing a map of an imag­i­nary city. The work started as a doo­dle done in the spare time he had while work­ing at a te­dious job. He con­tin­ued to add to that map through the years un­til, in 1983, he set it aside to put his free time to other use.

It was stored in the at­tic of his home in Cold Spring, New York. It gath­ered dust. Jerry’s son, Henry, found it one day while rum­mag­ing around. He brought it down and asked what it was. Seeing it then trig­gered Jerry to dust it off and con­tinue the pro­ject.

Years later, the Map is now a two-di­men­sional virtual world” art pro­ject which is now com­prised of over 4000 in­di­vid­ual eight by ten inch pan­els. When as­sem­bled, these pan­els form an ap­prox­i­mate cir­cle. The panel lo­ca­tions are de­fined by N, S, E, and W co­or­di­nates that orig­i­nate at the cen­ter of the cir­cle. The lo­ca­tions in the ma­trix do not change, but the pan­els them­selves are con­tin­u­ally re­vised based on in­struc­tions drawn from the artist’s cus­tom deck of cards.

Its ex­e­cu­tion, in acrylic, marker, col­ored pen­cil, ink, col­lage, and inkjet print on heavy pa­per, is dic­tated by the in­ter­play be­tween an elab­o­rate set of rules and ran­domly gen­er­ated in­struc­tions.

Jerry main­tained a blog about the pro­ject for many years. He no longer up­dates it, but the old posts are still avail­able on Blogger. And also be sure to check out r/​jer­rymap­ping,  an in­ter­est­ing sub­red­dit de­voted to map mak­ing in the style of Jerry’s Map.

Time lapse show­ing 20 years of changes to an 8 by 16 panel por­tion of the map. Sharp eyes will note that there are not ac­tu­ally 20 changes in this an­i­ma­tion. This is be­cause for some of those years, no cards were drawn that called for changes on these par­tic­u­lar pan­els.

The Creative Process

The Card Deck

The en­tire process is dri­ven by in­struc­tions on a card drawn from a spe­cial deck cre­ated by the artist. Each cy­cle be­gins only when the artist’s tasks from the pre­vi­ous card are com­plete. This could take any­where from a few min­utes to a few days.

The cards were first in­tro­duced as a sim­ple ran­dom num­ber gen­er­a­tor. When Jerry was first cre­at­ing the map it was sim­ple enough to work sheet to sheet, but as the map grew to hun­dreds of in­di­vid­ual pan­els it be­came very te­dious to make his way through the set.

I wanted to move through the stack faster, and the eas­i­est ran­dom num­ber sys­tem I could come up with was a deck of cards. I’d draw a card and move down that many pan­els in the stack.”

As Jerry be­gan work­ing on ways of sys­tem­atiz­ing the process of work­ing on the map he be­gan to in­cor­po­rate in­struc­tions on the cards. The con­tem­po­rary deck of cards has been adapted from play­ing cards and the to­tal num­ber varies as cards have been added, re­vised, and re­moved. Cur­rently there are ap­prox­i­mately 100 cards.

Sometimes I have feel­ings about the deck of cards. There’s a mes­sage in those cards. There’s no big man with a beard who has or­dered the cards, but I’m very in­ter­ested in see­ing what comes out of it. There’s a re­al­ity in there wait­ing to get out. It’s the map’s fu­ture pre­dic­tor and as it is al­ways chang­ing its alive…My hand puts the paint on the pa­per, I’ll step back and look at the sheets as though I was­n’t the per­pe­tra­tor but merely the ob­server.”

Interpreting the Cards

The in­struc­tions on each card have these three el­e­ments:

Card in­struc­tions for the Artist are in these five gen­eral cat­e­gories:

Next higher di­men­sion (void, red, black, zig­gu­rat)Spat­ter paint four con­tigu­ous pan­els (current panel plus the 3 clos­est to the cen­ter of that panel)Cre­ate a new seed pan­elMix a new paint col­orScreen print the 9 con­tigu­ous pan­els (current panel plus 8 sur­round­ing pan­els)

Next higher di­men­sion (void, red, black, zig­gu­rat)

Next higher di­men­sion (void, red, black, zig­gu­rat)

Spatter paint four con­tigu­ous pan­els (current panel plus the 3 clos­est to the cen­ter of that panel)

Spatter paint four con­tigu­ous pan­els (current panel plus the 3 clos­est to the cen­ter of that panel)

Create a new seed panel

Create a new seed panel

Mix a new paint color

Mix a new paint color

Screen print the 9 con­tigu­ous pan­els (current panel plus 8 sur­round­ing pan­els)

Screen print the 9 con­tigu­ous pan­els (current panel plus 8 sur­round­ing pan­els)

Update and copy the mas­ter (map el­e­ment) on the top of the stack of mas­tersCopy the cur­rent panel on la­bel pa­per so por­tions can be used in col­lage

Update and copy the mas­ter (map el­e­ment) on the top of the stack of mas­ters

Update and copy the mas­ter (map el­e­ment) on the top of the stack of mas­ters

Copy the cur­rent panel on la­bel pa­per so por­tions can be used in col­lage

Copy the cur­rent panel on la­bel pa­per so por­tions can be used in col­lage

Use a patch­work of re-used printed pa­per­board (e.g ce­real boxes)Use a photo from the artist’s file­sUse a lu­men print (objects scanned and printed)

Use a patch­work of re-used printed pa­per­board (e.g ce­real boxes)

Use a patch­work of re-used printed pa­per­board (e.g ce­real boxes)

Use a photo from the artist’s files

Use a photo from the artist’s files

Use a lu­men print (objects scanned and printed)

Use a lu­men print (objects scanned and printed)

Add or sub­tract from the num­ber on a spec­i­fied num­ber cards (e.g. add 3 to the num­ber on the last 2 cards”)Elim­i­nate a card from or add a card to the deck (eliminated cards are retired” for­ever, but the in­struc­tions are kept on a list for pos­si­ble re-use on a fu­ture card)Copy and re­tire the last 9 cards (physical cards are re­tired but the copies stay in the deck)Shuf­fle the deck

Add or sub­tract from the num­ber on a spec­i­fied num­ber cards (e.g. add 3 to the num­ber on the last 2 cards”)

Add or sub­tract from the num­ber on a spec­i­fied num­ber cards (e.g. add 3 to the num­ber on the last 2 cards”)

Eliminate a card from or add a card to the deck (eliminated cards are retired” for­ever, but the in­struc­tions are kept on a list for pos­si­ble re-use on a fu­ture card)

Eliminate a card from or add a card to the deck (eliminated cards are retired” for­ever, but the in­struc­tions are kept on a list for pos­si­ble re-use on a fu­ture card)

Copy and re­tire the last 9 cards (physical cards are re­tired but the copies stay in the deck)

Copy and re­tire the last 9 cards (physical cards are re­tired but the copies stay in the deck)

Shuffle the deck

Shuffle the deck

Do a blog en­tryDo a jour­nal en­try (also print and make col­lage ma­te­r­ial of the en­try)Do a Reddit post­Cal­cu­late the sales value of the en­tire set of pan­els (based lat­est eBay sale)

Do a blog en­try

Do a blog en­try

Do a jour­nal en­try (also print and make col­lage ma­te­r­ial of the en­try)

Do a jour­nal en­try (also print and make col­lage ma­te­r­ial of the en­try)

Do a Reddit post

Do a Reddit post

Calculate the sales value of the en­tire set of pan­els (based lat­est eBay sale)

Calculate the sales value of the en­tire set of pan­els (based lat­est eBay sale)

Card in­struc­tions for the Artist’s Helper are re­lated to:

The card in­di­cates the num­ber of pan­els to be scanned and added to the dig­i­tal li­brary.

The card in­di­cates the num­ber of pan­els to be scanned and added to the dig­i­tal li­brary.

The card in­di­cates the num­ber of pan­els to be scanned and added to the dig­i­tal li­brary.

The card asks the helper to sort re­tired pan­els and archive them.

The card asks the helper to sort re­tired pan­els and archive them.

The card asks the helper to sort re­tired pan­els and archive them.

The helper makes copies of the num­ber of cur­rent pan­els in­di­cated on the card, and the orig­i­nal pan­els are re­tired and archived.

The helper makes copies of the num­ber of cur­rent pan­els in­di­cated on the card, and the orig­i­nal pan­els are re­tired and archived.

The helper makes copies of the num­ber of cur­rent pan­els in­di­cated on the card, and the orig­i­nal pan­els are re­tired and archived.

The card asks the helper to up­date the in­ven­tory of the archives.

The card asks the helper to up­date the in­ven­tory of the archives.

The card asks the helper to up­date the in­ven­tory of the archives.

A TYPICAL DAY’S WORK

Jerry draws a card and works through the tasks it de­fines. This video gives some in­sight into what a typ­i­cal day’s work looks like.

The Principles

These are the in­struc­tions and rules which guide the Artist in the cre­ation of the map:

Each card has a large black or red num­ber in an up­per cor­ner. A task” is de­fined as the com­ple­tion of the num­ber of work units as spec­i­fied by the num­ber on the card that is drawn. A work unit is the num­ber of one inch squares to be cov­ered. The num­ber drawn and the ef­fort re­quired can be highly vari­able, so a day’s work could con­sist of one card’s work units, or just a por­tion of one. Work on an in­com­plete work unit con­tin­ues at the next work ses­sion.

Each card has a large black or red num­ber in an up­per cor­ner. A task” is de­fined as the com­ple­tion of the num­ber of work units as spec­i­fied by the num­ber on the card that is drawn. A work unit is the num­ber of one inch squares to be cov­ered. The num­ber drawn and the ef­fort re­quired can be highly vari­able, so a day’s work could con­sist of one card’s work units, or just a por­tion of one. Work on an in­com­plete work unit con­tin­ues at the next work ses­sion.

When a card is drawn you must fol­low the spe­cific in­struc­tions on the card, but those in­struc­tions may be changed for the next time that card is drawn.

When a card is drawn you must fol­low the spe­cific in­struc­tions on the card, but those in­struc­tions may be changed for the next time that card is drawn.

Work di­rec­tion is de­ter­mined by color of the drawn card - black is clock­wise, red is counter-clock­wise.

Work di­rec­tion is de­ter­mined by color of the drawn card - black is clock­wise, red is counter-clock­wise.

Every page has a center” point from which the work em­anates. The center” of the new page is the same as the par­en­t’s.

Every page has a center” point from which the work em­anates. The center” of the new page is the same as the par­en­t’s.

New pan­els are gen­er­ated by draw­ing a new panel” card, or a new panel is re­quired to com­plete a sec­tion of art.

New pan­els are gen­er­ated by draw­ing a new panel” card, or a new panel is re­quired to com­plete a sec­tion of art.

When a new page is added, the new page will use the color of the day”.

When a new page is added, the new page will use the color of the day”.

The lo­ca­tion of the new page is de­ter­mined by plac­ing a com­pass point in the center” of the par­ent page and de­ter­min­ing the clos­est edge of the map (this keeps the map roughly cir­cu­lar and grow­ing gen­er­ally equally in all di­rec­tions).

The lo­ca­tion of the new page is de­ter­mined by plac­ing a com­pass point in the center” of the par­ent page and de­ter­min­ing the clos­est edge of the map (this keeps the map roughly cir­cu­lar and grow­ing gen­er­ally equally in all di­rec­tions).

Master map shows the lo­ca­tions of the pan­els as de­fined by co­or­di­nates.

Master map shows the lo­ca­tions of the pan­els as de­fined by co­or­di­nates.

Colors are more ab­stract and do not nec­es­sar­ily rep­re­sent the phys­i­cal world. Colors may be ap­plied with ei­ther paint or mark­ers, or by us­ing col­lage. The 42 col­ors are con­tin­u­ally remixed to en­sure a spec­trum of paints.

Colors are more ab­stract and do not nec­es­sar­ily rep­re­sent the phys­i­cal world. Colors may be ap­plied with ei­ther paint or mark­ers, or by us­ing col­lage. The 42 col­ors are con­tin­u­ally remixed to en­sure a spec­trum of paints.

In memory of the man who put red and green squiggles under words

devblogs.microsoft.com

I re­cently learned of the pass­ing of some­one whose work nearly every­body knows, but no­body knows his name.

Tony Krueger is re­mem­bered in Wikipedia as the per­son who ported the game Chip’s Challenge to Windows for the Windows Entertainment Pack.¹ But that’s prob­a­bly not the code he wrote that touched the most peo­ple.

Tony worked on Word 1.0, 1.1, 2.0, then on Word for OS/2 and Word for Mac, then re­turned to Word 6.0 and sev­eral ver­sions be­yond that. He prob­a­bly holds the record for most ver­sions of Word shipped.”

In early ver­sions of Word, the Spell Check fea­ture was some­thing that you ex­plic­itly in­voked, and then you had to sit and wait while the pro­gram looked for all your po­ten­tially-mis­spelled words, and then showed them to you one at a time for a de­ci­sion on what to do for each one. Word did in­tro­duce an Auto Spell Check fea­ture to run spell check when the user was idle, so that when you hit the Spell Check but­ton, the re­sults were ready to go. However, the Auto Spell Check was still a block­ing op­er­a­tion. As a re­sult, a lot of users turned it off be­cause it al­ways seemed to de­cide Now would be a good time to spell-check the doc­u­ment” just as you wanted to do some­thing, forc­ing you to wait for the spell check pass to com­plete be­fore you could, say, save and exit.

Tony made the spell checker much more un­ob­tru­sive so that it did­n’t in­ter­fere with your fore­ground work. And when it found a prob­lem, in­stead of wait­ing for you to trig­ger a spell check, it im­me­di­ately drew red squig­gles un­der po­ten­tially-mis­spelled words (and later green squig­gles un­der po­ten­tial gram­mat­i­cal er­rors).

Tony was an early fan of the magic/​com­edy team Penn and Teller. A friend and col­league at­tended a show and hung out af­ter­ward to ask the duo to sign a photo for his friend Tony. He was on the team that did the red and green squig­gles in Word.”

Upon hear­ing this, Penn Jillette an­nounced in his sten­to­rian voice which filled the en­tire the­ater: The red and green squig­gles!? I love the red and green squig­gles!” Teller silently con­curred.

Tony re­ceived that au­to­graphed photo for his birth­day, and it was­n’t clear which he was more happy about, the au­to­graphed photo or the fact that Penn and Teller loved his fea­ture.

Many years later, Weird Al” Yankovic recorded a par­ody video ti­tled Word Crimes, in which the Word red squig­gles make a brief ap­pear­ance. That same friend got Weird Al” to au­to­graph the screen shot.

Today, there are red (and even green and blue) squig­gles in nearly every word proces­sor, and of­ten out­side word proces­sors. Tony did it first. The next time a red squig­gle catches one of your mis­takes, say thanks to Tony. I think he’d ap­pre­ci­ate it.

¹ Probably not as widely doc­u­mented is that he ac­com­plished this with­out the source code: He re­verse-en­gi­neered the MS-DOS ver­sion and then reim­ple­mented it for Windows.

Author

Raymond has been in­volved in the evo­lu­tion of Windows for more than 30 years. In 2003, he be­gan a Web site known as The Old New Thing which has grown in pop­u­lar­ity far be­yond his wildest imag­i­na­tion, a de­vel­op­ment which still gives him the hee­bie-jee­bies. The Web site spawned a book, co­in­ci­den­tally also ti­tled The Old New Thing (Addison Wesley 2007). He oc­ca­sion­ally ap­pears on the Windows Dev Docs Twitter ac­count to tell sto­ries which con­vey no use­ful in­for­ma­tion.

Founding a Company in Germany: €9,600, 152 Days, and I Still Can't Send an Invoice

paolino.me

I started found­ing my sec­ond com­pany in Germany in late January. It is now late June.

In that time, the state, two courts, a no­tary, a law firm, a tax firm, and soft­ware ven­dors have all found a way to bill me. Every sin­gle one of them, on time.

I have spent more than 9,600 eu­ros to start a com­pany: a lit­tle over 7,600 in fees and bills, plus 2,000 in share cap­i­tal frozen in an ac­count I am not al­lowed to touch. And af­ter five months, here is what I have to show for it:

I have not been able to send a sin­gle in­voice of my own.

Not one.

The work is hap­pen­ing. The clients are real. The one thing the state ex­ists to let me do, bill them cleanly, is the one thing I still can’t.

The time­line

23 JanFirst call with a law firm to set up the com­pany. The clock and the hourly billing start.

23 Jan

First call with a law firm to set up the com­pany. The clock and the hourly billing start.

5 FebI sign the man­date and send my ID. Drafting be­gins.

5 Feb

I sign the man­date and send my ID. Drafting be­gins.

18 FebThe struc­ture is set: PlentyLabs UG & Co. KG, tech­ni­cally two com­pa­nies. The name is a saga of its own.

18 Feb

The struc­ture is set: PlentyLabs UG & Co. KG, tech­ni­cally two com­pa­nies. The name is a saga of its own.

about 1 month of draft­ing

6 MarIncorporation doc­u­ments ready.

6 Mar

Incorporation doc­u­ments ready.

17 MarDocuments ap­proved. The hunt for a no­tary be­gins.

17 Mar

Documents ap­proved. The hunt for a no­tary be­gins.

7 days for the ap­point­ment

24 MarNotary in Berlin reads the deeds aloud and cer­ti­fies that I am who I say I am.€1,575.24No­tary fees

24 Mar

Notary in Berlin reads the deeds aloud and cer­ti­fies that I am who I say I am.

€1,575.24Notary fees

25 MarI pay in €2,000.00 of share cap­i­tal. Money I can­not touch; it has to stay there.€2,000.00Locked, not a fee

25 Mar

I pay in €2,000.00 of share cap­i­tal. Money I can­not touch; it has to stay there.

€2,000.00Locked, not a fee

26 MarThe reg­is­ter court de­mands a fee ad­vance.€300.00­Court ad­vance

26 Mar

The reg­is­ter court de­mands a fee ad­vance.

€300.00Court ad­vance

17 days af­ter the no­tary

10 AprFirst com­pany en­tered in the com­mer­cial reg­is­ter.

10 Apr

First com­pany en­tered in the com­mer­cial reg­is­ter.

1 week more

17 AprSecond com­pany en­tered.€260.00Reg­is­ter, 200 + 60

17 Apr

Second com­pany en­tered.

€260.00Register, 200 + 60

20 AprI ask the firm I al­ready pay to han­dle the tax reg­is­tra­tion too.

20 Apr

I ask the firm I al­ready pay to han­dle the tax reg­is­tra­tion too.

2.5 weeks just to start

6 MayBefore the tax work can be­gin, a fresh en­gage­ment is re­quired: pro­posal, power of at­tor­ney, ID checks, per com­pany.€630.00­Tax reg­is­tra­tion quote

6 May

Before the tax work can be­gin, a fresh en­gage­ment is re­quired: pro­posal, power of at­tor­ney, ID checks, per com­pany.

€630.00Tax reg­is­tra­tion quote

28 MayThe in­cor­po­ra­tion le­gal bill lands.€4,462.50Le­gal fees

28 May

The in­cor­po­ra­tion le­gal bill lands.

€4,462.50Legal fees

29 MayTax ques­tion­naires sub­mit­ted. I re­quest stan­dard VAT and a VAT ID, ur­gently.

29 May

Tax ques­tion­naires sub­mit­ted. I re­quest stan­dard VAT and a VAT ID, ur­gently.

3 JunFirst bill from the ac­count­ing soft­ware.€426.97Ac­count­ing soft­ware

3 Jun

First bill from the ac­count­ing soft­ware.

€426.97Accounting soft­ware

9 JunI am told the VAT ID will ar­rive by post. A let­ter.

9 Jun

I am told the VAT ID will ar­rive by post. A let­ter.

24 Jun, to­day­Seven weeks since the tax firm, al­most four weeks since the ques­tion­naires. No VAT ID. No in­voice sent.

24 Jun, to­day

Seven weeks since the tax firm, al­most four weeks since the ques­tion­naires. No VAT ID. No in­voice sent.

Billed by every­one else€7,654.71

Share cap­i­tal I can­not touch€2,000.00

Total gone€9,654.71

Invoices I have man­aged to send0

Everyone in this story could in­voice me. I am the only one who can’t in­voice any­one.

But you can in­voice your German clients”

The clients abroad need a VAT ID for re­verse charge, and that is ex­actly the one I am still wait­ing for. My German clients I could bill to­day. But a do­mes­tic in­voice now would have to be reis­sued the mo­ment the VAT ID ar­rives. Bill now, bill again later, for no rea­son. So those wait too.

This should have been a web form

Fill it in, pay a fee, get your com­pany and your VAT ID in a week. Estonia does it. The UK reg­is­ters a com­pany in a day, on­line, for the price of a din­ner. There is no law of na­ture that says in­cor­po­ra­tion has to take five months and ar­rive by post.

Germany has built a process that chains one de­pen­dency to the next, puts a fee on each, and lets a founder run up le­gal bills, no­tary bills, court fees, tax re­tain­ers, and soft­ware sub­scrip­tions on zero rev­enue, all be­fore grant­ing the one per­mis­sion a com­pany ex­ists for: the right to send an in­voice.

If you ask the gov­ern­ment, the rea­son is trust: the no­tary, the cap­i­tal, the reg­is­ters, the end­less checks, all there to keep bad ac­tors out. This is the same ma­chine that did not catch Wirecard, a two-bil­lion-euro scam. It does, some­how, gen­er­ate enough fric­tion to scare new founders out of the coun­try.

And no, I could not just leave in­stead. My first com­pany, Freshflow, is valu­able enough that walk­ing out of Germany would trig­ger a mas­sive six-fig­ure exit tax, on gains I have not even re­alised, purely for the priv­i­lege of leav­ing. But that is a story for an­other post.

This is a coun­try tax­ing am­bi­tion through the roof be­fore you’ve earned a cent, then won­der­ing why the am­bi­tious leave.

Bonus round: my com­pany name was too generic”

Have you heard of Apple? A piece of fruit, and one of the most valu­able brands ever built. That name would never have been ap­proved in Germany.

Naming a com­pany is hard. It is the word every­one who touches your work will re­mem­ber. After months of turn­ing it over, I found one I could stand be­hind, a name that says what I be­lieve soft­ware should be. (That be­lief will be its own post, soon.) Distinctive, I thought. The kind of name you do not for­get.

Plenty.

No,” said the lawyer. German com­pany names have to be dis­tinc­tive, and Plenty” is a plain English word. Berlin would re­ject it.

Plenty Group?” Two plain words. Plenty Labs?” Labs” is a plain word too. Plenty.is?” A generic word with a do­main on the end is still a generic word, and there was case law to prove it.

The sug­ges­tions were worse: stick my sur­name on the front, Paolino Plenty Labs. Or a pre­fix, PG Plenty Germany. Or make up a fan­tasy word.

Is Plenty. Its Plenty. IsPlenty. ItsPlenty. Rejected, all of it.

Fine. They wanted a mean­ing­less word; I gave them one. Plenty Labs, mi­nus the space: PlentyLabs.

Approved.

A name that started out of spite. Weeks of cor­re­spon­dence, re­solved by re­mov­ing a space. A rule that does not re­ward clar­ity. It re­wards non­sense.

Postscript: why a UG and Co. KG, two com­pa­nies?

Why does a one-per­son busi­ness need two com­pa­nies? Because the sim­ple ver­sion is worse, and be­cause I am build­ing it into some­thing big­ger.

The sim­plest setup is a sole pro­pri­etor­ship. Thirty eu­ros, no cap­i­tal, done in an af­ter­noon. It also makes me per­son­ally li­able for every­thing. A client sues? They are not su­ing a com­pany. They are su­ing me. My sav­ings, my apart­ment, my name.

So I wanted real lim­ited li­a­bil­ity, which means a com­pany. And for one per­son, the clean­est com­pany turns out not to be one com­pany. It is a KG, a part­ner­ship that does the work, with a tiny UG stand­ing in as the part­ner that car­ries the li­a­bil­ity. Strange, but stan­dard. You prob­a­bly have seen GmbH & Co. KG on German com­pa­nies a hun­dred times with­out won­der­ing why. This is why.

It is taxed the sane way, too. The part­ner­ship’s profit is taxed once, as my in­come, since I am the one who ends up with it. A plain UG would tax the com­pany first, then tax me again when I paid my­self.

Why a UG and not the fa­mous GmbH? A GmbH wants 25,000 eu­ros sit­ting in a bank ac­count be­fore it is al­lowed to ex­ist. The UG lets you start with al­most noth­ing, on one con­di­tion: lock away a quar­ter of every year’s profit un­til the re­serve reaches 25,000, then con­vert to a GmbH. The 25,000 does not go away. Germany just takes it in in­stal­ments.

Extreme Heat: Improving governance and strengthening action around the world - cancelled - Grantham Research Institute on climate change and the environment

www.lse.ac.uk

Credit: mam­muth/​is­tock

We re­gret that this event has been can­celled due to the red ex­treme heat warn­ing is­sued by the UK Met Office.

This London Climate Action Week event will open with the an­nounce­ment of the in­au­gural Adeline Stuart-Watt Award win­ner and will be fol­lowed by a ses­sion fo­cused on im­prov­ing ex­treme heat gov­er­nance and ac­tion around the world. Hosted in col­lab­o­ra­tion with the Zurich Climate Resilience Alliance.

The Adeline Stuart-Watt Award cel­e­brates the legacy of Ade­line Stuart-Watt, a highly re­spected and very sadly missed friend and col­league at the LSEs Grantham Research Institute on Climate Change and the Environment and the Zurich Climate Resilience Alliance. The Award recog­nises out­stand­ing, pol­icy-rel­e­vant re­search con­tri­bu­tions to the field of cli­mate adap­ta­tion and re­silience by post­grad­u­ate stu­dents.

An overview of the Award process will be pro­vided by Candice Howarth, fol­lowed by the Award win­ner be­ing an­nounced by Professor Lord Nicholas Stern. The win­ner will then re­ceive their award be­fore pre­sent­ing an overview of the win­ning topic. The Adeline Stuart-Watt Award is gen­er­ously sup­ported by the Z Zurich Foundation.

The sec­ond part of the event will fo­cus on Extreme Heat: Improving Governance and strength­en­ing ac­tion around the world.

Chaired by Swenja Suminski, this sec­tion of the event will ex­plore the crit­i­cal need to im­prove ex­treme heat gov­er­nance glob­ally. The event will bring to­gether ex­per­tise from the Grantham Research Institute and Zurich Climate Resilience Alliance part­ners Mercy Corps, Practical Action and the IFRC along­side in­put from lead­ing global part­ners.

The event will share new analy­sis of ex­treme heat gov­er­nance progress and chal­lenges across coun­tries where the Zurich Climate Resilience Alliance op­er­ates along with sto­ries from county pro­grammes.

The event will fin­ish with a fire side chat ses­sion where speak­ers will re­flect on key chal­lenges and op­por­tu­ni­ties for ad­vanc­ing ex­treme heat gov­er­nance glob­ally.

Meet our con­trib­u­tors:

Professor Lord Nicholas Stern, Chair of the LSE Grantham Research Institute

Candice Howarth, Research Director at Quadrature Climate Foundation and Visiting Professor in Practice at the LSE Grantham Research Institute

Swenja Surminski, Managing Director Climate and Sustainability at Marsh and Professor in Practice at the LSE Grantham Research Institute

Anna Beswick, Senior Policy Fellow (Adaptation and Resilience) at the LSE Grantham Research Institute

Martina Podesta, Policy Officer (Adaptation Policy and Governance) at the LSE Grantham Research Institute

Marc Gordon, Global Lead, Extreme Heat Risk Reduction & Senior Coordinator of the Midterm Review of the Sendai Framework (MTR SF), Centre of Excellence for Climate and Disaster Resilience at the United Nations

Ninni Ikkala Nyman, Lead, Climate Change, International Federation of Red Cross and Red Crescent Societies.

Olivia Shears, Head of CCRA and adap­ta­tion progress re­port­ing, UK Climate Change Committee.

Mary McBryde, Chief Program Officer, HERA.

The worthlessness of vitamin D is mildly exaggerated

dynomight.net

For a while there, many peo­ple thought vi­t­a­min D was mag­i­cal—that it could im­prove bones, the heart, in­fec­tions, can­cer, heart dis­ease, longevity, even men­tal health. But among peo­ple I re­spect, opin­ion is now over­whelm­ingly that tak­ing vi­t­a­min D does noth­ing un­less you’re se­verely de­fi­cient. The cen­tral ar­gu­ment is that while vi­t­a­min D lev­els are cor­re­lated with ~all pos­i­tive health out­comes, when you ac­tu­ally test vi­t­a­min D sup­ple­ments against placebo in ran­dom­ized tri­als, noth­ing ever hap­pens.

That’s what I used to think, too. But I’ve come to think the skep­tics have over-cor­rected. Yes, ran­dom­ized tri­als have shown the mag­i­cal cor­re­la­tions are not causal. But if you start with non-in­sane ex­pec­ta­tions, the tri­als look like weak but pos­i­tive ev­i­dence. And if you con­sider what we know about bi­ol­ogy and evo­lu­tion, I think the bal­ance of ev­i­dence tips pretty clearly in the di­rec­tion that peo­ple with low-ish lev­els would be wise to sup­ple­ment.

Am I cer­tain that vi­t­a­min D is ben­e­fi­cial for peo­ple with low-ish lev­els? Absolutely not! But I claim that’s the best bet given the lim­its of our knowl­edge.

The clas­si­cal view: Boring bone vi­t­a­min

Most vi­t­a­mins are ingredients” that the body uses to do stuff. Vitamin D is more like a signal” that the body uses to com­mu­ni­cate with it­self about what to do.1 The clas­si­cal endocrine” story of vi­t­a­min D is that your body uses it to tell your guts to take in more cal­cium from food. If you don’t get enough vi­t­a­min D, then you have cal­cium prob­lems.

That’s all you re­ally need to know about the clas­si­cal view. But if you en­joy gawk­ing at bi­ol­o­gy’s com­plex­ity, I rec­om­mend this di­a­gram and the fol­low­ing three para­graphs:

Ready for sci­ence? OK: Almost all the cells in your body make provi­t­a­min D.2 Usually, this is all con­verted to cho­les­terol, but your skin cells leave some sit­ting around. When UVB light hits those skin cells, provi­t­a­min D is trans­formed (physically by the light it­self) into pre­vi­t­a­min D and then (by heat) into vi­t­a­min D. This dif­fuses from the skin cells into blood ves­sels. There it binds to a pro­tein3 and starts cir­cu­lat­ing in the blood, where it is joined by vi­t­a­min D from food.4 Eventually, the liver con­verts it into more-sta­ble stor­age vi­t­a­min D. It also soaks in and out of fat and mus­cle tis­sue, which acts as a slow-re­lease reser­voir.

Now, a fun fact: If cal­cium lev­els in your blood get too low, then your heart will stop work­ing and you will die. To avoid this, you have parathy­roid glands in your neck that sense when cal­cium is get­ting low, and re­lease parathy­roid hor­mone into the blood. This tells your bones to re­lease some of their stored cal­cium. It also tells your kid­neys to con­vert some of the stor­age vi­t­a­min D from your blood into ac­tive vi­t­a­min D. And when that gets to your guts, they try to ab­sorb more cal­cium from food.

So what hap­pens if you don’t get enough vi­t­a­min D? Well, your body is not go­ing to let cal­cium lev­els drop too low, be­cause your body is de­signed to avoid death. Parathyroid hor­mone will still get se­creted, and cal­cium will still get scav­enged from your bones. But with­out vi­t­a­min D, your guts never get the sig­nal to gather ex­tra cal­cium from food. So the body scav­enges a lot of cal­cium from your bones, and you end up with weak bones, which is bad.

Now here’s the thing: In this story, only ac­tive vi­t­a­min D ac­tu­ally does any­thing. The kid­neys make this on de­mand in re­sponse to cal­cium lev­els, not in re­sponse to stor­age vi­t­a­min D lev­els. General opin­ion is that as long as the blood has above ~25 nmol/​L of stor­age vi­t­a­min D, then the kid­neys have no trou­ble mak­ing ac­tive vi­t­a­min D.5 Furthermore, sur­vey data sug­gests that only ~2% of the pop­u­la­tion has lev­els be­low that thresh­old. This sug­gests that for ~98% of peo­ple, sup­ple­ment­ing vi­t­a­min D should do ap­prox­i­mately noth­ing.

The cor­re­la­tion view: Magical mys­tery cure

Rickets is a ter­ri­ble dis­ease that in­volves soft bones, stunted growth, and skele­tal de­for­mi­ties. It’s prob­a­bly been with us since an­cient times, but it be­came com­mon in the West af­ter the in­dus­trial rev­o­lu­tion. In 1890, a Scottish mis­sion­ary named Theobald Palm ob­served that rick­ets was com­mon in smog-rid­den UK cities but al­most un­heard of in sunny coun­tries with poor san­i­ta­tion, sug­gest­ing sun­light it­self was the is­sue. This con­tributed to the dis­cov­ery that rick­ets could be cured with UV light or cod-liver oil, and even­tu­ally the dis­cov­ery of vi­t­a­min D.

In 1941, Apperly no­ticed that the amount of sun­light in dif­fer­ent US states was pos­i­tively cor­re­lated with skin can­cer but in­versely cor­re­lated with over­all can­cer mor­tal­ity.6 He gave this charm­ing graph:

Apperly never men­tions vi­t­a­min D, pre­sum­ably be­cause he thought it was a bor­ing bone vi­t­a­min.

Things took off in 1980, when Cedric and Frank Garland pub­lished, Do Sunlight and Vitamin D Reduce the Likelihood of Colon Cancer?” Seemingly un­aware of Apperly, they gave a sim­i­lar, but uglier, graph:

They point out that re­gional di­ets (like meat and fiber) did­n’t seem to ex­plain this pat­tern. Instead, they pro­pose a mech­a­nis­tic story:

Sunlight ↓ Vitamin D ↓ Adequate cal­cium in blood ↓ Reduced in­flam­ma­tion of ep­ithe­lial cells in the colon ↓ Less colon can­cer

(It’s al­ways in­flam­ma­tion.) This pa­per was re­jected many times be­fore fi­nally be­ing pub­lished. I wish I could find an un-gated copy to link to, be­cause it would have made a mag­nif­i­cent blog post.7

Following that pa­per, there was an ex­plo­sion of work that found neg­a­tive cor­re­la­tions be­tween sun­light (or lat­i­tude) and other types of can­cers as well as blood pres­sure, di­a­betes, and mul­ti­ple scle­ro­sis.

Then peo­ple started mea­sur­ing vi­t­a­min D in blood. In 1989, the Garlands and col­lab­o­ra­tors found blood sam­ples takin in 1974 from 25,000 peo­ple. They found that 34 of those peo­ple had since got­ten colon can­cer. They matched these with 67 de­mo­graph­i­cally sim­i­lar peo­ple and mea­sured vi­t­a­min D lev­els in the stored blood sam­ples for all 101 peo­ple. Among that group, peo­ple with vi­t­a­min D lev­els be­low 50 nmol/​L got colon can­cer more than three times as of­ten as peo­ple with higher lev­els.

Again, many sim­i­lar stud­ies fol­lowed. These linked higher vi­t­a­min D lev­els to bet­ter out­comes in car­dio­vas­cu­lar dis­ease, di­a­betes, obe­sity, in­fec­tious dis­ease, Parkinson’s, and mood dis­or­ders. While re­sults were mixed for non-col­orec­tal can­cer in­ci­dence, higher vi­t­a­min D lev­els pre­dicted bet­ter sur­vival of many can­cers. Amazingly, all-cause mor­tal­ity was roughly 30% lower for those at the 75th per­centile of vi­t­a­min D lev­els com­pared to the 25th.

Vitamin D was look­ing like a mir­a­cle. But how could it do all that stuff if it was just a bor­ing bone vi­t­a­min?

Meanwhile in bi­ol­ogy

While all these cor­re­la­tions were be­ing dis­cov­ered, we learned that the body does­n’t just use vi­t­a­min D for bone stuff.

In 1969, we dis­cov­ered the vi­t­a­min D re­cep­tor that ac­tive vi­t­a­min D binds to in the gut and bones. And in the 1980s came a shock: Almost all cells in the body have vi­t­a­min D re­cep­tors. These seem to do dif­fer­ent things in dif­fer­ent tis­sues. In the pan­creas, they sup­port in­sulin se­cre­tion. In im­mune cells, they boost an­timi­cro­bial pep­tides and re­duce in­flam­ma­tion. In neu­rons, they in­flu­ence pro­lif­er­a­tion and dif­fer­en­ti­a­tion.

So… What? When cal­cium drops and the kid­neys put out ac­tive vi­t­a­min D, does every part of the body start do­ing dif­fer­ent un­re­lated stuff?

In the late 1990s, we cloned the gene for the en­zyme that the kid­neys use to con­vert stor­age vi­t­a­min D to ac­tive vi­t­a­min D. Soon came an­other shock: This en­zyme also ex­ists in tons of other cells, in­clud­ing im­mune cells, the heart, the skin, the prostate, the breast, and colon. (Another win for the Garlands.)

So it’s not just the kid­neys mak­ing ac­tive vi­t­a­min D to trig­ger the gut. Cells every­where are mak­ing their own ac­tive vi­t­a­min D and us­ing it to trig­ger vi­t­a­min D re­cep­tors in neigh­bor­ing cells, or even in­side the same cell.8 This of­ten has lit­tle to do with cal­cium or bones.9

So:

The kid­neys use vi­t­a­min D as a bor­ing bone hor­mone.

As long as the blood con­tains at least ~25 nmol/​L of stor­age vi­t­a­min D, the kid­neys don’t care. They cre­ate the same amount of ac­tive vi­t­a­min D, in re­sponse to cal­cium lev­els.

But now cells every­where are us­ing stor­age vi­t­a­min D.

To do god-knows-what.

With god-knows-what sen­si­tiv­ity to cir­cu­lat­ing vi­t­a­min D lev­els.

And re­mem­ber how only ac­tive vi­t­a­min D does any­thing? That’s wrong. In the mid-1970s, we learned that stor­age vi­t­a­min D also binds to the vi­t­a­min D re­cep­tor. The affin­ity is 100 – 1000× lower, but have ~1000× more in your blood. So maybe cir­cu­lat­ing lev­els of stor­age vi­t­a­min D them­selves mat­ter, in­de­pen­dently of how much ac­tive vi­t­a­min D gets made?

If that’s not con­fus­ing enough, peo­ple also no­ticed that while ac­tive vi­t­a­min D lev­els in the blood aren’t cor­re­lated with stor­age vi­t­a­min D (above ~25 nmol/​L), lev­els of parathy­roid hor­mone (the thing your parathy­roid glands use to tell your kid­neys to make ac­tive vi­t­a­min D) seem to de­cline as lev­els of stor­age vi­t­a­min D rise from ~25 to 50 or 75 nmol/​L. Huh?10

On the one hand, all this makes the idea that vi­t­a­min D could be a mir­a­cle more plau­si­ble. On the other hand, this is get­ting com­pli­cated. And do we re­ally be­lieve that rais­ing your vi­t­a­min D lev­els from the 25th to the 75th per­centile could re­duce your risk of death from any cause by thirty per­cent? Maybe we should try giv­ing peo­ple vi­t­a­min D and see what hap­pens.

Then came the RCTs

There have been many ran­dom­ized tri­als. The right” thing to do in such cases is to look at meta analy­ses that care­fully com­bine all the data. We’ll get to those. But they con­ceal a lot of im­por­tant nu­ance about what ac­tu­ally hap­pens on the ground dur­ing these tri­als. So let’s start by go­ing over the three main megatrials”.

The Women’s Health Initiative (WHI) trial came out in 2006 and is still the largest vi­t­a­min D trial ever done. This took 36,000 post­menopausal American women and as­signed half to take 400 IU daily with cal­cium and the other half to placebo.11 After seven years, here’s what hap­pened:12

(The haz­ard ra­tio is the ra­tio of the rate that some­thing hap­pens in the treat­ment vs. placebo groups. So, a num­ber less than one sug­gests a ben­e­fit to tak­ing vi­t­a­min D, while a num­ber larger than one sug­gests a harm. The num­bers in paren­the­ses show a 95% con­fi­dence in­ter­val.)

The only sta­tis­ti­cally sig­nif­i­cant re­sult was a bad one: Extra kid­ney stones, likely from the ex­tra cal­cium.13 The other out­comes look vaguely good, but none were sta­tis­ti­cally sig­nif­i­cant de­spite the mas­sive sam­ple size.

This was dis­ap­point­ing. However, the WHI trial had lim­i­ta­tions: Many sub­jects in both the vi­t­a­min D and placebo groups were al­ready tak­ing vi­t­a­min D, and con­tin­ued tak­ing it through the trial. The dose of 400 IU was fairly low, many sub­jects stopped tak­ing their pills, and vi­t­a­min D lev­els did­n’t ac­tu­ally change that much. They also mea­sured vi­t­a­min D lev­els in only 6% of sub­jects, mean­ing we can’t com­pare the fates of sub­jects who started out with low ver­sus high lev­els.

The next big hope was VITAL, which came out in 2018. They re­cruited 26,000 older peo­ple across the United States, half of them men and 20% Black (and thus far more likely to be vi­t­a­min-D de­fi­cient). They mea­sured vi­t­a­min D lev­els in most peo­ple, and they gave the treat­ment group 2,000 IU per day.14 Here were the re­sults af­ter 5.3 years:

Some of the re­sults look good-ish, but car­dio­vas­cu­lar mor­tal­ity was higher in the treat­ment group, lead­ing to al­most no ef­fect on all-cause mor­tal­ity.15 More dis­ap­point­ment.

The last mega­trial was D-Health, which came out in 2022 based on 21,000 older Australians. Instead of daily sup­ple­ments, it used a monthly bolus” dose of 60,000 IU or placebo. Unlike in VITAL, there was no ex­clu­sion for peo­ple with a his­tory of car­dio­vas­cu­lar dis­ease or can­cer, and less re­stric­tion on how much vi­t­a­min D par­tic­i­pants could take on their own dur­ing the trial.16 Here were the re­sults af­ter 6 years:

Now, the treat­ment group did bet­ter in terms of car­dio­vas­cu­lar dis­ease, but worse in can­cer and worse in all-cause mor­tal­ity. Even more dis­ap­point­ment.

Just from these three large tri­als, the main les­son should al­ready be clear: Vitamin D is not a mir­a­cle. The cor­re­la­tions were wrong.17 There is es­sen­tially zero re­main­ing hope that tak­ing vi­t­a­min D could re­duce all-cause mor­tal­ity by a third.

In this sense, the vi­t­a­min D skep­tics are def­i­nitely right. But what about the other tri­als? And is there a more sub­tle les­son?

I made some ta­bles

I wanted a big table that sum­ma­rized all the ma­jor vi­t­a­min D RCTs and what they found for dif­fer­ent health out­comes. Annoyingly, no such overview ap­pears to ex­ist. So I made my own:18

Lots of the haz­ard ra­tios are less than one, sug­gest­ing a ben­e­fit to sup­ple­men­ta­tion. But lots of them are also higher than one, sug­gest­ing a harm. The num­bers that are far from one al­most al­ways come from smaller tri­als, which man­i­fest as larger con­fi­dence in­ter­vals. If you’re in­ter­ested in the de­tails of how these tri­als were run, I re­fer you to more gi­gan­tic ta­bles in a foot­note.19

If big ta­bles aren’t your thing, here are some for­mal meta-analy­ses, both some re­cent ones and an older but more com­pre­hen­sive Cochrane re­view:

Squinting at the data

There are var­i­ous ways you could try to squint at these RCT. In al­most all of them, most peo­ple al­ready had pretty high lev­els be­fore they started. So why don’t we sep­a­rate out peo­ple who started low? Usually we can’t, be­cause most tri­als did­n’t mea­sure base­line vi­t­a­min D.20 And among the tri­als that did, there are few peo­ple with low lev­els, so the re­sults are noisy and con­fus­ing.21

Or, you might the­o­rize that ben­e­fits would take time to show up, mean­ing the first cou­ple years just add noise. In some cases—no­tably VITAL—excluding the first two years seems to help, but in other cases things get worse.22

Finally, some peo­ple spec­u­late that tak­ing gi­gan­tic monthly or quar­terly bolus” doses of vi­t­a­min D might be dan­ger­ous. For ex­am­ple, here’s an en­joy­able para­graph from Kunzia et al. in their meta-analy­sis of vi­t­a­min D and can­cer mor­tal­ity:

Our re­sults show­ing ef­fi­cacy of daily, but not bo­lus, vi­t­a­min D3 sup­ple­men­ta­tion in re­duc­ing can­cer mor­tal­ity are con­sis­tent with pre­vi­ous meta-analy­ses on can­cer mor­tal­ity or all-cause mor­tal­ity (Guo et al., 2022; Keum et al., 2022; Keum et al., 2019; Zhang et al., 2022; Zhang et al., 2019). However, by in­clud­ing more tri­als than these pre­vi­ous meta-analy­ses, we were able to de­tect sta­tis­ti­cally sig­nif­i­cant ef­fect mod­i­fi­ca­tion by treat­ment reg­i­men for the first time with sta­tis­ti­cal sig­nif­i­cance (pinteraction=0.042). The pat­tern of in­take could be im­por­tant for a favourable steady state of the bioavail­abil­ity of the ac­tive 1,25 (OH)₂D hor­mone. Daily ad­min­is­tra­tion coun­ter­acts the fast ex­cre­tion of vi­t­a­min D from the cir­cu­la­tion (Hollis and Wagner, 2013; Keum et al., 2022). Moreover, the en­zymes CYP27B1 (converts 25(OH)D to 1,25 (OH)₂D) and CYP24A1 (inactivates 25(OH)D and 1,25(OH)₂D) fol­low first-or­der re­ac­tion ki­net­ics (Vieth, 2009). This means that dou­bling the con­cen­tra­tion of the pre­cur­sor dou­bles the yield of the prod­uct, un­like other steroid hor­mones (e.g., cor­ti­sol, oe­stro­gen, testos­terone) that fol­low zero-or­der ki­net­ics (Vieth, 2020). Intermittent, non-phys­i­o­log­i­cally large vi­t­a­min D3 bo­lus doses may lead to un­sta­ble cy­cling of 25(OH)D and 1,25(OH)₂D lev­els in blood be­cause the sys­tem needs time to adapt to the large doses (Hollis and Wagner, 2013; Keum et al., 2019; Vieth, 2020). In the long run, in­ter­mit­tent bo­lus reg­i­mens at weekly or larger in­ter­vals can lead to an up-reg­u­la­tion of coun­ter­vail­ing fac­tors (e.g., 24-hydroxylase (CYP24A1), 24,25(OH)2D and fi­brob­last growth fac­tor 23), all of which ul­ti­mately leads to lower syn­the­sis or higher degra­da­tion of 1,25(OH)₂D lev­els (Mazess et al., 2021). Bolus doses, un­like daily doses, failed to re­duce C-reactive pro­tein re­sponse and ac­tu­ally el­e­vated anti-in­flam­ma­tory cy­tokines and dou­bled the risk of hy­per­cal­cemia in pre­vi­ous stud­ies (Krishnan et al., 2012; Martineau et al., 2017; Mazess et al., 2021).

Our re­sults show­ing ef­fi­cacy of daily, but not bo­lus, vi­t­a­min D3 sup­ple­men­ta­tion in re­duc­ing can­cer mor­tal­ity are con­sis­tent with pre­vi­ous meta-analy­ses on can­cer mor­tal­ity or all-cause mor­tal­ity (Guo et al., 2022; Keum et al., 2022; Keum et al., 2019; Zhang et al., 2022; Zhang et al., 2019). However, by in­clud­ing more tri­als than these pre­vi­ous meta-analy­ses, we were able to de­tect sta­tis­ti­cally sig­nif­i­cant ef­fect mod­i­fi­ca­tion by treat­ment reg­i­men for the first time with sta­tis­ti­cal sig­nif­i­cance (pinteraction=0.042). The pat­tern of in­take could be im­por­tant for a favourable steady state of the bioavail­abil­ity of the ac­tive 1,25 (OH)₂D hor­mone. Daily ad­min­is­tra­tion coun­ter­acts the fast ex­cre­tion of vi­t­a­min D from the cir­cu­la­tion (Hollis and Wagner, 2013; Keum et al., 2022). Moreover, the en­zymes CYP27B1 (converts 25(OH)D to 1,25 (OH)₂D) and CYP24A1 (inactivates 25(OH)D and 1,25(OH)₂D) fol­low first-or­der re­ac­tion ki­net­ics (Vieth, 2009). This means that dou­bling the con­cen­tra­tion of the pre­cur­sor dou­bles the yield of the prod­uct, un­like other steroid hor­mones (e.g., cor­ti­sol, oe­stro­gen, testos­terone) that fol­low zero-or­der ki­net­ics (Vieth, 2020). Intermittent, non-phys­i­o­log­i­cally large vi­t­a­min D3 bo­lus doses may lead to un­sta­ble cy­cling of 25(OH)D and 1,25(OH)₂D lev­els in blood be­cause the sys­tem needs time to adapt to the large doses (Hollis and Wagner, 2013; Keum et al., 2019; Vieth, 2020). In the long run, in­ter­mit­tent bo­lus reg­i­mens at weekly or larger in­ter­vals can lead to an up-reg­u­la­tion of coun­ter­vail­ing fac­tors (e.g., 24-hydroxylase (CYP24A1), 24,25(OH)2D and fi­brob­last growth fac­tor 23), all of which ul­ti­mately leads to lower syn­the­sis or higher degra­da­tion of 1,25(OH)₂D lev­els (Mazess et al., 2021). Bolus doses, un­like daily doses, failed to re­duce C-reactive pro­tein re­sponse and ac­tu­ally el­e­vated anti-in­flam­ma­tory cy­tokines and dou­bled the risk of hy­per­cal­cemia in pre­vi­ous stud­ies (Krishnan et al., 2012; Martineau et al., 2017; Mazess et al., 2021).

Oh no, up-reg­u­la­tion of fi­brob­last growth fac­tor 23!23

I don’t feel like I un­der­stand this deeply enough to have any opin­ion be­yond the sur­face level that the body seems to adapt to large doses of vi­t­a­min D in ways that could pos­si­bly be bad.24 It seems in­tu­itive that small daily doses would be safer than gi­gan­tic monthly doses, but I’m al­ways sus­pi­cious of post-hoc mech­a­nis­tic spec­u­la­tion. Also, if peo­ple get enough sun, they can ap­par­ently syn­the­size 10,000 – 25,000 IU per day, which is­n’t that far from the 60,000 IU they got in the D-Health trial. But then again, I think Kunzia et al. are sug­gest­ing that the body is de­signed to adapt to reg­u­lar ex­po­sure to large doses but not in­ter­mit­tent ex­po­sure?

Well, if you split up the trails by daily vs. bo­lus dos­ing, there’s a de­cent pat­tern of daily dos­ing lead­ing to bet­ter re­sults:

If those bo­lus dos­ing tri­als did­n’t ex­ist, I’d think this looked pretty good. So, maybe? Or maybe this is a story made up to hal­lu­ci­nate a pos­i­tive trend. I would lean to­wards the lat­ter the­ory, but there are pa­pers like Mazess et al.’s Vitamin D: Bolus is Bogus”, that sug­gested this pat­tern be­fore D-Health’s dis­mal re­sults came out. There are even some tri­als that sug­gest bo­lus doses don’t even work for treat­ing rick­ets. So… I’m still not con­vinced. But maybe.

Aside: There are also many Mendelian ran­dom­iza­tion stud­ies that look at cor­re­la­tions be­tween health and genes that are re­lated to vi­t­a­min D. But I don’t think these pro­vide much in­for­ma­tion, be­cause the as­sump­tions are shaky and the genes don’t ex­plain much of the vari­ance.25

Where are we?

Still with me? Here’s a sum­mary of the above 5200 words:

The body uses vi­t­a­min D in all sorts of weird and com­pli­cated ways. It’s bi­o­log­i­cally plau­si­ble that vi­t­a­min D could mat­ter be­yond bone stuff with se­vere de­fi­ciency, but there’s no con­vinc­ing mech­a­nis­tic ev­i­dence that it is.

Vitamin D lev­els are strongly cor­re­lated with good health out­comes, but RCTs have con­clu­sively shown that most of these cor­re­la­tions are non-causal.

RCTs haven’t con­clu­sively shown any ben­e­fit for any­thing be­yond be­yond bone stuff. At best, they’ve given weak ev­i­dence for haz­ard ra­tios slightly be­low one.

So you might be won­der­ing: Isn’t that quite weak? Wasn’t this post sup­posed to be a de­fense of vi­t­a­min D?

The case for sup­ple­ment­ing any­way

It’s bi­o­log­i­cally plau­si­ble that vi­t­a­min D is good

Everyone agrees that se­vere vi­t­a­min D de­fi­ciency (below ~25 nmol/​L) is bad. It leads to rick­ets, adult rick­ets, os­teo­poro­sis, mus­cle weak­ness or even—with pro­found de­fi­ciency—to seizures or car­diac ar­rhyth­mia. This makes sense, be­cause be­low ~25 nmol/​L, the kid­neys have trou­ble con­vert­ing stor­age vi­t­a­min D into ac­tive vi­t­a­min D, mean­ing you don’t ab­sorb enough cal­cium from food.

The ques­tion is if tak­ing sup­ple­ment to fur­ther raise your lev­els (say to 50 or 90 nmol/​L) is im­por­tant. We have no mech­a­nis­tic proof, but it might be true, be­cause many parts of the body use vi­t­a­min D as a lo­cal sig­nal and be­cause cells are at least some­what sen­si­tive to cir­cu­lat­ing stor­age lev­els. There’s also this weird thing where parathy­roid hor­mone con­tin­ues to de­cline as vi­t­a­min D lev­els rise above ~25 nmol/​L even while this seems to make lit­tle dif­fer­ence to how much ac­tive vi­t­a­min D the kid­neys make.

Nothing in this world comes with­out trade-offs. Surely, sup­ple­ment­ing vi­t­a­min D comes with some down­sides. But it seems very un­likely that rais­ing vi­t­a­min D lev­els to a normal” level would cause more harm than ben­e­fit. Especially be­cause…

Humans evolved to have a lot of vi­t­a­min D

According to Luxwolda et al.’s 2012 pa­per, Traditionally liv­ing pop­u­la­tions in East Africa have a mean serum 25-hydroxyvitamin D con­cen­tra­tion of 115 nmol/​L”, tra­di­tion­ally liv­ing pop­u­la­tions in East Africa have a mean serum 25-hydroxyvitamin D con­cen­tra­tion of 115 nmol/​L.

Meanwhile, Wahl et al. 2012 try to es­ti­mate mean lev­els around the world to­day:

This map looks weird be­cause of vary­ing lifestyle, diet, sup­ple­men­ta­tion, and need­ing to com­bine frag­mented stud­ies. But you get the idea. And re­mem­ber, those are just av­er­ages. So there are lots of peo­ple with lev­els far lower than that in our evo­lu­tion­ary his­tory.

Of course, just the fact that vi­t­a­min D lev­els have dropped does­n’t mean it’s im­por­tant. Parasitic worm load, wood smoke in­hala­tion, and cousin mar­riage have also dropped, but we aren’t rush­ing to re­store those to an­ces­tral lev­els.

But there’s an­other piece of ev­i­dence: After hu­mans mi­grated out of East Africa, some of them evolved pale skin. Pale skin is bad, be­cause it al­lows light to de­stroy fo­late, which is cru­cial for preg­nancy.26 Evolution does­n’t typ­i­cally do things that harm fer­til­ity, be­cause evo­lu­tion wants to in­crease re­pro­duc­tive fit­ness. The most com­mon ex­pla­na­tion is that pale skin al­lows more UV light to pen­e­trate, and thus al­lows peo­ple to syn­the­size more vi­t­a­min D. If evo­lu­tion was will­ing to pay the high price” of fo­late de­struc­tion for more vi­t­a­min D, that seems like good ev­i­dence that vi­t­a­min D is im­por­tant.

Some even see con­trasts like the Inuits ver­sus Scandinavians as a kind of nat­ural ex­per­i­ment: They lived at sim­i­lar lat­i­tudes, but Inuits ate a diet with vi­t­a­min D (fatty fish and whale blub­ber) and Scandinavians did­n’t. The re­sult is that Inuits have darker skin than Scandinavians.27

This is all spec­u­la­tive, and even if true, might be dri­ven by se­vere de­fi­ciency and rick­ets. Or per­haps pre­his­toric ben­e­fits don’t trans­late to your lifestyle. But all the peo­ple in Luxwolda’s sam­ple in East Africa had lev­els above ~60 nmol/​L. I just don’t see how you can look at this and not see it as pro­vid­ing some sug­ges­tive ev­i­dence in fa­vor of the idea that rais­ing lev­els above se­vere de­fi­ciency is un­likely to be harm­ful, and could be im­por­tant. So I think the prior is fa­vor­able.

What do you ex­pect from vi­t­a­min D?

A haz­ard ra­tio like HR = 0.96 does­n’t look very im­pres­sive. But hold on. Suppose that life ex­pectancy is 80 years and that tak­ing vi­t­a­min D every day re­duces your risk of all-cause mor­tal­ity by a fac­tor of HR. A rea­son­able ap­prox­i­ma­tion in rich coun­tries is that this would in­crease your life ex­pectancy by

80 × 0.15 × (1-HR) years = 12 × (1-HR) years,

where 0.15 is de­rived from the en­tropy of lifes­pan in rich coun­tries.28 For ex­am­ple, if all-cause mor­tal­ity had a true haz­ard ra­tio of HR = 0.96, then tak­ing vi­t­a­min D every day of your life would in­crease life ex­pectancy by around

0.48 years.

I claim that this would be a lot. Certainly, if I were about to face my des­tiny, I would pay a lot of money for an ex­tra 0.48 years. Or, you can cal­cu­late that this cor­re­sponds to an in­crease of life ex­pectancy per-vi­t­a­min-D-pill of 8.6 min­utes.29 A com­mon rule-of-thumb is that smok­ing a cig­a­rette costs around 11 min­utes of life in ex­pec­ta­tion. If you think HR = 0.96 is triv­ial, do you also think that smok­ing one cig­a­rette each day is fine?30

The cor­re­la­tional stud­ies sug­gested that vi­t­a­min D might drop your risk of all-cause mor­tal­ity by a third. It’s dis­ap­point­ing that the RCTs re­futed this. But those cor­re­la­tional stud­ies were crazy. They im­ply31 an in­crease of life ex­pectancy of around 4 years or around 6.5 cig­a­rettes per day. Could we re­ally be­lieve that you could smoke 6.5 cig­a­rettes, then take a vi­t­a­min D pill, and you’re even?

Personally, I think haz­ard ra­tios just slightly less than one are the best we can rea­son­ably hope for. But I also think that they would be an ex­cel­lent re­turn on in­vest­ment. Arguably, mod­ern hu­man life ex­pectancy comes from stack­ing lots of mod­est haz­ard ra­tios on top of each other.

What do you ex­pect from vi­t­a­min D tri­als?

Let’s play a game. Let’s hal­lu­ci­nate some num­bers for what vi­t­a­min D might do, and then sim­u­late what tri­als would show. Here are the strongest ef­fects I con­sider plau­si­ble for dif­fer­ent base­line lev­els, along with how com­mon those lev­els are in the United States.

Suppose that were real. Now, say we pick 26,000 peo­ple at ran­dom, and give half of them vi­t­a­min D for give yars. Here are the re­sults of a mil­lion sim­u­lated tri­als, as­sum­ing a base­line mor­tal­ity risk of 0.7%: 32

Overall, 9% of tri­als would find a sig­nif­i­cant ben­e­fit, 63% would find a non-sig­nif­i­cant ben­e­fit, 27% would find a non-sig­nif­i­cant harm, and 1% would find a sig­nif­i­cant harm.

If you wanted to have an 80% chance of find­ing a sig­nif­i­cant de­crease, you’d need to run a trial with some­thing like 570,000 peo­ple, al­most five times more than in all the above tri­als com­bined.33 If you don’t like my num­bers, I’ve put up a page where you can run your own sim­u­la­tions with dif­fer­ent ones.

My point is, the re­sults we see in vi­t­a­min D RCTs are what we should ex­pect to see if vi­t­a­min D had plau­si­ble ben­e­fits. That’s not proof, of course—just that if you start with re­al­is­tic ex­pec­ta­tions, the tri­als don’t pro­vide much ev­i­dence in ei­ther di­rec­tion.

The tri­als do find slightly help­ful num­bers

Recent meta-analy­ses have not con­sis­tently found a sta­tis­ti­cally sig­nif­i­cant ben­e­fit to vi­t­a­min D sup­ple­men­ta­tion. But they do sug­gest a small ben­e­fit for can­cer mor­tal­ity and all-cause mor­tal­ity, and they’re close to be­ing sta­tis­ti­cally sig­nif­i­cant. That’s some­thing.

And if you buy the ar­gu­ment that bo­lus dos­ing is bad, the re­sults get even bet­ter. Kunzia et al. did a meta-analy­sis of can­cer mor­tal­ity us­ing only tri­als with daily dos­ing, and found a haz­ard ra­tio of 0.88 (confidence in­ter­val 0.78 to 0.98). I’d keep this at ar­m’s length. The bo­lus dos­ing tri­als might have done worse by ran­dom chance, mean­ing this is a kind of p-hack­ing. But there’s a rea­son­able chance (maybe 25 – 50%) that bo­lus dos­ing re­ally is bad, in which case those tri­als would be con­vinc­ing ev­i­dence.

Vulnerability Reports Are Not Special Anymore

words.filippo.io

A re­quire­ment for stay­ing sane while work­ing in pub­lic as an open source main­tainer is re­al­iz­ing that every is­sue, PR, and piece of feed­back is a pre­sent, not an oblig­a­tion. You can ac­cept it, ig­nore it, and use it par­tially or not at all.

Except…

For years, as lead of the Go Security team at the time,1 I’ve told new team mem­bers that it does­n’t ap­ply to vul­ner­a­bil­ity re­ports. No, vul­ner­a­bil­ity re­ports are spe­cial. Security re­searchers are do­ing us a fa­vor by re­port­ing things con­fi­den­tially in­stead of do­ing full dis­clo­sure, so we owe them some­thing, which is not true of reg­u­lar is­sues opened on the is­sue tracker.2

Different pro­jects have dif­fer­ent poli­cies, but the gen­eral ex­pec­ta­tions are re­spon­sive­ness and at­tri­bu­tion. We’re sup­posed to ac­knowl­edge re­ports quickly, in­ves­ti­gate them, keep the re­porter posted, and even­tu­ally credit them with the dis­cov­ery.

Why? Well, be­cause the re­porter is pro­vid­ing us a ser­vice, not ask­ing us to pro­vide one (such as a bug fix or a fea­ture im­ple­men­ta­tion). In ex­change for re­spon­sive­ness and at­tri­bu­tion, they are of­fer­ing pre­cious in­sight and the con­fi­den­tial­ity we need to ship a fix be­fore at­tack­ers ship an ex­ploit.3

Ultimately, it all stems from our re­spon­si­bil­ity to our users. The se­cu­rity re­searchers are not spe­cial, the in­sight and con­fi­den­tial­ity are, and we need them to keep our users safe. Ignoring a se­cu­rity re­port com­mu­ni­cates you don’t care about users’ se­cu­rity, and it’s rightly a rea­son for shame.

Except…

It’s 2026 and none of the premises are true any­more.

LLMs are as good as al­most any se­cu­rity re­searcher, and any­one4 can run them. The main­tain­ers can run them. The at­tack­ers can run them.

The in­sight is not scarce and pre­cious any­more. The bot­tle­neck now is not find­ing po­ten­tial is­sues but as­sess­ing which ones are real. Unless there’s al­ready a trust re­la­tion­ship, ex­ter­nal re­searchers can’t mean­ing­fully con­tribute to that triage process, and pick­ing through an LLMs out­put or through a se­cu­rity@ in­box has ap­prox­i­mately the same sig­nal-to-noise ra­tio.

Confidentiality, em­bar­goes, and co­or­di­na­tion also don’t mat­ter nearly as much as they used to. The at­tack­ers don’t need to read the full dis­clo­sure post to learn about the vul­ner­a­bil­ity: they can ask their own LLM and, in fact, they also prob­a­bly have the same triage bot­tle­neck as the de­fend­ers do.

The years of vul­ner­a­bil­ity re­ports be­ing spe­cial might be over, as weird and un­com­fort­able5 as that feels. Triage, rapid re­me­di­a­tion, and—as ever—pre­ven­tion are the job now. And we should all fig­ure out how to run LLM analy­sis in CI, I sup­pose.

This post rapidly gen­er­ated some in­ter­est­ing dis­cus­sion, which gives me the op­por­tu­nity to add some nu­ance.

On Bluesky, Avery Pennarun points out things will change again.

I’m not sure I agree. There’s been a step change in abil­ity to find vulns, but the only sta­ble out­come (once we get there) is fewer vulns get­ting re­leased. When that hap­pens there will be a new higher bar and find­ing them will be hard again. Unclear we should op­ti­mize for the short term dy­nam­ics.

I’m not sure I agree. There’s been a step change in abil­ity to find vulns, but the only sta­ble out­come (once we get there) is fewer vulns get­ting re­leased. When that hap­pens there will be a new higher bar and find­ing them will be hard again. Unclear we should op­ti­mize for the short term dy­nam­ics.

The cur­rent dy­namic will per­sist at least for as long as the mod­els keep get­ting bet­ter. I hon­estly have no idea how the pro­fes­sion will look af­ter that, so this whole post is more of a cur­rent ob­ser­va­tion than a long-term pre­dic­tion.

On Lobsters, Frederik Braun calls out how there are still some vul­ner­a­bil­ity re­ports that are spe­cial.

Special vul­ner­a­bil­ity re­ports should be treated as spe­cial and it is on the de­fender to work on bet­ter ver­i­fi­ca­tion and pub­lished threat mod­els such that peo­ple can meet (and ver­ify) a new, higher bar for what con­sti­tutes a great re­port.

Special vul­ner­a­bil­ity re­ports should be treated as spe­cial and it is on the de­fender to work on bet­ter ver­i­fi­ca­tion and pub­lished threat mod­els such that peo­ple can meet (and ver­ify) a new, higher bar for what con­sti­tutes a great re­port.

I agree, whether of­fi­cially or un­of­fi­cially there will need to be a process for spe­cial re­ports: the ex­tremely high sever­ity ones, the ones from highly trusted sources. Maybe the next task of se­cu­rity teams is get­ting good at clas­si­fy­ing re­ports rapidly into spe­cial and not spe­cial buck­ets.

On Hacker News, William Woodruff con­firms most re­ports are real, and not spe­cial any­more.

I agree with this. One of the con­se­quences of the vulnpocalpyse” is that it’s be­come even harder to sift through the noise: I triage well over a dozen re­ports a week, many of which are real” in the sense that they re­flect a gen­uine de­fect but oth­er­wise have an un­clear im­pact on a typ­i­cal user. This has al­ways been true of the me­dian vul­ner­a­bil­ity re­port, but the vol­ume means that I now lean much more heav­ily away from co­or­di­nated dis­clo­sure. One flip­side to this is that, be­cause many of these bugs are shallow” to LLMs, it’s ac­tu­ally eas­ier than ever to mod­er­ate the worst par­tic­i­pants in your vul­ner­a­bil­ity pro­gram — if some­one sends you slop, you can just ban them and wait for the next, bet­ter or­ches­trated LLM to send you a bet­ter re­port for the same vul­ner­a­bil­ity.

I agree with this. One of the con­se­quences of the vulnpocalpyse” is that it’s be­come even harder to sift through the noise: I triage well over a dozen re­ports a week, many of which are real” in the sense that they re­flect a gen­uine de­fect but oth­er­wise have an un­clear im­pact on a typ­i­cal user. This has al­ways been true of the me­dian vul­ner­a­bil­ity re­port, but the vol­ume means that I now lean much more heav­ily away from co­or­di­nated dis­clo­sure.

One flip­side to this is that, be­cause many of these bugs are shallow” to LLMs, it’s ac­tu­ally eas­ier than ever to mod­er­ate the worst par­tic­i­pants in your vul­ner­a­bil­ity pro­gram — if some­one sends you slop, you can just ban them and wait for the next, bet­ter or­ches­trated LLM to send you a bet­ter re­port for the same vul­ner­a­bil­ity.

Imagine be­ing able to freely ban re­searchers just one year ago!

Still on Hacker News, Juho Forsén, one of the most pro­lific re­porters of Go se­cu­rity is­sues, wrote a long in­ter­est­ing com­ment that makes the ar­gu­ment that in­stead we should lean harder into trust re­la­tion­ships with in­di­vid­ual re­searchers. It’d cer­tainly be worth it with Juho, in ret­ro­spect, but it’s un­clear if it would pay off of­ten enough, in the same way that train­ing new con­trib­u­tors who might leave the pro­ject in a month or two is not al­ways worth it.

For more, sub­scribe or fol­low me on Bluesky at @filippo.abyssdomain.expert or on Mastodon at @fil­ippo@abyss­do­main.ex­pert.

The pic­ture

A few weeks ago, like every year, I ran the CENTOPASSI, a GPS-tracked mo­tor­cy­cle com­pe­ti­tion in­volv­ing care­ful plan­ning, 100 co­or­di­nates, and 1700 km of sec­ondary roads over three days and a half. It al­ways takes me to in­cred­i­ble places, like this aban­doned baux­ite mine in Puglia.

My work is made pos­si­ble by Geomys, an or­ga­ni­za­tion of pro­fes­sional Go main­tain­ers, which is funded by Ava Labs, Teleport, Datadog, Tailscale, and Sentry. Through our re­tainer con­tracts they en­sure the sus­tain­abil­ity and re­li­a­bil­ity of our open source main­te­nance work and get a di­rect line to my ex­per­tise and that of the other Geomys main­tain­ers. (Learn more in the Geomys an­nounce­ment.) Here are a few words from some of them!

Teleport — For the past five years, at­tacks and com­pro­mises have been shift­ing from tra­di­tional mal­ware and se­cu­rity breaches to iden­ti­fy­ing and com­pro­mis­ing valid user ac­counts and cre­den­tials with so­cial en­gi­neer­ing, cre­den­tial theft, or phish­ing. Teleport Identity is de­signed to elim­i­nate weak ac­cess pat­terns through ac­cess mon­i­tor­ing, min­i­mize at­tack sur­face with ac­cess re­quests, and purge un­used per­mis­sions via manda­tory ac­cess re­views.

Ava Labs — We at Ava Labs, main­tainer of AvalancheGo (the most widely used client for in­ter­act­ing with the Avalanche Network), be­lieve the sus­tain­able main­te­nance and de­vel­op­ment of open source cryp­to­graphic pro­to­cols is crit­i­cal to the broad adop­tion of blockchain tech­nol­ogy. We are proud to sup­port this nec­es­sary and im­pact­ful work through our on­go­ing spon­sor­ship of Filippo and his team.

A role I passed on to ca­pa­ble hands when I left Google, so de­spite still be­ing in­volved in the main­te­nance of the Go pro­ject, none of this is the of­fi­cial po­si­tion of the Go Security team. ↩

A role I passed on to ca­pa­ble hands when I left Google, so de­spite still be­ing in­volved in the main­te­nance of the Go pro­ject, none of this is the of­fi­cial po­si­tion of the Go Security team. ↩

This gets messy quickly at the in­ter­sec­tion of vul­ner­a­bil­ity re­port han­dling and Code of Conduct en­force­ment. If a se­cu­rity vul­ner­a­bil­ity is re­ported by some­one who is also vi­o­lat­ing the CoC, what do you do? Do you ig­nore it? Fix it silently? Realistically, there’s no squar­ing the cir­cle. It comes down to a judg­ment call based on how egre­gious the be­hav­ior is, on whether it is pri­vate or af­fect­ing the com­mu­nity, and on the re­sources avail­able to the team mem­bers ser­vic­ing se­cu­rity@. It’s an in­ter­est­ing job. ↩

This gets messy quickly at the in­ter­sec­tion of vul­ner­a­bil­ity re­port han­dling and Code of Conduct en­force­ment. If a se­cu­rity vul­ner­a­bil­ity is re­ported by some­one who is also vi­o­lat­ing the CoC, what do you do? Do you ig­nore it? Fix it silently? Realistically, there’s no squar­ing the cir­cle. It comes down to a judg­ment call based on how egre­gious the be­hav­ior is, on whether it is pri­vate or af­fect­ing the com­mu­nity, and on the re­sources avail­able to the team mem­bers ser­vic­ing se­cu­rity@. It’s an in­ter­est­ing job. ↩

There’s ac­tu­ally a lot of com­plex his­tory to dis­clo­sure prac­tices, and in a dif­fer­ent era it was gen­uinely dan­ger­ous to re­port se­cu­rity is­sues: well-in­ten­tioned re­searchers were fre­quently met with le­gal threats or pros­e­cu­tion. It took the full dis­clo­sure move­ment to make the in­dus­try in­ter­nal­ize how coun­ter­pro­duc­tive and un­rea­son­able that was. Part of the co­or­di­nated dis­clo­sure (or responsible” dis­clo­sure, a morally loaded term I dis­like) trade was a promise, im­plicit or oth­er­wise, not to go af­ter re­searchers. Thankfully, that an­gle is mostly ir­rel­e­vant to the re­al­ity of open source in 2026: no re­searcher fears pros­e­cu­tion in re­port­ing a se­cu­rity vul­ner­a­bil­ity, and no pro­ject should even im­ply pros­e­cu­tion is on the table as the al­ter­na­tive to its doc­u­mented re­port­ing pol­icy. ↩

There’s ac­tu­ally a lot of com­plex his­tory to dis­clo­sure prac­tices, and in a dif­fer­ent era it was gen­uinely dan­ger­ous to re­port se­cu­rity is­sues: well-in­ten­tioned re­searchers were fre­quently met with le­gal threats or pros­e­cu­tion. It took the full dis­clo­sure move­ment to make the in­dus­try in­ter­nal­ize how coun­ter­pro­duc­tive and un­rea­son­able that was. Part of the co­or­di­nated dis­clo­sure (or responsible” dis­clo­sure, a morally loaded term I dis­like) trade was a promise, im­plicit or oth­er­wise, not to go af­ter re­searchers. Thankfully, that an­gle is mostly ir­rel­e­vant to the re­al­ity of open source in 2026: no re­searcher fears pros­e­cu­tion in re­port­ing a se­cu­rity vul­ner­a­bil­ity, and no pro­ject should even im­ply pros­e­cu­tion is on the table as the al­ter­na­tive to its doc­u­mented re­port­ing pol­icy. ↩

Welp. Sort of. But give it 1 – 3 months and the open mod­els will catch up. ↩

Welp. Sort of. But give it 1 – 3 months and the open mod­els will catch up. ↩

Just a few days ago, at the Geomys re­treat, I was ar­gu­ing that curl’s month-long sus­pen­sion of vul­ner­a­bil­ity re­port­ing chan­nels was go­ing too far, be­cause it feels vis­cer­ally wrong to drop a se­cu­rity re­port on the floor. And yet, as I write this, I have no ar­gu­ment for ser­vic­ing vul­ner­a­bil­ity re­ports be­ing the best way to spend time to pro­tect users. Gotta change to keep up with what the job ac­tu­ally is. ↩

Just a few days ago, at the Geomys re­treat, I was ar­gu­ing that curl’s month-long sus­pen­sion of vul­ner­a­bil­ity re­port­ing chan­nels was go­ing too far, be­cause it feels vis­cer­ally wrong to drop a se­cu­rity re­port on the floor. And yet, as I write this, I have no ar­gu­ment for ser­vic­ing vul­ner­a­bil­ity re­ports be­ing the best way to spend time to pro­tect users. Gotta change to keep up with what the job ac­tu­ally is. ↩

Just a moment...

www.patreon.com

Meta Pauses Employee-Tracking Program Following Internal Data Leak

www.wired.com

Meta is paus­ing a di­vi­sive em­ployee track­ing pro­gram af­ter an in­ter­nal se­cu­rity is­sue ex­posed po­ten­tially sen­si­tive data col­lected through the ini­tia­tive to other work­ers.

We have care­fully de­signed this pro­gram with pri­vacy safe­guards and while we have no in­di­ca­tion at this time that any data was im­prop­erly ac­cessed by Meta em­ploy­ees, we’re paus­ing it while we in­ves­ti­gate,” says com­pany spokesper­son Tracy Clayton.

Meta rolled out the Model Compatibility Initiative (MCI) tool in April to US em­ploy­ees. The tool collects com­puter in­puts such as mouse move­ments, click lo­ca­tions and key­strokes, as well as screen con­tent,” ac­cord­ing to work­ers who have been pe­ti­tion­ing against it over pri­vacy, se­cu­rity, and per­sonal lib­erty con­cerns. When MCI launched, em­ploy­ees could­n’t opt out, but that changed to a lim­ited de­gree af­ter work­ers protested.

Meta ex­ec­u­tives have re­peat­edly de­fended the data-gath­er­ing pro­ject, say­ing it was nec­es­sary to train AI sys­tems to op­er­ate com­puter soft­ware the way hu­mans do and that em­ploy­ees were the best ex­am­ples for the ar­ti­fi­cial in­tel­li­gence to learn from.

On Monday, a Meta en­gi­neer is­sued an in­ter­nal se­cu­rity no­tice stat­ing that data­bases filled with in­for­ma­tion gath­ered by MCI had been ex­posed to any­one in­side the com­pany.

A for­mer em­ployee ac­tively in­volved in push­ing back against MCI de­scribes the lapse as a mess” —and one that em­ploy­ees had ex­pected would oc­cur. When work­ers raised con­cerns, lead­er­ship dou­bled down and failed to ac­knowl­edge the risks work­ers raised about the safety and pri­vacy of worker and cus­tomer data,” the per­son says. Leadership has clearly cre­ated an au­thor­i­tar­ian en­vi­ron­ment where work­ers are no longer re­spected or heard.”

But af­ter crit­i­cal com­ments poured into in­ter­nal fo­rums on Monday ex­press­ing frus­tra­tion about the se­cu­rity is­sue, Meta shocked some of its staff by paus­ing MCI al­to­gether, telling WIRED about the de­vel­op­ment sev­eral hours be­fore an­nounc­ing it to em­ploy­ees. A few work­ers told WIRED they were con­fused in the mean­time be­cause the tool was con­tin­u­ing to run on their lap­tops.

Late on Monday, Stephane Kasriel, a Meta vice pres­i­dent over­see­ing AI re­search, an­nounced the pause and told staff that the se­cu­rity is­sue had been dis­cov­ered on June 18 and ad­dressed within four hours. But the ini­tial fix did­n’t stick and ac­cess to the data had to be fur­ther locked down. The is­sue made some MCI-derived data” ac­ces­si­ble to more peo­ple than in­tended, he wrote, with­out elab­o­rat­ing.

We will only re-en­able MCI when we are con­fi­dent in the ef­fec­tive­ness of our data pro­tec­tion con­trols,” Kasriel said. He added that Meta would be shar­ing more about the fu­ture of MCI be­cause it had now gathered suf­fi­cient data to as­sess the long-term value of the tool.”

Zoë Schiffer con­tributed to this re­port.

Update 12:15am EDT, 6/23/2026: This story was up­dated with de­tails from Stephane Kasriel’s note to staff and ad­di­tional com­ments from em­ploy­ees.

To add this web app to your iOS home screen tap the share button and select "Add to the Home Screen".

10HN is also available as an iOS App

If you visit 10HN only rarely, check out the the best articles from the past week.

Visit pancik.com for more.