GrapheneOS Discussion Forum

I’ll just work from the car, I thought. But af­ter a few min­utes of star­ing at my screen on quick moun­tain switch­backs I could feel the first signs of cold, co­ag­u­lated nau­sea bub­bling up from that sweaty place in my gut. I looked to the hori­zon for re­lief, but noth­ing helped… un­til I re­mem­bered Apple’s magic dots.

Introduced in 2024, Apple’s Vehicle Motion Cues promise to tap into your de­vice’s ac­celerom­e­ter and gy­ro­scope to re­duce or, in my case, even elim­i­nate the mo­tion sick­ness felt when try­ing to use an iPhone, iPad, or MacBook in­side a mov­ing ve­hi­cle.

According to big-S Science, this type of ve­hi­cle mo­tion sick­ness is caused by the eyes star­ing at a sta­tic dis­play while the in­ner ear feels the car turn­ing, brak­ing, and ac­cel­er­at­ing. Motion Cues solve this by plac­ing dots around the pe­riph­ery of the dis­play that move in har­mony with the mo­tion of the car. When the car turns right, the dots sweep across the screen to the left; when the car brakes the dots slide for­ward.

It sounds pre­pos­ter­ous, but I’m here to tell you that it ac­tu­ally works. Once en­abled, I’ve com­fort­ably read books in the Kindle app on my phone for a few hours at a go, and even writ­ten 1,000-word re­views while my wife drove our camper van to the next des­ti­na­tion. She uses Apple’s Vehicle Motion Cues now, too, be­cause they’ve been a game changer for how we bal­ance work with life on the road.

Vehicle Motion Cues can be con­fig­ured un­der ac­ces­si­bil­ity set­tings in iOS, iPa­dOS, and ma­cOS. They can be turned on, off, or set to ap­pear au­to­mat­i­cally when ve­hi­cle mo­tion is de­tected. I pre­fer to tog­gle the dots to avoid see­ing them when I’m dri­ving the car. The black dots are fairly un­ob­tru­sive, but they can in­ter­fere with maps, text, and im­agery on long straight stretches of road that cause the dots to sit mo­tion­less (Apple should dim all the dots in those sit­u­a­tions). You can also con­fig­ure the dot size, color, and den­sity if you want, but I found the de­faults to work just fine.

I made it easy to quickly tog­gle the Motion Cues on and off by dou­ble tap­ping the back of my iPhone. To do the same, head over to Accessibility –> Touch –> Back Tap and set the Double Tap ges­ture to Vehicle Motion Cues on de­vices sup­port­ing iOS 18 and above.

I’m for­tu­nate that I re­mem­bered this ob­scure ac­ces­si­bil­ity fea­ture that I used al­most daily on a re­cent two-month road trip around Europe. Hopefully you’ll find sim­i­lar suc­cess when trav­el­ing this sum­mer.

Follow top­ics and au­thors from this story to see more like this in your per­son­al­ized home­page feed and to re­ceive email up­dates.

Thomas Ricker

Hi — this is Gergely with a free is­sue of the Pragmatic Engineer Newsletter. In every is­sue, I cover chal­lenges at Big Tech and star­tups through the lens of se­nior en­gi­neers and en­gi­neer­ing lead­ers. Subscribe to get deep­dives like this in your in­box, weekly:

Many sub­scribers ex­pense this newslet­ter to their learn­ing and de­vel­op­ment bud­get. If you have such a bud­get, here’s an email you could send to your man­ager.

For two decades, Meta had a unique, high-per­for­mance en­gi­neer­ing org; right up un­til around April of this year. For the first 20 years of the com­pa­ny’s ex­is­tence, it had a “move-fast-and-break-things” cul­ture, and in the early 2020s this shifted to a “move-fast-with-stable-infra” one. Engineers I know at the com­pany were em­pow­ered to do good work, fo­cus on im­pact, and to bal­ance busi­ness in­ter­ests with solid en­gi­neer­ing.

But in the past few weeks, all that has changed, as if the lead­er­ship has been fol­low­ing de­tailed blue­prints on how to de­mol­ish a proven, suc­cess­ful en­gi­neer­ing cul­ture in the most ruth­lessly ef­fi­cient way pos­si­ble.

For the past few weeks, I’ve been shar­ing how bad things are in­side the so­cial me­dia com­pany for en­gi­neers in one of Silicon Valley’s most pres­ti­gious work­places. In this ar­ti­cle, we walk through what’s hap­pened, and ask what’s go­ing through the minds of lead­er­ship who are re­duc­ing soft­ware en­gi­neer­ing there from the profit cen­ter that it was be­tween 2004 un­til very re­cently, to the dis­dained cost cen­ter that it has be­come in just a few weeks.

We cover:

Meta’s pre-AI en­gi­neer­ing cul­ture

Meta’s pre-AI en­gi­neer­ing cul­ture

Investing in AI and press­ing en­gi­neers to al­ways use it

Investing in AI and press­ing en­gi­neers to al­ways use it

Core en­gi­neer­ing folks feel treated like trash

Core en­gi­neer­ing folks feel treated like trash

Most em­bar­rass­ing-ever out­age

Most em­bar­rass­ing-ever out­age

Internal mess

Internal mess

Self-inflicted wounds

Self-inflicted wounds

Is “AI psy­chosis” just a Meta is­sue?

Is “AI psy­chosis” just a Meta is­sue?

I’d split Meta’s en­gi­neer­ing cul­ture into two eras: “move fast and break things”, and then “move fast with sta­ble in­fra.”

In the 2010s, Facebook’s un­con­ven­tional en­gi­neer­ing cul­ture had grown some­what leg­endary in the tech in­dus­try, as the com­pany went against con­ven­tional best prac­tices and suc­ceeded mas­sively.

In 2012, when Facebook hit the bil­lion-users land­mark, the com­pany pro­duced a small phys­i­cal book about its cul­ture which was placed on em­ploy­ees’ desks. Presented with retro pro­pa­ganda de­sign, it was dubbed the “little red book”, co-opt­ing the name of a fa­mous vol­ume of the thoughts of Chairman Mao, (1964).

At around 70 pages long, Facebook’s ver­sion cod­i­fied its en­gi­neer­ing cul­ture: speed, fear­less­ness, tak­ing own­er­ship, and think­ing out­side of the box.

Back then, mantras in Facebook’s lit­tle red book were also in print across cam­pus, and in­cluded:

Move Fast and Break Things

Move Fast and Break Things

Done is Better Than Perfect

Done is Better Than Perfect

Fail Harder

Fail Harder

What Would You Do If You Weren’t Afraid?

What Would You Do If You Weren’t Afraid?

Every Day Feels Like a Week

Every Day Feels Like a Week

The Wright Brothers Did Not Have Pilot Licenses

The Wright Brothers Did Not Have Pilot Licenses

The Foolish Wait

The Foolish Wait

Fortune Favors the Bold

Fortune Favors the Bold

There was gen­uine fo­cus on build­ing good prod­ucts. Also from the book:

In 2022, I did what is one of the longest deep­dives we’ve pub­lished on the topic of Meta’s en­gi­neer­ing cul­ture. By then, things had evolved, and much of any for­mer reck­less­ness was gone, re­placed by the prin­ci­ple of mov­ing fast, but with sta­ble in­fra. Here’s how I de­scribed Meta’s en­gi­neer­ing cul­ture then:

“The cul­ture is in­cred­i­bly en­gi­neer­ing-cen­tric: much more than most of Big Tech. This might come from Mark Zuckerberg be­ing an en­gi­neer him­self, or be­cause much of the in­no­va­tion in the early days of Facebook came from en­gi­neers.Fo­cus on in­di­vid­ual im­pact. Impact has been the bread and but­ter of the fo­cus at Facebook. This is very true since the early days, and the fo­cus on gen­er­at­ing im­pact re­mains.One de­tail in com­mon with most Big Tech firms is that both the en­gi­neer­ing cul­ture and gen­eral cul­ture fo­cus so much on in­di­vid­ual im­pact. This re­sults in some peo­ple fo­cus­ing on short-term, mea­sur­able wins and as­sum­ing that team­work and split wins be­tween groups might be less re­warded.The lack of rigid processes. Facebook seems to have the least amount of processes or stan­dard­iza­tion across all of Big Tech. Don’t even try to com­pare it to Amazon’s en­gi­neer­ing cul­ture and the count­less for­mal processes there. But even com­pared to com­pa­nies like Google, Microsoft or Uber, Facebook’s processes are much looser. Most of this comes from the en­gi­neer­ing-cen­tric na­ture of the com­pany and en­gi­neers dis­lik­ing processes.Sur­pris­ingly lit­tle em­pha­sis on test­ing, doc­u­men­ta­tion or code com­ments. You’ll find shock­ingly lit­tle au­to­mated test­ing and doc­u­men­ta­tion at Facebook, com­pared to the rest of Big Tech. Inline code com­ments are also very rare.A founder-en­gi­neer dri­ven com­pany. Facebook is one of the few Big Tech firms whose founder is an en­gi­neer, and still is the CEO. Netflix is the other one where founder and co-CEO Reed Hastings was also a soft­ware en­gi­neer be­fore start­ing the com­pany. Amazon was the other ex­am­ple of this un­til re­cently, but it’s not the case at Google or Apple. There are good ex­am­ples of smaller com­pa­nies like Cloudflare, but they’re all younger than Facebook.Bootcamp. A unique on­board­ing process, un­like what any other Big Tech firms of­fer. We cover this more in the Bootcamp & on­board­ing sec­tion.”

“The cul­ture is in­cred­i­bly en­gi­neer­ing-cen­tric: much more than most of Big Tech. This might come from Mark Zuckerberg be­ing an en­gi­neer him­self, or be­cause much of the in­no­va­tion in the early days of Facebook came from en­gi­neers.

Focus on in­di­vid­ual im­pact. Impact has been the bread and but­ter of the fo­cus at Facebook. This is very true since the early days, and the fo­cus on gen­er­at­ing im­pact re­mains.

One de­tail in com­mon with most Big Tech firms is that both the en­gi­neer­ing cul­ture and gen­eral cul­ture fo­cus so much on in­di­vid­ual im­pact. This re­sults in some peo­ple fo­cus­ing on short-term, mea­sur­able wins and as­sum­ing that team­work and split wins be­tween groups might be less re­warded.

The lack of rigid processes. Facebook seems to have the least amount of processes or stan­dard­iza­tion across all of Big Tech. Don’t even try to com­pare it to Amazon’s en­gi­neer­ing cul­ture and the count­less for­mal processes there. But even com­pared to com­pa­nies like Google, Microsoft or Uber, Facebook’s processes are much looser. Most of this comes from the en­gi­neer­ing-cen­tric na­ture of the com­pany and en­gi­neers dis­lik­ing processes.

Surprisingly lit­tle em­pha­sis on test­ing, doc­u­men­ta­tion or code com­ments. You’ll find shock­ingly lit­tle au­to­mated test­ing and doc­u­men­ta­tion at Facebook, com­pared to the rest of Big Tech. Inline code com­ments are also very rare.

A founder-en­gi­neer dri­ven com­pany. Facebook is one of the few Big Tech firms whose founder is an en­gi­neer, and still is the CEO. Netflix is the other one where founder and co-CEO Reed Hastings was also a soft­ware en­gi­neer be­fore start­ing the com­pany. Amazon was the other ex­am­ple of this un­til re­cently, but it’s not the case at Google or Apple. There are good ex­am­ples of smaller com­pa­nies like Cloudflare, but they’re all younger than Facebook.

Bootcamp. A unique on­board­ing process, un­like what any other Big Tech firms of­fer. We cover this more in the Bootcamp & on­board­ing sec­tion.”

Also, Facebook, as a prod­uct, has one of the most so­phis­ti­cated auto roll­out sys­tems in the in­dus­try. Instagram has a bat­tle-tested in­fra­struc­ture where it was al­most triv­ial to launch a new so­cial net­work (Threads) with 100 mil­lion users served in its first week.

Engineers whom I knew in­side the com­pany are ca­pa­ble, mo­ti­vated, and prod­uct-minded, and their work was ap­pre­ci­ated. CEO, Mark Zuckerberg, was in­flu­en­tial: he per­son­ally coded the first ver­sion of Facebook, had stayed close to en­gi­neer­ing, and val­ued soft­ware en­gi­neers very much. Engineers there felt they were work­ing in­side a profit cen­ter.

Meta has been the only com­pany among the big five of Apple, Microsoft, Amazon, Google, and it­self not to own a hard­ware plat­form or op­er­at­ing sys­tem. Apple has the iPhone, iPad and Macs, Google has Android, ChromeOS and Pixel phones, Microsoft has Windows, and Amazon has the Kindle.

Stepping back, it looks as though the Mark Zuckerberg of to­day has re­solved not to miss a plat­form op­por­tu­nity, af­ter the com­pany failed to build its own mo­bile OS or mo­bile phone dur­ing the 2010s.

This is one rea­son for in­vest­ing so much in vir­tual re­al­ity (VR) with Oculus, and in aug­mented re­al­ity with the Meta Glasses. Facebook changed its name to Meta in 2021, back when it looked like VR — and the meta­verse — could be mas­sive. Billions was spent on en­sur­ing Meta would be the mar­ket leader in this space. But once again, VR did­n’t go main­stream; since the end of the pan­demic, pop­u­lar in­ter­est in the seg­ment has died down con­sid­er­ably.

When it be­came clear that AI would be­come a mega-trend in 2022, Zuckerberg did­n’t miss it: he as­sem­bled the in­ter­nal FAIR group (Fundamental AI Research team) as well as a GenAI prod­uct or­ga­ni­za­tion and re­leased a se­ries of open-weight AI mod­els:

Llama 1: re­leased in Feb 2023, three months af­ter ChatGPT, built by FAIR

Llama 1: re­leased in Feb 2023, three months af­ter ChatGPT, built by FAIR

Llama 2: in June 2023, built by the GenAI prod­uct or­ga­ni­za­tion (as well as all sub­se­quent Llama mod­els)

Llama 2: in June 2023, built by the GenAI prod­uct or­ga­ni­za­tion (as well as all sub­se­quent Llama mod­els)

Llama 3: in April 2024. This model was Meta’s most com­pet­i­tive LLM of all, and gained mo­men­tum in adop­tion across the in­dus­try

Llama 3: in April 2024. This model was Meta’s most com­pet­i­tive LLM of all, and gained mo­men­tum in adop­tion across the in­dus­try

Llama 4: in April 2025. This model was deeply dis­ap­point­ing

Llama 4: in April 2025. This model was deeply dis­ap­point­ing

In June that year, Meta ac­quired a 49% stake in Scale AI to re­boot its AI ef­forts for a whop­ping $14.8B, and brought in Scale AI’s CEO, Alexandr Wang to take over Meta’s AI strat­egy. The ac­qui­si­tion of Chinese startup Manus AI for $2B is cur­rently in ques­tion af­ter China blocked the deal from be­ing com­pleted.

Based on the in­vest­ment made into Scale AI and Wang, it’s pretty clear that Meta — and Zuckerberg — is de­ter­mined to build a state-of-the-art LLM that can be com­pet­i­tive with the lat­est ver­sions of Claude and ChatGPT. But Meta has to start pretty much from scratch, and it’s up to Alexandr Wang to de­liver.

Scale AI brings in a very spe­cific kind of ex­per­tise to Meta, as one of the best in the in­dus­try in:

Training data and la­bel­ing: Scale started, and is still best known, as a provider of high-qual­ity la­beled datasets for ma­chine learn­ing and AI train­ing, in­clud­ing code, text, im­age, video, etc.

Training data and la­bel­ing: Scale started, and is still best known, as a provider of high-qual­ity la­beled datasets for ma­chine learn­ing and AI train­ing, in­clud­ing code, text, im­age, video, etc.

RLHF and fine-tun­ing: A RLHF (reinforcement learn­ing from hu­man feed­back) flow which Scale runs, where peo­ple give feed­back for foun­da­tion mod­els, as a “human in the loop” data en­gine that many lead­ing AI labs use to cre­ate bet­ter LLMs.

RLHF and fine-tun­ing: A RLHF (reinforcement learn­ing from hu­man feed­back) flow which Scale runs, where peo­ple give feed­back for foun­da­tion mod­els, as a “human in the loop” data en­gine that many lead­ing AI labs use to cre­ate bet­ter LLMs.

Wang seems to have a very broad reign to do what he has been an ex­pert in: cre­at­ing train­ing data, do­ing data la­bel­ing and RLHF. This is be­ing pulled off with the la­bor of Meta’s en­gi­neer­ing work­force, and by sur­veilling it.

Problem #1: Tracking key­strokes and mouse clicks, with no op­tion to opt out. In late April, Meta told en­gi­neers they were be­ing en­rolled into a sys­tem that tracks every key­stroke and click, to pro­duce train­ing data for Meta’s new AI. There’s no way to opt out.

Needless to say, this is in­va­sive and raises pri­vacy ques­tions: If you log into your per­sonal bank ac­count, does the tool track you? What about when you’re writ­ing a per­sonal email, or re­spond­ing to a per­sonal call? Meta held no con­sul­ta­tion and there are no workarounds; just a top-down de­ci­sion be­ing pushed through.

This month, Reuters re­ported that peo­ple’s con­cerns there are fi­nally be­ing heard:

“Meta is di­al­ing back el­e­ments of its plan to col­lect em­ployee mouse move­ments, key­strokes and other ac­tions for use as AI ​training data, it said in an in­ter­nal memo on Tuesday, fol­low­ing weeks ​of an­gry push­back from staffers.New con­trols will al­low em­ploy­ees to pause ⁠the data col­lec­tion for up to 30 min­utes at a time and ​request ex­emp­tions from the ini­tia­tive, ac­cord­ing to the memo, au­thored by Stephane Kasriel, ​a vice pres­i­dent in Meta’s AI model-build­ing Superintelligence Labs unit.”

“Meta is di­al­ing back el­e­ments of its plan to col­lect em­ployee mouse move­ments, key­strokes and other ac­tions for use as AI ​training data, it said in an in­ter­nal memo on Tuesday, fol­low­ing weeks ​of an­gry push­back from staffers.

New con­trols will al­low em­ploy­ees to pause ⁠the data col­lec­tion for up to 30 min­utes at a time and ​request ex­emp­tions from the ini­tia­tive, ac­cord­ing to the memo, au­thored by Stephane Kasriel, ​a vice pres­i­dent in Meta’s AI model-build­ing Superintelligence Labs unit.”

From talk­ing with cur­rent Meta en­gi­neers, I un­der­stand the log­ging sys­tem has not been rolled out in the UK due to data pro­tec­tion reg­u­la­tion.

Problem #2: 30 – 50% of en­gi­neers on core teams have been force­fully re­as­signed to data la­bel­ing and RLHF, up­set­ting folks even more. Also start­ing in late April, prod­uct en­gi­neer­ing teams re­ceived a man­date from above, whereby 30 – 50% of en­gi­neers were to leave the team and join the ADO org (Agent Data Optimisation).

“Forceful” re­as­sign­ment is very rel­e­vant here be­cause of Meta’s tra­di­tional en­gi­neer­ing cul­ture. Between its found­ing in 2004 and un­til last year, Meta gave en­gi­neers au­ton­omy to choose where they work and what they work on. This was struc­tural to how the com­pany worked:

Engineers were not hired for a spe­cific team (save for at the Staff+, lev­els, in some cases). They were hired to the com­pany

Engineers were not hired for a spe­cific team (save for at the Staff+, lev­els, in some cases). They were hired to the com­pany

During a 6-week boot­camp, new hires got fa­mil­iar with Meta’s en­gi­neer­ing cul­ture and chose a team

During a 6-week boot­camp, new hires got fa­mil­iar with Meta’s en­gi­neer­ing cul­ture and chose a team

Team match­ing meant talk­ing with mul­ti­ple teams who had head­count, do­ing small work with them, and find­ing a match

Team match­ing meant talk­ing with mul­ti­ple teams who had head­count, do­ing small work with them, and find­ing a match

Internal trans­fers were easy, and of­ten ini­ti­ated by en­gi­neers

Internal trans­fers were easy, and of­ten ini­ti­ated by en­gi­neers

Team se­lec­tion via boot­camp started to die down in around 2024, but any Meta en­gi­neer with at least two years’ tenure knows that pre­vi­ously they chose what to work on, and of course, could pick the most im­pact­ful thing to work on. And then, out of the blue, they’re as­signed to a di­vi­sion where the im­pact is not clear, the work is me­nial, and do­ing it too long will surely hurt their ca­reer prospects.

“Data la­bel­ing” is more in­volved work, even though a bit repet­i­tive. There are la­bel­ing tasks, where you look at a web­site and de­cide if it looks good or not. But then there are more in­volved AI train­ing tasks, which looks like this:

Come up with a task that the AI should do

Come up with a task that the AI should do

Then write the tests that con­firm the re­sult

Then write the tests that con­firm the re­sult

Package all of this up into a Docker con­tainer, us­ing Harbor frame­work

Package all of this up into a Docker con­tainer, us­ing Harbor frame­work

Then read the code that the AI writes - of­ten do­ing this based on feed­back from sev­eral mod­els, and give it feed­back

Then read the code that the AI writes - of­ten do­ing this based on feed­back from sev­eral mod­els, and give it feed­back

Key find­ings

74%

con­sumers say the in­ter­net feels less hu­man than 10 years ago

40 min

av­er­age time be­fore con­sumers ex­pe­ri­ence “bot fa­tigue”

61%

con­sumers can’t name a brand us­ing AI well in its mes­sag­ing

16.6

av­er­age weekly hours en­ter­prise teams spend im­prov­ing AI vis­i­bil­ity

Brands have been chas­ing AI vis­i­bil­ity for two years. You’ve spent time and bud­get on it, yet your au­di­ence can’t name a sin­gle com­pany they think is do­ing it well. The brands build­ing for the next phase treat their web­site as the place where AI gets clean data and hu­mans get some­thing worth their time.

A less hu­man web costs you read­ers.

Your au­di­ence can sense when a ma­chine is talk­ing to them. Most are check­ing out be­fore they’ve de­cided whether they care. Bot fa­tigue sets in when the in­ter­net stops feel­ing hon­est. The small mo­ments that used to make the web worth vis­it­ing are dis­ap­pear­ing.

The AI web

7/10

con­sumers say the in­ter­net feels less hu­man than it did 10 years ago

Feels less hu­man

Still hu­man

40 min

the av­er­age time to “bot fa­tigue,” when in­ter­ac­tions start to feel syn­thetic

Can your con­tent in­fra­struc­ture mea­sure this shift and re­spond to it? We’ll cover how en­ter­prise teams re­struc­ture con­tent for AI dis­cov­ery with­out los­ing what feels hu­man in our up­com­ing we­bi­nar.

What is AI brand vis­i­bil­ity?

AI brand vis­i­bil­ity is how of­ten a brand ap­pears in an­swers gen­er­ated by AI en­gines like ChatGPT, Perplexity, Claude, and Gemini. It’s a dif­fer­ent prob­lem from search en­gine vis­i­bil­ity, which mea­sures rank­ings on re­sult pages. A brand can rank at the top of Google and not ap­pear in­side ChatGPT at all. As of 2026, no sin­gle dash­board tracks AI brand vis­i­bil­ity across every en­gine, and the cat­e­gory has no es­tab­lished leader.

Nobody has won AI brand vis­i­bil­ity yet.

Every an­swer in our con­sumer sur­vey pointed the same way: Nobody has done it well yet. Brands have spent the past year fund­ing AI strat­egy, but con­sumers can’t point to a sin­gle com­pany they think is get­ting it right.

The cat­e­gory has no in­cum­bent, and no tem­plate to copy. The brand that builds that recog­ni­tion first gets to de­fine the stan­dard.

Brand vis­i­bil­ity

61%

of con­sumers can’t name a brand that uses AI well in its mes­sag­ing

16%

say no brand is us­ing AI well at all

60%

say AI in a brand’s mes­sag­ing is a turnoff, not a fea­ture

“No cus­tomer or user wakes up and says, ‘I hope I get to talk to a chat bot or an AI agent to­day.’ Human-centered de­sign is truer to­day with ar­ti­fi­cial in­tel­li­gence. Ironically, the an­swer is us­ing AI to be more hu­man.”— Brian Solis, Head of Global Innovation, ServiceNow

“No cus­tomer or user wakes up and says, ‘I hope I get to talk to a chat bot or an AI agent to­day.’ Human-centered de­sign is truer to­day with ar­ti­fi­cial in­tel­li­gence. Ironically, the an­swer is us­ing AI to be more hu­man.”

Build for both au­di­ences at once.

AI needs to find the con­tent and a per­son needs a rea­son to stay once they ar­rive. The sec­ond part is harder, and most en­ter­prises are still guess­ing at it. The brands worth watch­ing are bet­ting that “staying” comes from giv­ing peo­ple some­thing to do: in­ter­ac­tive con­tent, dy­namic ex­pe­ri­ences, the small mo­ments a flat AI sum­mary can’t de­liver.

The web­site is the only place where both jobs run to­gether. AI gets struc­tured con­tent it can cite, and the reader gets some­thing worth their time. That’s the foun­da­tion you get on WordPress VIP.

The guide for build­ing that dual-pur­pose site is in Future-Proof Your Brand for the AI-Native Web, a frame­work for prepar­ing your web plat­form.

How en­ter­prises are mea­sur­ing AI brand vis­i­bil­ity

The cat­e­gory is barely two years old and the toolset is still set­tling. No sin­gle dash­board tracks every AI sur­face. No shared de­f­i­n­i­tion of “good” ex­ists yet. Pricing across the cat­e­gory swings from free to six fig­ures de­pend­ing on cov­er­age and cus­tomiza­tion. What en­ter­prises are us­ing right now sorts into five cat­e­gories, with real tools in­side each.

This is a snap­shot since the spe­cific prod­ucts will shift in the next 12 months. The cat­e­gories will out­last them, which is why the sec­tion is or­ga­nized around what the tools do.

This is the newest cat­e­gory, built specif­i­cally to track how of­ten a brand ap­pears in ChatGPT, Perplexity, Claude, and Gemini an­swers. These tools sim­u­late queries at scale and sur­face ci­ta­tion fre­quency and sen­ti­ment over time.

Tools in this cat­e­gory: Profound, BrightEdge, brand­vis­i­bil­ity.ai, Tryevergreen, and a hand­ful of smaller com­peti­tors that emerged in late 2025.

Best for: Teams that need to con­nect AI vis­i­bil­ity to busi­ness out­comes. AI ci­ta­tions are top-of-fun­nel. This cat­e­gory mea­sures what those ci­ta­tions turn into. The brands that fig­ure out which AI-referred vis­i­tors con­vert can de­fend their AI strat­egy spend.

Watch for: Pricing mod­els are still set­tling. Most plat­forms re­quire four to six weeks of data col­lec­tion be­fore bench­marks are mean­ing­ful. Sample-based query sim­u­la­tion has gaps, and tools that promise “complete cov­er­age” of every AI an­swer are over­stat­ing their method­ol­ogy.

These are the es­tab­lished SEO plat­forms that ex­tended into AI track­ing start­ing in 2024. These tools layer AI ci­ta­tion data on top of tra­di­tional search met­rics, which makes them use­ful for teams al­ready run­ning SEO work­flows.

Tools in this cat­e­gory: Similarweb (AI Intelligence), Semrush (AI Toolkit), Ahrefs (Brand Radar).

Best for: SEO teams that want AI vis­i­bil­ity data with­out a new ven­dor re­la­tion­ship. The in­te­gra­tion with ex­ist­ing search re­port­ing is the main value; it lets a team see or­ganic and AI traf­fic in the same view.

Watch for: AI cov­er­age in this cat­e­gory is gen­er­ally nar­rower than in ded­i­cated AI ci­ta­tion plat­forms. The tools were built for search and are still catch­ing up on the AI side. AI num­bers here have to be treated as di­rec­tional.

In this cat­e­gory: the an­a­lyt­ics plat­forms that de­tect and seg­ment traf­fic ar­riv­ing from AI en­gines. These are the ci­ta­tion mon­i­tor­ing tools that tell a brand it’s be­ing men­tioned. This cat­e­gory tells a brand what hap­pens af­ter.

Tools in this cat­e­gory: Parse.ly (part of the WordPress VIP prod­uct fam­ily), Plausible, Fathom Analytics, and most en­ter­prise an­a­lyt­ics plat­forms (Google Analytics 4) with cus­tom seg­men­ta­tion.

Best for: Teams that need to con­nect AI vis­i­bil­ity to busi­ness out­comes. AI ci­ta­tions are top-of-fun­nel. This cat­e­gory mea­sures what those ci­ta­tions turn into. The brands that fig­ure out which AI-referred vis­i­tors con­vert can de­fend their AI strat­egy spend.

Watch for: AI re­fer­rer de­tec­tion still varies by plat­form. Some AI en­gines pass clean re­fer­rer head­ers, oth­ers rely on UTM tag­ging. Coordination be­tween con­tent and an­a­lyt­ics teams is usu­ally re­quired to get clean data.

Broader brand mon­i­tor­ing plat­forms that added AI sur­face track­ing to ex­ist­ing so­cial lis­ten­ing and PR mon­i­tor­ing ca­pa­bil­i­ties. These cover AI en­gines as one in­put along­side so­cial and tra­di­tional me­dia men­tions.

Tools in this cat­e­gory: Brandwatch, Talkwalker, Meltwater.

Best for: Communications and PR teams that al­ready use these plat­forms for cri­sis mon­i­tor­ing and share-of-voice track­ing. The AI cov­er­age is an ex­ten­sion of an ex­ist­ing work­flow.

Watch for: AI cov­er­age in this cat­e­gory tends to be lighter than in ded­i­cated AI ci­ta­tion tools. Useful for a 30,000-foot view, less use­ful for gran­u­lar ci­ta­tion analy­sis.

This is what en­ter­prises with en­gi­neer­ing ca­pac­ity are build­ing them­selves. These so­lu­tions use LLM APIs to query AI en­gines on a sched­ule and sur­face re­sults in a dash­board the team con­trols. Pew Research Center’s work with WordPress VIP, cov­ered in Chapter 2, is one ex­am­ple of this ap­proach.

Best for: Enterprises with en­gi­neer­ing re­sources who want to de­fine their own queries and con­trol their own data. Ideal when the brand’s AI vis­i­bil­ity strat­egy de­pends on niche or in­dus­try-spe­cific queries that off-the-shelf tools don’t cover well.

Watch for: Maintenance bur­den. LLM API ac­cess is now sta­ble, though pric­ing and rate lim­its change fre­quently. Custom dash­boards re­quire on­go­ing en­gi­neer­ing at­ten­tion to keep cur­rent.

AI brand vis­i­bil­ity tools at a glance

How to choose

Match the tool cat­e­gory to the ques­tion the team needs to an­swer:

“Are we be­ing cited?” Use an AI ci­ta­tion mon­i­tor­ing plat­form.

“Are we be­ing cited rel­a­tive to our search per­for­mance?” Use search an­a­lyt­ics with AI over­lays.

“What hap­pens af­ter we’re cited?” Use web an­a­lyt­ics with AI re­fer­ral track­ing.

“How does AI fit into our broader brand sen­ti­ment?” Use a brand in­tel­li­gence plat­form.

“We need to track some­thing none of the above can an­swer.” Build a cus­tom so­lu­tion.

Most en­ter­prises use two cat­e­gories to­gether. The most com­mon com­bi­na­tion is a tool from the AI ci­ta­tion mon­i­tor­ing cat­e­gory to know whether the brand shows up, and a tool from the web an­a­lyt­ics cat­e­gory to know what that vis­i­bil­ity is worth. The brands that fig­ured this out first are the ones whose 2027 AI vis­i­bil­ity bud­gets won’t be re-lit­i­gated in bud­get meet­ings.

Continue read­ing

Chapter 2

Brands chase AI vis­i­bil­ity. Consumers chase the source.

Chapter 3

Consumers are wary of gate­keep­ing. More than mar­keters are.

Chapter 4

The web­site is still the de­fault trust layer.

Chapter 5

The next web­site does­n’t look like a web­site.

FAQs about AI brand vis­i­bil­ity

Bot fa­tigue is the point at which on­line in­ter­ac­tions start to feel syn­thetic. WordPress VIP’s 2026 sur­vey of 1,200 U.S. con­sumers found the av­er­age per­son hits bot fa­tigue in about 40 min­utes. The broader pat­tern: 74% of con­sumers say the in­ter­net feels less hu­man than it did 10 years ago, which is the con­sumer-mood shift dri­ving most of what brands are now try­ing to solve in their AI strat­egy.

Not yet. The cat­e­gory is too new and the mea­sure­ment tools are too im­ma­ture. Platforms cite dif­fer­ent sources for dif­fer­ent queries, the ci­ta­tions change as mod­els up­date, and the met­rics en­ter­prise teams use to track AI vis­i­bil­ity aren’t stan­dard­ized across ven­dors. What’s clear is that no brand has built a durable AI pres­ence. The brand that de­fines what “AI brand vis­i­bil­ity done well” looks like will be the one that fig­ures out the mea­sure­ment layer be­fore the rest of the mar­ket does.

The web­site has two jobs now and they have to run on the same foun­da­tion. AI en­gines need struc­tured con­tent they can find and cite ac­cu­rately. Human vis­i­tors need a rea­son to stay once they click through from an AI sum­mary. The brands solv­ing for both are treat­ing the web­site as the place where AI ex­tracts data and a per­son has an ex­pe­ri­ence worth their time. This is the cen­tral ar­gu­ment of WordPress VIP’s 2026 State of the Open Web re­port.

Dear Republic,

Maybe you liked Calvin and Hobbes as a kid but you prob­a­bly have no idea of the scrupu­lous moral in­tegrity that went into it, as Matthew Morgan demon­strates in this deeply-re­searched piece.

-ROL

CALVIN AND HOBBES AND THE PRICE OF INTEGRITY

I.

1978, Kenyon College, sopho­more year. Bill Watterson is ly­ing on his dorm room bed, star­ing up at the ceil­ing. He has­n’t yet in­vented six-year-old Calvin and his tiger, Hobbes — though his stud­ies have made him fa­mil­iar with their philo­soph­i­cal name­sakes — be­cause the strip that will make Watterson’s name is al­most a decade away. Right now, he’s think­ing that his dorm room needs an am­a­teur ren­di­tion of Michelangelo’s “Creation of Adam”.

There’s a num­ber of prob­lems up front. The first is that (as Watterson will tell you him­self) he’s not a tal­ented painter. Still, what the work will lack in “colour sense and tech­ni­cal flour­ish” it’ll make up for with com­edy — specif­i­cally “the in­con­gruity of hav­ing a High Renaissance mas­ter­piece in a col­lege dorm that had the un­mis­take­able odour of old beer cans and older laun­dry”. Besides, Michelangelo was­n’t Michelangelo un­til he’d painted and kept paint­ing and be­came Michelangelo the painter. Watterson de­cides to go ahead and start paint­ing.

The next prob­lem is struc­tural: how to reach the ceil­ing? He can stand on the bed, but that’ll mean hours with his head cocked all the way back, a young man de­vel­op­ing an old man’s spine. He needs a way to paint the ceil­ing with­out per­ma­nently dis­fig­ur­ing his pos­ture. His friends help him with a so­lu­tion: they stand two chairs on Watterson’s bed, then lie a table across the chairs. By climb­ing up this tower and ly­ing on the table, he comes two feet away from the ceil­ing. Watterson gets to work.

He’s sunk hours into hours of weeks upon weeks on his back when a third prob­lem oc­curs to him. He should have thought of this and acted on it be­fore the first brush stroke. He needs per­mis­sion to paint his dorm room ceil­ing. But Watterson once ad­mit­ted, “I never spent as much time or work on any au­tho­rised art pro­ject or any poli-sci pa­per as I spent on this one act of van­dal­ism.” He is­n’t giv­ing up on it now.

The hous­ing di­rec­tor is un­der­stand­ably sus­pi­cious of this kid want­ing to paint some elab­o­rate pic­ture on his ceil­ing with only a few weeks left of the aca­d­e­mic year. He re­alises that the idea is be­ing pro­posed retroac­tively. Maybe that’s why he plays along and grants per­mis­sion for some­thing that’s ob­vi­ously al­ready un­der­way. Watterson is al­lowed to com­plete the paint­ing on the con­di­tion that he re­turns the ceil­ing to nor­mal be­fore he leaves in sum­mer. Watterson goes back to his room, climbs up the makeshift scaf­fold­ing, and gets back to work.

A few weeks later, the pro­ject is fin­ished. Watterson prob­a­bly takes a mo­ment to stand in the mid­dle of the room and look up, con­tem­plat­ing the months of work, the tins of paint he went through, the things he learned about tech­nique, about the joy of a job done for its own sake, about him­self. Then he opens a tin of white­wash, climbs up the bed-chairs-table one last time, and paints over his work. He leaves the ceil­ing white, empty, fresh.

II.

In the years af­ter Kenyon, Watterson has a re­cur­ring dream about his old col­lege where he does­n’t know what class he’s tak­ing or where he’s meant to be. He roams the grounds, grow­ing more flus­tered with each con­fused step. Right be­fore he wakes, he thinks, “How many more years un­til I grad­u­ate…? Wait, did­n’t I grad­u­ate al­ready? How old am I?”

It’s 1995 and Watterson is thirty-seven. He’s sit­ting at the desk where he’s worked for the last ten years, draw­ing the ad­ven­tures of Calvin and his maybe-real or maybe-stuffed-toy tiger, Hobbes. Calvin and Hobbes runs in over 2,400 news­pa­pers across the world and, by a more mean­ing­ful met­ric, re-en­chants life for mil­lions of read­ers. It’s pop-cul­ture that tran­scends the ’pop” part of its na­ture; it feels like a pri­vate piece of each read­er’s soul. For a lot of grown-ups, Calvin and Hobbes is a bridge be­tween who we were as wide-eyed, won­der­ing chil­dren and who we are now.

A few years af­ter he found suc­cess with Calvin and Hobbes, Watterson told a grad­u­at­ing class at Kenyon that there’s noth­ing like the joy of work done for your own cre­ative sat­is­fac­tion, rather than for fame or a few bucks. Watterson is con­vinced that an artist should do what he does for love, even if it fails, even if it costs him, even though it and every­thing else will even­tu­ally end. In fact, his ed­i­tor has just okayed a strip in which Hobbes asks Calvin, “If good things lasted for­ever, would we ap­pre­ci­ate how pre­cious they are?”

Which is what brings Watterson to his desk to­day.

There’s a few pa­pers scat­tered be­neath and around the sin­gle page that he’s fo­cused on. He has an un­usual task this morn­ing: he’s not draw­ing, but writ­ing, deal­ing ex­clu­sively in words. Maybe he starts right away, know­ing ex­actly what he wants to tell the peo­ple who’ll read this let­ter. I imag­ine him tak­ing a mo­ment to con­sider the cur­rent that’s swept him along the last ten years to what he’s prepar­ing to do to­day. He takes a sip of what’s left in his cof­fee mug (damn, it’s gone cold), then starts to write.

I don’t know how Watterson drafted this let­ter, but in my telling of the story he’s scrib­bling it down on pa­per with a pen­cil, the way he does di­a­logue for his strip. He’ll type it up later and post it out to­mor­row af­ter­noon, or maybe he’ll type it up on a com­puter and elec­tron­i­cally mail it. For now, he’s writ­ing a let­ter by hand, and it’ll be sent to all the ed­i­tors of the var­i­ous news­pa­pers that run Calvin and Hobbes. The let­ter goes like this:

“I will be stop­ping Calvin and Hobbes at the end of the year. This was not a re­cent or an easy de­ci­sion, and I leave with some sad­ness. My in­ter­ests have shifted, how­ever, and I be­lieve I’ve done what I can do within the con­straints of daily dead­lines and small pan­els. I am ea­ger to work at a more thought­ful pace, with fewer artis­tic com­pro­mises. I have not yet de­cided on fu­ture pro­jects, but my re­la­tion­ship with Universal Press Syndicate will con­tinue.

That so many news­pa­pers would carry Calvin and Hobbes is an honor I’ll long be proud of, and I’ve greatly ap­pre­ci­ated your sup­port and in­dul­gence over the last decade. Drawing this comic strip has been a priv­i­lege and a plea­sure, and I thank you for giv­ing me the op­por­tu­nity.

Sincerely,

Bill Watterson”

III.

The let­ter is fin­ished, ready to be typed up and sent out. Time now for the real work. At one edge of Watterson’s desk are a cou­ple of pen­cils, an eraser, the curled zigzags of shav­ings. On the other side of the desk are tools for dif­fer­ent parts of the cre­ative process. A small sable brush (for ink­ing), a Rapidograph foun­tain pen (for let­ter­ing the di­a­logue), and a crowquill pen (for “odds and ends”). His set-up is “as low-tech as you can get”.

This is how he likes it. The sim­pler things are, the more con­trol he has over the work — which is the hill on which he’ll die and take every­one with him if he has to. For Watterson, it’s a ques­tion of main­tain­ing artis­tic in­tegrity. He de­rives an enor­mous amount of pride from the fact that he can say, “I write every word, draw every line, color every Sunday strip, and paint every book il­lus­tra­tion my­self.” The strip is a “one-man op­er­a­tion” be­cause he’s con­vinced it’s the only way to pre­serve the in­tegrity of his craft.

For Watterson, craft has never been a side dish to the main course. It’s in­ex­tri­ca­ble from the truths he wants to ex­press and the mean­ing he hopes his work might have for its read­ers. It’s his be­lief that half a cen­tury ago, the best comics were more than amus­ing to look at; they were beau­ti­ful and un­doubt­edly counted as cap­i­tal-A Art. Here in the mid-nineties, he “can’t think of a sin­gle strip to­day that comes close to that stan­dard of crafts­man­ship”.

His read­ers think he’s achieved that kind of qual­ity, from know-noth­ings like me who in­tuit some­thing spe­cial here that I haven’t found any­where else, to icons of the craft like Charles M. Schulz, cre­ator of Peanuts. In a fore­word to The Essential Calvin and Hobbes, Schulz praises Watterson’s abil­ity to show us the nu­mi­nous in the mun­dane by el­e­gantly draw­ing “bedside ta­bles … and liv­ing room couches and chairs and lamps … and all the things that make a comic strip fun to look at”. He adds that this at­ten­tion to the height­ened de­pic­tion of the small­est de­tails is what makes a strip truly great: if all the car­toon­ist does is “illustrate a joke”, the car­toon­ist “is go­ing to lose”.

It turns out there are a lot of ways a car­toon­ist can lose, and most of those wins and losses come out of one es­sen­tial bat­tle: cre­ativ­ity ver­sus com­merce. Here, com­merce is rep­re­sented by Universal Press Syndicate, which Watterson refers to as “the syn­di­cate”, like an or­gan­i­sa­tion of vil­lains in a comic book. (Once, he even pub­licly called them “a blood­suck­ing cor­po­rate par­a­site”.) The syn­di­cate act as mid­dle-man be­tween the artist and pub­lish­ing out­lets, and with­out them, there’s no re­al­is­tic chance of any car­toon­ist get­ting a strip printed in a ma­jor news­pa­per and mak­ing some­thing like a liv­able in­come. Middle-man, ex­cept that Watterson sees them as tak­ing a side — the side of the news­pa­pers.

The con­flict comes out of the fact that, Watterson laments, the “commercial, mass-mar­ket needs of news­pa­pers are not of­ten sym­pa­thetic to the con­cerns of artis­tic ex­pres­sion”. It’s a dy­namic that’s made him face “countless eth­i­cal de­ci­sions mas­querad­ing as sim­ple busi­ness de­ci­sions”. These things that oth­ers re­gard as only or mostly artis­tic con­cerns, he looks at as ques­tions of ethics, which ex­plains his re­fusal to back down even when giv­ing in was so much eas­ier (not to men­tion more prof­itable). It was never a ques­tion of draw­ing a lit­tle dif­fer­ently or work­ing to an al­tered sched­ule; it was a ques­tion of what truly mat­tered at a level Watterson per­haps thinks of as spir­i­tual.

Watterson’s way of speak­ing about these things oc­ca­sion­ally veers into the self-im­por­tant reg­is­ter of griev­ance, the eter­nal com­plaint of some­one for whom things-as-they-are never sat­isfy be­cause things-as-they-were al­ways seem bet­ter. But there’s no deny­ing the con­vic­tion with which he fought the fight, even be­fore he had the name-brand au­thor­ity he’d later earn, even back when it re­ally looked like he was go­ing to lose. And he came very close to los­ing some of his biggest bat­tles with the syn­di­cate.

IV.

“When car­toon­ists fight their syn­di­cates,” Watterson says, “it’s usu­ally to make more money, not less.” Yet for six years, Watterson kept his heels dug deep in the earth, fists up, box­ing stance against his syn­di­cate’s plan to make them all mil­lions of dol­lars.

Watterson’s con­tract meant the syn­di­cate re­tained the right to turn Calvin and Hobbes into toys, t-shirts, and other ephemera, and it be­came clear pretty early that they could all ex­pect stu­pid amounts of money from mer­chan­dis­ing. As Nevin Martell puts it in his inim­itable book Looking for Calvin and Hobbes, the eight­ies were a time when “big-name car­toon­ists were mak­ing big bucks by har­ness­ing the sell­ing power of their char­ac­ters”.

The cre­ator of Garfield, Jim Davis, be­came the head of his own em­pire just a few years af­ter he’d started draw­ing his mopey cat. There are Garfield plush toys, Garfield py­ja­mas, Garfield slot ma­chines, Garfield movies, Garfield-themed cruises, all of it bring­ing in a for­tune be­tween 750 mil­lion and one bil­lion dol­lars a year — and Davis gets a share of that. “So here’s a math prob­lem for the kids,” writes Martell. “If there were 255 mil­lion suc­tion-cupped Garfield dolls sold over the course of the decade, how many small trop­i­cal is­lands was Jim Davis able to buy with the pro­ceeds?”

Maths like this led Watterson’s syn­di­cate to in­clude li­cens­ing rights in their con­tract with him, as­sum­ing the artist would have no prob­lem with it. All that money for do­ing what he loves? Seemed a no-brainer. The prob­lem was that Watterson had an ex­act­ing idea of what it was he loved do­ing, and it was at odds with toys and tat, in­dif­fer­ent to si­los of cash. “I went into car­toon­ing to draw car­toons,” Watterson says, “not to run a cor­po­rate em­pire.”

It was still early days in the ten-year run of Calvin and Hobbes when the syn­di­cate ap­proached Watterson with its big ideas of Calvin sweat­shirts, Spaceman Spiff bumper stick­ers, an an­i­mated Calvin and Hobbes Saturday show, maybe a movie, and — worst of all — a Hobbes doll. Watterson re­ally loathed the Hobbes doll. To make sense of how much it both­ered him, we need to talk about the tiger in the room.

When Watterson cre­ated Hobbes, his fo­cus was on the char­ac­ter more than the con­ceit of a teddy that comes to life. Watterson told Rich West for The Comics Journal that “there’s some­thing a lit­tle pe­cu­liar about [Hobbes] that’s, hope­fully, not read­ily cat­e­gorised”. But Watterson’s read­ers of­ten wanted Hobbes cat­e­gorised into ei­ther “real” or “imaginary”. So Watterson came up with a com­pelling non-an­swer to the ques­tion:

“Calvin sees Hobbes one way, and every­one else sees Hobbes an­other way. I show two ver­sions of re­al­ity, and each makes com­plete sense to the par­tic­i­pant who sees it. I think that’s how life works.”

You could say Hobbes is both imag­i­na­tively real and re­ally imag­i­nary, de­pend­ing on your per­spec­tive. Hobbes can be ei­ther, which also means he’s both. Is Hobbes a tiger or a toy? Yes.

Watterson in­sisted that if he was­n’t go­ing to set­tle the ques­tion of Hobbes, then he def­i­nitely would­n’t let some toy man­u­fac­turer set­tle it by turn­ing Hobbes “into a stuffed toy for real, and de­prive the strip of an el­e­ment of its magic”. He’d sound off wher­ever he could on how “licensing usu­ally cheap­ens the orig­i­nal cre­ation” by sat­u­rat­ing a mar­ket with char­ac­ters un­til read­ers are bored of see­ing them; how a multi-pan­eled story with dy­namic ac­tion can­not be re­spected by the va­garies of a cof­fee mug il­lus­tra­tion; how sub­tlety is sac­ri­ficed for im­me­di­acy; how sell­ing off “everything fun and mag­i­cal” means “the strip’s world is di­min­ished”.

He has a hun­dred lines like these, ar­tic­u­la­tions of higher rea­son­ing against mer­chan­dis­ing, but just once, in the Tenth Anniversary Book, he drops the high-and-mighty in favour of I-the-mighty: “Calvin and Hobbes was de­signed to be a comic strip and that’s all I want it to be. It’s the one place where every­thing works the way I in­tend it to.”

Here again is the artist as lone ge­nius, the one-man op­er­a­tion he jeal­ously guards. Watterson’s con­vic­tions are sin­cere, and he’s put a lot of thought into de­fend­ing those val­ues, but maybe there’s some emo­tion in­volved too. Maybe part of Watterson’s aver­sion to what his syn­di­cate were ask­ing for has some­thing to do with not want­ing to play with oth­ers. It of­ten seems with Watterson like he’s never quite made peace with the pub­lic na­ture of the pri­vate world he cre­ated in Calvin and Hobbes. He seems un­com­fort­able with com­pro­mise.

There’s a Calvin and Hobbes strip where Calvin dis­cov­ers the world has lost all colour. There’s “no hue, value, or chroma” as he moves through his house de­picted in neg­a­tive re­lief. The cause of this aber­ra­tion was an ar­gu­ment with his dad, who in the fi­nal (full colour) panel says, “The prob­lem is, you see every­thing in terms of black and white, ” to which Calvin cries, “SOMETIMES THAT’S THE WAY THINGS ARE!”

Watterson wrote that strip to get onto the page and out of his head the way he felt when fight­ing his syn­di­cate. This move to a re­duc­tive black-white bi­nary is a dif­fi­cult cir­cle to square with the artist who re­fuses to set­tle the on­tol­ogy of Hobbes with a de­fin­i­tive an­swer. How is it that Watterson both adores and re­jects am­bi­gu­ity? Maybe Watterson is nei­ther one thing or the other. Maybe he’s both. Lee Salem, pres­i­dent of Universal Press Syndicate, says, “Bill is both re­fresh­ingly dif­fer­ent and ex­as­per­at­ingly dif­fer­ent, de­pend­ing on one’s per­spec­tive.”

The syn­di­cate found out just how dif­fer­ent he could be. Other car­toon­ists wanted fame, wanted to be printed in every news­pa­per in the world, wanted an ocean of money where the tide was al­ways in. But other car­toon­ists could swim; Watterson felt like he was drown­ing. So he told his syn­di­cate, “No.” No t-shirts, no mer­chan­dise, no stuffed Hobbes toys. But the ar­gu­ment would­n’t go away, and even Watterson un­der­stood why. As he put it, “Trainloads of money were at stake — mil­lions and mil­lions of dol­lars could be made with a few sig­na­tures. Syndicates are busi­nesses, and no busi­ness passes up that kind of op­por­tu­nity with­out an ar­gu­ment.”

So Watterson and the syn­di­cate had that ar­gu­ment. For six years.

V.

The strug­gle went on un­til 1991. It was a fight that few knew much about — cer­tainly not the happy ma­jor­ity of read­ers who met with Calvin each morn­ing in the pa­per and had noth­ing but fun — but Watterson viewed the con­flict as some­thing Biblical in its in­ten­sity and stakes.

On one side of the bat­tle: the con­glom­er­ated cor­po­rate power of the syn­di­cate-as-Go­liath, with their money and lawyers and bind­ing legalese, and their teams of peo­ple with vested in­ter­ests work­ing against the sim­ple artist.

On the other side: Bill Watterson, pen­cil in hand and heart full of un­com­pro­mis­able val­ues.

In one of Watterson’s strips from that time, Calvin re­fuses to get in the bath, shout­ing about how he’ll never com­pro­mise his prin­ci­ples; cut to Calvin in the bath, sullen and grum­bling, “I don’t need to com­pro­mise my prin­ci­ples, be­cause they don’t have the slight­est bear­ing on what hap­pens to me any­way.”

The way Watterson tells it, he was pow­er­less to stop them forc­ing him to mer­chan­dise Calvin and Hobbes, and all he could do about it was quit, in which case the syn­di­cate would just hire a team of anony­mous ghost-artists to churn out more sto­ries for Watterson’s duo. He was one small man fac­ing down a global be­he­moth that had risen from the swamp of mod­ern cap­i­tal­ism. What could he do?

I’m not so con­vinced by the case he makes. For any mer­chan­dis­ing op­por­tu­ni­ties to be worth much, the strip had to con­tinue en­chant­ing read­ers, and to do that it had to be writ­ten and drawn by the man who’d brought it into the world. Calvin and Hobbes, as Nevin Martell notes, “was not your run-of-the-mill, gag-a-day strip with av­er­age art­work that any­one could do”. That’s why Lee Salem ac­knowl­edged in a con­ver­sa­tion with Martell that the syn­di­cate was “lucky that [Watterson] did­n’t call one day and say, ‘I quit.’”

That’s not the only source of fishy odour around Watterson’s sug­ges­tion that all the cards were held by other play­ers. There’s also the plain fact of how long the ar­gu­ment ran for. Every week, month, and year that passed with Watterson hold­ing out and the syn­di­cate es­sen­tially shrug­ging and say­ing, “Okay, we won’t force it,” re­vealed how un­will­ing they were to sim­ply bend him to their will.

Then there’s the of­fer they made to him not long be­fore the whole thing was de­cided. Lee Salem went to Watterson’s house with a box of boot­leg t-shirts with Calvin and/​or Hobbes printed on them. (I first found out this was a thing when, in an episode of Friends, Joey tells Rachel he can’t take his sweater off in pub­lic be­cause his t-shirt “has a pic­ture of Calvin do­ing Hobbes”. I re­mem­ber scrunch­ing my young eyes up and think­ing no no no no no as if the word could scrub out the un­wanted men­tal pic­ture that had been forced on me.)

Salem told Watterson that the best way to choke off the flow of this stuff was to li­cense Calvin and Hobbes. Granting mer­chan­dis­ing rights would mean that an en­tirely sep­a­rate com­pany, whose in­ter­est would be con­trol­ling the le­gal use of those rights, would come down tough on the pi­rates mak­ing il­le­gal mer­chan­dise. On top of that, all the prof­its from mer­chan­dis­ing would go into a brand-new fund for sav­ing tigers across the world. This de­bate did­n’t take five years; sounds like it did­n’t take five min­utes: Watterson said no.

So, maybe it’s the clar­ity of hind­sight, an unim­pres­sive back­wards pre­dic­tion, but I don’t find it sur­pris­ing that when the dis­pute was fi­nally re­solved af­ter six years, it fell Watterson’s way. The syn­di­cate backed off, agreed not to li­cense any mer­chan­dise, and went as far as rewrit­ing their con­tract with Watterson in his favour. And it re­ally went in his favour.

VI.

Who knows for sure how the sab­bat­i­cals came about? Well, Watterson knows and the peo­ple at the syn­di­cate know, but they’re telling dif­fer­ent sto­ries. In Nevin Martell’s book, Watterson de­manded two sab­bat­i­cals as part of his rene­go­ti­ated con­tract, which is pre­sum­ably what Universal says went down. On the other hand, Watterson (who gave no in­put to Martell’s book ex­cept to ask Lee Salem of it, “Who cares?”) main­tains that Universal of­fered him the sab­bat­i­cals and he ac­cepted. This seems un­usu­ally gen­er­ous for a syn­di­cate Watterson also por­trays as es­sen­tially money-grub­bing, but again — who knows?

Sabbaticals were ba­si­cally un­known for syn­di­cated car­toon­ists. It was a huge ask of read­ers to take some months away from a strip and not lose in­ter­est or re­place it with an­other strip. For ed­i­tors, re-runs were a kick in the crotch, pay­ing for a strip they’d al­ready paid for. Universal knew all of this and ral­lied to craft a mes­sage sup­port­ing their artist’s need to recharge his cre­ative bat­ter­ies. Watterson knew all of this too and thought, They can have a worn out Calvin and Hobbes, or I can take this break and come back with work I’m proud of, that they’ll be ea­ger to print.

In May of 1991, Calvin and Hobbes went into re-runs.

For the next nine months, Watterson lived like he did­n’t have mil­lions of dol­lars in mer­chan­dise a mere sig­na­ture away, like his work was­n’t so widely adored that na­tional news­pa­pers were pub­licly count­ing down the days un­til he brought them all some­thing new. Instead, at the age of thirty-three and mid-ca­reer, he was liv­ing the low-key life of a re­tiree.

As the months slipped by, Watterson started meet­ing up with his art pro­fes­sor from Kenyon. They painted to­gether, the older man with skill and the younger man with in­el­e­gance slowly turn­ing into ba­sic pro­fi­ciency. A dy­namic de­vel­oped be­tween them that helped Watterson’s re­cov­ery. No longer teacher and stu­dent, they were (as the pro­fes­sor told Martell) “just two guys who liked to do a cou­ple of things re­ally well”.

The time off did what it was sup­posed to for Watterson, and in early 1992, he re­turned to draw­ing Calvin and Hobbes. He was three years away from quit­ting for­ever.

***

Watterson threw him­self into his next big swing with the syn­di­cate. He wanted to rad­i­cally change the Sunday strip.

In the Golden Age of comics, Sunday strips were given a whole page to cre­ate worlds and tell sto­ries. “With all that space to fill,” Watterson says, “cartoonists pro­duced works of ex­tra­or­di­nary beauty and power.” Around the mid­dle of the twen­ti­eth cen­tury, that page was re­duced to half a page, and most news­pa­pers saved more space by de­cap­i­tat­ing the strip of its top row. Whatever re­mained got strait­jack­eted into “specific and un­yield­ing” di­men­sions that forced all comics into the ex­act same se­quence of pan­els. Watterson says the re­sult was that he’d “often need to elim­i­nate di­a­logue or sim­plify the draw­ings so they’d fit in the ar­bi­trary space the for­mat al­lot­ted”.

On the back of so many wins for Calvin and Hobbes, Watterson de­cided it was time to push back. The pro­posal was sim­ple: his Sunday comic would come “exclusively as a half-page fea­ture with no panel re­stric­tions”. The story (and not the com­mer­cial needs of the news­pa­per) would de­ter­mine the shape of each Sunday strip.

The syn­di­cate warned Watterson they might lose half of the pa­pers run­ning Calvin and Hobbes and, with them, half of his in­come. Watterson wanted to hold him­self to a higher stan­dard than how many pa­pers pub­lished him and how much money he made. Whatever he lost would, he fig­ured, be worth it — “if I could work at the lim­its of my abil­i­ties for a change.” To his sur­prise (though maybe only to his), the syn­di­cate agreed to sell Calvin and Hobbes with this caveat in place.

Editors were fu­ri­ous. Watterson seemed ig­no­rant of the many and com­plex re­quire­ments of pub­lish­ing a pa­per in an ex­pen­sive in­dus­try al­ready strug­gling, one that had just paid him for nine months of re-runs. Watterson looked like a mega­lo­ma­niac snatch­ing pre­cious page space from other car­toon­ists. This lat­est act of ego was an af­front to both busi­ness and moral­ity.

Watterson was hav­ing none of it. No ed­i­tor had to buy Calvin and Hobbes, he said. He’d be of­fer­ing them a su­pe­rior prod­uct for the same price. With a lit­tle imag­i­na­tion, he told them, Sunday strips could be reimag­ined (and re­sized) with­out a zero-sum com­pe­ti­tion be­tween artists. Some ed­i­tors threat­ened to can­cel their con­tracts with Universal over this. It looked like the syn­di­cate’s warn­ings to Watterson were well-founded: Calvin and Hobbes was threat­ened with wide­spread can­cel­la­tion.

It says some­thing about the pop­u­lar­ity of Calvin and Hobbes — not to men­tion Watterson’s pulling power as a car­toon­ist — that af­ter all the out­rage and ar­gu­ments, only fif­teen of the 1,800 pa­pers run­ning Watterson’s strip threat­ened to re­move it from their pages. And only seven fol­lowed through. Calvin and Hobbes was still very much on top. But it was also get­ting on top of Watterson.

***

The stan­dard­ised Sunday strip was like a jig used by car­pen­ters: when they have to make the same cut over and over, they set up a jig to rest the saw and the wood against, and it au­to­mates the rep­e­ti­tion, mak­ing the process smoother, faster. Less cre­ative, by de­sign. The Sunday strip granted less cre­ative free­dom, but it stream­lined the process, mak­ing it smoother and faster to draw the strips.

Watterson no longer had the ben­e­fit of the jig. He had to go back to the lit­eral draw­ing board each time, so the Sunday strips that used to take a day to draw now re­quired a day and a half, some­times longer. They de­manded a slower process, more time to think and tin­ker. Watterson liked draw­ing his strip at this turtle’s pace, but the ever-loom­ing dead­line made it hard work to go so slowly.

His ap­proach to dead­lines was to stay well ahead of them. When you’re faced with a due date (as he told Lee Nordling in Your Career in Comics) “there’s no qual­ity con­trol. It’s just garbage in, garbage out.” This was be­cause “the never-end­ing pres­sure to meet dead­lines en­cour­ages car­toon­ists to pub­lish vir­tu­ally every­thing they think up”. Watterson’s ap­proach was in­stead to “stay far enough ahead of the dead­lines that I can throw away mediocre ma­te­r­ial and write some­thing bet­ter”. This is qual­ity de­ter­mined by quan­tity: the more stuff in the waste­bas­ket, the bet­ter the strip.

Watterson was al­ready work­ing weeks and some­times months ahead of sched­ule, re­fus­ing to sub­mit sec­ond-tier work, main­tain­ing this through­out six years of the li­cens­ing fight, and now he took more time than ever to think cre­atively about the Sunday strips sud­denly re­quir­ing triple the ef­fort to pro­duce. It took more and more of his time to keep Calvin and Hobbes go­ing. “I had to steal that ex­tra time,” he con­fessed in Sunday Pages 1985 – 1995, “from what would have been some sem­blance of an or­di­nary life.”

Watterson burned out. In April of 1994, he took his sec­ond nine-month sab­bat­i­cal. You have to as­sume he did a lot of the same as last time, some paint­ing, some walk­ing, nowhere to be and no time he had to be there. He never said or wrote much about that sec­ond sab­bat­i­cal, but it’s clear that it did­n’t do what it was meant to do. He re­turned to the strip in January, 1995, know­ing for sure what he’d sus­pected for a while: he was done with Calvin and Hobbes.

VII.

It’s early af­ter­noon. The let­ter he wrote this morn­ing is sit­ting at the top of his desk, and his sable brush is in hand. His fin­gers are cramp­ing a lit­tle, but the work is go­ing well. Maybe there’s an air of solem­nity hang­ing over him on this par­tic­u­lar day, and maybe he in­dulges it, or maybe he tries to ig­nore it so he can get on with what he’s do­ing. A lot of maybes hang over a sin­gle cer­tainty: that he’s draw­ing the last Calvin and Hobbes that will ever be printed.

Watterson’s wife was the first to know that it was all com­ing to an end, and sec­ond was Lee Salem and the syn­di­cate. No one be­hind the scenes was all that sur­prised. It was ob­vi­ous to every­one as far back as 1992, af­ter Watterson had come back from his first sab­bat­i­cal, that he would be quit­ting the strip, it just was­n’t clear when that would hap­pen. Now it’s a sure thing. They’ve agreed that the last day of 1995, not far off, will be the last time a new Calvin and Hobbes runs in the pa­pers. All that’s left is to tell his ed­i­tors and, more im­por­tantly, his read­ers, which he’ll do by way of the res­ig­na­tion let­ter.

And af­ter? He’s un­cer­tain. Does he know here to­day, as he works on his last out­ing with the kid and his tiger, that he’ll spend the next five years draw­ing noth­ing at all? That he’ll abruptly stop do­ing what he’s done for the last ten years of his life?

In the decades af­ter clos­ing shop on Calvin and Hobbes, Watterson will be­come sur­geon-thor­ough in scalpel­ing out his cre­ation from his pri­vate life. Over the next thirty years, he’ll give only three or four in­ter­views; he’ll put out a sin­gle book (entirely un­re­lated to Calvin and Hobbes) for which he’ll do his usual amount of hype and mar­ket­ing (none); he’ll say al­most noth­ing of note on the duo he gave up for adop­tion to his read­ers. Eventually, he’ll force his ed­i­tor to put a no­tice on his web­site in­form­ing fans that he won’t read any let­ters that re­fer even pass­ingly to Calvin and Hobbes.

This KEEP OUT sign on the gate to his per­sonal life will do very lit­tle to keep well-mean­ing if ob­sti­nate tres­passers off his men­tal lawn. There will be a day when Watterson will be on the lit­eral lawn of his front yard and a re­porter from the Cleveland Plain Dealer will show up. He’ll be nei­ther the first nor the last jour­nal­ist to try his luck like this, and he’ll be no more nor less suc­cess­ful than any of the oth­ers. The artist and the re­porter will get into an off-the-record “almost col­le­giate” de­bate about the na­ture of pri­vacy. It won’t mat­ter if the re­porter wins on points, be­cause Watterson is never go­ing to con­cede.

In any case, even as Watterson dis­tances him­self from Calvin and Hobbes, his read­ers will re­main as close to them as ever. We go on read­ing the comics, see­ing our­selves in Calvin, hop­ing for a Hobbes, look­ing for an ad­ven­ture.

“My strip,” Watterson once said, “is about pri­vate re­al­i­ties, the magic of imag­i­na­tion, and the spe­cial­ness of cer­tain friend­ships.” Well, ditto for my re­la­tion­ship with the boy Calvin and his tiger Hobbes. Those ad­ven­tures of a weirdo from an­other planet and his homi­ci­dal psy­cho jun­gle cat, their mag­i­cal world in which the days are just packed with de­ranged mu­tant killer mon­ster snow goons, that place where sci­en­tific progress goes “boink” and there’s trea­sure every­where — they be­long to their read­ers.

Back to his desk and that fi­nal strip. This one is made up of only five pan­els. The last panel is the largest. There’s a lot of un-inked page show­ing through here, just Calvin and Hobbes in the snow and a few trees to in­di­cate the for­est around them. Watterson has al­ready pen­cilled in the di­a­logue. “Wow,” Calvin says to Hobbes in the first panel, “it re­ally snowed last night! Isn’t it won­der­ful?” These two fig­ures have been lightly pen­cilled in car­ry­ing a sled through waist-deep snow. Now, Watterson’s ink­ing the lines, which sit with some weight on the oth­er­wise blank pa­per.

When Watterson first started draw­ing Calvin and Hobbes in the eight­ies, there were only 64 colours for him to choose from. Here at the end of his run, he has 125 colours avail­able to him. But he’s keep­ing it sim­ple. He usu­ally colours in the panel bor­ders and the word bal­loons, but he’s de­cided, here, to leave it all white. Only the char­ac­ters and their sled are given colour, be­cause he wants this draw­ing to have “a very spare and open look”.

He leans back in his chair to look at what he’s done. All that white he’s left of­fers the ef­fect of wip­ing the page clean, the strip stripped down un­til the page is empty for some­thing new. Something dif­fer­ent. Like he’s opened a tin of white­wash and painted over his work. He leaves the page white, empty, and fresh.

Matthew Morgan writes Volumes, a Substack about the places where books and life over­lap.

Sources used:

Unattributed quo­ta­tions come from Calvin and Hobbes: Tenth Anniversary Book (1995).

Unattributed quo­ta­tions come from Calvin and Hobbes: Tenth Anniversary Book (1995).

References to spe­cific Calvin and Hobbes strips come from the Calvin and Hobbes col­lec­tions pub­lished be­tween 1988 and 1996.

References to spe­cific Calvin and Hobbes strips come from the Calvin and Hobbes col­lec­tions pub­lished be­tween 1988 and 1996.

Looking for Calvin and Hobbes (2009) Nevin Martell;

Looking for Calvin and Hobbes (2009) Nevin Martell;

“Some thoughts on the real world by one who glimpsed it and fled” (1990) Kenyon College grad­u­a­tion speech;

“Some thoughts on the real world by one who glimpsed it and fled” (1990) Kenyon College grad­u­a­tion speech;

“The Bill Watterson Interview” in The Comics Journal (1989) Richard West;

Yesterday, June 15, 2026, a small and unim­por­tant an­nounce­ment ap­peared in Apple de­vel­oper news: New do­main for Sign in with Apple and iCloud+ Hide My Email.

Long story short: now both Sign in with Apple and Hide My Email aliases are go­ing to be is­sued on the @private.icloud.com sub­do­main. This makes it much eas­ier to ban all aliases with­out af­fect­ing non-re­lay mail­boxes on iCloud mail.

This is cer­tainly a big hit for iCloud pri­vacy, since some plau­si­ble de­ni­a­bil­ity to­gether with Apple’s back­ing made ban­ning iCloud aliases costly. But now a lot of ser­vices will just refuse to ac­cept these emails, just like what hap­pens with free tem­po­rary mail­boxes.

Hopefully, this can reach some­one at Apple so they can re­con­sider this de­ci­sion.

If you use iCloud+ and Hide My Email, there is still time to gen­er­ate more aliases on @icloud.com as the change has not yet landed and the rate limit for cre­at­ing aliases is at least 30 per hour.

Z ai’s GLM-5.2 is the new lead­ing open weights model on the Artificial Analysis Intelligence Index scor­ing 51 and it sits on the Pareto fron­tier of Intelligence vs Cost per Task

GLM-5.2 is the same size as GLM-5.1 (744B to­tal / 40B ac­tive pa­ra­me­ters) but scores 11 points higher on the Intelligence Index v4.1, plac­ing ahead of MiniMax-M3 (44) and DeepSeek V4 Pro (max, 44). On the first-party API it is priced in line with GLM-5.1 at $1.4/$4.4/$0.26 per 1M in­put/​out­put/​cache hit to­kens

Key re­sults:

➤ GLM-5.2 is the lead­ing open weights model on the Intelligence Index v4.1. At 51, it leads MiniMax-M3 (44), DeepSeek V4 Pro (max, 44) and Kimi K2.6 (43)

➤ Improvements across most eval­u­a­tions, par­tic­u­larly sci­en­tific rea­son­ing: GLM-5.2 gains over GLM-5.1 on most eval­u­a­tions, led by sci­en­tific rea­son­ing on CritPt (+16 points to 21%) and HLE (+12 points to 40%), along­side AA-LCR (+9 points to 71%), tau3 bank­ing (+15 points to 27%) and SciCode (+7 points to 50%). TerminalBench v2.1 also im­proves (+16 points to 78%) and GPQA Diamond gains 3 points to 89%

➤ Leading open weights model on GDPval-AA v2 and com­pet­i­tive with pro­pri­etary mod­els: GLM-5.2 scores 1524 on GDPval-AA v2, ahead of MiniMax-M3 (1418) and DeepSeek V4 Pro (max, 1328). This im­pres­sive re­sult places GLM-5.2 in-line with pro­pri­etary mod­els in­clud­ing GPT-5.5 (xhigh rea­son­ing). GDPval-AA v2 builds on the orig­i­nal GDPval-AA by baselin­ing Elo to hu­man per­for­mance at 1000, in­tro­duc­ing a ro­tat­ing panel of fron­tier-model judges, and rais­ing the turn limit from 100 to 250 for longer-hori­zon agent tra­jec­to­ries

➤ GLM-5.2 uses more out­put to­kens per task than other lead­ing open weights mod­els: the model uses 43k out­put to­kens per Intelligence Index task, up from GLM-5.1 (26k) and above MiniMax-M3 (24k), Kimi K2.6 (35k) and DeepSeek V4 Pro (max, 37k)

➤ On the Intelligence vs. Cost per Task Pareto Frontier: GLM-5.2 is on the Pareto fron­tier of the Intelligence vs Cost per Task chart, with the low­est cost per task among mod­els at its in­tel­li­gence level. GLM-5.2 costs ~$0.46 per task, com­pared to GLM-5.1 ($0.25), Kimi K2.6 ($0.31), MiniMax-M3 ($0.18) and DeepSeek V4 Pro (max, $0.05)

Additional Model Details:

➤ License: MIT

➤ Size: 744B to­tal pa­ra­me­ters, 40B ac­tive pa­ra­me­ters, equiv­a­lent to GLM-5.1

➤ Context win­dow: 1M to­kens, up from 200K on GLM-5.1

➤ Pricing: $1.4/$0.26/$4.4 per 1M in­put/​cache hit/​out­put to­kens

➤ Availability: Alongside Z ai’s first-party API, GLM-5.2 is avail­able across third-party providers in­clud­ing DeepInfra, Novita, Nebius, Parasail, Siliconflow, GMI Cloud, Baseten, and Fireworks

GLM-5.2 leads all open weights mod­els on GDPval-AA v2, our pri­mary met­ric for real-world agen­tic per­for­mance. At 1524 it places ahead of MiniMax-M3 (1418) and DeepSeek V4 Pro (max, 1328), and is ef­fec­tively level with GPT-5.5 (xhigh, 1514). We vi­su­ally in­spected GLM-5.2′s out­puts across a range of GDPval-AA tasks. We have at­tached a se­lec­tion be­low.

GLM-5.2 scores 4 on the AA-Omniscience Index, up from GLM-5.1 (2). The gain comes from both higher ac­cu­racy (25.1% vs 24.2%) and a lower hal­lu­ci­na­tion rate (28.1% vs 29.4%), with at­tempt rate flat at 47%.

GLM-5.2 uses 43k out­put to­kens per Intelligence Index task, of which 37k is rea­son­ing. This is up from GLM-5.1 (26k) and higher than open weights peers MiniMax-M3 (24k) and Kimi K2.6 (35k), plac­ing it among the less to­ken-ef­fi­cient open weights mod­els at its in­tel­li­gence level. GLM-5.2 sits off the most at­trac­tive quad­rant on the Intelligence vs Output Tokens chart.

Breakdown of the in­di­vid­ual eval­u­a­tions in the Artificial Analysis Intelligence Index v4.1.

Compare GLM-5.2 with other lead­ing mod­els at: https://​ar­ti­fi­cial­analy­sis.ai/​mod­els/​glm-5 – 2

Edit: This has gen­er­ated a nice dis­cus­sion on Hacker News — check it out too!

I needed to check that one con­tainer could reach an­other over an in­ter­nal Docker net­work: a plain GET /health against a ser­vice on a shared net­work. The ob­vi­ous move is curl http://​ser­vice:8642/​health. But this app im­age was stripped right down, with no curl or wget and noth­ing else around that I could use to open a socket.

As it turns out, bash can speak HTTP by it­self bash can open a TCP socket, and you can write a small HTTP re­quest to it by hand. Opening a con­nec­tion to a host and port and writ­ing the re­quest needs noth­ing be­yond the shell that’s al­ready there:

bash

exec 3<>/dev/tcp/service/8642 printf ‘GET /health HTTP/1.1\r\nHost: ser­vice\r\nCon­nec­tion: close\r\n\r\n’ >&3 cat <&3

ser­vice here is just the host­name of what­ever you’re talk­ing to. It has to re­solve and be reach­able from wher­ever you run this, so it needs to be set up first: a con­tainer or ser­vice name on a Docker net­work you’ve con­fig­ured, or any DNS name that re­solves. Swap in your own host and port.

That prints the whole re­sponse: the sta­tus line, the head­ers, the blank line, and the body. To add a header, such as an Authorization: Bearer to­ken, put an­other \r\n-terminated line be­fore the blank line that ends the re­quest:

bash

exec 3<>/dev/tcp/service/8642 printf ‘GET /v1/models HTTP/1.1\r\nHost: ser­vice\r\nAu­tho­riza­tion: Bearer %s\r\nConnection: close\r\n\r\n’ “$API_KEY” >&3 cat <&3

What caught me out the first time is that /dev/tcp is­n’t a real de­vice file. There’s no such path on disk; ls /dev/tcp finds noth­ing, and cat /dev/tcp/… from an­other shell just er­rors. It’s a redi­rec­tion that bash han­dles in­ter­nally. From the Bash man­ual:

/dev/tcp/host/port — If host is a valid host­name or Internet ad­dress, and port is an in­te­ger port num­ber or ser­vice name, bash at­tempts to open the cor­re­spond­ing TCP socket.

/dev/tcp/host/port — If host is a valid host­name or Internet ad­dress, and port is an in­te­ger port num­ber or ser­vice name, bash at­tempts to open the cor­re­spond­ing TCP socket.

The names were picked be­cause no real Unix has a /dev/tcp or /dev/udp hi­er­ar­chy, so there’s noth­ing to col­lide with. Bash does the DNS lookup and the con­nect(2) for you, and exec 3<> hands the socket a file de­scrip­tor (3) you read from and write to like any other.

A few things worth know­ing:

This is not a real HTTP client. It does not parse HTTP prop­erly, han­dle redi­rects, chun­ked re­sponses, com­pres­sion, re­tries, TLS, or any of the other things curl qui­etly does for you. It’s a quick con­nec­tiv­ity and de­bug­ging trick.

The Connection: close header mat­ters. Without it the server keeps the con­nec­tion open af­ter it re­sponds, which is the HTTP/1.1 de­fault, and cat <&3 then waits for­ever for bytes that never ar­rive. Asking the server to close means cat reaches EOF and re­turns. Wrapping the call in time­out 6 bash -c ‘…’ cov­ers you ei­ther way.

There’s no TLS. /dev/tcp opens a raw socket, so this only works for plain­text HTTP. For https you’d need openssl s_­client, and by then you may as well have the proper tools.

This is a bash fea­ture, not POSIX. dash (Debian’s /bin/sh) and zsh don’t have it, so a #!/bin/sh script can’t use it. Call bash di­rectly.

It’s a com­pile-time op­tion, switched on when bash is built with –enable-net-redirections. Most main­stream builds en­able it, and it worked with­out any fuss in the Debian-based im­age I was in, but Debian shipped it dis­abled for years, so on an old or very min­i­mal sys­tem it’s worth check­ing first.

For day-to-day work curl is still the right tool. But in­side a de­lib­er­ately small con­tainer where you can’t in­stall any­thing, this gets a quick check done with­out adding a pack­age.

TLDR: JWTs should not be used for keep­ing your user logged in. They are not de­signed for this pur­pose, they are not se­cure, and there is a much bet­ter tool which is de­signed for it: reg­u­lar cookie ses­sions.

If you’ve got a bit of time to watch a pre­sen­ta­tion on it, I highly rec­om­mend this talk: https://​www.youtube.com/​watch?v=pY­eek­wv3vC4 (Note that other top­ics are largely skimmed over, such as CSRF pro­tec­tion. You should learn about other top­ics from other sources. Also note that “valid” use­cases for JWTs at the end of the video can also be eas­ily han­dled by other, bet­ter, and more se­cure tools. Specifically, PASETO.)

A re­lated topic: Don’t use lo­cal­Stor­age (or ses­sion­Stor­age) for au­then­ti­ca­tion cre­den­tials, in­clud­ing JWT to­kens: https://​www.rdegges.com/​2018/​please-stop-us­ing-lo­cal-stor­age/

The rea­son to avoid JWTs comes down to a cou­ple dif­fer­ent points:

The JWT spec­i­fi­ca­tion is specif­i­cally de­signed only for very short-live to­kens (~5 minute or less). Sessions need to have longer lifes­pans than that.

“stateless” au­then­ti­ca­tion sim­ply is not fea­si­ble in a se­cure way. You must have some state to han­dle to­kens se­curely, and if you must have a data store, it’s bet­ter to just store all the data. Most of this ar­ti­cle and the fol­lowup it links to de­scribes the spe­cific is­sues: http://​cryto.net/~​joepie91/​blog/​2016/​06/​13/​stop-us­ing-jwt-for-ses­sions/

(Yes, peo­ple are do­ing it, and yes, their ap­pli­ca­tions are flawed, and you should not re­peat that mis­take.)

(Yes, peo­ple are do­ing it, and yes, their ap­pli­ca­tions are flawed, and you should not re­peat that mis­take.)

JWTs which just store a sim­ple ses­sion to­ken are in­ef­fi­cient and less flex­i­ble than a reg­u­lar ses­sion cookie, and don’t gain you any ad­van­tage.

The JWT spec­i­fi­ca­tion it­self is not trusted by se­cu­rity ex­perts. This should pre­clude all us­age of them for any­thing re­lated to se­cu­rity and au­then­ti­ca­tion. The orig­i­nal spec specif­i­cally made it pos­si­ble to cre­ate fake to­kens, and is likely to con­tain other mis­takes. This ar­ti­cle delves deeper into the prob­lems with the JWT (family) spec­i­fi­ca­tion.

Rebuttals

But Google uses JWTs! Google does not use JWTs for user ses­sions in the browser. They use reg­u­lar cookie ses­sions. JWTs are used purely as Single Sign On trans­ports so that your lo­gin ses­sion on one server or host can be trans­ferred to a ses­sion on an­other server or host. This is within the rea­son­able use­cases for JWTs, and Google has the re­sources (security ex­perts) to cre­ate and main­tain a more se­cure JWT im­ple­men­ta­tion. Their JWTs are ef­fec­tively not the same as any­one else’s.

But state­less is bet­ter! You can’t se­curely have truly state­less au­then­ti­ca­tion with­out hav­ing mas­sive re­sources, see the cryto.net link above. Also, Stateless is a lie.

I don’t know how to setup ses­sions! You don’t reg­u­larly see ar­ti­cles ex­plain­ing ses­sions be­cause the tech­nol­ogy is­n’t par­tic­u­larly new. You also should­n’t need third party in­for­ma­tion for setup. A ses­sion im­ple­men­ta­tion’s doc­u­men­ta­tion should take you through the setup process by it­self. Almost any web server frame­work will con­tain an im­ple­men­ta­tion for ses­sions, and usu­ally it’s very easy to en­able if it is­n’t en­abled by de­fault. Express and other Node.js frame­works are some­what ex­cep­tions to this rule, pri­mar­ily be­cause they are highly mod­u­lar and sin­gle pur­pose. For Express, you sim­ply use the ex­press-ses­sion mid­dle­ware and a store con­nec­tor which works with your store (I rec­om­mend con­nect-ses­sion-knex, to be used with Postgres, MySQL, or pos­si­bly SQLite).

Short term to­kens

If you do need a short-lived, signed to­ken for some­thing, there is a bet­ter spec called PASETO which is de­signed to be se­cure. Just make sure you aren’t us­ing them for ses­sions.

How ses­sions work

I rec­om­mend check­ing out this gist by joepie91 to learn more how ses­sions work.

Why have I been blocked?

This web­site is us­ing a se­cu­rity ser­vice to pro­tect it­self from on­line at­tacks. The ac­tion you just per­formed trig­gered the se­cu­rity so­lu­tion. There are sev­eral ac­tions that could trig­ger this block in­clud­ing sub­mit­ting a cer­tain word or phrase, a SQL com­mand or mal­formed data.

What can I do to re­solve this?

You can email the site owner to let them know you were blocked. Please in­clude what you were do­ing when this page came up and the Cloudflare Ray ID found at the bot­tom of this page.