Slack is ex­tort­ing us with a $195k/yr bill in­crease An open let­ter, or some­thing

For nearly 11 years, Hack Club - a non­profit that pro­vides cod­ing ed­u­ca­tion and com­mu­nity to teenagers world­wide - has used Slack as the tool for com­mu­ni­ca­tion. We weren’t free­load­ers. A few years ago, when Slack tran­si­tioned us from their free non­profit plan to a $5,000/year arrange­ment, we hap­pily paid. It was rea­son­able, and we val­ued the ser­vice they pro­vided to our com­mu­nity.

However, two days ago, Slack reached out to us and said that if we don’t agree to pay an ex­tra $50k this week and $200k a year, they’ll de­ac­ti­vate our Slack work­space and delete all of our mes­sage his­tory.

One could ar­gue that Slack is free to stop pro­vid­ing us the non­profit of­fer at any time, but in my opin­ion, a six month grace pe­riod is the bare min­i­mum for a mas­sive hike like this, if not more. Essentially, Salesforce (a $230 bil­lion com­pany) is strong-arm­ing a small non­profit for teens, by pro­vid­ing less than a week to pony up a pretty mas­sive sum of money, or risk cut­ting off all our com­mu­ni­ca­tions. That’s ab­surd.

The small amount of no­tice has also been cat­a­strophic for the pro­grams that we run. Dozens of our staff and vol­un­teers are now scram­bling to up­date sys­tems, re­build in­te­gra­tions and mi­grate years of in­sti­tu­tional knowl­edge. The op­por­tu­nity cost of this forced mi­gra­tion is sim­ply stag­ger­ing.

Anyway, we’re mov­ing to Mattermost. This ex­pe­ri­ence has taught us that own­ing your data is in­cred­i­bly im­por­tant, and if you’re a small busi­ness es­pe­cially, then I’d ad­vise you move away too.

This post was rushed out be­cause, well, this has been a shock! If you’d like any ad­di­tional de­tails then feel free to send me an email.

Three years ago, ver­sion 2.0 of the Wasm stan­dard was (essentially) fin­ished, which brought a num­ber of new fea­tures, such as vec­tor in­struc­tions, bulk mem­ory op­er­a­tions, mul­ti­ple re­turn val­ues, and sim­ple ref­er­ence types.

In the mean­time, the Wasm W3C Community Group and Working Group have not been lazy. Today, we are happy to an­nounce the re­lease of Wasm 3.0 as the new “live” stan­dard.

This is a sub­stan­tially larger up­date: sev­eral big fea­tures, some of which have been in the mak­ing for six or eight years, fi­nally made it over the fin­ish­ing line.

64-bit ad­dress space. Memories and ta­bles can now be de­clared to use i64 as their ad­dress type in­stead of just i32. That ex­pands the avail­able ad­dress space of Wasm ap­pli­ca­tions from 4 gi­ga­bytes to (theoretically) 16 ex­abytes, to the ex­tent that phys­i­cal hard­ware al­lows. While the web will nec­es­sar­ily keep en­forc­ing cer­tain lim­its — on the web, a 64-bit mem­ory is lim­ited to 16 gi­ga­bytes — the new flex­i­bil­ity is es­pe­cially in­ter­est­ing for non-web ecosys­tems us­ing Wasm, as they can sup­port much, much larger ap­pli­ca­tions and data sets now.

Multiple mem­o­ries. Contrary to pop­u­lar be­lief, Wasm ap­pli­ca­tions were al­ways able to use mul­ti­ple mem­ory ob­jects — and hence mul­ti­ple ad­dress spaces — si­mul­ta­ne­ously. However, pre­vi­ously that was only pos­si­ble by de­clar­ing and ac­cess­ing each of them in sep­a­rate mod­ules. This gap has been closed, a sin­gle mod­ule can now de­clare (define or im­port) mul­ti­ple mem­o­ries and di­rectly ac­cess them, in­clud­ing di­rectly copy­ing data be­tween them. This fi­nally al­lows tools like wasm-merge, which per­form “static link­ing” on two or more Wasm mod­ules by merg­ing them into one, to work for all Wasm mod­ules. It also paves the way for new uses of sep­a­rate ad­dress spaces, e.g., for se­cu­rity (separating pri­vate data), for buffer­ing, or for in­stru­men­ta­tion.

Garbage col­lec­tion. In ad­di­tion to ex­pand­ing the ca­pa­bil­i­ties of raw lin­ear mem­o­ries, Wasm also adds sup­port for a new (and sep­a­rate) form of stor­age that is au­to­mat­i­cally man­aged by the Wasm run­time via a garbage col­lec­tor. Staying true to the spirit of Wasm as a low-level lan­guage, Wasm GC is low-level as well: a com­piler tar­get­ing Wasm can de­clare the mem­ory lay­out of its run­time data struc­tures in terms of struct and ar­ray types, plus un­boxed tagged in­te­gers, whose al­lo­ca­tion and life­time is then han­dled by Wasm. But that’s it. Everything else, such as en­gi­neer­ing suit­able rep­re­sen­ta­tions for source-lan­guage val­ues, in­clud­ing im­ple­men­ta­tion de­tails like method ta­bles, re­mains the re­spon­si­bil­ity of com­pil­ers tar­get­ing Wasm. There are no built-in ob­ject sys­tems, nor clo­sures or other higher-level con­structs — which would in­evitably be heav­ily bi­ased to­wards spe­cific lan­guages. Instead, Wasm only pro­vides the ba­sic build­ing blocks for rep­re­sent­ing such con­structs and fo­cuses purely on the mem­ory man­age­ment as­pect.

Typed ref­er­ences. The GC ex­ten­sion is built upon a sub­stan­tial ex­ten­sion to the Wasm type sys­tem, which now sup­ports much richer forms of ref­er­ences. Reference types can now de­scribe the ex­act shape of the ref­er­enced heap value, avoid­ing ad­di­tional run­time checks that would oth­er­wise be needed to en­sure safety. This more ex­pres­sive typ­ing mech­a­nism, in­clud­ing sub­typ­ing and type re­cur­sion, is also avail­able for func­tion ref­er­ences, mak­ing it pos­si­ble to per­form safe in­di­rect func­tion calls with­out any run­time type or bounds check, through the new cal­l_ref in­struc­tion.

Tail calls. Tail calls are a vari­ant of func­tion calls that im­me­di­ately exit the cur­rent func­tion, and thereby avoid tak­ing up ad­di­tional stack space. Tail calls are an im­por­tant mech­a­nism that is used in var­i­ous lan­guage im­ple­men­ta­tions both in user-vis­i­ble ways (e.g., in func­tional lan­guages) and for in­ter­nal tech­niques (e.g., to im­ple­ment stubs). Wasm tail calls are fully gen­eral and work for callees both se­lected sta­t­i­cally (by func­tion in­dex) and dy­nam­i­cally (by ref­er­ence or table).

Exception han­dling. Exceptions pro­vide a way to lo­cally abort ex­e­cu­tion, and are a com­mon fea­ture in mod­ern pro­gram­ming lan­guages. Previously, there was no ef­fi­cient way to com­pile ex­cep­tion han­dling to Wasm, and ex­ist­ing com­pil­ers typ­i­cally re­sorted to con­vo­luted ways of im­ple­ment­ing them by es­cap­ing to the host lan­guage, e.g., JavaScript. This was nei­ther portable nor ef­fi­cient. Wasm 3.0 hence pro­vides na­tive ex­cep­tion han­dling within Wasm. Exceptions are de­fined by de­clar­ing ex­cep­tion tags with as­so­ci­ated pay­load data. As one would ex­pect, an ex­cep­tion can be thrown, and se­lec­tively be caught by a sur­round­ing han­dler, based on its tag. Exception han­dlers are a new form of block in­struc­tion that in­cludes a dis­patch list of tag/​la­bel pairs or catch-all la­bels to de­fine where to jump when an ex­cep­tion oc­curs.

Relaxed vec­tor in­struc­tions. Wasm 2.0 added a large set of vec­tor (SIMD) in­struc­tions, but due to dif­fer­ences in hard­ware, some of these in­struc­tions have to do ex­tra work on some plat­forms to achieve the spec­i­fied se­man­tics. In or­der to squeeze out max­i­mum per­for­mance, Wasm 3.0 in­tro­duces “relaxed” vari­ants of these in­struc­tions that are al­lowed to have im­ple­men­ta­tion-de­pen­dent be­hav­ior in cer­tain edge cases. This be­hav­ior must be se­lected from a pre-spec­i­fied set of le­gal choices.

Deterministic pro­file. To make up for the added se­man­tic fuzzi­ness of re­laxed vec­tor in­struc­tions, and in or­der to sup­port set­tings that de­mand or need de­ter­min­is­tic ex­e­cu­tion se­man­tics (such as blockchains, or re­playable sys­tems), the Wasm stan­dard now spec­i­fies a de­ter­min­is­tic de­fault be­hav­ior for every in­struc­tion with oth­er­wise non-de­ter­min­is­tic re­sults — cur­rently, this in­cludes float­ing-point op­er­a­tors and their gen­er­ated NaN val­ues and the afore­men­tioned re­laxed vec­tor in­struc­tions. Between plat­forms choos­ing to im­ple­ment this de­ter­min­is­tic ex­e­cu­tion pro­file, Wasm thereby is fully de­ter­min­is­tic, re­pro­ducible, and portable.

Custom an­no­ta­tion syn­tax. Finally, the Wasm text for­mat has been en­riched with generic syn­tax for plac­ing an­no­ta­tions in Wasm source code. Analogous to cus­tom sec­tions in the bi­nary for­mat, these an­no­ta­tions are not as­signed any mean­ing by the Wasm stan­dard it­self, and can be cho­sen to be ig­nored by im­ple­men­ta­tions. However, they pro­vide a way to rep­re­sent the in­for­ma­tion stored in cus­tom sec­tions in hu­man-read­able and writable form, and con­crete an­no­ta­tions can be spec­i­fied by down­stream stan­dards.

In ad­di­tion to these core fea­tures, em­bed­dings of Wasm into JavaScript ben­e­fit from a new ex­ten­sion to the JS API:

JS string builtins. JavaScript string val­ues can al­ready be passed to Wasm as ex­tern­refs. Functions from this new prim­i­tive li­brary can be im­ported into a Wasm mod­ule to di­rectly ac­cess and ma­nip­u­late such ex­ter­nal string val­ues in­side Wasm.

With these new fea­tures, Wasm has much bet­ter sup­port for com­pil­ing high-level pro­gram­ming lan­guages. Enabled by this, we have seen var­i­ous new lan­guages pop­ping up to tar­get Wasm, such as Java, OCaml, Scala, Kotlin, Scheme, or Dart, all of which use the new GC fea­ture.

On top of all these good­ies, Wasm 3.0 also is the first ver­sion of the stan­dard that has been pro­duced with the new SpecTec tool chain. We be­lieve that this makes for an even more re­li­able spec­i­fi­ca­tion.

Wasm 3.0 is al­ready ship­ping in most ma­jor web browsers, and sup­port in stand-alone en­gines like Wasmtime is on track to com­ple­tion as well. The Wasm fea­ture sta­tus page tracks sup­port across en­gines.

Over the past month or so, many YouTubers have been re­port­ing ma­jor drops to their video view counts. Theories have run wild, but there’s one ex­pla­na­tion in­volv­ing ad block­ers that makes the most sense, but YouTube is­n’t con­firm­ing any­thing di­rectly.

Since mid-Au­gust, many YouTubers have no­ticed their view counts are con­sid­er­ably lower than they were be­fore, in some cases with very dras­tic drops. The rea­son for the drop, though, has been shrouded in mys­tery for many cre­ators.

The most likely ex­pla­na­tion seems to be that YouTube is not count­ing views prop­erly for users with an ad blocker en­abled, an­other step in the plat­for­m’s con­tin­ued war on ad block­ers. This was first re­al­ized by Josh Strife Hayes, who no­ticed that view counts on TV, phones, and tablets have been steady, while views on com­put­ers have dropped by around 50% since the mid-Au­gust trend started. TechLinked, a chan­nel in the Linus Tech Tips fam­ily, con­firmed sim­i­lar num­bers within its sta­tis­tics.

This aligns with one of the pos­si­ble ex­pla­na­tions that YouTube it­self hinted at in an ac­knowl­edge­ment of lower view counts.

Viewers Using Ad Blockers & Other Content Blocking Tools: Ad block­ers and other ex­ten­sions can im­pact the ac­cu­racy of re­ported view counts. Channels whose au­di­ences in­clude a higher pro­por­tion of users uti­liz­ing such tools may see more fluc­tu­a­tions in traf­fic re­lated to up­dates to these tools.

The rest of the post ad­dresses prior spec­u­la­tion that YouTube’s new AI-powered age ver­i­fi­ca­tion tools were to blame — which YouTube adamantly says is not the case — while also of­fer­ing other pos­si­ble ex­pla­na­tions such as “seasonal view­ing habits” and com­pe­ti­tion on the plat­form.

YouTube says “there is no sys­temic is­sue that is im­pact­ing cre­ators” re­gard­ing lower view counts.

This ad blocker sit­u­a­tion does seem the most likely ex­pla­na­tion, though. In a prior video, Linus Tech Tips had noted that while view counts were down, ad rev­enue was not. If com­puter views are the only ones down, it stands to rea­son that view­ers us­ing an ad blocker are not be­ing counted cor­rectly, es­pe­cially if ad rev­enue is­n’t tak­ing a hit from the lower view counts. YouTube’s hint that ad block­ers “can im­pact the ac­cu­racy of re­ported view counts” cer­tainly sug­gests this is pos­si­ble, even if it’s not firm con­fir­ma­tion.

The keynote from this week’s Blender Conference, dur­ing which Ton Roosendaal — the orig­i­nal cre­ator of the open-source 3D soft­ware — an­nounced that he is stand­ing down as Blender CEO.

Ton Roosendaal is to step down as chair­man of the Blender Foundation and Blender CEO on 1 January 2026. The news was an­nounced dur­ing to­day’s keynote at the Blender Conference.

Roosendaal — the orig­i­nal au­thor of the open-source 3D soft­ware, and its pub­lic fig­ure­head for the past three decades — will pass on his roles to cur­rent Blender COO Francesco Siddi.

Roosendaal him­self will move to the newly es­tab­lished Blender Foundation su­per­vi­sory board.

Other new Blender Foundation board po­si­tions in­clude Head of Development Sergey Sharybin, Head of Product Dalai Felinto, and Head of Operations Fiona Cohen.

Ton Roosendaal in the early 1990s. Blender orig­i­nated as in-house soft­ware at NeoGeo, the an­i­ma­tion stu­dio he co-founded.

Blender’s orig­i­nal cre­ator and prin­ci­pal pro­moter

Blender be­gan life in the mid-1990s as in-house soft­ware at NeoGeo, the Dutch an­i­ma­tion stu­dio that Roosendaal co-founded, and for which he wrote the cus­tom tools.

Although ini­tially set to be­come a com­mer­cial prod­uct, when in­vestors de­cided to pull the plug on Not a Number, the firm he had founded to de­velop Blender, Roosendaal went open-source.

A crowd­fund­ing cam­paign quickly raised the €110,000 needed to buy back the code base, and on 13 October 2002, Blender was re­leased to the world un­der a GPL li­cence.

From the start, two things dis­tin­guished Blender from most other open-source CG ap­pli­ca­tions.

First, even be­fore the first re­lease, Roosendaal es­tab­lished the non-profit Blender Foundation, so de­vel­op­ment was run through an or­ga­ni­za­tion akin to a com­mer­cial soft­ware de­vel­oper.

Second, the soft­ware was tested in some­thing close to real-world pro­duc­tion con­di­tions, on a se­ries of ‘open movies’, be­gin­ning with 2006’s Elephants Dream, and con­tin­u­ing to this day.

The early open movies — funded through a com­bi­na­tion of DVD pre-sales and arts grants — also pro­vided valu­able ex­po­sure for Blender, and il­lus­trated Roosendaal’s gifts as a pro­ducer and pro­moter: a mix­ture of pas­sion, phys­i­cal pres­ence, and sheer force of per­son­al­ity.

Taking Blender from fan fa­vorite to sta­ple of pro­fes­sional pro­duc­tion

By the mid-2010s, Blender was qui­etly be­ing adopted in ma­jor stu­dios, with Pixar con­firm­ing that it was one of a small num­ber of third-party 3D tools that it ap­proved for use in­ter­nally.

A ma­jor turn­ing point was the re­lease of Blender 2.80: the 2019 up­date that ad­dressed many of the UI and work­flow is­sues that had pre­vi­ously alien­ated artists used to com­mer­cial 3D apps.

It catal­ysed sup­port for Blender among big tech firms, with a land­mark $1.2 mil­lion grant from Epic Games fol­lowed by back­ing from com­pa­nies in­clud­ing AMD, Intel, NVIDIA and Qualcomm.

Last year, the Corporate Patrons of the Blender Development Fund ac­counted for just un­der 40% of the col­lec­tive in­come of the Blender or­ga­ni­za­tions.

The re­sult­ing €3.1 mil­lion paid the salaries of over 15 full-time de­vel­op­ers, plus a sim­i­lar num­ber of tech sup­port staff and part-time devs: a larger team than many com­mer­cial CG tools.

Roosendaal (right) in­tro­duces the new Blender lead­er­ship team: (L-R) in­com­ing CEO Francesco Siddi, Sergey Sharybin, Dalai Felinto and Fiona Cohen.

The pub­lic ‘incarnation of Blender’

During that time, the soft­ware has been closely iden­ti­fied with Roosendaal him­self, both through his role as chair­man of the Blender Foundation, and as a kind of totemic fig­ure­head.

When I in­ter­viewed him in 2019 for an ar­ti­cle in a UK tech mag­a­zine, he de­scribed him­self, quite ac­cu­rately, as “the in­car­na­tion of Blender”.

But in the same in­ter­view, he noted that “Blender is much big­ger than me” and com­mented: “Now I need to get the or­gan­i­sa­tion work­ing with­out me, and then I can move on.”

Six years later, those plans have fi­nally come to fruition, with the ap­point­ment of a new Blender lead­er­ship team.

From sin­gle fig­ure­head to four core lead­er­ship roles

Announcing his de­par­ture at the Blender Conference, Roosendaal iden­ti­fied four of his key skills that he felt had led to Blender’s suc­cess: as an or­ga­nizer, de­vel­oper, de­signer and en­tre­pre­neur.

It’s telling that rather than try­ing to ap­point one per­son to ful­fil all four roles, they have been split out to four sep­a­rate peo­ple.

Former an­i­ma­tion pro­ducer Fiona Cohen takes on the or­ga­ni­za­tional role, as the Blender Foundation’s new Head of Operations.

The de­vel­oper role will be taken by Blender Lead Engineer Sergey Sharybin in his new role as the Foundation’s Head of Development.

The de­signer role passes to an­other long-time Blender artist and de­vel­oper, Dalai Felinto, as the new Head of Product.

The crit­i­cal en­tre­pre­neur­ial role will be ful­filled by cur­rent Blender COO Francesco Siddi, who takes on Roosendaal’s job tiles of chair­man and CEO.

Siddi first be­gan work­ing with the Blender or­ga­ni­za­tion in 2012, as a VFX artist on the open movie Tears of Steel, later act­ing as a web de­vel­oper and pipeline de­vel­oper.

He be­gan work­ing as a pro­ducer on sub­se­quent open movies in 2017, and has man­aged Blender’s in­dus­try re­la­tions since 2020.

“I am very proud to have such a won­der­fully tal­ented young team around me to bring our free and open source pro­ject into the next decade,” com­mented Roosendaal.

Read the of­fi­cial an­nounce­ment that Ton Roosendaal is step­ping down as Blender CEO

Read more about Blender’s lead­er­ship struc­ture in the lat­est Blender Foundation an­nual re­port

Between August and early September, three in­fra­struc­ture bugs in­ter­mit­tently de­graded Claude’s re­sponse qual­ity. We’ve now re­solved these is­sues and want to ex­plain what hap­pened.

In early August, a num­ber of users be­gan re­port­ing de­graded re­sponses from Claude. These ini­tial re­ports were dif­fi­cult to dis­tin­guish from nor­mal vari­a­tion in user feed­back. By late August, the in­creas­ing fre­quency and per­sis­tence of these re­ports prompted us to open an in­ves­ti­ga­tion that led us to un­cover three sep­a­rate in­fra­struc­ture bugs.

To state it plainly: We never re­duce model qual­ity due to de­mand, time of day, or server load. The prob­lems our users re­ported were due to in­fra­struc­ture bugs alone.

We rec­og­nize users ex­pect con­sis­tent qual­ity from Claude, and we main­tain an ex­tremely high bar for en­sur­ing in­fra­struc­ture changes don’t af­fect model out­puts. In these re­cent in­ci­dents, we did­n’t meet that bar. The fol­low­ing post­mortem ex­plains what went wrong, why de­tec­tion and res­o­lu­tion took longer than we would have wanted, and what we’re chang­ing to pre­vent sim­i­lar fu­ture in­ci­dents.

We don’t typ­i­cally share this level of tech­ni­cal de­tail about our in­fra­struc­ture, but the scope and com­plex­ity of these is­sues jus­ti­fied a more com­pre­hen­sive ex­pla­na­tion.

We serve Claude to mil­lions of users via our first-party API, Amazon Bedrock, and Google Cloud’s Vertex AI. We de­ploy Claude across mul­ti­ple hard­ware plat­forms, namely AWS Trainium, NVIDIA GPUs, and Google TPUs. This ap­proach pro­vides the ca­pac­ity and ge­o­graphic dis­tri­b­u­tion nec­es­sary to serve users world­wide.

Each hard­ware plat­form has dif­fer­ent char­ac­ter­is­tics and re­quires spe­cific op­ti­miza­tions. Despite these vari­a­tions, we have strict equiv­a­lence stan­dards for model im­ple­men­ta­tions. Our aim is that users should get the same qual­ity re­sponses re­gard­less of which plat­form serves their re­quest. This com­plex­ity means that any in­fra­struc­ture change re­quires care­ful val­i­da­tion across all plat­forms and con­fig­u­ra­tions.

The over­lap­ping na­ture of these bugs made di­ag­no­sis par­tic­u­larly chal­leng­ing. The first bug was in­tro­duced on August 5, af­fect­ing ap­prox­i­mately 0.8% of re­quests made to Sonnet 4. Two more bugs arose from de­ploy­ments on August 25 and 26.

Although ini­tial im­pacts were lim­ited, a load bal­anc­ing change on August 29 started to in­crease af­fected traf­fic. This caused many more users to ex­pe­ri­ence is­sues while oth­ers con­tin­ued to see nor­mal per­for­mance, cre­at­ing con­fus­ing and con­tra­dic­tory re­ports.

Below we de­scribe the three bugs that caused the degra­da­tion, when they oc­curred, and how we re­solved them:

On August 5, some Sonnet 4 re­quests were mis­routed to servers con­fig­ured for the up­com­ing 1M to­ken con­text win­dow. This bug ini­tially af­fected 0.8% of re­quests. On August 29, a rou­tine load bal­anc­ing change un­in­ten­tion­ally in­creased the num­ber of short-con­text re­quests routed to the 1M con­text servers. At the worst im­pacted hour on August 31, 16% of Sonnet 4 re­quests were af­fected.

Approximately 30% of Claude Code users who made re­quests dur­ing this pe­riod had at least one mes­sage routed to the wrong server type, re­sult­ing in de­graded re­sponses. On Amazon Bedrock, mis­routed traf­fic peaked at 0.18% of all Sonnet 4 re­quests from August 12. Incorrect rout­ing af­fected less than 0.0004% of re­quests on Google Cloud’s Vertex AI be­tween August 27 and September 16.

However, some users were af­fected more se­verely, as our rout­ing is “sticky”. This meant that once a re­quest was served by the in­cor­rect server, sub­se­quent fol­low-ups were likely to be served by the same in­cor­rect server.

Resolution: We fixed the rout­ing logic to en­sure short- and long-con­text re­quests were di­rected to the cor­rect server pools. We de­ployed the fix on September 4. A roll­out to our first-party plat­forms and Google Cloud’s Vertex was com­pleted by September 16. The fix is in the process of be­ing rolled out on Bedrock.

On August 25, we de­ployed a mis­con­fig­u­ra­tion to the Claude API TPU servers that caused an er­ror dur­ing to­ken gen­er­a­tion. An is­sue caused by a run­time per­for­mance op­ti­miza­tion oc­ca­sion­ally as­signed a high prob­a­bil­ity to to­kens that should rarely be pro­duced given the con­text, for ex­am­ple pro­duc­ing Thai or Chinese char­ac­ters in re­sponse to English prompts, or pro­duc­ing ob­vi­ous syn­tax er­rors in code. A small sub­set of users that asked a ques­tion in English might have seen “สวัสดี” in the mid­dle of the re­sponse, for ex­am­ple.

This cor­rup­tion af­fected re­quests made to Opus 4.1 and Opus 4 on August 25-28, and re­quests to Sonnet 4 August 25–September 2. Third-party plat­forms were not af­fected by this is­sue.

Resolution: We iden­ti­fied the is­sue and rolled back the change on September 2. We’ve added de­tec­tion tests for un­ex­pected char­ac­ter out­puts to our de­ploy­ment process.

On August 25, we de­ployed code to im­prove how Claude se­lects to­kens dur­ing text gen­er­a­tion. This change in­ad­ver­tently trig­gered a la­tent bug in the XLA:TPU[1] com­piler, which has been con­firmed to af­fect re­quests to Claude Haiku 3.5.

We also be­lieve this could have im­pacted a sub­set of Sonnet 4 and Opus 3 on the Claude API. Third-party plat­forms were not af­fected by this is­sue.

Resolution: We first ob­served the bug af­fect­ing Haiku 3.5 and rolled it back on September 4. We later no­ticed user re­ports of prob­lems with Opus 3 that were com­pat­i­ble with this bug, and rolled it back on September 12. After ex­ten­sive in­ves­ti­ga­tion we were un­able to re­pro­duce this bug on Sonnet 4 but de­cided to also roll it back out of an abun­dance of cau­tion.

Simultaneously, we have (a) been work­ing with the XLA:TPU team on a fix for the com­piler bug and (b) rolled out a fix to use ex­act top-k with en­hanced pre­ci­sion. For de­tails, see the deep dive be­low.

To il­lus­trate the com­plex­ity of these is­sues, here’s how the XLA com­piler bug man­i­fested and why it proved par­tic­u­larly chal­leng­ing to di­ag­nose.

When Claude gen­er­ates text, it cal­cu­lates prob­a­bil­i­ties for each pos­si­ble next word, then ran­domly chooses a sam­ple from this prob­a­bil­ity dis­tri­b­u­tion. We use “top-p sam­pling” to avoid non­sen­si­cal out­puts—only con­sid­er­ing words whose cu­mu­la­tive prob­a­bil­ity reaches a thresh­old (typically 0.99 or 0.999). On TPUs, our mod­els run across mul­ti­ple chips, with prob­a­bil­ity cal­cu­la­tions hap­pen­ing in dif­fer­ent lo­ca­tions. To sort these prob­a­bil­i­ties, we need to co­or­di­nate data be­tween chips, which is com­plex.[2]

In December 2024, we dis­cov­ered our TPU im­ple­men­ta­tion would oc­ca­sion­ally drop the most prob­a­ble to­ken when tem­per­a­ture was zero. We de­ployed a workaround to fix this case.

The root cause in­volved mixed pre­ci­sion arith­metic. Our mod­els com­pute next-to­ken prob­a­bil­i­ties in bf16 (16-bit float­ing point). However, the vec­tor proces­sor is fp32-na­tive, so the TPU com­piler (XLA) can op­ti­mize run­time by con­vert­ing some op­er­a­tions to fp32 (32-bit). This op­ti­miza­tion pass is guarded by the xla_al­low_ex­cess_­pre­ci­sion flag which de­faults to true.

This caused a mis­match: op­er­a­tions that should have agreed on the high­est prob­a­bil­ity to­ken were run­ning at dif­fer­ent pre­ci­sion lev­els. The pre­ci­sion mis­match meant they did­n’t agree on which to­ken had the high­est prob­a­bil­ity. This caused the high­est prob­a­bil­ity to­ken to some­times dis­ap­pear from con­sid­er­a­tion en­tirely.

On August 26, we de­ployed a rewrite of our sam­pling code to fix the pre­ci­sion is­sues and im­prove how we han­dled prob­a­bil­i­ties at the limit that reach the top-p thresh­old. But in fix­ing these prob­lems, we ex­posed a trick­ier one.

Our fix re­moved the December workaround be­cause we be­lieved we’d solved the root cause. This led to a deeper bug in the ap­prox­i­mate top-k op­er­a­tion—a per­for­mance op­ti­miza­tion that quickly finds the high­est prob­a­bil­ity to­kens.[3] This ap­prox­i­ma­tion some­times re­turned com­pletely wrong re­sults, but only for cer­tain batch sizes and model con­fig­u­ra­tions. The December workaround had been in­ad­ver­tently mask­ing this prob­lem.

The bug’s be­hav­ior was frus­trat­ingly in­con­sis­tent. It changed de­pend­ing on un­re­lated fac­tors such as what op­er­a­tions ran be­fore or af­ter it, and whether de­bug­ging tools were en­abled. The same prompt might work per­fectly on one re­quest and fail on the next.

While in­ves­ti­gat­ing, we also dis­cov­ered that the ex­act top-k op­er­a­tion no longer had the pro­hib­i­tive per­for­mance penalty it once did. We switched from ap­prox­i­mate to ex­act top-k and stan­dard­ized some ad­di­tional op­er­a­tions on fp32 pre­ci­sion.[4] Model qual­ity is non-ne­go­tiable, so we ac­cepted the mi­nor ef­fi­ciency im­pact.

Our val­i­da­tion process or­di­nar­ily re­lies on bench­marks along­side safety eval­u­a­tions and per­for­mance met­rics. Engineering teams per­form spot checks and de­ploy to small “canary” groups first.

These is­sues ex­posed crit­i­cal gaps that we should have iden­ti­fied ear­lier. The eval­u­a­tions we ran sim­ply did­n’t cap­ture the degra­da­tion users were re­port­ing, in part be­cause Claude of­ten re­cov­ers well from iso­lated mis­takes. Our own pri­vacy prac­tices also cre­ated chal­lenges in in­ves­ti­gat­ing re­ports. Our in­ter­nal pri­vacy and se­cu­rity con­trols limit how and when en­gi­neers can ac­cess user in­ter­ac­tions with Claude, in par­tic­u­lar when those in­ter­ac­tions are not re­ported to us as feed­back. This pro­tects user pri­vacy but pre­vents en­gi­neers from ex­am­in­ing the prob­lem­atic in­ter­ac­tions needed to iden­tify or re­pro­duce bugs.

Each bug pro­duced dif­fer­ent symp­toms on dif­fer­ent plat­forms at dif­fer­ent rates. This cre­ated a con­fus­ing mix of re­ports that did­n’t point to any sin­gle cause. It looked like ran­dom, in­con­sis­tent degra­da­tion.

More fun­da­men­tally, we re­lied too heav­ily on noisy eval­u­a­tions. Although we were aware of an in­crease in re­ports on­line, we lacked a clear way to con­nect these to each of our re­cent changes. When neg­a­tive re­ports spiked on August 29, we did­n’t im­me­di­ately make the con­nec­tion to an oth­er­wise stan­dard load bal­anc­ing change.

As we con­tinue to im­prove our in­fra­struc­ture, we’re also im­prov­ing the way we eval­u­ate and pre­vent bugs like those dis­cussed above across all plat­forms where we serve Claude. Here’s what we’re chang­ing:

* More sen­si­tive eval­u­a­tions: To help dis­cover the root cause of any given is­sue, we’ve de­vel­oped eval­u­a­tions that can more re­li­ably dif­fer­en­ti­ate be­tween work­ing and bro­ken im­ple­men­ta­tions. We’ll keep im­prov­ing these eval­u­a­tions to keep a closer eye on model qual­ity.

* Quality eval­u­a­tions in more places: Although we run reg­u­lar eval­u­a­tions on our sys­tems, we will run them con­tin­u­ously on true pro­duc­tion sys­tems to catch is­sues such as the con­text win­dow load bal­anc­ing er­ror.

* Faster de­bug­ging tool­ing: We’ll de­velop in­fra­struc­ture and tool­ing to bet­ter de­bug com­mu­nity-sourced feed­back with­out sac­ri­fic­ing user pri­vacy. Additionally, some be­spoke tools de­vel­oped here will be used to re­duce the re­me­di­a­tion time in fu­ture sim­i­lar in­ci­dents, if those should oc­cur.

Evals and mon­i­tor­ing are im­por­tant. But these in­ci­dents have shown that we also need con­tin­u­ous sig­nal from users when re­sponses from Claude aren’t up to the usual stan­dard. Reports of spe­cific changes ob­served, ex­am­ples of un­ex­pected be­hav­ior en­coun­tered, and pat­terns across dif­fer­ent use cases all helped us iso­late the is­sues.

It re­mains par­tic­u­larly help­ful for users to con­tinue to send us their feed­back di­rectly. You can use the /bug com­mand in Claude Code or you can use the “thumbs down” but­ton in the Claude apps to do so. Developers and re­searchers of­ten cre­ate new and in­ter­est­ing ways to eval­u­ate model qual­ity that com­ple­ment our in­ter­nal test­ing. If you’d like to share yours, reach out to feed­back@an­thropic.com.

We re­main grate­ful to our com­mu­nity for these con­tri­bu­tions.

We have an air bike in our base­ment. If you are un­fa­mil­iar with air bikes, they are sim­i­lar to sta­tion­ary bikes with foot ped­als but also have han­dles you push and pull with your arms. It uses air re­sis­tance, so the harder you pedal and move your arms, the higher the re­sis­tance.

It’s also known as an as­sault bike. 😬

Which is apt, be­cause it’s a butt-kicker of a work­out. I use it about once a week, more fre­quently in the win­ter when it’s too cold to run, and less of­ten in the sum­mer when I can get out­side more. And I kind of hate it!

Before I even drag my­self to our base­ment, I’m al­ready dread­ing it. The only way I can con­vince my­self to do it is by find­ing a suit­ably en­gag­ing show I can dis­tract my­self with on my phone while I huff and puff.

Every time, I start my warm-up and think to my­self,

“It’s only 30 min­utes, I can do this!”

Like clock­work, within the first three min­utes, I think, “Maybe I will only do ten min­utes to­day and do some pi­lates or weights in­stead.”

After ten min­utes, I think, “OK, surely I can make it to 20 min­utes, and that will be enough”.

After 20 min­utes, as I gasp for air and sweat soaks through my shirt, I think “Well, I al­ready made it to 20 min­utes… I guess I will just fin­ish it.”

And then I pro­ceed to huff and puff to the end, wherein I walk my wob­bly legs back up the stairs to do a cooldown. At which point I think, “That su­u­u­u­u­ucked…” And then con­grat­u­late my­self on fin­ish­ing as I try to get my heart rate back to nor­mal. 🥵

This men­tal dance hap­pens, with­out fail, every sin­gle time I ride.

I share this anec­dote be­cause it il­lus­trates how tricky mo­ti­va­tion can be, es­pe­cially when faced with some­thing you don’t want to do or have been pro­cras­ti­nat­ing on. There are any num­ber of things you have to deal with in your life that you don’t want to. There are even things you might gen­er­ally en­joy that feel like they are hang­ing over you.

The pat­tern of­ten goes like this:

* Before you start, it feels daunt­ing, and the prospect lingers in the back of your mind. You know it needs to be done, but you re­ally, re­ally don’t feel like it. You leave it un­til it starts to loom larger and larger.

* When you fi­nally con­vince your­self to start, it’s not what you want to be do­ing, but it’s gen­er­ally fine. It’s of­ten not even as bad as you thought it would be, and it feels good to make progress.

* As you near the end, you can even push your­self a lit­tle to wrap it up and get it off your plate.

* When it’s over, you feel re­lieved, like a weight has been taken off your shoul­ders, and you are both pleased with your­self and a lit­tle an­noyed that it took you so long to deal with.

Motivation is a topic that comes up with nearly all my clients, as they nav­i­gate the var­i­ous com­plex­i­ties of their lives. In some ways, mo­ti­va­tion seems sim­ple. You ask your­self, “Why can’t I just make my­self be mo­ti­vated to do the thing?”, what­ever the thing might be. However, as you beat your­self up about it, con­sider that many fac­tors in­flu­ence our de­ci­sion-mak­ing and the feel­ing of be­ing mo­ti­vated.

Humans are com­plex crea­tures, with nu­mer­ous brain chem­i­cals and hor­mones in­flu­enc­ing our over­all phys­i­cal and emo­tional state, which them­selves are con­stantly im­pacted, some­times dras­ti­cally, by things like:

* Have you been sleep­ing well and enough?

* Have you been eat­ing well and the right amount for you?

* Have you been im­bib­ing in al­co­hol or other things?

* Have you been mov­ing your body reg­u­larly?

* Do you have any phys­i­cal or men­tal con­di­tions?

* Are you in pain?

* Do you have sig­nif­i­cant life stres­sors at this time?

* What time of day is it?

* Where are you in your nat­ural hor­mone cy­cles?

* How old are you?

* Have you had any con­flicts in your life re­cently?

* Did you move your body in a way en­tirely within your usual rou­tines, but ap­par­ently in a way that is no longer ac­cept­able?

* Did you sleep in a slightly dif­fer­ent po­si­tion than usual, and now your back will never be the same again?

I could go on, but you get the idea.😅

All of these fac­tors (and more) con­spire to shift your mood, phys­i­cal en­ergy, and men­tal en­ergy, of­ten mak­ing it harder to muster the mo­ti­va­tion to do things. What, then, can you do to move things in the right di­rec­tion? How do you mo­ti­vate your­self to do a thing you don’t want to do?

Here are sev­eral ways to help en­cour­age ac­tion when you feel un­mo­ti­vated.

There are many ex­ter­nal and in­ter­nal fac­tors, as listed above, that con­tribute to mo­ti­va­tion.

* When your body is­n’t feel­ing good, it’s harder to make it do things.

* When your mind is tired, dis­tracted, or over­whelmed, it’s chal­leng­ing to fo­cus and ac­com­plish tasks.

* When the thing you need to do is­n’t im­por­tant to you or some­thing you don’t like, it’s hard to make your­self do it.

When you know why you aren’t mo­ti­vated, you can think about what you could change to make things eas­ier on your­self. What fac­tors do you have con­trol over?

* Environment - Is there a place you can go or a thing you can add that will make it feel eas­ier? For ex­am­ple, I have my writ­ing desk set up in a quiet cor­ner of my bed­room (not the of­fice I share with my hus­band) to help make writ­ing eas­ier, even when I am not feel­ing it.

* Mood - Is there some­thing that will help boost your mood? Go for a ten-minute walk, treat your­self to a donut, text your best friend for a pep talk, turn on your favourite tunes… any­thing that will give you a lit­tle pick-me-up.

* Body - Are there things you can do to take care of your body to make it feel bet­ter? Try some stretch­ing, take a nap, med­i­tate, read a book, get some fresh air, go for a run, eat a com­fort meal, or do any­thing that will help your body feel less stressed.

* Negative or fear mo­ti­va­tors - Is the thing you are not mo­ti­vated to do be­ing mo­ti­vated by neg­a­tive or fear mo­ti­va­tors? These in­clude things like fear of judg­ment, fear of con­flict, shame, guilt, or oblig­a­tion. These mo­ti­va­tors only go so far and de­serve fur­ther ex­am­i­na­tion to de­ter­mine their place in your pri­or­i­ties. Maybe they aren’t things you need to do in the first place.

The key point here is to iden­tify where you have con­trol and where you don’t, and then do your best to adapt your cir­cum­stances to make it eas­ier to take ac­tion.

When you think about the var­i­ous ac­tiv­i­ties and tasks you do each day, what is it that en­cour­ages you to do them? Some of those things will be neg­a­tive mo­ti­va­tors, as I men­tioned above, but oth­ers will be things you do for fun, be­cause they are in­ter­est­ing or re­ward­ing. These are some tac­tics to con­sider for things that might help mo­ti­vate you:

You know what makes clean­ing out the garage a lot bet­ter? Some good tunes. Throw on an au­dio­book while you cook din­ner. Watch a good show while you huff and puff on the air bike! Think about the things you en­joy and con­sider how you can com­bine them with the thing you’re try­ing to mo­ti­vate your­self to do.

Sometimes it can be chal­leng­ing to push your­self to do some­thing when there are no ex­ter­nal mo­ti­va­tors. Ask a friend to be your ac­count­abil­ity buddy, or hire a pro­fes­sional to help you stay ac­count­able for the thing you’re try­ing to do, such as a coach, trainer, teacher, or di­etit­ian. I know that one of the sig­nif­i­cant value-added ben­e­fits my clients get from work­ing with me for a few months is hav­ing some­one they have to re­port back to on their progress!

Is there any way to turn the process or thing you are un­mo­ti­vated to do into a game? Can you add re­wards if you do a cer­tain amount, or set a goal for how many days you make progress in a row? For ex­am­ple, one of my mo­ti­va­tors for do­ing some kind of fit­ness every day is keep­ing up my streak! 2817 days in a row as of pub­lish­ing. 😁

Beyond small planned re­wards, hav­ing some­thing to look for­ward to as you make progress on your task or ac­tiv­ity can also help en­cour­age you to con­tinue mov­ing for­ward. Maybe you take a day off, or­der your favourite take­out, or sim­ply share it with some­one you care about.

For more specifics on types of mo­ti­va­tion, read my ar­ti­cle, What Motivates You? Learn the Types of Motivation and How to Use Them, where I get into more de­tail about in­trin­sic and ex­trin­sic mo­ti­va­tion.

If part of why you feel un­mo­ti­vated is that what you need to do feels big and over­whelm­ing, of­ten the best thing you can do is try to break it down into smaller, more man­age­able pieces. What is the small­est amount you can do to make a bit of progress?

* Commit to spend­ing 5 min­utes on it

* Choose a small cor­ner of a room you need to clean

* Write the text, even if you don’t send it

* Plan in your cal­en­dar when you will do it, so you don’t have it sit­ting in the back of your mind

* Talk about it with your part­ner or a friend

* Switch tasks to take a break and come back to it

Often, get­ting over the hump of start­ing some­thing is enough to help push you through it. Even if it is­n’t, at the very least, you have made some amount of progress, which you can build on.

If the thing you need to do is some­thing you need to do reg­u­larly, like writ­ing, fit­ness, prac­tic­ing an in­stru­ment, or clean­ing, you can’t rely purely on mo­ti­va­tion to drive you. Even for things you en­joy, it’s easy to push some­thing off “until you feel like it”. But with so many fac­tors af­fect­ing your mood and en­ergy, the times when you feel like it will be fleet­ing. Instead of re­ly­ing on mo­ti­va­tion, try to es­tab­lish a rou­tine that fos­ters con­sis­tency.

* Plan your in­ten­tional week so you have an idea of when you in­tend to do it

* Book it in your cal­en­dar

* Set a cer­tain amount of time you will put aside each day or week to chip away at it

A lit­tle bit, con­sis­tently, will go a long way.

Sometimes, when you are not feel­ing mo­ti­vated to do some­thing, it’s rea­son­able to just put it on the back burner. Maybe it’s just not a pri­or­ity right now, and that’s to­tally fine! Ask your­self, is this a glass ball or a plas­tic ball? If it’s plas­tic, set it aside for a bit and fo­cus your time and en­ergy on other things.

It’s ok to de­cide now is not the right time, but make it an in­ten­tional de­ci­sion in­stead of some­thing you avoid and feel bad about!

If you’re strug­gling with mo­ti­va­tion, you’re not alone! It’s nor­mal, it’s nat­ural, and there are tons of dif­fer­ent, ever-chang­ing fac­tors that will change how you feel. Do your best to ex­am­ine where you are at, con­trol what you can con­trol, and make progress where you can!

Need some help get­ting mo­ti­vated? Get in touch!

Hacker, red teamer, re­searcher. Likes to write in­fosec-fo­cussed Python tools. This is my per­sonal blog con­tain­ing re­search on top­ics I find in­ter­est­ing, such as (Azure) Active Directory in­ter­nals, pro­to­cols and vul­ner­a­bil­i­ties.

Looking for a se­cu­rity test or train­ing? Business con­tact via out­sider­se­cu­rity.nl

One Token to rule them all - ob­tain­ing Global Admin in every Entra ID ten­ant via Actor to­kens

While prepar­ing for my Black Hat and DEF CON talks in July of this year, I found the most im­pact­ful Entra ID vul­ner­a­bil­ity that I will prob­a­bly ever find. This vul­ner­a­bil­ity could have al­lowed me to com­pro­mise every Entra ID ten­ant in the world (except prob­a­bly those in na­tional cloud de­ploy­ments). If you are an Entra ID ad­min read­ing this, yes that means com­plete ac­cess to your ten­ant. The vul­ner­a­bil­ity con­sisted of two com­po­nents: un­doc­u­mented im­per­son­ation to­kens, called “Actor to­kens”, that Microsoft uses in their back­end for ser­vice-to-ser­vice (S2S) com­mu­ni­ca­tion. Additionally, there was a crit­i­cal flaw in the (legacy) Azure AD Graph API that failed to prop­erly val­i­date the orig­i­nat­ing ten­ant, al­low­ing these to­kens to be used for cross-ten­ant ac­cess.

Effectively this means that with a to­ken I re­quested in my lab ten­ant I could au­then­ti­cate as any user, in­clud­ing Global Admins, in any other ten­ant. Because of the na­ture of these Actor to­kens, they are not sub­ject to se­cu­rity poli­cies like Conditional Access, which means there was no set­ting that could have mit­i­gated this for spe­cific hard­ened ten­ants. Since the Azure AD Graph API is an older API for man­ag­ing the core Azure AD / Entra ID ser­vice, ac­cess to this API could have been used to make any mod­i­fi­ca­tion in the ten­ant that Global Admins can do, in­clud­ing tak­ing over or cre­at­ing new iden­ti­ties and grant­ing them any per­mis­sion in the ten­ant. With these com­pro­mised iden­ti­ties the ac­cess could also be ex­tended to Microsoft 365 and Azure.

I re­ported this vul­ner­a­bil­ity the same day to the Microsoft Security Response Center (MSRC). Microsoft fixed this vul­ner­a­bil­ity on their side within days of the re­port be­ing sub­mit­ted and has rolled out fur­ther mit­i­ga­tions that block ap­pli­ca­tions from re­quest­ing these Actor to­kens for the Azure AD Graph API. Microsoft also is­sued CVE-2025-55241 for this vul­ner­a­bil­ity.

These to­kens al­lowed full ac­cess to the Azure AD Graph API in any ten­ant. Requesting Actor to­kens does not gen­er­ate logs. Even if it did they would be gen­er­ated in my ten­ant in­stead of in the vic­tim ten­ant, which means there is no record of the ex­is­tence of these to­kens.

Furthermore, the Azure AD Graph API does not have API level log­ging. Its suc­ces­sor, the Microsoft Graph, does have this log­ging, but for the Azure AD Graph this teleme­try source is still in a very lim­ited pre­view and I’m not aware of any ten­ant that cur­rently has this avail­able. Since there is no API level log­ging, it means the fol­low­ing Entra ID data could be ac­cessed with­out any traces:

User in­for­ma­tion in­clud­ing all their per­sonal de­tails stored in Entra ID.

This in­for­ma­tion could be ac­cessed by im­per­son­at­ing a reg­u­lar user in the vic­tim ten­ant. If you want to know the full im­pact, my tool road­recon uses the same API, if you run it then every­thing you find in the GUI of the tool could have been ac­cessed and mod­i­fied by an at­tacker abus­ing this flaw.

If a Global Admin was im­per­son­ated, it would also be pos­si­ble to mod­ify any of the above ob­jects and set­tings. This would re­sult in full ten­ant com­pro­mise with ac­cess to any ser­vice that uses Entra ID for au­then­ti­ca­tion, such as SharePoint Online and Exchange Online. It would also pro­vide full ac­cess to any re­source hosted in Azure, since these re­sources are con­trolled from the ten­ant level and Global Admins can grant them­selves rights on Azure sub­scrip­tions. Modifying ob­jects in the ten­ant does (usually) re­sult in au­dit logs be­ing gen­er­ated. That means that while the­o­ret­i­cally all data in Microsoft 365 could have been com­pro­mised, do­ing any­thing other than read­ing the di­rec­tory in­for­ma­tion would leave au­dit logs that could alert de­fend­ers, though with­out knowl­edge of the spe­cific ar­ti­facts that mod­i­fi­ca­tions with these Actor to­kens gen­er­ate, it would ap­pear as if a le­git­i­mate Global Admin per­formed the ac­tions.

Based on Microsoft’s in­ter­nal teleme­try, they did not de­tect any abuse of this vul­ner­a­bil­ity. If you want to search for pos­si­ble abuse ar­ti­facts in your own en­vi­ron­ment, a KQL de­tec­tion is in­cluded at the end of this post.

Actor to­kens are to­kens that are is­sued by the “Access Control Service”. I don’t know the ex­act ori­gins of this ser­vice, but it ap­pears to be a legacy ser­vice that is used for au­then­ti­ca­tion with SharePoint ap­pli­ca­tions and also seems to be used by Microsoft in­ter­nally. I came across this ser­vice while in­ves­ti­gat­ing hy­brid Exchange se­tups. These hy­brid se­tups used to pro­vi­sion a cer­tifi­cate cre­den­tial on the Exchange Online Service Principal (SP) in the ten­ant, with which it can per­form au­then­ti­ca­tion. These hy­brid at­tacks were the topic of some talks I did this sum­mer, the slides are on the talks page. In this case the hy­brid part is not rel­e­vant, as in my lab I could also have added a cre­den­tial on the Exchange Online SP with­out the com­plete hy­brid setup. Exchange is not the only app which can do this, but since I found this in Exchange we will keep talk­ing about these to­kens in the con­text of Exchange.

Exchange will re­quest Actor to­kens when it wants to com­mu­ni­cate with other ser­vices on be­half of a user. The Actor to­ken al­lows it to “act” as an­other user in the ten­ant when talk­ing to Exchange Online, SharePoint and as it turns out the Azure AD Graph. The Actor to­ken (a JSON Web Token / JWT) looks as fol­lows when de­coded:

There are a few fields here that dif­fer from reg­u­lar Entra ID ac­cess to­kens:

The aud field con­tains the GUID of the Azure AD Graph API, as well as the URL graph.win­dows.net and the ten­ant it was is­sued to 6287f28f-4f7f-4322-9651-a8697d8fe1bc.

The ex­piry is ex­actly 24 hours af­ter the to­ken was is­sued.

The iss con­tains the GUID of the Entra ID to­ken ser­vice it­self, called “Azure ESTS Service”, and again the ten­ant GUID where it was is­sued.

The to­ken con­tains the claim trust­ed­fordel­e­ga­tion, which is True in this case, mean­ing we can use this to­ken to im­per­son­ate other iden­ti­ties. Many Microsoft apps could re­quest such to­kens. Non-Microsoft apps re­quest­ing an Actor to­ken would re­ceive a to­ken with this field set to False in­stead.

When us­ing this Actor to­ken, Exchange would em­bed this in an un­signed JWT that is then sent to the re­source provider, in this case the Azure AD graph. In the rest of the blog I call these im­per­son­ation to­kens since they are used to im­per­son­ate users.

The sip, smtp, upn fields are used when ac­cess­ing re­sources in Exchange on­line or SharePoint, but are ig­nored when talk­ing to the Azure AD Graph, which only cares about the nameid. This nameid orig­i­nates from an at­tribute of the user that is called the netId on the Azure AD Graph. You will also see it re­flected in to­kens is­sued to users, in the puid claim, which stands for Passport UID. I be­lieve these iden­ti­fiers are an ar­ti­fact from the orig­i­nal code­base which Microsoft used for its Microsoft Accounts (consumer ac­counts or MSA). They are still used in Entra ID, for ex­am­ple to map guest users to the orig­i­nal iden­tity in their home ten­ant.

As I men­tioned be­fore, these im­per­son­ation to­kens are not signed. That means that once Exchange has an Actor to­ken, it can use the one Actor to­ken to im­per­son­ate any­one against the tar­get ser­vice it was re­quested for, for 24 hours. In my per­sonal opin­ion, this whole Actor to­ken de­sign is some­thing that never should have ex­isted. It lacks al­most every se­cu­rity con­trol that you would want:

There are no logs when Actor to­kens are is­sued.

Since these ser­vices can craft the un­signed im­per­son­ation to­kens with­out talk­ing to Entra ID, there are also no logs when they are cre­ated or used.

They can­not be re­voked within their 24 hours va­lid­ity.

They com­pletely by­pass any re­stric­tions con­fig­ured in Conditional Access.

We have to rely on log­ging from the re­source provider to even know these to­kens were used in the ten­ant.

Microsoft uses these to­kens to talk to other ser­vices in their back­end, some­thing that Microsoft calls ser­vice-to-ser­vice (S2S) com­mu­ni­ca­tion. If one of these to­kens leaks, it can be used to ac­cess all the data in an en­tire ten­ant with­out any use­ful teleme­try or mit­i­ga­tion. In July of this year, Microsoft did pub­lish a blog about re­mov­ing these in­se­cure legacy prac­tices from their en­vi­ron­ment, but they do not pro­vide any trans­parency about how many ser­vices still use these to­kens.

As I was re­fin­ing my slide deck and pol­ished up my proof-of-con­cept code for re­quest­ing and gen­er­at­ing these to­kens, I tested more vari­ants of us­ing these to­kens, chang­ing var­i­ous fields to see if the to­kens still worked with the mod­i­fied in­for­ma­tion. As one of the tests I changed the ten­ant ID of the im­per­son­ation to­ken to a dif­fer­ent ten­ant in which none of my test ac­counts ex­isted. The Actor to­kens ten­ant ID was my iminy­our.cloud ten­ant, with ten­ant ID 6287f28f-4f7f-4322-9651-a8697d8fe1bc and the un­signed JWT gen­er­ated had the ten­ant ID b9f­b93c1-c0c8-4580-99f3-d1b540­cada32.

I sent this to­ken to graph.win­dows.net us­ing my CLI tool roadtx, ex­pect­ing a generic ac­cess de­nied since I had a ten­ant ID mis­match. However, I was in­stead greeted by a cu­ri­ous er­ror mes­sage:

Note that these are the ac­tual screen­shots I made dur­ing my re­search, which is why the for­mat­ting may not work as well in this blog

The er­ror mes­sage sug­gested that while my to­ken was valid, the iden­tity could not be found in the ten­ant. Somehow the API seemed to ac­cept my to­ken even with the mis­match­ing ten­ant. I quickly looked up the netId of a user that did ex­ist in the tar­get ten­ant, crafted a to­ken and the Azure AD Graph hap­pily re­turned the data I re­quested. I tested this in a few more test ten­ants I had ac­cess to, to make sure I was not crazy, but I could in­deed ac­cess data in other ten­ants, as long as I knew their ten­ant ID (which is pub­lic in­for­ma­tion) and the netId of a user in that ten­ant.

To demon­strate the vul­ner­a­bil­ity, here I am us­ing a Guest user in the tar­get ten­ant to query the netId of a Global Admin. Then I im­per­son­ate the Global Admin us­ing the same Actor to­ken, and can per­form any ac­tion in the ten­ant as that Global Admin over the Azure AD Graph.

First I craft an im­per­son­ation to­ken for a Guest user in my vic­tim ten­ant:

I use this to­ken to query the netId of a Global Admin:

Then I cre­ate an im­per­son­ation to­ken for this Global Admin (the UPN is kept the same since it is not val­i­dated by the API):

And fi­nally this to­ken is used to ac­cess the ten­ant as the Global Admin, list­ing the users, some­thing the guest user was not able to do:

I can even run road­recon with this im­per­son­ation to­ken, which queries all Azure AD Graph API end­points to enu­mer­ate the avail­able in­for­ma­tion in the ten­ant.

None of these ac­tions would gen­er­ate any logs in the vic­tim ten­ant.

With this vul­ner­a­bil­ity it would be pos­si­ble to com­pro­mise any Entra ID ten­ant. Starting with an Actor to­ken from an at­tacker con­trolled ten­ant, the fol­low­ing steps would lead to full con­trol over the vic­tim ten­ant:

Find the ten­ant ID for the vic­tim ten­ant, this can be done us­ing pub­lic APIs based on the do­main name.

Find a valid netId of a reg­u­lar user in the ten­ant. Methods for this will be dis­cussed be­low.

Craft an im­per­son­ation to­ken with the Actor to­ken from the at­tacker ten­ant, us­ing the ten­ant ID and netId of the user in the vic­tim ten­ant.

List all Global Admins in the ten­ant and their netId.

Craft an im­per­son­ation to­ken for the Global Admin ac­count.

Perform any read or write ac­tion over the Azure AD Graph API.

If an at­tacker makes any mod­i­fi­ca­tions in the ten­ant in step 6, that would be the only event in this chain that gen­er­ates any teleme­try in the vic­tim ten­ant. An at­tacker could for ex­am­ple cre­ate new user ac­counts, grant these Global Admin priv­i­leges and then sign in in­ter­ac­tively to any Entra ID, Microsoft 365 or third party ap­pli­ca­tion that in­te­grates with the vic­tim ten­ant. Alternatively they could add cre­den­tials on ex­ist­ing ap­pli­ca­tions, grant these apps API per­mis­sions and use that to ex­fil­trate emails or files from Microsoft 365, a tech­nique that is pop­u­lar among threat ac­tors. An at­tacker could also add cre­den­tials to Microsoft Service Principals in the vic­tim ten­ant, sev­eral of which can re­quest Actor to­kens that al­low im­per­son­ation against SharePoint or Exchange. For my DEF CON and Black Hat talks I made a demo video about us­ing these Actor to­kens to ob­tain Global Admin ac­cess. The video uses Actor to­kens within a ten­ant, but the same tech­nique could have been ap­plied to any other ten­ant by abus­ing this vul­ner­a­bil­ity.

Since ten­ant IDs can be re­solved when the do­main name of a ten­ant is known, the only iden­ti­fier that is not im­me­di­ately avail­able to the at­tacker is a valid netId for a user in that spe­cific ten­ant. As I men­tioned above, these IDs are added to Entra ID ac­cess to­kens as the puid claim. Any to­ken found on­line, in screen­shots, ex­am­ples or logs, even those that are long ex­pired or with an ob­fus­cated sig­na­ture, would pro­vide an at­tacker with enough in­for­ma­tion to breach the ten­ant. Threat ac­tors that still have old to­kens for any ten­ant from pre­vi­ous breaches can im­me­di­ately ac­cess those ten­ants again as long as the vic­tim ac­count still ex­ists.

The above is prob­a­bly not a very com­mon oc­cur­rence. What is a more re­al­is­tic at­tack is sim­ply brute-forc­ing the netId. Unlike ob­ject IDs, which are ran­domly gen­er­ated, netIds are ac­tu­ally in­cre­men­tal. Looking at the dif­fer­ences in netIds be­tween my ten­ant and those of some ten­ants I an­a­lyzed, I found the dif­fer­ence be­tween a newly cre­ated user in my ten­ant and their newest user to be in the range of 100.000 to 100 mil­lion. Simply brute forc­ing the netId could be ac­com­plished in min­utes to hours for any tar­get ten­ant, and the more user ex­ist in a ten­ant the eas­ier it is to find a match. Since this does not gen­er­ate any logs it is­n’t a noisy at­tack ei­ther. Because of the pos­si­bil­ity to brute force these netIds I would say this vul­ner­a­bil­ity could have been used to take over any ten­ant with­out any pre­req­ui­sites. There is how­ever a third tech­nique which is even more ef­fec­tive (and more fun from a tech­ni­cal level).

I pre­vi­ously men­tioned that a users netId is used to es­tab­lish links be­tween a user ac­count in mul­ti­ple ten­ants. This is some­thing that I re­searched a few years ago when I gave a talk at Black Hat USA 22 about ex­ter­nal iden­ti­ties. The be­low screen­shot is taken from one of my slides, which il­lus­trates this:

The way this works is as fol­lows. Suppose we have ten­ant A and ten­ant B. A user in ten­ant B is in­vited into ten­ant A. In the new guest ac­count that is cre­ated in ten­ant A, their netId is stored on the al­ter­na­tiveSe­cu­ri­tyIds at­tribute. That means that an at­tacker want­ing to abuse this bug can sim­ply read that at­tribute in ten­ant A, put it in an im­per­son­ation to­ken for ten­ant B and then im­per­son­ate the vic­tim in their home ten­ant. It should be noted that this works against the di­rec­tion of in­vite. Any user in any ten­ant where you ac­cept an in­vite will be able to read your netId, and with this bug could have im­per­son­ated you in your home ten­ant. In your home ten­ant you have a full user ac­count, which can enu­mer­ate other users. This is not a bug or risk with B2B trusts, but is sim­ply an un­in­tended con­se­quence of the B2B de­sign mech­a­nism. A guest ac­count in some­one else’s ten­ant would also be suf­fi­cient with the de­fault Entra ID guest set­tings be­cause the de­fault set­tings al­low users to query the netId of a user as long as the UPN is known.

To abuse this, a threat ac­tor could per­form the fol­low­ing steps, given that they have ac­cess to at least one ten­ant with a guest user:

Query the guest users and their al­ter­na­tiveSe­cu­ri­tyIds at­tribute which gives the netId.

Query the ten­ant ID of the guest users home ten­ant based on the do­main name in their UPN.

Create an im­per­son­ation to­ken, im­per­son­at­ing the vic­tim in their home ten­ant.

Optionally list Global Admins and im­per­son­ate those to com­pro­mise the en­tire ten­ant.

Repeat step 1 for each ten­ant that was com­pro­mised.

The steps above can be done in 2 API calls per ten­ant, which do not gen­er­ate any logs. Most ten­ants will have guest users from mul­ti­ple dis­tinct other ten­ants. This means the num­ber of ten­ants you com­pro­mise with this scales ex­po­nen­tially and the in­for­ma­tion needed to com­pro­mise the ma­jor­ity of all ten­ants world­wide could have been gath­ered within min­utes us­ing a sin­gle Actor to­ken. After at least 1 user is known per vic­tim ten­ant, the at­tacker can se­lec­tively per­form post-com­pro­mise ac­tions in these ten­ants by im­per­son­at­ing Global Admins.

Looking at the list of guest users in the ten­ants of some of my clients, this tech­nique would be ex­tremely pow­er­ful. I also ob­served that one of the first ten­ants you will likely com­pro­mise is Microsoft’s own ten­ant, since Microsoft con­sul­tants of­ten get in­vited to cus­tomer ten­ants. Many MSPs and Microsoft Partners will have a guest ac­count in the Microsoft ten­ant, so from the Microsoft ten­ant a com­pro­mise of most ma­jor ser­vice provider ten­ants is one step away.

Needless to say, as much as I would have liked to test this tech­nique in prac­tice to see how fast this would spread out, I only tested the in­di­vid­ual steps in my own ten­ants and did not ac­cess any data I’m not au­tho­rized to.

While query­ing data over the Azure AD Graph does not leave any logs, mod­i­fy­ing data does (usually) gen­er­ate au­dit logs. If mod­i­fi­ca­tions are done with Actor to­kens, these logs look a bit cu­ri­ous.

Since Actor to­kens in­volve both the app and the user be­ing im­per­son­ated, it seems Entra ID gets con­fused about who ac­tu­ally made the change, and it will log the UPN of the im­per­son­ated Global Admin, but the dis­play name of Exchange. Luckily for de­fend­ers this cre­ates a nice give­away when Actor to­kens are used in the ten­ant. After some test­ing and fil­ter­ing with some fel­low re­searchers that work on the blue side (thanks to Fabian Bader and Olaf Hartong) we came up with the fol­low­ing de­tec­tion query:

AuditLogs

| where not(Op­er­a­tionName has “group”)

| where not(Op­er­a­tionName == “Set di­rec­tory fea­ture on ten­ant”)

| where InitiatedBy has “user”

| where InitiatedBy.user.displayName has_any ( “Office 365 Exchange Online”, “Skype for Business Online”, “Dataverse”, “Office 365 SharePoint Online”, “Microsoft Dynamics ERP”)

The ex­clu­sion for group op­er­a­tions is there be­cause some of these prod­ucts do ac­tu­ally use Actor to­kens to per­form op­er­a­tions on your be­half. For ex­am­ple cre­at­ing spe­cific groups via the Exchange Online PowerShell mod­ule will make Exchange use an Actor to­ken on your be­half and cre­ate the group in Entra ID.

This blog dis­cussed a crit­i­cal to­ken val­i­da­tion fail­ure in the Azure AD Graph API. While the vul­ner­a­bil­ity it­self was a bad over­sight in the to­ken han­dling, the whole con­cept of Actor to­kens is a pro­to­col that was de­signed to be­have with all the prop­er­ties men­tioned in the para­graphs above. If it weren’t for the com­plete lack of se­cu­rity mea­sures in these to­kens, I don’t think such a big im­pact with such lim­ited teleme­try would have been pos­si­ble.

Thanks to the peo­ple at MSRC who im­me­di­ately picked up the vul­ner­a­bil­ity re­port, searched for po­ten­tial vari­ants in other re­sources, and to the en­gi­neers who fol­lowed up with fixes for the Azure AD Graph and blocked Actor to­kens for the Azure AD Graph API re­quested with cre­den­tials stored on Service Principals, es­sen­tially re­strict­ing the us­age of these Actor to­kens to only Microsoft in­ter­nal ser­vices.

July 15, 2025 - re­ported fur­ther de­tails on the im­pact.

July 15, 2025 - MSRC re­quested to halt fur­ther test­ing of this vul­ner­a­bil­ity.

July 17, 2025 - Microsoft pushed a fix for the is­sue glob­ally into pro­duc­tion.

August 6, 2025 - Further mit­i­ga­tions pushed out pre­vent­ing Actor to­kens be­ing is­sued for the Azure AD Graph with SP cre­den­tials.