At the end of last year, our up­time was pretty shaky. You can see this trend on our sta­tus page, and that in­sta­bil­ity con­tin­ued into the new year. Many of these out­ages were caused by a sin­gle bug, deep in SQLite. It took months of in­tense foren­sics to track it down.

Now we’re in sum­mer, we’re con­fi­dent that we’ve found the bug, that we un­der­stand it—and more im­por­tantly, that we’ve fixed it.

We know our cus­tomers ex­pect Tailscale to be a re­li­able ser­vice, and for sev­eral months we did­n’t live up to that promise. That’s dis­rup­tive, and we’re sorry. We’re pub­lish­ing this blog post to ex­plain what went wrong, how we re­sponded, and how we ul­ti­mately helped to un­cover a long-stand­ing bug in the heart of the SQLite data­base.

Tailscale’s data­base ar­chi­tec­ture

While our clients in­ter­act with our con­trol plane as a sin­gle pub­lic end­point (controlplane.tailscale.com), in­ter­nally, our con­trol plane is split into a se­ries of co­or­di­na­tion servers (or “shards”). Each tail­net lives on one in­ter­nal shard at a time, but can mi­grate seam­lessly from one to an­other. These shards are an in­ter­nal im­ple­men­ta­tion de­tail: you don’t know what shard your tail­net is on, and you never need to.

Each shard has an SQLite data­base that holds all the in­for­ma­tion about the tail­nets on that shard. A sin­gle Go process ex­clu­sively ac­cesses that data­base, and serves the con­trol plane for those tail­nets. This sin­gle-writer de­sign is ex­actly how SQLite is meant to be used.

We’ve used SQLite as our pri­mary data­base since 2022, and we chose it be­cause it’s well-known, re­li­able, and widely used. SQLite is “boring tech­nol­ogy”—in a good way. Many com­pa­nies use SQLite in much larger de­ploy­ments with­out is­sue, and we ex­pected the same stress-free us­age.

In our cur­rent backup pipeline, we take a com­plete snap­shot of the data­base every few min­utes, then up­load the en­tire SQLite file to an S3 bucket. We’d been run­ning this setup with­out in­ci­dent since early 2023.

Fast for­ward to August last year, when a data pipeline that reads those S3 back­ups re­ported an er­ror in one of our data­bases. We ran SQLite’s PRAGMA in­tegri­ty_check com­mand against the backup, and found it was in­deed cor­rupted. SQLite cor­rup­tion is pos­si­ble, but it’s highly un­usual and not some­thing you should en­counter in nor­mal op­er­a­tion. We re­paired the af­fected data­base, and in­ves­ti­gated the cause, but to no avail.

When op­er­at­ing at scale, even rare events can oc­cur with some fre­quency, so we should have been un­sur­prised when it hap­pened again—and again, and again, and again. In to­tal, we faced 19 sep­a­rate in­stances of data­base cor­rup­tion over six months be­fore we fi­nally re­solved the un­der­ly­ing bug.

When you hear the phrase “database cor­rup­tion”, it’s nat­ural to worry about data loss. Because our con­trol plane only han­dles con­fig­u­ra­tion data, these data­bases con­tain meta­data about your tail­net and de­vices, but never your pri­vate en­cryp­tion keys or net­work traf­fic. In the ear­li­est in­ci­dents, the re­cov­ery process meant a hand­ful of newly added de­vices or con­fig­u­ra­tion changes did­n’t per­sist, and a small amount of meta­data had to be re-en­tered.

Whenever cor­rup­tion oc­curred, we had to stop the con­trol plane process on the shard while we re­paired or re­stored the data­base. This was painful for tail­nets on that shard, be­cause their en­tire con­trol plane dis­ap­peared dur­ing that re­cov­ery win­dow. In the early in­ci­dents, that down­time was over an hour, but we grad­u­ally sped up the re­cov­ery process over sub­se­quent in­ci­dents.

Each tail­net is a mesh net­work, where de­vices make peer-to-peer WireGuard® con­nec­tions to each other. When a de­vice joins the tail­net, it has to get a list of other de­vices from the con­trol plane be­fore it can es­tab­lish new con­nec­tions—so if a de­vice came on­line dur­ing the SQLite down­time, it could­n’t con­nect. While the data­base was be­ing re­paired, de­vices al­ready on­line re­mained con­nected to each other, but they could­n’t learn about changes to the net­work. Those tail­nets also tem­porar­ily lost ac­cess to the web-based ad­min con­sole and the Tailscale API.

There’s also a broader im­pact on trust. We post a global in­ci­dent on our sta­tus page even when only a small num­ber of tail­nets are af­fected. Many peo­ple saw a sta­tus page event for an in­ci­dent that did­n’t af­fect them. Indeed, the ma­jor­ity of shards and tail­nets were never in­volved in a data­base cor­rup­tion in­ci­dent! Nonetheless, re­peated down­time erodes trust, whether or not you’re di­rectly af­fected.

From the very first in­stance of cor­rup­tion, we knew this was a se­ri­ous threat to our re­li­a­bil­ity, and we threw a lot of en­gi­neer­ing time at the prob­lem—but the fix was­n’t easy.

Trying to find the fault

This bug re­sisted all our ini­tial at­tempts to find it.

We looked at re­cent changes, but there weren’t any that seemed rel­e­vant. Nobody had been work­ing on our low-level code that in­ter­acts with SQLite, be­cause it had all been writ­ten years ago and pre­sented no is­sues up un­til that point. We re-re­viewed all of that code with a fine-toothed comb to look for pre­vi­ously missed bugs, but we did­n’t find any­thing that would cause the cor­rup­tion we were see­ing.

We looked for com­mon fac­tors be­tween cor­rup­tion in­ci­dents, but we could­n’t find any. It was­n’t tied to a sin­gle shard, or cus­tomer, or tail­net fea­ture, or time of day, or load level. We were at a loss for what might be trig­ger­ing the be­hav­iour.

This lack of re­li­able trig­ger con­di­tions meant we could­n’t re­pro­duce the bug syn­thet­i­cally. Instead, we had to rely on de­ploy­ing pas­sive, foren­sic teleme­try in our live en­vi­ron­ment to catch the cor­rup­tion red-handed. Gathering live di­ag­nos­tics for a data­base is­sue is the last thing we wanted to do, but we had no choice.

As an ad­di­tional com­pli­ca­tion, the cor­rup­tion did­n’t oc­cur on a reg­u­lar sched­ule. Sometimes in­ci­dents would be hours apart, other times weeks. This made it dif­fi­cult to pre­dict progress or plan fur­ther work, be­cause we were never sure when we’d get our next di­ag­nos­tic dump. We had a six-week pe­riod be­tween October and December when there were no cor­rup­tion in­ci­dents, be­fore they re­turned as an un­wel­come Christmas pre­sent.

Because this would­n’t be a quick or easy fix, we reached out to the SQLite de­vel­op­ers for a pro­fes­sional sup­port con­tract. This was a great de­ci­sion. It gave us di­rect ac­cess to their deep ex­per­tise and ex­pe­ri­ence, and we had many de­tailed tech­ni­cal con­ver­sa­tions about our ar­chi­tec­ture and our in­ci­dents.

Between Tailscale en­gi­neer­ing and the SQLite core de­vel­op­ers, we mapped out sev­eral the­o­ries for what might be caus­ing the cor­rup­tion—in­clud­ing bro­ken POSIX locks on close(), mis­man­ag­ing mem­ory owned by SQLite, or ac­ci­den­tally us­ing SQLite from mul­ti­ple threads while dis­abling thread safety. After every in­ci­dent, we gath­ered more data, added more di­ag­nos­tics, and sys­tem­at­i­cally ruled out these the­o­ries. We were grad­u­ally con­verg­ing on the true bug.

The trans­ac­tions that did­n’t bark

While we were in­ves­ti­gat­ing the root cause, we still had a live plat­form to run. We took ag­gres­sive steps to au­to­mate re­cov­ery and min­i­mize down­time:

Configuring our con­trol plane shards to hard-stop im­me­di­ately upon en­coun­ter­ing cor­rup­tion

Deploying an au­to­mated backup mon­i­tor that con­tin­u­ously ran PRAGMA in­tegri­ty_check over our back­ups

Improving our run­books and on-call train­ing

These ef­forts cut our re­sponse time to un­der an hour—and then we dis­cov­ered an un­ex­pected clue.

We wanted a way to re­store ser­vice that did­n’t in­volve rolling back to the last known-good backup (which would lose a lot of data) or re­pair­ing the known-cor­rupted data­base (which was po­ten­tially risky).

To do this, we built a trans­ac­tion log­ging pipeline. We streamed every SQL state­ment that mod­i­fied the data­base to a sep­a­rate log file. Because SQLite is a sin­gle-writer data­base with se­ri­al­is­able trans­ac­tions, our trans­ac­tion his­tory was com­pletely lin­ear and de­ter­min­is­tic. (This would­n’t be true in a multi-writer data­base like Postgres or MySQL.) Replaying those trans­ac­tions against the lat­est known-good backup should re­store the data­base to its most re­cent state, safely by­pass­ing the cor­rup­tion.

This pipeline worked, but then it did some­thing even bet­ter: it gave us a clue.

In two in­ci­dents, our trans­ac­tion logs failed to re­play cleanly. Upon closer in­spec­tion, we dis­cov­ered that data writ­ten and com­mit­ted by one trans­ac­tion was in­ex­plic­a­bly in­vis­i­ble to later trans­ac­tions. A write had van­ished into thin air with­out rais­ing an er­ror. That should be im­pos­si­ble!

The writ­ing on the WAL

As these in­ci­dents were on­go­ing, the SQLite de­vel­op­ers had been de­vel­op­ing a new de­bug­ging tool. For a while, we’d sus­pected that the bug was some­where in the check­point process. They were build­ing a new tool to give bet­ter vis­i­bil­ity into what was hap­pen­ing dur­ing check­points.

To un­der­stand what this tool found, we need to briefly ex­plain how SQLite check­points work.

A SQLite data­base is made of a se­ries of “pages”, tiny blocks of in­for­ma­tion. When you up­date the data­base, some of those pages need to be re­placed with new pages with the up­dated in­for­ma­tion.

For bet­ter per­for­mance and greater con­cur­rency, we run SQLite with Write-Ahead Logging, which means new pages aren’t writ­ten di­rectly to the data­base file. Instead, they’re writ­ten to the “write-ahead log” or “WAL file”.

New pages can’t be writ­ten to the WAL file in­def­i­nitely; at some point they have to be copied back to the main data­base file. This process is called “checkpointing”.

In most de­ploy­ments, SQLite it­self de­cides when to do a check­point, and the process is in­vis­i­ble to the end user and de­vel­oper. In our con­trol plane, we take man­ual con­trol of the check­point process so we can run fast and con­sis­tent back­ups. This non-stan­dard ap­proach seemed sus­pi­cious as we steadily elim­i­nated po­ten­tial causes.

One clue was that dur­ing cor­rup­tion in­ci­dents, our met­rics showed that SQLite would re­port copy­ing more pages from the WAL file than were ac­tu­ally avail­able. If there are 10 pages in the WAL file and 20 pages get copied to the data­base, some­thing is clearly wrong.

To un­der­stand what was hap­pen­ing dur­ing these faulty check­points, the SQLite de­vel­op­ers cre­ated a new de­bug­ging tool for the vir­tual filesys­tem layer.

SQLite is split into sev­eral lay­ers. The top layer is the parser and code gen­er­a­tor, which con­verts SQL state­ments into SQLite’s in­ter­nal data struc­tures. These data struc­tures get passed to the pager, which splits them into the in­di­vid­ual pages to be writ­ten to disk. Actually writ­ing them to disk is han­dled by the OS in­ter­face, or “virtual filesys­tem”. Currently SQLite has two main­stream vir­tual filesys­tem im­ple­men­ta­tions—Unix and Windows.

If you’re in­ter­ested in a deeper dive on these in­ter­nals, I rec­om­mend this lec­ture by Richard Hipp, the pri­mary au­thor of SQLite.

This ap­proach al­lows you to re­place dif­fer­ent lay­ers with dif­fer­ent im­ple­men­ta­tions, or wrap an ex­ist­ing layer to get more in­for­ma­tion. To help di­ag­nose our prob­lem, the SQLite de­vel­op­ers cre­ated a wrap­per around the vir­tual filesys­tem that writes ad­di­tional trac­ing in­for­ma­tion and logs about changes to the data­base. This wrap­per is called the tm­stm­pvfs shim, and the source code is avail­able in the SQLite pub­lic repos­i­tory.

We de­ployed the shim into our live en­vi­ron­ment, and waited for the next cor­rup­tion to oc­cur. Fortunately, we did­n’t have to wait long.

The WAL-Reset bug

After our next cor­rup­tion in­ci­dent, the ad­di­tional logs from the new tm­stm­pvfs shim al­lowed the SQLite de­vel­op­ers to find and fix the bug: a rare data race in the SQLite source code be­tween a check­point and a write trans­ac­tion.

In par­tic­u­lar, if a write oc­curs at a spe­cific time dur­ing a check­point, the check­point­ing process gets con­fused—it thinks some of the pages have been copied from the WAL into the main data­base file, but they haven’t. Those pages never get writ­ten to the data­base file, and that data is per­ma­nently lost. The data­base file be­comes cor­rupt, be­cause other pages which ref­er­ence those pages—such as an in­dex—are writ­ten to the data­base.

The SQLite de­vel­op­ers named this the “WAL-Reset bug”, and they es­ti­mate it was pre­sent in SQLite for at least 16 years. It could ex­ist that long be­cause it was rare—so rare, the SQLite de­vel­op­ers had to add code to de­lib­er­ately trig­ger it in their test­ing en­vi­ron­ments. Their fix adds an ad­di­tional check to the check­point­ing func­tion which de­tects when the WAL has been re­set by an­other thread.

They con­firmed that this bug caused all of the baf­fling be­hav­iour we’d seen. It ex­plained the cor­rup­tion, the trans­ac­tion logs that would­n’t ap­ply cleanly, and the in­con­sis­tent check­point sta­tis­tics. They also ex­plained why we were more likely to hit the bug than other SQLite users: we take man­ual con­trol of the check­point­ing process, and we check­point very ag­gres­sively. Even a bug trig­gered by a rare con­di­tion was bound to hit us even­tu­ally.

This was an ex­cit­ing mo­ment. After months of con­fu­sion and un­cer­tainty, we fi­nally had a plau­si­ble the­ory for why the cor­rup­tion was oc­cur­ring, and a fix we could de­ploy to pre­vent it.

The SQLite de­vel­op­ers re­leased the fix as SQLite 3.52.0, and we pre­pared to de­ploy it as soon as it was avail­able.

Fixed, with a false alarm

We rolled out SQLite 3.52.0 care­fully—first to a few ca­nary shards, then, when we saw it run­ning smoothly, we de­ployed it to the rest of the con­trol plane.

Our backup mon­i­tor promptly turned red, and re­ported cor­rup­tion in 13 dif­fer­ent data­bases. This was ex­tremely alarm­ing, but we fol­lowed our re­cov­ery pro­ce­dures to fix all the sup­posed cor­rup­tion, and every­thing was happy. It turned out these data­bases had not suf­fered real cor­rup­tion, but were sub­ject to a sec­ond prob­lem in the ver­sion of SQLite.

We shared our er­rors with the SQLite de­vel­op­ers, which un­cov­ered a bug in SQLite re­lated to stale ex­pres­sion in­dexes. If you cre­ate an in­dex on a com­puted value, and then the com­pu­ta­tion changes, the in­dex will con­tain mis­matched val­ues, which gets re­ported as cor­rup­tion by PRAGMA in­tegri­ty_check.

In our case, we were stor­ing some high-pre­ci­sion time­stamps as text, con­vert­ing them to a float­ing-point num­ber in a VIRTUAL gen­er­ated col­umn, and the SQLite 3.52.0 re­lease that fixed our data race also made an op­ti­mi­sa­tion that sub­tly changed the round­ing be­hav­iour for text-to-float­ing-point con­ver­sions. Our ca­nary shards did­n’t have any time­stamps that trig­gered the changed round­ing be­hav­iour, so we missed this in our phased roll­out.

Because this change caused false cor­rup­tion warn­ings, the SQLite de­vel­op­ers with­drew the 3.52.0 re­lease and in­stead pub­lished 3.51.3, which only con­tained a fix for the WAL-Reset bug.

We fixed the is­sue on our side by re­duc­ing the pre­ci­sion of our time­stamps to in­te­ger sec­onds; text-to-in­te­ger con­ver­sions are un­am­bigu­ous. Meanwhile, the SQLite de­vel­op­ers cre­ated an au­to­mated, self-heal­ing in­dex fea­ture in 3.53.0, which pre­vents the stale ex­pres­sion in­dex prob­lem.

Party time!

With the fix rolled out to our en­tire con­trol plane, we were ready to de­clare vic­tory, but we were still cau­tious. An ab­sence of cor­rup­tion in­ci­dents does­n’t mean things are fixed—we’d al­ready had one six-week pe­riod of de­cep­tive calm.

We wanted pos­i­tive proof that this data race was ac­tively oc­cur­ring in our pro­duc­tion en­vi­ron­ment. Now that we un­der­stood the cause of the bug—a col­li­sion be­tween a write trans­ac­tion and a WAL-reset—we patched our SQLite dri­ver to log a warn­ing when these two op­er­a­tions over­lap. If the warn­ing fired but the data­base re­mained un­cor­rupted, we’d know the fix had saved us from a po­ten­tial cor­rup­tion in­ci­dent.

We de­ployed the warn­ing, and we waited. And we waited. And waited. And waited. As weeks slipped by, we be­gan to won­der why we did­n’t see it. Was the warn­ing bro­ken? Was our the­ory wrong? Was the true bug still lurk­ing in the dark­ness?

Then, two months later, the alert we were wait­ing for fi­nally fired:

This alert proved that the pre­cise con­di­tions for the WAL-Reset bug do oc­cur in our pro­duc­tion en­vi­ron­ment, which means it was the likely cul­prit for our six months of shaky up­time.

Since that weirdly joy­ous alert fired, we’ve run for an­other four months with­out any data­base in­ci­dents, as of this writ­ing. Finally, we could breathe a sigh of re­lief.

Off the well-trod­den path

Nobody wanted us to spend six months look­ing for bugs in SQLite. This was an im­mensely frus­trat­ing ex­pe­ri­ence for both our cus­tomers and staff, and we’re all glad to put this in­sta­bil­ity be­hind us.

This in­ves­ti­ga­tion is a use­ful re­minder: run­ning bor­ing tech­nol­ogy in a non-stan­dard way is a risk. The com­mon paths and stan­dard con­fig­u­ra­tions are in­cred­i­bly well-tested and re­li­able. Most peo­ple use SQLite in a stan­dard con­fig­u­ra­tion and never face this sort of is­sue. Everything we were do­ing was a pub­lic, doc­u­mented, sup­ported con­fig­u­ra­tion—but by tak­ing man­ual con­trol of the check­point­ing process and run­ning at our own ag­gres­sive pace, we stepped off the well-trod­den op­er­a­tional path.

Resolving these in­ci­dents was a mas­sive, cross-func­tional ef­fort in­volv­ing dozens of peo­ple—in­clud­ing Tailscale’s en­gi­neer­ing and sup­port teams, and the core main­tain­ers of SQLite. It is to all of their credit that the im­pact of these in­ci­dents was not much worse.

We know that re­peated down­time erodes trust, no mat­ter how many peo­ple are af­fected, and we’re grate­ful to our cus­tomers for their pa­tience and sup­port while we chased this down.

Frustrating as this pe­riod was, we’re left in a stronger po­si­tion than we were be­fore. The long-stand­ing bug in SQLite has been patched, and we fixed dozens of other in­ci­den­tal is­sues that we spot­ted while look­ing for it. We funded the open-source SQLite VFS shim that helped iso­late the race con­di­tion al­most im­me­di­ately, and will help track down sim­i­lar bugs in the fu­ture. Finally, we’ve re­fined our data­base backup and re­cov­ery processes, and live-tested them over a dozen times.

Hopefully there won’t be an­other data­base in­ci­dent like this—but if there is, we’ll be ready.

Skip to con­tent

It’s 2020. You’re the most se­nior per­son on your team, in charge of code qual­ity and ar­chi­tec­ture. You’ve set up good en­gi­neer­ing prac­tices, you thor­oughly re­view PRs from peo­ple who are less ex­pe­ri­enced than you and work hard to main­tain a healthy code­base.

Then at some point, you go on hol­i­day. When you come back, the code­base is a mess. Everyone merged each oth­er’s PRs with­out re­ally pay­ing much at­ten­tion, some­one added a bunch of new ta­bles to the data­base to de­nor­malise it be­cause it was eas­ier and they added server­less or Kafka to the stack with­out any solid ev­i­dence that they needed ei­ther.

It’s okay. You can fix this.

Fast for­ward to 2026. You haven’t been on hol­i­day. It’s just a nor­mal Monday morn­ing. You make your­self a nice cof­fee, open your com­puter and find your­self with 7 PRs to re­view. You open the first one: +24506 – 3938 lines, ac­com­pa­nied by some AI-generated de­scrip­tion of what they’re sup­posed to do. Somehow, your team has made more changes since Friday than they used to make while you were away for a few weeks.

AI re­moved the speed limit

AI makes pro­jects with weak en­gi­neer­ing cul­ture fail much faster.

There used to be a time when peo­ple sat down and talked about how they’d do some­thing. Now they can just prompt an agent for a few hours and open a PR.

The most tragic as­pect of this way of work­ing is that, to the un­trained eye, it works.

If you pull the branch and test it, you’ll prob­a­bly get some­thing some­what func­tional. So what do they do? They keep go­ing. Again and again. Until the pro­ject reaches a point where no one knows how any­thing works.

Just like some­one buy­ing a new lux­ury car on a credit card. You don’t see the debt. You just see the car that looks great.

But then users start to re­port a weird bug. It’s the 4th time your team has been try­ing to fix it. I mean… ask­ing AI to fix it. Unfortunately, it seems like not even Fable can fig­ure it out.

You go talk to the per­son who worked on this fea­ture.

“So where does the data come from?”

“Hmm… ac­tu­ally I don’t know. Let me ask Claude.”

You sit next to each other watch­ing an end­less wall of text ap­pear on the screen. Neither of you has any idea whether any of it is true but Claude seems very con­fi­dent.

“Let’s just turn on ul­tra­code and ask it to dou­ble-check?”

This one will take a while. You start talk­ing about the lat­est drama on X.

You fi­nally get an an­swer back.

“Does this make any sense to you?”

“I’m not sure.”

“Didn’t you build this like… last week?”

Silence.

This pro­ject has be­come so con­vo­luted, with so many lay­ers and ser­vices, that no one on your team could pos­si­bly start to un­der­stand what’s go­ing on.

So, what do you do?

Fixing it would re­quire such a colos­sal amount of work that it would be im­pos­si­ble to even start jus­ti­fy­ing it to any­one in man­age­ment.

And what are you even think­ing about? It would end up in the ex­act same state again in just a few months any­way.

“Let’s just ask Claude to fix it.”

“Okay. I’ll cre­ate a loop and goal so it does­n’t stop un­til it’s checked that every­thing works.”

“Sounds good”

“Actually, I ran out of Fable us­age for to­day so I’ll run it to­mor­row”

You grab an­other cof­fee and walk back to your com­puter. You now have 13 PRs left to re­view. You see some­thing you don’t quite un­der­stand, so you mes­sage the per­son who wrote it.

“Why are we do­ing this here?”

They send you a link. It’s a Claude con­ver­sa­tion.

Somewhere in that con­ver­sa­tion, buried be­tween Claude con­fi­dently rec­om­mend­ing one ar­chi­tec­ture, apol­o­gis­ing, chang­ing its mind, your coworker ask­ing it to re­con­sider again and an­other 15 rounds of changes, is ap­par­ently the de­sign de­ci­sion be­hind this code.

“Which part should I read?”

“Probably all of it.”

Does this sound fa­mil­iar?

Whenever I talk about this, some­one even­tu­ally tells me that no­body ever fully un­der­stood large sys­tems any­way. It’s true.

You were never ex­pected to un­der­stand every ser­vice and every data­base. But at least some­one did and would ex­plain it to you.

Now they ask an LLM be­cause they don’t ac­tu­ally know them­selves.

You can’t af­ford bad en­gi­neers any­more

In every team, there are com­pe­tent peo­ple who make the pro­ject pos­si­ble. There are also peo­ple who es­sen­tially make it harder for every­one else. And now any­one can pro­duce more code in a day than they used to in a year.

In the story above, every­one is fail­ing:

The en­gi­neer open­ing a 25,000-line PR should have stopped the agent long be­fore it got there. They should have un­der­stood what it was do­ing, bro­ken the work into smaller pieces and ques­tioned every new ab­strac­tion it in­tro­duced.

The per­son re­view­ing it should have re­fused to re­view some­thing that large in­stead of giv­ing in.

The per­son adding Kafka should have been able to ex­plain ex­actly why it was needed.

The per­son who built the fea­ture should have been able to ex­plain where the data came from with­out send­ing a link to a Claude con­ver­sa­tion.

But what’s the prob­lem then? Just use AI to fix it. Well, it’s not that easy…

Before any­one jumps on this, none of this means tech­ni­cal debt is al­ways bad. The im­por­tant part is that you know it’s a short­cut.

Anyway, re­vert­ing a bad de­ci­sion is hard. Very hard.

For ex­am­ple, how long would it take an LLM to add a bunch of ta­bles and columns to the data­base? 10 min­utes?

But once you start stor­ing data there, you can’t just re­move them. You have to come up with a mi­gra­tion plan, make sure you don’t dis­rupt the sys­tem be­cause peo­ple are pay­ing to use this every day. You have to think about what you’ll do if the mi­gra­tion fails. Make sure you don’t end up with or­phaned for­eign keys. It’s just so much harder to fix. Even with the best model you can get.

And while you’re fix­ing it, more PRs keep com­ing in. More code, more ab­strac­tions, more de­ci­sions. A per­son can gen­er­ate 20,000 lines of code in an af­ter­noon, but you still have to sit there and un­der­stand what those lines ac­tu­ally do.

By the time you’ve un­tan­gled one bad de­ci­sion, five more have been merged.

The new AI econ­omy

Of course, bad en­gi­neers were al­ways a li­a­bil­ity.

It has been like this for decades, well be­fore OpenAI or Anthropic ex­isted. Bad de­ci­sions com­pounded, un­nec­es­sary com­plex­ity ac­cu­mu­lated and teams ended up main­tain­ing sys­tems no­body re­ally un­der­stood.

The dif­fer­ence is that there used to be a limit to how fast you could do it.

Today, im­ple­men­ta­tion is cheap. You are paid to make good de­ci­sions. To build soft­ware that will scale while man­ag­ing com­plex­ity.

Ask your­self why com­pa­nies are pay­ing six-fig­ure salaries for en­gi­neers in London or San Francisco in the first place.

If all they needed was some­one who could turn a spec­i­fi­ca­tion into work­ing code, why were they pay­ing that much when they could al­ready get it done cheaply else­where?

Why are the tech com­pa­nies claim­ing that “software is solved” still pay­ing top salaries to at­tract the best peo­ple they can?

My bet is that AI pushes salaries fur­ther apart. To be em­ploy­able, there’s a bar you have to clear and that bar is what­ever the cur­rent best model du jour can do.

Good en­gi­neers have be­come more valu­able be­cause AI lets them move much faster. They don’t need as many peo­ple around them just to do the im­ple­men­ta­tion work any­more.

At the same time, bad en­gi­neers have be­come much more ex­pen­sive to hire.

I wrote about this be­fore when I said the vibe coder ca­reer path is doomed.

You need to con­tribute be­yond what every­one al­ready gets by giv­ing an agent a prompt.

If you lack the judg­ment re­quired to eval­u­ate the LLM’s rec­om­men­da­tion, ask­ing for more judg­ment does­n’t solve the prob­lem.

At some point, some­one still has to know what is go­ing on. And that’s the most valu­able per­son on the team.

The peo­ple who don’t will be­come much cheaper to hire or get re­placed en­tirely while the money gets fun­nelled to­wards an in­creas­ingly smaller num­ber of peo­ple who can ac­tu­ally be trusted.

I don’t think this is go­ing to be lim­ited to soft­ware en­gi­neer­ing ei­ther. I be­lieve the same thing is go­ing to hap­pen across most knowl­edge work. AI will make the best peo­ple much more pro­duc­tive and the bad ones al­most im­pos­si­ble to hire. Before, there was a good chance some­one would catch their bad de­ci­sions be­fore they went too far. Now they can make changes faster than any­one around them can re­al­is­ti­cally re­view or un­der­stand them.

So while I work with po­lice de­part­ments reg­u­larly, I think it is crit­i­cally im­por­tant that tech­nol­ogy be used rea­son­ably.

While this may be off-putting to some of my clients, I worked with the Institute for Justice as an ex­pert wit­ness in their trial Schmidt v City of Norfolk. (Any opin­ions herein are my own and not those of IJ, to be clear.) The gist of that case was whether searches of his­tor­i­cally cached ALPR data (automated-license-plate-reader) con­sti­tuted an il­le­gal search.1

The judge ruled against plain­tiffs in that case. Here is a quote from the judg­ment:

Consistent with Plaintiffs’ claims in this case and con­trol­ling prece­dent in­volv­ing mass sur­veil­lance in pub­lic spaces, ALPR sur­veil­lance could be­come too in­tru­sive and run afoul of [constitutional pri­vacy stan­dards] at some point. But when? While a de­fin­i­tive an­swer to that ques­tion is elu­sive, what is read­ily ap­par­ent to this Court is that, at least in Norfolk, Virginia, the an­swer is: not to­day.

Consistent with Plaintiffs’ claims in this case and con­trol­ling prece­dent in­volv­ing mass sur­veil­lance in pub­lic spaces, ALPR sur­veil­lance could be­come too in­tru­sive and run afoul of [constitutional pri­vacy stan­dards] at some point. But when? While a de­fin­i­tive an­swer to that ques­tion is elu­sive, what is read­ily ap­par­ent to this Court is that, at least in Norfolk, Virginia, the an­swer is: not to­day.

The im­por­tant point to note about this quote is “not to­day”. This will be a long winded post, but to try to keep it sim­ple:

I think cam­eras will be­come ubiq­ui­tous in the fore­see­able fu­ture. So the ques­tion is not if this data will re­quire a war­rant, it is when. It is go­ing to hap­pen even­tu­ally un­der cur­rent case law.

I think cam­eras are good, and can be used to re­duce crime in a cost ef­fec­tive man­ner.

There is a dif­fer­ence be­tween ac­tive flags (e.g. this car is stolen and it pings the PD when it dri­ves past a cam­era) vs his­tor­i­cal searches (e.g. look to see where li­cense plate XYZ1000 was the last 30 days).

Requiring a war­rant for his­tor­i­cal searches will not se­ri­ously im­pede po­lice in­ves­ti­ga­tions.

The cur­rent sta­tus quo of not re­tain­ing data is VERY BAD; it does not pre­vent il­le­gal searches, and cur­rently lim­its the util­ity of ac­tu­ally us­ing that data for le­git­i­mate in­ves­ti­ga­tions.

Current stan­dards to pre­vent abuse of the search­ing ALPR data sys­tems are laugh­able.

Long story short in my opin­ion every­one would be bet­ter off if states just man­dated war­rant pro­ce­dures through state statutes.

Current Legal Standards

To try to not get too much into the weeds of what his­tor­i­cally con­sti­tutes a search, I think the eas­i­est place to start is via Carpenter vs US. So cur­rent US case law re­quires po­lice de­part­ments to ob­tain a war­rant to re­quest cel­lu­lar providers pro­vide law en­force­ment with cell phone tower pings (cell-site lo­ca­tion in­for­ma­tion, CSLI).

This de­vi­ated from his­tor­i­cal prece­dent in re­quir­ing a war­rant mainly be­cause it was pri­vate com­pa­nies that had the in­for­ma­tion. Before Carpenter, mostly it was ar­gued you did not have a rea­son­able ex­pec­ta­tion of pri­vacy if a pri­vate com­pany could ac­cess the same data. The court in Carpenter ba­si­cally made a de­ter­mi­na­tion that cell phone data was so com­pre­hen­sive it jus­ti­fied a dif­fer­ent stan­dard — that you could track the whole of a per­son’s move­ments with the de­tailed CSLI data. And that this level of in­va­sive­ness vi­o­lated a rea­son­able per­son’s ex­pec­ta­tion of pri­vacy. Even if Google had all that info, you did not ex­pect them to give it away.

This opin­ion was reaf­firmed with the re­cent Chatrie de­ci­sion (for ge­ofence war­rants, e.g. give me a ping for all cell phones in area X and date­time-range Y). Another rel­e­vant de­ci­sion to be aware of is also Beautiful Struggle v Baltimore, in which search­ing his­tor­i­cal aer­ial im­agery via drones also con­sti­tuted a search.

So this is why I am say­ing the ques­tion is when, not if, ALPR data will re­quire a war­rant. If a city hap­pened to have a cam­era on lit­er­ally every in­ter­sec­tion (which I think will hap­pen in the fu­ture), un­der cur­rent case law it would clearly be the same sit­u­a­tion as you have for your cell phone data.

Cameras are Good

To be brief, again I mostly work with po­lice de­part­ments in my ca­reer and was a for­mer crime an­a­lyst. I do think ALPR cam­eras are good in­vest­ments, mainly be­cause they are cheap enough to have a rea­son­able re­turn on in­vest­ment. (Note I do not think this about all po­lice tech, I am par­tic­u­larly crit­i­cal of the price tag for acoustic-gun-shot-de­tec­tion.)

So ALPRs are well un­der $3,000 per cam­era. The ma­chine learn­ing mod­els, cam­era, and com­pu­ta­tion nec­es­sary to flag a plate when it passes can eas­ily fit on cur­rent cell phones. (The harder part is pow­er­ing the phone and pro­tect­ing it from the el­e­ments.) ALPRs for the most part just take sta­tic im­ages and then ex­tract out the li­cense plate (and for some ven­dors ex­tract out ad­di­tional in­for­ma­tion, like car make and color).

The over­all ev­i­dence that ALPRs re­duce crime is pretty meh at the mo­ment (see my slides at a Wake Libertarian talk I did in 2024), but be­cause they are so cheap they re­ally only need to in­crease a few ar­rests per cam­era to likely have a pos­i­tive re­turn on in­vest­ment.

It is pretty hand-wavy, as we do not have es­ti­mates for the value of in­creased clear­ances I find per­sua­sive. But I think say­ing “I would pay $500 to help solve one case” is on the low side if any­thing. So a sin­gle cam­era if it helps catch just a hand­ful of crimes a year is likely in my opin­ion to be a pos­i­tive ROI.

I think cam­eras in all pub­lic spaces are go­ing to hap­pen. Imagine Ring comes out with a nicer cam­era sys­tem for home­own­ers that has more com­pre­hen­sive views around your house and is just as cheap. And we will ul­ti­mately be safer for it. So even for folks ad­vo­cat­ing that cities do not pay for Flock, this is com­ing any­way in the near fu­ture.

Historical Searches vs Active Flags

ALPRs have been around a long time. The first ones I worked with at Troy, NY when I was an an­a­lyst were in-car cam­eras. Basically a go pro at­tached to the win­dow that alerted when an of­fi­cer drove by a stolen plate.

While ALPRs ini­tial use was al­ways pitched as this ac­tive flag­ging of stolen ve­hi­cles, they were used right away to retroac­tively search the his­tor­i­cal lo­ca­tions of plates. They had a log of every plate, lat/​lon, and time­stamp of when that car passed a cam­era.

So imag­ine you are con­duct­ing an in­ves­ti­ga­tion of Joe Schmo, you know his li­cense plate, and then you can type in his plate and see where his car passed a cam­era. Based on this in­for­ma­tion, same as CSLI data, you can ba­si­cally trace where Joe went, where he re­peat­edly vis­ited, where he likely slept, etc. (The first time I used this at Troy, we fig­ured out a par­tic­u­lar in­di­vid­ual we were ac­tively in­ves­ti­gat­ing was liv­ing with his girl­friend for ex­am­ple. I was hon­estly amazed how densely filled in the map was of hits for a sin­gle plate based on the in-car cam­eras.)

You tech­ni­cally do not need to cache any data at all to ac­com­plish this “flag a stolen ve­hi­cle” (or any other sce­nario where you are ac­tively look­ing for a spe­cific li­cense plate). There are le­git­i­mate sce­nar­ios though where ALPR searches for re­cent data in a real time con­text can be very help­ful.

One of the more com­mon ex­am­ples — some­one robs a gas sta­tion, and they drove a ve­hi­cle. You don’t know the plate, but can look at the im­ages that passed by the fixed lo­ca­tion ALPRs in the time range, and then es­pe­cially if you have a car de­scrip­tion from the gas sta­tion at­ten­dant can fig­ure out the plate as­so­ci­ated with the ve­hi­cle.

To be clear I am not a lawyer, but in my opin­ion I think ex­i­gent cir­cum­stances make search­ing a few min­utes of cached ALPR lo­ca­tion data to­tally rea­son­able. In prac­tice, New Hampshire’s 3 minute data re­ten­tion is far too short. I could see ar­gu­ments for sev­eral hours (imagine “I found a dead body on the side of the road”, that re­quires more time for it to be re­ported.) But we are me­an­der­ing into the ter­ri­tory where it is not an ac­tive emer­gency “need to find some­one who may have a gun and hurt peo­ple” that would jus­tify those ex­i­gent cir­cum­stances. Those are the sce­nar­ios where get­ting a war­rant is rea­son­able (no dif­fer­ent than a ge­ofence war­rant if you do not have a plate and want to just search what cars passed by a cam­era within a cer­tain date-time win­dow, or no dif­fer­ent than a CSLI war­rant if you have an ac­tive sus­pect and want to search for a spe­cific li­cense plate).

Most states are re­tain­ing ALPR data for longer pe­ri­ods. While the Norfolk case was on­go­ing, Virginia set a stan­dard across the state at 21 days. Before that it was up to the in­di­vid­ual agency. It varies state by state, but states of­ten man­date data re­ten­tion around 30 days, or leave it up to the dis­cre­tion of the po­lice de­part­ment.

Deleting Data does not pre­vent abuses

These data re­ten­tion statutes are ar­gued as a mech­a­nism to pre­vent abuse. They do not ac­com­plish this.

If you look through the cases in which of­fi­cers abused the sys­tem to search for in­di­vid­u­als, all of them searched for spe­cific plates over-and-over again, some­times hun­dreds of times.

If you re­tain data for 20 days, you can just go and do a search every 20 days, keep notes on the data as you so wish, and then do an­other search 20 days later. Getting rid of old data, in-and-of-it­self, does noth­ing to pre­vent that abuse. In fact if some­one is ac­tively stalk­ing a per­son, you would ex­pect them to reg­u­larly do searches, see­ing where their vic­tim is go­ing on a reg­u­lar ba­sis while they have ac­cess to the sys­tem.

Simultaneously, delet­ing data does pre­vent its le­git­i­mate use in long term law en­force­ment in­ves­ti­ga­tions. It is to­tally nor­mal for a mur­der in­ves­ti­ga­tion to take more than 30 days to iden­tify a sus­pect. Gosh, sure would be nice to be able to then query the ALPR data to show whether a per­son was in the vicin­ity of the mur­der. Simultaneously it could be used by the de­fense for ex­cul­pa­tory pur­poses (which as­suredly would take longer than 30 days).

So folks ad­vo­cat­ing for delet­ing data as a mech­a­nism to pre­vent abuse are mak­ing things worse. It does not pre­vent abuse, and lim­its the util­ity of ALPR for his­tor­i­cal in­ves­ti­ga­tions. The only way data re­ten­tion by it­self pre­vents abuse is if you do not cache data at all (like in New Hampshire), and only use ALPRs for the ac­tive alert sit­u­a­tion.

What Smart Regulation Looks Like

One of the rea­sons I say that the cur­rent stan­dards to pre­vent abuse are laugh­able is that data re­ten­tion poli­cies and in­ter­nal PD poli­cies on when the data should be searched have been in place in most de­part­ments for years (if not a decade) at this point. The ex­am­ples where search­ing ALPR data to stalk an in­ti­mate part­ner were ob­vi­ously not pre­vented via data re­ten­tion poli­cies.

Alas, my sug­ges­tion that some data is cached for real time in­ves­ti­ga­tions (longer than 3 min­utes), and that a war­rant should be re­quired out­side of this win­dow, does not pre­vent that type of abuse ei­ther. Most de­part­ments have in place rea­sons why a search can be con­ducted, and some states have spe­cific statutes iden­ti­fy­ing im­per­mis­si­ble rea­sons for con­duct­ing searches. In the Norfolk IJ case, of­fi­cers, when en­ter­ing a rea­son for a search (which was of­ten omit­ted), some­times sup­plied rea­sons that ap­peared prima fa­cie il­le­gal, such as “protest”.

Departments, even if they have a stan­dard to do in­ter­nal au­dits, of­ten do not fol­low them. It took Tyler Dukes ask­ing Raleigh PD for their au­dit re­sults for them to even con­duct their first au­dit.

This is a long stand­ing prob­lem for PDs, not just with ALPRs, but also with search­ing crim­i­nal his­tory il­le­gally. IJ col­lat­ing a dozen cases of ar­rests of ALPR mis­use across the coun­try is not ev­i­dence these sys­tems are work­ing, as it is likely the case that only the most egre­gious abuses are ever caught.

In ad­di­tion to cre­at­ing state statutes to man­date that a war­rant be used for his­tor­i­cal ALPR searches, states should, at a min­i­mum, have clear pun­ish­ments for il­le­gal searches. These should in­clude at a min­i­mum los­ing your job, and be­ing banned from ac­cess­ing the sys­tem for­ever. When I was a crime an­a­lyst in New York (and ditto for when I worked at DCJS), this was the stan­dard for mis­us­ing the crim­i­nal his­tory search data­base.

If there is a stan­dard for just re­tain­ing ac­tive search data for less than 24 hours, it does pre­sent a po­ten­tial sim­ple check that should be flagged — if a spe­cific plate or spe­cific cam­era is searched twice within 2 days, it should be flagged to re­view more closely. Flock does have their own sys­tem to iden­tify sus­pi­cious search his­tory.

The big­ger is­sue to me though is who is do­ing the re­view­ing. It does not make sense to put this on ven­dors, and PDs just have not se­ri­ously de­voted re­sources to this, even in re­sponse to pub­lic crit­i­cism. This au­dit mech­a­nism should be del­e­gated to a third party, ei­ther a spe­cific group in the state at­tor­ney gen­er­al’s of­fice, or a state crim­i­nal jus­tice agency (like DCJS in New York).

So that of course needs to be ex­plic­itly set by state statute as well. Who is do­ing the au­dit­ing?

My fo­cus so far has been on abuses via po­lice de­part­ments them­selves, but smart reg­u­la­tion should also spec­ify au­dit­ing of the ven­dors them­selves, as well as pun­ish­ments if they fail to meet data stan­dards. (I am not think­ing so much TEMPEST at­tacks here, but more so “I left an unau­then­ti­cated end­point willy nilly on the in­ter­net”.)

Indeed, many of the re­quire­ments I am sug­gest­ing are likely al­ready on the books; the prob­lem is that the en­tity re­spon­si­ble for au­dit­ing is of­ten un­spec­i­fied or lacks the re­sources to do the work. (Also it is of­ten un­clear what the pun­ish­ments are for fail­ing to abide by statutes. That also needs to be specif­i­cally stated.)

The Future

So while I hope (although I have no ex­pec­ta­tion) that my blog post can some­how in­flu­ence cur­rent stan­dards across the coun­try, I think it is im­por­tant to keep in mind sur­veil­lance not just as the world ex­ists now, but how it may look in the fore­see­able fu­ture.

I think states should just pull the band aid off and cre­ate statutes that re­quire a war­rant to search the his­tor­i­cal ALPR data. (And this makes data shar­ing be­tween agen­cies mostly moot, the real time searches only need to be done within your own ju­ris­dic­tion.) Like I said at the be­gin­ning, the cur­rent case law on be­ing able to re­con­struct the whole of a per­son’s move­ments (which I think is quite rea­son­able) will even­tu­ally be met if the ALPR cam­eras be­come dense enough. So states can ei­ther cre­ate the statutes to dic­tate that a war­rant is nec­es­sary them­selves, or even­tu­ally have the court sys­tem thrust it upon them.

In a world filled with pri­vately owned cam­eras in pub­lic spaces, I think these sug­ges­tions are still rel­e­vant. So sim­i­lar to Carpenter for CSLI data, and Chatrie for ge­ofence war­rants, there should just be war­rant stan­dards for his­tor­i­cally search­ing any sur­veil­lance footage. There need be no spe­cial dis­tinc­tion be­tween ALPR data (public or pri­vate) or video cam­eras.

Even if the groups call­ing for the ban­ning of Flock cam­eras get their way, this does not stop pri­vate own­ers from col­lect­ing the data. So ban­ning Flock, by it­self, does not pre­vent abuse of search­ing pri­vate cam­eras. Again I think it is bet­ter to just let the gov­ern­ment re­tain the data (same as pri­vate ven­dors will re­tain the data), and have con­sis­tent war­rant stan­dards for po­lice to ob­tain that his­tor­i­cal data.

This, of course, is a bur­den to de­tec­tives. I be­lieve that trade-off in pro­tect­ing our per­sonal lib­er­ties while still al­low­ing po­lice ef­fec­tive means to in­ves­ti­gate cases is a rea­son­able one.

There are some tech­ni­cal­i­ties be­tween whether just col­lect­ing the data is a search (which was the sce­nario in the Norfolk case) or whether do­ing an ac­tive search (e.g. an of­fi­cer query­ing the sys­tem for li­cense plate ABC1234). The Norfolk case was the for­mer, but for this post I am fo­cus­ing on of­fi­cers ac­tu­ally search­ing the data (the lat­ter sce­nario).↩︎

There are some tech­ni­cal­i­ties be­tween whether just col­lect­ing the data is a search (which was the sce­nario in the Norfolk case) or whether do­ing an ac­tive search (e.g. an of­fi­cer query­ing the sys­tem for li­cense plate ABC1234). The Norfolk case was the for­mer, but for this post I am fo­cus­ing on of­fi­cers ac­tu­ally search­ing the data (the lat­ter sce­nario).↩︎

This repos­i­tory con­tains model weights and con­fig­u­ra­tion files for the post-trained model in the Hugging Face Transformers for­mat. These ar­ti­facts are com­pat­i­ble with vLLM, SGLang, TokenSpeed, etc.

This repos­i­tory con­tains model weights and con­fig­u­ra­tion files for the post-trained model in the Hugging Face Transformers for­mat.

These ar­ti­facts are com­pat­i­ble with vLLM, SGLang, TokenSpeed, etc.

For users seek­ing man­aged, scal­able in­fer­ence with­out in­fra­struc­ture main­te­nance, the of­fi­cial Qwen API ser­vice is pro­vided by Qwen Cloud. In par­tic­u­lar, Qwen3.8-Max is the of­fi­cial ver­sion based on Qwen3.8 – 2.4T-A95B with more fea­tures, such as vi­sion in­put & non-think­ing sup­port, 1M con­text length by de­fault, of­fi­cial built-in tools, etc. For more in­for­ma­tion, please re­fer to the Qwen3.8-Max Overview.

For users seek­ing man­aged, scal­able in­fer­ence with­out in­fra­struc­ture main­te­nance, the of­fi­cial Qwen API ser­vice is pro­vided by Qwen Cloud.

In par­tic­u­lar, Qwen3.8-Max is the of­fi­cial ver­sion based on Qwen3.8 – 2.4T-A95B with more fea­tures, such as vi­sion in­put & non-think­ing sup­port, 1M con­text length by de­fault, of­fi­cial built-in tools, etc. For more in­for­ma­tion, please re­fer to the Qwen3.8-Max Overview.

Following the wide­spread com­mu­nity adop­tion of the Qwen3.5 and Qwen3.6 se­ries, we are pleased to in­tro­duce Qwen3.8, the most ca­pa­ble gen­er­a­tion in the Qwen open-model fam­ily to date.

For the first time, Qwen3.8 brings a Qwen-Max-class model to open re­lease. Built on the ar­chi­tec­tural foun­da­tion of Qwen3.5, Qwen3.8 de­liv­ers sub­stan­tial gains across cod­ing, pro­fes­sional work, re­search, and long-hori­zon agen­tic tasks. Beyond an­swer­ing harder ques­tions, Qwen3.8 is de­signed to carry com­plex, multi-step tasks through to com­ple­tion with greater re­li­a­bil­ity.

Qwen3.8 Highlights

Qwen3.8 fea­tures the fol­low­ing en­hance­ments:

Core Capabilities: Comprehensive im­prove­ments across cod­ing, pro­fes­sional work, re­search, and long-hori­zon agen­tic tasks.

Agent Execution: Stronger au­tonomous plan­ning and bet­ter han­dling of en­vi­ron­ment feed­back, lead­ing to more re­li­able end-to-end task com­ple­tion.

Downstream Compatibility: Broader sup­port for pop­u­lar har­nesses and de­vel­op­ment tools, mak­ing it eas­ier to in­te­grate into your ex­ist­ing stack.

Flexible Thinking Control: Reasoning depth can be tuned with rea­son­ing_­ef­fort, and rea­son­ing con­text from his­tor­i­cal mes­sages is re­tained via pre­serve_­think­ing.

For more de­tails, please re­fer to our blog post Qwen3.8-Max.

Model Overview

Type: Causal Language Model

Training Stage: Pre-training & Post-training

Language Model Number of Parameters: 2.4T in to­tal and 95B ac­ti­vated Hidden Dimension: 8192 Token Embedding: 248,320 (Padded) Number of Layers: 92 Hidden Layout: 23 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE)) Gated DeltaNet: Number of Linear Attention Heads: 128 for V and 16 for QK Head Dimension: 128

Gated Attention: Number of Attention Heads: 64 for Q and 4 for KV Head Dimension: 256 Rotary Position Embedding Dimension: 64

Mixture of Experts: Number of Experts: 512 Number of Activated Experts: 10 Routed + 1 Shared Expert Intermediate Dimension: 2048

LM Output: 248,320 (Padded) MTP (Multi-Token Prediction): trained with mul­ti­ple steps

Number of Parameters: 2.4T in to­tal and 95B ac­ti­vated

Hidden Dimension: 8192

Token Embedding: 248,320 (Padded)

Number of Layers: 92

Hidden Layout: 23 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE))

Gated DeltaNet: Number of Linear Attention Heads: 128 for V and 16 for QK Head Dimension: 128

Number of Linear Attention Heads: 128 for V and 16 for QK

Head Dimension: 128

Gated Attention: Number of Attention Heads: 64 for Q and 4 for KV Head Dimension: 256 Rotary Position Embedding Dimension: 64

Number of Attention Heads: 64 for Q and 4 for KV

Head Dimension: 256

Rotary Position Embedding Dimension: 64

Mixture of Experts: Number of Experts: 512 Number of Activated Experts: 10 Routed + 1 Shared Expert Intermediate Dimension: 2048

Number of Experts: 512

Number of Activated Experts: 10 Routed + 1 Shared

Expert Intermediate Dimension: 2048

LM Output: 248,320 (Padded)

MTP (Multi-Token Prediction): trained with mul­ti­ple steps

Context Length: 262,144 na­tively and ex­ten­si­ble up to 1,010,000 to­kens.

Benchmark Results

1. Fable5 re­sults may in­volve fall­backs.2. Terminal Bench 2.1: Evaluated with Claude Code (avg@10), us­ing a 5-hour time­out and max_­to­kens=131,072. For all other mod­els, we re­port the best pub­lished score across har­nesses: Claude Opus 4.8 and Claude Fable 5 with Terminus 2 from Artificial Analysis (https://​ar­ti­fi­cial­analy­sis.ai/​eval­u­a­tions/​ter­mi­nal­bench-v2 – 1); GPT-5.6 Sol with Codex (https://​ope­nai.com/​in­dex/​pre­view­ing-gpt-5 – 6-sol/).3. SWE-bench Pro: Evaluated with the Claude Code har­ness, temp=1.0, top_p=0.95, and a 256K con­text win­dow. Problematic tasks cor­rected and all base­lines eval­u­ated on the re­fined bench­mark.4. DeepSWE 1.1: Evaluated with the Claude Code and mini-SWE-agent har­nesses, temp=1.0, top_p=0.95, and a 256K con­text win­dow. We re­port the high­est score among both har­nesses; no­tably, Qwen3.8-Max per­forms best on Claude Code.5. NL2Repo-Bench: Evaluated with the Claude Code har­ness. To pre­vent re­ward hack­ing, we dis­able Bash com­mands that at­tempt to ac­cess the spe­cific repos­i­tory, such as pip down­load, pip in­stall, and git clone.6. FrontierSWE: Evaluated with the Claude Code har­ness. All other avail­able MEAN@5 re­sults are taken from the of­fi­cial FrontierSWE leader­board (https://​www.fron­tier­swe.com) as of August 3, 2026. Dominance scores are re­com­puted from the raw scores us­ing the of­fi­cial eval­u­a­tion script. “–” in­di­cates that no of­fi­cial MEAN@5 re­sult was avail­able as of that date.7. MLS-Bench-Lite: Evaluated with Claude Code us­ing a 5-hour time­out and max_­to­kens=131,072. All other model scores are taken from the of­fi­cial leader­board.8. PaperBench: Evaluated in the BasicAgent set­ting un­der Code-Dev mode, judged by Claude Opus 4.6, and av­er­aged over 3 runs (max 12 hours per run).9. AndroidBench: Evaluated on the 95-task pub­lic sub­set, re­port­ing avg@3 scores.10. QwenSWEBench: Inhouse cod­ing bench­mark to eval­u­ate mod­els’ soft­ware en­gi­neer­ing ca­pa­bil­i­ties. Evaluated with the Claude Code har­ness. Reporting avg@3 with an 8-hour time­out, max_­to­kens=32,768, tem­per­a­ture=1.0, and a 256K-token con­text win­dow.11. QwenQoderBench: Inhouse cod­ing bench­mark to eval­u­ate user ex­pe­ri­ence on Qoder. Evaluated with the Claude Code har­ness. Reporting avg@5 with a 6-hour time­out, max_­to­kens=32,768, tem­per­a­ture=1.0, and a 256K-token con­text win­dow.12. QwenReactBench: Inhouse React pro­ject build­ing bench­mark us­ing Claude Code as the har­ness, bilin­gual (EN/CN), 7 cat­e­gories; auto-ren­der + mul­ti­modal judge; BT/Elo rat­ing.13. QwenSVGBench: Inhouse SVG code gen­er­a­tion bench­mark; bilin­gual (EN/CN), auto-ren­der + mul­ti­modal judge; BT/Elo rat­ing.14. CoWorkBench: Inhouse cowork bench­mark for eval­u­at­ing long-hori­zon tasks across com­puter sci­ence, fi­nance, law, med­ical, and other pro­duc­tiv­ity do­mains.15. SkillsBench: Evaluated on the pub­lic SkillsBench v1.1 bench­mark across 87 tasks, re­port­ing the av­er­age score over three runs per task. Opus 4.8 and Fable 5 are eval­u­ated on Claude Code; GPT-5.6 Sol is eval­u­ated on Codex; the Qwen-series are eval­u­ated on OpenCode. All re­sults are from our own test­ing.16. Automation-Bench: Evaluated on the 600-task pub­lic sub­set.17. WideSearch: Evaluated with the Claude Code har­ness for ex­ter­nal mod­els and the Qwen-Agent har­ness for ours, re­port­ing the av­er­age item-F1 over four runs.18. $OneMillion-Bench: Evaluated us­ing gem­ini-3.1-pro-pre­view.19. PLawBench: Evaluated us­ing gem­ini-3.1-pro-pre­view.20. Empty cells (–): Scores are not yet avail­able or are not ap­plic­a­ble.

Quickstart

For stream­lined in­te­gra­tion, we rec­om­mend us­ing Qwen3.8 via APIs.

Serving Qwen3.8

Inference ef­fi­ciency and through­put vary sig­nif­i­cantly across frame­works. We rec­om­mend us­ing the lat­est frame­work ver­sions to en­sure op­ti­mal per­for­mance and com­pat­i­bil­ity. For pro­duc­tion work­loads or high-through­put sce­nar­ios, ded­i­cated serv­ing en­gines such as SGLang, vLLM, or TokenSpeed are rec­om­mended.

Inference ef­fi­ciency and through­put vary sig­nif­i­cantly across frame­works. We rec­om­mend us­ing the lat­est frame­work ver­sions to en­sure op­ti­mal per­for­mance and com­pat­i­bil­ity. For pro­duc­tion work­loads or high-through­put sce­nar­ios, ded­i­cated serv­ing en­gines such as SGLang, vLLM, or TokenSpeed are rec­om­mended.

Qwen3.8 can be de­ployed with pop­u­lar in­fer­ence frame­works, e.g.:

SGLang: Qwen3.8 Cookbook

vLLM: Qwen3.8 Recipe

TokenSpeed: Qwen3.8 Recipe

API Usage

Qwen3.8 – 2.4T-A95B is a text-only model that re­quires think­ing mode for all in­ter­ac­tions. Multimodal in­puts are not sup­ported, and think­ing can­not be dis­abled. Every re­sponse will au­to­mat­i­cally be­gin with rea­son­ing en­closed in <think>\n…</think>\n\n be­fore the fi­nal out­put.

Qwen3.8 – 2.4T-A95B is a text-only model that re­quires think­ing mode for all in­ter­ac­tions. Multimodal in­puts are not sup­ported, and think­ing can­not be dis­abled. Every re­sponse will au­to­mat­i­cally be­gin with rea­son­ing en­closed in <think>\n…</think>\n\n be­fore the fi­nal out­put.

We rec­om­mend us­ing the fol­low­ing set of sam­pling pa­ra­me­ters for gen­er­a­tion:

tem­per­a­ture=1.0, top_p=0.95, top_k=20, min_p=0.0, pres­ence_penalty=0.0, rep­e­ti­tion_penalty=1.0

Please note that the sup­port for sam­pling pa­ra­me­ters varies ac­cord­ing to in­fer­ence frame­works.

We rec­om­mend us­ing the fol­low­ing set of sam­pling pa­ra­me­ters for gen­er­a­tion:

tem­per­a­ture=1.0, top_p=0.95, top_k=20, min_p=0.0, pres­ence_penalty=0.0, rep­e­ti­tion_penalty=1.0

Please note that the sup­port for sam­pling pa­ra­me­ters varies ac­cord­ing to in­fer­ence frame­works.

Qwen3.8 comes with of­fi­cial sup­port for rea­son­ing_­ef­fort, which can be used to ad­just rea­son­ing depth and con­trol cost:

xhigh (default): for com­plex tasks de­mand­ing thor­ough analy­sis

medium: bal­anc­ing ac­cu­racy and speed

low: ef­fi­cient rea­son­ing op­ti­miz­ing for speed and cost

In ad­di­tion, pre­serve_­think­ing is en­abled by de­fault for all work­loads for the best out-of-the-box ex­pe­ri­ence.

Chat Completions API

The Chat Completions API can be used with most in­fer­ence frame­works, as well as Qwen Cloud. Before start­ing, make sure the OpenAI Python SDK is in­stalled and the API key and the API base URL are con­fig­ured, e.g.:

pip in­stall -U ope­nai

# Set the fol­low­ing ac­cord­ingly ex­port OPENAI_BASE_URL=‘your-base-url’ ex­port OPENAI_API_KEY=‘your-api-key’

Text-Only Input

from ope­nai im­port OpenAI # Configured by en­vi­ron­ment vari­ables client = OpenAI()

mes­sages = [{“role”: “user”, “content”: “Write a Python func­tion to merge two sorted linked lists.“}]

com­ple­tion = client.chat.com­ple­tions.cre­ate( model=“Qwen/​Qwen3.8 – 2.4T-A95B”, mes­sages=mes­sages, ex­tra_­body={ “chat_template_kwargs”: { “enable_thinking”: True, # on by de­fault; should not be turned off “preserve_thinking”: True, # on by de­fault }, }, rea­son­ing_­ef­fort=“xhigh”, # xhigh by de­fault; sup­ported lev­els are xhigh, medium, and low stream=True, stream_op­tions={“in­clude_us­age”: True}, )

rea­son­ing_­con­tent = “” an­swer_­con­tent = “” is_an­swer­ing = False print(“\n” + “=” * 20 + “Reasoning” + “=” * 20 + “\n”)

for chunk in com­ple­tion: if not chunk.choices: print(“\nUsage:“) print(chunk.us­age) con­tinue

delta = chunk.choices[0].delta

if hasattr(delta, “reasoning_content”) and delta.rea­son­ing_­con­tent is not None: if not is_an­swer­ing: print(delta.rea­son­ing_­con­tent, end=“”, flush=True) rea­son­ing_­con­tent += delta.rea­son­ing_­con­tent

if hasattr(delta, “content”) and delta.con­tent: if not is_an­swer­ing: print(“\n” + “=” * 20 + “Answer” + “=” * 20 + “\n”) is_an­swer­ing = True print(delta.con­tent, end=“”, flush=True) an­swer_­con­tent += delta.con­tent

If you are us­ing APIs from Qwen Cloud, in ad­di­tion to chang­ing model, please pass ex­tra_­body={“en­able_­think­ing”: True, “preserve_thinking”: True} in­stead of ex­tra_­body={“chat_tem­plate_k­wargs”: {“enable_thinking”: True, “preserve_thinking”: True}}.

If you are us­ing APIs from Qwen Cloud, in ad­di­tion to chang­ing model, please pass ex­tra_­body={“en­able_­think­ing”: True, “preserve_thinking”: True} in­stead of ex­tra_­body={“chat_tem­plate_k­wargs”: {“enable_thinking”: True, “preserve_thinking”: True}}.

Best Practices

To achieve op­ti­mal per­for­mance, we rec­om­mend the fol­low­ing set­tings:

Sampling Parameters:

We sug­gest us­ing the fol­low­ing set of sam­pling pa­ra­me­ters: tem­per­a­ture=1.0, top_p=0.95, top_k=20, min_p=0.0, pres­ence_penalty=0.0, rep­e­ti­tion_penalty=1.0

For sup­ported frame­works, you can ad­just the pres­ence_penalty pa­ra­me­ter be­tween 0 and 2 to re­duce end­less rep­e­ti­tion. However, us­ing a higher value may oc­ca­sion­ally re­sult in lan­guage mix­ing and a slight de­crease in model per­for­mance.

Sampling Parameters:

We sug­gest us­ing the fol­low­ing set of sam­pling pa­ra­me­ters: tem­per­a­ture=1.0, top_p=0.95, top_k=20, min_p=0.0, pres­ence_penalty=0.0, rep­e­ti­tion_penalty=1.0

tem­per­a­ture=1.0, top_p=0.95, top_k=20, min_p=0.0, pres­ence_penalty=0.0, rep­e­ti­tion_penalty=1.0

For sup­ported frame­works, you can ad­just the pres­ence_penalty pa­ra­me­ter be­tween 0 and 2 to re­duce end­less rep­e­ti­tion. However, us­ing a higher value may oc­ca­sion­ally re­sult in lan­guage mix­ing and a slight de­crease in model per­for­mance.

Adequate Output Length: To op­ti­mize per­for­mance on agen­tic tasks, we rec­om­mend al­lo­cat­ing suf­fi­cient out­put length to al­low the model to gen­er­ate de­tailed and com­pre­hen­sive re­sponses. For frame­works that sup­port sep­a­rate to­ken lim­its for in­ter­nal rea­son­ing and fi­nal out­puts, we sug­gest the fol­low­ing con­fig­u­ra­tion within the 1M con­text length:

Reasoning Content: Set the max­i­mum out­put length to 262,144 to­kens. Final Response: Set the max­i­mum out­put length to 131,072 to­kens.

These set­tings pro­vide the nec­es­sary ca­pac­ity for com­plex rea­son­ing while en­sur­ing am­ple space for high-qual­ity fi­nal de­liv­er­ables.

Adequate Output Length: To op­ti­mize per­for­mance on agen­tic tasks, we rec­om­mend al­lo­cat­ing suf­fi­cient out­put length to al­low the model to gen­er­ate de­tailed and com­pre­hen­sive re­sponses. For frame­works that sup­port sep­a­rate to­ken lim­its for in­ter­nal rea­son­ing and fi­nal out­puts, we sug­gest the fol­low­ing con­fig­u­ra­tion within the 1M con­text length:

Reasoning Content: Set the max­i­mum out­put length to 262,144 to­kens.

Final Response: Set the max­i­mum out­put length to 131,072 to­kens.

These set­tings pro­vide the nec­es­sary ca­pac­ity for com­plex rea­son­ing while en­sur­ing am­ple space for high-qual­ity fi­nal de­liv­er­ables.

Citation

If you find our work help­ful, feel free to give us a cite.

@misc{qwen38, ti­tle = {{Qwen3.8-Max}: A New Bar for Coding and Cowork}, url = {https://​qwen.ai/​blog?id=qwen3.8}, au­thor = {{Qwen Team}}, month = {August}, year = {2026} }

A white na­tion­al­ist with neo-Nazi links who hurled racist abuse at the Indian prime min­is­ter dur­ing his visit last month is be­ing paid by Facebook’s par­ent com­pany, Meta, to pro­duce con­tent.

An ABC NEWS Verify in­ves­ti­ga­tion into sev­eral con­tro­ver­sial Australian pages on Facebook has found that some are ben­e­fit­ing di­rectly from so­cial me­dia mon­eti­sa­tion pro­grams run by Meta.

Hugo Lennon is a far-right ag­i­ta­tor who was for­mally moved on by Victoria Police for shout­ing pro­fan­i­ties and racist abuse at Narendra Modi at a Melbourne ho­tel.

He has been re­ceiv­ing rev­enue through Facebook’s “Content Monetization” pro­gram since September 2025.

White na­tion­al­ist Hugo Lennon be­ing con­fronted by Victoria Police af­ter yelling racist abuse at India’s prime min­is­ter. (Supplied)

Mr Lennon has been pho­tographed with known neo-Nazis and has in­ter­acted with them on­line.

His re­cent Facebook videos in­clude his out­burst at the Indian PM, and oth­ers ref­er­enc­ing white su­prema­cist con­cepts like re­place­ment the­ory and rem­i­gra­tion.

The Noticer, a far-right Australian news web­site that reg­u­larly pro­motes white su­prema­cist and neo-Nazi ide­olo­gies, has been mak­ing money through the same Facebook rev­enue pro­gram since November 2025.

There is a short pe­riod from January 11 to February 15 when it stopped earn­ing money for un­known rea­sons.

The Noticer is cur­rently sus­pended from its X ac­count with a no­tice that the ac­count is “temporarily un­avail­able be­cause it vi­o­lates the X Hateful Profile Policy”.

A page pro­mot­ing the anti-im­mi­gra­tion group “March for Australia”, which also has close ties to neo-Nazis, reg­is­tered for the pro­gram in December 2025.

Monica Smit, the founder of the anti-vac­cine and anti-lock­down group Reignite Democracy Australia, joined Facebook’s con­tent mon­eti­sa­tion pro­gram in September 2025.

She ap­pears to have been earn­ing ad­ver­tis­ing rev­enue through Facebook from as far back as 2017.

Her page fea­tures vac­cine mis­in­for­ma­tion and pro­motes the sale of “radiation pro­tec­tion” bracelets through her per­sonal web­site.

Hugo Lennon, Monica Smit, The Noticer and March for Australia did not re­spond to re­quests for com­ment.

Facebook told ABC NEWS Verify “it is not Meta’s role to po­lice of­fen­sive­ness”. (Reuters: Dado Ruvic/illustration)

Invitation only

Facebook dis­trib­uted nearly $US3 bil­lion ($4.27 bil­lion) to an es­ti­mated 16.2 mil­lion mon­e­tised ac­counts in 2025.

Meta has pub­lished reg­u­lar dis­clo­sures re­lat­ing to these “partner-publishers” since 2019.

Tech pol­icy and ac­count­abil­ity non-profit What To Fix col­lates this raw data into a search­able archive that was used by ABC NEWS Verify for this in­ves­ti­ga­tion.

According to Facebook, the mon­eti­sa­tion scheme that Hugo Lennon, Monica Smit, March for Australia and The Noticer ap­pear to be prof­it­ing from is “an in­vi­ta­tion-only pro­gramme that lets you earn money from the per­for­mance of your el­i­gi­ble pub­lic reels, pho­tos, sto­ries and text posts.”

What To Fix ex­ec­u­tive di­rec­tor Victoire Rio said it meant that, in ef­fect, Facebook was in a busi­ness re­la­tion­ship with these peo­ple and or­gan­i­sa­tions.

“If we con­sider that Meta is in a di­rect com­mer­cial re­la­tion­ship with its pub­lish­ers, in­so­far that they’re pay­ing them roy­al­ties and that they have a mon­eti­sa­tion agree­ment with them, then you could ar­gue that they are li­able for the con­tent that gets pro­duced by their busi­ness part­ner in some ways.”

Ms Rio said she was no longer sur­prised by some of the pages Meta was choos­ing to mon­e­tise.

“We’ve ac­tu­ally en­coun­tered ac­counts af­fil­i­ated with sanc­tioned en­ti­ties that were part of those pro­grams,” she said.

“It’s quite com­mon that we see ac­counts that are very clearly in vi­o­la­tion of their poli­cies get­ting through.

“I think the ques­tion here re­ally is: is the qual­ity of the en­force­ment of those poli­cies that they have.”

RDA founder Monica Smit de­liv­ers a speech out­side Daniel Andrews’ elec­torate of­fice on Tuesday. (Facebook)

Policy Violation

The pages ex­am­ined by ABC NEWS Verify all ap­peared at times to be in di­rect vi­o­la­tion of Facebook’s own poli­cies around con­tent mon­eti­sa­tion.

They state that “debated so­cial is­sues” such as race “may face re­duced or re­stricted mon­eti­sa­tion”, while “misleading med­ical in­for­ma­tion” is “ineligible to mon­e­tise”.

Independent right-wing ex­trem­ism re­searcher Kaz Ross said pay­ing con­tro­ver­sial cre­ators was a de­lib­er­ate strat­egy from Meta.

“Their fi­nan­cial model is to re­ward con­tent cre­ators who get en­gage­ment, and as we know, the best way of get­ting en­gage­ment is to pro­duce rage bait, ex­trem­ist ma­te­r­ial, ag­gra­vat­ing ma­te­r­ial: any­thing that will get peo­ple up­set and ar­gu­ing is the best way to get en­gage­ment,” Dr Ross said.

Meta did not an­swer spe­cific ques­tions about these pages and their con­tent but pro­vided a gen­eral state­ment.

“We have clear poli­cies in place which any­one us­ing our mon­eti­sa­tion tools must com­ply with.“When Content Monetisation Pages vi­o­late our Community Standards, we ap­ply penal­ties such as tem­porar­ily or per­ma­nently dis­abling their abil­ity to earn on our plat­forms.“Cre­ators or pub­lish­ers who re­peat­edly breach these poli­cies will be re­moved from our plat­forms.”

“We have clear poli­cies in place which any­one us­ing our mon­eti­sa­tion tools must com­ply with.

“When Content Monetisation Pages vi­o­late our Community Standards, we ap­ply penal­ties such as tem­porar­ily or per­ma­nently dis­abling their abil­ity to earn on our plat­forms.

“Creators or pub­lish­ers who re­peat­edly breach these poli­cies will be re­moved from our plat­forms.”

The com­pany also said it was, “important to dis­tin­guish be­tween speech that is of­fen­sive and con­tent that could po­ten­tially lead to of­fline vi­o­lence”.

“Such speech may be of­fen­sive to many, but it is not Meta’s role to po­lice of­fen­sive­ness,” it said.

‘Any amount is too much’

It is un­clear ex­actly how much money these spe­cific cre­ators have made from their Facebook pages as this in­for­ma­tion is not made pub­lic by Meta.

Dr Ross said any amount was too much.

“It’s morally bank­rupt to do that, to know that these peo­ple are pro­duc­ing con­tent that’s in breach of your own reg­u­la­tions, yet you’re ac­tu­ally not just al­low­ing them, but you’re en­cour­ag­ing them by pay­ing them,” she said.

Meta is also one of the few so­cial me­dia gi­ants that dis­closes any in­for­ma­tion about who it is pay­ing. Other plat­forms, like YouTube, TikTok and Twitter, have very lit­tle trans­parency in this space.

“I think in gen­eral, Meta has been more forth­com­ing with trans­parency and that’s some­thing that should be lauded,” What To Fix’s Victoire Rio said.

“It’s still a very low bar, and the ques­tion here is to re­ally push for all plat­forms to at least meet that bar and if not ac­tu­ally go higher,′ she said.

🌑 Totality be­gins in …

📹 Reaches first we­b­cam in …

made by jonty

Today we are re­leas­ing Grok 4.6. Grok 4.6 builds on Grok 4.5 with a par­tic­u­lar fo­cus on long-run­ning agents and more am­bi­tious in­ter­ac­tive and vi­sual work. It stays with com­plex tasks across many steps, whether re­search­ing a topic, an­a­lyz­ing in­for­ma­tion, work­ing across a code­base, or turn­ing an idea into a pol­ished ap­pli­ca­tion or work ar­ti­fact.

Grok 4.6 achieves fron­tier in­tel­li­gence across sev­eral agen­tic cod­ing and knowl­edge work bench­marks. It matches GPT-5.6 Sol on the Artificial Analysis Intelligence Index, which is a com­pos­ite score of nine bench­marks.

Competitor fig­ures are drawn from the re­spec­tive de­vel­op­ers’ pub­lished sys­tem cards or bench­mark leader­boards

Grok 4.6 is avail­able to­day in Cursor and Grok Build. We’re of­fer­ing 2x in­cluded us­age in­side Grok Build and Cursor for the first week so you can start try­ing 4.6 im­me­di­ately.

Training Grok 4.6

Grok 4.6 un­der­went a longer sup­ple­men­tal train­ing run than Grok 4.5, with cu­rated model-gen­er­ated data for rea­son­ing and ad­vanced tech­ni­cal con­cepts, high-qual­ity en­gi­neer­ing data, and an im­proved op­ti­mizer and train­ing recipe. This pro­duced a stronger foun­da­tion for the SFT and RL stages that fol­lowed.

We then used Grok 4.5 to re­gen­er­ate the SFT tra­jec­to­ries across rea­son­ing ef­forts, agent har­nesses, and do­mains such as STEM, soft­ware en­gi­neer­ing, and knowl­edge work, and fil­tered out prob­lem­atic traces with model-based checks. The re­sult­ing SFT check­point shows strong per­for­mance and im­proved be­hav­ior.

Grok 4.6 is trained on a wide range of agen­tic RL tasks, in­clud­ing knowl­edge work, gen­eral cod­ing, and do­main-spe­cific en­vi­ron­ments for ker­nel op­ti­miza­tion, web de­vel­op­ment, com­puter-aided de­sign, and more.

Turning am­bi­tious ideas into work­ing pro­jects

We tested Grok 4.6 on pro­jects de­signed to stretch its range and abil­ity to sus­tain work over many steps. We found the model is es­pe­cially strong at turn­ing a broad prod­uct idea into a work­ing first ver­sion. It can re­search un­fa­mil­iar do­mains, struc­ture the ap­pli­ca­tion, im­ple­ment the core in­ter­ac­tions, and con­tinue re­fin­ing the re­sult through sev­eral rounds of feed­back.

On longer tra­jec­to­ries, we also started to see more self-test­ing and ver­i­fi­ca­tion, with the model check­ing its own work be­fore mov­ing on.

Grok 4.6 pro­duces stronger first passes on vi­sual and in­ter­ac­tive pro­jects than we typ­i­cally saw with Grok 4.5. Given a con­crete prod­uct idea, it is able to es­tab­lish struc­ture and vi­sual lan­guage for an ap­pli­ca­tion in one pass. This has made it es­pe­cially use­ful for pro­jects where the fastest route to a good re­sult was to be­gin with some­thing sub­stan­tial and then it­er­ate in the loop.

Safety and ca­pa­bil­i­ties

Grok 4.6’s safe­guards have been im­proved and cal­i­brated in line with the mod­el’s ca­pa­bil­i­ties.

Our safety stack is de­signed to max­i­mize util­ity and se­cu­rity across le­git­i­mate use cases, al­low­ing Grok 4.6 to be help­ful and safe in do­mains such as vul­ner­a­bil­ity patch­ing, ac­cel­er­at­ing the en­gi­neer­ing de­sign cy­cle, and aug­ment­ing AI re­search.

Our safe­guard eval­u­a­tion work re­flects Grok 4.6’s ex­panded ca­pa­bil­i­ties, with our widest-ever suite of pre-de­ploy­ment test­ing for ca­pa­bil­i­ties and safe­guard cal­i­bra­tion, as well as ex­ten­sive post-de­ploy­ment and third-party test­ing.

Evals

Grok 4.6 High

Grok 4.5 High

GPT-5.6 Sol Max

Fable 5 Max

AA Intelligence Index

61

56

61

62

GDPVal-AA v2

1753

1526

1728

1741

CursorBench v3.2

69.9%

66.7%

67.2%

70.5%

DeepSWE v1.1

65.9%

54%

73%

70%

FrontierCode v1.1 (Extended)

61.3%

56.6%

60.6%

63.6%

APEX-Agents

57.5%

47.1%

56.7%

59.2%

Terminal-Bench v3.0

26%

15.7%

34.6%

34.1%

APEX-SWE

56.4%

53.6%

—

58.8%

AA-Briefcase

1577

1313

1502

1574

Harvey LAB (Vals)

15.8%

12.9%

2.5%

11.3%

Best score per eval­u­a­tion in bold. Third-party model scores are the best of self-re­ported or pub­licly avail­able re­sults.

Get started with Grok 4.6

Grok 4.6 is avail­able to­day in Cursor and Grok Build. It’s also avail­able in the API and other part­ners like OpenRouter, Vercel, and Cloudflare.

Pricing starts at $2 per mil­lion in­put to­kens and $6 per mil­lion out­put to­kens. Additionally, there is a fast vari­ant which is twice the price.

We’re of­fer­ing 2x in­cluded us­age in­side Grok Build and Cursor for the first week so you can start try­ing 4.6 im­me­di­ately.

Create an API Key

Start build­ing with Grok 4.6 to­day via the SpaceXAI API.

API Docs

Read the docs and in­te­grate Grok 4.6 into your stack.

Try it in Grok Build for free

Get started to­day at x.ai/​build.

Pair it with a lo­cal cod­ing agent.

Run llama serve, in­stall the pi-llama plu­gin and launch Pi. It will au­to­mat­i­cally dis­cover your lo­cal model. No con­fig, no API keys. Files stay on your ma­chine, re­quests never leave it.

# 1. Serve a model llama serve

# 2. Install the pi-llama plu­gin pi in­stall git:github.com/​hug­ging­face/​pi-llama

# 3. Run Pi, every­thing is set pi

Optimized for any hard­ware.

From your lap­top to a clus­ter, llama.cpp runs on what­ever you have. Same bi­nary, same mod­els, same hand-tuned ker­nels for every GPU and CPU.

Apple Silicon

M Ultra

RTX 5090

CPU

Jetson

H100

MI300

RTX 4090

A100

M Pro

M Max

DGX Spark

T4

Radeon RX

B200

Intel Arc

RTX 3090