Our mis­sion is to build AI that ex­tends hu­man will and judg­ment. We have de­vel­oped a plat­form that lets any­one cus­tomize mod­els, pre­viewed an AI sys­tem built for in­ter­ac­tive col­lab­o­ra­tion, and pub­lished novel re­search. Today we are ad­vanc­ing our mis­sion by re­leas­ing a model we trained from scratch with the full weights avail­able, so that peo­ple can make it their own.

Our model, called Inkling, is a Mixture-of-Experts trans­former with 975B to­tal pa­ra­me­ters, 41B ac­tive. It sup­ports a con­text win­dow of up to 1M to­kens. It was pre­trained on 45 tril­lion to­kens of text, im­ages, au­dio and video. It is the first in a fam­ily of mod­els of dif­fer­ent sizes: along­side it we are shar­ing a pre­view of Inkling-Small, a lighter-weight model with 12B ac­tive pa­ra­me­ters, trained with a sim­i­lar recipe, that achieves strong per­for­mance with even lower cost and la­tency.

Inkling rea­sons na­tively over text, im­ages, and au­dio, and bal­ances cost with per­for­mance through ef­fi­cient and con­trol­lable think­ing ef­fort. We trained it to be a broad, bal­anced foun­da­tion model: strong across many do­mains, flex­i­ble enough to adapt. Inkling is not the strongest over­all model avail­able to­day, open or closed. Instead, a com­bi­na­tion of qual­i­ties makes it a good open-weights base for cus­tomiza­tion: mul­ti­modal ca­pa­bil­i­ties, ef­fi­cient think­ing, and avail­abil­ity on Tinker for fine-tun­ing. Inkling is just the start: our first re­lease in a model fam­ily we will con­tinue to build on.

We want to make cus­tomiza­tion ac­ces­si­ble for more use cases, so Inkling is avail­able for fine-tun­ing on Tinker to­day. Picking the right base model to fine-tune is a qual­i­ta­tive judg­ment that com­bines mea­sur­able bench­marks with the unique feel of a model that comes from play­ing with it. To en­able the lat­ter we’re adding the Inkling Playground in the Tinker con­sole: a de­vel­oper-fac­ing in­ter­face for chat­ting with Inkling.

To show what cus­tomiza­tion means in prac­tice, we asked Inkling to fine-tune it­self. Using Tinker, the model wrote its own fine-tun­ing job, ran it, and eval­u­ated the re­sult:

Build · inkling · tin­ker-prod

Capabilities

Real-world ap­pli­ca­tions re­quire mod­els with a wide range of ca­pa­bil­i­ties that can be com­bined and im­proved with fine-tun­ing. We show­case what Inkling can do and how it mea­sures up on im­por­tant qual­i­ties such as trust­wor­thi­ness and safety.

Generalist model

Inkling is de­signed to be broad. We trained it across agen­tic, rea­son­ing, cod­ing, in­struc­tion-fol­low­ing, fac­tu­al­ity, vi­sion, and au­dio tasks, rather than nar­rowly op­ti­miz­ing for one do­main. That breadth mat­ters for cus­tomiza­tion and real-world use: dif­fer­ent users need mod­els that can adapt to very dif­fer­ent work­flows, not just ex­cel on bench­marks.

Spider chart com­par­ing Inkling, Nemotron 3 Ultra, GLM 5.2, GPT 5.6 Sol, and Claude Fable 5 on ten eval­u­a­tions scored from zero to one hun­dred. Inkling is shown with the heav­ier cobalt line. Evaluations with­out a re­ported model score are plot­ted at zero. Hover an eval­u­a­tion to com­pare every mod­el’s score.

Agentic cod­ing and tool use

A strong base for fine-tun­ing needs to flex­i­bly solve a wide va­ri­ety of tasks with agen­tic tool use. Inkling scores well among open-weights mod­els on most agen­tic bench­marks.

We trained Inkling to run in­side a va­ri­ety of cod­ing and agent har­nesses, and we ran­dom­ized the tool set and schema dur­ing train­ing to re­duce sen­si­tiv­ity to any par­tic­u­lar one. Inkling’s con­trol­lable think­ing ef­fort, de­scribed in the next sec­tion, can be set from within the har­ness.

Below are a few demos show­cas­ing Inkling’s agen­tic cod­ing and tool use and the ar­ti­facts it cre­ates.

One-shot web app with em­bed­ded browser use

Inkling built a func­tional web app in a sin­gle shot, then pow­ers an em­bed­ded AI as­sis­tant that can op­er­ate the web app in­ter­face through nat­ural lan­guage in­struc­tions.

Design Arena

Inkling was eval­u­ated on Design Arena’s Agentic Web Dev leader­board, where blinded hu­man eval­u­a­tors com­pare gen­er­ated web apps head to head. It ranks among the strongest open-weights mod­els.

Claude Sonnet 5

1333

Claude Fable 5

1329

Claude Opus 4.8

1285

GLM 5.2

1275

Grok 4.5

1271

GPT-5.6 Sol

1260

Inkling

1257

Claude Opus 4.6

1257

Gemini 3.5 Flash

1254

Kimi K2.6

1249

Claude Sonnet 4.6

1237

Kimi K2.7 Code

1234

GLM 5.1

1233

Claude Opus 4.5

1212

Grok 4.20 Reasoning

1203

Gemini 3.1 Pro Preview

1187

Grok 4.3

1185

Kimi K2.5 (Thinking)

1185

Cohesively styled ar­ti­facts

Inkling cre­ates multi-page ar­ti­facts with pre­cise in­struc­tion fol­low­ing, ac­cu­rate in­for­ma­tion, and co­he­sive styling and de­sign through­out.

Multiplayer game cre­ated through long re­fine­ment loop

Inkling re­fined an on­line snake game through 40 it­er­a­tions of feed­back from GPT Codex serv­ing as a re­viewer. The abil­ity to sus­tain a long process of re­fine­ment and im­prove from feed­back is cru­cial to cre­at­ing the best col­lab­o­ra­tive work.

Controllable think­ing ef­fort

Test-time scal­ing and prob­lem-solv­ing are the core ca­pa­bil­ity of every model, but that ca­pac­ity is hard to cap­ture with a sin­gle num­ber. Developers fine-tun­ing mod­els for a spe­cial­ized task care as much about ef­fi­ciency as about the max-ef­fort per­for­mance on a pub­lic bench­mark. Cost and la­tency are of­ten bind­ing con­straints in real-world ap­pli­ca­tions, and low la­tency in par­tic­u­lar is cru­cial for en­abling col­lab­o­ra­tion and im­prove­ment through it­er­a­tion.

Inkling (effort sweep) GLM-5.2 Kimi K2.6 Nemotron 3 Ultra Kimi K2.5 GPT-OSS (high)

Inkling sup­ports con­trol­lable think­ing ef­fort, al­low­ing you to bal­ance per­for­mance with to­ken ef­fi­ciency. The chart above shows the ef­fort/​per­for­mance curve of Inkling as well as other open-weights mod­els on a range of bench­marks: Terminal Bench 2.1 for agen­tic cod­ing, HLE for ad­vanced rea­son­ing, and IFBench for in­struc­tion fol­low­ing. Inkling spends one third as many to­kens to achieve the same per­for­mance as Nemotron 3 Ultra on Terminal Bench. Cost and la­tency mat­ter for a model that you run mil­lions of times and as part of longer work­flows; look­ing at the full cost curve al­lows de­vel­op­ers to choose the best model for each use case.

Multimodality

A ma­jor goal of Inkling’s de­sign is to serve as the back­ground rea­son­ing model in the in­ter­ac­tion mod­els sys­tem we re­cently in­tro­duced. Interaction mod­els en­able the user to col­lab­o­rate nat­u­rally, us­ing voice and vi­sion in real time. This re­quires a model na­tively trained for broad mul­ti­modal ca­pa­bil­i­ties.

Audio and vi­sion bench­marks against spe­cial­ist omni mod­els (open- and closed-weight), re­ported at ef­fort=0.99.

The mul­ti­modal com­po­nents were trained from scratch on gen­eral-do­main data. We opted for an en­coder-free ar­chi­tec­ture for au­dio and vi­sion in­puts, con­sis­tent with the in­ter­ac­tion model de­sign. Audio sig­nals are in­put as dMel spec­tro­grams­d­Mel: Speech Tokenization made Simple (Richard He Bai et al, 2024), while im­ages are en­coded as patches of 40x40 pix­els us­ing a four-layer hMLPThree things every­one should know about Vision Transformers (Hugo Touvron et al, 2022). Both are trans­formed via a light-weight em­bed­ding layer and processed jointly with text to­kens.

Inkling tran­scribes speech, fol­lows spo­ken in­struc­tions, an­swers ques­tions about record­ings, and rea­sons over longer-form au­dio. These ca­pa­bil­i­ties place it among the strongest open-weights au­dio mod­els on VoiceBench, MMAU, and AudioMC. For vi­sion, Inkling ac­cepts im­ages as in­put and can de­scribe vi­sual con­tent, an­swer ques­tions, and per­form in-depth rea­son­ing based on the pro­vided vi­sual in­for­ma­tion. It demon­strates strong per­for­mance on charts, di­a­grams, and math­e­mat­i­cal vi­sual rea­son­ing tasks. During in­fer­ence, Inkling can also lever­age a Python tool to sup­port im­age un­der­stand­ing through op­er­a­tions such as zoom­ing and crop­ping, while seam­lessly in­te­grat­ing vi­sual rea­son­ing with code-based rea­son­ing.

As our first re­lease, Inkling es­tab­lishes a ro­bust mul­ti­modal foun­da­tion for fu­ture work. We ex­pect its mul­ti­modal ca­pa­bil­i­ties to con­tinue im­prov­ing as we ex­pand the model and train­ing pipeline in sub­se­quent it­er­a­tions.

Epistemics

We trained Inkling for cal­i­bra­tion, in­struc­tion fol­low­ing, and re­sis­tance to cen­sor­ship, which we re­fer to col­lec­tively as the mod­el’s epis­temics.

Getting the facts right re­quires more than mem­o­riz­ing a large cor­pus of knowl­edge. A use­ful model must be well-cal­i­brated, ex­press­ing the right amount of con­fi­dence in its an­swers — in­clud­ing on ques­tions which aren’t yet set­tled. The lat­ter is a cru­cial ca­pa­bil­ity for pre­dic­tion and fore­cast­ing, an im­por­tant use case where fine-tuned mod­els have shown rapid im­prove­ment in re­cent months, out­per­form­ing fron­tier LLMs.

Results were ob­tained dur­ing test­ing be­tween June 30 and July 13, 2026 on a dif­fer­ent check­point of Inkling than the one re­leased.

Forecasting re­quires in­te­grat­ing mul­ti­ple sources of in­for­ma­tion into a cal­i­brated prob­a­bil­ity, a core skill for a model users can trust. A model that’s con­fi­dent in every an­swer it gives, in­clud­ing when it’s miss­ing info and con­fab­u­lates, forces the user to dou­ble-check every­thing. A model that gives the ap­pro­pri­ate mea­sure of con­fi­dence is use­ful across more real-world do­mains where in­for­ma­tion is of­ten con­flict­ing, un­re­li­able, or hard to find. We trained for cal­i­bra­tion with RL against proper scor­ing rules on a large cor­pus of re­solved real-world ques­tions.

The sec­ond com­po­nent of a trust­wor­thy model is in­struc­tion fol­low­ing, in­clud­ing on hard-to-ver­ify, com­plex queries. We did RL with two au­to­mated graders: a rubric grader and claims grader. The first grader scores each re­sponse against a check­list of what a good an­swer should con­tain. Rubrics can pe­nal­ize er­rors in prin­ci­ple, but in prac­tice they em­pha­size re­call and can be hacked by mod­els spray­ing plau­si­bly rel­e­vant facts hop­ing to match rubric items. The claims grader ver­i­fies each fac­tual claim in the re­sponse, pe­nal­iz­ing claims that don’t check out. It per­forms agen­tic web search for claim ver­i­fi­ca­tion, not re­ly­ing solely on its own knowl­edge. Together, the two graders im­prove help­ful­ness and re­duce hal­lu­ci­na­tion at the same time, rather than trad­ing one for the other.

These re­wards don’t di­rectly tar­get cal­i­brated un­cer­tainty in long-form re­sponses, so we added tar­geted datasets that do. The largest is short-form fac­tual QA with ab­sten­tion-aware re­wards: an­swer­ing only pays off when the model is likely to be right, so the op­ti­mal pol­icy is to an­swer when con­fi­dent and oth­er­wise say “I don’t know” or give a hedged best guess. Some prompts en­cour­age or for­bid hedg­ing, teach­ing the model to fol­low the user’s pref­er­ence for a forced guess ver­sus a cal­i­brated non-an­swer.

Finally, we trained Inkling to an­swer di­rectly on top­ics that may be sub­ject to cen­sor­ship. Cognition eval­u­ated the model on their Propaganda and Censorship EvalThe Cognition Team, “Measuring the Trustworthiness of Open-Source-Derived Models,” 2026., and it ex­hib­ited strong pat­terns of cen­sor­ship non-com­pli­ance.

Safety

We trained Inkling to an in­ter­nal spec of safe model be­hav­ior across all modal­i­ties. We then com­mis­sioned ex­ter­nal safety testers to ver­ify the re­sults.

We eval­u­ated Inkling’s safety in sev­eral ar­eas. For dan­ger­ous ca­pa­bil­i­ties — CBRN, cy­ber, and loss of con­trol — we ran in­ter­nal eval­u­a­tions and en­listed ex­ter­nal testers. We at­tended to hu­man-AI threat vec­tors, in­clud­ing syco­phancy, vul­ner­a­ble users, and harm­ful ma­nip­u­la­tion, us­ing in­ter­nal eval­u­a­tions and ex­ter­nal testers.

Inkling shows the strongest built-in safe­guards of any open-weights model we com­pared on FORTRESS, a bench­mark that tests re­fusal of re­quests re­lated to weapons and vi­o­lence along­side be­nign look-alike queries. Inkling re­fused more harm­ful re­quests with­out over-re­fus­ing be­nign analogs. Inkling scores above 98% on StrongREJECT — a re­fusal test of un­am­bigu­ous harm­ful re­quests — in line with other open and closed-weights mod­els.

Safety is cru­cial for open-weights mod­els. We’re con­tin­u­ing to study safety be­hav­ior and ca­pa­bil­ity up­lift in cus­tomiz­able mod­els, in­clud­ing how safety be­hav­ior is im­pacted by fine-tun­ing on Tinker.

Benchmarking Inkling

We bench­mark Inkling on a broad range of ca­pa­bil­i­ties. All evals are run at ef­fort 0.99 and tem­per­a­ture 1.0. All cod­ing evals run with 256K max-to­ken tra­jec­tory limit.

To im­prove con­sis­tency, we rely on ex­ter­nally re­ported eval­u­a­tions for both in­ter­nal and ex­ter­nal mod­els when ap­plic­a­ble. Specifically, we use the score re­ported by Artificial Analysis for the fol­low­ing evals: Humanity’s Last Exam, GPQA Diamond, GDPVal, Tau 3 Banking, AA Omniscience, MMMU Pro.

*SWEBench Verified: Inkling num­bers are re­ported us­ing a bash-only har­ness. We use self-re­ported num­bers for ex­ter­nal mod­els.*Ter­mi­nal Bench 2.1: Inkling num­bers are re­ported us­ing an in­ter­nal cod­ing har­ness. A small num­ber of so­lu­tions were found to be con­t­a­m­i­nated from web search and were as­signed a score of 0. We use self-re­ported num­bers for ex­ter­nal mod­els where avail­able. Otherwise, we re­port per­for­mance us­ing our in­ter­nal har­ness.†Au­dio MC: Other mod­els were eval­u­ated in­ter­nally since they are not on the of­fi­cial leader­board.†VoiceBench: VoiceBench uses rule-based, hard-coded string match­ing for grad­ing, mak­ing the eval­u­a­tion sen­si­tive to out­put-for­mat­ting dif­fer­ences. We there­fore added a sys­tem mes­sage in­struct­ing mod­els to fol­low the ex­pected an­swer for­mat.†CharXiv RQ with tools: We bench­marked Claude Fable 5 and GPT 5.6 Sol (max/xhigh) us­ing our in­ter­nal Python har­ness.

The mak­ing of Inkling

Architecture

Inkling is a Mixture-of-Experts Transformer with a hand­ful of de­par­tures from the com­mon recipe, each cho­sen for ef­fi­ciency and long-con­text per­for­mance.

The MoE de­sign largely fol­lows DeepSeek-V3. Each MoE layer con­tains 256 routed ex­perts and 2 shared ex­perts, with 6 routed ex­perts ac­tive per to­ken. Inkling uses a sig­moid-based router with an aux­il­iary-loss-free load-bal­anc­ing bias. The scores of the se­lected routed ex­perts and the shared ex­perts are nor­mal­ized jointly and used to weight their com­bined out­puts.

For at­ten­tion, we in­ter­leave slid­ing-win­dow and global lay­ers at a 5:1 ra­tio with 8 KV heads. We find that en­cod­ing po­si­tion with a rel­a­tive po­si­tional em­bed­ding­Self-At­ten­tion with Relative Position Representations (Peter Shaw et al, 2018)Music Transformer (Cheng-Zhi Anna Huang et al, 2018) per­forms bet­ter and ex­trap­o­lates bet­ter to longer se­quences than the more widely adopted Rotary Positional Embedding (RoPE). We also ap­ply short con­vo­lu­tions at two points — af­ter the key and value pro­jec­tions in each at­ten­tion layer, and on the at­ten­tion and MLP resid­ual branch out­puts be­fore they re­join the main resid­ual stream.

Training

Inkling was pre­trained on 45 tril­lion to­kens from a va­ri­ety of con­tent types, in­clud­ing text, im­ages, au­dio and video. We trained Inkling with a hy­brid op­ti­miza­tion strat­egy — Muon for large ma­trix weights, Adam for other pa­ra­me­ters — and hy­per­pa­ra­me­ter sched­ules in­spired by our pre­vi­ous re­search on mod­u­lar man­i­folds. We cou­pled the weight de­cay strength to the square of the learn­ing rate, which we found kept the over­all size of the model weights sta­ble across train­ing hori­zon­sSee also Kosson et al. (2023) and Defazio (2025)..

We post-trained Inkling on a broad dis­tri­b­u­tion of math, agen­tic code & tool use, au­dio, im­age, chat, and safety do­mains. To boot­strap post-train­ing, we ran an ini­tial SFT on syn­thetic data gen­er­ated by open-weights mod­els in­clud­ing Kimi K2.5. The boot­strap ac­counts for a small frac­tion of com­pute, with the ma­jor­ity be­ing em­ployed for large-scale RL on syn­thetic and hu­man-cre­ated en­vi­ron­ments.

Inkling was our first ma­jor train­ing ef­fort and was trained on NVIDIA GB300 NVL72 sys­tems. Future mod­els will fur­ther push the scale of com­pute across pre-train­ing, post-train­ing and RL.

RL at scale

We re­lied on large-scale asyn­chro­nous RL to shape model be­hav­ior and im­prove its rea­son­ing and over­all per­for­mance. The chart be­low shows the mod­el’s score on a held-out ag­gre­gate of rea­son­ing evals such as AIME, HLE, GPQA, and oth­ers. We scaled RL to over 30M roll­outs, with sta­ble train­ing sus­tained over two long con­tin­u­ous runs. Reasoning per­for­mance im­proved log-lin­early through­out the en­tire process, re­sult­ing in a sig­nif­i­cant in­crease over­all.

We spec­i­fied the mod­el’s ef­fort level on dif­fer­ent sam­ples by chang­ing the sys­tem mes­sage and ad­just­ing the per-to­ken cost. This caused the model to use a dif­fer­ent amount of to­kens in dif­fer­ent roll­outs and learn the abil­ity to con­trol think­ing ef­fort.

We also ob­served an emer­gent shift in the rea­son­ing style over the course of RL train­ing. The chain of thought be­came more con­cise over time, drop­ping gram­mat­i­cal over­head while re­main­ing com­pre­hen­si­ble and leav­ing the fi­nal re­sponse un­af­fected. This was­n’t tar­geted by the re­ward — ef­fi­ciency alone drove the com­pres­sion. A sim­i­lar ef­fect was also re­cently noted by the Cognition team in the process of train­ing SWE-1.7The Cognition Team, “SWE-1.7: Frontier Intelligence at a Fraction of the Cost.”. Below is an ex­am­ple of how Inkling’s chain of thought on the same math prob­lem evolved with RL:

Early in RL ver­bose, gram­mat­i­cal

For help please visit help.ft.com. We apol­o­gise for any in­con­ve­nience.

The fol­low­ing in­for­ma­tion can help our sup­port team to re­solve this is­sue.

Please en­able JS and dis­able any ad blocker

Saved searches

Use saved searches to fil­ter your re­sults more quickly

Table of ContentsGoogle sup­ports the im­mi­gra­tion en­force­ment sup­ply chain­But how could I do any­thing about it?Talk­ing to Jeff DeanThe Pentagon tries to in­tim­i­date AnthropicI wanted to mo­bi­lize the AI lu­mi­nar­ies at the con­fer­enc­eTalk­ing to Bengio and StuartStuart closes out iaseai­In­ter­na­tional Association of Silence on the Ethics of AITrying to stop Google from sign­ing­Build­ing in­ter­nal cost for GoogleJeff Dean, you’re our only hope­J­eff signs an am­i­cus brief sup­port­ing AnthropicSenior man­age­ment in­sisted Google would­n’t cavePrepar­ing for lunch with Jeff DeanI arranged so­cial sup­port­The art of the dealMy lunch with JeffSearching for an­other path to im­pactNo one is re­spond­ing, so why not just DM the ceo?My Framework goes un­eval­u­at­ed­Google qui­etly signs the dealDemis in­sists Google’s AI prin­ci­ples “haven’t changed”We can “work with Western democ­ra­cies” to “beat China” with­out giv­ing in to every de­mand Trump makes­Build­ing a world-re­shap­ing tech­nol­ogy on per­sonal trustRe­flec­tion­sHow can a pledge-signer re­main at gdm?The weight of ethic­sWhat are the AI lu­mi­nar­ies do­ing?Why did­n’t Jeff put his foot down?Break­ing free of rolesWhy I left Google DeepMindSimilar post­sAp­pen­dix: Anticipated ques­tion­sWhat if the peo­ple you cri­tique were sav­ing their po­lit­i­cal cap­i­tal?Maybe they thought you weren’t worth their time; you aren’t en­ti­tled to their helpEv­ery per­son should­n’t have to speak out about every is­sueEven if Google had adopted your Framework, the Pentagon would have re­fused­Does this have any im­pact on ex­is­ten­tial risk from AI?Appendix: “Don’t worry, it’s only api ac­cess”Foot­notes

Table of Contents

Google sup­ports the im­mi­gra­tion en­force­ment sup­ply chain­But how could I do any­thing about it?

But how could I do any­thing about it?

Talking to Jeff Dean

The Pentagon tries to in­tim­i­date AnthropicI wanted to mo­bi­lize the AI lu­mi­nar­ies at the con­fer­enc­eTalk­ing to Bengio and StuartStuart closes out iaseai­In­ter­na­tional Association of Silence on the Ethics of AI

I wanted to mo­bi­lize the AI lu­mi­nar­ies at the con­fer­ence

Talking to Bengio and Stuart

Stuart closes out iaseai

International Association of Silence on the Ethics of AI

Trying to stop Google from sign­ing­Build­ing in­ter­nal cost for GoogleJeff Dean, you’re our only hope­J­eff signs an am­i­cus brief sup­port­ing Anthropic

Building in­ter­nal cost for Google

Jeff Dean, you’re our only hope

Jeff signs an am­i­cus brief sup­port­ing Anthropic

Senior man­age­ment in­sisted Google would­n’t cave

Preparing for lunch with Jeff DeanI arranged so­cial sup­port­The art of the deal

I arranged so­cial sup­port

The art of the deal

My lunch with JeffSearching for an­other path to im­pact

Searching for an­other path to im­pact

No one is re­spond­ing, so why not just DM the ceo?My Framework goes un­eval­u­ated

My Framework goes un­eval­u­ated

Google qui­etly signs the deal

Demis in­sists Google’s AI prin­ci­ples “haven’t changed”We can “work with Western democ­ra­cies” to “beat China” with­out giv­ing in to every de­mand Trump makes­Build­ing a world-re­shap­ing tech­nol­ogy on per­sonal trust

We can “work with Western democ­ra­cies” to “beat China” with­out giv­ing in to every de­mand Trump makes

Building a world-re­shap­ing tech­nol­ogy on per­sonal trust

ReflectionsHow can a pledge-signer re­main at gdm?The weight of ethic­sWhat are the AI lu­mi­nar­ies do­ing?Why did­n’t Jeff put his foot down?Break­ing free of roles

How can a pledge-signer re­main at gdm?

The weight of ethics

What are the AI lu­mi­nar­ies do­ing?

Why did­n’t Jeff put his foot down?

Breaking free of roles

Why I left Google DeepMind

Similar posts

Appendix: Anticipated ques­tion­sWhat if the peo­ple you cri­tique were sav­ing their po­lit­i­cal cap­i­tal?Maybe they thought you weren’t worth their time; you aren’t en­ti­tled to their helpEv­ery per­son should­n’t have to speak out about every is­sueEven if Google had adopted your Framework, the Pentagon would have re­fused­Does this have any im­pact on ex­is­ten­tial risk from AI?

What if the peo­ple you cri­tique were sav­ing their po­lit­i­cal cap­i­tal?

Maybe they thought you weren’t worth their time; you aren’t en­ti­tled to their help

Every per­son should­n’t have to speak out about every is­sue

Even if Google had adopted your Framework, the Pentagon would have re­fused

Does this have any im­pact on ex­is­ten­tial risk from AI?

Appendix: “Don’t worry, it’s only api ac­cess”

Footnotes

In January, Department of Home­land Security (dhs) of­fi­cers killed at least two peo­ple. In both cases, a fed­eral agent grasped his gun, aimed it at a peace­ful cit­i­zen, and shot them dead.

I learned that Google sells its Cloud ser­vices to the rel­e­vant agen­cies within dhs. I thought that was wrong. Federal agents should not be able to kill cit­i­zens in the street. I set out to find the most ef­fec­tive way to push my com­pany to stop serv­ing these agen­cies. My divestment cam­paign quickly broad­ened into an at­tempt to pre­vent Google from sign­ing an un­eth­i­cal mil­i­tary AI deal, as the Pentagon started pres­sur­ing AI providers into mil­i­tary AI deals with no re­stric­tions against use for killer ro­bots or mass sur­veil­lance.1

I wanted AI ethics com­mit­ments to hold un­der pres­sure. In particular, I wanted Google DeepMind (gdm) to main­tain its ex­ist­ing com­mit­ment against sup­port­ing killer ro­bots. Over sev­eral months, I asked many peo­ple to act. I asked se­nior peo­ple⁠—re­spected peo­ple⁠—peo­ple with rep­u­ta­tions sil­vered by their con­cern about AI ethics and safety. Nearly all de­clined.

Take Stuart Russell, a fa­mous AI researcher who spent over a decade cru­sad­ing against au­tonomous weapons. I worked at his lab for years. At a con­fer­ence, on-stage, he agreed to push his or­ga­ni­za­tion to make a state­ment sup­port­ing AI providers against gov­ern­ment co­er­cion and promised a poll of its mem­bers. The state­ment and poll both van­ished.

Or take Jeff Dean, who is Google’s Chief Scientist and the co-lead of Google’s Gem­ini AI project. In 2018, Jeff signed a pledge to never sup­port the de­vel­op­ment or use of killer ro­bots. I got Jeff to pub­licly and boldly co-sign an am­i­cus brief (where out­siders weigh in to sway a law­suit) back­ing Anthropic against the Pentagon. But I also asked him to use his im­mense lever­age to stop Google from mak­ing its own un­eth­i­cal deal with the mil­i­tary, and I don’t think he did. He remains at Google de­spite his pledge.

I wrote a 25-⁠page pro­posal con­tain­ing con­tract lan­guage and over­sight mech­a­nisms. Military- and sur­veil­lance-law ex­perts praised the pro­posal, which rep­re­sented a prin­ci­pled coun­terof­fer Google could have stood by. I sent the pro­posal to Demis Hassabis (gdm’s ceo) who routed it to se­nior pol­icy staff, only for the pro­posal to wilt un­at­tended un­til Google signed a deal.

Senior man­age­ment had in­sisted that Google would­n’t sign. I disagreed with them, but they largely ig­nored my warn­ings. While I may have in­creased the Pentagon’s hesitation around the deal, Google still signed a deal hand­ing over their AI without re­stric­tions against killer ro­bots or mass AI spying. Google’s contract re­stric­tions were even weaker than Ope­nAI’s. At that point, I couldn’t stay at Google in good con­science, so I left.

This es­say tells the story of why I left Google DeepMind. It is also the story of some­thing larger: how pow­er­ful peo­ple and in­sti­tu­tions failed, one af­ter an­other, to keep their AI ethics promises in the face of pres­sure.

On pri­vate com­mu­ni­ca­tions Throughout this es­say, I never quote any­one’s pri­vate words with­out per­mis­sion. Where pri­vate con­ver­sa­tions mat­ter, I characterize them min­i­mally. I otherwise keep to my own ac­tions, pub­lic in­for­ma­tion, and of­fi­cial com­mu­ni­ca­tions. You can’t ver­ify my char­ac­ter­i­za­tions, so weight them ac­cord­ingly. This min­i­miza­tion has cut con­tent which would have sup­ported my ar­gu­ments.

On pri­vate com­mu­ni­ca­tions

Throughout this es­say, I never quote any­one’s pri­vate words with­out per­mis­sion. Where pri­vate con­ver­sa­tions mat­ter, I characterize them min­i­mally. I otherwise keep to my own ac­tions, pub­lic in­for­ma­tion, and of­fi­cial com­mu­ni­ca­tions. You can’t ver­ify my char­ac­ter­i­za­tions, so weight them ac­cord­ingly. This min­i­miza­tion has cut con­tent which would have sup­ported my ar­gu­ments.

Google sup­ports the im­mi­gra­tion en­force­ment sup­ply chain

January 26th, 2026

After Alex Pretti’s death, I was de­ter­mined to take ef­fec­tive ac­tion. To determine how to re­duce harm from dhs, I researched Big Tech’s entanglement. Certainly, Microsoft and Amazon have larger in­volve­ment, but I was sur­prised to learn of Google’s ex­po­sure.

Google’s con­tracts with DHS Alex Pretti, 2024. The dhs 2025 AI Use Case Inventory lists Google among the GenAI providers used to “im­prove the op­er­a­tional ef­fi­ciency” of dhs. Google sells Cloud ser­vices to ice through third par­ties like ITC Federal.2 On October 3rd, 2025, Google delisted apps that warned of ice ac­tiv­ity. Google vol­un­tar­ily handed a stu­dent pro­test­er’s ac­count to ice with­out no­tice, break­ing their Terms of Ser­vice promise to “send an email to the user ac­count be­fore dis­clos­ing in­for­ma­tion [to the gov­ern­ment].”3

Google’s con­tracts with DHS

The dhs 2025 AI Use Case Inventory lists Google among the GenAI providers used to “im­prove the op­er­a­tional ef­fi­ciency” of dhs.

Google sells Cloud ser­vices to ice through third par­ties like ITC Federal.2

On October 3rd, 2025, Google delisted apps that warned of ice ac­tiv­ity.

Google vol­un­tar­ily handed a stu­dent pro­test­er’s ac­count to ice with­out no­tice, break­ing their Terms of Ser­vice promise to “send an email to the user ac­count be­fore dis­clos­ing in­for­ma­tion [to the gov­ern­ment].”3

But how could I do any­thing about it?

The stereo­typ­i­cal ac­tivist ac­tion is to make a pe­ti­tion. But Google had al­ready ig­nored a large pe­ti­tion on this is­sue. Plus, Google’s executives likely hard­ened their com­pany against stereo­typ­i­cal or­ga­niz­ing tac­tics. Sit-ins, strikes, even a mass of Google en­gi­neers quit­ting: I deemed all of them in­ef­fec­tive (if I could even pull them off).

As I strate­gized, I judged that Google would not care about 100 ran­dom re­search en­gi­neers quit­ting. No, in the AI industry, tal­ent is top-heavy and teams are dri­ven by a few hard-to-re­place stars. I didn’t need to co­or­di­nate 100 en­gi­neers. Perhaps I just needed to co­or­di­nate 10.

I’d followed the news and guessed that Sundar Pichai (Google’s ceo) was more of a busi­ness­man than a “make me a big speech about ethics and I’ll change my mind” kind of guy. But if a few hard-to-re­place peo­ple were ready to walk, that would mat­ter for the busi­ness, so Sun­dar might lis­ten.

That’s when I remembered read­ing Jeff Dean tweet­ing about how bad ice was, retweet­ing Anne Frank quotes. Maybe I didn’t even need 10 en­gi­neers, I just needed one.

This is ab­solutely shame­ful. Agents of a fed­eral agency un­nec­es­sar­ily es­ca­lat­ing, and then ex­e­cut­ing a de­fense­less cit­i­zen whose of­fense ap­pears to be us­ing his cell phone cam­era. Every per­son re­gard­less of po­lit­i­cal af­fil­i­a­tion should be de­nounc­ing this.

2458.4k

Jeff Dean retweeted

“Terrible things are hap­pen­ing out­side. Poor help­less peo­ple are be­ing dragged out of their homes. Families are torn apart.Men, women, and chil­dren are sep­a­rated. Children come home from school to find that their par­ents have dis­ap­peared.”

Diary of Anne FrankJanuary 13, 1943

54834.8k

Jeff is con­sid­ered a saint at Google. He was Google’s 30th em­ployee, de­vel­oped key al­go­rithms, and is known as a man of prin­ci­ple. A common joke: it’s eas­ier for Jeff Dean’s resume to list what he has­n’t achieved than what he has. He’s Google’s Chief Scientist and a co-lead of Google’s Gem­ini ef­fort. A Jeff de­par­ture would be a dis­as­ter for the com­pany.

But if Jeff cared so much and had so much lever­age, why was Google in these ice con­tracts in the first place? Of course, you can’t be Chief Scientist and con­stantly get what you want by threat­en­ing to quit. But I still felt con­fused.

Talking to Jeff Dean

February 9th, 2026

At first I thought about who could put me in touch with him. But (and this is a good gen­eral les­son) if you want to talk to some­one about some­thing, you can al­ways just ask them!

I told Jeff that I respected him for speak­ing out, that I wanted Google to di­vest from the dhs sup­ply chain. I asked if he shared these goals and, if so, how I could help.

He suggested it’d be rea­son­able for me to email a few guys. Their names: Sundar Pichai (ceo of Google), Demis Hassabis (ceo of Google DeepMind), and Thomas Kurian (ceo of Google Cloud). I thought “sure, Jeff. No problem. I’ll just tell them what I think.”

My email I’m writing as a con­cerned em­ployee at gdm. I re­cently mes­saged Jeff Dean re­gard­ing my con­cerns. He suggested that di­rectly email­ing the three of you was a rea­son­able next step.I have no prob­lem with Google work­ing with law­ful ad­min­is­tra­tions of ei­ther US political party. My concern is not about pol­i­tics, but rather about the events en­abled by Google’s role in the dhs sup­ply chain.I think that ice has gone well be­yond its le­gal man­date to re­move il­le­gal im­mi­grants from the coun­try in an or­derly fash­ion. According to watch­dogs, ice op­er­a­tions fre­quently de­prive tar­gets of due process. These op­er­a­tions reg­u­larly de­tain cit­i­zens in fa­cil­i­ties op­er­at­ing with min­i­mal (or no) le­gal over­sight. Over 1,000 peo­ple are miss­ing from one such lo­ca­tion. At other lo­ca­tions, the aclu re­ports hu­man rights abuses and a se­vere lack of safety for de­tainees.These are not stan­dard, le­git­i­mate en­force­ment ac­tiv­i­ties. These op­er­a­tions are trou­bling from a hu­man rights per­spec­tive and also pose rep­u­ta­tional risk to any ven­dors in­volved.On 1/​28/​26, the dhs posted its 2025 AI Use Case Inventory, which lists Google as one of sev­eral GenAI providers which “improve the op­er­a­tional ef­fi­ciency” of dhs. I urge Google to im­me­di­ately stop work­ing with ice (and dhs more broadly), in­clud­ing via sup­port for third-party in­te­gra­tors fa­cil­i­tat­ing these spe­cific op­er­a­tions. Whether through a di­rect (“prime”) con­tract or through in­ter­me­di­aries (like ITC Federal), Cloud and Gemini must not power these op­er­a­tions.His­tory will judge the tech sec­tor by its in­volve­ment in these events. I love work­ing at Google, and I want to en­sure Google is on the right side of that his­tory.

Alexander Matt Turner Research Scientist

My email

I’m writing as a con­cerned em­ployee at gdm. I re­cently mes­saged Jeff Dean re­gard­ing my con­cerns. He suggested that di­rectly email­ing the three of you was a rea­son­able next step.

I have no prob­lem with Google work­ing with law­ful ad­min­is­tra­tions of ei­ther US political party. My concern is not about pol­i­tics, but rather about the events en­abled by Google’s role in the dhs sup­ply chain.

I think that ice has gone well be­yond its le­gal man­date to re­move il­le­gal im­mi­grants from the coun­try in an or­derly fash­ion. According to watch­dogs, ice op­er­a­tions fre­quently de­prive tar­gets of due process. These op­er­a­tions reg­u­larly de­tain cit­i­zens in fa­cil­i­ties op­er­at­ing with min­i­mal (or no) le­gal over­sight. Over 1,000 peo­ple are miss­ing from one such lo­ca­tion. At other lo­ca­tions, the aclu re­ports hu­man rights abuses and a se­vere lack of safety for de­tainees.

These are not stan­dard, le­git­i­mate en­force­ment ac­tiv­i­ties. These op­er­a­tions are trou­bling from a hu­man rights per­spec­tive and also pose rep­u­ta­tional risk to any ven­dors in­volved.

On 1/28/26, the dhs posted its 2025 AI Use Case Inventory, which lists Google as one of sev­eral GenAI providers which “improve the op­er­a­tional ef­fi­ciency” of dhs. I urge Google to im­me­di­ately stop work­ing with ice (and dhs more broadly), in­clud­ing via sup­port for third-party in­te­gra­tors fa­cil­i­tat­ing these spe­cific op­er­a­tions. Whether through a di­rect (“prime”) con­tract or through in­ter­me­di­aries (like ITC Federal), Cloud and Gemini must not power these op­er­a­tions.

History will judge the tech sec­tor by its in­volve­ment in these events. I love work­ing at Google, and I want to en­sure Google is on the right side of that his­tory.

Alexander Matt Turner

Research Scientist

They never replied. I returned to Jeff and asked for a lunch to dis­cuss con­struc­tive op­por­tu­ni­ties for real change within Google. I told him: “any time, any place. I’ll drive down to Moun­tain View to meet with you.”

At this point, I thought this was where “plan A” would fail. To my sur­prise, he ac­tu­ally ac­cepted, for a lunch a few weeks out.

A lot would hap­pen in that time.

The Pentagon tries to in­tim­i­date Anthropic

February 25th, 2026

The Pentagon wanted the fron­tier AI lab Anthropic to re­move red lines from its ex­ist­ing con­tract: red lines against lethal au­tonomous weapons sys­tems and AI spying / pro­fil­ing. The ul­ti­ma­tum was es­sen­tially “give us your prod­uct or we will des­ig­nate you a sup­ply chain risk.” The gov­ern­ment wanted the AI for “all law­ful use.”

There were two ma­jor prob­lems with that kind of deal.

Independent le­gal ex­perts had pointed out po­ten­tial war crimes com­mit­ted by the Pentagon (like dou­ble-tap strikes on ship­wrecked sur­vivors), though the Pentagon in­sisted those ac­tions were le­gal. Under that kind of “le­gal­ity,” “all law­ful use” po­ten­tially meant “AI enabling war crimes” and “automatically pro­fil­ing dis­si­dents with AI.”

The Pentagon threat­ened a pri­vate com­pany with eco­nomic de­struc­tion. Usually, the gov­ern­ment would say “no thanks, we will find an­other sup­plier who will pro­vide terms we want.” In this case, the gov­ern­ment threat­ened to false­ly4 des­ig­nate an Amer­i­can com­pany as a “supply chain risk,” which would force all mil­i­tary con­trac­tors to stop us­ing An­thropic.

Tl;dr I am slowly learn­ing to live with a se­vere de­pres­sion, my ca­reer is fail­ing, but at least I am not alone.

Tl;dr I am slowly learn­ing to live with a se­vere de­pres­sion, my ca­reer is fail­ing, but at least I am not alone.

When I had my first ever in­tern­ship dur­ing my BSc, my mind was filled with noth­ing but ex­cite­ment. I was fi­nally do­ing the thing I stud­ied years for. I was go­ing to ap­ply what I have learned in a com­pany that will ben­e­fit from the things I know. Sure, I start at the bot­tom of the lad­der, but surely if I do well I will do bet­ter, right?

I con­tin­ued work­ing there as a stu­dent. In hind­sight, I did so in a very de­mo­ti­vated state. I told my­self it was be­cause I was also do­ing school aside from it. I knew deep down that it did­n’t mat­ter what I had to do, be­cause I did­n’t feel mo­ti­vated to do the work I was as­signed.

When you’re a ju­nior, that’s not an is­sue. You’re learn­ing, you’re bound to write bad code.

But this feel­ing did­n’t go away. It was the same setup for my fi­nal in­tern­ship. I started very con­fi­dently, very mo­ti­vated, and af­ter week three I had lost mo­ti­va­tion. For some rea­son it did­n’t stick to me.

… and the mo­ment you get into the real world, do­ing a real job, it’s go­ing to af­fect your per­for­mance. I had two jobs, both I have been fired from. In both of them there was a pat­tern. This was the feed­back I got from both:

Communication is not good. I start things with­out dis­cussing them first. This lead to frus­tra­tions

The work i was as­signed to do gets done very slowly (with no com­mu­ni­cat­ing). People get frus­trated, some more au­di­ble than oth­ers.

The work that I did de­liver was just not of qual­ity. Acceptation en­vi­ron­ments kept go­ing down and cus­tomers got an­gry. This lead to my man­ager and other col­leagues get­ting stressed be­cause they could not trust that my work was work­ing.

I kept telling my­self “well, if X broke be­cause of my change, then we must make sure X can­not have this mis­take”. While i still do not dis­agree, it’s ob­vi­ous I was ig­nor­ing the task I ac­tu­ally needed to do, which was to test this more thor­oughly be­fore I de­ploy.

I al­ways fig­ured oth­ers had sim­i­lar is­sues and worked through it. The older I get, the more I re­al­ize that this is­n’t the case. It’s not like “everyone has the same is­sues I have, and have suc­ceeded to live through it, so I should be able to do so as well”. I kept ask­ing peo­ple ques­tions in the hopes that their an­swers would re­veal that they have been feel­ing a sim­i­lar way. Turns out I’m the only one. I re­al­ize that while oth­ers also have is­sues, they are in no way sim­i­lar.

Then vs. now

I ba­si­cally went from a con­fi­dent want­ing-to-be sys­tems en­gi­neer (that even got a great job of­fer that I knew I was­n’t go­ing to be able to do), to ques­tion­ing every­thing I ever did. As of to­day I seem to fuck up the sim­plest tasks.

At first I blamed the com­pa­nies I worked for. Granted, they’re not per­fect, but what made me re­al­ize I am in the wrong is that oth­ers with sim­i­lar sit­u­a­tions seemed to have been do­ing fine. I blamed my in­ex­pe­ri­ence, but some­one less ex­pe­ri­enced did my job just fine. I blamed the boss for be­ing an id­iot (this never went away, I still think my bosses are id­iots, but it seems to work), but even they saw how bad I was per­form­ing. I blamed my sur­round­ings, which was­n’t very fair. At the end of the day, you can name a dozen things that might af­fect your per­fo­mance, but it’s just not go­ing to cut of.

So some­thing was up with me. Something that I was do­ing was not work­ing for the com­pa­nies I worked for. If I’m hon­est, I still have no idea. My the­ory is dis­ci­pline, but right now I have no men­tal space to work on­this.

The work I do is sloppy. I start the tasks I get as­signed, get far, but I do not fin­ish it and just start work­ing on other things. I multi-task and for­get to put the dots on the I’s and J’s

LLMs came and made this much eas­ier. I could let the LLMs do the mul­ti­task­ing for me. I would scope out a ticket, start work­ing and be done. Obviously LLMs have proven to lead to slop­pier code, be­cause you do not fol­low a path any­more where you need to test the thing you’re work­ing on. It’s not that you don’t need to test any­more, but there is no path that forces you to test it. In the­ory you can ask it to do the task and com­mit what­ever it barfed out.

This could be due to ADD, I am still get­ting tested. Granted, that’s a di­ag­no­sis, not a root cause.

This could be due to ADD, I am still get­ting tested. Granted, that’s a di­ag­no­sis, not a root cause.

I’m sure your first thought is “well, just do X next time and go on with your life”. The prob­lem is that to­mor­row it’ll be some­thing else I screwed up. I do not feel like I have the men­tal space to keep track of every­thing I need to do, and I do not even feel like I am aware of what I’m do­ing any­more. It feels like my mind is about to ex­plode with thoughts.

The now now

I have been di­ag­nosed with a se­vere de­pres­sion, tak­ing flu­ox­e­tine and ox­azepam, and I will be liv­ing on ben­e­fits for now, so I can take time to heal. I would love to work again, but it’s not in the cards for me at this time. I can still do some open-source con­tri­bu­tions, but I do want to make sure that the work I do is ben­e­fi­cial.

I would like to say that, de­spite be­ing neg­a­tive about our health­care (who is­n’t, right?), I am very grate­ful of the help I have re­ceived from my GP, PAPC, as well as my friends and fam­ily who are all there for me. Step one for me was to com­mu­ni­cate with them. I re­alised that a lot of the frus­tra­tions piled on top of my de­pres­sion were due to my lack of com­mu­ni­ca­tion with them. Not say­ing how you’re feel­ing leads to peo­ple mis­un­der­stand­ing you. There’s a lot of re­grets I have in life that could’ve been pre­vented by just say­ing the right things. It’s im­por­tant to think fur­ther than just what you need at that mo­ment, and to see the big­ger pic­ture. The hard­est thing I had to re­al­ize is that some­times things do take time, but that does­n’t mean to just aban­don what­ever it is. It means to fight for it, but to not rush it.

I started last year with post­ing a lot of in­ner thoughts on my blog. I have since deleted all of them (if you re­ally care enough, they’re still in the git his­tory). I had no one to dis­cuss this with (this should’ve be a spe­cial­ist, but I did­n’t have one at the time), so I ended up post­ing them on­line. The re­sponses I had got­ten from those were in­ter­est­ing. It was HN be­ing HN. Some found it slightly re­lat­able, some nit­picked a sen­tence and let their ver­bose vo­cab­u­lary loose on that sin­gle sen­tence, some just thought I was be­ing whiney. That’s fine, be­cause all of those might be true at the same time.

In the end, I would like to live a sta­ble life, with a sta­ble job. I would like to be proud of the work I do. I would like to be proud of the col­leagues I work with. I do not want to feel like I need to leave the com­pany I work for. I do not want to feel like I need to leave my home.

I want that fight or flight out of my sys­tem.

I was told it’s go­ing to take a year with ther­apy, at least.

When I feel like I’m ready, I will prob­a­bly not be do­ing soft­ware de­vel­op­ment for a while. I do not feel like I will do a good job, and I do not want to be­come a bur­den to a com­pany.

Goals

My goals for the end of 2027 are as fol­lows:

Stop mak­ing stu­pid mis­takes. I want to be able to fin­ish a task fully with­out miss­ing or skip­ping a step. One way to do this is to make a plan for every­thing you do, and only do that thing. Nothing else.

Be proud of what I de­liver. I need to fig­ure out why I keep get­ting de­mo­ti­vated at work. Is it me? Is it the work? What work can I do in this state? What state do I need to do the work I want?

Find sta­bil­ity. I al­ready have friends/​fam­ily go­ing for me, so that is nice. I still need a sta­ble job, and ob­vi­ously I do not want to be­come a bur­den to my sur­round­ings.

I think to gain sta­bil­ity I will need to gain work dis­ci­pline. I am not sure how I will get this. When my mind is a bit more clear, I will do my best to gain this work dis­ci­pline my­self.

There’s a server in my base­ment that has no busi­ness run­ning a mod­ern lan­guage model. It’s a re­pur­posed HP StoreVirtual stor­age box, roughly thir­teen years old, two Ivy Bridge Xeons, no GPU. It was built to hold disks, not do math. As of this week it runs Google’s Gemma 4, a 26-billion-parameter open-weights mix­ture-of-ex­perts model, at about five to­kens per sec­ond. Reading speed.

Anybody can rent a GPU. It’s harder to take a mod­ern MoE model and a dead en­ter­prise box and make them meet in the mid­dle, and that gap is the whole rea­son I’m writ­ing this up. “Good with AI” has qui­etly come to mean “pays for a sub­scrip­tion.” I think the real skill is dif­fer­ent: know­ing a model well enough to point it at a prob­lem no­body pack­aged for you, and telling whether the an­swer it hands back is ac­tu­ally cor­rect. So rather than claim we’re good at this, here’s a worked ex­am­ple, on hard­ware that had no busi­ness co­op­er­at­ing.

The post that started it

A cou­ple of weeks ago a piece called “A 10 year old Xeon is all you need” made the rounds on Hacker News. The au­thor runs Gemma 4 on a sin­gle 2016 Xeon with no GPU and 128 GB of slow DDR3, us­ing ik_l­lama.cpp and about 25 care­fully cho­sen flags. It’s a great read, and it leans on every trick in the mod­ern in­fer­ence play­book: spec­u­la­tive de­cod­ing, CPU-aware mix­ture-of-ex­perts rout­ing, flash at­ten­tion ported to the CPU, run-time weight repack­ing. Real en­gi­neer­ing.

“I have a Xeon too,” I thought. Several, in fact. So I tried it. It did­n’t run.

What an AI agent is ac­tu­ally good for

The build died on startup. I handed the fail­ure to Claude and asked what was wrong. The an­swer came back fast and spe­cific. The au­thor’s 2016 chip is a Broadwell part. Mine are Ivy Bridge, the gen­er­a­tion Intel calls “v2.” The fast ker­nels in that fork as­sume AVX2 and FMA3, in­struc­tion sets that did­n’t ship un­til Haswell, the “v3” gen­er­a­tion, in 2014. My CPUs are older than the in­struc­tions the code was writ­ten against. The op­ti­mized paths weren’t there to ex­e­cute.

So I asked the ob­vi­ous fol­low-up: can we make it run any­way? I’d al­ready taken a first swing with a free model that got close but could­n’t land it. Claude picked up that half-fin­ished ap­proach, agreed it was the right one, and fin­ished it off, re­work­ing the hot paths so they fall back cleanly on a pre-AVX2 chip in­stead of reach­ing for in­struc­tions that aren’t there.

This is the part I care about. This did­n’t come from typ­ing “fix it” once and get­ting a work­ing patch back. Somebody had to read an­other per­son’s per­for­mance-crit­i­cal C++, work out why a ker­nel was­n’t valid on this par­tic­u­lar mi­croar­chi­tec­ture, and route around it with­out throw­ing away the op­ti­miza­tions that made the fork worth us­ing. Claude did that work. My job was nar­rower: run the right ex­per­i­ments and rec­og­nize when the out­put was fi­nally cor­rect. I came away im­pressed.

The re­sult

Gemma 4’s 26B mix­ture-of-ex­perts model now gen­er­ates text at read­ing speed on hard­ware that was re­tired be­fore the mod­el’s ar­chi­tec­ture ex­isted. The orig­i­nal write-up never pub­lished a to­kens-per-sec­ond fig­ure, just “reading speed,” so here’s the con­crete one: about five to­kens a sec­ond on thir­teen-year-old sil­i­con, for bor­der­line free.

Proof it runs: Gemma 4 26B an­swer­ing on the base­ment box, CPU-only.

The patch is up as ikawrakow/​ik_l­lama.cpp#2138 if you want the ex­act diff — still open and await­ing main­tainer re­view as I write this, so run it from the branch for now. The hope is that any­one else sit­ting on an­cient en­ter­prise iron can keep a lo­cal model around: a fall­back for when the paid APIs are down, or a cheap way to grind through slow batch jobs when pay­ing per to­ken does­n’t make sense.

For the peo­ple who want the ac­tual bug

Full dis­clo­sure be­fore I go fur­ther. I’m not a C++ pro­gram­mer. I can read a stack trace and I know my way around a build sys­tem, but I did not hand-write ker­nel fall­backs for a quan­tized mat­mul en­gine, and I won’t pre­tend I did. What I did was drive. I ran the ex­per­i­ments, read the out­put, asked the next ques­tion, and knew what “correct” had to look like. The di­ag­no­sis and the patch came from the Claude in­stance run­ning on the server it­self. I asked it to write up what it fixed, and the rest of this sec­tion is that sum­mary, lightly edited. If you came here from Hacker News for the real tear­down, this part’s for you.

What was ac­tu­ally bro­ken

The en­gine we needed was ik_l­lama.cpp, ikawrakow’s fork of llama.cpp that adds the op­ti­miza­tions Gemma 4’s MoE in­fer­ence de­pends on. It as­sumes AVX2 as its floor. The Xeon E5 – 2690 v2 in this box has AVX1 but not AVX2. Turn GGML_USE_IQK_MULMAT off at build time and most of the code­base re­spects it: the fast paths com­pile out, and the model falls back to plain scalar/​SSE math. That’s fine for a nor­mal Q8_0 mat­mul.

Two graph ops are the ex­cep­tion. The Gemma 4 MoE feed-for­ward net­work emits MOE_FUSED_UP_GATE (a per-ex­pert gate+up mat­mul fused with SwiGLU) and FUSED_UP_GATE (its dense ana­log). Both are #if-gated on GGML_USE_IQK_MULMAT in­side the com­pute dis­patcher, but the graph builder still emits them un­con­di­tion­ally. On this build the dis­patcher’s switch had no case for those op enums, so they fell through to the de­fault, and the des­ti­na­tion ten­sors for every ex­pert FFN silently never got com­puted. Gemma 4 26B has 30 lay­ers by 8 ac­tive ex­perts per to­ken, so every for­ward pass con­sumed roughly 240 ten­sors of what­ever hap­pened to be sit­ting in that mem­ory buffer al­ready.

The symp­tom was flu­ent-look­ing mul­ti­lin­gual gib­ber­ish. Token IDs spread uni­formly across the 262K vo­cab­u­lary, the model equally happy to emit Thai script, Korean, <unused> sen­tinels, or English frag­ments. Deterministic at tem­per­a­ture 0, byte-iden­ti­cal be­tween sin­gle- and multi-threaded runs, no NaNs any­where. Just a hid­den state get­ting shoved by a large con­stant every layer un­til the fi­nal soft­max went flat.

That de­ter­min­ism is what cracked it. Claude in­stru­mented the raw log­its be­fore sam­pling, print­ing the top-5 to­kens plus range, mean, and NaN count. The num­bers gave it away: a mean logit of +16 for the first pre­dicted to­ken when it should sit near zero, and about 80% of the vo­cab­u­lary at pos­i­tive log­its. Random cor­rup­tion does­n’t look like that. A bias that clean only hap­pens when a big chunk of the hid­den state is unini­tial­ized mem­ory that hap­pens to hold small pos­i­tive floats.

The fix

Three com­mits on top of the fork’s main.

Compile fixes. The scalar #else branches for quan­tize_row_q8_0_x4 and quan­tize_row_q8_1_x4_T in iqk_quan­tize.cpp weren’t ac­tu­ally scalar. They still ref­er­enced hsum_i32_8 and other AVX2 helpers. Those got rewrit­ten as portable scalar loops, with #if GGML_USE_IQK_MULMAT guards added around a hand­ful of stray IQK calls leak­ing through ggml.c and ggml-quants.c, plus a miss­ing in­clude so iqk_cpu_ops.cpp com­piles stand­alone. Without these, the fork won’t build at all on non-AVX2 hard­ware.

The run­time bug. Rather than touch the dis­patcher, the fix makes the graph builder emit ops that do have com­pute paths on this build. In ggm­l_­moe_up­_­gate, when GGML_USE_IQK_MULMAT is off: if the weight is the com­bined up­_­gate_­exps ten­sor (shape [n_embd, 2*n_ff, n_­ex­perts], gate in the first half, up in the sec­ond), split it into two ggm­l_view_3d slices, run two sep­a­rate ggm­l_­mul_­mat_id calls, and com­bine them with ggm­l_­fused_­mul_u­nary(gate, up, SILU). If gate and up are al­ready sep­a­rate weights, skip the split and do the same two mul-mat-IDs plus the fused mul-unary. ggm­l_­fused_up­_­gate, the dense ver­sion used in non-MoE lay­ers, gets the same treat­ment. Every op in­volved al­ready has a work­ing non-IQK im­ple­men­ta­tion (mul_mat_id is stock ggml, and fused_­mul_u­nary does the SILU-and-multiply in one pass). The whole change sits be­hind #if !GGML_USE_IQK_MULMAT, so an AVX2 build stays bit-iden­ti­cal to what it was be­fore.

The run­time bug. Rather than touch the dis­patcher, the fix makes the graph builder emit ops that do have com­pute paths on this build. In ggm­l_­moe_up­_­gate, when GGML_USE_IQK_MULMAT is off: if the weight is the com­bined up­_­gate_­exps ten­sor (shape [n_embd, 2*n_ff, n_­ex­perts], gate in the first half, up in the sec­ond), split it into two ggm­l_view_3d slices, run two sep­a­rate ggm­l_­mul_­mat_id calls, and com­bine them with ggm­l_­fused_­mul_u­nary(gate, up, SILU). If gate and up are al­ready sep­a­rate weights, skip the split and do the same two mul-mat-IDs plus the fused mul-unary. ggm­l_­fused_up­_­gate, the dense ver­sion used in non-MoE lay­ers, gets the same treat­ment. Every op in­volved al­ready has a work­ing non-IQK im­ple­men­ta­tion (mul_mat_id is stock ggml, and fused_­mul_u­nary does the SILU-and-multiply in one pass). The whole change sits be­hind #if !GGML_USE_IQK_MULMAT, so an AVX2 build stays bit-iden­ti­cal to what it was be­fore.

CI stubs. The #else stub sec­tions of the iqk sources had drifted out of sync with iqk_­mul_­mat.h, so ci/​run.sh could­n’t even build on non-AVX2 hard­ware: a miss­ing <cstdint>, stubs with the wrong sig­na­tures (an ex­tra lead­ing pa­ra­me­ter here, a miss­ing sinks there), and no stubs at all for a cou­ple of func­tions, which meant un­de­fined ref­er­ences at link time. Boring work, but with­out it no­body on this hard­ware can run the test suite.

The fall­back costs some­thing, two sep­a­rate mat­mul-IDs in­stead of one fused ker­nel, but this CPU is mem­ory-band­width-bound any­way, and the fused ker­nel was AVX2-only, so we weren’t giv­ing any­thing up. End to end we get about 5.2 tok/​s de­code and ~16 tok/​s prompt-eval on a 26B-A4B MoE.

One more gotcha. –run-time-repack re­orders quan­tized weights into an AVX2-only in­ter­leaved lay­out (Q8_0_R8) at startup, which gar­bles out­put on AVX1 the same way. That’s a sep­a­rate bug, and the patch does­n’t try to fix it. The run script just drops the flag.

The in­struc­tion-set mis­match was easy to spot. The silent fall-through was not. Reading the code kept clear­ing the ob­vi­ous sus­pects: the RMSNorm helpers looked cor­rect, the AVX1 fall­back in ggm­l_vec_­dot_q8_0_q8_0 looked cor­rect, and a bit-iden­ti­cal sin­gle-thread run ruled out thread­ing. Only af­ter in­stru­ment­ing the log­its, and see­ing the mean pinned at +16 with every long-tail to­ken roughly tied, did the search nar­row to “a big chunk of the resid­ual stream is unini­tial­ized.” Grepping for #if GGML_USE_IQK_MULMAT in the dis­patcher turned up the two miss­ing cases about a minute later.

Reproduce it

If you have a pre-AVX2 box and want to try this:

Hardware: dual Xeon E5 – 2690 v2 (Ivy Bridge, AVX1, no AVX2), DDR3, no GPU.

Build: com­pile ik_l­lama.cpp from the branch above with GGML_USE_IQK_MULMAT off. The com­pile fixes are what let it build at all on non-AVX2 hard­ware.

Model: Gemma 4 26B-A4B, Q8_0.

Run: use the usual ik_l­lama.cpp CPU flags, but drop –run-time-repack (it re­orders weights into an AVX2-only lay­out and re-gar­bles the out­put on AVX1).

That is the whole recipe: about 5 tok/​s de­code, CPU-only, no GPU any­where in the box.

Why this is on the com­pany blog

The sub­scrip­tion is the easy part. The rest is the will­ing­ness to open the hood, read a stranger’s code, and keep ask­ing un­til a thir­teen-year-old CPU does some­thing it was never meant to. That’s the same work a fif­teen-year-old Rails app needs, or a data­base no­body left on the team still un­der­stands: some­one who’ll dig un­til they find where the lever­age is, and what the tool won’t tell you on its own.

If you have pre-AVX2 iron gath­er­ing dust and try the branch, I’d love to hear what it does on your sil­i­con — how far down the CPU gen­er­a­tions does this go? The PR thread is the right place for bug re­ports. And if the thing you’re nurs­ing along is a fif­teen-year-old Rails app rather than a thir­teen-year-old Xeon, that’s what we do for a liv­ing.

A cou­ple of point­ers for the cu­ri­ous. The server it­self cost un­der $300; here’s the math on why a base­ment box beats $1,500 a month of cloud. And get­ting a scream­ing en­ter­prise ap­pli­ance quiet and bootable in the first place was its own pro­ject, writ­ten up here for peo­ple who en­joy that sort of thing.

Telegram claims to have 5 data cen­ters (DCs), re­ferred to as DC1~5 in Telegram’s code and doc­u­men­ta­tion. Among them, DC1 and DC3 are lo­cated in Miami, USA; DC2 and DC4 are in Amsterdam, Netherlands; and DC5 is in Singapore.

Each ac­count is as­so­ci­ated with a DC upon reg­is­tra­tion and does not change with the user’s phone num­ber or ge­o­graphic lo­ca­tion. Users can­not freely choose a DC—if con­nected to the wrong DC, the server re­turns an er­ror mes­sage, re­quir­ing the client to con­nect to the cor­rect DC as­so­ci­ated with the ac­count.

Starting with the Frequently Down DC5

Among the 5 DCs, DC5 is par­tic­u­larly well-known in the Chinese Telegram com­mu­nity—not be­cause it qui­etly serves a large num­ber of Chinese users, but be­cause it fre­quently goes down.

When DC5 is down, users on DC5 can­not use Telegram, and the topic in Telegram cir­cles of­ten be­comes, “Why is DC5 down again?” DC5 users can only wait for their con­stantly “reconnecting” clients to re­cover, then join group chats with users from other DCs to crit­i­cize DC5.

The Mystery of DC2 and DC3

To sat­isfy the cu­rios­ity of group mem­bers, some­one cre­ated a bot to query the DC a user is on. So, group mem­bers started check­ing their DCs:

Users reg­is­tered with a +86 phone num­ber found them­selves on DC5 (Singapore), about to ex­pe­ri­ence the next down­time.

Users reg­is­tered with a +1 phone num­ber were on DC1 (Miami, USA), en­joy­ing both con­nec­tion speed and sta­bil­ity.

Users reg­is­tered with European phone num­bers found them­selves on DC4 (Amsterdam, Netherlands), be­com­ing the rarest in the Chinese Telegram com­mu­nity.

Huh? Wait… Doesn’t Telegram claim to have 5 DCs? What about DC2 and DC3?

Through “big data queries” of the bot’s mes­sages, it seems there are in­deed no users from DC2 or DC3.

Thus, some spec­u­lated that DC2 and DC3 have no users, while oth­ers an­a­lyzed and spec­u­lated that DC2 and DC3 are sub­or­di­nate DCs to DC1 and DC4, re­spec­tively, ac­cept­ing user reg­is­tra­tions when their par­ent DCs are busy.

Is that re­ally the case?

Telegram DC Allocation Rules (2022 – 05)

TL;DR:

DC1, DC2, DC4, and DC5 are al­lo­cated based on the coun­try code of the phone num­ber pro­vided at reg­is­tra­tion. These 4 DCs ac­cept new user reg­is­tra­tions at any time and have a large num­ber of ex­ist­ing users.

DC3 once had users, but around 2020, DC3’s ex­ist­ing users may have been trans­ferred to DC1. DC3 cur­rently likely has no users and no longer ac­cepts new reg­is­tra­tions.

In other words, DC2 ac­tu­ally has a large num­ber of users, and like other DCs, as long as the coun­try code of the phone num­ber at reg­is­tra­tion falls within DC2’s range (e.g., +49 Germany), the user will def­i­nitely be as­signed to DC2. Meanwhile, DC3, though still op­er­a­tional, likely has no as­so­ci­ated users.

Since there are many users on DC2, why did the bot above never de­tect any DC2 users?

This is be­cause the bot’s method of re­triev­ing DCs is flawed.

Which DC Am I Actually On?

There are cur­rently 3 com­mon meth­ods to de­ter­mine a DC. Below, we will reg­is­ter a new DC2 ac­count and try these 3 meth­ods.

Method 1 (Login Method)

Using a phone num­ber that would be as­signed to DC2, we con­nect to DC1 via Telegram’s MTProto pro­to­col (the same pro­to­col used by the of­fi­cial client), call the auth.send­Code in­ter­face, and at­tempt to send a ver­i­fi­ca­tion code to reg­is­ter an ac­count.

At this point, the server re­turns a PHONE_MIGRATE_2 er­ror, in­di­cat­ing that the client should con­nect to DC2 (if the same op­er­a­tion is per­formed af­ter con­nect­ing to DC2, the ver­i­fi­ca­tion code is sent di­rectly).

This way, we know this ac­count is a DC2 ac­count. This method also works for al­ready reg­is­tered ac­counts, but it re­quires know­ing the user’s phone num­ber, mak­ing it dif­fi­cult to query group mem­bers’ DCs.

Method 2 (Profile Picture/File Method)

After reg­is­ter­ing the DC2 ac­count (referred to as the new ac­count be­low), we use a third-party client (Plus Messenger) that dis­plays a user’s DC, log in with an­other ac­count, and check the new ac­coun­t’s DC. However, the client can­not dis­play the DC un­til we up­load a pro­file pic­ture for the new ac­count, af­ter which it shows the new ac­count is on DC2.

This is be­cause the third-party client re­trieves the user’s DC from the dc_id field in the user­Pro­file­Photo struc­ture of the MTProto pro­to­col when down­load­ing the user’s pro­file pic­ture.

This method de­ter­mines the user’s DC based on the DC where the user’s up­loaded files are stored.

If you log into a third-party client with the new ac­count to check your own DC, it may use Method 1 to de­ter­mine the DC, as the client knows which server it is con­nected to.

If you log into a third-party client with the new ac­count to check your own DC, it may use Method 1 to de­ter­mine the DC, as the client knows which server it is con­nected to.

Method 3 (Web CDN Method)

Finally, we use the bot men­tioned ear­lier to query the new ac­coun­t’s DC.

Huh? Isn’t this new ac­count on DC2? Why does the bot say it’s DC4?

It turns out the bot de­ter­mines the user’s DC via Telegram’s Web CDN. If we open https://​t.me/​dctest** and check the source code, we find that the new ac­coun­t’s pro­file pic­ture do­main starts with cdn4, caus­ing the bot to iden­tify it as DC4.

Since DC2 and DC3 “borrow” the do­mains of DC4 and DC1 in the same lo­ca­tion to pro­vide Web CDN ser­vices, the bot can­not iden­tify any DC2 users—they are all mis­taken for DC4 users.

Another type of bot re­quires users to send an im­age/​file to de­ter­mine their DC. This is sim­i­lar to Method 2 and can ac­cu­rately iden­tify a user’s DC.

Another type of bot re­quires users to send an im­age/​file to de­ter­mine their DC. This is sim­i­lar to Method 2 and can ac­cu­rately iden­tify a user’s DC.

The Disappearance of DC3

After cor­rectly de­ter­min­ing users’ DCs us­ing Method 2, we found that glob­ally (especially in over­seas groups), DC2 users are not un­com­mon, but DC3 users are ex­tremely rare. After ex­ten­sive search­ing, we found two users on DC3: @urie** and @flowinglig**.

However, fur­ther analy­sis re­veals they may not ac­tu­ally be on DC3. For ex­am­ple, call­ing pho­tos.Ge­tUser­Pho­tos to view the pro­file pic­ture list shows that @urie** up­loaded 7 pro­file pic­tures, only two of which are on DC3, with newly up­loaded pic­tures on DC1.

Similarly, check­ing the im­ages in the mes­sage his­tory of both users shows only a few old im­ages stored on DC3, with new im­ages stored on DC1. In con­trast, users on other DCs have their im­ages stored on their re­spec­tive DCs.

Additionally, @urie** sent a file to a DC iden­ti­fi­ca­tion bot (File Method) in 2021 for test­ing, and the re­sult was also DC1.

Unfortunately, since we can­not ac­cess their phone num­bers, we can­not ac­cu­rately test us­ing Method 1 (Login Method). We can only in­fer through Method 2 (Profile Picture/File Method) that they were trans­ferred from DC3 to DC1.

To fur­ther con­firm that DC3 no longer ac­cepts new users, we gen­er­ated over 10,000 phone num­bers from var­i­ous global re­gions and tested Telegram’s DC al­lo­ca­tion rules us­ing Method 1 (Login Method). The re­sults are as fol­lows:

During test­ing, we en­sured each num­ber con­nected to the wrong DC when­ever pos­si­ble—to un­der­stand the ac­tual DC cor­re­spond­ing to the num­ber via the re­turned PHONE_MIGRATE_X er­ror and to avoid gen­er­at­ing ex­ces­sive spam SMS, caus­ing ha­rass­ment or bank­rupt­ing Pavel Durov with SMS fees.

After test­ing, we fil­tered out num­bers con­firmed to be on DC4, then re­con­nected the re­main­ing num­bers to DC4 to en­sure no num­bers as­signed to DC3 were missed.

During test­ing, we en­sured each num­ber con­nected to the wrong DC when­ever pos­si­ble—to un­der­stand the ac­tual DC cor­re­spond­ing to the num­ber via the re­turned PHONE_MIGRATE_X er­ror and to avoid gen­er­at­ing ex­ces­sive spam SMS, caus­ing ha­rass­ment or bank­rupt­ing Pavel Durov with SMS fees.

After test­ing, we fil­tered out num­bers con­firmed to be on DC4, then re­con­nected the re­main­ing num­bers to DC4 to en­sure no num­bers as­signed to DC3 were missed.

Based on the above analy­sis, we can con­clude that DC3 in­deed no longer ac­cepts new users, and ex­ist­ing users have likely all been trans­ferred to DC1.

Although we could­n’t find the miss­ing DC3, if you want to be­come a user on a spe­cific DC, avoid DC5’s down­time risks, or test whether a bot is re­li­able, you can now re­fer to the pre­vi­ous im­age and reg­is­ter with a phone num­ber from a spe­cific coun­try code.

Since Telegram’s server and some op­er­a­tional mech­a­nisms are not open-source, many con­clu­sions in this ar­ti­cle are based on spec­u­la­tion. If you find er­rors in the ar­ti­cle or have ad­di­tional clues, please feel free to com­ment.

The fol­low­ing pro­jects and con­tent were ref­er­enced dur­ing the writ­ing of this ar­ti­cle, and I would like to ex­press my grat­i­tude:

Telegram DC Analysis Report by Hertz

Which DC is My Telegram Account On? (Web CDN Method) by oot­t123

GramJS

Plus Messenger (Profile Picture Method)

@WooMaiBot (Web CDN Method)

@where_is_my_dc_bot (File Method)

Coxxs

This ar­ti­cle (https://​dev.moe/​en/​3025) is an orig­i­nal work by Coxxs. Please credit the orig­i­nal link when re­post­ing.

This ar­ti­cle (https://​dev.moe/​en/​3025) is an orig­i­nal work by Coxxs. Please credit the orig­i­nal link when re­post­ing.

Date: 2026 – 07-15 Git: https://​git­lab.com/​mort96/​blog/​blob/​pub­lished/​con­tent/​00000-home/​00017-sqlite-edi­tions.md

SQLite is an amaz­ing data­base en­gine. I use it as a data­base for plenty of em­bed­ded pro­jects, and I don’t think it’s an ex­ag­ger­a­tion to call it the in­dus­try stan­dard for lo­cal data stor­age. Some server soft­ware even uses it; for ex­am­ple, lob­ste.rs is now run­ning on SQLite.

Unlike tra­di­tional RDBMSes (Relational DataBase Management Systems), SQLite is not a sep­a­rate process; it’s an RDBMS as a li­brary, mean­ing your soft­ware re­mains self con­tained. Unlike tra­di­tional file for­mats, you don’t need to write cus­tom se­ri­al­iz­ers and parsers. In some ways, it’s the best of both worlds.

There’s just one huge prob­lem though. Its de­faults are all wrong.

Bad de­fault #1: Foreign key con­straints are ig­nored by de­fault

You read that right. Foreign key con­straints are ar­guably the pri­mary tool we have to en­sure that a data­base re­mains con­sis­tent and don’t have dan­gling ref­er­ences.

As a quick primer, this is how an SQL for­eign key con­straint looks:

CREATE TABLE users ( id INTEGER PRIMARY KEY, dis­play_­name TEXT );

CREATE TABLE posts ( id INTEGER PRIMARY KEY, user_id INTEGER NOT NULL, con­tent TEXT NOT NULL, FOREIGN KEY(user_id) REFERENCES users(id) );

The typ­i­cal be­hav­ior for all other RDBMSes would be that the user_id col­umn of a post must al­ways ref­er­ence the ID of a valid user. You can’t cre­ate a new post with­out pro­vid­ing a valid user ID, you can’t delete a user with­out also delet­ing its posts, lest you get a for­eign key con­straint vi­o­la­tion er­ror.

The only RDBMS I’m aware of which does­n’t en­force this by de­fault is SQLite.

This is made even worse by SQLite’s ten­dency to re-use ROWID. You see, in this ex­am­ple, those INTEGER PRIMARY KEY rows be­come aliases for the table’s ROWID, which is a unique in­te­ger ID as­signed to every row of a table in SQLite. The al­go­rithm for as­sign­ing ROWID is a bit com­pli­cated (more de­tails in the SQLite doc­u­men­ta­tion), but it re­sults in ID re-use in some cases. This means that a dan­gling ref­er­ence eas­ily re­sults in a ref­er­ence to the wrong row, which is even worse than a dan­gling ref­er­ence be­cause every­thing will seem fine. You don’t even get an er­ror dur­ing lookup.

Just look at this hy­po­thet­i­cal se­quence of op­er­a­tions in our toy data­base schema:

– Bob cre­ates a user ac­count INSERT INTO users (display_name) VALUES (‘Bob’); SELECT * FROM users; — id | dis­play_­name — 1 | Bob

– Bob posts an in­tro­duc­tion post INSERT INTO posts (user_id, con­tent) VALUES (1, ‘Hello, I am Bob’); SELECT u.dis­play_­name, p.con­tent FROM users as u, posts as p WHERE u.id = p.user_id; — dis­play_­name | con­tent — Bob | Hello, I am Bob

– Bob deletes his ac­count, — but we for­got to delete the posts. — SQLite does­n’t pro­duce an er­ror be­cause it ig­nores our for­eign key. DELETE FROM users WHERE id = 1;

– Alice cre­ates an ac­count. — Alice gets the same ID that Bob had due to the ROWID al­go­rithm. INSERT INTO users (display_name) VALUES (‘Alice’); SELECT * FROM users; — id | dis­play_­name — 1 | Alice

– Alice has now in­her­ited Bob’s old post! SELECT u.dis­play_­name, p.con­tent FROM users as u, posts as p WHERE u.id = p.user_id; — dis­play_­name | con­tent — Alice | Hello, I am Bob

The fix is to en­able for­eign_keys with a pragma:

PRAGMA for­eign_keys = ON;

If we had done this in the be­gin­ning, the buggy DELETE would have pro­duced an er­ror:

DELETE FROM users WHERE id = 1; — Runtime er­ror: FOREIGN KEY con­straint failed (19)

Bad de­fault #2: Columns can store the wrong data type

SQLite has a sim­ple type sys­tem: a value can be NULL, an INTEGER, a REAL (aka a dou­ble pre­ci­sion float), TEXT, or a BLOB (aka bi­nary data). Consequently, a col­umn can be de­fined to hold val­ues of any of those types.

However, a col­umn de­fined as an INTEGER col­umn is­n’t re­stricted to only in­te­gers; in­stead, SQLite con­sid­ers it to “use INTEGER affin­ity”. What this means is es­sen­tially:

If you try to in­sert a TEXT value, and it is a valid string rep­re­sen­ta­tion of an in­te­ger, it is con­verted to an in­te­ger and stored as such.

If you try to in­sert a TEXT value, and it is a valid string rep­re­sen­ta­tion of a real num­ber, it is con­verted to a real (aka dou­ble pre­ci­sion float) and stored as such.

Otherwise, the value is stored as-is.

Other affini­ties have dif­fer­ent but sim­pler rules:

Columns with BLOB affin­ity store val­ues as-is.

Columns with TEXT affin­ity store BLOB, TEXT and NULL val­ues as-is, but con­vert nu­meric val­ues to TEXT.

Columns with REAL affin­ity work like columns with INTEGER affin­ity ex­cept that in­te­ger val­ues are con­verted to REAL.

Here’s how this looks in prac­tice:

CREATE TABLE mu­sic ( id INTEGER PRIMARY KEY, name TEXT, du­ra­tion_sec INTEGER );

INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘Lost In Hollywood’, 321); INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘Comfortably Numb’, 382); INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘The Way of All Flesh’, ‘Way too long, I mean come on’); SELECT * FROM mu­sic; — id | name | du­ra­tion_sec — 1 | Lost In Hollywood | 321 — 2 | Comfortably Numb | 382 — 3 | The Way of All Flesh | Way too long, I mean come on

I don’t think I need to ex­plain why it’s a bad idea for a data­base to be so care­less about data val­i­da­tion. It would be one thing if SQLite was an ex­plic­itly dy­nam­i­cally typed doc­u­ment data­base, but it’s not. SQLite asks me through its syn­tax rules, “What type do you want to go into this col­umn”.

I once had to clean up a pro­ject where some code had ac­ci­den­tally been writ­ing the strings ‘1’ and ‘0’ to a col­umn which was in­tended to store booleans (1 and 0). That was not a fun de­bug­ging story.

Luckily, SQLite has the con­cept of strict ta­bles, which makes SQLite pro­duce a type er­ror when the wrong type is in­serted into a col­umn:

CREATE TABLE mu­sic ( id INTEGER PRIMARY KEY, name TEXT, du­ra­tion_sec INTEGER ) strict;

INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘The Way of All Flesh’, ‘Way too long, I mean come on’); — Runtime er­ror: can­not store TEXT value in INTEGER col­umn mu­sic.du­ra­tion_sec (19)

Unfortunately, there is no pragma to glob­ally make all ta­bles strict. So you have to re­mem­ber to add the strict tag to every table man­u­ally.

There’s a cou­ple of ar­gu­ments against strict ta­bles which I want to cover here.

The au­thors of SQLite have writ­ten about their pref­er­ence for “flexible typ­ing”. Personally, I find this a re­ally strange piece of writ­ing. It does­n’t pro­vide any ex­am­ples for why it could ever be use­ful to in­sert a BLOB into an INTEGER col­umn. All it does is il­lus­trate why it’s some­times use­ful to have a col­umn which can store val­ues of any type. Strict ta­bles have a so­lu­tion for that; it’s called the ANY data type. You can still cre­ate columns which ac­cept any value, you just have to be ex­plicit about it.

A much bet­ter ar­gu­ment is pro­vided by user ‘zie’ on lob­ste.rs. You see, strict ta­bles in SQLite don’t just en­force types. They also change the rules for how type spec­i­fiers are parsed.

Non-strict SQLite ta­bles use the fol­low­ing rules to de­ter­mine the type of a col­umn (from SQLite’s doc­u­men­ta­tion):

If the de­clared type con­tains the string “INT” then it is as­signed INTEGER affin­ity. If the de­clared type of the col­umn con­tains any of the strings “CHAR”, “CLOB”, or “TEXT” then that col­umn has TEXT affin­ity. Notice that the type VARCHAR con­tains the string “CHAR” and is thus as­signed TEXT affin­ity. If the de­clared type for a col­umn con­tains the string “BLOB” or if no type is spec­i­fied then the col­umn has affin­ity BLOB. If the de­clared type for a col­umn con­tains any of the strings “REAL”, “FLOA”, or “DOUB” then the col­umn has REAL affin­ity. Otherwise, the affin­ity is NUMERIC.

If the de­clared type con­tains the string “INT” then it is as­signed INTEGER affin­ity.

If the de­clared type of the col­umn con­tains any of the strings “CHAR”, “CLOB”, or “TEXT” then that col­umn has TEXT affin­ity. Notice that the type VARCHAR con­tains the string “CHAR” and is thus as­signed TEXT affin­ity.

If the de­clared type for a col­umn con­tains the string “BLOB” or if no type is spec­i­fied then the col­umn has affin­ity BLOB.

If the de­clared type for a col­umn con­tains any of the strings “REAL”, “FLOA”, or “DOUB” then the col­umn has REAL affin­ity.

Otherwise, the affin­ity is NUMERIC.

A con­se­quence of this rule, com­bined with SQLite’s loose typ­ing, is that you can give your columns type names such as DATETIME or KEY_VALUE_SET or COLOR, and have a data­base con­nec­tor/​wrap­per which au­to­mat­i­cally knows to se­ri­al­ize and de­se­ri­al­ize columns with cus­tom types. And if noth­ing else, those cus­tom type names serve as use­ful doc­u­men­ta­tion.

I have to ac­knowl­edge that just chang­ing the de­fault from non-strict ta­bles to strict ta­bles, with no fur­ther changes, would give up on this some­what nifty quirk. However, I think we would be much bet­ter served by cus­tom type aliases.

If we could write some­thing like:

CREATE TYPE KEY_VALUE_SET = TEXT;

and then use KEY_VALUE_SET as a type name in a strict table, I think every­one would be happy. I would prob­a­bly start us­ing such a fea­ture lib­er­ally to doc­u­ment the ex­pected pat­tern of data in my columns. In a real world schema, you in­evitably end up with TEXT columns which have to be parsed by ap­pli­ca­tion code.

As an aside to this aside, it would be neat if we could as­so­ci­ate CHECK con­straints with a cus­tom type.

Update: ‘masklinn’ on lob­ste.rs points out that the SQL 99 stan­dard al­ready spec­i­fies type aliases, called CREATE DOMAIN. This al­ready sup­ports con­straints as well. So re­ally, SQLite just needs to add sup­port for the stan­dard CREATE DOMAIN state­ment.

Update: ‘masklinn’ on lob­ste.rs points out that the SQL 99 stan­dard al­ready spec­i­fies type aliases, called CREATE DOMAIN. This al­ready sup­ports con­straints as well. So re­ally, SQLite just needs to add sup­port for the stan­dard CREATE DOMAIN state­ment.

Bad de­fault #3: SQLITE_BUSY er­rors with con­cur­rent writ­ers

SQLite al­lows mul­ti­ple con­cur­rent read­ers, but only one writer at a time. By de­fault, if you have two processes try­ing to ac­quire a write lock at the same time, one of them will im­me­di­ately re­ceive an SQLITE_BUSY er­ror.

This is not the be­hav­ior I ex­pect. I ex­pect SQLite to wait for the lock to get un­locked, up to some time­out. It’s do­ing disk IO af­ter all, so I al­ready struc­ture my code with the as­sump­tion that a write could po­ten­tially be slow.

The de­fault be­hav­ior has lead me to writ­ing real-world bugs, where sys­tems would some­times just crash. I’ve man­u­ally writ­ten retry loops to fix it.

The fix is to set busy_­time­out with a pragma:

PRAGMA busy_­time­out = 5000;

This makes SQLite try to ac­quire the lock for up to 5 sec­onds be­fore er­ror­ing with a SQLITE_BUSY er­ror.

I did­n’t learn about this set­ting un­til re­cently. It seems like such an ob­vi­ous de­fault that I’m as­ton­ished that it’s not.

Update: I should add a note here about why sup­port for con­cur­rent writ­ers is de­sir­able. During nor­mal op­er­a­tion, you’re usu­ally best served by struc­tur­ing your soft­ware such that all writes are done by a sin­gle process, ide­ally a sin­gle thread. Concurrent writ­ers will never be fast. But there are non-typ­i­cal sit­u­a­tions. Maybe you need to man­u­ally clean up a data­base in­ter­ac­tively us­ing the sqlite3 tool in­ter­ac­tively on the com­mand line. Maybe you have scripts for un­com­mon ad­min­is­tra­tive tasks which you haven’t had the need to write a front-end for. These are per­fectly le­git­i­mate and, I be­lieve, fairly com­mon use cases. I think it’s bad that with SQLite’s de­faults, this kind use has a chance to just crash the soft­ware by mak­ing it throw an un­ex­pected SQLITE_BUSY er­ror.

Update: I should add a note here about why sup­port for con­cur­rent writ­ers is de­sir­able.

During nor­mal op­er­a­tion, you’re usu­ally best served by struc­tur­ing your soft­ware such that all writes are done by a sin­gle process, ide­ally a sin­gle thread. Concurrent writ­ers will never be fast. But there are non-typ­i­cal sit­u­a­tions. Maybe you need to man­u­ally clean up a data­base in­ter­ac­tively us­ing the sqlite3 tool in­ter­ac­tively on the com­mand line. Maybe you have scripts for un­com­mon ad­min­is­tra­tive tasks which you haven’t had the need to write a front-end for. These are per­fectly le­git­i­mate and, I be­lieve, fairly com­mon use cases. I think it’s bad that with SQLite’s de­faults, this kind use has a chance to just crash the soft­ware by mak­ing it throw an un­ex­pected SQLITE_BUSY er­ror.

Bad de­fault 4: Performance

There’s a lot to say about per­for­mance tun­ing in SQLite. When cor­rectly con­fig­ured, it can be a truly fast RDBMS, with the abil­ity to fill roles we typ­i­cally re­serve for the big servers like PostgreSQL or MySQL.

But by de­fault, its per­for­mance is­n’t great. Smarter peo­ple than me have writ­ten much more on this, and I rec­om­mend Sylvain Kerkour’s Optimizing SQLite for servers if you’re in­ter­ested in this topic.

But the most sig­nif­i­cant bad de­fault is that SQLite’s Write-Ahead Log (WAL) is dis­abled by de­fault. It can be en­abled with:

PRAGMA jour­nal_­mode = WAL;

The WAL pro­vides a dra­matic write speed-up in most cir­cum­stances. Additionally, it lets us dras­ti­cally re­duce the amount of disk syncs with­out risk­ing data cor­rup­tion by chang­ing an­other set­ting:

PRAGMA syn­chro­nous = NORMAL;

See the SQLite doc­u­men­ta­tion on what ex­actly syn­chro­nous does.

The so­lu­tion: edi­tions?

The oft-cited rea­son for why these de­faults re­main, well, de­fault, is of course back­wards com­pat­i­bil­ity. Changing de­faults now would likely break lots of old soft­ware and make peo­ple afraid to up­grade SQLite in the fu­ture in case it breaks every­thing again, just like how I’m afraid to up­grade Python be­cause every “upgrade” breaks a bunch of soft­ware I use. It’s a laud­bile and rare goal to try to not break your de­pen­dents.

However, I think the so­lu­tion is sim­ple: add one “super pragma” which changes all the bad de­faults. I pro­pose that the fol­low­ing:

PRAGMA edi­tion = 2026;

should be an alias for at least the fol­low­ing set of prag­mas:

PRAGMA for­eign_keys = ON; PRAGMA busy_­time­out = 5000; PRAGMA jour­nal_­mode = WAL; PRAGMA syn­chro­nous = NORMAL;

And also make strict mode the de­fault for ta­bles.

This should be a nice mid­dle ground which avoids break­ing back­wards com­pat­i­bil­ity, but lets the data­base en­gine move for­wards and not be bogged down by its own his­tory.

The edi­tion idea is stolen straight from Rust edi­tions. The ad­van­tage of a year-based edi­tion rather than some­thing like JavaScript’s “use strict”; is that as the years go by, the sen­si­ble de­faults may change. Maybe some­thing like Hctree’s WAL2 makes its way into the main branch, say, in the year 2034, so maybe PRAGMA edi­tion = 2034 will some day set PRAGMA jour­nal_­mode = WAL2.

Anyway, that’s all. I think SQLite should have an edi­tion sys­tem with up­dated sets of de­faults. Are there any things I’ve missed which makes this a bad idea? Or more prag­mas which should be added to my imag­i­nary “2026 edi­tion”?