Ask any­thing, or task an agent…

Explore in­spi­ra­tion

Today, we are in­tro­duc­ing Kimi K3 — our most ca­pa­ble model. Kimi K3 is a 2.8T-parameter model built on our Kimi Delta Attention and Attention Residuals, with na­tive vi­sion ca­pa­bil­i­ties and a 1-million-token con­text win­dow. It is the world’s first open 3T-class model, de­signed for fron­tier in­tel­li­gence across long-hori­zon cod­ing, knowl­edge work, and rea­son­ing.

While its over­all per­for­mance still trails the most pow­er­ful pro­pri­etary mod­els, Claude Fable 5 and GPT 5.6 Sol, Kimi K3 demon­strated fron­tier-level per­for­mance across our eval­u­a­tion suite, con­sis­tently out­per­form­ing other tested mod­els.

Kimi K3 is avail­able to­day on Kimi.com, Kimi Work, Kimi Code, and the Kimi API. At launch, Kimi K3 will use max think­ing ef­fort by de­fault, with low- and high-ef­fort modes to be in­tro­duced in sub­se­quent up­dates. We are cur­rently work­ing closely with in­fer­ence part­ners and open-source main­tain­ers to align tech­ni­cal de­tails and en­sure a re­li­able roll­out across the ecosys­tem. The full model weights will be re­leased by July 27, 2026. Further de­tails on the ar­chi­tec­ture, train­ing, and eval­u­a­tions will be re­leased along­side the Kimi K3 tech­ni­cal re­port.

An Open 3T-Class Model

Kimi K3 is the first open model to reach 2.8 tril­lion pa­ra­me­ters. It marks the lat­est step in Kimi’s sus­tained push at the scal­ing fron­tier: for nine of the past twelve months, Kimi mod­els have set the up­per bound of open-model sizes.

Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), two ar­chi­tec­tural up­dates de­signed to im­prove how in­for­ma­tion flows across se­quence length and model depth. We have also scaled up Mixture of Experts (MoE) spar­sity, ef­fec­tively ac­ti­vat­ing 16 out of 896 ex­perts when paired with a Stable LatentMoE frame­work. Together with re­fined train­ing and data recipes, these struc­tural changes yield an ap­prox­i­mate 2.5× im­prove­ment in over­all scal­ing ef­fi­ciency com­pared to Kimi K2, al­low­ing the model to con­vert com­pute into in­tel­li­gence more ef­fec­tively.

Coding

Kimi K3 has strong long-hori­zon cod­ing per­for­mance. Operating with min­i­mal hu­man over­sight, it can sus­tain long en­gi­neer­ing ses­sions, nav­i­gate mas­sive repos­i­to­ries, and or­ches­trate ter­mi­nal tools.

Kimi K3 also ex­cels in tasks blend­ing soft­ware en­gi­neer­ing with vi­sual rea­son­ing — it lever­ages screen­shots and vi­su­als to op­ti­mize game dev, fron­tend, and CAD.

The case stud­ies be­low show how Kimi K3′s cod­ing ca­pa­bil­ity trans­lates into open-ended soft­ware cre­ation and sci­en­tific re­search.

Kernel Optimization

We tested the mod­els’ ca­pa­bil­ity to op­ti­mize GPU ker­nels. Each model works in­de­pen­dently in an iden­ti­cal sand­box, with up to 24 hours to pro­file, rewrite, and bench­mark four tasks span­ning AttnRes, KDA, and a 512-head-dimension MLA ker­nel across NVIDIA H200 and GPGPU from an al­ter­na­tive ven­dor. Kimi K3 per­formed com­pet­i­tively with Fable 5 (with fall­back) and sub­stan­tially out­per­formed Opus 4.8, GPT 5.6 Sol, and GPT 5.5.

Claude Fable 5 was eval­u­ated by a third party, and its re­sults may in­clude fall­back be­hav­ior. Across most mod­els, some tra­jec­to­ries in­clude small, ac­cept­able pre­ci­sion short­cuts that re­main within our nu­mer­i­cal tol­er­ance. GPGPU de­notes gen­eral-pur­pose GPUs used for com­pu­ta­tion be­yond graph­ics ren­der­ing.

In the late stages of Kimi K3 de­vel­op­ment, an early ver­sion of Kimi K3 han­dled the ma­jor­ity of the team’s ker­nel op­ti­miza­tion works.

GPU Compiler Development

We fur­ther tested whether Kimi K3 could build a GPU pro­gram­ming sys­tem from scratch. Kimi K3 de­vel­oped MiniTriton, a com­pact Triton-like com­piler with its own tile-level IR layer over MLIR, op­ti­miza­tion passes, and a PTX code-gen­er­a­tion pipeline. Across sup­ported roofline bench­marks, MiniTriton de­liv­ers per­for­mance on par with or bet­ter than Triton and torch.com­pile — beat­ing Triton on cer­tain work­loads. Beyond mi­crobench­marks, MiniTriton sus­tains end-to-end nanoGPT train­ing with sta­ble con­ver­gence, the loss curve closely track­ing the ref­er­ence with only mi­nor di­ver­gence — val­i­dat­ing the full pipeline on a re­al­is­tic work­load. These re­sults demon­strate that Kimi K3 can build a co­her­ent end-to-end com­piler — from DSL fron­tend and IR passes to PTX code­gen and run­time — rather than iso­lated ker­nels; its from-scratch Tensor Core path al­ready ri­vals Triton’s ex­ten­sively op­ti­mized stack.

Game Dev and Digital Creation

Kimi K3 com­bines strong 3D rea­son­ing, cod­ing, and vi­sion ca­pa­bil­i­ties to turn con­cepts, im­ages, and videos into fully playable in­ter­ac­tive ex­pe­ri­ences. Kimi K3 achieves true “vision in the loop” by seam­lessly it­er­at­ing be­tween code and live screen­shots—in­stantly see­ing and re­fin­ing out­puts.

Case 1: 3D Open World

Kimi K3 built a fully pro­ce­dural browser-based 3D ex­plo­ration game us­ing Three.js WebGPU and GPU com­pute. It pro­ce­du­rally gen­er­ated the en­vi­ron­ment, while us­ing a 3D as­set gen­er­a­tion tool to cre­ate the rider and horse mod­els, pro­duc­ing an ex­pan­sive open world with forests, a log-cabin vil­lage, snowy moun­tains, and dy­namic weather. External as­sets used: an­i­mated cow­boy and horse mod­els and ter­rain data.

Chip Design

As an early proof of con­cept, Kimi K3 de­signed a chip to serve a nano model built on its own ar­chi­tec­ture. In a sin­gle 48-hour au­tonomous run, K3 built, op­ti­mized, and ver­i­fied the chip us­ing open-source EDA tools on the Nangate 45nm li­brary. Within 4 mm², the chip closes tim­ing at 100 MHz and sus­tains over 8,700 to­kens/​s de­code through­put in sim­u­la­tion, pack­ing 1.46M stan­dard cells, 0.277 MB of SRAM, and an INT4 MAC ar­ray with fused de­quan­ti­za­tion. A chip built by a model, for a model, re­flects K3′s long-hori­zon agen­tic ca­pa­bil­i­ties.

Coding for Research

Kimi K3 bridges sci­en­tific lit­er­a­ture and ex­e­cutable code, au­tonomously im­ple­ment­ing, val­i­dat­ing, and an­a­lyz­ing com­plex com­pu­ta­tional re­search work­flows.

In one case, Kimi K3 com­pleted in about two hours what would typ­i­cally re­quire one to two weeks of work by an ex­pe­ri­enced re­searcher. To re­pro­duce the I–Love–Q uni­ver­sal re­la­tions in com­pu­ta­tional as­tro­physics, it re­viewed and cross-val­i­dated 20+ pa­pers, im­ple­mented the full nu­mer­i­cal pipeline, eval­u­ated 300+ equa­tions of state, iden­ti­fied in­con­sis­ten­cies in pub­lished for­mu­las, gen­er­ated 3,000+ lines of Python code, and pro­duced an in­ter­ac­tive HTML dash­board for ex­plor­ing the re­sults.

Knowledge Work

Kimi K3 ad­vances end-to-end knowl­edge work. Beyond pub­lic bench­marks, Kimi K3 (max) demon­strates con­sis­tent gains across our in­ter­nal eval­u­a­tions, which are de­rived from re­cur­ring pat­terns and chal­lenges ob­served in real-world user-agent work­flows. These con­sis­tent ad­van­tages across dis­tinct pro­duc­tion-ori­ented work­flows re­flect a broad im­prove­ment in Kimi K3′s agen­tic knowl­edge work ca­pa­bil­i­ties.

Research with Interactive Visualization

Below are a few ex­am­ples of what Kimi K3 in Kimi Work can pro­duce across fi­nan­cial con­sult­ing and sci­en­tific re­search:

Case 1: Interactive 42 years of AI ASIC in­dus­try re­search web­site

An in­ter­ac­tive re­search re­port you can drill into: 42 years of the ASIC in­dus­try, cre­ated through 120+ rounds of re­cur­sive self-im­prove­ment. Kimi K3 trans­forms ev­i­dence into be­spoke charts, an­i­mated di­a­grams, and in­ter­ac­tive vi­sual nar­ra­tives. It pulled data via 2.8k+ web searches/​fetches and 1.1k+ ter­mi­nal data pulls, across 11k+ pages span­ning 87 quar­terly re­ports and 99 orig­i­nal PDFs.

Case 2: Fusion Industry Research

A con­sult­ing-style in­dus­try re­port with in­ter­ac­tive vi­su­al­iza­tions—in­clud­ing time­lines, Funnel Chart, Range Bar Chart, Gantt Charts, and pub­li­ca­tion-qual­ity slides.

Case 3: GWTC-5 Gravitational-wave Analysis

An analy­sis of 391 grav­i­ta­tional-wave events us­ing 20+ con­cur­rent sub­agents, pro­duc­ing 7 sci­en­tific vi­su­al­iza­tions, 2 ta­bles, and a lit­er­a­ture syn­the­sis from 10+ pa­pers.

Kimi K3 is also par­tic­u­larly ef­fec­tive at pro­duc­ing in­fo­graphic-style pre­sen­ta­tions, such as the fully ed­itable heatmap and an­nual re­port shown be­low:

Widgets and Dashboard

In Kimi Work, we in­tro­duce two new fea­tures - Widgets and Dashboard - which make in­ter­ac­tions with Kimi K3 more vi­sual and per­sis­tent. Widgets let you gen­er­ate in­ter­ac­tive com­po­nents di­rectly within a chat, with con­nec­tions to lo­cal data or ex­ter­nal plu­g­ins for con­tin­u­ous up­dates. Dashboard brings the wid­gets you care about most into one per­sis­tent, per­son­al­ized view or­ga­nized around a topic, pro­ject, or goal.

Video Editing

Kimi K3 ex­cels at mo­tion de­sign, an­i­ma­tion, and video edit­ing be­cause its na­tive mul­ti­modal ar­chi­tec­ture un­der­stands text, im­ages, and video within the same model.

In one ex­am­ple, K3 cre­ated a 3Blue1Brown-style mo­tion-graph­ics ex­plainer of its own ar­chi­tec­ture, trans­lat­ing tech­ni­cal ideas into an­i­mated di­a­grams and tran­si­tions.

In an­other, Kimi K3 edited its own teaser video from 56 source clips, han­dling clip se­lec­tion, mo­tion-matched cuts, frame-ac­cu­rate beat syn­chro­niza­tion, au­dio pro­cess­ing, and mul­ti­ple rounds of re­vi­sion. A high-den­sity short video like this would typ­i­cally take an ex­pe­ri­enced ed­i­tor one to two work­ing days, or a be­gin­ner three to five.

Architecture and Infrastructure

Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes). KDA pro­vides an ef­fi­cient foun­da­tion for scal­ing at­ten­tion, while AttnRes se­lec­tively re­trieves rep­re­sen­ta­tions across depth rather than ac­cu­mu­lat­ing them uni­formly. Together, they form the ar­chi­tec­tural back­bone of a model de­signed to scale well be­yond the tril­lion-pa­ra­me­ter regime.

Kimi K3 uses Stable LatentMoE, ef­fec­tively ac­ti­vat­ing 16 of 896 ex­perts. At this level of spar­sity, rout­ing and op­ti­miza­tion be­come first-or­der chal­lenges. Quantile Balancing de­rives ex­pert al­lo­ca­tion di­rectly from router-score quan­tiles, elim­i­nat­ing heuris­tic up­dates and a sen­si­tive bal­anc­ing hy­per­pa­ra­me­ter, while Per-Head Muon ex­tends Muon by op­ti­miz­ing at­ten­tion heads in­de­pen­dently for more adap­tive learn­ing at scale. Sigmoid Tanh Unit (SiTU) and Gated MLA im­prove ac­ti­va­tion con­trol and at­ten­tion se­lec­tiv­ity re­spec­tively. Together, these ad­vances en­able sta­ble and ef­fi­cient train­ing at the 2.8-trillion-parameter scale.

Kimi K3 ap­plies quan­ti­za­tion-aware train­ing from the SFT stage on­ward, us­ing MXFP4 weights with MXFP8 ac­ti­va­tions for broad hard­ware com­pat­i­bil­ity. To pre­vent ex­pert im­bal­ance from de­grad­ing through­put at large ex­pert-par­al­lel scales, we in­tro­duce a fully bal­anced ex­pert-par­al­lel train­ing method with sta­tic shapes and no host syn­chro­niza­tion on the crit­i­cal path. Since in­fer­ence ef­fi­ciency like­wise ben­e­fits from larger high-band­width com­mu­ni­ca­tion do­mains, we rec­om­mend de­ploy­ing Kimi K3 on su­pern­ode con­fig­u­ra­tions with 64 or more ac­cel­er­a­tors. Finally, as KDA poses new chal­lenges for con­ven­tional pre­fix caching, we have con­tributed a cor­re­spond­ing im­ple­men­ta­tion to the vLLM com­mu­nity, to be re­leased along­side the model. KDA with pre­fill cache al­lows us to serve Kimi K3 at a highly com­pet­i­tive to­ken price de­spite its scale and long con­text.

More tech­ni­cal de­tails will be avail­able in our com­ing re­port.

Availability

Kimi K3 Agents: Download or up­date to the lat­est Kimi app from your mo­bile app store, avail­able on iOS, Android, and HarmonyOS, or visit kimi.com.

Work with Kimi K3: Download the lat­est Kimi Work desk­top app, ver­sion 3.1.0 or later, avail­able for Windows and Apple sil­i­con Macs.

Code with Kimi K3: Run Kimi Code in your ter­mi­nal and se­lect Kimi K3 us­ing the /model com­mand.

Build with the Kimi API: Visit the Kimi API Platform and se­lect kimi-k3. Pricing is $0.30/MTok for cache-hit in­put, $3.00/MTok for cache-miss in­put, and $15.00/MTok for out­put. Powered by Mooncake’s dis­ag­gre­gated in­fer­ence ar­chi­tec­ture, the of­fi­cial Kimi API achieves a cache hit rate above 90% in cod­ing work­loads.

Bring Kimi to your or­ga­ni­za­tion: Kimi Enterprise pro­vides en­ter­prise-grade data pri­vacy and mem­ber man­age­ment, with com­plete sep­a­ra­tion be­tween per­sonal and or­ga­ni­za­tion ac­counts. Visit the pric­ing page and se­lect “Get Kimi Enterprise” to sub­scribe for your team.

Full Benchmark Table

Footnotes

All Kimi K3 re­sults re­ported be­low are ob­tained with the rea­son­ing ef­fort set to ‘max’, set­ting tem­per­a­ture = 1.0 and top-p = 1.0. Depending on the bench­mark, each model is eval­u­ated un­der one of three agen­tic har­nesses — KimiCode, Claude Code, or Codex — as spec­i­fied in the notes be­low.

Coding bench­marks

DeepSWE. Kimi K3 is eval­u­ated with the KimiCode har­ness. The GLM-5.2 score is taken from the GLM-5.2 re­lease blog (https://​z.ai/​blog/​glm-5.2); all re­main­ing scores are from the of­fi­cial DeepSWE leader­board (https://​deep­swe.dat­acurve.ai/), un­der which Kimi K3 at­tains 67.3 with the mini-SWE-agent har­ness.

Terminal-Bench 2.1. Kimi K3 is eval­u­ated with the KimiCode har­ness. For all other mod­els, we re­port the best score across har­nesses: GLM-5.2 with Claude Code (https://​z.ai/​blog/​glm-5.2); Claude Opus 4.8 and Claude Fable 5 with Terminus 2 (https://​ar­ti­fi­cial­analy­sis.ai/​eval­u­a­tions/​ter­mi­nal­bench-v2 – 1); GPT 5.5 and GPT 5.6 Sol with Codex (https://​ope­nai.com/​in­dex/​pre­view­ing-gpt-5 – 6-sol/).

Program Bench. Kimi K3 is eval­u­ated with the KimiCode har­ness. The GLM-5.2 score is from https://​z.ai/​blog/​glm-5.2; all other scores are from https://​www.vals.ai/​bench­marks/​pro­gram­bench.

SWE Marathon. Kimi K3, Claude Opus 4.8, and Claude Fable 5 are eval­u­ated with the Claude Code har­ness; GPT 5.6 Sol is eval­u­ated with the Codex har­ness. The GLM-5.2 score is from https://​z.ai/​blog/​glm-5.2.

FrontierSWE. Kimi K3 is eval­u­ated with the KimiCode har­ness and GPT 5.6 Sol with the Codex har­ness; all other re­sults are from https://​www.fron­tier­swe.com/. Dominance scores are re­com­puted from the raw scores us­ing the of­fi­cial eval­u­a­tion script and are cur­rent as of July 16, 2026.

PostTrain Bench. Scores for GLM-5.2, GPT 5.5, and Claude Opus 4.8 are adopted from the of­fi­cial PostTrainBench re­sults. Kimi K3, Claude Fable 5, and GPT 5.6 Sol are eval­u­ated with the of­fi­cial Harbor im­ple­men­ta­tion at max­i­mum rea­son­ing ef­fort, av­er­aged over three runs — Kimi K3 and Claude Fable 5 with the Claude Code har­ness, and GPT 5.6 Sol with the Codex har­ness. Under the Claude Code har­ness, re­quests re­fused by Claude Fable 5 due to its us­age pol­icy au­to­mat­i­cally fall back to Claude Opus 4.8.

MLS Bench Lite. Kimi K3 is eval­u­ated with the KimiCode har­ness; GLM-5.2 and the Claude mod­els with the Claude Code har­ness; GPT 5.5 and GPT 5.6 Sol with the Codex har­ness.

KCB 2.0. Kimi K3 is eval­u­ated with both the KimiCode and Claude Code har­nesses; GLM-5.2, Claude Opus 4.8, and Claude Fable 5 with the Claude Code har­ness; GPT 5.5 and GPT 5.6 Sol with the Codex har­ness. All mod­els are eval­u­ated at max­i­mum rea­son­ing ef­fort, ex­cept GPT 5.5, which uses the “xhigh” set­ting.

Productivity and agen­tic bench­marks

OfficeQA Pro and SpreadsheetBench 2. Kimi K3, GLM-5.2, Claude Opus 4.8, and Claude Fable 5 are eval­u­ated with the Claude Code har­ness; GPT 5.5 and GPT 5.6 Sol are eval­u­ated with the Codex har­ness.

MCP Atlas. All mod­els are eval­u­ated on the 500-task pub­lic sub­set with a 100-turn limit, us­ing Gemini 3.1 Pro as the judge.

AutomationBench. All mod­els are eval­u­ated on the 600-task pub­lic sub­set, fol­low­ing the of­fi­cial GitHub setup in all other re­spects.

BrowseComp. We adopt the con­text-com­paction strat­egy used in the Claude model cards, trig­gered at 300K to­kens. When eval­u­ated with a 1M-token con­text win­dow and no con­text man­age­ment, Kimi K3 achieves a score of 90.4. The re­sults of Claude Fable 5, Claude Opus 4.8, GPT 5.6 Sol, and GPT 5.5 are cited from https://​www.an­thropic.com/​news/​claude-fa­ble-5-mythos-5 and https://​ope­nai.com/​in­dex/​gpt-5 – 6/.

GDPval-AA v2 and AA-Briefcase scores are cited from https://​ar­ti­fi­cial­analy­sis.ai/.

Multimodal bench­marks

Except for ZeroBench, which fol­lows the of­fi­cial set­ting and is run five times, all mul­ti­modal scores are av­er­aged over three runs. MMMU-Pro is eval­u­ated fol­low­ing the of­fi­cial pro­to­col, pre­serv­ing the orig­i­nal in­put or­der and prepend­ing im­ages to the text in­put.

PerceptionBench. PerceptionBench is an in-house bench­mark that fo­cuses on atomic vi­sual per­cep­tion ca­pa­bil­i­ties.

Limitations

Sensitivity to think­ing his­tory. K3 was trained in the pre­served think­ing his­tory mode. If the agent har­ness fails to pass back all the his­tor­i­cal think­ing con­tent as re­quired, or if an on­go­ing ses­sion with an­other model is switched over to K3, gen­er­a­tion qual­ity may be­come highly un­sta­ble. We rec­om­mend us­ing a har­ness with ver­i­fied com­pat­i­bil­ity, such as Kimi Code, and avoid­ing switch­ing to K3 in the mid­dle of a ses­sion.

Excessive proac­tive­ness. K3′s train­ing places par­tic­u­lar em­pha­sis on long-hori­zon, chal­leng­ing tasks. As a re­sult, when it en­coun­ters mi­nor is­sues or am­bigu­ous user in­tent dur­ing task ex­e­cu­tion, it may make un­ex­pected de­ci­sions on the user’s be­half. If your ap­pli­ca­tion re­quires the agent to op­er­ate within well-de­fined bound­aries and re­frain from ex­ces­sive im­pro­vi­sa­tion, please im­pose more ex­plicit be­hav­ioral con­straints on K3 in the sys­tem prompt or in AGENTS.md.

Despite be­ing a highly com­pet­i­tive model over­all, K3 nonethe­less ex­hibits a no­tice­able gap in user ex­pe­ri­ence com­pared with Claude Fable 5 and GPT 5.6 Sol.

Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing.

from the poof-it’s-gone dept

In all of our dis­cus­sions about how the dig­i­tal rev­o­lu­tion has cre­ated a sys­tem in which peo­ple don’t ac­tu­ally own the things they think they’re buy­ing, I get par­tic­u­larly frus­trated by the lack of change in it all. We’ve spilled much ink com­plain­ing that this clearly anti-con­sumer prac­tice needs to be done away with, where an un­sus­pect­ing pub­lic thinks they’re buy­ing “a thing” only to learn months or years later that “the thing” they bought was ac­tu­ally a li­cense to use/​view/​lis­ten to an­other “thing”, and that li­cense ex­ists at the plea­sure of the com­pany that col­lected the money for it. And if you want to see the lack of change or ac­tion re­ally honed in upon, let’s take a look at Sony’s PlayStation Store.

In 2022, due to “evolving li­cens­ing agree­ments” with dis­trib­u­tor StudioCanal, German and Austrian users had hun­dreds of movies dis­ap­pear from their PS ac­counts, long af­ter buy­ing them through Sony. Then in 2023, it hap­pened again in America, specif­i­cally when Sony ended its li­cens­ing agree­ment with Discovery af­ter the Warner Bros. merger, which, of course, has since been bought by Paramount Skydance. That re­sulted in cus­tomers hav­ing hun­dreds and hun­dreds of episodes of TV shows deleted from their ac­counts. Nowhere in any of this were there re­funds, of course. No rec­om­pense at all, ac­tu­ally. Just a thing you thought you’d bought taken away from you by the very peo­ple you thought you bought it from.

And now it’s hap­pen­ing again. Due to an­other li­cens­ing agree­ment fall­out with StudioCanal, hun­dreds of movies and TV shows are be­ing ripped from the ac­counts of PS Store cus­tomers, and there ap­pears to be fuck all that they can do about it.

This news was brought to peo­ple’s at­ten­tion by X user so­matyk, who posted the no­ti­fi­ca­tion they had re­ceived from PlayStation this week. Along with the un­apolo­getic news that the pur­chased movies would be deleted from their ac­count on September 1, the mes­sage con­cluded with, “Click here for a full list of af­fected ti­tles that will no longer be sup­ported. Thank you.” The same warn­ing is now re­pro­duced in full on the PlayStation web­site, along with the list of 551 films and TV se­ries that are be­ing pulled from peo­ple’s li­braries.

This news was brought to peo­ple’s at­ten­tion by X user so­matyk, who posted the no­ti­fi­ca­tion they had re­ceived from PlayStation this week. Along with the un­apolo­getic news that the pur­chased movies would be deleted from their ac­count on September 1, the mes­sage con­cluded with, “Click here for a full list of af­fected ti­tles that will no longer be sup­ported. Thank you.” The same warn­ing is now re­pro­duced in full on the PlayStation web­site, along with the list of 551 films and TV se­ries that are be­ing pulled from peo­ple’s li­braries.

As Kotaku notes later in their post, part of what is strik­ing in all of this is the sheer mun­dan­ity of the an­nounce­ment. Because there have been no con­se­quences, or any ac­tion at all from the pub­lic or gov­ern­ment, Sony treats this all as if it’s per­fectly nor­mal and no big deal. You can tell me all you want about how the Ts and Cs in these pur­chases do in fact note that the na­ture of the pur­chase is a tem­po­rary li­cens­ing of the con­tent for an un­de­ter­mined time pe­riod… but I can promise you that the pub­lic in gen­eral does­n’t un­der­stand that. They think they’re buy­ing a thing, not a li­cense.

And that’s be­cause of the pur­pose­ful ob­fus­ca­tion of that fact. Sony damned well knows that the vast ma­jor­ity of peo­ple don’t read those Ts and Cs. It knows that the pub­lic largely does­n’t un­der­stand how these back­end li­cens­ing agree­ments with dis­trib­u­tors work, or that they even ex­ist. And Sony is­n’t ex­actly putting out a big blink­ing sign on its store pages in­form­ing the pub­lic of all of this. Instead, the com­pany is only too happy to col­lect money from a pub­lic that is be­ing pur­pose­fully kept ig­no­rant of what they’re buy­ing.

Of course, when you scroll past the end­less EULAs when you first use your PlayStation, and click “Agree” the first time you load the store, you’re un­wit­tingly agree­ing that noth­ing you buy is re­ally truly bought, and that it can be taken away from you at any point, and there’s noth­ing you can do. The same is true of your games.

Of course, when you scroll past the end­less EULAs when you first use your PlayStation, and click “Agree” the first time you load the store, you’re un­wit­tingly agree­ing that noth­ing you buy is re­ally truly bought, and that it can be taken away from you at any point, and there’s noth­ing you can do. The same is true of your games.

This, too, will prob­a­bly pass with­out any real ac­tion. The gov­ern­ment has done its best to gut our con­sumer pro­tec­tion agen­cies, so they won’t be any help. Angry cus­tomers won’t co­a­lesce into ac­tivism or ac­tion, most likely. And I’ll prob­a­bly be writ­ing an­other one of these posts in a cou­ple of years when it all hap­pens again.

But it should­n’t be that way. There are com­mon sense things that can be done to bet­ter in­form the pub­lic. Rules for how the store should in­form peo­ple with each and every pur­chase. Someone just needs to de­mand it be done.

Filed Under: eula, own­er­ship, playsta­tion, playsta­tion store, video games

Companies: sony, stu­dio­canal

The chat client that brought Comic Sans to the world is now on GitHub

Today, we’re ex­cited to an­nounce the open-source re­lease of Microsoft Comic Chat, the chat client that au­to­mat­i­cally turned con­ver­sa­tions within Internet Relay Chat (IRC) into comic pan­els fea­tur­ing il­lus­trated char­ac­ters, speech bub­bles, and ex­pres­sions, and helped in­tro­duce the world to a lit­tle font called Comic Sans.

Yes, that Comic Sans. Originally de­signed by Microsoft ty­pog­ra­pher Vincent Connare in 1994, Comic Sans found its first real home in Comic Chat, where its in­for­mal, hand-let­tered feel matched the soft­ware’s speech-bub­ble con­ver­sa­tions per­fectly.

For many peo­ple, Comic Chat is a nos­tal­gic ar­ti­fact from the early days of the in­ter­net as we tran­si­tioned from tech­nolo­gies like tel­net, Usenet, and IRC to the largely vi­sual web that we en­joy to­day. For oth­ers, it’s a leg­endary piece of Microsoft his­tory they have only heard about in sto­ries, screen­shots, and de­bates about ty­pog­ra­phy. Now, de­vel­op­ers, his­to­ri­ans, retro com­put­ing en­thu­si­asts, and any­one who ap­pre­ci­ates a won­der­fully un­con­ven­tional idea can ex­plore the source code for them­selves.

A dif­fer­ent vi­sion for on­line com­mu­ni­ca­tion

Today we’re ac­cus­tomed to mes­sag­ing apps with re­ac­tions, stick­ers, GIFs, avatars, video, and AI-generated con­tent. But in the mid-1990s, in­ter­net chat was largely walls of scrolling text.

Rather than dis­play­ing mes­sages as plain text, Comic Chat pre­sented par­tic­i­pants as il­lus­trated char­ac­ters. Conversations un­folded in comic pan­els, with speech bub­bles, ex­pres­sions, and ges­tures gen­er­ated from what peo­ple typed. If some­one wrote “I like that,” the char­ac­ter might point to it­self. If the text sug­gested anger, the char­ac­ter might frown or cross its arms. It was quirky, am­bi­tious, oc­ca­sion­ally chaotic, and sur­pris­ingly for­ward-look­ing.

Many ideas we now take for granted in on­line com­mu­ni­ca­tion can trace some of their spirit to ex­per­i­ments like Comic Chat.

The peo­ple who built it

David “DJ” Kurlander, work­ing in the Microsoft Research Virtual Worlds Group, con­ceived the idea of a new vi­sual rep­re­sen­ta­tion of con­ver­sa­tional his­to­ries, and started de­vel­op­ing Comic Chat in 1995. Built in Visual C++ 4.0 and MFC, Comic Chat was re­leased in 1996 with the Internet Explorer 3 web browser.

Under the hood, Comic Chat was more than a clever skin for IRC. It was able to in­ter­pret con­ver­sa­tional cues in the text and choose ap­pro­pri­ate poses, fa­cial ex­pres­sions, ges­tures, and panel lay­outs. That meant Comic Chat was not sim­ply dis­play­ing mes­sages but also mak­ing real-time ed­i­to­r­ial de­ci­sions about how a con­ver­sa­tion should look and feel as a comic. DJ, Tim Skelly, and David Salesin pub­lished a pa­per on the tech­nol­ogy in Comic Chat at SIGGRAPH ’96, a com­puter graph­ics con­fer­ence, de­scrib­ing what they had built as an ex­per­i­ment in au­to­matic il­lus­tra­tion con­struc­tion and lay­out.

The vi­sual world of Comic Chat was the work of Jim Woodring, a highly re­garded in­de­pen­dent comic artist whose char­ac­ters gave the soft­ware its dis­tinc­tive look. The team would hand Jim tran­scripts of real chat ses­sions to il­lus­trate, then use the re­sults to fig­ure out whether the whole idea was worth pur­su­ing. It was.

Why open source it now?

Comic Chat rep­re­sents a fas­ci­nat­ing chap­ter in the evo­lu­tion of on­line com­mu­ni­ca­tion. It emerged dur­ing a pe­riod when the in­ter­net was still dis­cov­er­ing what it wanted to be­come. Many rules had not yet been writ­ten, which gave de­vel­op­ers per­mis­sion to try bold con­cepts that might seem un­usual even to­day.

By re­leas­ing Comic Chat as open source, we’re pre­serv­ing an im­por­tant piece of soft­ware his­tory and giv­ing the com­mu­nity an op­por­tu­nity to ex­plore, learn, and build upon it.

The source is avail­able now for ex­plo­ration, study, and ex­per­i­men­ta­tion. Alongside the orig­i­nal snap­shots, we’ve in­cluded a few AI-powered mod­ern­iza­tion at­tempts that demon­strate what’s pos­si­ble—get­ting this 1990s-era C++ and MFC code build­ing with cur­rent Visual Studio tools, con­nect­ing to mod­ern IRC servers, and run­ning leg­i­bly on to­day’s high-res­o­lu­tion Windows ma­chines. These are not pol­ished re-re­leases, but worked ex­am­ples that show Comic Chat can still come alive on mod­ern sys­tems. We’re ex­cited to see what im­prove­ments, ports, ex­per­i­ments, and en­tirely new forms the com­mu­nity brings to it next.

A time cap­sule of in­ter­net op­ti­mism

Looking back, Comic Chat cap­tures some­thing spe­cial about the era in which it was cre­ated.

The early web was filled with ex­per­i­men­ta­tion. “What if chat rooms looked like comics?” That ques­tion sounds won­der­fully un­rea­son­able. And yet it was built, shipped, lo­cal­ized into 24 lan­guages, and bun­dled with Windows 98.

That’s part of what makes Comic Chat mem­o­rable decades later. It re­minds us that in­no­va­tion of­ten starts with ideas that are play­ful, un­con­ven­tional, and cre­ative.

One last speech bub­ble

Comic Chat was cre­ated dur­ing a pe­riod when soft­ware teams were will­ing to color out­side the lines, lit­er­ally and fig­u­ra­tively. DJ Kurlander, Tim Skelly, David Salesin, Jim Woodring, and every­one else who touched this pro­ject made some­thing that peo­ple still re­mem­ber and still run thirty years later.

Take a look at the source code, ex­plore what they built, and use its story as in­spi­ra­tion to come up with new un­con­ven­tion­ally de­light­ful things to cre­ate.

And if you hap­pen to read the source code in Comic Sans, we promise not to judge.

For the past year and a half, the team build­ing Roc’s com­piler has been rewrit­ing our 300,000 lines of Rust code into Zig, for rea­sons I’ll re­cap be­low. We re­cently passed an ex­cit­ing mile­stone: fea­ture par­ity with the orig­i­nal com­piler!

Since the Bun pro­ject re­cently shared an ex­pe­ri­ence re­port of their rewrite in the other di­rec­tion (from Zig to Rust, al­though that’s only the tip of the ice­berg of dif­fer­ences be­tween our rewrites), this seems like a nice time to re­flect on how our move from Rust to Zig is go­ing.

Passing Feature Parity

Hitting this mile­stone made it pos­si­ble to up­date Brendan Hansknecht’s charm­ing 2024 WASM-4 game, Rocci Bird (with art by Luke DeVault) to use the new com­piler. It’s a nice ex­am­ple be­cause the whole game is un­der a thou­sand lines of Roc code, and you can play it on itch.io or right here via WebAssembly:

Click or tap the game, then press Space (or tap) to flap. On mo­bile you don’t have a right ar­row key, so re­fresh the page to restart the game.

Rocci Bird’s up­dated source code is a bit more con­cise than the orig­i­nal, and roc build –opt=size now out­puts a 31KB wasm bi­nary. (The orig­i­nal com­piler pro­duced a bi­nary more than dou­ble that size.) Rocci Bird is by no means a large code base, but get­ting it to run at all re­quired land­ing a lot of fea­tures in the new com­piler. Seeing those chunky pur­ple pix­els brought a smile to my face when we fi­nally got there!

To be clear, this is a mile­stone but not a for­mal re­lease. (We aim to land ver­sion 0.1.0 later this year.) That said, it’s a won­der­ful mile­stone to have reached, and I’m ex­tremely grate­ful to all the peo­ple who came to­gether to make this hap­pen! I want to thank some in par­tic­u­lar who have been es­pe­cially help­ful in get­ting the lan­guage and com­piler to this point:

Anthony Bullard and Sam Mohr for col­lab­o­rat­ing on the new parser

Jared Ramirez for the new type-checker (among many other things!)

Ayaz Hafiz for the new lambda set res­o­lu­tion sys­tem, plus tons of the orig­i­nal com­piler

Aurélien Geron for hand-up­dat­ing 108 (!) be­gin­ner ex­er­cises in the Roc Exercism course he orig­i­nally cre­ated

Stephan for get­ting the com­pil­er’s new “echo” plat­form run­ning in the browser, so that any­one can now write and run ba­sic Roc pro­grams from the roc-lang.org home­page via a 2.5MB WebAssembly bi­nary!

Niclas Åhdén, Roc’s most pro­lific pro­duc­tion user, for pa­tiently fil­ing help­ful bug re­ports and giv­ing ac­tion­able feed­back about the up­grade process

JRI98 for me­thod­i­cally re­pro­duc­ing and in­ves­ti­gat­ing fuzzer er­rors and other bugs, clos­ing out is­sues that no longer re­pro­duced, and more

Jasper Woudenberg for it­er­at­ing on API de­signs for user­space pack­ages us­ing the new com­piler

Folkert de Vries, Brendan Hansknecht, Brian Carroll, Josh Warner, Agus Zubiaga, and Jelle Teeuwissen for build­ing the foun­da­tion of the orig­i­nal com­piler, with­out which the new com­piler never would have ex­isted

I’ve saved the undis­puted biggest con­trib­u­tors to the new com­piler for last: Anton-4 and Luke Boswell for so many things I can’t even keep track of them all—com­piler work, builtins, plat­forms, pack­ages, ex­am­ples, fix­ing bugs, help­ing be­gin­ners on Roc Zulip…enumerating it all could take up a whole sec­ond post! It’s been in­cred­i­ble see­ing how much you’ve built.

Thank you all so much! I feel hon­ored that you’ve put so much of your valu­able time into this pro­ject. Also thanks to our past and pre­sent spon­sors—rwx, Lambda Class, ohne-mak­ler, mar­t­ian, tweede golf, Vendr, NoRedInk, and many gen­er­ous in­di­vid­ual spon­sors—who have helped get us to this point by sup­port­ing our con­trib­u­tors.

Speaking of time: our 487-day rewrite took 476 days longer than Bun’s 11-day rewrite from their ~500K lines of Zig into Rust. There are many rea­sons for this dif­fer­ence which have noth­ing to do with Rust or Zig, in­clud­ing the fact that theirs was a di­rect port whereas we’d de­cided to rewrite be­cause of how much we were go­ing to change. The tech­niques they used would­n’t have worked in our case.

The laun­dry list of changes we made also means com­par­ing our orig­i­nal Rust code base and new Zig code base won’t be ap­ples-to-ap­ples. Still, we’ve reached a nice point to re­flect on how the rewrite has gone, both in terms of what new fea­tures it has un­locked for Roc pro­gram­mers, as well as how our ex­pe­ri­ences with Rust and Zig have com­pared.

Let’s get into it!

Hot Code Loading + Cross-Compiled Binaries

Roc’s new com­piler au­to­mat­i­cally does hot code load­ing dur­ing de­vel­op­ment. For ex­am­ple, I can run roc server.roc to start a Web server, then change some of its code while it’s run­ning. The next time that server han­dles a re­quest, it’ll au­to­mat­i­cally be han­dled us­ing the new code. Here it is in ac­tion, both in a server and in a sim­ple 2D game:

Hot load­ing is stan­dard be­hav­ior for in­ter­preted lan­guages like Python, but not so much for high-per­for­mance com­piled lan­guages like Roc. When I’m ready to de­ploy, roc build server.roc gets me an LLVM-optimized, self-con­tained bi­nary that I can drop onto a ma­chine and run.

Roc also cross-com­piles; build­ing a sta­tic bi­nary that runs on Alpine Linux is as sim­ple as roc build –target=x64musl, and that com­mand will pro­duce the same out­put bytes (for the same in­put source code bytes) when run on a Mac or any other sys­tem—which not all com­pil­ers guar­an­tee.

Pattern Matching with String Interpolation

The HTTP re­quest-han­dling logic from that video looks like this:

match (verb, path) { (“GET”, “/users/${id}/${page}“) => match page { “” | “profile” => ok(id) “settings” => ok(with­_de­fault(user_a­gent, id)) “posts/${post_id}” => ok(“Post ID: ${post_id}“) _ => not_­found }

(“GET”, “/users/${id}“) => ok(id)

(“POST”, “/posts/new”) => cre­ated(with­_de­fault(…))

_ => not_­found }

This uses sev­eral fea­tures we in­tro­duced in the new com­piler. For ex­am­ple, that “/users/${id}” syn­tax is not im­ple­mented with pars­ing tem­plate strings at run­time, but rather with a new lan­guage fea­ture: string in­ter­po­la­tion in­side pat­tern match­ing.

Not only is this type-safe at com­pile time, this en­tire code snip­pet per­forms zero heap al­lo­ca­tions. I’d ex­pect the typ­i­cal lan­guage that ships with hot code load­ing to av­er­age closer to 1 al­lo­ca­tion per line of code here…but Roc is aim­ing high on er­gonom­ics, type safety, and per­for­mance!

You can play around with this syn­tax on the new roc-lang.org home­page - if you scroll down a bit, there’s an WebAssembly build of the com­piler right there on the page that you can use to try out the lan­guage.

By the way, if you’re in­ter­ested in a post on the tech­ni­cal de­tails of how we used the new com­pil­er’s com­pile-time ex­e­cu­tion of pure func­tions to get HTTP re­quest rout­ing down to zero al­lo­ca­tions, let me know on Roc Zulip.

By the way, if you’re in­ter­ested in a post on the tech­ni­cal de­tails of how we used the new com­pil­er’s com­pile-time ex­e­cu­tion of pure func­tions to get HTTP re­quest rout­ing down to zero al­lo­ca­tions, let me know on Roc Zulip.

Why a Scratch-Rewrite?

Unlike Rust, C, and Zig, Roc is not a sys­tems lan­guage; it has au­to­matic mem­ory man­age­ment (using ref­er­ence count­ing, both to avoid trac­ing col­lec­tor pauses and also for Perceus op­ti­miza­tions and op­por­tunis­tic mu­ta­tion like Koka’s). Roc would have way more heap al­lo­ca­tions if it needed one heap al­lo­ca­tion per clo­sure cap­ture (like most non-sys­tems lan­guages do), but our clo­sure cap­tures don’t heap-al­lo­cate be­cause Roc is the first non-aca­d­e­mic lan­guage to im­ple­ment poly­mor­phic de­func­tion­al­iza­tion through lambda set spe­cial­iza­tion.

This might sound like a niche op­ti­miza­tion, but in a func­tional lan­guage like Roc, de­func­tion­al­iza­tion turns out to be sim­i­lar to in­lin­ing in that it un­locks a trea­sure trove of fol­low-up op­ti­miza­tions. Although this sys­tem proved in­cred­i­bly ben­e­fi­cial to Roc’s run­time per­for­mance, it also proved in­cred­i­bly dif­fi­cult for us to im­ple­ment cor­rectly. We strug­gled with nasty bugs in the orig­i­nal im­ple­men­ta­tion, and only af­ter Ayaz Hafiz pro­to­typed a new ar­chi­tec­ture in OCaml were we able to fi­nally get it right in the new com­piler.

Ayaz’s pro­to­type showed that the root of our prob­lems was ar­chi­tec­tural across sev­eral com­piler phases, and fix­ing it would re­quire rewrit­ing most of the com­piler. This was one rea­son we de­cided to rewrite in the first place—that, and sev­eral con­trib­u­tors in­de­pen­dently men­tion­ing they planned to rewrite var­i­ous parts of the com­piler for other rea­sons. We re­al­ized we were about to rewrite al­most all of the com­piler any­way, so it made sense to con­sider a full rewrite as an al­ter­na­tive to the Ship of Theseus ap­proach.

Compilers are un­usual in that scratch-rewrites are the norm among suc­cess­ful pro­jects. It’s of­ten the only way to self-host, al­though not all com­pil­ers rewrite into their own lan­guage; see for ex­am­ple TypeScript’s rewrite to Go. My po­si­tion has al­ways been that Roc’s com­piler should not self-host, so the idea that some­day the ben­e­fits of a rewrite might seem to out­weigh their no­to­ri­ous costs had frankly never oc­curred to me.

The more we talked about it, the more sense it made to do what ba­si­cally every main­stream com­piler to­day has done at some point: rewrite from scratch.

Why Zig?

Once we’d de­cided to scratch-rewrite, the next ques­tion was whether to choose Rust again. Based on our ex­pe­ri­ences with both Rust and Zig (we were al­ready us­ing Zig for a bunch of prim­i­tives in our stan­dard li­brary), we de­cided to build the en­tire com­piler in Zig this time.

I en­joy Rust, I’ve taught a course on it, and I hap­pily use it daily for my work at Zed. Despite what Internet com­ments might have us be­lieve, it’s ex­tremely nor­mal for one lan­guage to be the best fit for one pro­ject, while a dif­fer­ent lan­guage turns out to be the best fit for a dif­fer­ent pro­ject. One size does not ac­tu­ally fit all!

I’ve talked in depth about our rea­sons for go­ing with Zig else­where—in writ­ing, on pod­casts, and so on—and we only se­ri­ously con­sid­ered Rust and Zig, be­cause those were the only sys­tems lan­guages our team knew well enough. The biggest con­sid­er­a­tions on our minds when de­cid­ing be­tween Rust and Zig were:

Build times. Our cargo build times were a ma­jor pain point, even for in­cre­men­tal builds, and get­ting worse as our code base grew. We ex­pected build times in a Zig rewrite to be much faster.

Memory con­trol. We use a va­ri­ety of dif­fer­ent mem­ory al­lo­ca­tors through­out com­pi­la­tion, es­pe­cially are­nas, and struct-of-ar­rays lay­outs all over the place. Rust’s ecosys­tem con­sis­tently as­sumes one global al­lo­ca­tor, in­clud­ing soa_rs. Zig’s whole ecosys­tem as­sumes gran­u­lar al­lo­ca­tors, and struct-of-ar­rays sup­port is stan­dard.

Ecosystem rel­e­vance. Rust’s ecosys­tem is much big­ger than Zig’s over­all…but al­most no pack­ages in ei­ther ecosys­tem are rel­e­vant to our par­tic­u­lar needs. For the niche things we wanted to get off the shelf—such as a faster way to emit LLVM bit­code than wrap­ping LLVM’s C++ li­brary—more of that code ex­isted in Zig than in Rust.

Memory-unsafety as­sis­tance. Rust is de­signed to iso­late mem­ory-un­safe code in­side rare un­safe blocks, and use things like miri or Valgrind to vet those. Memory-unsafe code was­n’t rare for us, though (more on this later) and we ended up with about 1,200 uses of un­safe (out of our 300K lines of Rust code; com­pare to about 40,000 uses of un­safe in rust’s 3.5M lines, and re­mem­ber that for com­pil­ers which emit ma­chine code, like roc and rustc, do­ing mem­ory-un­safe things is a big part of the job). Zig has more fea­tures than Rust for mak­ing mem­ory-un­safe code work cor­rectly, and that was the area where we wanted the most help.

After a year and a half of rewrit­ing, how did our ex­pec­ta­tions of Zig’s ben­e­fits line up with the re­al­ity of what we got? And which parts of Rust did we end up miss­ing once we no longer had ac­cess to them?

Life Without Borrow-Checking

Let’s start with mem­ory safety. There’s a fa­mous 2019 Microsoft pre­sen­ta­tion that says, on slide 10:

~70% of the vul­ner­a­bil­i­ties ad­dressed through a se­cu­rity up­date each year con­tinue to be mem­ory safety is­sues.

~70% of the vul­ner­a­bil­i­ties ad­dressed through a se­cu­rity up­date each year con­tinue to be mem­ory safety is­sues.

The pre­sen­ta­tion’s next slide has a break­down by type of mem­ory safety is­sue, which paints the fol­low­ing pic­ture when it comes to Rust and Zig specif­i­cally:

83.6% of vul­ner­a­bil­i­ties ad­dressed through a se­cu­rity up­date in 2018 would have been com­pletely un­af­fected by the choice of Rust or Zig, be­cause both lan­guages han­dle all of these sce­nar­ios (out-of-bounds reads/​write, un­safe casts, unini­tial­ized reads, stack over­flows, and non-mem­ory-safety is­sues) in the same way.

16.4% of the vul­ner­a­bil­i­ties were specif­i­cally use-af­ter-free er­rors. These could have been caught by Zig’s ReleaseSafe run­time mem­ory-safety checks, or Rust’s bor­row checker, or the checks Fil-C uses…mod­ern lan­guages have a va­ri­ety of ways to help catch UAFs, al­though these CVEs from 2018 would have al­most cer­tainly been from C or C++ code in­stead.

ReleaseSafe catches use-af­ter-free er­rors through run­time checks which panic if the pro­gram tries to use freed mem­ory. Compared to Rust’s safe sub­set, Zig’s checks are less com­pre­hen­sive, have a run­time cost, and can panic. That said, Zig with ReleaseSafe has worked great in prac­tice for the TigerBeetle data­base, which re­cently un­der­went a leg­en­dar­ily metic­u­lous Jepsen re­port that found only two safety bugs, nei­ther re­lated to mem­ory safety.

ReleaseFast skips these checks in pro­duc­tion builds to avoid their over­head, but keeps them in de­bug builds and tests to catch mem­ory-safety is­sues dur­ing de­vel­op­ment. If your tests cov­ered every pos­si­ble real-world code path, ReleaseFast would give you the same safety as ReleaseSafe, but that level of test cov­er­age is rarely prac­ti­cal; the real ques­tion is what slips through the cov­er­age cracks in prac­tice. Bun talked about their strug­gles with use-af­ter-frees, but other widely-used pro­jects build­ing with ReleaseFast have had no CVEs caused by mem­ory un­safety in their Zig code. Ghostty is one, and Zig’s com­piler it­self is an­other.

If you want to learn more about these pro­jects, I’ve recorded in-depth con­ver­sa­tions with their cre­ators: Joran Greef on TigerBeetle, Mitchell Hashimoto on Ghostty, and Andrew Kelley on Zig.

If you want to learn more about these pro­jects, I’ve recorded in-depth con­ver­sa­tions with their cre­ators: Joran Greef on TigerBeetle, Mitchell Hashimoto on Ghostty, and Andrew Kelley on Zig.

Rust code has a dif­fer­ent source of mem­ory-safety gaps: the un­safe sec­tions that nearly every Rust pro­gram has some­where in its de­pen­den­cies. Unsafe Rust has all the mem­ory un­safety risk of ReleaseFast Zig code, but none of the run­time checks to catch is­sues dur­ing de­vel­op­ment. The Rust ecosyt­sem has miri to find bugs in non-FFI un­safe code, and Valgrind can help too, but few Rust pro­jects use ei­ther. That said, the cul­tural norm of us­ing un­safe rarely, and au­dit­ing it ex­tra care­fully, has worked out well enough to earn Rust a strong rep­u­ta­tion for mem­ory safety in prac­tice.

Of course, Rust mem­ory un­safety er­rors can and do still slip through the cracks. Deno, a Bun com­peti­tor which is writ­ten in Rust, has had mem­ory-un­safety CVEs in­clud­ing an out-of-bounds read as well as a use-af­ter-free, both in­volv­ing the use of Unsafe Rust. Rocket, a Rust Web Framework, has had a use-af­ter-free CVE, and Actix has had a va­ri­ety of mem­ory-un­safety CVEs from a pe­riod when its use of un­safe was ab­nor­mally high.

When we were de­cid­ing be­tween Rust and Zig for the new com­piler, we were aware of all of this. We knew Rust had a well-de­served rep­u­ta­tion for mem­ory safety, but that mem­ory un­safety could still hap­pen, and we’d ex­pe­ri­enced all of that first­hand with the orig­i­nal com­piler. We also knew we’d be us­ing un­safe way more than typ­i­cal Rust pro­jects, and even though we were al­ready us­ing Valgrind, get­ting help with in­nately mem­ory-un­safe code from Zig’s ad­di­tional checks sounded ap­peal­ing. We wanted the hard stuff to get eas­ier, and we weren’t wor­ried about use-af­ter-free is­sues in a com­piler where al­lo­ca­tions would be over­whelm­ingly done in are­nas with straight­for­ward life­times.

We knew high-pro­file Zig pro­jects had achieved great per­for­mance and mem­ory safety in prac­tice, and we de­cided to aim for be­com­ing an­other of those suc­cess sto­ries.

Memory Safety Post-Rewrite

It’s easy to the­o­rize about how things will go with a par­tic­u­lar tech­nol­ogy choice, but where the rub­ber meets the road is what end users en­counter in real-world us­age. So how has Zig with ReleaseFast worked out for us in prac­tice? How many mem­ory cor­rup­tion in­ci­dents—from use-af­ter-frees or any other cause—have we seen since rewrit­ing our com­piler from Rust to Zig?

Here’s a break­down of bug re­ports in Roc’s is­sue tracker, as clas­si­fied by Claude Opus 4.8:

You might be won­der­ing how the Rust-based com­piler had any mem­ory cor­rup­tion bugs at all, let alone more than dou­ble the to­tal count of the Zig-based one. Is it be­cause of that pesky Unsafe Rust again?

Actually, no. None of those 21 mem­ory cor­rup­tion bugs oc­curred in the com­pil­er’s logic it­self, which is a tes­ta­ment to Rust’s bor­row-checker work­ing as in­tended. The rea­son we had mem­ory cor­rup­tion bugs in our Rust-based com­piler is that it’s a com­piler.

Compilers emit ma­chine in­struc­tions. When a ma­chine ex­e­cutes those in­struc­tions, they can cause mem­ory cor­rup­tion, re­sult­ing in mem­ory cor­rup­tion bug re­ports from the peo­ple who ex­pe­ri­enced them. Regardless of which process had the bug—the com­piler or com­piled pro­gram—in both cases the proces­sor only did the bad thing be­cause the com­piler told it to. And in both cases the fix is the same: the com­pil­er’s code must change, since that code was what caused the mem­ory cor­rup­tion.

Just like every com­piler, Roc’s has had bugs, and some of those have been mis­com­pi­la­tions that led to mem­ory cor­rup­tion. That said, while 8 of the 10 mem­ory cor­rup­tion bugs in the Zig-based com­piler were also mis­com­pi­la­tions, the re­main­ing 2 were in the com­piler it­self. Both were use-af­ter-free bugs in er­ror re­port­ing, with the same symp­tom: file­names in er­ror mes­sages (one in roc check and the other in roc bun­dle) ren­dered as use­less ques­tion-mark-in-di­a­mond char­ac­ters. Rust’s bor­row checker would have caught both.

Now let’s sup­pose we had in­stead cho­sen Rust for our rewrite, or Zig with ReleaseSafe. What would have been the im­pact in prac­tice, hold­ing all else equal?

After 18 months of de­vel­op­ment, hun­dreds of to­tal bug re­ports, and hun­dreds of thou­sands of lines of code, my main take­away from ret­ro­spect­ing on this table is that pick­ing a dif­fer­ent row would have made no ap­pre­cia­ble dif­fer­ence to the pro­ject. So far our choice has got­ten us the out­come we’d hoped for.

As I noted ear­lier, every pro­ject has dif­fer­ent needs. When Bun rewrote in the op­po­site di­rec­tion—from Zig to Rust—their ac­com­pa­ny­ing post noted:

For Bun, cor­rectly han­dling the life­times of garbage-col­lected val­ues [from JavaScript] and man­u­ally-man­aged val­ues has been a ma­jor source of sta­bil­ity is­sues - most of­ten small mem­ory leaks and oc­ca­sion­ally, crashes. Every mem­ory al­lo­ca­tion has to be metic­u­lously re­viewed. Where do these bytes get freed? How do we en­sure it only gets freed once? Did we check for JavaScript ex­cep­tions prop­erly? Is this garbage-col­lected pointer vis­i­ble to the con­ser­v­a­tive stack scan­ner? Is this garbage col­lected mem­ory or man­u­ally man­aged mem­ory?

For Bun, cor­rectly han­dling the life­times of garbage-col­lected val­ues [from JavaScript] and man­u­ally-man­aged val­ues has been a ma­jor source of sta­bil­ity is­sues - most of­ten small mem­ory leaks and oc­ca­sion­ally, crashes. Every mem­ory al­lo­ca­tion has to be metic­u­lously re­viewed. Where do these bytes get freed? How do we en­sure it only gets freed once? Did we check for JavaScript ex­cep­tions prop­erly? Is this garbage-col­lected pointer vis­i­ble to the con­ser­v­a­tive stack scan­ner? Is this garbage col­lected mem­ory or man­u­ally man­aged mem­ory?

Roc’s com­piler does­n’t have these par­tic­u­lar chal­lenges be­cause it does­n’t in­ter­face with JavaScript or any other trac­ing garbage col­lec­tor. For Bun, “use-after-free, dou­ble-free, and ‘forgot to free’” er­rors have been “a large per­cent­age of bugs,” whereas er­rors like these have been a small per­cent­age of Roc’s bugs. And of course Roc’s com­piler faces other chal­lenges that Bun does­n’t. Different pro­jects have dif­fer­ent needs!

In our case, I’m not sure how I could look back at what’s ac­tu­ally hap­pened and con­clude that what we needed was a big­ger in­vest­ment in tool­ing to pre­vent mem­ory safety bugs in the com­piler it­self. There’s a much stronger case that we would ben­e­fit from bet­ter tool­ing to catch mem­ory safety bugs in our com­piled out­put, which has al­ways been out of scope for the bor­row checker.

Build Times

We wanted faster builds from Zig. Did we get them?

Well, the good news is that zig build –watch -fincremental can re­build a change to our cur­rent ~450K lines of Zig code in about 35 mil­lisec­onds. That’s even faster than what we were hop­ing for when we con­sid­ered Zig’s build speed a sell­ing point for the rewrite!

The bad news is that Zig’s cur­rent sta­ble 0.16.0 re­lease has a bug that breaks -fincremental on our code base. The fix al­ready landed, but to get it we’d have to build on a nightly 0.17.0 pre­re­lease build (which has break­ing lan­guage changes), along with ven­dor­ing and up­grad­ing our af­fected de­pen­den­cies to 0.17.0. We de­cided to wait for the next sta­ble re­lease in­stead.

As of the last com­mit that had Rust sources in our code base, here’s a tim­ing com­par­i­son on my Intel desk­top ma­chine run­ning Ubuntu 26 for build­ing cold (no cache, but pack­ages down­loaded lo­cally) com­pared to do­ing an in­cre­men­tal re­build af­ter mak­ing a triv­ial edit to our parser:

Note that our Zig build con­fig­u­ra­tion as of the fea­ture-par­ity com­mit was re­build­ing rarely-chang­ing ar­ti­facts on every build that we later de­cided to re­build only on de­mand. That’s why to­day’s cold builds are faster than they were back at 300K LoC, even though our lines of code have in­creased by ~50% since then.

Note that our Zig build con­fig­u­ra­tion as of the fea­ture-par­ity com­mit was re­build­ing rarely-chang­ing ar­ti­facts on every build that we later de­cided to re­build only on de­mand. That’s why to­day’s cold builds are faster than they were back at 300K LoC, even though our lines of code have in­creased by ~50% since then.

Rust 1.97 is the cur­rent sta­ble re­lease to­day, and 1.85 was the cur­rent sta­ble re­lease 487 days ago (the time our rewrite took to reach to fea­ture par­ity). So if we’d stayed on Rust for the same du­ra­tion, we could have seen our in­cre­men­tal build times de­crease from 10 sec­onds to 3.4. That’s a big jump! I re­ally ap­pre­ci­ate all the hard work that Rust con­trib­u­tors have done to im­prove build times. Eliminating 2/3 of our in­cre­men­tal build times over 18 months would have been a very wel­come change if we’d stayed on Rust, and it’s a big­ger im­prove­ment than I would have an­tic­i­pated in an 18-month pe­riod. Bravo!

As im­pres­sive as that im­prove­ment is, Zig’s 35ms is still way ahead. Not only is it 1/100th the build time of 3.4 sec­onds, it’s also in a dif­fer­ent per­for­mance cat­e­gory—and that 35ms is on a Zig code base with ~50% more lines of code than the Rust one that got 3.4s. I ex­pect Roc’s code base to keep grow­ing, and for this gap to keep grow­ing with it; I’ve never heard of any ini­tia­tive on Rust’s roadmap com­pa­ra­ble to -fincremental.

So while our de­ci­sion to re­main on sta­ble 0.16.0 (plus how many of our con­trib­u­tors run Mac lap­tops with ARM proces­sors; -fincremental only works on x86 – 64 CPUs right now) means we haven’t yet reaped the an­tic­i­pated build-time re­wards of choos­ing Zig for the rewrite, we cer­tainly have some­thing to look for­ward to in the next sta­ble Zig re­lease!

Memory Control: Zero-Parse Deserialization

Roc’s new on-disk caching sys­tem uses a tech­nique I first learned about from Zig’s com­piler, and which Casey Muratori told me is com­mon prac­tice in game pro­gram­ming. It re­lies on the happy co­in­ci­dence that if you’re or­ga­niz­ing your mem­ory in the way that runs fastest on mod­ern hard­ware any­way, you can also load it from disk di­rectly into mem­ory and start us­ing it with­out pars­ing any­thing.

Here’s how it works:

All of our com­piler data struc­tures are rep­re­sented as ar­rays with 32-bit in­dices over point­ers (and of­ten in struc­ture-of-ar­rays form).

This not only saves mem­ory and runs faster, it also means our data struc­tures can be writ­ten di­rectly to disk with­out need­ing to be se­ri­al­ized into a dif­fer­ent for­mat first.

The big­ger ben­e­fit is that this lets us de­se­ri­al­ize them back into mem­ory with­out pars­ing the on-disk bytes in any way. We load the bytes into mem­ory, do some re­lo­ca­tions to point our ex­ist­ing data struc­tures to the newly-loaded ar­rays, and we’re ready to go.

This means we de­se­ri­al­ize at the speed of load­ing the bytes from disk into mem­ory—so, ac­tu­ally I/O bound. If those bytes are al­ready in the op­er­at­ing sys­tem’s disk cache, it means we load cached work from pre­vi­ous builds at roughly the speed of mem­cpy.

When you run roc check twice in a row, the first time it caches all of its out­puts on disk us­ing this strat­egy. The sec­ond time, if the in­put source code files haven’t changed, all the parsed/​type-checked/​etc. data struc­tures jump straight from disk into mem­ory. It’s ex­tremely fast. roc test sim­i­larly caches the out­comes for tests of pure func­tions (which are de­ter­min­is­tic), and all of this is done with file-level gran­u­lar­ity, so if you change one file you’ll only be pay­ing for re­do­ing work of that file and any oth­ers that de­pend on it.

This zero-parse de­se­ri­al­iza­tion strat­egy only works be­cause we’re fol­low­ing this pro­gram­ming with­out point­ers style for all of our com­piler data struc­tures. If we in­stead used point­ers every­where (like al­most all com­pil­ers do), de­se­ri­al­iza­tion could­n’t be zero-parse.

This ap­proach has safety risks, how­ever. Similarly to how a pointer in mem­ory can point to the wrong ad­dress (e.g. lead­ing to a use-af­ter-free), any in­dex can be used as a lookup into the wrong ar­ray at run­time, at which point you end up with what­ever ran­dom bytes hap­pened to be at that lo­ca­tion. Rust’s bor­row checker is de­signed to help with pointer life­times, but it does­n’t at­tempt to an­swer the ques­tion “which in­dex goes with which ar­ray?” be­cause that has never been in scope for its de­sign.

Date: 2026 – 07-15 Git: https://​git­lab.com/​mort96/​blog/​blob/​pub­lished/​con­tent/​00000-home/​00017-sqlite-edi­tions.md

SQLite is an amaz­ing data­base en­gine. I use it as a data­base for plenty of em­bed­ded pro­jects, and I don’t think it’s an ex­ag­ger­a­tion to call it the in­dus­try stan­dard for lo­cal data stor­age. Some server soft­ware even uses it; for ex­am­ple, lob­ste.rs is now run­ning on SQLite.

Unlike tra­di­tional RDBMSes (Relational DataBase Management Systems), SQLite is not a sep­a­rate process; it’s an RDBMS as a li­brary, mean­ing your soft­ware re­mains self con­tained. Unlike tra­di­tional file for­mats, you don’t need to write cus­tom se­ri­al­iz­ers and parsers. In some ways, it’s the best of both worlds.

There’s just one huge prob­lem though. Its de­faults are all wrong.

Bad de­fault #1: Foreign key con­straints are ig­nored by de­fault

You read that right. Foreign key con­straints are ar­guably the pri­mary tool we have to en­sure that a data­base re­mains con­sis­tent and don’t have dan­gling ref­er­ences.

As a quick primer, this is how an SQL for­eign key con­straint looks:

CREATE TABLE users ( id INTEGER PRIMARY KEY, dis­play_­name TEXT );

CREATE TABLE posts ( id INTEGER PRIMARY KEY, user_id INTEGER NOT NULL, con­tent TEXT NOT NULL, FOREIGN KEY(user_id) REFERENCES users(id) );

The typ­i­cal be­hav­ior for all other RDBMSes would be that the user_id col­umn of a post must al­ways ref­er­ence the ID of a valid user. You can’t cre­ate a new post with­out pro­vid­ing a valid user ID, you can’t delete a user with­out also delet­ing its posts, lest you get a for­eign key con­straint vi­o­la­tion er­ror.

The only RDBMS I’m aware of which does­n’t en­force this by de­fault is SQLite.

This is made even worse by SQLite’s ten­dency to re-use ROWID. You see, in this ex­am­ple, those INTEGER PRIMARY KEY rows be­come aliases for the table’s ROWID, which is a unique in­te­ger ID as­signed to every row of a table in SQLite. The al­go­rithm for as­sign­ing ROWID is a bit com­pli­cated (more de­tails in the SQLite doc­u­men­ta­tion), but it re­sults in ID re-use in some cases. This means that a dan­gling ref­er­ence eas­ily re­sults in a ref­er­ence to the wrong row, which is even worse than a dan­gling ref­er­ence be­cause every­thing will seem fine. You don’t even get an er­ror dur­ing lookup.

Just look at this hy­po­thet­i­cal se­quence of op­er­a­tions in our toy data­base schema:

– Bob cre­ates a user ac­count INSERT INTO users (display_name) VALUES (‘Bob’); SELECT * FROM users; — id | dis­play_­name — 1 | Bob

– Bob posts an in­tro­duc­tion post INSERT INTO posts (user_id, con­tent) VALUES (1, ‘Hello, I am Bob’); SELECT u.dis­play_­name, p.con­tent FROM users as u, posts as p WHERE u.id = p.user_id; — dis­play_­name | con­tent — Bob | Hello, I am Bob

– Bob deletes his ac­count, — but we for­got to delete the posts. — SQLite does­n’t pro­duce an er­ror be­cause it ig­nores our for­eign key. DELETE FROM users WHERE id = 1;

– Alice cre­ates an ac­count. — Alice gets the same ID that Bob had due to the ROWID al­go­rithm. INSERT INTO users (display_name) VALUES (‘Alice’); SELECT * FROM users; — id | dis­play_­name — 1 | Alice

– Alice has now in­her­ited Bob’s old post! SELECT u.dis­play_­name, p.con­tent FROM users as u, posts as p WHERE u.id = p.user_id; — dis­play_­name | con­tent — Alice | Hello, I am Bob

The fix is to en­able for­eign_keys with a pragma:

PRAGMA for­eign_keys = ON;

If we had done this in the be­gin­ning, the buggy DELETE would have pro­duced an er­ror:

DELETE FROM users WHERE id = 1; — Runtime er­ror: FOREIGN KEY con­straint failed (19)

Bad de­fault #2: Columns can store the wrong data type

SQLite has a sim­ple type sys­tem: a value can be NULL, an INTEGER, a REAL (aka a dou­ble pre­ci­sion float), TEXT, or a BLOB (aka bi­nary data). Consequently, a col­umn can be de­fined to hold val­ues of any of those types.

However, a col­umn de­fined as an INTEGER col­umn is­n’t re­stricted to only in­te­gers; in­stead, SQLite con­sid­ers it to “use INTEGER affin­ity”. What this means is es­sen­tially:

If you try to in­sert a TEXT value, and it is a valid string rep­re­sen­ta­tion of an in­te­ger, it is con­verted to an in­te­ger and stored as such.

If you try to in­sert a TEXT value, and it is a valid string rep­re­sen­ta­tion of a real num­ber, it is con­verted to a real (aka dou­ble pre­ci­sion float) and stored as such.

Otherwise, the value is stored as-is.

Other affini­ties have dif­fer­ent but sim­pler rules:

Columns with BLOB affin­ity store val­ues as-is.

Columns with TEXT affin­ity store BLOB, TEXT and NULL val­ues as-is, but con­vert nu­meric val­ues to TEXT.

Columns with REAL affin­ity work like columns with INTEGER affin­ity ex­cept that in­te­ger val­ues are con­verted to REAL.

Here’s how this looks in prac­tice:

CREATE TABLE mu­sic ( id INTEGER PRIMARY KEY, name TEXT, du­ra­tion_sec INTEGER );

INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘Lost In Hollywood’, 321); INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘Comfortably Numb’, 382); INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘The Way of All Flesh’, ‘Way too long, I mean come on’); SELECT * FROM mu­sic; — id | name | du­ra­tion_sec — 1 | Lost In Hollywood | 321 — 2 | Comfortably Numb | 382 — 3 | The Way of All Flesh | Way too long, I mean come on

I don’t think I need to ex­plain why it’s a bad idea for a data­base to be so care­less about data val­i­da­tion. It would be one thing if SQLite was an ex­plic­itly dy­nam­i­cally typed doc­u­ment data­base, but it’s not. SQLite asks me through its syn­tax rules, “What type do you want to go into this col­umn”.

I once had to clean up a pro­ject where some code had ac­ci­den­tally been writ­ing the strings ‘1’ and ‘0’ to a col­umn which was in­tended to store booleans (1 and 0). That was not a fun de­bug­ging story.

Luckily, SQLite has the con­cept of strict ta­bles, which makes SQLite pro­duce a type er­ror when the wrong type is in­serted into a col­umn:

CREATE TABLE mu­sic ( id INTEGER PRIMARY KEY, name TEXT, du­ra­tion_sec INTEGER ) strict;

INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘The Way of All Flesh’, ‘Way too long, I mean come on’); — Runtime er­ror: can­not store TEXT value in INTEGER col­umn mu­sic.du­ra­tion_sec (19)

Unfortunately, there is no pragma to glob­ally make all ta­bles strict. So you have to re­mem­ber to add the strict tag to every table man­u­ally.

There’s a cou­ple of ar­gu­ments against strict ta­bles which I want to cover here.

The au­thors of SQLite have writ­ten about their pref­er­ence for “flexible typ­ing”. Personally, I find this a re­ally strange piece of writ­ing. It does­n’t pro­vide any ex­am­ples for why it could ever be use­ful to in­sert a BLOB into an INTEGER col­umn. All it does is il­lus­trate why it’s some­times use­ful to have a col­umn which can store val­ues of any type. Strict ta­bles have a so­lu­tion for that; it’s called the ANY data type. You can still cre­ate columns which ac­cept any value, you just have to be ex­plicit about it.

A much bet­ter ar­gu­ment is pro­vided by user ‘zie’ on lob­ste.rs. You see, strict ta­bles in SQLite don’t just en­force types. They also change the rules for how type spec­i­fiers are parsed.

Non-strict SQLite ta­bles use the fol­low­ing rules to de­ter­mine the type of a col­umn (from SQLite’s doc­u­men­ta­tion):

If the de­clared type con­tains the string “INT” then it is as­signed INTEGER affin­ity. If the de­clared type of the col­umn con­tains any of the strings “CHAR”, “CLOB”, or “TEXT” then that col­umn has TEXT affin­ity. Notice that the type VARCHAR con­tains the string “CHAR” and is thus as­signed TEXT affin­ity. If the de­clared type for a col­umn con­tains the string “BLOB” or if no type is spec­i­fied then the col­umn has affin­ity BLOB. If the de­clared type for a col­umn con­tains any of the strings “REAL”, “FLOA”, or “DOUB” then the col­umn has REAL affin­ity. Otherwise, the affin­ity is NUMERIC.

If the de­clared type con­tains the string “INT” then it is as­signed INTEGER affin­ity.

If the de­clared type of the col­umn con­tains any of the strings “CHAR”, “CLOB”, or “TEXT” then that col­umn has TEXT affin­ity. Notice that the type VARCHAR con­tains the string “CHAR” and is thus as­signed TEXT affin­ity.

If the de­clared type for a col­umn con­tains the string “BLOB” or if no type is spec­i­fied then the col­umn has affin­ity BLOB.

If the de­clared type for a col­umn con­tains any of the strings “REAL”, “FLOA”, or “DOUB” then the col­umn has REAL affin­ity.

Otherwise, the affin­ity is NUMERIC.

A con­se­quence of this rule, com­bined with SQLite’s loose typ­ing, is that you can give your columns type names such as DATETIME or KEY_VALUE_SET or COLOR, and have a data­base con­nec­tor/​wrap­per which au­to­mat­i­cally knows to se­ri­al­ize and de­se­ri­al­ize columns with cus­tom types. And if noth­ing else, those cus­tom type names serve as use­ful doc­u­men­ta­tion.

I have to ac­knowl­edge that just chang­ing the de­fault from non-strict ta­bles to strict ta­bles, with no fur­ther changes, would give up on this some­what nifty quirk. However, I think we would be much bet­ter served by cus­tom type aliases.

If we could write some­thing like:

CREATE TYPE KEY_VALUE_SET = TEXT;

and then use KEY_VALUE_SET as a type name in a strict table, I think every­one would be happy. I would prob­a­bly start us­ing such a fea­ture lib­er­ally to doc­u­ment the ex­pected pat­tern of data in my columns. In a real world schema, you in­evitably end up with TEXT columns which have to be parsed by ap­pli­ca­tion code.

As an aside to this aside, it would be neat if we could as­so­ci­ate CHECK con­straints with a cus­tom type.

Update: ‘masklinn’ on lob­ste.rs points out that the SQL 99 stan­dard al­ready spec­i­fies type aliases, called CREATE DOMAIN. This al­ready sup­ports con­straints as well. So re­ally, SQLite just needs to add sup­port for the stan­dard CREATE DOMAIN state­ment.

Update: ‘masklinn’ on lob­ste.rs points out that the SQL 99 stan­dard al­ready spec­i­fies type aliases, called CREATE DOMAIN. This al­ready sup­ports con­straints as well. So re­ally, SQLite just needs to add sup­port for the stan­dard CREATE DOMAIN state­ment.

Bad de­fault #3: SQLITE_BUSY er­rors with con­cur­rent writ­ers

SQLite al­lows mul­ti­ple con­cur­rent read­ers, but only one writer at a time. By de­fault, if you have two processes try­ing to ac­quire a write lock at the same time, one of them will im­me­di­ately re­ceive an SQLITE_BUSY er­ror.

This is not the be­hav­ior I ex­pect. I ex­pect SQLite to wait for the lock to get un­locked, up to some time­out. It’s do­ing disk IO af­ter all, so I al­ready struc­ture my code with the as­sump­tion that a write could po­ten­tially be slow.

The de­fault be­hav­ior has lead me to writ­ing real-world bugs, where sys­tems would some­times just crash. I’ve man­u­ally writ­ten retry loops to fix it.

The fix is to set busy_­time­out with a pragma:

PRAGMA busy_­time­out = 5000;

This makes SQLite try to ac­quire the lock for up to 5 sec­onds be­fore er­ror­ing with a SQLITE_BUSY er­ror.

I did­n’t learn about this set­ting un­til re­cently. It seems like such an ob­vi­ous de­fault that I’m as­ton­ished that it’s not.

Update: I should add a note here about why sup­port for con­cur­rent writ­ers is de­sir­able. During nor­mal op­er­a­tion, you’re usu­ally best served by struc­tur­ing your soft­ware such that all writes are done by a sin­gle process, ide­ally a sin­gle thread. Concurrent writ­ers will never be fast. But there are non-typ­i­cal sit­u­a­tions. Maybe you need to man­u­ally clean up a data­base in­ter­ac­tively us­ing the sqlite3 tool in­ter­ac­tively on the com­mand line. Maybe you have scripts for un­com­mon ad­min­is­tra­tive tasks which you haven’t had the need to write a front-end for. These are per­fectly le­git­i­mate and, I be­lieve, fairly com­mon use cases. I think it’s bad that with SQLite’s de­faults, this kind use has a chance to just crash the soft­ware by mak­ing it throw an un­ex­pected SQLITE_BUSY er­ror.

Update: I should add a note here about why sup­port for con­cur­rent writ­ers is de­sir­able.

During nor­mal op­er­a­tion, you’re usu­ally best served by struc­tur­ing your soft­ware such that all writes are done by a sin­gle process, ide­ally a sin­gle thread. Concurrent writ­ers will never be fast. But there are non-typ­i­cal sit­u­a­tions. Maybe you need to man­u­ally clean up a data­base in­ter­ac­tively us­ing the sqlite3 tool in­ter­ac­tively on the com­mand line. Maybe you have scripts for un­com­mon ad­min­is­tra­tive tasks which you haven’t had the need to write a front-end for. These are per­fectly le­git­i­mate and, I be­lieve, fairly com­mon use cases. I think it’s bad that with SQLite’s de­faults, this kind use has a chance to just crash the soft­ware by mak­ing it throw an un­ex­pected SQLITE_BUSY er­ror.

Bad de­fault 4: Performance

There’s a lot to say about per­for­mance tun­ing in SQLite. When cor­rectly con­fig­ured, it can be a truly fast RDBMS, with the abil­ity to fill roles we typ­i­cally re­serve for the big servers like PostgreSQL or MySQL.

But by de­fault, its per­for­mance is­n’t great. Smarter peo­ple than me have writ­ten much more on this, and I rec­om­mend Sylvain Kerkour’s Optimizing SQLite for servers if you’re in­ter­ested in this topic.

But the most sig­nif­i­cant bad de­fault is that SQLite’s Write-Ahead Log (WAL) is dis­abled by de­fault. It can be en­abled with:

PRAGMA jour­nal_­mode = WAL;

The WAL pro­vides a dra­matic write speed-up in most cir­cum­stances. Additionally, it lets us dras­ti­cally re­duce the amount of disk syncs with­out risk­ing data cor­rup­tion by chang­ing an­other set­ting:

PRAGMA syn­chro­nous = NORMAL;

See the SQLite doc­u­men­ta­tion on what ex­actly syn­chro­nous does.

The so­lu­tion: edi­tions?

The oft-cited rea­son for why these de­faults re­main, well, de­fault, is of course back­wards com­pat­i­bil­ity. Changing de­faults now would likely break lots of old soft­ware and make peo­ple afraid to up­grade SQLite in the fu­ture in case it breaks every­thing again, just like how I’m afraid to up­grade Python be­cause every “upgrade” breaks a bunch of soft­ware I use. It’s a laudi­ble and rare goal to try to not break your de­pen­dents.

However, I think the so­lu­tion is sim­ple: add one “super pragma” which changes all the bad de­faults. I pro­pose that the fol­low­ing:

PRAGMA edi­tion = 2026;

should be an alias for at least the fol­low­ing set of prag­mas:

PRAGMA for­eign_keys = ON; PRAGMA busy_­time­out = 5000; PRAGMA jour­nal_­mode = WAL; PRAGMA syn­chro­nous = NORMAL;

And also make strict mode the de­fault for ta­bles.

This should be a nice mid­dle ground which avoids break­ing back­wards com­pat­i­bil­ity, but lets the data­base en­gine move for­wards and not be bogged down by its own his­tory.

The edi­tion idea is stolen straight from Rust edi­tions. The ad­van­tage of a year-based edi­tion rather than some­thing like JavaScript’s “use strict”; is that as the years go by, the sen­si­ble de­faults may change. Maybe some­thing like Hctree’s WAL2 makes its way into the main branch, say, in the year 2034, so maybe PRAGMA edi­tion = 2034 will some day set PRAGMA jour­nal_­mode = WAL2.

Anyway, that’s all. I think SQLite should have an edi­tion sys­tem with up­dated sets of de­faults. Are there any things I’ve missed which makes this a bad idea? Or more prag­mas which should be added to my imag­i­nary “2026 edi­tion”?

I like my new-to-me MacBook be­cause it en­ables me to cre­ate more stuff com­pared to my now al­most 8 year old Thinkpad I had be­fore, mainly due to bat­tery and screen rea­sons. There is one thing about it that I can­not wrap my head around from a de­sign per­spec­tive and it’s the sharp edges es­pe­cially around the wrist area. On a flat sur­face it’s no prob­lem at all, but low and be­hold a lap­top will be fre­quently used on a lap, mean­ing the wrists will touch the sharp edge at an an­gle which is very un­com­fort­able.

I’ve found some posts on­line about peo­ple that have filed the edges off of their Macs be­fore, like this nice one by Kent Walters. In my case how­ever, al­most no one freaks out about this be­cause no one ac­tu­ally no­tices it. It went well, so I thought why not doc­u­ment the process.

Note I am ab­solutely not a good crafts­man, so do not use this as a guide!

I thought a long time about how to do it so as not to have it re­sult in a wavy bezel. Although I re­ally wanted to use this pro­ject as an ex­cuse to buy a ran­dom or­bital sander, I very quickly piv­oted from this idea be­cause I would prob­a­bly cause more dam­age with it than I would with a hand file. Another idea was to 3D print some sort of cham­fer to use as a guide, just to have the ini­tial part cleanly taken off at an an­gle. I played around with this idea for a while but con­cluded that I could prob­a­bly not get to the re­quired level of pre­ci­sion I wanted, es­pe­cially since it would have to hold the file/​sand­pa­per. I ended up just us­ing gen­eral pur­pose metal file I had at hand and pro­gres­sive sand pa­per (in block form, strongly rec­om­mended).

Some tape helped me to mark the ar­eas from be­low and above the bezel that I wanted to have file off. This worked re­ally well. Of course I taped off the track­pad, the key­board and head­phone plug etc. to avoid any residue from get­ting in­side. Additionally, I used very lit­tle soapy wa­ter that I ap­plied to the sand­pa­per through­out to keep the dust con­tained some­what. The ini­tial fil­ing with the gen­eral metal file was a bit scary, but it did not take off too much and the tape helped me to achieve an even level. The sand­pa­per blocks also helped with that, and I went up to 1200 coarse­ness.

The tricky part for me were these pointy dots on the lit­tle gap in the mid­dle. I did not want to go too heavy here with the fil­ing so I used some model mak­ing files very, very lightly and then just used the 1200 sand­pa­per un­til I was sat­is­fied with the re­sult. In the end, I was care­ful to re­move any dust that had ac­cu­mu­late, for which this lit­tle air blower sque­egy thing came in handy.

This is a “blue” M4 MacBook Air, so I’m cu­ri­ous how it will look over time due to the an­odized alu­minum. Another note would be to test it out a bit and see how it feels. I have quite large arms so when typ­ing I also touch the cor­ners, so I had to go back and file them too, but that might not be nec­es­sary de­pend­ing on how you do it.

What I liked about the Thinkpad was that I could just chuck it any­where with­out wor­ry­ing about it. I made it a point when I bought this Mac to treat it just the same way. Ultimately, it is a tool and should be used ap­pro­pri­ately. Even if it is nice and shiny, if mod­i­fy­ing it would make it serve it’s pri­mary pur­pose as a tool bet­ter, it’s worth a con­sid­er­a­tion. So I like this re­cent wave of en­cour­age­ment around sim­i­lar mod­i­fi­ca­tions, and it is also far eas­ier and ap­proach­able than it seems ini­tially.

Counting each YC chap­ter by batch year shows when these founders came through. A founder with two YC star­tups ap­pears in both years.