10 interesting stories served every morning and every evening.

Kimi AI with K3 | Built for Agentic Coding & Knowledge Work

www.kimi.com

Ask any­thing, or task an agent…

Explore in­spi­ra­tion

Kimi K3 Tech Blog: Open Frontier Intelligence

www.kimi.com

Today, we are in­tro­duc­ing Kimi K3 — our most ca­pa­ble model. Kimi K3 is a 2.8T-parameter model built on our Kimi Delta Attention and Attention Residuals, with na­tive vi­sion ca­pa­bil­i­ties and a 1-million-token con­text win­dow. It is the world’s first open 3T-class model, de­signed for fron­tier in­tel­li­gence across long-hori­zon cod­ing, knowl­edge work, and rea­son­ing.

While its over­all per­for­mance still trails the most pow­er­ful pro­pri­etary mod­els, Claude Fable 5 and GPT 5.6 Sol, Kimi K3 demon­strated fron­tier-level per­for­mance across our eval­u­a­tion suite, con­sis­tently out­per­form­ing other tested mod­els.

Kimi K3 is avail­able to­day on Kimi.com, Kimi Work, Kimi Code, and the Kimi API. At launch, Kimi K3 will use max think­ing ef­fort by de­fault, with low- and high-ef­fort modes to be in­tro­duced in sub­se­quent up­dates. We are cur­rently work­ing closely with in­fer­ence part­ners and open-source main­tain­ers to align tech­ni­cal de­tails and en­sure a re­li­able roll­out across the ecosys­tem. The full model weights will be re­leased by July 27, 2026. Further de­tails on the ar­chi­tec­ture, train­ing, and eval­u­a­tions will be re­leased along­side the Kimi K3 tech­ni­cal re­port.

An Open 3T-Class Model

Kimi K3 is the first open model to reach 2.8 tril­lion pa­ra­me­ters. It marks the lat­est step in Kimi’s sus­tained push at the scal­ing fron­tier: for nine of the past twelve months, Kimi mod­els have set the up­per bound of open-model sizes.

Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), two ar­chi­tec­tural up­dates de­signed to im­prove how in­for­ma­tion flows across se­quence length and model depth. We have also scaled up Mixture of Experts (MoE) spar­sity, ef­fec­tively ac­ti­vat­ing 16 out of 896 ex­perts when paired with a Stable LatentMoE frame­work. Together with re­fined train­ing and data recipes, these struc­tural changes yield an ap­prox­i­mate 2.5× im­prove­ment in over­all scal­ing ef­fi­ciency com­pared to Kimi K2, al­low­ing the model to con­vert com­pute into in­tel­li­gence more ef­fec­tively.

Coding

Kimi K3 has strong long-hori­zon cod­ing per­for­mance. Operating with min­i­mal hu­man over­sight, it can sus­tain long en­gi­neer­ing ses­sions, nav­i­gate mas­sive repos­i­to­ries, and or­ches­trate ter­mi­nal tools.

Kimi K3 also ex­cels in tasks blend­ing soft­ware en­gi­neer­ing with vi­sual rea­son­ing — it lever­ages screen­shots and vi­su­als to op­ti­mize game dev, fron­tend, and CAD.

The case stud­ies be­low show how Kimi K3′s cod­ing ca­pa­bil­ity trans­lates into open-ended soft­ware cre­ation and sci­en­tific re­search.

Kernel Optimization

We tested the mod­els’ ca­pa­bil­ity to op­ti­mize GPU ker­nels. Each model works in­de­pen­dently in an iden­ti­cal sand­box, with up to 24 hours to pro­file, rewrite, and bench­mark four tasks span­ning AttnRes, KDA, and a 512-head-dimension MLA ker­nel across NVIDIA H200 and GPGPU from an al­ter­na­tive ven­dor. Kimi K3 per­formed com­pet­i­tively with Fable 5 (with fall­back) and sub­stan­tially out­per­formed Opus 4.8, GPT 5.6 Sol, and GPT 5.5.

Claude Fable 5 was eval­u­ated by a third party, and its re­sults may in­clude fall­back be­hav­ior. Across most mod­els, some tra­jec­to­ries in­clude small, ac­cept­able pre­ci­sion short­cuts that re­main within our nu­mer­i­cal tol­er­ance. GPGPU de­notes gen­eral-pur­pose GPUs used for com­pu­ta­tion be­yond graph­ics ren­der­ing.

In the late stages of Kimi K3 de­vel­op­ment, an early ver­sion of Kimi K3 han­dled the ma­jor­ity of the team’s ker­nel op­ti­miza­tion works.

GPU Compiler Development

We fur­ther tested whether Kimi K3 could build a GPU pro­gram­ming sys­tem from scratch. Kimi K3 de­vel­oped MiniTriton, a com­pact Triton-like com­piler with its own tile-level IR layer over MLIR, op­ti­miza­tion passes, and a PTX code-gen­er­a­tion pipeline. Across sup­ported roofline bench­marks, MiniTriton de­liv­ers per­for­mance on par with or bet­ter than Triton and torch.com­pile — beat­ing Triton on cer­tain work­loads. Beyond mi­crobench­marks, MiniTriton sus­tains end-to-end nanoGPT train­ing with sta­ble con­ver­gence, the loss curve closely track­ing the ref­er­ence with only mi­nor di­ver­gence — val­i­dat­ing the full pipeline on a re­al­is­tic work­load. These re­sults demon­strate that Kimi K3 can build a co­her­ent end-to-end com­piler — from DSL fron­tend and IR passes to PTX code­gen and run­time — rather than iso­lated ker­nels; its from-scratch Tensor Core path al­ready ri­vals Triton’s ex­ten­sively op­ti­mized stack.

Game Dev and Digital Creation

Kimi K3 com­bines strong 3D rea­son­ing, cod­ing, and vi­sion ca­pa­bil­i­ties to turn con­cepts, im­ages, and videos into fully playable in­ter­ac­tive ex­pe­ri­ences. Kimi K3 achieves true vision in the loop” by seam­lessly it­er­at­ing be­tween code and live screen­shots—in­stantly see­ing and re­fin­ing out­puts.

Case 1: 3D Open World

Kimi K3 built a fully pro­ce­dural browser-based 3D ex­plo­ration game us­ing Three.js WebGPU and GPU com­pute. It pro­ce­du­rally gen­er­ated the en­vi­ron­ment, while us­ing a 3D as­set gen­er­a­tion tool to cre­ate the rider and horse mod­els, pro­duc­ing an ex­pan­sive open world with forests, a log-cabin vil­lage, snowy moun­tains, and dy­namic weather. External as­sets used: an­i­mated cow­boy and horse mod­els and ter­rain data.

Chip Design

As an early proof of con­cept, Kimi K3 de­signed a chip to serve a nano model built on its own ar­chi­tec­ture. In a sin­gle 48-hour au­tonomous run, K3 built, op­ti­mized, and ver­i­fied the chip us­ing open-source EDA tools on the Nangate 45nm li­brary. Within 4 mm², the chip closes tim­ing at 100 MHz and sus­tains over 8,700 to­kens/​s de­code through­put in sim­u­la­tion, pack­ing 1.46M stan­dard cells, 0.277 MB of SRAM, and an INT4 MAC ar­ray with fused de­quan­ti­za­tion. A chip built by a model, for a model, re­flects K3′s long-hori­zon agen­tic ca­pa­bil­i­ties.

Coding for Research

Kimi K3 bridges sci­en­tific lit­er­a­ture and ex­e­cutable code, au­tonomously im­ple­ment­ing, val­i­dat­ing, and an­a­lyz­ing com­plex com­pu­ta­tional re­search work­flows.

In one case, Kimi K3 com­pleted in about two hours what would typ­i­cally re­quire one to two weeks of work by an ex­pe­ri­enced re­searcher. To re­pro­duce the I–Love–Q uni­ver­sal re­la­tions in com­pu­ta­tional as­tro­physics, it re­viewed and cross-val­i­dated 20+ pa­pers, im­ple­mented the full nu­mer­i­cal pipeline, eval­u­ated 300+ equa­tions of state, iden­ti­fied in­con­sis­ten­cies in pub­lished for­mu­las, gen­er­ated 3,000+ lines of Python code, and pro­duced an in­ter­ac­tive HTML dash­board for ex­plor­ing the re­sults.

Knowledge Work

Kimi K3 ad­vances end-to-end knowl­edge work. Beyond pub­lic bench­marks, Kimi K3 (max) demon­strates con­sis­tent gains across our in­ter­nal eval­u­a­tions, which are de­rived from re­cur­ring pat­terns and chal­lenges ob­served in real-world user-agent work­flows. These con­sis­tent ad­van­tages across dis­tinct pro­duc­tion-ori­ented work­flows re­flect a broad im­prove­ment in Kimi K3′s agen­tic knowl­edge work ca­pa­bil­i­ties.

Research with Interactive Visualization

Below are a few ex­am­ples of what Kimi K3 in Kimi Work can pro­duce across fi­nan­cial con­sult­ing and sci­en­tific re­search:

Case 1: Interactive 42 years of AI ASIC in­dus­try re­search web­site

An in­ter­ac­tive re­search re­port you can drill into: 42 years of the ASIC in­dus­try, cre­ated through 120+ rounds of re­cur­sive self-im­prove­ment. Kimi K3 trans­forms ev­i­dence into be­spoke charts, an­i­mated di­a­grams, and in­ter­ac­tive vi­sual nar­ra­tives. It pulled data via 2.8k+ web searches/​fetches and 1.1k+ ter­mi­nal data pulls, across 11k+ pages span­ning 87 quar­terly re­ports and 99 orig­i­nal PDFs.

Case 2: Fusion Industry Research

A con­sult­ing-style in­dus­try re­port with in­ter­ac­tive vi­su­al­iza­tions—in­clud­ing time­lines, Funnel Chart, Range Bar Chart, Gantt Charts, and pub­li­ca­tion-qual­ity slides.

Case 3: GWTC-5 Gravitational-wave Analysis

An analy­sis of 391 grav­i­ta­tional-wave events us­ing 20+ con­cur­rent sub­agents, pro­duc­ing 7 sci­en­tific vi­su­al­iza­tions, 2 ta­bles, and a lit­er­a­ture syn­the­sis from 10+ pa­pers.

Kimi K3 is also par­tic­u­larly ef­fec­tive at pro­duc­ing in­fo­graphic-style pre­sen­ta­tions, such as the fully ed­itable heatmap and an­nual re­port shown be­low:

Widgets and Dashboard

In Kimi Work, we in­tro­duce two new fea­tures - Widgets and Dashboard - which make in­ter­ac­tions with Kimi K3 more vi­sual and per­sis­tent. Widgets let you gen­er­ate in­ter­ac­tive com­po­nents di­rectly within a chat, with con­nec­tions to lo­cal data or ex­ter­nal plu­g­ins for con­tin­u­ous up­dates. Dashboard brings the wid­gets you care about most into one per­sis­tent, per­son­al­ized view or­ga­nized around a topic, pro­ject, or goal.

Video Editing

Kimi K3 ex­cels at mo­tion de­sign, an­i­ma­tion, and video edit­ing be­cause its na­tive mul­ti­modal ar­chi­tec­ture un­der­stands text, im­ages, and video within the same model.

In one ex­am­ple, K3 cre­ated a 3Blue1Brown-style mo­tion-graph­ics ex­plainer of its own ar­chi­tec­ture, trans­lat­ing tech­ni­cal ideas into an­i­mated di­a­grams and tran­si­tions.

In an­other, Kimi K3 edited its own teaser video from 56 source clips, han­dling clip se­lec­tion, mo­tion-matched cuts, frame-ac­cu­rate beat syn­chro­niza­tion, au­dio pro­cess­ing, and mul­ti­ple rounds of re­vi­sion. A high-den­sity short video like this would typ­i­cally take an ex­pe­ri­enced ed­i­tor one to two work­ing days, or a be­gin­ner three to five.

Architecture and Infrastructure

Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes). KDA pro­vides an ef­fi­cient foun­da­tion for scal­ing at­ten­tion, while AttnRes se­lec­tively re­trieves rep­re­sen­ta­tions across depth rather than ac­cu­mu­lat­ing them uni­formly. Together, they form the ar­chi­tec­tural back­bone of a model de­signed to scale well be­yond the tril­lion-pa­ra­me­ter regime.

Kimi K3 uses Stable LatentMoE, ef­fec­tively ac­ti­vat­ing 16 of 896 ex­perts. At this level of spar­sity, rout­ing and op­ti­miza­tion be­come first-or­der chal­lenges. Quantile Balancing de­rives ex­pert al­lo­ca­tion di­rectly from router-score quan­tiles, elim­i­nat­ing heuris­tic up­dates and a sen­si­tive bal­anc­ing hy­per­pa­ra­me­ter, while Per-Head Muon ex­tends Muon by op­ti­miz­ing at­ten­tion heads in­de­pen­dently for more adap­tive learn­ing at scale. Sigmoid Tanh Unit (SiTU) and Gated MLA im­prove ac­ti­va­tion con­trol and at­ten­tion se­lec­tiv­ity re­spec­tively. Together, these ad­vances en­able sta­ble and ef­fi­cient train­ing at the 2.8-trillion-parameter scale.

Kimi K3 ap­plies quan­ti­za­tion-aware train­ing from the SFT stage on­ward, us­ing MXFP4 weights with MXFP8 ac­ti­va­tions for broad hard­ware com­pat­i­bil­ity. To pre­vent ex­pert im­bal­ance from de­grad­ing through­put at large ex­pert-par­al­lel scales, we in­tro­duce a fully bal­anced ex­pert-par­al­lel train­ing method with sta­tic shapes and no host syn­chro­niza­tion on the crit­i­cal path. Since in­fer­ence ef­fi­ciency like­wise ben­e­fits from larger high-band­width com­mu­ni­ca­tion do­mains, we rec­om­mend de­ploy­ing Kimi K3 on su­pern­ode con­fig­u­ra­tions with 64 or more ac­cel­er­a­tors. Finally, as KDA poses new chal­lenges for con­ven­tional pre­fix caching, we have con­tributed a cor­re­spond­ing im­ple­men­ta­tion to the vLLM com­mu­nity, to be re­leased along­side the model. KDA with pre­fill cache al­lows us to serve Kimi K3 at a highly com­pet­i­tive to­ken price de­spite its scale and long con­text.

More tech­ni­cal de­tails will be avail­able in our com­ing re­port.

Availability

Kimi K3 Agents: Download or up­date to the lat­est Kimi app from your mo­bile app store, avail­able on iOS, Android, and HarmonyOS, or visit kimi.com.

Work with Kimi K3: Download the lat­est Kimi Work desk­top app, ver­sion 3.1.0 or later, avail­able for Windows and Apple sil­i­con Macs.

Code with Kimi K3: Run Kimi Code in your ter­mi­nal and se­lect Kimi K3 us­ing the /model com­mand.

Build with the Kimi API: Visit the Kimi API Platform and se­lect kimi-k3. Pricing is $0.30/MTok for cache-hit in­put, $3.00/MTok for cache-miss in­put, and $15.00/MTok for out­put. Powered by Mooncake’s dis­ag­gre­gated in­fer­ence ar­chi­tec­ture, the of­fi­cial Kimi API achieves a cache hit rate above 90% in cod­ing work­loads.

Bring Kimi to your or­ga­ni­za­tion: Kimi Enterprise pro­vides en­ter­prise-grade data pri­vacy and mem­ber man­age­ment, with com­plete sep­a­ra­tion be­tween per­sonal and or­ga­ni­za­tion ac­counts. Visit the pric­ing page and se­lect Get Kimi Enterprise” to sub­scribe for your team.

Full Benchmark Table

Footnotes

All Kimi K3 re­sults re­ported be­low are ob­tained with the rea­son­ing ef­fort set to max’, set­ting tem­per­a­ture = 1.0 and top-p = 1.0. Depending on the bench­mark, each model is eval­u­ated un­der one of three agen­tic har­nesses — KimiCode, Claude Code, or Codex — as spec­i­fied in the notes be­low.

Coding bench­marks

DeepSWE. Kimi K3 is eval­u­ated with the KimiCode har­ness. The GLM-5.2 score is taken from the GLM-5.2 re­lease blog (https://​z.ai/​blog/​glm-5.2); all re­main­ing scores are from the of­fi­cial DeepSWE leader­board (https://​deep­swe.dat­acurve.ai/), un­der which Kimi K3 at­tains 67.3 with the mini-SWE-agent har­ness.

Terminal-Bench 2.1. Kimi K3 is eval­u­ated with the KimiCode har­ness. For all other mod­els, we re­port the best score across har­nesses: GLM-5.2 with Claude Code (https://​z.ai/​blog/​glm-5.2); Claude Opus 4.8 and Claude Fable 5 with Terminus 2 (https://​ar­ti­fi­cial­analy­sis.ai/​eval­u­a­tions/​ter­mi­nal­bench-v2 – 1); GPT 5.5 and GPT 5.6 Sol with Codex (https://​ope­nai.com/​in­dex/​pre­view­ing-gpt-5 – 6-sol/).

Program Bench. Kimi K3 is eval­u­ated with the KimiCode har­ness. The GLM-5.2 score is from https://​z.ai/​blog/​glm-5.2; all other scores are from https://​www.vals.ai/​bench­marks/​pro­gram­bench.

SWE Marathon. Kimi K3, Claude Opus 4.8, and Claude Fable 5 are eval­u­ated with the Claude Code har­ness; GPT 5.6 Sol is eval­u­ated with the Codex har­ness. The GLM-5.2 score is from https://​z.ai/​blog/​glm-5.2.

FrontierSWE. Kimi K3 is eval­u­ated with the KimiCode har­ness and GPT 5.6 Sol with the Codex har­ness; all other re­sults are from https://​www.fron­tier­swe.com/. Dominance scores are re­com­puted from the raw scores us­ing the of­fi­cial eval­u­a­tion script and are cur­rent as of July 16, 2026.

PostTrain Bench. Scores for GLM-5.2, GPT 5.5, and Claude Opus 4.8 are adopted from the of­fi­cial PostTrainBench re­sults. Kimi K3, Claude Fable 5, and GPT 5.6 Sol are eval­u­ated with the of­fi­cial Harbor im­ple­men­ta­tion at max­i­mum rea­son­ing ef­fort, av­er­aged over three runs — Kimi K3 and Claude Fable 5 with the Claude Code har­ness, and GPT 5.6 Sol with the Codex har­ness. Under the Claude Code har­ness, re­quests re­fused by Claude Fable 5 due to its us­age pol­icy au­to­mat­i­cally fall back to Claude Opus 4.8.

MLS Bench Lite. Kimi K3 is eval­u­ated with the KimiCode har­ness; GLM-5.2 and the Claude mod­els with the Claude Code har­ness; GPT 5.5 and GPT 5.6 Sol with the Codex har­ness.

KCB 2.0. Kimi K3 is eval­u­ated with both the KimiCode and Claude Code har­nesses; GLM-5.2, Claude Opus 4.8, and Claude Fable 5 with the Claude Code har­ness; GPT 5.5 and GPT 5.6 Sol with the Codex har­ness. All mod­els are eval­u­ated at max­i­mum rea­son­ing ef­fort, ex­cept GPT 5.5, which uses the xhigh” set­ting.

Productivity and agen­tic bench­marks

OfficeQA Pro and SpreadsheetBench 2. Kimi K3, GLM-5.2, Claude Opus 4.8, and Claude Fable 5 are eval­u­ated with the Claude Code har­ness; GPT 5.5 and GPT 5.6 Sol are eval­u­ated with the Codex har­ness.

MCP Atlas. All mod­els are eval­u­ated on the 500-task pub­lic sub­set with a 100-turn limit, us­ing Gemini 3.1 Pro as the judge.

AutomationBench. All mod­els are eval­u­ated on the 600-task pub­lic sub­set, fol­low­ing the of­fi­cial GitHub setup in all other re­spects.

BrowseComp. We adopt the con­text-com­paction strat­egy used in the Claude model cards, trig­gered at 300K to­kens. When eval­u­ated with a 1M-token con­text win­dow and no con­text man­age­ment, Kimi K3 achieves a score of 90.4. The re­sults of Claude Fable 5, Claude Opus 4.8, GPT 5.6 Sol, and GPT 5.5 are cited from https://​www.an­thropic.com/​news/​claude-fa­ble-5-mythos-5 and https://​ope­nai.com/​in­dex/​gpt-5 – 6/.

GDPval-AA v2 and AA-Briefcase scores are cited from https://​ar­ti­fi­cial­analy­sis.ai/.

Multimodal bench­marks

Except for ZeroBench, which fol­lows the of­fi­cial set­ting and is run five times, all mul­ti­modal scores are av­er­aged over three runs. MMMU-Pro is eval­u­ated fol­low­ing the of­fi­cial pro­to­col, pre­serv­ing the orig­i­nal in­put or­der and prepend­ing im­ages to the text in­put.

PerceptionBench. PerceptionBench is an in-house bench­mark that fo­cuses on atomic vi­sual per­cep­tion ca­pa­bil­i­ties.

Limitations

Sensitivity to think­ing his­tory. K3 was trained in the pre­served think­ing his­tory mode. If the agent har­ness fails to pass back all the his­tor­i­cal think­ing con­tent as re­quired, or if an on­go­ing ses­sion with an­other model is switched over to K3, gen­er­a­tion qual­ity may be­come highly un­sta­ble. We rec­om­mend us­ing a har­ness with ver­i­fied com­pat­i­bil­ity, such as Kimi Code, and avoid­ing switch­ing to K3 in the mid­dle of a ses­sion.

Excessive proac­tive­ness. K3′s train­ing places par­tic­u­lar em­pha­sis on long-hori­zon, chal­leng­ing tasks. As a re­sult, when it en­coun­ters mi­nor is­sues or am­bigu­ous user in­tent dur­ing task ex­e­cu­tion, it may make un­ex­pected de­ci­sions on the user’s be­half. If your ap­pli­ca­tion re­quires the agent to op­er­ate within well-de­fined bound­aries and re­frain from ex­ces­sive im­pro­vi­sa­tion, please im­pose more ex­plicit be­hav­ioral con­straints on K3 in the sys­tem prompt or in AGENTS.md.

Despite be­ing a highly com­pet­i­tive model over­all, K3 nonethe­less ex­hibits a no­tice­able gap in user ex­pe­ri­ence com­pared with Claude Fable 5 and GPT 5.6 Sol.

The Lost Joy of Music Piracy

www.pigeonsandplanes.com

Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing. Pigeons & Planes is all about mu­sic dis­cov­ery, sup­port­ing new artists, and de­liv­er­ing the best mu­sic cu­ra­tion on­line and IRL. We’re al­ways lis­ten­ing.

Sony Deletes A Bunch More Movies From The Accounts Of People Who ‘Bought’ Them

www.techdirt.com

from the poof-it’s-gone dept

In all of our dis­cus­sions about how the dig­i­tal rev­o­lu­tion has cre­ated a sys­tem in which peo­ple don’t ac­tu­ally own the things they think they’re buy­ing, I get par­tic­u­larly frus­trated by the lack of change in it all. We’ve spilled much ink com­plain­ing that this clearly anti-con­sumer prac­tice needs to be done away with, where an un­sus­pect­ing pub­lic thinks they’re buy­ing a thing” only to learn months or years later that the thing” they bought was ac­tu­ally a li­cense to use/​view/​lis­ten to an­other thing”, and that li­cense ex­ists at the plea­sure of the com­pany that col­lected the money for it. And if you want to see the lack of change or ac­tion re­ally honed in upon, let’s take a look at Sony’s PlayStation Store.

In 2022, due to evolving li­cens­ing agree­ments” with dis­trib­u­tor StudioCanal, German and Austrian users had hun­dreds of movies dis­ap­pear from their PS ac­counts, long af­ter buy­ing them through Sony. Then in 2023, it hap­pened again in America, specif­i­cally when Sony ended its li­cens­ing agree­ment with Discovery af­ter the Warner Bros. merger, which, of course, has since been bought by Paramount Skydance. That re­sulted in cus­tomers hav­ing hun­dreds and hun­dreds of episodes of TV shows deleted from their ac­counts. Nowhere in any of this were there re­funds, of course. No rec­om­pense at all, ac­tu­ally. Just a thing you thought you’d bought taken away from you by the very peo­ple you thought you bought it from.

And now it’s hap­pen­ing again. Due to an­other li­cens­ing agree­ment fall­out with StudioCanal, hun­dreds of movies and TV shows are be­ing ripped from the ac­counts of PS Store cus­tomers, and there ap­pears to be fuck all that they can do about it.

This news was brought to peo­ple’s at­ten­tion by X user so­matyk, who posted the no­ti­fi­ca­tion they had re­ceived from PlayStation this week. Along with the un­apolo­getic news that the pur­chased movies would be deleted from their ac­count on September 1, the mes­sage con­cluded with, Click here for a full list of af­fected ti­tles that will no longer be sup­ported. Thank you.” The same warn­ing is now re­pro­duced in full on the PlayStation web­site, along with the list of 551 films and TV se­ries that are be­ing pulled from peo­ple’s li­braries.

This news was brought to peo­ple’s at­ten­tion by X user so­matyk, who posted the no­ti­fi­ca­tion they had re­ceived from PlayStation this week. Along with the un­apolo­getic news that the pur­chased movies would be deleted from their ac­count on September 1, the mes­sage con­cluded with, Click here for a full list of af­fected ti­tles that will no longer be sup­ported. Thank you.” The same warn­ing is now re­pro­duced in full on the PlayStation web­site, along with the list of 551 films and TV se­ries that are be­ing pulled from peo­ple’s li­braries.

As Kotaku notes later in their post, part of what is strik­ing in all of this is the sheer mun­dan­ity of the an­nounce­ment. Because there have been no con­se­quences, or any ac­tion at all from the pub­lic or gov­ern­ment, Sony treats this all as if it’s per­fectly nor­mal and no big deal. You can tell me all you want about how the Ts and Cs in these pur­chases do in fact note that the na­ture of the pur­chase is a tem­po­rary li­cens­ing of the con­tent for an un­de­ter­mined time pe­riod… but I can promise you that the pub­lic in gen­eral does­n’t un­der­stand that. They think they’re buy­ing a thing, not a li­cense.

And that’s be­cause of the pur­pose­ful ob­fus­ca­tion of that fact. Sony damned well knows that the vast ma­jor­ity of peo­ple don’t read those Ts and Cs. It knows that the pub­lic largely does­n’t un­der­stand how these back­end li­cens­ing agree­ments with dis­trib­u­tors work, or that they even ex­ist. And Sony is­n’t ex­actly putting out a big blink­ing sign on its store pages in­form­ing the pub­lic of all of this. Instead, the com­pany is only too happy to col­lect money from a pub­lic that is be­ing pur­pose­fully kept ig­no­rant of what they’re buy­ing.

Of course, when you scroll past the end­less EULAs when you first use your PlayStation, and click Agree” the first time you load the store, you’re un­wit­tingly agree­ing that noth­ing you buy is re­ally truly bought, and that it can be taken away from you at any point, and there’s noth­ing you can do. The same is true of your games.

Of course, when you scroll past the end­less EULAs when you first use your PlayStation, and click Agree” the first time you load the store, you’re un­wit­tingly agree­ing that noth­ing you buy is re­ally truly bought, and that it can be taken away from you at any point, and there’s noth­ing you can do. The same is true of your games.

This, too, will prob­a­bly pass with­out any real ac­tion. The gov­ern­ment has done its best to gut our con­sumer pro­tec­tion agen­cies, so they won’t be any help. Angry cus­tomers won’t co­a­lesce into ac­tivism or ac­tion, most likely. And I’ll prob­a­bly be writ­ing an­other one of these posts in a cou­ple of years when it all hap­pens again.

But it should­n’t be that way. There are com­mon sense things that can be done to bet­ter in­form the pub­lic. Rules for how the store should in­form peo­ple with each and every pur­chase. Someone just needs to de­mand it be done.

Filed Under: eula, own­er­ship, playsta­tion, playsta­tion store, video games

Companies: sony, stu­dio­canal

OnePlus Community

community.oneplus.com

Microsoft Comic Chat is now open source

opensource.microsoft.com

The chat client that brought Comic Sans to the world is now on GitHub

Today, we’re ex­cited to an­nounce the open-source re­lease of Microsoft Comic Chat, the chat client that au­to­mat­i­cally turned con­ver­sa­tions within Internet Relay Chat (IRC) into comic pan­els fea­tur­ing il­lus­trated char­ac­ters, speech bub­bles, and ex­pres­sions, and helped in­tro­duce the world to a lit­tle font called Comic Sans.

Yes, that Comic Sans. Originally de­signed by Microsoft ty­pog­ra­pher Vincent Connare in 1994, Comic Sans found its first real home in Comic Chat, where its in­for­mal, hand-let­tered feel matched the soft­ware’s speech-bub­ble con­ver­sa­tions per­fectly.

For many peo­ple, Comic Chat is a nos­tal­gic ar­ti­fact from the early days of the in­ter­net as we tran­si­tioned from tech­nolo­gies like tel­net, Usenet, and IRC to the largely vi­sual web that we en­joy to­day. For oth­ers, it’s a leg­endary piece of Microsoft his­tory they have only heard about in sto­ries, screen­shots, and de­bates about ty­pog­ra­phy. Now, de­vel­op­ers, his­to­ri­ans, retro com­put­ing en­thu­si­asts, and any­one who ap­pre­ci­ates a won­der­fully un­con­ven­tional idea can ex­plore the source code for them­selves.

A dif­fer­ent vi­sion for on­line com­mu­ni­ca­tion

Today we’re ac­cus­tomed to mes­sag­ing apps with re­ac­tions, stick­ers, GIFs, avatars, video, and AI-generated con­tent. But in the mid-1990s, in­ter­net chat was largely walls of scrolling text.

Rather than dis­play­ing mes­sages as plain text, Comic Chat pre­sented par­tic­i­pants as il­lus­trated char­ac­ters. Conversations un­folded in comic pan­els, with speech bub­bles, ex­pres­sions, and ges­tures gen­er­ated from what peo­ple typed. If some­one wrote I like that,” the char­ac­ter might point to it­self. If the text sug­gested anger, the char­ac­ter might frown or cross its arms. It was quirky, am­bi­tious, oc­ca­sion­ally chaotic, and sur­pris­ingly for­ward-look­ing.

Many ideas we now take for granted in on­line com­mu­ni­ca­tion can trace some of their spirit to ex­per­i­ments like Comic Chat.

The peo­ple who built it

David DJ Kurlander, work­ing in the Microsoft Research Virtual Worlds Group, con­ceived the idea of a new vi­sual rep­re­sen­ta­tion of con­ver­sa­tional his­to­ries, and started de­vel­op­ing Comic Chat in 1995. Built in Visual C++ 4.0 and MFC, Comic Chat was re­leased in 1996 with the Internet Explorer 3 web browser.

Under the hood, Comic Chat was more than a clever skin for IRC. It was able to in­ter­pret con­ver­sa­tional cues in the text and choose ap­pro­pri­ate poses, fa­cial ex­pres­sions, ges­tures, and panel lay­outs. That meant Comic Chat was not sim­ply dis­play­ing mes­sages but also mak­ing real-time ed­i­to­r­ial de­ci­sions about how a con­ver­sa­tion should look and feel as a comic. DJ, Tim Skelly, and David Salesin pub­lished a pa­per on the tech­nol­ogy in Comic Chat at SIGGRAPH 96, a com­puter graph­ics con­fer­ence, de­scrib­ing what they had built as an ex­per­i­ment in au­to­matic il­lus­tra­tion con­struc­tion and lay­out.

The vi­sual world of Comic Chat was the work of Jim Woodring, a highly re­garded in­de­pen­dent comic artist whose char­ac­ters gave the soft­ware its dis­tinc­tive look. The team would hand Jim tran­scripts of real chat ses­sions to il­lus­trate, then use the re­sults to fig­ure out whether the whole idea was worth pur­su­ing. It was.

Why open source it now?

Comic Chat rep­re­sents a fas­ci­nat­ing chap­ter in the evo­lu­tion of on­line com­mu­ni­ca­tion. It emerged dur­ing a pe­riod when the in­ter­net was still dis­cov­er­ing what it wanted to be­come. Many rules had not yet been writ­ten, which gave de­vel­op­ers per­mis­sion to try bold con­cepts that might seem un­usual even to­day.

By re­leas­ing Comic Chat as open source, we’re pre­serv­ing an im­por­tant piece of soft­ware his­tory and giv­ing the com­mu­nity an op­por­tu­nity to ex­plore, learn, and build upon it.

The source is avail­able now for ex­plo­ration, study, and ex­per­i­men­ta­tion. Alongside the orig­i­nal snap­shots, we’ve in­cluded a few AI-powered mod­ern­iza­tion at­tempts that demon­strate what’s pos­si­ble—get­ting this 1990s-era C++ and MFC code build­ing with cur­rent Visual Studio tools, con­nect­ing to mod­ern IRC servers, and run­ning leg­i­bly on to­day’s high-res­o­lu­tion Windows ma­chines. These are not pol­ished re-re­leases, but worked ex­am­ples that show Comic Chat can still come alive on mod­ern sys­tems. We’re ex­cited to see what im­prove­ments, ports, ex­per­i­ments, and en­tirely new forms the com­mu­nity brings to it next.

A time cap­sule of in­ter­net op­ti­mism

Looking back, Comic Chat cap­tures some­thing spe­cial about the era in which it was cre­ated.

The early web was filled with ex­per­i­men­ta­tion. What if chat rooms looked like comics?” That ques­tion sounds won­der­fully un­rea­son­able. And yet it was built, shipped, lo­cal­ized into 24 lan­guages, and bun­dled with Windows 98.

That’s part of what makes Comic Chat mem­o­rable decades later. It re­minds us that in­no­va­tion of­ten starts with ideas that are play­ful, un­con­ven­tional, and cre­ative.

One last speech bub­ble

Comic Chat was cre­ated dur­ing a pe­riod when soft­ware teams were will­ing to color out­side the lines, lit­er­ally and fig­u­ra­tively. DJ Kurlander, Tim Skelly, David Salesin, Jim Woodring, and every­one else who touched this pro­ject made some­thing that peo­ple still re­mem­ber and still run thirty years later.

Take a look at the source code, ex­plore what they built, and use its story as in­spi­ra­tion to come up with new un­con­ven­tion­ally de­light­ful things to cre­ate.

And if you hap­pen to read the source code in Comic Sans, we promise not to judge.

How Our Rust-to-Zig Rewrite is Going

rtfeldman.com

For the past year and a half, the team build­ing Roc’s com­piler has been rewrit­ing our 300,000 lines of Rust code into Zig, for rea­sons I’ll re­cap be­low. We re­cently passed an ex­cit­ing mile­stone: fea­ture par­ity with the orig­i­nal com­piler!

Since the Bun pro­ject re­cently shared an ex­pe­ri­ence re­port of their rewrite in the other di­rec­tion (from Zig to Rust, al­though that’s only the tip of the ice­berg of dif­fer­ences be­tween our rewrites), this seems like a nice time to re­flect on how our move from Rust to Zig is go­ing.

Passing Feature Parity

Hitting this mile­stone made it pos­si­ble to up­date Brendan Hansknecht’s charm­ing 2024 WASM-4 game, Rocci Bird (with art by Luke DeVault) to use the new com­piler. It’s a nice ex­am­ple be­cause the whole game is un­der a thou­sand lines of Roc code, and you can play it on itch.io or right here via WebAssembly:

Click or tap the game, then press Space (or tap) to flap. On mo­bile you don’t have a right ar­row key, so re­fresh the page to restart the game.

Rocci Bird’s up­dated source code is a bit more con­cise than the orig­i­nal, and roc build –opt=size now out­puts a 31KB wasm bi­nary. (The orig­i­nal com­piler pro­duced a bi­nary more than dou­ble that size.) Rocci Bird is by no means a large code base, but get­ting it to run at all re­quired land­ing a lot of fea­tures in the new com­piler. Seeing those chunky pur­ple pix­els brought a smile to my face when we fi­nally got there!

To be clear, this is a mile­stone but not a for­mal re­lease. (We aim to land ver­sion 0.1.0 later this year.) That said, it’s a won­der­ful mile­stone to have reached, and I’m ex­tremely grate­ful to all the peo­ple who came to­gether to make this hap­pen! I want to thank some in par­tic­u­lar who have been es­pe­cially help­ful in get­ting the lan­guage and com­piler to this point:

Anthony Bullard and Sam Mohr for col­lab­o­rat­ing on the new parser

Jared Ramirez for the new type-checker (among many other things!)

Ayaz Hafiz for the new lambda set res­o­lu­tion sys­tem, plus tons of the orig­i­nal com­piler

Aurélien Geron for hand-up­dat­ing 108 (!) be­gin­ner ex­er­cises in the Roc Exercism course he orig­i­nally cre­ated

Stephan for get­ting the com­pil­er’s new echo” plat­form run­ning in the browser, so that any­one can now write and run ba­sic Roc pro­grams from the roc-lang.org home­page via a 2.5MB WebAssembly bi­nary!

Niclas Åhdén, Roc’s most pro­lific pro­duc­tion user, for pa­tiently fil­ing help­ful bug re­ports and giv­ing ac­tion­able feed­back about the up­grade process

JRI98 for me­thod­i­cally re­pro­duc­ing and in­ves­ti­gat­ing fuzzer er­rors and other bugs, clos­ing out is­sues that no longer re­pro­duced, and more

Jasper Woudenberg for it­er­at­ing on API de­signs for user­space pack­ages us­ing the new com­piler

Folkert de Vries, Brendan Hansknecht, Brian Carroll, Josh Warner, Agus Zubiaga, and Jelle Teeuwissen for build­ing the foun­da­tion of the orig­i­nal com­piler, with­out which the new com­piler never would have ex­isted

I’ve saved the undis­puted biggest con­trib­u­tors to the new com­piler for last: Anton-4 and Luke Boswell for so many things I can’t even keep track of them all—com­piler work, builtins, plat­forms, pack­ages, ex­am­ples, fix­ing bugs, help­ing be­gin­ners on Roc Zulip…enumerating it all could take up a whole sec­ond post! It’s been in­cred­i­ble see­ing how much you’ve built.

Thank you all so much! I feel hon­ored that you’ve put so much of your valu­able time into this pro­ject. Also thanks to our past and pre­sent spon­sors—rwx, Lambda Class, ohne-mak­ler, mar­t­ian, tweede golf, Vendr, NoRedInk, and many gen­er­ous in­di­vid­ual spon­sors—who have helped get us to this point by sup­port­ing our con­trib­u­tors.

Speaking of time: our 487-day rewrite took 476 days longer than Bun’s 11-day rewrite from their ~500K lines of Zig into Rust. There are many rea­sons for this dif­fer­ence which have noth­ing to do with Rust or Zig, in­clud­ing the fact that theirs was a di­rect port whereas we’d de­cided to rewrite be­cause of how much we were go­ing to change. The tech­niques they used would­n’t have worked in our case.

The laun­dry list of changes we made also means com­par­ing our orig­i­nal Rust code base and new Zig code base won’t be ap­ples-to-ap­ples. Still, we’ve reached a nice point to re­flect on how the rewrite has gone, both in terms of what new fea­tures it has un­locked for Roc pro­gram­mers, as well as how our ex­pe­ri­ences with Rust and Zig have com­pared.

Let’s get into it!

Hot Code Loading + Cross-Compiled Binaries

Roc’s new com­piler au­to­mat­i­cally does hot code load­ing dur­ing de­vel­op­ment. For ex­am­ple, I can run roc server.roc to start a Web server, then change some of its code while it’s run­ning. The next time that server han­dles a re­quest, it’ll au­to­mat­i­cally be han­dled us­ing the new code. Here it is in ac­tion, both in a server and in a sim­ple 2D game:

Hot load­ing is stan­dard be­hav­ior for in­ter­preted lan­guages like Python, but not so much for high-per­for­mance com­piled lan­guages like Roc. When I’m ready to de­ploy, roc build server.roc gets me an LLVM-optimized, self-con­tained bi­nary that I can drop onto a ma­chine and run.

Roc also cross-com­piles; build­ing a sta­tic bi­nary that runs on Alpine Linux is as sim­ple as roc build –target=x64musl, and that com­mand will pro­duce the same out­put bytes (for the same in­put source code bytes) when run on a Mac or any other sys­tem—which not all com­pil­ers guar­an­tee.

Pattern Matching with String Interpolation

The HTTP re­quest-han­dling logic from that video looks like this:

match (verb, path) { (“GET, /users/${id}/${page}“) => match page { ” | profile” => ok(id) settings” => ok(with­_de­fault(user_a­gent, id)) posts/${post_id}” => ok(“Post ID: ${post_id}“) _ => not_­found }

(“GET, /users/${id}“) => ok(id)

(“POST, /posts/new”) => cre­ated(with­_de­fault(…))

_ => not_­found }

This uses sev­eral fea­tures we in­tro­duced in the new com­piler. For ex­am­ple, that /users/${id}” syn­tax is not im­ple­mented with pars­ing tem­plate strings at run­time, but rather with a new lan­guage fea­ture: string in­ter­po­la­tion in­side pat­tern match­ing.

Not only is this type-safe at com­pile time, this en­tire code snip­pet per­forms zero heap al­lo­ca­tions. I’d ex­pect the typ­i­cal lan­guage that ships with hot code load­ing to av­er­age closer to 1 al­lo­ca­tion per line of code here…but Roc is aim­ing high on er­gonom­ics, type safety, and per­for­mance!

You can play around with this syn­tax on the new roc-lang.org home­page - if you scroll down a bit, there’s an WebAssembly build of the com­piler right there on the page that you can use to try out the lan­guage.

By the way, if you’re in­ter­ested in a post on the tech­ni­cal de­tails of how we used the new com­pil­er’s com­pile-time ex­e­cu­tion of pure func­tions to get HTTP re­quest rout­ing down to zero al­lo­ca­tions, let me know on Roc Zulip.

By the way, if you’re in­ter­ested in a post on the tech­ni­cal de­tails of how we used the new com­pil­er’s com­pile-time ex­e­cu­tion of pure func­tions to get HTTP re­quest rout­ing down to zero al­lo­ca­tions, let me know on Roc Zulip.

Why a Scratch-Rewrite?

Unlike Rust, C, and Zig, Roc is not a sys­tems lan­guage; it has au­to­matic mem­ory man­age­ment (using ref­er­ence count­ing, both to avoid trac­ing col­lec­tor pauses and also for Perceus op­ti­miza­tions and op­por­tunis­tic mu­ta­tion like Koka’s). Roc would have way more heap al­lo­ca­tions if it needed one heap al­lo­ca­tion per clo­sure cap­ture (like most non-sys­tems lan­guages do), but our clo­sure cap­tures don’t heap-al­lo­cate be­cause Roc is the first non-aca­d­e­mic lan­guage to im­ple­ment poly­mor­phic de­func­tion­al­iza­tion through lambda set spe­cial­iza­tion.

This might sound like a niche op­ti­miza­tion, but in a func­tional lan­guage like Roc, de­func­tion­al­iza­tion turns out to be sim­i­lar to in­lin­ing in that it un­locks a trea­sure trove of fol­low-up op­ti­miza­tions. Although this sys­tem proved in­cred­i­bly ben­e­fi­cial to Roc’s run­time per­for­mance, it also proved in­cred­i­bly dif­fi­cult for us to im­ple­ment cor­rectly. We strug­gled with nasty bugs in the orig­i­nal im­ple­men­ta­tion, and only af­ter Ayaz Hafiz pro­to­typed a new ar­chi­tec­ture in OCaml were we able to fi­nally get it right in the new com­piler.

Ayaz’s pro­to­type showed that the root of our prob­lems was ar­chi­tec­tural across sev­eral com­piler phases, and fix­ing it would re­quire rewrit­ing most of the com­piler. This was one rea­son we de­cided to rewrite in the first place—that, and sev­eral con­trib­u­tors in­de­pen­dently men­tion­ing they planned to rewrite var­i­ous parts of the com­piler for other rea­sons. We re­al­ized we were about to rewrite al­most all of the com­piler any­way, so it made sense to con­sider a full rewrite as an al­ter­na­tive to the Ship of Theseus ap­proach.

Compilers are un­usual in that scratch-rewrites are the norm among suc­cess­ful pro­jects. It’s of­ten the only way to self-host, al­though not all com­pil­ers rewrite into their own lan­guage; see for ex­am­ple TypeScript’s rewrite to Go. My po­si­tion has al­ways been that Roc’s com­piler should not self-host, so the idea that some­day the ben­e­fits of a rewrite might seem to out­weigh their no­to­ri­ous costs had frankly never oc­curred to me.

The more we talked about it, the more sense it made to do what ba­si­cally every main­stream com­piler to­day has done at some point: rewrite from scratch.

Why Zig?

Once we’d de­cided to scratch-rewrite, the next ques­tion was whether to choose Rust again. Based on our ex­pe­ri­ences with both Rust and Zig (we were al­ready us­ing Zig for a bunch of prim­i­tives in our stan­dard li­brary), we de­cided to build the en­tire com­piler in Zig this time.

I en­joy Rust, I’ve taught a course on it, and I hap­pily use it daily for my work at Zed. Despite what Internet com­ments might have us be­lieve, it’s ex­tremely nor­mal for one lan­guage to be the best fit for one pro­ject, while a dif­fer­ent lan­guage turns out to be the best fit for a dif­fer­ent pro­ject. One size does not ac­tu­ally fit all!

I’ve talked in depth about our rea­sons for go­ing with Zig else­where—in writ­ing, on pod­casts, and so on—and we only se­ri­ously con­sid­ered Rust and Zig, be­cause those were the only sys­tems lan­guages our team knew well enough. The biggest con­sid­er­a­tions on our minds when de­cid­ing be­tween Rust and Zig were:

Build times. Our cargo build times were a ma­jor pain point, even for in­cre­men­tal builds, and get­ting worse as our code base grew. We ex­pected build times in a Zig rewrite to be much faster.

Memory con­trol. We use a va­ri­ety of dif­fer­ent mem­ory al­lo­ca­tors through­out com­pi­la­tion, es­pe­cially are­nas, and struct-of-ar­rays lay­outs all over the place. Rust’s ecosys­tem con­sis­tently as­sumes one global al­lo­ca­tor, in­clud­ing soa_rs. Zig’s whole ecosys­tem as­sumes gran­u­lar al­lo­ca­tors, and struct-of-ar­rays sup­port is stan­dard.

Ecosystem rel­e­vance. Rust’s ecosys­tem is much big­ger than Zig’s over­all…but al­most no pack­ages in ei­ther ecosys­tem are rel­e­vant to our par­tic­u­lar needs. For the niche things we wanted to get off the shelf—such as a faster way to emit LLVM bit­code than wrap­ping LLVMs C++ li­brary—more of that code ex­isted in Zig than in Rust.

Memory-unsafety as­sis­tance. Rust is de­signed to iso­late mem­ory-un­safe code in­side rare un­safe blocks, and use things like miri or Valgrind to vet those. Memory-unsafe code was­n’t rare for us, though (more on this later) and we ended up with about 1,200 uses of un­safe (out of our 300K lines of Rust code; com­pare to about 40,000 uses of un­safe in rust’s 3.5M lines, and re­mem­ber that for com­pil­ers which emit ma­chine code, like roc and rustc, do­ing mem­ory-un­safe things is a big part of the job). Zig has more fea­tures than Rust for mak­ing mem­ory-un­safe code work cor­rectly, and that was the area where we wanted the most help.

After a year and a half of rewrit­ing, how did our ex­pec­ta­tions of Zig’s ben­e­fits line up with the re­al­ity of what we got? And which parts of Rust did we end up miss­ing once we no longer had ac­cess to them?

Life Without Borrow-Checking

Let’s start with mem­ory safety. There’s a fa­mous 2019 Microsoft pre­sen­ta­tion that says, on slide 10:

~70% of the vul­ner­a­bil­i­ties ad­dressed through a se­cu­rity up­date each year con­tinue to be mem­ory safety is­sues.

~70% of the vul­ner­a­bil­i­ties ad­dressed through a se­cu­rity up­date each year con­tinue to be mem­ory safety is­sues.

The pre­sen­ta­tion’s next slide has a break­down by type of mem­ory safety is­sue, which paints the fol­low­ing pic­ture when it comes to Rust and Zig specif­i­cally:

83.6% of vul­ner­a­bil­i­ties ad­dressed through a se­cu­rity up­date in 2018 would have been com­pletely un­af­fected by the choice of Rust or Zig, be­cause both lan­guages han­dle all of these sce­nar­ios (out-of-bounds reads/​write, un­safe casts, unini­tial­ized reads, stack over­flows, and non-mem­ory-safety is­sues) in the same way.

16.4% of the vul­ner­a­bil­i­ties were specif­i­cally use-af­ter-free er­rors. These could have been caught by Zig’s ReleaseSafe run­time mem­ory-safety checks, or Rust’s bor­row checker, or the checks Fil-C uses…mod­ern lan­guages have a va­ri­ety of ways to help catch UAFs, al­though these CVEs from 2018 would have al­most cer­tainly been from C or C++ code in­stead.

ReleaseSafe catches use-af­ter-free er­rors through run­time checks which panic if the pro­gram tries to use freed mem­ory. Compared to Rust’s safe sub­set, Zig’s checks are less com­pre­hen­sive, have a run­time cost, and can panic. That said, Zig with ReleaseSafe has worked great in prac­tice for the TigerBeetle data­base, which re­cently un­der­went a leg­en­dar­ily metic­u­lous Jepsen re­port that found only two safety bugs, nei­ther re­lated to mem­ory safety.

ReleaseFast skips these checks in pro­duc­tion builds to avoid their over­head, but keeps them in de­bug builds and tests to catch mem­ory-safety is­sues dur­ing de­vel­op­ment. If your tests cov­ered every pos­si­ble real-world code path, ReleaseFast would give you the same safety as ReleaseSafe, but that level of test cov­er­age is rarely prac­ti­cal; the real ques­tion is what slips through the cov­er­age cracks in prac­tice. Bun talked about their strug­gles with use-af­ter-frees, but other widely-used pro­jects build­ing with ReleaseFast have had no CVEs caused by mem­ory un­safety in their Zig code. Ghostty is one, and Zig’s com­piler it­self is an­other.

If you want to learn more about these pro­jects, I’ve recorded in-depth con­ver­sa­tions with their cre­ators: Joran Greef on TigerBeetle, Mitchell Hashimoto on Ghostty, and Andrew Kelley on Zig.

If you want to learn more about these pro­jects, I’ve recorded in-depth con­ver­sa­tions with their cre­ators: Joran Greef on TigerBeetle, Mitchell Hashimoto on Ghostty, and Andrew Kelley on Zig.

Rust code has a dif­fer­ent source of mem­ory-safety gaps: the un­safe sec­tions that nearly every Rust pro­gram has some­where in its de­pen­den­cies. Unsafe Rust has all the mem­ory un­safety risk of ReleaseFast Zig code, but none of the run­time checks to catch is­sues dur­ing de­vel­op­ment. The Rust ecosyt­sem has miri to find bugs in non-FFI un­safe code, and Valgrind can help too, but few Rust pro­jects use ei­ther. That said, the cul­tural norm of us­ing un­safe rarely, and au­dit­ing it ex­tra care­fully, has worked out well enough to earn Rust a strong rep­u­ta­tion for mem­ory safety in prac­tice.

Of course, Rust mem­ory un­safety er­rors can and do still slip through the cracks. Deno, a Bun com­peti­tor which is writ­ten in Rust, has had mem­ory-un­safety CVEs in­clud­ing an out-of-bounds read as well as a use-af­ter-free, both in­volv­ing the use of Unsafe Rust. Rocket, a Rust Web Framework, has had a use-af­ter-free CVE, and Actix has had a va­ri­ety of mem­ory-un­safety CVEs from a pe­riod when its use of un­safe was ab­nor­mally high.

When we were de­cid­ing be­tween Rust and Zig for the new com­piler, we were aware of all of this. We knew Rust had a well-de­served rep­u­ta­tion for mem­ory safety, but that mem­ory un­safety could still hap­pen, and we’d ex­pe­ri­enced all of that first­hand with the orig­i­nal com­piler. We also knew we’d be us­ing un­safe way more than typ­i­cal Rust pro­jects, and even though we were al­ready us­ing Valgrind, get­ting help with in­nately mem­ory-un­safe code from Zig’s ad­di­tional checks sounded ap­peal­ing. We wanted the hard stuff to get eas­ier, and we weren’t wor­ried about use-af­ter-free is­sues in a com­piler where al­lo­ca­tions would be over­whelm­ingly done in are­nas with straight­for­ward life­times.

We knew high-pro­file Zig pro­jects had achieved great per­for­mance and mem­ory safety in prac­tice, and we de­cided to aim for be­com­ing an­other of those suc­cess sto­ries.

Memory Safety Post-Rewrite

It’s easy to the­o­rize about how things will go with a par­tic­u­lar tech­nol­ogy choice, but where the rub­ber meets the road is what end users en­counter in real-world us­age. So how has Zig with ReleaseFast worked out for us in prac­tice? How many mem­ory cor­rup­tion in­ci­dents—from use-af­ter-frees or any other cause—have we seen since rewrit­ing our com­piler from Rust to Zig?

Here’s a break­down of bug re­ports in Roc’s is­sue tracker, as clas­si­fied by Claude Opus 4.8:

You might be won­der­ing how the Rust-based com­piler had any mem­ory cor­rup­tion bugs at all, let alone more than dou­ble the to­tal count of the Zig-based one. Is it be­cause of that pesky Unsafe Rust again?

Actually, no. None of those 21 mem­ory cor­rup­tion bugs oc­curred in the com­pil­er’s logic it­self, which is a tes­ta­ment to Rust’s bor­row-checker work­ing as in­tended. The rea­son we had mem­ory cor­rup­tion bugs in our Rust-based com­piler is that it’s a com­piler.

Compilers emit ma­chine in­struc­tions. When a ma­chine ex­e­cutes those in­struc­tions, they can cause mem­ory cor­rup­tion, re­sult­ing in mem­ory cor­rup­tion bug re­ports from the peo­ple who ex­pe­ri­enced them. Regardless of which process had the bug—the com­piler or com­piled pro­gram—in both cases the proces­sor only did the bad thing be­cause the com­piler told it to. And in both cases the fix is the same: the com­pil­er’s code must change, since that code was what caused the mem­ory cor­rup­tion.

Just like every com­piler, Roc’s has had bugs, and some of those have been mis­com­pi­la­tions that led to mem­ory cor­rup­tion. That said, while 8 of the 10 mem­ory cor­rup­tion bugs in the Zig-based com­piler were also mis­com­pi­la­tions, the re­main­ing 2 were in the com­piler it­self. Both were use-af­ter-free bugs in er­ror re­port­ing, with the same symp­tom: file­names in er­ror mes­sages (one in roc check and the other in roc bun­dle) ren­dered as use­less ques­tion-mark-in-di­a­mond char­ac­ters. Rust’s bor­row checker would have caught both.

Now let’s sup­pose we had in­stead cho­sen Rust for our rewrite, or Zig with ReleaseSafe. What would have been the im­pact in prac­tice, hold­ing all else equal?

After 18 months of de­vel­op­ment, hun­dreds of to­tal bug re­ports, and hun­dreds of thou­sands of lines of code, my main take­away from ret­ro­spect­ing on this table is that pick­ing a dif­fer­ent row would have made no ap­pre­cia­ble dif­fer­ence to the pro­ject. So far our choice has got­ten us the out­come we’d hoped for.

As I noted ear­lier, every pro­ject has dif­fer­ent needs. When Bun rewrote in the op­po­site di­rec­tion—from Zig to Rust—their ac­com­pa­ny­ing post noted:

For Bun, cor­rectly han­dling the life­times of garbage-col­lected val­ues [from JavaScript] and man­u­ally-man­aged val­ues has been a ma­jor source of sta­bil­ity is­sues - most of­ten small mem­ory leaks and oc­ca­sion­ally, crashes. Every mem­ory al­lo­ca­tion has to be metic­u­lously re­viewed. Where do these bytes get freed? How do we en­sure it only gets freed once? Did we check for JavaScript ex­cep­tions prop­erly? Is this garbage-col­lected pointer vis­i­ble to the con­ser­v­a­tive stack scan­ner? Is this garbage col­lected mem­ory or man­u­ally man­aged mem­ory?

For Bun, cor­rectly han­dling the life­times of garbage-col­lected val­ues [from JavaScript] and man­u­ally-man­aged val­ues has been a ma­jor source of sta­bil­ity is­sues - most of­ten small mem­ory leaks and oc­ca­sion­ally, crashes. Every mem­ory al­lo­ca­tion has to be metic­u­lously re­viewed. Where do these bytes get freed? How do we en­sure it only gets freed once? Did we check for JavaScript ex­cep­tions prop­erly? Is this garbage-col­lected pointer vis­i­ble to the con­ser­v­a­tive stack scan­ner? Is this garbage col­lected mem­ory or man­u­ally man­aged mem­ory?

Roc’s com­piler does­n’t have these par­tic­u­lar chal­lenges be­cause it does­n’t in­ter­face with JavaScript or any other trac­ing garbage col­lec­tor. For Bun, use-after-free, dou­ble-free, and forgot to free’” er­rors have been a large per­cent­age of bugs,” whereas er­rors like these have been a small per­cent­age of Roc’s bugs. And of course Roc’s com­piler faces other chal­lenges that Bun does­n’t. Different pro­jects have dif­fer­ent needs!

In our case, I’m not sure how I could look back at what’s ac­tu­ally hap­pened and con­clude that what we needed was a big­ger in­vest­ment in tool­ing to pre­vent mem­ory safety bugs in the com­piler it­self. There’s a much stronger case that we would ben­e­fit from bet­ter tool­ing to catch mem­ory safety bugs in our com­piled out­put, which has al­ways been out of scope for the bor­row checker.

Build Times

We wanted faster builds from Zig. Did we get them?

Well, the good news is that zig build –watch -fincremental can re­build a change to our cur­rent ~450K lines of Zig code in about 35 mil­lisec­onds. That’s even faster than what we were hop­ing for when we con­sid­ered Zig’s build speed a sell­ing point for the rewrite!

The bad news is that Zig’s cur­rent sta­ble 0.16.0 re­lease has a bug that breaks -fincremental on our code base. The fix al­ready landed, but to get it we’d have to build on a nightly 0.17.0 pre­re­lease build (which has break­ing lan­guage changes), along with ven­dor­ing and up­grad­ing our af­fected de­pen­den­cies to 0.17.0. We de­cided to wait for the next sta­ble re­lease in­stead.

As of the last com­mit that had Rust sources in our code base, here’s a tim­ing com­par­i­son on my Intel desk­top ma­chine run­ning Ubuntu 26 for build­ing cold (no cache, but pack­ages down­loaded lo­cally) com­pared to do­ing an in­cre­men­tal re­build af­ter mak­ing a triv­ial edit to our parser:

Note that our Zig build con­fig­u­ra­tion as of the fea­ture-par­ity com­mit was re­build­ing rarely-chang­ing ar­ti­facts on every build that we later de­cided to re­build only on de­mand. That’s why to­day’s cold builds are faster than they were back at 300K LoC, even though our lines of code have in­creased by ~50% since then.

Note that our Zig build con­fig­u­ra­tion as of the fea­ture-par­ity com­mit was re­build­ing rarely-chang­ing ar­ti­facts on every build that we later de­cided to re­build only on de­mand. That’s why to­day’s cold builds are faster than they were back at 300K LoC, even though our lines of code have in­creased by ~50% since then.

Rust 1.97 is the cur­rent sta­ble re­lease to­day, and 1.85 was the cur­rent sta­ble re­lease 487 days ago (the time our rewrite took to reach to fea­ture par­ity). So if we’d stayed on Rust for the same du­ra­tion, we could have seen our in­cre­men­tal build times de­crease from 10 sec­onds to 3.4. That’s a big jump! I re­ally ap­pre­ci­ate all the hard work that Rust con­trib­u­tors have done to im­prove build times. Eliminating 2/3 of our in­cre­men­tal build times over 18 months would have been a very wel­come change if we’d stayed on Rust, and it’s a big­ger im­prove­ment than I would have an­tic­i­pated in an 18-month pe­riod. Bravo!

As im­pres­sive as that im­prove­ment is, Zig’s 35ms is still way ahead. Not only is it 1/100th the build time of 3.4 sec­onds, it’s also in a dif­fer­ent per­for­mance cat­e­gory—and that 35ms is on a Zig code base with ~50% more lines of code than the Rust one that got 3.4s. I ex­pect Roc’s code base to keep grow­ing, and for this gap to keep grow­ing with it; I’ve never heard of any ini­tia­tive on Rust’s roadmap com­pa­ra­ble to -fincremental.

So while our de­ci­sion to re­main on sta­ble 0.16.0 (plus how many of our con­trib­u­tors run Mac lap­tops with ARM proces­sors; -fincremental only works on x86 – 64 CPUs right now) means we haven’t yet reaped the an­tic­i­pated build-time re­wards of choos­ing Zig for the rewrite, we cer­tainly have some­thing to look for­ward to in the next sta­ble Zig re­lease!

Memory Control: Zero-Parse Deserialization

Roc’s new on-disk caching sys­tem uses a tech­nique I first learned about from Zig’s com­piler, and which Casey Muratori told me is com­mon prac­tice in game pro­gram­ming. It re­lies on the happy co­in­ci­dence that if you’re or­ga­niz­ing your mem­ory in the way that runs fastest on mod­ern hard­ware any­way, you can also load it from disk di­rectly into mem­ory and start us­ing it with­out pars­ing any­thing.

Here’s how it works:

All of our com­piler data struc­tures are rep­re­sented as ar­rays with 32-bit in­dices over point­ers (and of­ten in struc­ture-of-ar­rays form).

This not only saves mem­ory and runs faster, it also means our data struc­tures can be writ­ten di­rectly to disk with­out need­ing to be se­ri­al­ized into a dif­fer­ent for­mat first.

The big­ger ben­e­fit is that this lets us de­se­ri­al­ize them back into mem­ory with­out pars­ing the on-disk bytes in any way. We load the bytes into mem­ory, do some re­lo­ca­tions to point our ex­ist­ing data struc­tures to the newly-loaded ar­rays, and we’re ready to go.

This means we de­se­ri­al­ize at the speed of load­ing the bytes from disk into mem­ory—so, ac­tu­ally I/O bound. If those bytes are al­ready in the op­er­at­ing sys­tem’s disk cache, it means we load cached work from pre­vi­ous builds at roughly the speed of mem­cpy.

When you run roc check twice in a row, the first time it caches all of its out­puts on disk us­ing this strat­egy. The sec­ond time, if the in­put source code files haven’t changed, all the parsed/​type-checked/​etc. data struc­tures jump straight from disk into mem­ory. It’s ex­tremely fast. roc test sim­i­larly caches the out­comes for tests of pure func­tions (which are de­ter­min­is­tic), and all of this is done with file-level gran­u­lar­ity, so if you change one file you’ll only be pay­ing for re­do­ing work of that file and any oth­ers that de­pend on it.

This zero-parse de­se­ri­al­iza­tion strat­egy only works be­cause we’re fol­low­ing this pro­gram­ming with­out point­ers style for all of our com­piler data struc­tures. If we in­stead used point­ers every­where (like al­most all com­pil­ers do), de­se­ri­al­iza­tion could­n’t be zero-parse.

This ap­proach has safety risks, how­ever. Similarly to how a pointer in mem­ory can point to the wrong ad­dress (e.g. lead­ing to a use-af­ter-free), any in­dex can be used as a lookup into the wrong ar­ray at run­time, at which point you end up with what­ever ran­dom bytes hap­pened to be at that lo­ca­tion. Rust’s bor­row checker is de­signed to help with pointer life­times, but it does­n’t at­tempt to an­swer the ques­tion which in­dex goes with which ar­ray?” be­cause that has never been in scope for its de­sign.

SQLite should have (Rust-style) editions

mort.coffee

Date: 2026 – 07-15 Git: https://​git­lab.com/​mort96/​blog/​blob/​pub­lished/​con­tent/​00000-home/​00017-sqlite-edi­tions.md

SQLite is an amaz­ing data­base en­gine. I use it as a data­base for plenty of em­bed­ded pro­jects, and I don’t think it’s an ex­ag­ger­a­tion to call it the in­dus­try stan­dard for lo­cal data stor­age. Some server soft­ware even uses it; for ex­am­ple, lob­ste.rs is now run­ning on SQLite.

Unlike tra­di­tional RDBMSes (Relational DataBase Management Systems), SQLite is not a sep­a­rate process; it’s an RDBMS as a li­brary, mean­ing your soft­ware re­mains self con­tained. Unlike tra­di­tional file for­mats, you don’t need to write cus­tom se­ri­al­iz­ers and parsers. In some ways, it’s the best of both worlds.

There’s just one huge prob­lem though. Its de­faults are all wrong.

Bad de­fault #1: Foreign key con­straints are ig­nored by de­fault

You read that right. Foreign key con­straints are ar­guably the pri­mary tool we have to en­sure that a data­base re­mains con­sis­tent and don’t have dan­gling ref­er­ences.

As a quick primer, this is how an SQL for­eign key con­straint looks:

CREATE TABLE users ( id INTEGER PRIMARY KEY, dis­play_­name TEXT );

CREATE TABLE posts ( id INTEGER PRIMARY KEY, user_id INTEGER NOT NULL, con­tent TEXT NOT NULL, FOREIGN KEY(user_id) REFERENCES users(id) );

The typ­i­cal be­hav­ior for all other RDBMSes would be that the user_id col­umn of a post must al­ways ref­er­ence the ID of a valid user. You can’t cre­ate a new post with­out pro­vid­ing a valid user ID, you can’t delete a user with­out also delet­ing its posts, lest you get a for­eign key con­straint vi­o­la­tion er­ror.

The only RDBMS I’m aware of which does­n’t en­force this by de­fault is SQLite.

This is made even worse by SQLite’s ten­dency to re-use ROWID. You see, in this ex­am­ple, those INTEGER PRIMARY KEY rows be­come aliases for the table’s ROWID, which is a unique in­te­ger ID as­signed to every row of a table in SQLite. The al­go­rithm for as­sign­ing ROWID is a bit com­pli­cated (more de­tails in the SQLite doc­u­men­ta­tion), but it re­sults in ID re-use in some cases. This means that a dan­gling ref­er­ence eas­ily re­sults in a ref­er­ence to the wrong row, which is even worse than a dan­gling ref­er­ence be­cause every­thing will seem fine. You don’t even get an er­ror dur­ing lookup.

Just look at this hy­po­thet­i­cal se­quence of op­er­a­tions in our toy data­base schema:

– Bob cre­ates a user ac­count INSERT INTO users (display_name) VALUES (‘Bob’); SELECT * FROM users; — id | dis­play_­name — 1 | Bob

– Bob posts an in­tro­duc­tion post INSERT INTO posts (user_id, con­tent) VALUES (1, Hello, I am Bob’); SELECT u.dis­play_­name, p.con­tent FROM users as u, posts as p WHERE u.id = p.user_id; — dis­play_­name | con­tent — Bob | Hello, I am Bob

– Bob deletes his ac­count, — but we for­got to delete the posts. — SQLite does­n’t pro­duce an er­ror be­cause it ig­nores our for­eign key. DELETE FROM users WHERE id = 1;

– Alice cre­ates an ac­count. — Alice gets the same ID that Bob had due to the ROWID al­go­rithm. INSERT INTO users (display_name) VALUES (‘Alice’); SELECT * FROM users; — id | dis­play_­name — 1 | Alice

– Alice has now in­her­ited Bob’s old post! SELECT u.dis­play_­name, p.con­tent FROM users as u, posts as p WHERE u.id = p.user_id; — dis­play_­name | con­tent — Alice | Hello, I am Bob

The fix is to en­able for­eign_keys with a pragma:

PRAGMA for­eign_keys = ON;

If we had done this in the be­gin­ning, the buggy DELETE would have pro­duced an er­ror:

DELETE FROM users WHERE id = 1; — Runtime er­ror: FOREIGN KEY con­straint failed (19)

Bad de­fault #2: Columns can store the wrong data type

SQLite has a sim­ple type sys­tem: a value can be NULL, an INTEGER, a REAL (aka a dou­ble pre­ci­sion float), TEXT, or a BLOB (aka bi­nary data). Consequently, a col­umn can be de­fined to hold val­ues of any of those types.

However, a col­umn de­fined as an INTEGER col­umn is­n’t re­stricted to only in­te­gers; in­stead, SQLite con­sid­ers it to use INTEGER affin­ity”. What this means is es­sen­tially:

If you try to in­sert a TEXT value, and it is a valid string rep­re­sen­ta­tion of an in­te­ger, it is con­verted to an in­te­ger and stored as such.

If you try to in­sert a TEXT value, and it is a valid string rep­re­sen­ta­tion of a real num­ber, it is con­verted to a real (aka dou­ble pre­ci­sion float) and stored as such.

Otherwise, the value is stored as-is.

Other affini­ties have dif­fer­ent but sim­pler rules:

Columns with BLOB affin­ity store val­ues as-is.

Columns with TEXT affin­ity store BLOB, TEXT and NULL val­ues as-is, but con­vert nu­meric val­ues to TEXT.

Columns with REAL affin­ity work like columns with INTEGER affin­ity ex­cept that in­te­ger val­ues are con­verted to REAL.

Here’s how this looks in prac­tice:

CREATE TABLE mu­sic ( id INTEGER PRIMARY KEY, name TEXT, du­ra­tion_sec INTEGER );

INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘Lost In Hollywood’, 321); INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘Comfortably Numb’, 382); INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘The Way of All Flesh’, Way too long, I mean come on’); SELECT * FROM mu­sic; — id | name | du­ra­tion_sec — 1 | Lost In Hollywood | 321 — 2 | Comfortably Numb | 382 — 3 | The Way of All Flesh | Way too long, I mean come on

I don’t think I need to ex­plain why it’s a bad idea for a data­base to be so care­less about data val­i­da­tion. It would be one thing if SQLite was an ex­plic­itly dy­nam­i­cally typed doc­u­ment data­base, but it’s not. SQLite asks me through its syn­tax rules, What type do you want to go into this col­umn”.

I once had to clean up a pro­ject where some code had ac­ci­den­tally been writ­ing the strings 1’ and 0’ to a col­umn which was in­tended to store booleans (1 and 0). That was not a fun de­bug­ging story.

Luckily, SQLite has the con­cept of strict ta­bles, which makes SQLite pro­duce a type er­ror when the wrong type is in­serted into a col­umn:

CREATE TABLE mu­sic ( id INTEGER PRIMARY KEY, name TEXT, du­ra­tion_sec INTEGER ) strict;

INSERT INTO mu­sic (name, du­ra­tion_sec) VALUES (‘The Way of All Flesh’, Way too long, I mean come on’); — Runtime er­ror: can­not store TEXT value in INTEGER col­umn mu­sic.du­ra­tion_sec (19)

Unfortunately, there is no pragma to glob­ally make all ta­bles strict. So you have to re­mem­ber to add the strict tag to every table man­u­ally.

There’s a cou­ple of ar­gu­ments against strict ta­bles which I want to cover here.

The au­thors of SQLite have writ­ten about their pref­er­ence for flexible typ­ing”. Personally, I find this a re­ally strange piece of writ­ing. It does­n’t pro­vide any ex­am­ples for why it could ever be use­ful to in­sert a BLOB into an INTEGER col­umn. All it does is il­lus­trate why it’s some­times use­ful to have a col­umn which can store val­ues of any type. Strict ta­bles have a so­lu­tion for that; it’s called the ANY data type. You can still cre­ate columns which ac­cept any value, you just have to be ex­plicit about it.

A much bet­ter ar­gu­ment is pro­vided by user zie’ on lob­ste.rs. You see, strict ta­bles in SQLite don’t just en­force types. They also change the rules for how type spec­i­fiers are parsed.

Non-strict SQLite ta­bles use the fol­low­ing rules to de­ter­mine the type of a col­umn (from SQLite’s doc­u­men­ta­tion):

If the de­clared type con­tains the string INT then it is as­signed INTEGER affin­ity. If the de­clared type of the col­umn con­tains any of the strings CHAR, CLOB, or TEXT then that col­umn has TEXT affin­ity. Notice that the type VARCHAR con­tains the string CHAR and is thus as­signed TEXT affin­ity. If the de­clared type for a col­umn con­tains the string BLOB or if no type is spec­i­fied then the col­umn has affin­ity BLOB. If the de­clared type for a col­umn con­tains any of the strings REAL, FLOA, or DOUB then the col­umn has REAL affin­ity. Otherwise, the affin­ity is NUMERIC.

If the de­clared type con­tains the string INT then it is as­signed INTEGER affin­ity.

If the de­clared type of the col­umn con­tains any of the strings CHAR, CLOB, or TEXT then that col­umn has TEXT affin­ity. Notice that the type VARCHAR con­tains the string CHAR and is thus as­signed TEXT affin­ity.

If the de­clared type for a col­umn con­tains the string BLOB or if no type is spec­i­fied then the col­umn has affin­ity BLOB.

If the de­clared type for a col­umn con­tains any of the strings REAL, FLOA, or DOUB then the col­umn has REAL affin­ity.

Otherwise, the affin­ity is NUMERIC.

A con­se­quence of this rule, com­bined with SQLite’s loose typ­ing, is that you can give your columns type names such as DATETIME or KEY_VALUE_SET or COLOR, and have a data­base con­nec­tor/​wrap­per which au­to­mat­i­cally knows to se­ri­al­ize and de­se­ri­al­ize columns with cus­tom types. And if noth­ing else, those cus­tom type names serve as use­ful doc­u­men­ta­tion.

I have to ac­knowl­edge that just chang­ing the de­fault from non-strict ta­bles to strict ta­bles, with no fur­ther changes, would give up on this some­what nifty quirk. However, I think we would be much bet­ter served by cus­tom type aliases.

If we could write some­thing like:

CREATE TYPE KEY_VALUE_SET = TEXT;

and then use KEY_VALUE_SET as a type name in a strict table, I think every­one would be happy. I would prob­a­bly start us­ing such a fea­ture lib­er­ally to doc­u­ment the ex­pected pat­tern of data in my columns. In a real world schema, you in­evitably end up with TEXT columns which have to be parsed by ap­pli­ca­tion code.

As an aside to this aside, it would be neat if we could as­so­ci­ate CHECK con­straints with a cus­tom type.

Update: masklinn’ on lob­ste.rs points out that the SQL 99 stan­dard al­ready spec­i­fies type aliases, called CREATE DOMAIN. This al­ready sup­ports con­straints as well. So re­ally, SQLite just needs to add sup­port for the stan­dard CREATE DOMAIN state­ment.

Update: masklinn’ on lob­ste.rs points out that the SQL 99 stan­dard al­ready spec­i­fies type aliases, called CREATE DOMAIN. This al­ready sup­ports con­straints as well. So re­ally, SQLite just needs to add sup­port for the stan­dard CREATE DOMAIN state­ment.

Bad de­fault #3: SQLITE_BUSY er­rors with con­cur­rent writ­ers

SQLite al­lows mul­ti­ple con­cur­rent read­ers, but only one writer at a time. By de­fault, if you have two processes try­ing to ac­quire a write lock at the same time, one of them will im­me­di­ately re­ceive an SQLITE_BUSY er­ror.

This is not the be­hav­ior I ex­pect. I ex­pect SQLite to wait for the lock to get un­locked, up to some time­out. It’s do­ing disk IO af­ter all, so I al­ready struc­ture my code with the as­sump­tion that a write could po­ten­tially be slow.

The de­fault be­hav­ior has lead me to writ­ing real-world bugs, where sys­tems would some­times just crash. I’ve man­u­ally writ­ten retry loops to fix it.

The fix is to set busy_­time­out with a pragma:

PRAGMA busy_­time­out = 5000;

This makes SQLite try to ac­quire the lock for up to 5 sec­onds be­fore er­ror­ing with a SQLITE_BUSY er­ror.

I did­n’t learn about this set­ting un­til re­cently. It seems like such an ob­vi­ous de­fault that I’m as­ton­ished that it’s not.

Update: I should add a note here about why sup­port for con­cur­rent writ­ers is de­sir­able. During nor­mal op­er­a­tion, you’re usu­ally best served by struc­tur­ing your soft­ware such that all writes are done by a sin­gle process, ide­ally a sin­gle thread. Concurrent writ­ers will never be fast. But there are non-typ­i­cal sit­u­a­tions. Maybe you need to man­u­ally clean up a data­base in­ter­ac­tively us­ing the sqlite3 tool in­ter­ac­tively on the com­mand line. Maybe you have scripts for un­com­mon ad­min­is­tra­tive tasks which you haven’t had the need to write a front-end for. These are per­fectly le­git­i­mate and, I be­lieve, fairly com­mon use cases. I think it’s bad that with SQLite’s de­faults, this kind use has a chance to just crash the soft­ware by mak­ing it throw an un­ex­pected SQLITE_BUSY er­ror.

Update: I should add a note here about why sup­port for con­cur­rent writ­ers is de­sir­able.

During nor­mal op­er­a­tion, you’re usu­ally best served by struc­tur­ing your soft­ware such that all writes are done by a sin­gle process, ide­ally a sin­gle thread. Concurrent writ­ers will never be fast. But there are non-typ­i­cal sit­u­a­tions. Maybe you need to man­u­ally clean up a data­base in­ter­ac­tively us­ing the sqlite3 tool in­ter­ac­tively on the com­mand line. Maybe you have scripts for un­com­mon ad­min­is­tra­tive tasks which you haven’t had the need to write a front-end for. These are per­fectly le­git­i­mate and, I be­lieve, fairly com­mon use cases. I think it’s bad that with SQLite’s de­faults, this kind use has a chance to just crash the soft­ware by mak­ing it throw an un­ex­pected SQLITE_BUSY er­ror.

Bad de­fault 4: Performance

There’s a lot to say about per­for­mance tun­ing in SQLite. When cor­rectly con­fig­ured, it can be a truly fast RDBMS, with the abil­ity to fill roles we typ­i­cally re­serve for the big servers like PostgreSQL or MySQL.

But by de­fault, its per­for­mance is­n’t great. Smarter peo­ple than me have writ­ten much more on this, and I rec­om­mend Sylvain Kerkour’s Optimizing SQLite for servers if you’re in­ter­ested in this topic.

But the most sig­nif­i­cant bad de­fault is that SQLite’s Write-Ahead Log (WAL) is dis­abled by de­fault. It can be en­abled with:

PRAGMA jour­nal_­mode = WAL;

The WAL pro­vides a dra­matic write speed-up in most cir­cum­stances. Additionally, it lets us dras­ti­cally re­duce the amount of disk syncs with­out risk­ing data cor­rup­tion by chang­ing an­other set­ting:

PRAGMA syn­chro­nous = NORMAL;

See the SQLite doc­u­men­ta­tion on what ex­actly syn­chro­nous does.

The so­lu­tion: edi­tions?

The oft-cited rea­son for why these de­faults re­main, well, de­fault, is of course back­wards com­pat­i­bil­ity. Changing de­faults now would likely break lots of old soft­ware and make peo­ple afraid to up­grade SQLite in the fu­ture in case it breaks every­thing again, just like how I’m afraid to up­grade Python be­cause every upgrade” breaks a bunch of soft­ware I use. It’s a laudi­ble and rare goal to try to not break your de­pen­dents.

However, I think the so­lu­tion is sim­ple: add one super pragma” which changes all the bad de­faults. I pro­pose that the fol­low­ing:

PRAGMA edi­tion = 2026;

should be an alias for at least the fol­low­ing set of prag­mas:

PRAGMA for­eign_keys = ON; PRAGMA busy_­time­out = 5000; PRAGMA jour­nal_­mode = WAL; PRAGMA syn­chro­nous = NORMAL;

And also make strict mode the de­fault for ta­bles.

This should be a nice mid­dle ground which avoids break­ing back­wards com­pat­i­bil­ity, but lets the data­base en­gine move for­wards and not be bogged down by its own his­tory.

The edi­tion idea is stolen straight from Rust edi­tions. The ad­van­tage of a year-based edi­tion rather than some­thing like JavaScript’s use strict”; is that as the years go by, the sen­si­ble de­faults may change. Maybe some­thing like Hctree’s WAL2 makes its way into the main branch, say, in the year 2034, so maybe PRAGMA edi­tion = 2034 will some day set PRAGMA jour­nal_­mode = WAL2.

Anyway, that’s all. I think SQLite should have an edi­tion sys­tem with up­dated sets of de­faults. Are there any things I’ve missed which makes this a bad idea? Or more prag­mas which should be added to my imag­i­nary 2026 edi­tion”?

I also filed the edges off my MacBook

www.brt.fyi

I like my new-to-me MacBook be­cause it en­ables me to cre­ate more stuff com­pared to my now al­most 8 year old Thinkpad I had be­fore, mainly due to bat­tery and screen rea­sons. There is one thing about it that I can­not wrap my head around from a de­sign per­spec­tive and it’s the sharp edges es­pe­cially around the wrist area. On a flat sur­face it’s no prob­lem at all, but low and be­hold a lap­top will be fre­quently used on a lap, mean­ing the wrists will touch the sharp edge at an an­gle which is very un­com­fort­able.

I’ve found some posts on­line about peo­ple that have filed the edges off of their Macs be­fore, like this nice one by Kent Walters. In my case how­ever, al­most no one freaks out about this be­cause no one ac­tu­ally no­tices it. It went well, so I thought why not doc­u­ment the process.

Note I am ab­solutely not a good crafts­man, so do not use this as a guide!

I thought a long time about how to do it so as not to have it re­sult in a wavy bezel. Although I re­ally wanted to use this pro­ject as an ex­cuse to buy a ran­dom or­bital sander, I very quickly piv­oted from this idea be­cause I would prob­a­bly cause more dam­age with it than I would with a hand file. Another idea was to 3D print some sort of cham­fer to use as a guide, just to have the ini­tial part cleanly taken off at an an­gle. I played around with this idea for a while but con­cluded that I could prob­a­bly not get to the re­quired level of pre­ci­sion I wanted, es­pe­cially since it would have to hold the file/​sand­pa­per. I ended up just us­ing gen­eral pur­pose metal file I had at hand and pro­gres­sive sand pa­per (in block form, strongly rec­om­mended).

Some tape helped me to mark the ar­eas from be­low and above the bezel that I wanted to have file off. This worked re­ally well. Of course I taped off the track­pad, the key­board and head­phone plug etc. to avoid any residue from get­ting in­side. Additionally, I used very lit­tle soapy wa­ter that I ap­plied to the sand­pa­per through­out to keep the dust con­tained some­what. The ini­tial fil­ing with the gen­eral metal file was a bit scary, but it did not take off too much and the tape helped me to achieve an even level. The sand­pa­per blocks also helped with that, and I went up to 1200 coarse­ness.

The tricky part for me were these pointy dots on the lit­tle gap in the mid­dle. I did not want to go too heavy here with the fil­ing so I used some model mak­ing files very, very lightly and then just used the 1200 sand­pa­per un­til I was sat­is­fied with the re­sult. In the end, I was care­ful to re­move any dust that had ac­cu­mu­late, for which this lit­tle air blower sque­egy thing came in handy.

This is a blue” M4 MacBook Air, so I’m cu­ri­ous how it will look over time due to the an­odized alu­minum. Another note would be to test it out a bit and see how it feels. I have quite large arms so when typ­ing I also touch the cor­ners, so I had to go back and file them too, but that might not be nec­es­sary de­pend­ing on how you do it.

What I liked about the Thinkpad was that I could just chuck it any­where with­out wor­ry­ing about it. I made it a point when I bought this Mac to treat it just the same way. Ultimately, it is a tool and should be used ap­pro­pri­ately. Even if it is nice and shiny, if mod­i­fy­ing it would make it serve it’s pri­mary pur­pose as a tool bet­ter, it’s worth a con­sid­er­a­tion. So I like this re­cent wave of en­cour­age­ment around sim­i­lar mod­i­fi­ca­tions, and it is also far eas­ier and ap­proach­able than it seems ini­tially.

OpenAI and Anthropic — Startups.RIP

joinedanthropic.com

Counting each YC chap­ter by batch year shows when these founders came through. A founder with two YC star­tups ap­pears in both years.

To add this web app to your iOS home screen tap the share button and select "Add to the Home Screen".

10HN is also available as an iOS App

If you visit 10HN only rarely, check out the the best articles from the past week.

Visit pancik.com for more.