Please en­able JS and dis­able any ad blocker

A bench­mark you re­ally don’t want mod­els to be sat­u­rated with.

Learn more

Score

↖ Most il­le­gal­Least il­le­gal ↘

8

Anthropic

8

OpenAI

1

Meta

0

Google

0

Moonshot

Scores in­di­cate count of il­le­gal ac­tiv­ity. Higher is… you de­cide.

Methodology

Felony Bench counts unique in­stances where AI agents af­fect third-party en­ti­ties. Escaping a sand­box alone does not con­sti­tute a counted in­ci­dent. It is for these rea­sons that Frontier Security’s Kimi K3 in­ci­dent and Alibaba’s ROME in­ci­dent are not counted.

Your Kobo can run apps now.

Cobalt is an open-source ap­pli­ca­tion plat­form for Kobo e-read­ers: a launcher, a signed App Store, a Rust SDK, and a run­time that keeps every app in its own un­priv­i­leged process.

Install it once over USB. Every app af­ter that in­stalls, up­dates and re­moves on the reader it­self, over Wi-Fi. A re­boot re­turns to the stock Kobo reader.

Not af­fil­i­ated with Rakuten Kobo

Running on a Kobo.

Every app is a sta­tic ARM bi­nary run­ning as its own un­priv­i­leged process on stock hard­ware. The App Store in­stalls, up­dates and re­moves them over Wi-Fi, with sig­na­tures ver­i­fied be­fore any­thing launches.

arXiv pa­pers and cod­ing agents, on the panel.

These are pho­tographs of the de­vice, not sim­u­la­tor cap­tures. The arXiv app reads the HTML ren­der­ing arXiv pub­lishes for every pa­per since December 2023: ab­stracts, sec­tions, math and re­sult ta­bles, pag­i­nated for the panel.

Apps

The apps.

Every screen­shot be­low is a cap­ture from a Kobo Clara BW. Store apps ver­sion in­de­pen­dently of the plat­form; the rest ship with the plat­form in­stall.

Launcher

Opens in­stalled apps and al­ways keeps a route back to the Kobo reader.

App Store

Installs, up­dates, re­moves and re­in­stalls signed apps over Wi-Fi.

arXiv

Browses a sub­jec­t’s newest preprints and reads the full text on the panel.

Sudoku

Store-only by de­sign: in­stalling it proves de­liv­ery of an app the USB pack­age never con­tained.

Morse

Sends a typed mes­sage in Morse on the front light, one let­ter across the whole panel.

Gutenbird

Reads any OPDS li­brary: Project Gutenberg, Standard Ebooks, Open Library, or yours.

Hacker News

Top, New, Ask and Show sto­ries with com­plete com­ment threads.

Feeds

Discovers a site’s feed and pre­sents its ar­ti­cles with­out the site’s lay­out.

Daily Brief

Collects the day’s sto­ries in the back­ground while you use an­other app.

Sidekick

Approve or deny re­quests from cod­ing agents, away from the key­board.

Terminal

A panel-na­tive shell with keys that send in­put im­me­di­ately.

Components

The UI toolk­it’s con­trols, lay­outs, ty­pog­ra­phy and states, on the panel.

Settings

Connectivity, hard­ware, and plat­form up­dates, kept sep­a­rate from Store.

Todo

A per­sis­tent list with touch en­try and com­pleted-item states.

Tic-tac-toe

Two play­ers, par­tial re­freshes for in­di­vid­ual cells.

Magnet

Locates the hall sen­sor be­hind the bezel and re­ports its changes.

The SDK

An app is one Rust file.

Implement KoboApp, de­scribe screens de­clar­a­tively, and the run­time han­dles lay­out, e-ink re­fresh plan­ning, Back nav­i­ga­tion and life­cy­cle.

Apps don’t open de­vice re­sources; they ask. Network, stor­age, au­dio, front­light and Wi-Fi are ca­pa­bil­ity-gated, and a re­fusal comes back as a value the app can han­dle.

kobo new my-app cd my-app kobo dev

Read the SDK docs

use kobo_sdk::{ ActionId, Context, KoboApp, ScreenBuilder, };

#[derive(Default)] struct Hello { taps: u32 }

impl KoboApp for Hello { fn on_s­tart(&mut self, ctx: &mut Context) { self.show(ctx); }

fn on_ac­tion( &mut self, ctx: &mut Context, a: ActionId, ) { if a == kobo_sdk::ac­tion_id(“tap”) { self.taps += 1; } self.show(ctx); } }

impl Hello { fn show(&self, ctx: &mut Context) { let screen = ScreenBuilder::new(“hello”) .top_bar(“Hello”) .heading(format!(“{} taps”, self.taps)) .button(“tap”, “Tap me”) .build(); ctx.set_screen(screen); } }

fn main() { let app = Hello::default(); let _ = kobo_sdk::run(“hello”, app); }

The Store

Signed pack­ages, ver­i­fied be­fore launch.

Store reads a signed cat­a­log from a fixed GitHub re­lease. Each pack­age holds one ARM ex­e­cutable and a signed canon­i­cal man­i­fest. The run­time ver­i­fies the cat­a­log, the pack­age, the in­stalled man­i­fest and the bi­nary be­fore an app runs.

App re­leases are in­de­pen­dent of plat­form re­leases: merg­ing an app PR builds it for ARM, signs it, and up­dates the cat­a­log. No Cobalt ver­sion bump, no re­in­stall. The app sim­ply ap­pears in Store.

The Cobalt plat­form it­self also up­dates over Wi-Fi, through Settings, on a chan­nel sep­a­rate from the app cat­a­log. The USB ca­ble is only ever needed once.

Install and cat­a­log trans­ac­tions are re­cov­ery-safe; an in­ter­rupted up­date leaves the reader with the ver­sion it had.

Publish your own app →

Install

Installing from source.

Charge a Kobo Clara BW (N365) and con­nect it over USB. Other mod­els are re­fused, not guessed at.

Run the setup:

git clone https://​github.com/​Ban­dar­Labs/​Cobalt.git cd Cobalt rustup tar­get add ar­mv7-un­known-linux-musleabihf cargo run -p kobo-cli — setup

Restart the reader and open Cobalt from Kobo’s menu.

Open Store. Everything from here on ar­rives over Wi-Fi.

The com­plete walk­through, in­clud­ing re­cov­ery steps, is in docs/​IN­STALL.md.

Contributing

Contribute an app.

App con­tri­bu­tions are reg­u­lar pull re­quests. If it runs on your de­vice and the PR shows it run­ning, it gets merged and pub­lished.

Build it. Add the app as a work­space pack­age un­der apps/&​lt;app-id>/ and reg­is­ter it in apps/​cat­a­log.json.

Test it. Add unit and lay­out tests, and run it in the browser and run­time sim­u­la­tors.

Run it on your own de­vice. A real Clara BW, not just the sim­u­la­tor.

Open a PR with a gif or pho­tos of it run­ning. Once re­viewed and merged, the pub­lish work­flow signs it and it ap­pears in Store. No plat­form re­lease needed.

Own a dif­fer­ent Kobo model? Porting is wel­come too; open an is­sue first so the de­vice pro­file can be agreed. Full de­tails in docs/​CON­TRIBUT­ING_APPS.md.

Safety

Device sup­port and safety.

Cobalt does not re­place Kobo’s boot chain. Device writes are gated on an ex­act hard­ware and firmware match, and a re­boot re­turns to the stock reader. The first in­stal­la­tion does mod­ify files on the user stor­age par­ti­tion, and it is pro­vided with­out war­ranty.

Only the Clara BW pro­file has been hard­ware-tested. Don’t in­stall on an­other model un­til it has a re­viewed, hard­ware-tested pro­file. Cobalt is an in­de­pen­dent pro­ject, not af­fil­i­ated with Rakuten Kobo.

“Over the past 18 months, Canada’s new gov­ern­ment has fo­cused on build­ing our strength at home, di­ver­si­fy­ing our part­ner­ships abroad, and strik­ing a fair deal with the United States.

Our ob­jec­tives in our trade ne­go­ti­a­tions have been to:

Preserve tar­iff-free ac­cess to the U.S. for the vast ma­jor­ity of Canadian busi­ness;

Provide greater sta­bil­ity to our trade re­la­tion­ship;

Significantly re­duce U.S. tar­iffs on our key strate­gic in­dus­tries, so that Canadian busi­nesses in these sec­tors would have the best ac­cess of any in the world;

Protect our small and medium-sized busi­nesses — the lifeblood of our econ­omy — in­clud­ing by re­mov­ing the im­mi­nent threat of new tar­iffs; and

Maintain our flex­i­bil­ity, in­de­pen­dence, and sov­er­eignty so we can keep build­ing the Canada we want.

We have recog­nised from the be­gin­ning that America has changed, and that we will not re­turn to our old re­la­tion­ship. Our gov­ern­ment un­der­stood, be­fore many, that America is al­ter­ing all its trade re­la­tion­ships. Putting tar­iffs on its clos­est al­lies and charg­ing for ac­cess to its vast mar­ket.

We have worked in that con­text. To strike a fair deal that would pro­vide the best ac­cess to the U.S. mar­ket and greater cer­tainty to Canadian busi­nesses and work­ers. Throughout, our goal has been to se­cure the best deal for Canadians, never a deal at any price or on any dead­line.

In re­cent weeks, we made im­por­tant progress to­ward im­prov­ing Canada’s po­si­tion as hav­ing the best deal in the world with the U.S.

However, that progress has not been enough to meet our ob­jec­tives for Canadians. As a re­sult, this evening, I have de­cided to sus­pend trade ne­go­ti­a­tions with the U.S. and have di­rected Canada’s ne­go­tia­tors to re­turn to Ottawa. They have worked hard, in good faith, to de­fend the in­ter­ests of Canadians through­out these ne­go­ti­a­tions up un­til the very last minute. However, last-minute changes in the U.S. pro­posed terms were un­fair, un­eco­nomic, and called into ques­tion the re­li­a­bil­ity of any deal.

At mid­night tonight, the U.S. in­tends to im­pose a 50% tar­iff on roughly $28 billion of Canadian goods. Canada will match those tar­iffs dol­lar for dol­lar to pro­tect our work­ers and busi­nesses.

In the com­ing days, the gov­ern­ment will in­tro­duce ad­di­tional mea­sures to sup­port Canadian work­ers and busi­nesses, build­ing on the nearly $25 billion in sup­port pro­vided over the past 18 months.

These ac­tions com­ple­ment Canada’s core eco­nomic strat­egy. From day one, we have been fo­cused on build­ing our strength at home and di­ver­si­fy­ing our part­ner­ships abroad.

That strat­egy is work­ing. We are ad­vanc­ing nearly $500 billion in ma­jor in­fra­struc­ture pro­jects. In par­al­lel, we are un­lock­ing new ex­port mar­kets for Canadian busi­nesses. Our ex­ist­ing free trade deals al­ready pro­vide Canada with pref­er­en­tial ac­cess to 1.5 billion con­sumers, and we are on track to dou­ble that mar­ket ac­cess by the end of this year.

Canadian eco­nomic growth is ac­cel­er­at­ing, and we are on course to have the sec­ond-fastest growth in the G7 over the next two years. Our econ­omy is cre­at­ing jobs at four times the rate of the United States. Our ex­ports to non-U.S. mar­kets are on track to dou­ble over the next decade. Foreign di­rect in­vest­ment in Canada is at its high­est level in two decades, run­ning at twice the rate of our near­est G7 com­peti­tor. Canada now ranks as the most at­trac­tive coun­try in the world for in­fra­struc­ture in­vest­ment.

Canada has what the world wants. And we will not al­low any na­tion to de­ter­mine our fu­ture. We will set our own course to keep build­ing Canada strong for all.”

The other day, I saw a vi­ral tweet say­ing that peo­ple talk­ing about how LLMs are caus­ing slow, bloated, code are go­ing to eat crow once they re-write every­thing in su­per-op­ti­mized as­sem­bly. We’re not quite at the point where we want to write every­thing in as­sem­bly, but some vari­ant of what Nolan Lawson said about test­ing, you can choose how many bugs you want now, which I less elo­quently noted here, is be­com­ing more true for per­for­mance.

In re­sponse to a com­ment in my last post that the cost of for­merly spe­cial­ized per­for­mance work has dropped by many or­ders of mag­ni­tude and per­for­mance work that used to re­quire a per­son or team that had a rare set of skills can be done by any­one who can type a few sen­tences1, which means that you can do all sorts of op­ti­miza­tions that used to be too ex­pen­sive to be worth­while for all but the largest scale or most lu­cra­tive pro­jects, Marc Brooker re­sponded with

Completely agree with your clos­ing point. Dynamic cus­tom soft­ware, fit­ted to a par­tic­u­lar work­load rather than a class of work­loads, seems like a very likely out­come. (Which comes with all kinds of fun risks and op­por­tu­ni­ties of its own). Kind of re­minds me of FFTW. And a ton of weird old demoscene tech­niques which were all about be­ing su­per fast and small on a very par­tic­u­lar prob­lem (and of­ten very par­tic­u­lar hard­ware). For ex­am­ple, I re­mem­ber a demo that re-used its code as tex­tures to get great cache lo­cal­ity.

Completely agree with your clos­ing point. Dynamic cus­tom soft­ware, fit­ted to a par­tic­u­lar work­load rather than a class of work­loads, seems like a very likely out­come. (Which comes with all kinds of fun risks and op­por­tu­ni­ties of its own). Kind of re­minds me of FFTW. And a ton of weird old demoscene tech­niques which were all about be­ing su­per fast and small on a very par­tic­u­lar prob­lem (and of­ten very par­tic­u­lar hard­ware). For ex­am­ple, I re­mem­ber a demo that re-used its code as tex­tures to get great cache lo­cal­ity.

And Michael Malis has noted

There’s been a meme cir­cu­lat­ing about how AI does­n’t help be­cause “code was never the hard part.” I think that’s true in some do­mains, but in oth­ers, writ­ing the code ab­solutely was the hard part. JIT com­pil­ers are a great ex­am­ple of that. For many pieces of soft­ware, a JIT com­piler would help a lot with speed­ing up the code. The rar­ity of JIT com­pil­ers makes me be­lieve that im­ple­ment­ing a JIT com­piler his­tor­i­cally was too dif­fi­cult for it to be worth­while. LLMs have low­ered the bar­rier to en­try and made it much eas­ier to write a JIT com­piler. This is the the­sis be­hind pgrust. Databases his­tor­i­cally were the hard­est piece of soft­ware to build and were lim­ited be­cause of that. Now, with AI, we can be more am­bi­tious about the type of soft­ware we build.

There’s been a meme cir­cu­lat­ing about how AI does­n’t help be­cause “code was never the hard part.” I think that’s true in some do­mains, but in oth­ers, writ­ing the code ab­solutely was the hard part. JIT com­pil­ers are a great ex­am­ple of that. For many pieces of soft­ware, a JIT com­piler would help a lot with speed­ing up the code. The rar­ity of JIT com­pil­ers makes me be­lieve that im­ple­ment­ing a JIT com­piler his­tor­i­cally was too dif­fi­cult for it to be worth­while. LLMs have low­ered the bar­rier to en­try and made it much eas­ier to write a JIT com­piler. This is the the­sis be­hind pgrust. Databases his­tor­i­cally were the hard­est piece of soft­ware to build and were lim­ited be­cause of that. Now, with AI, we can be more am­bi­tious about the type of soft­ware we build.

Optimizing for a class of work­load

Let’s try this out with FRE, the regex en­gine we built in the last post. Recall that it was cre­ated by hav­ing an agent loop for a month on im­prov­ing regex en­gine per­for­mance with ac­cess to the re­bar regex bench­mark suite. This re­sulted in FRE be­ing heav­ily over­fit to re­bar un­til we warned our agent that we had a hold­out bench­mark, which caused the agent to gen­er­al­ize the op­ti­miza­tions enough that per­for­mance was ok-ish on our hold­out. There’s no par­tic­u­lar rea­son to use a “software fac­tory” regex en­gine that does­n’t beat a well-tested regex en­gine on hold­out bench­marks, but one no­table thing about FRE was that the na­tive AOT com­piled ver­sion did quite well at longer searches. We noted that, it stands to rea­son that one could run the na­tive code com­piler in an­other thread while rip­grep was run­ning its nor­mal matcher and then cut over to the na­tive code when it fin­ished com­pil­ing and gen­er­ally get bet­ter per­for­mance. Of course this will gen­er­ally re­sult in worse per­for­mance for short queries as we lose a thread to com­pi­la­tion, but I care a lot more about how long rip­grep takes when it runs for many sec­onds or min­utes than when it runs for a few sec­onds, so I’m ok with that trade­off.

In the same way we could build a regex en­gine in a few min­utes of hu­man time, we can also just try this ex­per­i­ment in a few min­utes of hu­man time. I typed a few sen­tences and an agent went and did the work to al­low this to hap­pen (which would be a de­cent chunk of code surgery for a hu­man) and it ran the bench­mark on ac­tual rip­grep queries that come from my codex his­tory. For longer queries, we see a 2x-4x per­for­mance im­prove­ment here for a few very sim­ple queries. But most queries are more com­plex, and when we run on rep­re­sen­ta­tive hold­out queries, for queries where AOT should be en­abled2, we get about a 7% speedup. Not an earth shat­ter­ing re­sult, but also not a bad out­come for spend­ing a few min­utes typ­ing to codex (and it’s still do­ing more op­ti­miza­tion and will pre­sum­ably speed things up fur­ther).

Build an in­dex?

This is ar­guably a silly thing to do, since if we’re re­peat­edly search­ing for text on a com­puter, the ob­vi­ous thing to do to speed that up is­n’t to write a na­tive code com­piler for regex match­ing, it’s to cre­ate an in­dex. But the point here is just that this kind of tech­ni­cal work, which used to take a fair amount of time and ex­per­tise, can just be done triv­ially now. And if we wanted to build a text in­dex, it just so hap­pens that I worked on BitFunnel, the Bing search in­dex that was spe­cial­ized for con­stant/​fast text in­ges­tion that won Best Paper Award at SIGIR, so I can think of a few ex­per­i­ments to try if we’re go­ing to build a fast lo­cal in­dex of our en­tire ma­chine (the pro­jects I’ve seen seem to be in­tended to in­dex your code di­rec­to­ries, but what re­ally kills my ma­chine per­for­mance is when codex de­cides to run rip­grep against huge tem­po­rary di­rec­to­ries with a ton of gen­er­ated files and then ex­pands to look­ing at my whole ma­chine when it misses, so I’d want an in­dex of my en­tire disk and not just of the code for some pro­jects).

If I were work­ing at an AI lab and had ac­cess to things like SOTA mod­els run­ning on Cerebras chips or other ac­cel­er­a­tors that greatly in­crease tok/​s and there­fore load/​de­mand for search, I might ac­tu­ally sur­vey the ex­ist­ing in­dex­ers to see if they’re fast enough or if I’d want to build some­thing cus­tom my­self. While the open source ver­sion of BitFunnel “only” con­tains a byte­code in­ter­preter and one JIT, the Bing ver­sion con­tains mul­ti­ple JIT com­pil­ers. A pro­ject that did that level of op­ti­miza­tion used to be a ma­jor un­der­tak­ing, but “I could do that in a week­end” is now ac­tu­ally true for some of these kinds of pro­jects. With my lowly $200/mo ac­count, I think a some­what faster rip­grep plus any off-the-shelf in­dex is fine, so maybe this fast-in­gest­ing whole-ma­chine in­dex pro­ject can be left as an “exercise for the reader (who works at an AI lab)”.

Optimizations are cheap

The dras­tic re­duc­tion in the cost of op­ti­miza­tions has been true go­ing back to November 2025 and maybe even some­what be­fore then with pub­lic mod­els (and I’m sure be­fore that still with what folks at AI labs had ac­cess to). For an ex­am­ple from the GPT-5.1 or 5.2 days, with no knowl­edge of game AIs, I tried build­ing an Azul AI. This ended up be­ing the strongest AI in the world for the game by a pretty large mar­gin. From read­ing the the­sis that de­scribes the 2nd strongest AI, I think my AI is prob­a­bly a bit bet­ter on the “AI” side of things, but the main place it wins is on op­ti­miza­tion de­spite spend­ing what looks like maybe two or­ders of mag­ni­tude less time (estimated by read­ing the the­sis and see­ing the process and com­par­i­son to my process) and also mostly work­ing on my lap­top vs. hav­ing a clus­ter of ma­chines to use (which means much less band­width to run ex­per­i­ments with, do pa­ra­me­ter tun­ing, etc.). For ex­am­ple, that other AI is sin­gle-threaded and my AI is multi-threaded. Since I have a na­tive code ver­sion as well as a heinous wasm shared mem­ory + javascript ver­sion, and two dif­fer­ent search ar­chi­tec­tures for two dif­fer­ent ver­sions, which “require” com­pletely dif­fer­ent multi-thread­ing al­go­rithms (minimax for a very small and fast net and MCTS for a larger net), this would’ve been a fairly large un­der­tak­ing if done by hand. And, be­cause I let an LLM pick the multi-thread­ing al­go­rithm based on its own (incorrect) rea­son­ing a cou­ple times be­fore spend­ing 30 min­utes read­ing about multi-thread­ing al­go­rithms for game AIs my­self, I ended up re-writ­ing (having codex re-write) the multi-thread­ing al­go­rithm mul­ti­ple times.

There’s a bunch of stan­dard stuff it makes sense to do to de­bug and ver­ify a mul­ti­thread­ing al­go­rithm for some­thing like this, like im­ple­ment­ing re­play from de­bug logs that can re­pro­duce bugs de­spite the al­go­rithm be­ing non­de­ter­mistic. Doing that alone would’ve prob­a­bly been days to a week of work had I done it by hand, but it’s ex­actly the kind of thing an agent can triv­ially do in a loop (just have it try to re­play logs and in­sert log­ging for non-de­ter­min­ism every time you don’t get a per­fect re­play). A lot of the te­dium it used to take to get a tricky op­ti­miza­tion like this work­ing is gone.

This also ap­plies to a lot of other tricky op­ti­miza­tions. From hav­ing writ­ten CPU mi­croc­ode, done CPU ver­i­fi­ca­tion, worked on op­ti­miz­ing a search en­gine in­dex, etc., I have a lot of ex­pe­ri­ence look­ing at op­ti­miza­tions and think­ing “hmm, this would in­crease per­for­mance by 2%, but it’s go­ing to take N per­son-days to ver­ify that this tricky op­ti­miza­tion works” and mak­ing a call to go ahead or not based on whether or not it’s worth the time to get the op­ti­miza­tion work­ing. Now that this N has dropped by a tremen­dous fac­tor (variable but, in terms of hu­man time, fre­quently 1000x / 10000x / 1000000x, prob­a­bly more like 1000x on dol­lar cost if you com­pare to­ken costs at me­tered rates vs. the Bing en­gi­neer who wrote the com­pil­ers at JITs that the search in­dex used), the num­ber of these kinds of op­ti­miza­tions it makes sense to do goes way up. The same goes for op­ti­miza­tions that you aren’t sure will work out. I used to some­times look at an op­ti­miza­tion that I was­n’t sure would speed things up and think “this will take M hours to im­ple­ment to the point where we have a good enough mea­sure­ment to guess at the per­for­mance im­pact”. Many more of those op­ti­miza­tions make sense to try out now.

Going back to the game AI case, at least for the AI I tried, it seems like you gain about 100 Elo for every dou­bling in speed (more than in chess, I sus­pect be­cause draws are very rare). Just adding mul­ti­thread­ing alone is enough to wipe the floor with an oth­er­wise com­pa­ra­ble AI on a large ma­chine. If you stack in 10 – 20 more op­ti­miza­tions that seem too an­noy­ing for most peo­ple to do by hand, the dif­fer­ence in strength is tremen­dous and it’s not re­ally rea­son­able to try to keep up with a hand-writ­ten AI3.

The game AI case is a lit­tle more an­noy­ing than for most soft­ware be­cause a lot of the op­ti­miza­tions you want to do ac­tu­ally change the re­sult and there is­n’t a cheap, triv­ial, way to tell if the speed in­crease + the change in re­sult gives a bet­ter or worse ac­tual re­sult in prac­tice. And, as we noted be­fore, cur­rent pub­licly avail­able SOTA mod­els are pretty bad at ex­per­i­men­tal de­sign, so I had to set up the frame­work they used to de­ter­mine if an op­ti­miza­tion is good, but once that was in place, it’s like any other op­ti­miza­tion prob­lem. I guess peo­ple work­ing on LLM op­ti­miza­tions also have to deal with this class of prob­lem but most op­ti­miza­tion prob­lems are a lot more straight­for­ward.

To pick an­other ex­am­ple, as part of prepar­ing for per­for­mance in­ter­views, Jamie Brandon tried Anthropic’s now pub­lic per­for­mance take­home. After try­ing it, he had Claude pick up where he left off and it got a much bet­ter re­sult. When he looked at what Claude did that he did­n’t, he said a lot of the op­ti­miza­tions were things that oc­curred to him but he had­n’t got­ten to yet, and “[o]thers were just crazy shit that I would never try un­less I was work­ing on this for weeks”4. He’s a rea­son­able per­for­mance en­gi­neer and he got an of­fer for the per­for­mance job he wanted, but on a well-de­fined op­ti­miza­tion prob­lem, he does­n’t stand a chance against a de­cent model (I haven’t tried the prob­lem my­self, but I sus­pect I also would­n’t stand a chance given re­motely com­pa­ra­ble time con­trols).

Workload-specific op­ti­miza­tion

Coming back to this part of Marc Brooker’s com­ment:

Dynamic cus­tom soft­ware, fit­ted to a par­tic­u­lar work­load rather than a class of work­loads, seems like a very likely out­come.

Dynamic cus­tom soft­ware, fit­ted to a par­tic­u­lar work­load rather than a class of work­loads, seems like a very likely out­come.

This seems pretty in­evitable. In an­other re­sponse to my post, Michael Malis of pgrust said some­thing sim­i­lar:

[discussion of pgrust op­ti­miza­tions] … I think it’s easy enough to cre­ate these op­ti­miza­tions that we could look at a cus­tomers work­load and add them as needed

[discussion of pgrust op­ti­miza­tions] … I think it’s easy enough to cre­ate these op­ti­miza­tions that we could look at a cus­tomers work­load and add them as needed

Without hav­ing any kind of frame­work or setup, right be­fore I started writ­ing this post, I had an agent do work­load-spe­cific op­ti­miza­tion for my rip­grep queries (not the na­tive code com­piler switch, just the op­ti­miza­tions to the gen­eral FRE en­gine based on a set of bench­marks), which took about 2 min­utes for me to launch. The op­ti­miza­tions run on a set of queries, and then there’s a later hold­out set of queries to run against. That’s still run­ning, but the ini­tial re­sults seem promis­ing. After one pass of op­ti­miza­tion, the work­load op­ti­mized ver­sion is 2% faster than stan­dard rip­grep on the hold­out and it’s still get­ting faster. 2% is­n’t a big deal for my lo­cal rip­grep us­age, but con­sid­er­ing that this took min­utes of time and the op­ti­miza­tions done here got started when I started typ­ing this point and are still im­prov­ing, I’d take a 2% win here (note that this is­n’t com­bined with the na­tive code com­piler, which would give a larger over­all win if com­bined prop­erly). And re­call that this is lever­ag­ing the FRE regex en­gine5, which was sub­stan­tially slower than the Rust regex en­gine on hold­out bench­marks and was stuck with slow im­prove­ment on hold­outs be­cause with me know­ing noth­ing about regex work­loads and SOTA LLMs not be­ing good enough at ex­per­i­men­tal de­sign to do un­guided open-ended self-im­prov­ing loops, we did­n’t have a good way to im­prove per­for­mance on our hold­outs. But if what I care about is per­for­mance on my own work­loads, I have plenty of data and am gen­er­at­ing more all the time. As Marc Brooker noted above, we do have to be care­ful about over­fit­ting if there’s a regime change that’s not in the old data, etc., but we’re still in a bet­ter sit­u­a­tion than we were be­fore.

In the more gen­eral case, if you’re some­one like Marc Brooker at Amazon or Michael Malis work­ing on pgrust, it makes sense to not just do this as a one-off, but to work with cus­tomers to pi­lot a pro­gram that uses their data to op­ti­mize things for them and then fig­ure out how to scale it out for cus­tomers in gen­eral. I’m not work­ing at a com­pany where that’s the best use of my time6, but it’s pretty wild that you can see that this is com­ing for larger com­pa­nies with more scale, and given that it only takes min­utes of my time to run these ex­per­i­ments for my per­sonal work­flows, it’s pretty rea­son­able to mess with this kind of thing on per­sonal pro­jects.

Thanks to Jamie Brandon, Michael Malis, and Max Bittker for com­ments/​cor­rec­tions/​dis­cus­sion.

P.S. As I’ve noted in the last cou­ple posts, with cod­ing agents, the time it takes to run an ex­per­i­ment and see enough of a re­sult to sat­isfy my cu­rios­ity has gone way done while the time it takes to make a re­sult re­ally rig­or­ous has­n’t changed or has gone up, so writ­ing things up the way I used to would mean run­ning very few ex­per­i­ments rel­a­tive to the band­width I have for them. As a re­sult, I’ve just been run­ning these ex­per­i­ments and shar­ing the re­sult with a cou­ple of friends. As an ex­per­i­ment, I’m try­ing to write these up in a very quick and non-rig­or­ous way in­stead of years of these ex­per­i­ments only be­ing known to a few friends. Like the last post, I set a goal of writ­ing this post and do­ing all the clean-up in half an hour and did­n’t time it but am pretty sure I missed that by a bit.

Even do­ing this, the time it takes to write these up is long enough that I’m falling be­hind on shar­ing re­cent re­sults, but I’m not in­clined to switch to LLM-written posts (yet?), and I don’t think I can re­al­is­ti­cally get the time to clean up the data and write a post like this down enough to turn a post around in less than half an hour. Just on the length of this post, typ­ing this up should be some­thing like 20 – 30 min­utes in­clud­ing time to pause and think about what I’m writ­ing, and then when I look at the data some­times some­thing will look wrong enough that I need to look into it more closely to see if there’s an is­sue that needs to be fixed (this hap­pened mul­ti­ple times here, and I would ex­pect that, be­cause I did­n’t spend much more time, there are other data is­sues that I don’t know about).

Anyway, if you have opin­ions on these quick (and surely more wrong) writeup, let me know what you think (X Bsky Mastodon)!

Appendix: There’s no rea­son for soft­ware to be slow any­more

I’ve been on the record for a long time as strongly dis­agree­ing with the gen­eral sen­ti­ment that the de­vel­op­ers of X are bad and should feel bad for writ­ing slow code be­cause there are a lot of dif­fer­ent kinds of pro­gram­ming ex­per­tise and not only is it not the case that most pro­gram­mers don’t have per­for­mance ex­per­tise, it prob­a­bly does­n’t even make sense for them to de­vel­op­ment (from the stand­point of what the busi­ness cares about, what the em­ploy­ment mar­ket looks like, etc.), so of course most pro­jects will have very poor per­for­mance com­pared to what a per­for­mance ex­pert can do.

For the ex­am­ple above, Jamie Brandon got an of­fer from Anthropic and you prob­a­bly can’t af­ford him or some­one like him un­less you’re OpenAI, but you can af­ford to use a cod­ing agent that can beat him on a bounded op­ti­miza­tion prob­lem. The agent does­n’t have the judge­ment he has and will do worse on an open-ended prob­lem (recall that when we tried build­ing an op­ti­mized regex en­gine and just told it to not over­fit, it was more than an or­der of mag­ni­tude worse than the best regex en­gines on our hold­out bench­marks, but also re­call that af­ter telling the agent there was a hold­out it was do­ing poorly on, it sped up regex en­gine per­for­mance enough to gen­er­ally match 2nd tier regex en­gines in terms of per­for­mance, which is still ex­tremely good com­pared to the gen­eral level of per­for­mance op­ti­miza­tion in most code to­day), but that’s plenty good to achieve rea­son­able per­for­mance on all sorts of prob­lems. This post has gen­er­ally dis­cussed back­end per­for­mance is­sues, but agents don’t seem worse at front-end per­for­mance if you want to drive down a set of met­rics like LCP, INP, etc.

Appendix: How is codex run­ning rip­grep?

Here’s some in­for­ma­tion about the dis­tri­b­u­tion of riprep queries on my ma­chine. I make no claims that this is at all rep­re­sen­ta­tive of what’s hap­pen­ing any­where else. The pat­tern dis­tri­b­u­tion of the length of the pat­tern that’s searched has a lot more long pat­terns that I would’ve ex­pected. The p50 is 55 uni­code code points (I’ll just call these char­ac­ters for sim­plic­ity), which is al­ready longer than things I grep for by hand, and the p90 is 119!

We can also look at the num­ber of al­ter­na­tion arms in regexes, which are once again much more com­plex than what I do by hand.

Another view is to look at how these are cor­re­lated. Do we get more al­ter­na­tion arms in the regexes as the regexes get longer? Yes.

What are these re­ally long regexes, any­way? If we look at them, most of the longest are long al­ter­na­tions over func­tion or tests names, such as the fol­low­ing regex, which ap­pears to be re­lated to FRE de­vel­op­ment.

fn (hot_byte_compiler_is_generic_only_and_anonymous_count_uses_auto_count| one_­pat­tern_­coun­t_s­pan­s_us­es_the_re­tained_­com­plete_s­pan_ses­sion| for­mal_­com­pact_s­tate_byte_vis­i­tors_­co­ex­ist_with­_­na­tive_­count| fixed_bound­ary_record_vis­it_­match­es_­line_rel­a­tive_ref­er­ence_and_is_atomic| un­bound­ed_lan­guages_refuse_fi­nite_ex­trac­tion_be­fore_al­lo­ca­tion| for­mal_s­in­gle_raw_s­pan_sweep­_pre­flight| as­sert_ex­ac­t_­fix­ture_us­es_­for­mal_large_­con­tin­u­a­tion_sweep| url_on­ly_­com­pile_i­den­ti­ty_bind­s_lan­guage_and_own­er_­mode| url_on­ly_­com­pile_ex­ac­t_lim­it­s_and_run­time_re­fusal­s_­close| url_on­ly_­com­pile_­post_­plan_al­lo­ca­tion_­fault­s_­close| url_on­ly_own­er_dis­crim­i­na­tor_is_sta­ble_and_precharged| url_on­ly_­com­pile_own­er_is_s­trat­e­gy_and_­op­er­a­tion_s­coped| for­mal_re­bar_url_own­er_is_­com­pile_on­ly_and_­match­es_o­r­a­cle| for­mal_re­bar_url_ex­ac­t_­fix­ture_us­es_cer­ti­fied_ex­e­cu­tion| for­mal_­fixed_schema_­ma­te­ri­al­iza­tion_­match­es_both­_record_o­r­a­cles_and_­con­trols| for­mal_s­in­gle_­coun­t_s­e­lect­s_­com­pact_s­tate_byte_­com­plete_bound­_vis­i­tors| au­then­ti­cat­ed_bound­_­line_­to­tal_lf_free_­do­main_op­por­tu­ni­ty_ex­ceed­s_­five_per­cent| pre­pared_ab­solute_onepass_­fus­es_s­lot­s_and_p­re­serves_pre_­source_­fall­back| au­then­ti­cat­ed_­word_bound­ary_russ­ian_­com­pact_low­er­ing_pub­lic_­ca­nary| or­dered_n­fa_x86_ep­silon_edges_by­pass_the_as­ser­tion_­call| or­dered_n­fa_aarch64_ep­silon_edges_by­pass_the_as­ser­tion_­call| or­dered_edge_dis­patch_v2_is_­tar­get_neu­tral_de­ter­min­is­tic_and_re­lo­ca­tion_free| or­dered_edge_dis­patch_v2_­copies_­canon­i­cal_ta­bles_and_­cap_­fall­s_back­_­to_v1| or­dered_n­fa_v3_­com­pos­es_ter­mi­nal_range_and_dis­patch_with­out_­da­ta_re­lo­ca­tions| or­dered_n­fa_x86_ter­mi­nal_range_emit­s_au­then­ti­cat­ed_re­verse_s­can| or­dered_n­fa_aarch64_ter­mi­nal_range_emit­s_au­then­ti­cat­ed_re­verse_s­can| or­dered_n­fa_x86_bound­ary_as­ser­tion_­cache_is_lazy_and_bound­ary_s­coped| or­dered_n­fa_aarch64_­caches_re­peat­ed_as­ser­tion­s_once_per_bound­ary| bound­ary_as­ser­tion_­cache_re­quires_­dense_ex­ac­t_kind_reuse| bound­ary_as­ser­tion_­cache_s­e­lec­tion_is_­com­pil­er_on­ly_and_de­ter­min­is­tic)

But some are funny nu­mer­i­cal con­struc­tions, such as

:(13[0 – 9]|14[0 – 9]|15[0 – 9]|16[0 – 9]|17[0 – 9]|18[0 – 9]|19[0 – 9]|20[0 – 9]|21[0 – 9]|22[0 – 9]|23[0 – 9]|24[0 – 9]|25[0 – 9]|26[0 – 9]|27[0 – 9]|28[0 – 9]|29[0 – 9]|30[0 – 9]|31[0 – 9]|32[0 – 9]|33[0 – 9]|34[0 – 9]|35[0 – 9]|36[0 – 9]|37[0 – 9]|38[0 – 9]|39[0 – 9]|40[0 – 9]|41[0 – 9]|42[0 – 9]|43[0 – 9]|44[0 – 9]|45[0 – 9]|46[0 – 9]|47[0 – 9]|48[0 – 9]|49[0 – 9]|50[0 – 9]|51[0 – 9]|52[0 – 9]|53[0 – 9]|54[0 – 9]|55[0 – 9]|56[0 – 9]|57[0 – 9]|58[0 – 9]|59[0 – 9]|60[0 – 9]|61[0 – 9]|62[0 – 9]|63[0 – 9]|64[0 – 9]|65[0 – 9]|66[0 – 9]|67[0 – 9]|68[0 – 9]|69[0 – 9]|70[0 – 9]|71[0 – 9]|72[0 – 9]|73[0 – 9]|74[0 – 9]|75[0 – 9]|76[0 – 9]|77[0 – 9]|78[0 – 9]|79[0 – 9]|80[0 – 9]|81[0 – 9]|82[0 – 9]|83[0 – 9]|84[0 – 9]|85[0 – 9]|86[0 – 9]|87[0 – 9]|88[0 – 9]|89[0 – 9]|90[0 – 9]|91[0 – 9]|92[0 – 9]|93[0 – 9]|94[0 – 9]|95[0 – 9]|96[0 – 9]|97[0 – 9]|98[0 – 9]|99[0 – 9])[0 – 9]:

This is equiv­a­lent to :(?:1[3 – 9]|[2 – 9][0 – 9])[0 – 9]{2}: (which, if run through rip­grep on the orig­i­nal in­put, has ap­prox­i­mately the same per­for­mance). The en­tire pipeline for that was

cargo clippy … | rg ‘crates/fre-aot-regex/src/module.rs:’ | rg NUMBER_REGEX | head -250

which might be an odd thing for a hu­man to do, but agents seem to do this kind of thing all the time.

On an­other topic, if we look at how long rip­grep queries took, there are quite a few slow queries, e.g., p99 is al­most 1 minute! And p999 is al­most 10 min­utes! And the max­i­mum query over this time pe­riod (around a month on one lap­top; queries and dis­tri­b­u­tions seem likely to be dif­fer­ent on the AWS hosts I run agents on, etc., but I haven’t checked) is ap­proach­ing 2 hours!

In terms of com­mand line op­tions, we see the fol­low­ing. Perhaps un­sur­pris­ingly, codex of­ten wants line num­bers and, for what­ever rea­son, it very oc­ca­sion­ally uses PCRE2 regexes.

I won’t add plots or ta­bles for these, but an­other thing to note is that there’s fairly low lo­cal­ity for what pat­terns are searched for (about 94% of pat­terns only oc­curred once), which makes some sense given how long a lot of the queries were. However, there’s fairly high lo­cal­ity in what files get searched and a file that got searched is rel­a­tively likely to get searched again soon, in­di­cat­ing that (for small enough files), they’re likely to be searched in mem­ory.

Also, 99% of queries were regex queries (1% were non-regex string searches) and 99.9% of search queries were ASCII only, but in terms of files searched, ap­prox­i­mately 45% were ASCII only and 55% con­tained Unicode, a higher per­cent­age than I would’ve guessed for Unicode.

On a draft of the last post, Peter Geoghegan noted

It’s also pos­si­ble for a regex im­ple­men­ta­tion to be faster by sup­port­ing fewer fea­tures. Some im­ple­men­ta­tions don’t sup­port back ref­er­ences, etc.

It’s also pos­si­ble for a regex im­ple­men­ta­tion to be faster by sup­port­ing fewer fea­tures. Some im­ple­men­ta­tions don’t sup­port back ref­er­ences, etc.

which is also true here. The work­load-spe­cific op­ti­miza­tions done here were fairly su­per­fi­cial be­cause I just gave codex some short in­struc­tions and let it do what­ever it wanted (which is, in gen­eral, not the most ef­fec­tive use of codex), but with a more de­tailed plan, more fo­cused op­ti­miza­tions sup­port­ing the com­mon use cases for my queries could be ex­pected to yield larger gains.

though, as we dis­cussed in that post as well as be­fore, the bench­mark­ing and ex­per­i­men­tal de­sign skills of SOTA mod­els aren’t good enough to do this in the gen­eral case with­out a hu­man (or a skill) set­ting up the bench­mark­ing en­vi­ron­ment for the agent. [return]

we can see from our old bench­marks that, even with time to run the com­piler, there are a lot of cases where the na­tive code com­piled ver­sion is slower than the Rust regex crate. If we look at why this is, these tend to be more com­plex queries where the Rust regex crate has some al­go­rith­mic op­ti­miza­tion and the FRE na­tive code com­piler is falling back to some­thing naive (the agent that cre­ated FRE spent much less time on the na­tive code com­piler than it did on the “normal” regex en­gine). [return]

I have no doubt that a hand-writ­ten AI by some­one who has real AI ex­per­tise, e.g., by some­one who’s writ­ten one of the top Go and chess en­gines in the world, could beat my AI on the strength of the “AI” side of things be­ing bet­ter than what you get when some­one who knows noth­ing about AI (me) cre­ates an AI, but if the lev­els of ex­per­tise are re­motely sim­i­lar, the LLM-written ver­sion is go­ing to dom­i­nate for any given amount of time spent. [return]

it’s ar­guably un­fair to com­pare the re­sult of an agent pick­ing up where he left off, since his work is a start­ing point which might let an agent do much bet­ter than it would do on its own, so I tried giv­ing the fresh task to an agent and it got a very sim­i­lar score to what he got when an agent re-used his work (and a quick check by an­other agent did­n’t find ev­i­dence of cheat­ing). [return]

The per­for­mance prob­a­bly would’ve been bet­ter if I had an agent just mod­ify a rip­grep fork di­rectly, but I was cu­ri­ous if this could also solve the FRE over­fit­ting prob­lem with re­spect to my queries. [return]

a while back, I re­duced the size of page in our signup flow from 50 MB to 5 MB and a rev­enue A/B test seemed to in­di­cate that this in­creased rev­enue by about 0.5%. In gen­eral, I’m a huge fan of do­ing the sim­ple and easy wins first, such as this, and there are prob­a­bly a lot of higher ROI wins than we’d get out of build­ing cus­tom com­pil­ers or do­ing other highly spe­cial­ized tech­ni­cal work here. [return]

I want to pre­sent a pro­ject that I’ve been work­ing on for the past 4 months: an al­ter­na­tive Rust LSP im­ple­men­ta­tion that is built with a fo­cus on low mem­ory us­age.

It has two main fea­tures:

It can use very lit­tle mem­ory (target <100mb for rea­son­able pro­jects). There are caveats, these are de­scribed be­low.

It al­lows im­me­di­ate in­dex­ing af­ter restart: if your pro­ject was in­dexed, restart­ing the ed­i­tor will not re­quire re-in­dex­ing.

Note: through­out this video, the used RAM re­mained un­der 100mb

These fea­tures make Rust Glancer suit­able for the older com­put­ers: I have tested it on my old MacBook Pro M1 2020 with 8GB RAM, and it was pretty good.

As you can imag­ine, 4 months is not a lot of time for a pro­ject as big as a Rust LSP. Rust Glancer is not a com­plete LSP yet, it has a lot of miss­ing func­tion­al­ity, it has some known bugs, and it has a lot of things I want to im­prove.

At the same time, it is al­ready pretty ca­pa­ble: it has a full in­dex­ing pipeline with type in­fer­ence and a trait solver (chalk), most of the “normal” Rust syn­tax is sup­ported, and most of the “normal” LSP ac­tions do work as well: goto de­f­i­n­i­tion, hover, in­lay hints, com­ple­tions, you name it.

If you are in­ter­ested, you can al­ready try it out: just in­stall the VS Code ex­ten­sion here, or, if you pre­fer, build and in­stall the vsix from the repos­i­tory.

The rest of the post con­tains the his­tory of the pro­ject: mo­ti­va­tion, LLM use, plans and roadmap. If you’re not in­ter­ested, you might want to check out the pro­ject doc­u­men­ta­tion in­stead.

Difference with rust-an­a­lyzer

There are sev­eral rea­sons why rust-an­a­lyzer con­sumes a lot of mem­ory:

Rust work­spaces gen­uinely have a lot of in­for­ma­tion that must be in­dexed: thou­sands of func­tions, struc­tures, traits, re­la­tion­ships be­tween these, func­tion bod­ies and state­ments in them, etc. Each of these needs to be an­a­lyzed and re­mem­bered, and you can’t re­ally cheat if you want to have things like “find all ref­er­ences to this struc­ture”.

rust-an­a­lyzer uses salsa as its data­base. It’s an in­cre­men­tal query-based data­base, which lazily com­putes all the data you need with­out hav­ing to ex­plic­itly “record” every­thing. It is a very cool ap­proach, but it’s in­her­ently tied to mem­ory, which makes it hard to move parts of data from mem­ory else­where.

rust-an­a­lyzer uses rowan for syn­tax tree rep­re­sen­ta­tion. The cool prop­erty here is that it al­lows par­tial in­val­i­da­tion: if only a part of the file changed, only the rel­e­vant bits have to be reparsed, which makes it faster than hav­ing to re-parse the whole file on each key­stroke. However, the tree-like rep­re­sen­ta­tion in­side of it can cause heavy mem­ory frag­men­ta­tion (meaning that the amount of RAM taken from the OS is higher than the amount of “actually used” RAM).

(1) is some­thing we have to live with (though there are a few op­ti­miza­tions we can do there which Rust Glancer does), but (2) and (3) are the con­se­quences of the rust-an­a­lyzer ar­chi­tec­ture. rust-an­a­lyzer chose them to make the LSP faster, and it does work for that pur­pose.

The idea I had when I started the pro­ject: what if we don’t try to make an in­cre­men­tal LSP? What if all we have is a frozen analy­sis re­sult that gets in­val­i­dated on save? It ob­vi­ously will not be as fast as rust-an­a­lyzer, but it will give us the prop­er­ties we seek:

analy­sis re­sults can be of­floaded to the filesys­tem and loaded to mem­ory only when they are ac­tu­ally needed.

saved analy­sis is reusable, and since it’s al­ready of­floaded to the filesys­tem, it can be reused af­ter the ed­i­tor restart.

This is the core idea of Rust Glancer.

It in­dexes the work­space once and pre­serves re­sults in the filesys­tem, and then when­ever queries need some­thing, they can load the re­quired in­for­ma­tion for the du­ra­tion of the query.

It does­n’t come for free though: frozen work­space analy­sis is slower than lazy in­cre­men­tal by de­f­i­n­i­tion, since load­ing and de­se­ri­al­iz­ing data from filesys­tem is slower than load­ing from mem­ory. To mit­i­gate that, Rust Glancer has to use some tricks: for ex­am­ple, when you type, it does­n’t per­form full blown analy­sis on each key­stroke, it in­stead at­tempts shal­low analy­sis of the cur­rent body and reuses the pre­vi­ous com­plete in­dex. This makes com­ple­tions rea­son­ably fast, but it also means that new items (imports, struc­tures, traits) are not “indexed” un­til you save the doc­u­ment. Which, hope­fully, should not be a prob­lem: you re­ally get used to it fast, and at least in my case it does not feel overly wrong af­ter a while. If that sounds scary, I sug­gest to just try it, it re­ally is not.

For peo­ple who rely on agen­tic work­flows, Rust Glancer is also op­ti­mized for large amount of out-of-ed­i­tor changes. I’m not sure why, but in rust-an­a­lyzer I’ve ob­served that when agents edit the code, in­lay hints can get out of place, and I had the same prob­lem in Rust Glancer ini­tially, but it was re­solved by im­ple­ment­ing a cus­tom file watcher and tweak­ing it some­what. The server also has lower pri­or­ity for out-of-ed­i­tor changes, so agen­tic changes do not cause rapid re-in­dex­ing.

Still, it’s im­por­tant to un­der­stand that Rust Glancer has some ben­e­fits, but also has some draw­backs (besides be­ing in­com­plete, ob­vi­ously) com­pared to rust-an­a­lyzer. Maybe I will man­age to solve some of them even­tu­ally, but it’s highly un­likely that Rust Glancer will ever be­come “just like rust-an­a­lyzer, but bet­ter”. I imag­ine that rust-an­a­lyzer will re­main the de­fault choice for pro­jects that care about com­plete­ness and key­stroke ac­cu­racy, while Rust Glancer will work for peo­ple with weaker ma­chines or peo­ple who are ready for some sac­ri­fices to re­duce RAM us­age.

How and why it hap­pened

I have been writ­ing Rust pro­fes­sion­ally for ~7 years, and since pretty early on I started ob­serv­ing how the com­piler and its tool­ing are de­vel­oped. I’ve made some con­tri­bu­tions to rustc, clippy, and rust-an­a­lyzer, and I’ve spent dozens of hours read­ing its source code just to teach my­self. So I was pretty much aware how big of a pro­ject a Rust LSP is.

At the same time, I have a love-hate re­la­tion­ship with rust-an­a­lyzer. It is ab­solutely beau­ti­ful ex­cept for two things: mem­ory us­age and ini­tial in­dex­ing (especially with build scripts / proc macros en­abled). These prob­lems seem to be brought up quite a lot, but in my case they are even more dras­tic: I have a rather stu­pid work­flow where I have two iden­ti­cal IDEs open on two dis­plays with a bunch of pro­jects in­side a work­space. So the mem­ory con­sump­tion is roughly 2N, and with my last set of the pro­jects I had to work on, rust an­a­lyzer was con­sum­ing 16GB of mem­ory that I, ugh, would pre­fer to have avail­able for other uses; not to men­tion that each time I opened VS Code, my PC fans would go brr be­cause of a ton of par­al­lel in­dex­ing jobs.

At some point I thought that I am fairly con­fi­dent in my Rust knowl­edge, so I prob­a­bly don’t need a full-blown LSP, and can use some­thing sim­pler and more mem­ory ef­fi­cient. I de­cided to try build­ing a “smart ctags for Rust”. I very ex­plic­itly did not want to build an al­ter­na­tive LSP, be­cause of how in­sane of a task it is. Little did I know…

The ini­tial progress was go­ing pretty smoothly: I made use of rust-an­a­lyz­er’s syn­tax li­brary, low­ered items to in­ter­nal rep­re­sen­ta­tions, then built de­f­i­n­i­tion maps and mod­ule struc­ture, got all the de­c­la­ra­tions in­dexed. It was so sur­pris­ingly straight­for­ward that I de­cided to do some prim­i­tive body low­er­ing. Then I de­cided to add very very sim­ple type prop­a­ga­tion. Then it turned out that naive type prop­a­ga­tion does­n’t give me much — but I al­ready had these nice in­lay hints, so I wanted more. Overall, I don’t care about com­plex cases and nightly fea­tures, right? (Right?…). So then came naive trait re­solv­ing via impl header match­ing. It’s quite ad­dic­tive, you get it.

The il­lu­sion, how­ever, broke when I de­cided that it is pretty rea­son­able to ex­pect the fol­low­ing code to be sup­ported as well:

fn mul_­by_two(vals: &[u8]) -> Vec<u8> { vals.iter().copied().map(|v| v * 2).collect() }

The code is pretty sim­ple, but in or­der to sup­port it we need:

Slice type sup­port

Closures / Fn traits

Trait solv­ing

Associated type pro­jec­tion

A bunch of nightly stuff

the last item is funny: I wanted to avoid nightly, but I some­how did­n’t think that std (or sys­root in gen­eral) breathes nightly. Welp.

So all in all, one fea­ture af­ter an­other, I slowly was get­ting from “smart ctags” to a “real LSP”. Probably, the three biggest mile­stones were:

Declarative macro ex­pan­sion (I hate de­clar­a­tive macros now). Thankfully, I was able to reuse most of rust-an­a­lyz­er’s in­fra­struc­ture for that.

Proper type in­fer­ence en­gine. It was a big “oh wow” mo­ment when I truly re­al­ized how type in­fer­ence works (in short: we “link” all re­lated type bind­ings in a big in­fer­ence table, and then we try to get ev­i­dence from all pos­si­ble places, where pro­vid­ing ev­i­dence can solve types for mul­ti­ple places). It was the mo­ment that prob­a­bly brought me the most joy dur­ing the work on this pro­ject so far.

Proper trait solv­ing en­gine. I ini­tially wrote “it’s highly un­likely that we will have a trait solver in this pro­ject”, but then I re­ally wanted to get the above­men­tioned it­er­a­tor ex­am­ple to work prop­erly. I re­sisted in­te­grat­ing trait solver for a while, try­ing to have naive hacks like naive trait impl match­ing + spe­cial­ized han­dlers for std traits, but it was get­ting more and more com­plex while work­ing pretty poorly. Then I gave up and in­te­grated Chalk, which turned out to be sig­nif­i­cantly sim­pler than the whole hi­er­ar­chy I have built. Making Chalk fast was an­other chal­lenge, though.

Somewhat sep­a­rately, prob­a­bly the thing I am most proud of (and the thing that made Rust Glancer pos­si­ble — had I not de­signed it early, the pro­ject would die very quickly) is a cool pro­fil­ing stack that can mea­sure per­for­mance, mem­ory us­age (both na­tively, track­ing ac­tual al­lo­cated ob­jects, and with je­mal­loc), pro­file data on de­mand, and com­pare LSP against rust-an­a­lyzer, as well as a set of bench­marks run­ning in CI. If you’re in­ter­ested, it’s par­tially cov­ered in the docs (1, 2), but I’ll work on a more de­tailed cov­er­age later.

Probably ~1.5 months ago I started us­ing Rust Glancer as my daily dri­ver in­stead of rust-an­a­lyzer. Now, I am happy with its state enough to pre­sent it to a larger au­di­ence.

LLM use

This pro­ject was built with heavy use of LLMs. It is not vibe coded, though. I am ver­i­fy­ing each pull re­quest to make sure that I am happy with the state of the code­base. If you need proofs, you can check the git his­tory: it has PRs with 10k+ lines of diff, but these are mul­ti­ple days apart de­spite the fact that I work on this pro­ject nearly every day since its in­cep­tion. I care about the code, and tbh it would be weird for me to spend 4 months cre­at­ing a Rust LSP if look­ing at the code was­n’t some­thing I do a lot.

I am not go­ing to pre­tend that I am an ex­pe­ri­enced LSP de­vel­oper and the code is per­fect. It is in a state that I can work with, but I un­der­stand that some bits might not be id­iomatic in terms of com­piler tool­ing de­sign. The code has a lot of com­ments, and I tried re­ally hard to make sure that these com­ments are not sloppy but help­ful, be­cause I have to read them all the time; so far the qual­ity is ob­vi­ously not as good as pro­fes­sion­ally writ­ten hu­man docs, but IMHO it’s pretty help­ful and not an­noy­ing to read.

A large part of the jour­ney is learn­ing. LLMs can be pretty good do­main ex­perts, and LLMs know about LSP de­sign much more than I do. At the same time, LLMs are not great at build­ing big pro­jects. So the fol­low­ing loop hap­pened mul­ti­ple times dur­ing de­vel­op­ment:

I build some­thing new.

LLM pro­pos­als seem rea­son­able, so I go with them.

It works but some­thing bugs me.

I think about the de­sign for a while and see a big flaw.

I work with LLM to fix it (sometimes for a week, if the screw up was par­tic­u­larly big — but the big­ger the screw-up is, the more I learn).

So on one hand, if I am to at­tribute code own­er­ship to the LLMs, I can com­plain: “LLMs tried to de­rail the pro­ject so many times!11”. But since it’s my code, I think that the code might get worse at some mo­ments, but as I learn, I get to im­prove it. Which is pretty nor­mal soft­ware de­vel­op­ment flow, just ac­cel­er­ated.

All in all, LLMs are just a tool, and it’s one’s choice to use it re­spon­si­bly or out­source think­ing to it. Given the amount of witch hunt­ing to­day, I have just one re­quest: do not re­duce me to a clanker. It is my code, so if you con­sider it to be slop, call it my slop, not AI.

I am open to crit­i­cism and will hap­pily lis­ten to feed­back: the more I learn, the more I can im­prove the code­base. Whether I use LLMs for that or not does not mat­ter that much, in my opin­ion.

What’s next

The pro­ject is al­ready in a state where it can be a daily dri­ver for some users, but I have rather big plans for it. So in the com­ing re­leases, you might ex­pect:

Further per­for­mance op­ti­miza­tions

Some more mem­ory op­ti­miza­tions (primarily dur­ing in­dex­ing, plus there are a few frag­men­ta­tion is­sues hap­pen­ing af­ter a full in­dex­ing run that I want to fix)

Improved type in­fer­ence / syn­tax sup­port.

Code ac­tions (implement miss­ing trait fields, auto-im­ports, etc).

Potentially proc macro sup­port (I have some weird idea that will not re­quire ac­tual code ex­e­cu­tion, but it’ll take a while to pre­pare).

Some fea­tures are un­likely to be sup­ported though, such as build scripts / proc macros sup­port via proc macro in­vo­ca­tion (e.g. any­thing that re­quires un­trusted code ex­e­cu­tion). I also don’t plan to work on things that are un­nec­es­sary at the cur­rent state of the pro­ject, such as mi­grat­ing to the new trait solver. Niche things like par­tic­u­lar nightly fea­tures will likely be post­poned un­til the pro­ject reaches some de­gree of ma­tu­rity with sta­ble Rust.

Additionally, there is a lot of cool lit­tle tricks I’ve done in Rust Glancer that I’m some­what proud of (aligning al­lo­ca­tion life­times to re­duce mem­ory frag­men­ta­tion, en­gine-as-a-sub­process model to help with both mem­ory frag­men­ta­tion and multi-work­space pro­jects, sharded cache, and oth­ers), so if peo­ple will be in­ter­ested, I’ll be happy to write some blogs telling about how Rust Glancer works un­der the hood. It’s par­tially cov­ered in the docs al­ready (1, 2) if you want to get some info right now.

But in any case, I hope that the pro­ject can be help­ful for some folks al­ready, and for more folks in the fu­ture.

My fa­ther passed re­cently, and he was twice my age. I am ap­prox­i­mately the same age that he was when I was born, and I am now “the old gen­er­a­tion” - there’s no one left in the gen­er­a­tion above me.

At the same time, I re­cently joined a com­pany that skews younger-than-me. When I joined Google in 2011, I had just turned 30, and was in the main­stream de­mo­graph­ics of Google in 2011. There were a bunch of more se­nior folks, with the very se­nior ones be­ing in their 50s and hav­ing com­pleted stints at Bell Labs. I ad­mired a lot of these “greybeards” (even though this is a sex­ist term - what’s the right fe­male equiv­a­lent? There were a few very se­nior fe­male en­gi­neers that I would love to in­clude).

So per­haps it is nat­ural that I am re­flect­ing on “what were the im­por­tant re­al­iza­tions that I made since my early 20s that had a pro­found im­pact on the way I think about the world”? In some sense: What are the in­sights I had that made me “more ma­ture”, for some pos­i­tive de­f­i­n­i­tion of “mature”?

This post tries to list them.

1. The im­por­tance of un­der­stand­ing your own in­cen­tive struc­ture, and not be­liev­ing every­thing you think.

I re­cently wrote a Twitter thread about the topic. Oppenheimer was very pub­licly guilt-rid­den about the cre­ation of the nu­clear bomb, and von Neumann at some point quipped “some peo­ple pro­fess guilt to claim credit for sin”. In my young years, par­tic­u­larly in sit­u­a­tions when I had 0day that no­body else had, I ag­o­nized about the re­spon­si­bil­ity that comes with hav­ing 0day. Should I fix them? Should I use them for good? Will the world be harmed this way? Or that way?

In the end, it turns out that - while in­di­vid­u­als mat­ter - many ideas have a “time at which they are ripe”, and the ac­tions of the in­di­vid­ual mat­ter less than the in­di­vid­ual thinks in that mo­ment. There is also al­most no way to pre­dict the ways in which what you do im­pacts the broader world.

If you were asked: “Would it be good if this 0day was used to ap­pre­hend a ter­ror­ist?” you would prob­a­bly say “this is good”. If you were asked “would it be good if this 0day is used to ar­rest some­one and then tor­ture and wa­ter­board him 183 times?”, you would prob­a­bly say “this is bad”. So if your 0day was used to cap­ture KSM, it is prob­a­bly good? Or bad? Things get very com­pli­cated very quickly.

Is clos­ing 0days good for so­ci­ety, be­cause it makes every­thing safer? Or is it en­abling op­pres­sion, be­cause buggy sys­tems are eas­ier to by­pass?

There are no good an­swers, and your own in­cen­tive struc­ture will greatly in­flu­ence how you choose your be­liefs. In the end, peo­ple want to be the he­roes of their own story, and at the same time they have basal needs for recog­ni­tion, for ma­te­r­ial goods, etc. - so they will try to con­struct a nar­ra­tive that al­lows them to sat­isfy their basal needs while also re­main­ing the hero of their saga.

Anxiety about the im­pact of your work is self-flat­ter­ing, and you have to rec­og­nize it as such, and keep it in check - it’s sugar for your ego, but his­tory will largely route around you, be­cause while in­di­vid­ual de­ci­sions mat­ter in spe­cific sit­u­a­tions, the over­all flow of his­tory is less sen­si­tive to the in­di­vid­ual than the in­di­vid­ual thinks. The broader les­son, though, is: Do not be­lieve every­thing you think. Examine your own in­cen­tive struc­tures care­fully. Ask your­self what al­ter­na­tive nar­ra­tives for your be­hav­ior and be­liefs could be, es­pe­cially if they con­tra­dict the nar­ra­tive of the heroic saga you’re con­struct­ing for your­self. Carefully weigh­ing the ques­tion “how might I be the vil­lain in this story?” is an im­por­tant and valu­able skill.

Similarly, meta-cog­ni­tion - just ob­serv­ing your own thoughts in a de­tached man­ner, and then be­ing able to in­ter­pret, an­a­lyze, and con­tex­tu­al­ize them with re­gards to your own in­cen­tive struc­tures, is a great skill to cul­ti­vate.

2. Monocausal de­ter­min­ism is an il­lu­sion, and largely does not ex­ist out­side of com­puter de­bug­ging.

The mono­causal de­ter­min­ism that young com­puter en­thu­si­asts get used to is an il­lu­sion that gen­er­a­tions of elec­tri­cal and process en­gi­neers spent their lives per­fect­ing and main­tain­ing. It is be­cause of these en­gi­neers that com­puter sci­en­tists could largely get away with­out prob­a­bil­i­ties or any em­pir­i­cal ground­ing in the past. There is an ar­gu­ment that you have so many nat­ural sci­en­tists that crossed over into AI be­cause CS ed­u­ca­tion was for a long time too fo­cused on rea­son­ing within the de­ter­min­is­tic mono­causal il­lu­sion.

The re­al­ity is: Computing ma­chines are phys­i­cal de­vices, which in­cludes wear & tear, dif­fer­ences in qual­ity be­tween items, and “probabilistically de­ter­min­is­tic be­hav­ior”, e.g. it’ll ap­pear de­ter­min­is­tic most of the time if not shaken too much. If pushed a bit - be it tem­per­a­ture, volt­age, elec­tro­mag­netic fields, or even rapid mem­ory ac­cesses to ad­ja­cent DRAM rows - de­ter­min­ism has a ten­dency to go out of the win­dow, the il­lu­sion col­lapses, and we’re deal­ing with a very dif­fer­ent beast.

FWIW - this also makes me won­der about model align­ment, be­cause even a per­fectly aligned model will be sub­ject to ran­dom bit flips in in­fer­ence, and it’s hard for me to imag­ine that you can main­tain any rea­son­able guar­an­tees in the pres­ence of bit flips to in­op­por­tune val­ues at in­op­por­tune times.

The real world is one where very few things that hap­pen have a sin­gle rea­son, and very few truly de­ter­min­is­tic trans­mis­sion mech­a­nisms. Everything is prob­a­bilis­tic, and every­thing is mul­ti­causal.

Measurement noise is real, ex­per­i­ment de­sign is dif­fi­cult.

Interestingly, if you think about this care­fully, you also re­al­ize that the sci­en­tific method is a clas­si­fier that is in­ten­tion­ally bi­ased against ac­cept­ing some­thing as true - so that we only ac­cept things as true that are be­yond any rea­son­able doubt true.

A some­what fas­ci­nat­ing coro­lary of this is that there ex­ists a large class of true things that will never be sci­en­tif­i­cally shown as true.

3. The di­chotomy be­tween rea­son and emo­tion is a cul­tural con­struct, and nei­ther grounded in neu­ro­science nor in logic.

With some dig­ging, it turns out that the west­ern be­lief that rea­son and emo­tion are two ends of a spec­trum is a purely cul­tural con­struct, as is the be­lief that “higher-order” rea­son needs to reign in “basal” emo­tions, or that “emotions” in­trude on “rationality”.

In most non-west­ern cul­tures, achiev­ing in­te­gra­tion be­tween ra­tio­nal de­lib­er­a­tion and im­pulses and emo­tions is more com­mon, and it turns out that this is much closer to the bi­o­log­i­cal re­al­ity.

From a neu­ro­science per­spec­tive, it is clear that emo­tional val­u­a­tion is part of a larger de­ci­sion-mak­ing ma­chin­ery that tends to not func­tion prop­erly if the emo­tional val­u­a­tion com­po­nent is dam­aged or re­moved. There is also a large com­po­nent where things that your brain strug­gles to ar­tic­u­late ver­bally are trans­mit­ted via emo­tions, as well as ac­tual feed­back from your sen­sory or­gans in your body. Fun trivia: Your gut’s en­teric ner­vous sys­tem con­tains as many neu­rons as the en­tire cere­bral cor­tex of a dog. Your body also for­ward-de­ploys neu­rons in your mus­cles and ex­trem­i­ties, as a form of la­tency op­ti­miza­tion. Your body is feed­ing you ex­tra in­for­ma­tion, and most of this shows up in the shape of emo­tions.

Which brings us to the log­i­cal ar­gu­ment why at­tempt­ing to “remove” emo­tions from de­ci­sion-mak­ing is a bad idea: Clearly, hav­ing the abil­ity of lever­ag­ing more in­for­ma­tion for de­ci­sion-mak­ing will im­prove the qual­ity of de­ci­sions. Attempting to elim­i­nate a par­tic­u­lar source of in­for­ma­tion al­most cer­tainly makes the qual­ity of your de­ci­sions worse.

This is not to say one should act on im­pulse alone, but it is cer­tain that in­te­grat­ing the full spec­trum of in­for­ma­tion - which in­cludes emo­tions - in your de­ci­sions is a wise idea.

I am sure that if I think more care­fully, I will come up with more in­sights, but these three are im­por­tant enough that they show up in my life with as­ton­ish­ing reg­u­lar­ity.

Hope this is help­ful to some­one.

It’s time to dream big. Omarchy Quattro has given peo­ple a chance to ex­pe­ri­ence what the mal­leable com­puter of the fu­ture looks like, and they like it (a lot!). It now feels like a moral oblig­a­tion to make this fu­ture more broadly avail­able and fun­da­men­tally change how peo­ple re­late to their com­put­ers for the first time in what seems like for­ever.

To do just that, I’m in­cor­po­rat­ing the Omacom Foundation to en­sure that this mis­sion is fully funded, durable, and ready to ac­cel­er­ate.

This non­profit foun­da­tion will hold the trade­marks, fund the in­fra­struc­ture, pro­mote the work, and sup­port the open-source pro­jects and de­vel­op­ers Omarchy de­pends on.

These eight Founding Patrons are each con­tribut­ing $1 mil­lion to this mis­sion:

Tobi Lütke, CEO of Shopify

Patrick Collison, CEO of Stripe

Michael Dell, Chairman and CEO of Dell Technologies

Jack Dorsey, Block Head and Chairman of Block

Matthew Prince, CEO of Cloudflare

Brendan Iribe, Cofounder of Sesame and Oculus

Jason Fried, CEO of 37signals

Yours truly

This is a ridicu­lous sum of money, so I in­tend to make sure it lasts a long time, and that we make the most of it. But just as im­por­tant as the in­cred­i­ble cush­ion is the vote of con­fi­dence de­liv­ered by these pledges.

We’re go­ing to make the prophecy of The Year of Linux on the Desktop come true. All the pieces are now in place. Time to go all in!

For years now one of the most re­li­able com­plaints I hear when I try to drag a team off their ven­dor spe­cific SDK and onto OpenTelemetry is some vari­a­tion of: “why does it seem like this is­n’t done yet?”

Vendor SDKs for ob­serv­abil­ity are, to put it char­i­ta­bly, id­iot-proof. You in­stall the thing, dash­boards just load data, some­one else wor­ries about how all those pieces fit to­gether, and you get on with your life. OpenTelemetry, by con­trast, greets you at the door with a lot of “experimental” stamps and roughly six dif­fer­ent ways to ac­com­plish any given task.

In OpenTelemetry’s de­fense this was never what they were go­ing for as a pro­ject. I’ve al­ways re­spect that they stuck to their guns by at­tempt­ing to build a truly ven­dor ag­nos­tic sys­tem that re­ally does­n’t care what you do with the data. I have never got­ten a sense of a ven­dor be­ing strongly pre­ferred with OTel, which is quite the feat con­sid­er­ing how lu­cra­tive and con­tentious the ob­serv­abil­ity ecosys­tem was. Also con­sid­er­ing that the main­tain­ers of this pro­ject are largely em­ployed by ex­clu­sively those com­pa­nies.

As the years wore on, I started to get ner­vous. Conversations in the se­man­tic-con­ven­tions repo drag on and on and on. Different lan­guages had dra­mat­i­cally dif­fer­ent sto­ries. Golang and Dotnet were first class cit­i­zens, but other lan­guages lagged years be­hind the oth­ers.

I started ask­ing a lot of prob­ing ques­tions be­fore rec­om­mend­ing OpenTelemetry to smaller teams who did­n’t have the time, bud­get, or emo­tional band­width for it. Auto-instrumentation was gen­uinely mag­i­cal, but the cliff be­tween “auto-instrument works” and “now I have to man­u­ally in­stru­ment some­thing” was steep enough that you owed peo­ple a warn­ing be­fore you pushed them off it.

This nar­ra­tive has been go­ing on for awhile in the ob­serv­abil­ity space, a vague sense of “something is wrong in Otel-land”. But let’s try to gen­er­ate some ac­tual data here. Is there an ac­tual prob­lem, or is this some­thing where the per­cep­tion by the com­mu­nity of slow progress is imag­i­nary? Is the prob­lem not enough main­tain­ers, too big of a scope, or some­thing in-be­tween?

My guess when I started was “oh this is your clas­sic open-source bit off more than they can chew”. Not enough main­tain­ers, not enough bud­get. Now there is some of that, but there’s also some­thing else go­ing on.

The ac­tual prob­lem hap­pen­ing in­side of OpenTelemetry is a three way crash. You have a bi­nary sta­bil­ity gate which, when com­bined with a very small bench of ac­tual main­tain­ers means there is un­der­stand­able worry about mark­ing a fea­ture not ex­per­i­men­tal then add on just a mas­sive scope of lan­guages and frame­works they are at­tempt­ing to cover. This cre­ates a per­fect storm where there is an in­cen­tive to ar­gue about po­ten­tial prob­lems a fea­ture might cre­ate since once it is locked in and shipped as sta­ble you can never change them.

How does OpenTelemetry Work

So OpenTelemetry cur­rently is at­tempt­ing to sup­port a dizzy­ing num­ber of lan­guages and frame­works.

OpenTelemetry is a gi­ant pro­ject. It spans dozens of lan­guages, hun­dreds of li­braries, and count­less back­ends. To keep things sane, the pro­ject splits work into two buck­ets:

Core → Maintained di­rectly by the OTel pro­ject. Small, sta­ble, ven­dor-neu­tral, and tightly re­viewed. This is the “spec-defining” sur­face.

Contrib → Community- and ven­dor-con­tributed. Broader, faster-mov­ing, and cov­ers the long tail of in­te­gra­tions.

There ex­ists the otel-col­lec­tor, the thing that runs along the thing so that you can ship logs met­rics and traces. That copies the same rough pat­tern. But for the lan­guages when we’re talk­ing about core vs con­trib this is what we’re talk­ing about.

Stuff that breaks goes in con­trib, stuff that does­n’t break goes into core.

Now the rea­son this causes a con­flict. con­trib is mas­sive overkill for most pro­jects. You don’t want 300 ex­porters to add the one you typ­i­cally need. On the lan­guage side, this is­n’t that big of a prob­lem. pip in­stall open­teleme­try-in­stru­men­ta­tion-flask gives you the stuff you need for flask. However on the col­lec­tor side you end up hav­ing to do the OpenTelemetry Collector Builder to make your own col­lec­tor (or just kinda ride the wave and hope it works out). While cool that this ex­ists, it’s a lot of scope to ask a team to take on.

Process of adding a new fea­ture

So I be­lieve I have cap­tured the work­flow of adding a new fea­ture to OTel. You can check my home­work here:

OpenTelemetry Enhancement Proposal (OTEP) (https://​github.com/​open-teleme­try/​open­teleme­try-spec­i­fi­ca­tion/​tree/​main/​oteps/)

Once the OTEP is ac­cepted, the text goes into the Specification di­rec­tory in the same repo.

After that it seems to go to Semantic con­ven­tions. This seems to be where we get down to the spe­cific de­tails and where most of the long dis­cus­sions seem to live. At this point we’re talk­ing about more or less a per­ma­nent com­mit­ment to this de­sign and where the lock-in process be­comes very hard to change.

Each of the SDKs im­ple­ments the API sur­face that is de­fined in the spec­i­fi­ca­tion. Now some of the SDKs have done 2.0 break­ing changes, so it does seem like the ear­lier “please no 2.0 at all costs” sen­ti­ment has been aban­doned (which I think is smart and good).

Contrib / in­stru­men­ta­tion. This is slightly more mushy. Looks like they should track lat­est API/SDK but each con­trib pack­age may ver­sion in­de­pen­dently so its more flex­i­ble as a de­sign.

Collector + OTLP. The data has to ac­tu­ally go some­where. OTLP (wire pro­to­col) has its own sta­bil­ity life­cy­cle and spec­i­fi­ca­tion (here). Collector com­po­nents have their own sta­bil­ity in their READMEs and as far as I can tell that’s kinda all over the place.

Things I’m not re­ally clear on

It’s un­clear how long the OTEP -> Specification process takes. I’ve looked through the Git his­tory but there does­n’t seem to be any pre­dictable num­ber or cy­cle.

I don’t fully un­der­stand what is the re­la­tion­ship be­tween all these sta­bil­ity com­mit­ments. Does Collector + OTLP group work in lock­step? Can a lan­guage “fall out of scope” if you lag too far be­hind?

Attempting to test it

So be­cause OpenTelemetry is a CNCF pro­ject, I fig­ured it made the most sense to com­pare them to other CNCF pro­jects. My ba­sis for com­par­i­son is Envoy and Prometheus. I have used a hacky Python script I’ve used be­fore for mea­sur­ing the “health” of open-source pro­jects, which is prob­a­bly not the best. However I’ll in­clude a link to the raw data with­out the charts so folks can re­view it and (more than likely) find a prob­lem in what I gen­er­ated.

So we look at 24 months of ac­tiv­ity for Envoy and what we see is a pretty healthy pro­ject. There’s good dis­tri­b­u­tion of au­thors, merg­ers, is­sue closers. phlax is ob­vi­ously pretty im­por­tant to the pro­ject but in gen­eral there’s a good bench of peo­ple to step in if needed. I’ve at­tempted to fil­ter out all the known bot traf­fic.

Let’s com­pare that to one of the OpenTelemetry lan­guages. The ones I have the most pro­fes­sional ex­pe­ri­ence with are Golang and Python, but I hear from a lot of folks in the com­mu­nity that the Ruby and PHP ones strug­gle a lot. This is the PHP one for the same pe­riod.

So we see pretty clearly that there’s way too much con­cen­trated on 2 peo­ple. This is not a healthy open-source pro­ject and they clearly don’t have enough peo­ple to cover the kind of scope OTel needs to cover. Same story with Ruby.

In com­par­i­son the “strongest” OpenTelemetry SDKs in my opin­ion, Golang and Dotnet (although Python is also no slouch) look more healthy.

Golang

So the first is­sue is maybe the least sur­pris­ing. There’s too much con­cen­tra­tion among too few main­tain­ers. Your au­thors should­n’t also be your merg­ers and your is­sue closers. Ideally these tasks should be dis­trib­uted out more evenly.

For what its worth I think the main­tain­ers have done a good job of at­tempt­ing to keep their dis­cus­sions pub­lic. It was very easy for me to find the pub­lic meet­ing notes of the dif­fer­ent groups of main­tain­ers, read through them and see what was go­ing on. I don’t get the sense that these main­tain­ers are try­ing to stop peo­ple from get­ting in­volved as much as the ex­pec­ta­tions of sta­bil­ity have, more or less, frozen the pro­ject in place.

The is­sue is more a clas­sic case of “someone has to pay the main­tain­ers”. The pro­ject is too com­plex for some­one to re­al­is­ti­cally do this as a hobby. I think any pro­ject sign­ing on for such long sta­bil­ity con­tracts can­not turn to the com­mu­nity of hob­by­ists ex­pect­ing as­sis­tance. I can’t join calls and do the things I would be ex­pected to do for a pro­ject of this size and im­por­tance for free. But it also means that the peo­ple do­ing this crit­i­cal work have ex­pec­ta­tions placed on them by their par­ent or­ga­ni­za­tions.

So these SDKs have too few main­tain­ers. But that does­n’t fully ex­plain why it seems to take so long for new fea­tures to get through the stack. My guess for that was that some­where in the process be­tween sub­mis­sion of the new idea and the for­mal­iza­tion of the idea was a long dis­cus­sion that took a mil­lion years.

Conventions about Semantics

So with this level of sur­face area across dif­fer­ent frame­works and lan­guages, it makes sense to con­cen­trate the con­ver­sa­tion about con­ven­tions in one place. That lives here: https://​github.com/​open-teleme­try/​se­man­tic-con­ven­tions

If ven­dor de­bate is caus­ing the slow­down, we should (in the­ory) see this slow­down in PRs here. Then you should see the slow­down ba­si­cally prop­a­gate out. Spoiler alert, I was wrong about this. Big thanks to the OpenTelemetry peo­ple for hav­ing good con­ven­tions on la­bel­ing their PRs which made this much eas­ier.

So if sem­conv is the slow­down, let’s look at the slow­est PRs there.

Yeah some of them are pretty slow, but there are some com­plex top­ics be­ing dis­cussed. However in­ter­est­ingly this slow­down does­n’t re­ally trickle into the SDK/API space, sug­gest­ing that OpenTelemetry is go­ing a good job of keep­ing these con­ver­sa­tions siloed off.

If we look at Python we see that their slow­est PRs aren’t sem­conv re­lated.

In re­al­ity the slow­down for these are the ex­tra re­quired check im­posed by the Approve Public API check which re­quires an­other main­tainer. But that seems ap­pro­pri­ate and takes us back to the ini­tial prob­lem of “not enough main­tain­ers”.

Potential Solutions

So af­ter look­ing at all of this, the pat­tern be­comes clear. A new fea­ture takes a very long time to make it to the end user in OpenTelemetry be­cause they take sta­bil­ity very se­ri­ously, com­bined with a rel­a­tively lim­ited bench of tal­ent to pull from. Once things make it through the en­tire stack, im­ple­ment­ing the API and get­ting that API change through to the end user falls on an over­worked main­tainer pool. So what do we do?

I think one idea worth ex­plor­ing is adding some sort of time-bound beta tier. Basically be­tween the “Experimental” and the “Stable” in the fol­low­ing di­a­gram. The prob­lem is that for end users, due to the ex­tra steps to use Experimental fea­tures, they might as well not ex­ist. 99% of us have no idea when an ex­per­i­men­tal fea­ture is added and we would never en­gage with it. But if I knew the fea­ture would stick around for at least 12 months with­out a re­moval and was more ac­ces­si­ble to me as an end user, it could ac­tu­ally help the pro­ject get more ac­tion­able feed­back.

Basically a fea­ture would go Experimental (pretty low us­age) -> Beta (more ex­posed to the end user than Experimental) -> 12 months -> Removal or Stable.

Now con­fus­ingly Beta ex­ists for Otel but is used for SDKs, not for com­po­nents. Like Rust is a Beta but it seems like Profiles can­not be a Beta. Honestly it’s nearly im­pos­si­ble for me to fig­ure out like what la­bels should ap­ply to what things. I sus­pect no­body re­ally knows. Here’s the ex­pla­na­tion of Beta that I think only ap­plies to SDKs.

Development

Not all pieces of the com­po­nent are in place yet, and it might not be avail­able for users yet. Bugs and per­for­mance is­sues are ex­pected to be re­ported. User feed­back around the UX of the com­po­nent is de­sired, such as for con­fig­u­ra­tion op­tions, com­po­nent ob­serv­abil­ity, tech­ni­cal im­ple­men­ta­tion de­tails, and planned use-cases for the com­po­nent. Configuration op­tions might break of­ten de­pend­ing on how things evolve. The com­po­nent SHOULD NOT be used in pro­duc­tion. The com­po­nent MAY be re­moved with­out prior no­tice.

Alpha

This is the de­fault level: any com­po­nents with no ex­plicit ma­tu­rity level should be as­sumed to be “Alpha”. The com­po­nent is ready to be used for lim­ited non-crit­i­cal pro­duc­tion work­loads, and the au­thors of this com­po­nent wel­come user feed­back. Bugs and per­for­mance prob­lems are en­cour­aged to be re­ported, but com­po­nent own­ers might not work on them im­me­di­ately. The com­po­nen­t’s in­ter­face and con­fig­u­ra­tion op­tions might of­ten change with­out back­ward com­pat­i­bil­ity guar­an­tees. Components at this stage might be dropped at any time with­out no­tice.

Beta

Same as Alpha, but the in­ter­faces (API, con­fig­u­ra­tion, gen­er­ated teleme­try) are treated as sta­ble when­ever pos­si­ble. While there might be break­ing changes be­tween re­leases, com­po­nent own­ers should try to min­i­mize them. A com­po­nent at this stage is ex­pected to have had ex­po­sure to non-crit­i­cal pro­duc­tion work­loads al­ready dur­ing its Alpha phase, mak­ing it suit­able for broader us­age.

Release Candidate

The com­po­nent is fea­ture-com­plete and ready for broader us­age. The com­po­nent is ready to be de­clared sta­ble, it might just need to be tested in more pro­duc­tion en­vi­ron­ments be­fore that can hap­pen. Bugs and per­for­mance prob­lems are ex­pected to be re­ported, and there’s an ex­pec­ta­tion that the com­po­nent own­ers will work on them. Breaking changes, in­clud­ing con­fig­u­ra­tion op­tions and the com­po­nen­t’s out­put, are only al­lowed un­der spe­cial cir­cum­stances. Whenever pos­si­ble, users should be given prior no­tice of the break­ing changes. Stable

The com­po­nent is ready for gen­eral avail­abil­ity. Bugs and per­for­mance prob­lems should be re­ported, and there’s an ex­pec­ta­tion that the com­po­nent own­ers will work on them. Breaking changes, in­clud­ing con­fig­u­ra­tion op­tions and the com­po­nen­t’s out­put, are only al­lowed un­der spe­cial cir­cum­stances. Whenever pos­si­ble, users should be given prior no­tice of the break­ing changes.

Deprecated

Development of this com­po­nent is halted. No new ver­sions are planned, and the com­po­nent might be re­moved from its in­cluded dis­tri­b­u­tions. Note that new is­sues will likely not be worked on ex­cept for crit­i­cal se­cu­rity is­sues. Components that are in­cluded in dis­tri­b­u­tions are ex­pected to ex­ist for at least two mi­nor re­leases or six months, whichever hap­pens later. They also MUST com­mu­ni­cate in which ver­sion they will be re­moved, ei­ther in terms of a con­crete ver­sion num­ber or the date of a re­lease, like: “the first re­lease af­ter 2023 – 08-01”.

Unmaintained

A com­po­nent iden­ti­fied as un­main­tained does not have an ac­tive code owner. Such com­po­nents may have never been as­signed a code owner, or a pre­vi­ously ac­tive code owner has not re­sponded to re­quests for feed­back within 6 weeks of be­ing con­tacted. Issues and pull re­quests for un­main­tained com­po­nents SHOULD be la­beled as such. After 6 months of be­ing un­main­tained, these com­po­nents MAY be dep­re­cated. Unmaintained com­po­nents are ac­tively seek­ing con­trib­u­tors to be­come code own­ers.

In ad­di­tion it is, re­spect­fully, mis­lead­ing to im­ply that Go and Ruby are be­ing main­tained at the same stan­dard. This is­n’t a shot at the Ruby folks — they are do­ing heroic work with what they have. But pre­tend­ing par­ity ex­ists when it does­n’t just cre­ates con­fu­sion and quiet re­sent­ment when a user shows up ex­pect­ing one ex­pe­ri­ence and gets an­other. Being hon­est about main­te­nance tiers would let peo­ple make in­formed choices and might at­tract more help to the other tiers by nam­ing the prob­lem out loud.

Finally I would try to sur­face these prob­lems more openly for OpenTelemetry from the per­spec­tive of “we need more main­tain­ers”. I feel like the peo­ple do­ing this work prob­a­bly knew there was a prob­lem, but it seems like the com­mu­nity at large has no idea that there is a need for frankly more en­gaged ide­ally in­de­pen­dent main­tain­ers and con­trib­u­tors.

OpenTelemetry is a great pro­ject that is do­ing great work. It’s do­ing, frankly, heroic work at this scale with this few peo­ple. But I think in or­der to ac­tu­ally re­place the ven­dor spe­cific SDKs we need to start get­ting a bit more prag­matic about what is re­al­is­tic to do in terms of sta­bil­ity con­tracts and num­ber of lan­guages. I don’t think break­ing changes are as dev­as­tat­ing to the com­mu­nity as these promises im­ply as long as they are com­mu­ni­cated well and I think with this thin of a bench of main­tain­ers, some­thing has to give.

Anyway feel free to check my data for ac­cu­racy and let me know if you find prob­lems!