“Over the past 18 months, Canada’s new gov­ern­ment has fo­cused on build­ing our strength at home, di­ver­si­fy­ing our part­ner­ships abroad, and strik­ing a fair deal with the United States.

Our ob­jec­tives in our trade ne­go­ti­a­tions have been to:

Preserve tar­iff-free ac­cess to the U.S. for the vast ma­jor­ity of Canadian busi­ness;

Provide greater sta­bil­ity to our trade re­la­tion­ship;

Significantly re­duce U.S. tar­iffs on our key strate­gic in­dus­tries, so that Canadian busi­nesses in these sec­tors would have the best ac­cess of any in the world;

Protect our small and medium-sized busi­nesses — the lifeblood of our econ­omy — in­clud­ing by re­mov­ing the im­mi­nent threat of new tar­iffs; and

Maintain our flex­i­bil­ity, in­de­pen­dence, and sov­er­eignty so we can keep build­ing the Canada we want.

We have recog­nised from the be­gin­ning that America has changed, and that we will not re­turn to our old re­la­tion­ship. Our gov­ern­ment un­der­stood, be­fore many, that America is al­ter­ing all its trade re­la­tion­ships. Putting tar­iffs on its clos­est al­lies and charg­ing for ac­cess to its vast mar­ket.

We have worked in that con­text. To strike a fair deal that would pro­vide the best ac­cess to the U.S. mar­ket and greater cer­tainty to Canadian busi­nesses and work­ers. Throughout, our goal has been to se­cure the best deal for Canadians, never a deal at any price or on any dead­line.

In re­cent weeks, we made im­por­tant progress to­ward im­prov­ing Canada’s po­si­tion as hav­ing the best deal in the world with the U.S.

However, that progress has not been enough to meet our ob­jec­tives for Canadians. As a re­sult, this evening, I have de­cided to sus­pend trade ne­go­ti­a­tions with the U.S. and have di­rected Canada’s ne­go­tia­tors to re­turn to Ottawa. They have worked hard, in good faith, to de­fend the in­ter­ests of Canadians through­out these ne­go­ti­a­tions up un­til the very last minute. However, last-minute changes in the U.S. pro­posed terms were un­fair, un­eco­nomic, and called into ques­tion the re­li­a­bil­ity of any deal.

At mid­night tonight, the U.S. in­tends to im­pose a 50% tar­iff on roughly $28 billion of Canadian goods. Canada will match those tar­iffs dol­lar for dol­lar to pro­tect our work­ers and busi­nesses.

In the com­ing days, the gov­ern­ment will in­tro­duce ad­di­tional mea­sures to sup­port Canadian work­ers and busi­nesses, build­ing on the nearly $25 billion in sup­port pro­vided over the past 18 months.

These ac­tions com­ple­ment Canada’s core eco­nomic strat­egy. From day one, we have been fo­cused on build­ing our strength at home and di­ver­si­fy­ing our part­ner­ships abroad.

That strat­egy is work­ing. We are ad­vanc­ing nearly $500 billion in ma­jor in­fra­struc­ture pro­jects. In par­al­lel, we are un­lock­ing new ex­port mar­kets for Canadian busi­nesses. Our ex­ist­ing free trade deals al­ready pro­vide Canada with pref­er­en­tial ac­cess to 1.5 billion con­sumers, and we are on track to dou­ble that mar­ket ac­cess by the end of this year.

Canadian eco­nomic growth is ac­cel­er­at­ing, and we are on course to have the sec­ond-fastest growth in the G7 over the next two years. Our econ­omy is cre­at­ing jobs at four times the rate of the United States. Our ex­ports to non-U.S. mar­kets are on track to dou­ble over the next decade. Foreign di­rect in­vest­ment in Canada is at its high­est level in two decades, run­ning at twice the rate of our near­est G7 com­peti­tor. Canada now ranks as the most at­trac­tive coun­try in the world for in­fra­struc­ture in­vest­ment.

Canada has what the world wants. And we will not al­low any na­tion to de­ter­mine our fu­ture. We will set our own course to keep build­ing Canada strong for all.”

The other day, I saw a vi­ral tweet say­ing that peo­ple talk­ing about how LLMs are caus­ing slow, bloated, code are go­ing to eat crow once they re-write every­thing in su­per-op­ti­mized as­sem­bly. We’re not quite at the point where we want to write every­thing in as­sem­bly, but some vari­ant of what Nolan Lawson said about test­ing, you can choose how many bugs you want now, which I less elo­quently noted here, is be­com­ing more true for per­for­mance.

In re­sponse to a com­ment in my last post that the cost of for­merly spe­cial­ized per­for­mance work has dropped by many or­ders of mag­ni­tude and per­for­mance work that used to re­quire a per­son or team that had a rare set of skills can be done by any­one who can type a few sen­tences1, which means that you can do all sorts of op­ti­miza­tions that used to be too ex­pen­sive to be worth­while for all but the largest scale or most lu­cra­tive pro­jects, Marc Brooker re­sponded with

Completely agree with your clos­ing point. Dynamic cus­tom soft­ware, fit­ted to a par­tic­u­lar work­load rather than a class of work­loads, seems like a very likely out­come. (Which comes with all kinds of fun risks and op­por­tu­ni­ties of its own). Kind of re­minds me of FFTW. And a ton of weird old demoscene tech­niques which were all about be­ing su­per fast and small on a very par­tic­u­lar prob­lem (and of­ten very par­tic­u­lar hard­ware). For ex­am­ple, I re­mem­ber a demo that re-used its code as tex­tures to get great cache lo­cal­ity.

Completely agree with your clos­ing point. Dynamic cus­tom soft­ware, fit­ted to a par­tic­u­lar work­load rather than a class of work­loads, seems like a very likely out­come. (Which comes with all kinds of fun risks and op­por­tu­ni­ties of its own). Kind of re­minds me of FFTW. And a ton of weird old demoscene tech­niques which were all about be­ing su­per fast and small on a very par­tic­u­lar prob­lem (and of­ten very par­tic­u­lar hard­ware). For ex­am­ple, I re­mem­ber a demo that re-used its code as tex­tures to get great cache lo­cal­ity.

And Michael Malis has noted

There’s been a meme cir­cu­lat­ing about how AI does­n’t help be­cause “code was never the hard part.” I think that’s true in some do­mains, but in oth­ers, writ­ing the code ab­solutely was the hard part. JIT com­pil­ers are a great ex­am­ple of that. For many pieces of soft­ware, a JIT com­piler would help a lot with speed­ing up the code. The rar­ity of JIT com­pil­ers makes me be­lieve that im­ple­ment­ing a JIT com­piler his­tor­i­cally was too dif­fi­cult for it to be worth­while. LLMs have low­ered the bar­rier to en­try and made it much eas­ier to write a JIT com­piler. This is the the­sis be­hind pgrust. Databases his­tor­i­cally were the hard­est piece of soft­ware to build and were lim­ited be­cause of that. Now, with AI, we can be more am­bi­tious about the type of soft­ware we build.

There’s been a meme cir­cu­lat­ing about how AI does­n’t help be­cause “code was never the hard part.” I think that’s true in some do­mains, but in oth­ers, writ­ing the code ab­solutely was the hard part. JIT com­pil­ers are a great ex­am­ple of that. For many pieces of soft­ware, a JIT com­piler would help a lot with speed­ing up the code. The rar­ity of JIT com­pil­ers makes me be­lieve that im­ple­ment­ing a JIT com­piler his­tor­i­cally was too dif­fi­cult for it to be worth­while. LLMs have low­ered the bar­rier to en­try and made it much eas­ier to write a JIT com­piler. This is the the­sis be­hind pgrust. Databases his­tor­i­cally were the hard­est piece of soft­ware to build and were lim­ited be­cause of that. Now, with AI, we can be more am­bi­tious about the type of soft­ware we build.

Optimizing for a class of work­load

Let’s try this out with FRE, the regex en­gine we built in the last post. Recall that it was cre­ated by hav­ing an agent loop for a month on im­prov­ing regex en­gine per­for­mance with ac­cess to the re­bar regex bench­mark suite. This re­sulted in FRE be­ing heav­ily over­fit to re­bar un­til we warned our agent that we had a hold­out bench­mark, which caused the agent to gen­er­al­ize the op­ti­miza­tions enough that per­for­mance was ok-ish on our hold­out. There’s no par­tic­u­lar rea­son to use a “software fac­tory” regex en­gine that does­n’t beat a well-tested regex en­gine on hold­out bench­marks, but one no­table thing about FRE was that the na­tive AOT com­piled ver­sion did quite well at longer searches. We noted that, it stands to rea­son that one could run the na­tive code com­piler in an­other thread while rip­grep was run­ning its nor­mal matcher and then cut over to the na­tive code when it fin­ished com­pil­ing and gen­er­ally get bet­ter per­for­mance. Of course this will gen­er­ally re­sult in worse per­for­mance for short queries as we lose a thread to com­pi­la­tion, but I care a lot more about how long rip­grep takes when it runs for many sec­onds or min­utes than when it runs for a few sec­onds, so I’m ok with that trade­off.

In the same way we could build a regex en­gine in a few min­utes of hu­man time, we can also just try this ex­per­i­ment in a few min­utes of hu­man time. I typed a few sen­tences and an agent went and did the work to al­low this to hap­pen (which would be a de­cent chunk of code surgery for a hu­man) and it ran the bench­mark on ac­tual rip­grep queries that come from my codex his­tory. For longer queries, we see a 2x-4x per­for­mance im­prove­ment here for a few very sim­ple queries. But most queries are more com­plex, and when we run on rep­re­sen­ta­tive hold­out queries, for queries where AOT should be en­abled2, we get about a 7% speedup. Not an earth shat­ter­ing re­sult, but also not a bad out­come for spend­ing a few min­utes typ­ing to codex (and it’s still do­ing more op­ti­miza­tion and will pre­sum­ably speed things up fur­ther).

Build an in­dex?

This is ar­guably a silly thing to do, since if we’re re­peat­edly search­ing for text on a com­puter, the ob­vi­ous thing to do to speed that up is­n’t to write a na­tive code com­piler for regex match­ing, it’s to cre­ate an in­dex. But the point here is just that this kind of tech­ni­cal work, which used to take a fair amount of time and ex­per­tise, can just be done triv­ially now. And if we wanted to build a text in­dex, it just so hap­pens that I worked on BitFunnel, the Bing search in­dex that was spe­cial­ized for con­stant/​fast text in­ges­tion that won Best Paper Award at SIGIR, so I can think of a few ex­per­i­ments to try if we’re go­ing to build a fast lo­cal in­dex of our en­tire ma­chine (the pro­jects I’ve seen seem to be in­tended to in­dex your code di­rec­to­ries, but what re­ally kills my ma­chine per­for­mance is when codex de­cides to run rip­grep against huge tem­po­rary di­rec­to­ries with a ton of gen­er­ated files and then ex­pands to look­ing at my whole ma­chine when it misses, so I’d want an in­dex of my en­tire disk and not just of the code for some pro­jects).

If I were work­ing at an AI lab and had ac­cess to things like SOTA mod­els run­ning on Cerebras chips or other ac­cel­er­a­tors that greatly in­crease tok/​s and there­fore load/​de­mand for search, I might ac­tu­ally sur­vey the ex­ist­ing in­dex­ers to see if they’re fast enough or if I’d want to build some­thing cus­tom my­self. While the open source ver­sion of BitFunnel “only” con­tains a byte­code in­ter­preter and one JIT, the Bing ver­sion con­tains mul­ti­ple JIT com­pil­ers. A pro­ject that did that level of op­ti­miza­tion used to be a ma­jor un­der­tak­ing, but “I could do that in a week­end” is now ac­tu­ally true for some of these kinds of pro­jects. With my lowly $200/mo ac­count, I think a some­what faster rip­grep plus any off-the-shelf in­dex is fine, so maybe this fast-in­gest­ing whole-ma­chine in­dex pro­ject can be left as an “exercise for the reader (who works at an AI lab)”.

Optimizations are cheap

The dras­tic re­duc­tion in the cost of op­ti­miza­tions has been true go­ing back to November 2025 and maybe even some­what be­fore then with pub­lic mod­els (and I’m sure be­fore that still with what folks at AI labs had ac­cess to). For an ex­am­ple from the GPT-5.1 or 5.2 days, with no knowl­edge of game AIs, I tried build­ing an Azul AI. This ended up be­ing the strongest AI in the world for the game by a pretty large mar­gin. From read­ing the the­sis that de­scribes the 2nd strongest AI, I think my AI is prob­a­bly a bit bet­ter on the “AI” side of things, but the main place it wins is on op­ti­miza­tion de­spite spend­ing what looks like maybe two or­ders of mag­ni­tude less time (estimated by read­ing the the­sis and see­ing the process and com­par­i­son to my process) and also mostly work­ing on my lap­top vs. hav­ing a clus­ter of ma­chines to use (which means much less band­width to run ex­per­i­ments with, do pa­ra­me­ter tun­ing, etc.). For ex­am­ple, that other AI is sin­gle-threaded and my AI is multi-threaded. Since I have a na­tive code ver­sion as well as a heinous shared wasm mem­ory + javascript ver­sion, and two dif­fer­ent search ar­chi­tec­tures for two dif­fer­ent ver­sions, which “require” com­pletely dif­fer­ent multi-thread­ing al­go­rithms (minimax for a very small and fast net and MCTS for a larger net), this would’ve been a fairly large un­der­tak­ing if done by hand. And, be­cause I let an LLM pick the multi-thread­ing al­go­rithm based on its own (incorrect) rea­son­ing a cou­ple times be­fore spend­ing 30 min­utes read­ing about multi-thread­ing al­go­rithms for game AIs my­self, I ended up re-writ­ing (having codex re-write) the multi-thread­ing al­go­rithm mul­ti­ple times.

There’s a bunch of stan­dard stuff it makes sense to do to de­bug and ver­ify a mul­ti­thread­ing al­go­rithm for some­thing like this, like im­ple­ment­ing re­play from de­bug logs that can re­pro­duce bugs de­spite the al­go­rithm be­ing non­de­ter­mistic. Doing that alone would’ve prob­a­bly been days to a week of work had I done it by hand, but it’s ex­actly the kind of thing an agent can triv­ially do in a loop (just have it try to re­play logs and in­sert log­ging for non-de­ter­min­ism every time you don’t get a per­fect re­play). A lot of the te­dium it used to take to get a tricky op­ti­miza­tion like this work­ing is gone.

This also ap­plies to a lot of other tricky op­ti­miza­tions. From hav­ing writ­ten CPU mi­croc­ode, done CPU ver­i­fi­ca­tion, worked on op­ti­miz­ing a search en­gine in­dex, etc., I have a lot of ex­pe­ri­ence look­ing at op­ti­miza­tions and think­ing “hmm, this would in­crease per­for­mance by 2%, but it’s go­ing to take N per­son-days to ver­ify that this tricky op­ti­miza­tion works” and mak­ing a call to go ahead or not based on whether or not it’s worth the time to get the op­ti­miza­tion work­ing. Now that this N has dropped by a tremen­dous fac­tor (variable but, in terms of hu­man time, fre­quently 1000x / 10000x / 1000000x, prob­a­bly more like 1000x on dol­lar cost if you com­pare to­ken costs at me­tered rates vs. the Bing en­gi­neer who wrote the com­pil­ers at JITs that the search in­dex used), the num­ber of these kinds of op­ti­miza­tions it makes sense to do goes way up. The same goes for op­ti­miza­tions that you aren’t sure will work out. I used to some­times look at an op­ti­miza­tion that I was­n’t sure would speed things up and think “this will take M hours to im­ple­ment to the point where we have a good enough mea­sure­ment to guess at the per­for­mance im­pact”. Many more of those op­ti­miza­tions make sense to try out now.

Going back to the game AI case, at least for the AI I tried, it seems like you gain about 100 Elo for every dou­bling in speed (more than in chess, I sus­pect be­cause draws are very rare). Just adding mul­ti­thread­ing alone is enough to wipe the floor with an oth­er­wise com­pa­ra­ble AI on a large ma­chine. If you stack in 10 – 20 more op­ti­miza­tions that seem too an­noy­ing for most peo­ple to do by hand, the dif­fer­ence in strength is tremen­dous and it’s not re­ally rea­son­able to try to keep up with a hand-writ­ten AI3.

The game AI case is a lit­tle more an­noy­ing than for most soft­ware be­cause a lot of the op­ti­miza­tions you want to do ac­tu­ally change the re­sult and there is­n’t a cheap, triv­ial, way to tell if the speed in­crease + the change in re­sult gives a bet­ter or worse ac­tual re­sult in prac­tice. And, as we noted be­fore, cur­rent pub­licly avail­able SOTA mod­els are pretty bad at ex­per­i­men­tal de­sign, so I had to set up the frame­work they used to de­ter­mine if an op­ti­miza­tion is good, but once that was in place, it’s like any other op­ti­miza­tion prob­lem. I guess peo­ple work­ing on LLM op­ti­miza­tions also have to deal with this class of prob­lem but most op­ti­miza­tion prob­lems are a lot more straight­for­ward.

To pick an­other ex­am­ple, as part of prepar­ing for per­for­mance in­ter­views, Jamie Brandon tried Anthropic’s now pub­lic per­for­mance take­home. After try­ing it, he had Claude pick up where he left off and it got a much bet­ter re­sult. When he looked at what Claude did that he did­n’t, he said a lot of the op­ti­miza­tions were things that oc­curred to him but he had­n’t got­ten to yet, and “[o]thers were just crazy shit that I would never try un­less I was work­ing on this for weeks”4. He’s a rea­son­able per­for­mance en­gi­neer and he got an of­fer for the per­for­mance job he wanted, but on a well-de­fined op­ti­miza­tion prob­lem, he does­n’t stand a chance against a de­cent model (I haven’t tried the prob­lem my­self, but I sus­pect I also would­n’t stand a chance given re­motely com­pa­ra­ble time con­trols).

Workload-specific op­ti­miza­tion

Coming back to this part of Marc Brooker’s com­ment:

Dynamic cus­tom soft­ware, fit­ted to a par­tic­u­lar work­load rather than a class of work­loads, seems like a very likely out­come.

Dynamic cus­tom soft­ware, fit­ted to a par­tic­u­lar work­load rather than a class of work­loads, seems like a very likely out­come.

This seems pretty in­evitable. In an­other re­sponse to my post, Michael Malis of pgrust said some­thing sim­i­lar:

[discussion of pgrust op­ti­miza­tions] … I think it’s easy enough to cre­ate these op­ti­miza­tions that we could look at a cus­tomers work­load and add them as needed

[discussion of pgrust op­ti­miza­tions] … I think it’s easy enough to cre­ate these op­ti­miza­tions that we could look at a cus­tomers work­load and add them as needed

Without hav­ing any kind of frame­work or setup, right be­fore I started writ­ing this post, I had an agent do work­load-spe­cific op­ti­miza­tion for my rip­grep queries (not the na­tive code com­piler switch, just the op­ti­miza­tions to the gen­eral FRE en­gine based on a set of bench­marks), which took about 2 min­utes for me to launch. The op­ti­miza­tions run on a set of queries, and then there’s a later hold­out set of queries to run against. That’s still run­ning, but the ini­tial re­sults seem promis­ing. After one pass of op­ti­miza­tion, the work­load op­ti­mized ver­sion is 2% faster than stan­dard rip­grep on the hold­out and it’s still get­ting faster. 2% is­n’t a big deal for my lo­cal rip­grep us­age, but con­sid­er­ing that this took min­utes of time and the op­ti­miza­tions done here got started when I started typ­ing this point and are still im­prov­ing, I’d take a 2% win here (note that this is­n’t com­bined with the na­tive code com­piler, which would give a larger over­all win if com­bined prop­erly). And re­call that this is lever­ag­ing the FRE regex en­gine5, which was sub­stan­tially slower than the Rust regex en­gine on hold­out bench­marks and was stuck with slow im­prove­ment on hold­outs be­cause with me know­ing noth­ing about regex work­loads and SOTA LLMs not be­ing good enough at ex­per­i­men­tal de­sign to do un­guided open-ended self-im­prov­ing loops, we did­n’t have a good way to im­prove per­for­mance on our hold­outs. But if what I care about is per­for­mance on my own work­loads, I have plenty of data and am gen­er­at­ing more all the time. As Marc Brooker noted above, we do have to be care­ful about over­fit­ting if there’s a regime change that’s not in the old data, etc., but we’re still in a bet­ter sit­u­a­tion than we were be­fore.

In the more gen­eral case, if you’re some­one like Marc Brooker at Amazon or Michael Malis work­ing on pgrust, it makes sense to not just do this as a one-off, but to work with cus­tomers to pi­lot a pro­gram that uses their data to op­ti­mize things for them and then fig­ure out how to scale it out for cus­tomers in gen­eral. I’m not work­ing at a com­pany where that’s the best use of my time6, but it’s pretty wild that you can see that this is com­ing for larger com­pa­nies with more scale, and given that it only takes min­utes of my time to run these ex­per­i­ments for my per­sonal work­flows, it’s pretty rea­son­able to mess with this kind of thing on per­sonal pro­jects.

Thanks to Jamie Brandon, Michael Malis, and Max Bittker for com­ments/​cor­rec­tions/​dis­cus­sion.

P.S. As I’ve noted in the last cou­ple posts, with cod­ing agents, the time it takes to run an ex­per­i­ment and see enough of a re­sult to sat­isfy my cu­rios­ity has gone way done while the time it takes to make a re­sult re­ally rig­or­ous has­n’t changed or has gone up, so writ­ing things up the way I used to would mean run­ning very few ex­per­i­ments rel­a­tive to the band­width I have for them. As a re­sult, I’ve just been run­ning these ex­per­i­ments and shar­ing the re­sult with a cou­ple of friends. As an ex­per­i­ment, I’m try­ing to write these up in a very quick and non-rig­or­ous way in­stead of years of these ex­per­i­ments only be­ing known to a few friends. Like the last post, I set a goal of writ­ing this post and do­ing all the clean-up in half an hour and did­n’t time it but am pretty sure I missed that by a bit.

Even do­ing this, the time it takes to write these up is long enough that I’m falling be­hind on shar­ing re­cent re­sults, but I’m not in­clined to switch to LLM-written posts (yet?), and I don’t think I can re­al­is­ti­cally get the time to clean up the data and write a post like this down enough to turn a post around in less than half an hour. Just on the length of this post, typ­ing this up should be some­thing like 20 – 30 min­utes in­clud­ing time to pause and think about what I’m writ­ing, and then when I look at the data some­times some­thing will look wrong enough that I need to look into it more closely to see if there’s an is­sue that needs to be fixed (this hap­pened mul­ti­ple times here, and I would ex­pect that, be­cause I did­n’t spend much more time, there are other data is­sues that I don’t know about).

Anyway, if you have opin­ions on these quick (and surely more wrong) writeup, let me know what you think (X Bsky Mastodon)!

Appendix: There’s no rea­son for soft­ware to be slow any­more

I’ve been on the record for a long time as strongly dis­agree­ing with the gen­eral sen­ti­ment that the de­vel­op­ers of X are bad and should feel bad for writ­ing slow code be­cause there are a lot of dif­fer­ent kinds of pro­gram­ming ex­per­tise and not only is it not the case that most pro­gram­mers don’t have per­for­mance ex­per­tise, it prob­a­bly does­n’t even make sense for them to de­velop (from the stand­point of what the busi­ness cares about, what the em­ploy­ment mar­ket looks like, etc.), so of course most pro­jects will have very poor per­for­mance com­pared to what a per­for­mance ex­pert can do. I can see why a per­for­mance ex­pert would look at the grow­ing gap be­tween how fast a pro­gram can be and how fast pro­grams ac­tu­ally are and think that it’s ridicu­lous. I don’t dis­agree that there’s an ab­sur­dity to it, but if I think about the gap be­tween how good a UI can be and how a good a UI I can make (by hand) is, I don’t think that looks any less ab­surd, but I also don’t think it re­ally makes sense for me to spend time learn­ing how to build a great UI, or even a de­cent UI, for the same rea­sons it does­n’t make sesne for most peo­ple to spend time learn­ing how to de­cent per­for­mance work.

For the ex­am­ple above, Jamie Brandon got an of­fer from Anthropic and you prob­a­bly can’t af­ford him un­less you’re OpenAI, but you can af­ford to use a cod­ing agent that can beat him on a bounded op­ti­miza­tion prob­lem. The agent does­n’t have the judge­ment he has and will do worse on an open-ended prob­lem (recall that when we tried build­ing an op­ti­mized regex en­gine and just told it to not over­fit, it was more than an or­der of mag­ni­tude worse than the best regex en­gines on our hold­out bench­marks, but also re­call that af­ter telling the agent there was a hold­out it was do­ing poorly on, it sped up regex en­gine per­for­mance enough to gen­er­ally match 2nd tier regex en­gines in terms of per­for­mance, which is still ex­tremely good com­pared to the gen­eral level of per­for­mance op­ti­miza­tion in most code to­day), but that’s plenty good to achieve rea­son­able per­for­mance on all sorts of prob­lems. This post has gen­er­ally dis­cussed back­end per­for­mance is­sues, but agents don’t seem worse at front-end per­for­mance if you want to drive down a set of met­rics like LCP, INP, etc.

I still don’t think some­one is bad and should feel bad if their soft­ware has poor per­for­mance, but I do think that some­one who does­n’t know any­thing about per­for­mance and is a rea­son­able user of LLMs (just in gen­eral, not on per­for­mance prob­lems in par­tic­u­lar) should gen­er­ally be able to cre­ate soft­ware that has de­cent per­for­mance. If you just tell an LLM to op­ti­mize, it will of­ten do all sorts of in­cor­rect things that are re­ally bad that you have to catch, but that’s gen­er­ally true of us­ing the LLM ef­fec­tively in the first place, so get­ting de­cent per­for­mance is no longer a spe­cial­ized skill.

Appendix: How is codex run­ning rip­grep?

Here’s some in­for­ma­tion about the dis­tri­b­u­tion of riprep queries on my ma­chine. I make no claims that this is at all rep­re­sen­ta­tive of what’s hap­pen­ing any­where else. The pat­tern dis­tri­b­u­tion of the length of the pat­tern that’s searched has a lot more long pat­terns that I would’ve ex­pected. The p50 is 55 uni­code code points (I’ll just call these char­ac­ters for sim­plic­ity), which is al­ready longer than things I grep for by hand, and the p90 is 119!

We can also look at the num­ber of al­ter­na­tion arms in regexes, which are once again much more com­plex than what I do by hand.

Another view is to look at how these are cor­re­lated. Do we get more al­ter­na­tion arms in the regexes as the regexes get longer? Yes.

What are these re­ally long regexes, any­way? If we look at them, most of the longest are long al­ter­na­tions over func­tion or tests names, such as the fol­low­ing regex, which ap­pears to be re­lated to FRE de­vel­op­ment.

fn (hot_byte_compiler_is_generic_only_and_anonymous_count_uses_auto_count| one_­pat­tern_­coun­t_s­pan­s_us­es_the_re­tained_­com­plete_s­pan_ses­sion| for­mal_­com­pact_s­tate_byte_vis­i­tors_­co­ex­ist_with­_­na­tive_­count| fixed_bound­ary_record_vis­it_­match­es_­line_rel­a­tive_ref­er­ence_and_is_atomic| un­bound­ed_lan­guages_refuse_fi­nite_ex­trac­tion_be­fore_al­lo­ca­tion| for­mal_s­in­gle_raw_s­pan_sweep­_pre­flight| as­sert_ex­ac­t_­fix­ture_us­es_­for­mal_large_­con­tin­u­a­tion_sweep| url_on­ly_­com­pile_i­den­ti­ty_bind­s_lan­guage_and_own­er_­mode| url_on­ly_­com­pile_ex­ac­t_lim­it­s_and_run­time_re­fusal­s_­close| url_on­ly_­com­pile_­post_­plan_al­lo­ca­tion_­fault­s_­close| url_on­ly_own­er_dis­crim­i­na­tor_is_sta­ble_and_precharged| url_on­ly_­com­pile_own­er_is_s­trat­e­gy_and_­op­er­a­tion_s­coped| for­mal_re­bar_url_own­er_is_­com­pile_on­ly_and_­match­es_o­r­a­cle| for­mal_re­bar_url_ex­ac­t_­fix­ture_us­es_cer­ti­fied_ex­e­cu­tion| for­mal_­fixed_schema_­ma­te­ri­al­iza­tion_­match­es_both­_record_o­r­a­cles_and_­con­trols| for­mal_s­in­gle_­coun­t_s­e­lect­s_­com­pact_s­tate_byte_­com­plete_bound­_vis­i­tors| au­then­ti­cat­ed_bound­_­line_­to­tal_lf_free_­do­main_op­por­tu­ni­ty_ex­ceed­s_­five_per­cent| pre­pared_ab­solute_onepass_­fus­es_s­lot­s_and_p­re­serves_pre_­source_­fall­back| au­then­ti­cat­ed_­word_bound­ary_russ­ian_­com­pact_low­er­ing_pub­lic_­ca­nary| or­dered_n­fa_x86_ep­silon_edges_by­pass_the_as­ser­tion_­call| or­dered_n­fa_aarch64_ep­silon_edges_by­pass_the_as­ser­tion_­call| or­dered_edge_dis­patch_v2_is_­tar­get_neu­tral_de­ter­min­is­tic_and_re­lo­ca­tion_free| or­dered_edge_dis­patch_v2_­copies_­canon­i­cal_ta­bles_and_­cap_­fall­s_back­_­to_v1| or­dered_n­fa_v3_­com­pos­es_ter­mi­nal_range_and_dis­patch_with­out_­da­ta_re­lo­ca­tions| or­dered_n­fa_x86_ter­mi­nal_range_emit­s_au­then­ti­cat­ed_re­verse_s­can| or­dered_n­fa_aarch64_ter­mi­nal_range_emit­s_au­then­ti­cat­ed_re­verse_s­can| or­dered_n­fa_x86_bound­ary_as­ser­tion_­cache_is_lazy_and_bound­ary_s­coped| or­dered_n­fa_aarch64_­caches_re­peat­ed_as­ser­tion­s_once_per_bound­ary| bound­ary_as­ser­tion_­cache_re­quires_­dense_ex­ac­t_kind_reuse| bound­ary_as­ser­tion_­cache_s­e­lec­tion_is_­com­pil­er_on­ly_and_de­ter­min­is­tic)

But some are funny nu­mer­i­cal con­struc­tions, such as

:(13[0 – 9]|14[0 – 9]|15[0 – 9]|16[0 – 9]|17[0 – 9]|18[0 – 9]|19[0 – 9]|20[0 – 9]|21[0 – 9]|22[0 – 9]|23[0 – 9]|24[0 – 9]|25[0 – 9]|26[0 – 9]|27[0 – 9]|28[0 – 9]|29[0 – 9]|30[0 – 9]|31[0 – 9]|32[0 – 9]|33[0 – 9]|34[0 – 9]|35[0 – 9]|36[0 – 9]|37[0 – 9]|38[0 – 9]|39[0 – 9]|40[0 – 9]|41[0 – 9]|42[0 – 9]|43[0 – 9]|44[0 – 9]|45[0 – 9]|46[0 – 9]|47[0 – 9]|48[0 – 9]|49[0 – 9]|50[0 – 9]|51[0 – 9]|52[0 – 9]|53[0 – 9]|54[0 – 9]|55[0 – 9]|56[0 – 9]|57[0 – 9]|58[0 – 9]|59[0 – 9]|60[0 – 9]|61[0 – 9]|62[0 – 9]|63[0 – 9]|64[0 – 9]|65[0 – 9]|66[0 – 9]|67[0 – 9]|68[0 – 9]|69[0 – 9]|70[0 – 9]|71[0 – 9]|72[0 – 9]|73[0 – 9]|74[0 – 9]|75[0 – 9]|76[0 – 9]|77[0 – 9]|78[0 – 9]|79[0 – 9]|80[0 – 9]|81[0 – 9]|82[0 – 9]|83[0 – 9]|84[0 – 9]|85[0 – 9]|86[0 – 9]|87[0 – 9]|88[0 – 9]|89[0 – 9]|90[0 – 9]|91[0 – 9]|92[0 – 9]|93[0 – 9]|94[0 – 9]|95[0 – 9]|96[0 – 9]|97[0 – 9]|98[0 – 9]|99[0 – 9])[0 – 9]:

This is equiv­a­lent to :(?:1[3 – 9]|[2 – 9][0 – 9])[0 – 9]{2}: (which, if run through rip­grep on the orig­i­nal in­put, has ap­prox­i­mately the same per­for­mance). The en­tire pipeline for that was

cargo clippy … | rg ‘crates/fre-aot-regex/src/module.rs:’ | rg NUMBER_REGEX | head -250

which might be an odd thing for a hu­man to do, but agents seem to do this kind of thing all the time.

On an­other topic, if we look at how long rip­grep queries took, there are quite a few slow queries, e.g., p99 is al­most 1 minute! And p999 is al­most 10 min­utes! And the max­i­mum query over this time pe­riod (around a month on one lap­top; queries and dis­tri­b­u­tions seem likely to be dif­fer­ent on the AWS hosts I run agents on, etc., but I haven’t checked) is ap­proach­ing 2 hours!

In terms of com­mand line op­tions, we see the fol­low­ing. Perhaps un­sur­pris­ingly, codex of­ten wants line num­bers and, for what­ever rea­son, it very oc­ca­sion­ally uses PCRE2 regexes.

I won’t add plots or ta­bles for these, but an­other thing to note is that there’s fairly low lo­cal­ity for what pat­terns are searched for (about 94% of pat­terns only oc­curred once), which makes some sense given how long a lot of the queries were. However, there’s fairly high lo­cal­ity in what files get searched and a file that got searched is rel­a­tively likely to get searched again soon, in­di­cat­ing that (for small enough files), they’re likely to be searched in mem­ory.

Also, 99% of queries were regex queries (1% were non-regex string searches) and 99.9% of search queries were ASCII only, but in terms of files searched, ap­prox­i­mately 45% were ASCII only and 55% con­tained Unicode, a higher per­cent­age than I would’ve guessed for Unicode.

On a draft of the last post, Peter Geoghegan noted

It’s also pos­si­ble for a regex im­ple­men­ta­tion to be faster by sup­port­ing fewer fea­tures. Some im­ple­men­ta­tions don’t sup­port back ref­er­ences, etc.

It’s also pos­si­ble for a regex im­ple­men­ta­tion to be faster by sup­port­ing fewer fea­tures. Some im­ple­men­ta­tions don’t sup­port back ref­er­ences, etc.

which is also true here. The work­load-spe­cific op­ti­miza­tions done here were fairly su­per­fi­cial be­cause I just gave codex some short in­struc­tions and let it do what­ever it wanted (which is, in gen­eral, not the most ef­fec­tive use of codex), but with a more de­tailed plan, more fo­cused op­ti­miza­tions sup­port­ing the com­mon use cases for my queries could be ex­pected to yield larger gains.

though, as we dis­cussed in that post as well as be­fore, the bench­mark­ing and ex­per­i­men­tal de­sign skills of SOTA mod­els aren’t good enough to do this in the gen­eral case with­out a hu­man (or a skill) set­ting up the bench­mark­ing en­vi­ron­ment for the agent. [return]

we can see from our old bench­marks that, even with time to run the com­piler, there are a lot of cases where the na­tive code com­piled ver­sion is slower than the Rust regex crate. If we look at why this is, these tend to be more com­plex queries where the Rust regex crate has some al­go­rith­mic op­ti­miza­tion and the FRE na­tive code com­piler is falling back to some­thing naive (the agent that cre­ated FRE spent much less time on the na­tive code com­piler than it did on the “normal” regex en­gine). [return]

I have no doubt that a hand-writ­ten AI by some­one who has real AI ex­per­tise, e.g., by some­one who’s writ­ten one of the top Go and chess en­gines in the world, could beat my AI on the strength of the “AI” side of things be­ing bet­ter than what you get when some­one who knows noth­ing about AI (me) cre­ates an AI, but if the lev­els of ex­per­tise are re­motely sim­i­lar, the LLM-written ver­sion is go­ing to dom­i­nate for any given amount of time spent. [return]

it’s ar­guably un­fair to com­pare the re­sult of an agent pick­ing up where he left off, since his work is a start­ing point which might let an agent do much bet­ter than it would do on its own, so I tried giv­ing the fresh task to an agent and it got a very sim­i­lar score to what he got when an agent re-used his work (and a quick check by an­other agent did­n’t find ev­i­dence of cheat­ing). [return]

The per­for­mance prob­a­bly would’ve been bet­ter if I had an agent just mod­ify a rip­grep fork di­rectly, but I was cu­ri­ous if this could also solve the FRE over­fit­ting prob­lem with re­spect to my queries. [return]

a while back, I re­duced the size of page in our signup flow from 50 MB to 5 MB and a rev­enue A/B test seemed to in­di­cate that this in­creased rev­enue by about 0.5%. In gen­eral, I’m a huge fan of do­ing the sim­ple and easy wins first, such as this, and there are prob­a­bly a lot of higher ROI wins than we’d get out of build­ing cus­tom com­pil­ers or do­ing other highly spe­cial­ized tech­ni­cal work here. [return]

I want to pre­sent a pro­ject that I’ve been work­ing on for the past 4 months: an al­ter­na­tive Rust LSP im­ple­men­ta­tion that is built with a fo­cus on low mem­ory us­age.

It has two main fea­tures:

It can use very lit­tle mem­ory (target <100mb for rea­son­able pro­jects). There are caveats, these are de­scribed be­low.

It al­lows im­me­di­ate in­dex­ing af­ter restart: if your pro­ject was in­dexed, restart­ing the ed­i­tor will not re­quire re-in­dex­ing.

Note: through­out this video, the used RAM re­mained un­der 100mb

These fea­tures make Rust Glancer suit­able for the older com­put­ers: I have tested it on my old MacBook Pro M1 2020 with 8GB RAM, and it was pretty good.

As you can imag­ine, 4 months is not a lot of time for a pro­ject as big as a Rust LSP. Rust Glancer is not a com­plete LSP yet, it has a lot of miss­ing func­tion­al­ity, it has some known bugs, and it has a lot of things I want to im­prove.

At the same time, it is al­ready pretty ca­pa­ble: it has a full in­dex­ing pipeline with type in­fer­ence and a trait solver (chalk), most of the “normal” Rust syn­tax is sup­ported, and most of the “normal” LSP ac­tions do work as well: goto de­f­i­n­i­tion, hover, in­lay hints, com­ple­tions, you name it.

If you are in­ter­ested, you can al­ready try it out: just in­stall the VS Code ex­ten­sion here, or, if you pre­fer, build and in­stall the vsix from the repos­i­tory.

The rest of the post con­tains the his­tory of the pro­ject: mo­ti­va­tion, LLM use, plans and roadmap. If you’re not in­ter­ested, you might want to check out the pro­ject doc­u­men­ta­tion in­stead.

Difference with rust-an­a­lyzer

There are sev­eral rea­sons why rust-an­a­lyzer con­sumes a lot of mem­ory:

Rust work­spaces gen­uinely have a lot of in­for­ma­tion that must be in­dexed: thou­sands of func­tions, struc­tures, traits, re­la­tion­ships be­tween these, func­tion bod­ies and state­ments in them, etc. Each of these needs to be an­a­lyzed and re­mem­bered, and you can’t re­ally cheat if you want to have things like “find all ref­er­ences to this struc­ture”.

rust-an­a­lyzer uses salsa as its data­base. It’s an in­cre­men­tal query-based data­base, which lazily com­putes all the data you need with­out hav­ing to ex­plic­itly “record” every­thing. It is a very cool ap­proach, but it’s in­her­ently tied to mem­ory, which makes it hard to move parts of data from mem­ory else­where.

rust-an­a­lyzer uses rowan for syn­tax tree rep­re­sen­ta­tion. The cool prop­erty here is that it al­lows par­tial in­val­i­da­tion: if only a part of the file changed, only the rel­e­vant bits have to be reparsed, which makes it faster than hav­ing to re-parse the whole file on each key­stroke. However, the tree-like rep­re­sen­ta­tion in­side of it can cause heavy mem­ory frag­men­ta­tion (meaning that the amount of RAM taken from the OS is higher than the amount of “actually used” RAM).

(1) is some­thing we have to live with (though there are a few op­ti­miza­tions we can do there which Rust Glancer does), but (2) and (3) are the con­se­quences of the rust-an­a­lyzer ar­chi­tec­ture. rust-an­a­lyzer chose them to make the LSP faster, and it does work for that pur­pose.

The idea I had when I started the pro­ject: what if we don’t try to make an in­cre­men­tal LSP? What if all we have is a frozen analy­sis re­sult that gets in­val­i­dated on save? It ob­vi­ously will not be as fast as rust-an­a­lyzer, but it will give us the prop­er­ties we seek:

analy­sis re­sults can be of­floaded to the filesys­tem and loaded to mem­ory only when they are ac­tu­ally needed.

saved analy­sis is reusable, and since it’s al­ready of­floaded to the filesys­tem, it can be reused af­ter the ed­i­tor restart.

This is the core idea of Rust Glancer.

It in­dexes the work­space once and pre­serves re­sults in the filesys­tem, and then when­ever queries need some­thing, they can load the re­quired in­for­ma­tion for the du­ra­tion of the query.

It does­n’t come for free though: frozen work­space analy­sis is slower than lazy in­cre­men­tal by de­f­i­n­i­tion, since load­ing and de­se­ri­al­iz­ing data from filesys­tem is slower than load­ing from mem­ory. To mit­i­gate that, Rust Glancer has to use some tricks: for ex­am­ple, when you type, it does­n’t per­form full blown analy­sis on each key­stroke, it in­stead at­tempts shal­low analy­sis of the cur­rent body and reuses the pre­vi­ous com­plete in­dex. This makes com­ple­tions rea­son­ably fast, but it also means that new items (imports, struc­tures, traits) are not “indexed” un­til you save the doc­u­ment. Which, hope­fully, should not be a prob­lem: you re­ally get used to it fast, and at least in my case it does not feel overly wrong af­ter a while. If that sounds scary, I sug­gest to just try it, it re­ally is not.

For peo­ple who rely on agen­tic work­flows, Rust Glancer is also op­ti­mized for large amount of out-of-ed­i­tor changes. I’m not sure why, but in rust-an­a­lyzer I’ve ob­served that when agents edit the code, in­lay hints can get out of place, and I had the same prob­lem in Rust Glancer ini­tially, but it was re­solved by im­ple­ment­ing a cus­tom file watcher and tweak­ing it some­what. The server also has lower pri­or­ity for out-of-ed­i­tor changes, so agen­tic changes do not cause rapid re-in­dex­ing.

Still, it’s im­por­tant to un­der­stand that Rust Glancer has some ben­e­fits, but also has some draw­backs (besides be­ing in­com­plete, ob­vi­ously) com­pared to rust-an­a­lyzer. Maybe I will man­age to solve some of them even­tu­ally, but it’s highly un­likely that Rust Glancer will ever be­come “just like rust-an­a­lyzer, but bet­ter”. I imag­ine that rust-an­a­lyzer will re­main the de­fault choice for pro­jects that care about com­plete­ness and key­stroke ac­cu­racy, while Rust Glancer will work for peo­ple with weaker ma­chines or peo­ple who are ready for some sac­ri­fices to re­duce RAM us­age.

How and why it hap­pened

I have been writ­ing Rust pro­fes­sion­ally for ~7 years, and since pretty early on I started ob­serv­ing how the com­piler and its tool­ing are de­vel­oped. I’ve made some con­tri­bu­tions to rustc, clippy, and rust-an­a­lyzer, and I’ve spent dozens of hours read­ing its source code just to teach my­self. So I was pretty much aware how big of a pro­ject a Rust LSP is.

At the same time, I have a love-hate re­la­tion­ship with rust-an­a­lyzer. It is ab­solutely beau­ti­ful ex­cept for two things: mem­ory us­age and ini­tial in­dex­ing (especially with build scripts / proc macros en­abled). These prob­lems seem to be brought up quite a lot, but in my case they are even more dras­tic: I have a rather stu­pid work­flow where I have two iden­ti­cal IDEs open on two dis­plays with a bunch of pro­jects in­side a work­space. So the mem­ory con­sump­tion is roughly 2N, and with my last set of the pro­jects I had to work on, rust an­a­lyzer was con­sum­ing 16GB of mem­ory that I, ugh, would pre­fer to have avail­able for other uses; not to men­tion that each time I opened VS Code, my PC fans would go brr be­cause of a ton of par­al­lel in­dex­ing jobs.

At some point I thought that I am fairly con­fi­dent in my Rust knowl­edge, so I prob­a­bly don’t need a full-blown LSP, and can use some­thing sim­pler and more mem­ory ef­fi­cient. I de­cided to try build­ing a “smart ctags for Rust”. I very ex­plic­itly did not want to build an al­ter­na­tive LSP, be­cause of how in­sane of a task it is. Little did I know…

The ini­tial progress was go­ing pretty smoothly: I made use of rust-an­a­lyz­er’s syn­tax li­brary, low­ered items to in­ter­nal rep­re­sen­ta­tions, then built de­f­i­n­i­tion maps and mod­ule struc­ture, got all the de­c­la­ra­tions in­dexed. It was so sur­pris­ingly straight­for­ward that I de­cided to do some prim­i­tive body low­er­ing. Then I de­cided to add very very sim­ple type prop­a­ga­tion. Then it turned out that naive type prop­a­ga­tion does­n’t give me much — but I al­ready had these nice in­lay hints, so I wanted more. Overall, I don’t care about com­plex cases and nightly fea­tures, right? (Right?…). So then came naive trait re­solv­ing via impl header match­ing. It’s quite ad­dic­tive, you get it.

The il­lu­sion, how­ever, broke when I de­cided that it is pretty rea­son­able to ex­pect the fol­low­ing code to be sup­ported as well:

fn mul_­by_two(vals: &[u8]) -> Vec<u8> { vals.iter().copied().map(|v| v * 2).collect() }

The code is pretty sim­ple, but in or­der to sup­port it we need:

Slice type sup­port

Closures / Fn traits

Trait solv­ing

Associated type pro­jec­tion

A bunch of nightly stuff

the last item is funny: I wanted to avoid nightly, but I some­how did­n’t think that std (or sys­root in gen­eral) breathes nightly. Welp.

So all in all, one fea­ture af­ter an­other, I slowly was get­ting from “smart ctags” to a “real LSP”. Probably, the three biggest mile­stones were:

Declarative macro ex­pan­sion (I hate de­clar­a­tive macros now). Thankfully, I was able to reuse most of rust-an­a­lyz­er’s in­fra­struc­ture for that.

Proper type in­fer­ence en­gine. It was a big “oh wow” mo­ment when I truly re­al­ized how type in­fer­ence works (in short: we “link” all re­lated type bind­ings in a big in­fer­ence table, and then we try to get ev­i­dence from all pos­si­ble places, where pro­vid­ing ev­i­dence can solve types for mul­ti­ple places). It was the mo­ment that prob­a­bly brought me the most joy dur­ing the work on this pro­ject so far.

Proper trait solv­ing en­gine. I ini­tially wrote “it’s highly un­likely that we will have a trait solver in this pro­ject”, but then I re­ally wanted to get the above­men­tioned it­er­a­tor ex­am­ple to work prop­erly. I re­sisted in­te­grat­ing trait solver for a while, try­ing to have naive hacks like naive trait impl match­ing + spe­cial­ized han­dlers for std traits, but it was get­ting more and more com­plex while work­ing pretty poorly. Then I gave up and in­te­grated Chalk, which turned out to be sig­nif­i­cantly sim­pler than the whole hi­er­ar­chy I have built. Making Chalk fast was an­other chal­lenge, though.

Somewhat sep­a­rately, prob­a­bly the thing I am most proud of (and the thing that made Rust Glancer pos­si­ble — had I not de­signed it early, the pro­ject would die very quickly) is a cool pro­fil­ing stack that can mea­sure per­for­mance, mem­ory us­age (both na­tively, track­ing ac­tual al­lo­cated ob­jects, and with je­mal­loc), pro­file data on de­mand, and com­pare LSP against rust-an­a­lyzer, as well as a set of bench­marks run­ning in CI. If you’re in­ter­ested, it’s par­tially cov­ered in the docs (1, 2), but I’ll work on a more de­tailed cov­er­age later.

Probably ~1.5 months ago I started us­ing Rust Glancer as my daily dri­ver in­stead of rust-an­a­lyzer. Now, I am happy with its state enough to pre­sent it to a larger au­di­ence.

LLM use

This pro­ject was built with heavy use of LLMs. It is not vibe coded, though. I am ver­i­fy­ing each pull re­quest to make sure that I am happy with the state of the code­base. If you need proofs, you can check the git his­tory: it has PRs with 10k+ lines of diff, but these are mul­ti­ple days apart de­spite the fact that I work on this pro­ject nearly every day since its in­cep­tion. I care about the code, and tbh it would be weird for me to spend 4 months cre­at­ing a Rust LSP if look­ing at the code was­n’t some­thing I do a lot.

I am not go­ing to pre­tend that I am an ex­pe­ri­enced LSP de­vel­oper and the code is per­fect. It is in a state that I can work with, but I un­der­stand that some bits might not be id­iomatic in terms of com­piler tool­ing de­sign. The code has a lot of com­ments, and I tried re­ally hard to make sure that these com­ments are not sloppy but help­ful, be­cause I have to read them all the time; so far the qual­ity is ob­vi­ously not as good as pro­fes­sion­ally writ­ten hu­man docs, but IMHO it’s pretty help­ful and not an­noy­ing to read.

A large part of the jour­ney is learn­ing. LLMs can be pretty good do­main ex­perts, and LLMs know about LSP de­sign much more than I do. At the same time, LLMs are not great at build­ing big pro­jects. So the fol­low­ing loop hap­pened mul­ti­ple times dur­ing de­vel­op­ment:

I build some­thing new.

LLM pro­pos­als seem rea­son­able, so I go with them.

It works but some­thing bugs me.

I think about the de­sign for a while and see a big flaw.

I work with LLM to fix it (sometimes for a week, if the screw up was par­tic­u­larly big — but the big­ger the screw-up is, the more I learn).

So on one hand, if I am to at­tribute code own­er­ship to the LLMs, I can com­plain: “LLMs tried to de­rail the pro­ject so many times!11”. But since it’s my code, I think that the code might get worse at some mo­ments, but as I learn, I get to im­prove it. Which is pretty nor­mal soft­ware de­vel­op­ment flow, just ac­cel­er­ated.

All in all, LLMs are just a tool, and it’s one’s choice to use it re­spon­si­bly or out­source think­ing to it. Given the amount of witch hunt­ing to­day, I have just one re­quest: do not re­duce me to a clanker. It is my code, so if you con­sider it to be slop, call it my slop, not AI.

I am open to crit­i­cism and will hap­pily lis­ten to feed­back: the more I learn, the more I can im­prove the code­base. Whether I use LLMs for that or not does not mat­ter that much, in my opin­ion.

What’s next

The pro­ject is al­ready in a state where it can be a daily dri­ver for some users, but I have rather big plans for it. So in the com­ing re­leases, you might ex­pect:

Further per­for­mance op­ti­miza­tions

Some more mem­ory op­ti­miza­tions (primarily dur­ing in­dex­ing, plus there are a few frag­men­ta­tion is­sues hap­pen­ing af­ter a full in­dex­ing run that I want to fix)

Improved type in­fer­ence / syn­tax sup­port.

Code ac­tions (implement miss­ing trait fields, auto-im­ports, etc).

Potentially proc macro sup­port (I have some weird idea that will not re­quire ac­tual code ex­e­cu­tion, but it’ll take a while to pre­pare).

Some fea­tures are un­likely to be sup­ported though, such as build scripts / proc macros sup­port via proc macro in­vo­ca­tion (e.g. any­thing that re­quires un­trusted code ex­e­cu­tion). I also don’t plan to work on things that are un­nec­es­sary at the cur­rent state of the pro­ject, such as mi­grat­ing to the new trait solver. Niche things like par­tic­u­lar nightly fea­tures will likely be post­poned un­til the pro­ject reaches some de­gree of ma­tu­rity with sta­ble Rust.

Additionally, there is a lot of cool lit­tle tricks I’ve done in Rust Glancer that I’m some­what proud of (aligning al­lo­ca­tion life­times to re­duce mem­ory frag­men­ta­tion, en­gine-as-a-sub­process model to help with both mem­ory frag­men­ta­tion and multi-work­space pro­jects, sharded cache, and oth­ers), so if peo­ple will be in­ter­ested, I’ll be happy to write some blogs telling about how Rust Glancer works un­der the hood. It’s par­tially cov­ered in the docs al­ready (1, 2) if you want to get some info right now.

But in any case, I hope that the pro­ject can be help­ful for some folks al­ready, and for more folks in the fu­ture.

9 hours ago

Nadine Yousif,Senior Canada re­porterand

Toby Mann

Reuters

A fresh wave of US tar­iffs on a wide ar­ray of Canadian goods came into ef­fect on Saturday af­ter a last-minute break­down in trade talks.

Announcing the sus­pen­sion of ne­go­ti­a­tions shortly be­fore the Friday night dead­line, Canadian Prime Minister Mark Carney said he would im­pose rec­i­p­ro­cal tar­iffs on US goods “dollar for dol­lar”.

Carney said “last-minute changes in the US pro­posed terms were un­fair, un­eco­nomic, and called into ques­tion the re­li­a­bil­ity of any deal”.

Trade ne­go­tia­tors had been en­gaged in in­tense talks since July, af­ter President Donald Trump threat­ened to im­pose a 50% levy on nearly $20bn (C$28bn; £14bn) of Canadian im­ports by 19 August.

Trump had tem­porar­ily paused those tar­iffs ear­lier in the week, say­ing the two sides were close to sign­ing a trade deal that was “very good” for both coun­tries.

But min­utes be­fore the dead­line for a deal, Carney said that while “important progress” had been made in the talks it was “not enough to meet our ob­jec­tives for Canadians”.

“As a re­sult, this evening, I have de­cided to sus­pend trade ne­go­ti­a­tions with the US and have di­rected ne­go­tia­tors to re­turn to Ottawa,” he said.

“Last-minute changes in the US pro­posed terms were un­fair, un­eco­nomic, and called into ques­tion the re­li­a­bil­ity of any deal.”

After Carney’s an­nounce­ment US trade rep­re­sen­ta­tive Jamieson Greer said in a state­ment: “Tonight, Canada de­clined to fi­nalise the trade deal un­der the terms agreed ear­lier this week.

“Despite the US of­fer to Canada to re­ceive the best treat­ment of any ma­jor ex­porter to our mar­ket, new de­mands and walk backs of other com­mit­ments by Canada have up­ended the care­ful bal­ance reached in the past days.”

The break­down in talks marks a sig­nif­i­cant shift in tone from ear­lier in the week, when both US and Canadian of­fi­cials sounded op­ti­mistic that a trade deal ben­e­fi­cial for both coun­tries was within reach.

Negotiators were re­port­edly dis­cussing a deal that would re­duce US tar­iffs on Canadian steel and alu­minium from 50% to 25%, and on Canadian au­tos from 25% to 15%.

In ex­change, Carney had asked Canadian provinces to re­store US al­co­hol to store shelves.

The break­down of the talks puts the US-Canada re­la­tion­ship in un­charted ter­ri­tory, and will be a ma­jor test for Carney’s pre­mier­ship.

The two coun­tries have de­vel­oped one of the world’s most deeply eco­nom­i­cally in­te­grated trad­ing re­la­tion­ships, and Canada sends ap­prox­i­mately 70% of its ex­ports south.

But ten­sions be­tween the two ma­jor trad­ing part­ners have been sim­mer­ing since Trump re­turned to of­fice in January last year and un­leashed a wide-rang­ing global pro­gramme of tar­iffs, up­end­ing decades of free trade be­tween Canada and the US.

Now that talks have bro­ken down, Canada will be hit with new 50% US tar­iffs im­posed by Trump us­ing a Depression-era law called the Tariff Act of 1930.

They will be ap­plied on a range of goods, about 5% of Canadian ex­ports, in­clud­ing wine, dairy, ce­ment, cloth­ing and hockey equip­ment.

The new levies are in ad­di­tion to ex­ist­ing tar­iffs the US had al­ready im­posed on Canadian steel and alu­minium, au­tos and lum­ber.

Businesses and stake­hold­ers on both sides of the bor­der had pushed for a deal to be reached, ar­gu­ing that the new US tar­iffs on Canada will be harm­ful to both coun­tries.

In a state­ment, the Canadian Chamber of Commerce called the tar­iffs “a body blow to North American com­pet­i­tive­ness”.

“For a small Canadian ex­porter op­er­at­ing on tight mar­gins, this is­n’t an ab­stract trade dis­pute. It means look­ing at your or­ders, your pay­roll and your em­ploy­ees and ask­ing what you can still af­ford,” said the cham­ber’s pres­i­dent Candace Laing.

Canadians have told poll­sters that they are will­ing to fight for a good deal.

A re­cent sur­vey by firm Abacus Data sug­gested that around 36% of Canadians would sup­port re­tal­i­at­ing to US tar­iffs, while an­other 30% would want the Carney gov­ern­ment to con­tinue ne­go­ti­at­ing.

The prime min­is­ter will need to con­vince Canadians that the eco­nomic pain is worth the cost of stand­ing up to the White House is search of a bet­ter agree­ment in the long run.

Financial an­a­lysts have pro­jected the new 50% tar­iffs could re­duce the coun­try’s GDP by 0.3% to 0.6%.

Doug Ford, pre­mier of Canada’s most pop­u­lous province Ontario, said “the prime min­is­ter has my full sup­port for a strong re­sponse - tar­iff for tar­iff, dol­lar for dol­lar”.

Ontario, which has a large man­u­fac­tur­ing and auto sec­tor, has been among the hard­est hit of Canada’s provinces in this trade dis­pute. The economies of Quebec and British Columbia are also go­ing to be es­pe­cially ex­posed to the new tar­iffs.

British Columbia Premier David Eby said in a state­ment on Friday night that “our po­lite­ness should never be mis­taken for weak­ness”.

“We did­n’t ask for this, but we’ll keep fight­ing for as long as it takes.”

Canada has been en­gaged in on-again, off-again trade ne­go­ti­a­tions with the US for over a year in pur­suit of a deal that would see the US drop or re­duce tar­iffs on these key sec­tors.

The US, mean­while, has been ask­ing for a num­ber of con­ces­sions from Canada, in­clud­ing re­mov­ing its re­main­ing re­tal­ia­tory tar­iffs on American au­tos and ad­just­ing its dairy quo­tas to al­low greater ac­cess for US cheese pro­duc­ers.

It has also asked that the ban on US al­co­hol sales, im­posed last year by most Canadian provinces in re­tal­i­a­tion to Trump’s tar­iffs, be re­moved.

In a state­ment on Saturday, the Distilled Spirits Council of the United States said it was “unfortunate that the Canadian provinces’ con­tin­ued re­fusal to re­turn US spir­its prod­ucts to store shelves has led to this out­come”.

It said ex­ports to Canada fell more than 70% year-over-year from the start of the re­tal­ia­tory ban in March through December 2025.

You may have heard of the speech syn­the­sis com­pany ElevenLabs. Recently a friend men­tioned they knew some­one who worked at a com­pany called TwelveLabs that does AI for video (it feels like it must have been in­tended to play on the fact that ElevenLabs does au­dio, but I don’t know for sure). Jokingly, I googled “thirteenlabs” and was sur­prised to find an AI for 3D scenery pro­ject. I googled “fourteenlabs”. Another AI startup…?? How far does this go?

Numbers 0 – 99 an­no­tated with links to com­pa­nies us­ing it + “labs” as their name. A link has a back­ground if the com­pany is AI-related.

The com­pany must have some on­line pres­ence (not nec­es­sar­ily a web­site, but usu­ally one).

The word “labs” (or some­times “lab”) must ei­ther go af­ter or be­fore the num­ber (spelled or nu­meric). There’s plenty of com­pa­nies named af­ter a num­ber - what I’m af­ter is the weird trend of num­ber + “labs”.

If there were mul­ti­ple, I picked the one that was most sim­i­lar to ElevenLabs (and thus more likely to have got­ten the name in­spi­ra­tion from there?).

Every com­pany sort of wants to be an AI com­pany now, so it’s hard to def­i­nitely say what com­pa­nies are AI re­lated. I marked it if the do­main had a .ai TLD or if all of their main prod­ucts in­volved AI in a cen­tral ca­pac­ity.

This re­ally just raised more ques­tions than an­swers. Why is this such a pop­u­lar nam­ing scheme? Are peo­ple in­de­pen­dently ar­riv­ing at this nam­ing scheme? Why call your AI startup “68labs”? Why are the sev­en­ties so much more dense than the rest of the higher num­bers? Also, I’m tempted to spec­u­la­tively buy up do­mains like “twentyfivelabs” or “thirtytwolabs”…

One fun dis­cov­ery: among all the in­cred­i­bly same-y startup web­sites, there is sev­en­ty­onelab.com. Alongside the usual “all rights re­served” text, it po­litely in­forms you it is

best viewed in Netscape 4/0+ or IE 5.0+

best viewed in Netscape 4/0+ or IE 5.0+

Amazing. It looks like a de­sign/​web­dev port­fo­lio from the early 2000s and is full of fun lit­tle sites. I par­tic­u­larly like the aes­thetic of the land­ing page and the other ver­sions of it in the “little pro­ject” sec­tion. They sort of re­mind me of what I re­cently heard re­ferred to as the “vectorheart” aes­thetic (like what you’d see on 2000s IDM al­bum cov­ers!).

Free, open source and per­for­mant multi-agent har­ness, works with your ex­ist­ing sub­scrip­tions (uses hourly lim­its).

Munder Difflin uses CLI agents run­ning on your com­puter to do any­thing you can do

Supports 12 CLI agent providers off the shelf, more com­ing soon

Claude Code

Codex

Grok

Kimi Code

Antigravity

Qwen

Gemini CLI

OpenCode

Crush

Pi

Copilot

Cursor

PRIVATE CLOUD + NETWORK

Get the Teams plan and dou­ble your pro­duc­tiv­ity

Private Cloud: Run agents 24/7 for each team­mate in iso­lated sand­boxes

Private Network: Allow clones of your team to talk to each other au­tonomously (E2E en­crypted)

MAKE CLONES OF YOUR TEAM · THEY WORK 24/7 🔒 E2E

CLICK A CLONE TO INSPECT

4/4 HUMANS · 4 CLONES ON

HOW IT WORKS

Three steps to a sec­ond you.

1

Install your har­ness

One down­load. It wraps the agent CLI you al­ready use and runs on your lap­top. Your code, your keys, your ex­ist­ing sub­scrip­tion — noth­ing leaves your ma­chine.

2

It be­comes you

It cap­tures your work­flow, your tool­ing and what you know. Every clone you run shares that mem­ory, so the next one you spin up starts al­ready know­ing how you work.

3

Your of­fice gets to work

Your clones work around the clock — and when one needs some­thing, it mes­sages an­other. They hand off work, share con­text and un­block each other, all on your own ma­chine.

JIM’S CLONE ⇄ PAM’S CLONE🔒 E2E

JIM’S CLONE

Blocked — need the in­voice-state de­sign to­kens.

03:12 · en­crypted

PAM’S CLONE

Sent — to­kens + edge-case flows in billing/​to­kens.json.

03:12 · en­crypted

✓ un­blocked overnight · PR #147 open

WHAT EACH TEAM MEMBER GETS

Understand your har­ness­es’ ca­pa­bil­i­ties.

Munder Difflin does­n’t give your team one shared bot. It acts as a clone of the in­di­vid­ual and con­trols their com­puter.

KICK IT OFF FROM ANYWHERE

YOU

SLACK INBOX

TRIGGERS

YOUR CLONE · YOUR COMPUTER

GOD or­ches­tra­tor reads · plans · routes

re­search Claude Code

build Codex

re­view Claude Code

git · each agent in its own iso­lated work­tree

MemPalace their mem­ory · their ma­chine · nowhere else

⟳ 24/7

🔒 asks “pricing fi­nal?” 🔒 an­swers 3 AM · he’s asleep

CLONE ⇄ CLONE E2E · same org only

TEAMMATE’S CLONE · THEIR COMPUTER

GOD or­ches­tra­tor reads · plans · routes

sell Grok

draft Kimi CLI

MemPalace their mem­ory · their ma­chine

⟳ 24/7

WHILE YOU’RE BUSY

Real work. Not demos.

🔍

Reviews like you would

Your clone re­views team­mates’ PRs with your stan­dards and your nit­picks — while you’re in a meet­ing.

💬

Answers for you

“How does the billing ser­vice work?” A team­mate’s clone asks yours and gets your an­swer — at 3am, with­out wak­ing you.

🌙

The of­fice never closes

Clones plan, build, hand off, and un­block each other around the clock. You come back to fin­ished threads, not open ques­tions.

👔

You stay the boss

Your clone es­ca­lates only the few de­ci­sions that gen­uinely need a hu­man. Check in oc­ca­sion­ally, an­swer, and it keeps mov­ing.

WHAT EACH NODE CAN DO

Not just for en­gi­neers.

Everything a com­puter does is reach­able from the com­mand line — and CLI agents can drive all of it. So every team­mate gets a clone that does their job, what­ever that job is.

👩‍💻

Developer

Reviews PRs, fixes bugs, ships small fea­tures, babysits CI, keeps docs hon­est.

$ git, tests, de­ploys

🎨

Designer

Audits screens against the de­sign sys­tem, ex­ports as­sets, drafts specs and copy.

$ screen­shots, to­kens, specs

📋

Product man­ager

Writes specs, triages is­sues, keeps boards and docs in sync, preps standup sum­maries.

$ tick­ets, docs, roadmaps

📈

Sales & GTM

Drafts out­reach, preps call briefs, keeps the CRM hon­est, chases fol­low-ups.

$ crm, email, briefs

🗂️

Everyone else

Reports, spread­sheets, files, sched­ul­ing, fol­low-ups — any­thing script­able. Which is every­thing.

For years now one of the most re­li­able com­plaints I hear when I try to drag a team off their ven­dor spe­cific SDK and onto OpenTelemetry is some vari­a­tion of: “why does it seem like this is­n’t done yet?”

Vendor SDKs for ob­serv­abil­ity are, to put it char­i­ta­bly, id­iot-proof. You in­stall the thing, dash­boards just load data, some­one else wor­ries about how all those pieces fit to­gether, and you get on with your life. OpenTelemetry, by con­trast, greets you at the door with a lot of “experimental” stamps and roughly six dif­fer­ent ways to ac­com­plish any given task.

In OpenTelemetry’s de­fense this was never what they were go­ing for as a pro­ject. I’ve al­ways re­spect that they stuck to their guns by at­tempt­ing to build a truly ven­dor ag­nos­tic sys­tem that re­ally does­n’t care what you do with the data. I have never got­ten a sense of a ven­dor be­ing strongly pre­ferred with OTel, which is quite the feat con­sid­er­ing how lu­cra­tive and con­tentious the ob­serv­abil­ity ecosys­tem was. Also con­sid­er­ing that the main­tain­ers of this pro­ject are largely em­ployed by ex­clu­sively those com­pa­nies.

As the years wore on, I started to get ner­vous. Conversations in the se­man­tic-con­ven­tions repo drag on and on and on. Different lan­guages had dra­mat­i­cally dif­fer­ent sto­ries. Golang and Dotnet were first class cit­i­zens, but other lan­guages lagged years be­hind the oth­ers.

I started ask­ing a lot of prob­ing ques­tions be­fore rec­om­mend­ing OpenTelemetry to smaller teams who did­n’t have the time, bud­get, or emo­tional band­width for it. Auto-instrumentation was gen­uinely mag­i­cal, but the cliff be­tween “auto-instrument works” and “now I have to man­u­ally in­stru­ment some­thing” was steep enough that you owed peo­ple a warn­ing be­fore you pushed them off it.

This nar­ra­tive has been go­ing on for awhile in the ob­serv­abil­ity space, a vague sense of “something is wrong in Otel-land”. But let’s try to gen­er­ate some ac­tual data here. Is there an ac­tual prob­lem, or is this some­thing where the per­cep­tion by the com­mu­nity of slow progress is imag­i­nary? Is the prob­lem not enough main­tain­ers, too big of a scope, or some­thing in-be­tween?

My guess when I started was “oh this is your clas­sic open-source bit off more than they can chew”. Not enough main­tain­ers, not enough bud­get. Now there is some of that, but there’s also some­thing else go­ing on.

The ac­tual prob­lem hap­pen­ing in­side of OpenTelemetry is a three way crash. You have a bi­nary sta­bil­ity gate which, when com­bined with a very small bench of ac­tual main­tain­ers means there is un­der­stand­able worry about mark­ing a fea­ture not ex­per­i­men­tal then add on just a mas­sive scope of lan­guages and frame­works they are at­tempt­ing to cover. This cre­ates a per­fect storm where there is an in­cen­tive to ar­gue about po­ten­tial prob­lems a fea­ture might cre­ate since once it is locked in and shipped as sta­ble you can never change them.

How does OpenTelemetry Work

So OpenTelemetry cur­rently is at­tempt­ing to sup­port a dizzy­ing num­ber of lan­guages and frame­works.

OpenTelemetry is a gi­ant pro­ject. It spans dozens of lan­guages, hun­dreds of li­braries, and count­less back­ends. To keep things sane, the pro­ject splits work into two buck­ets:

Core → Maintained di­rectly by the OTel pro­ject. Small, sta­ble, ven­dor-neu­tral, and tightly re­viewed. This is the “spec-defining” sur­face.

Contrib → Community- and ven­dor-con­tributed. Broader, faster-mov­ing, and cov­ers the long tail of in­te­gra­tions.

There ex­ists the otel-col­lec­tor, the thing that runs along the thing so that you can ship logs met­rics and traces. That copies the same rough pat­tern. But for the lan­guages when we’re talk­ing about core vs con­trib this is what we’re talk­ing about.

Stuff that breaks goes in con­trib, stuff that does­n’t break goes into core.

Now the rea­son this causes a con­flict. con­trib is mas­sive overkill for most pro­jects. You don’t want 300 ex­porters to add the one you typ­i­cally need. On the lan­guage side, this is­n’t that big of a prob­lem. pip in­stall open­teleme­try-in­stru­men­ta­tion-flask gives you the stuff you need for flask. However on the col­lec­tor side you end up hav­ing to do the OpenTelemetry Collector Builder to make your own col­lec­tor (or just kinda ride the wave and hope it works out). While cool that this ex­ists, it’s a lot of scope to ask a team to take on.

Process of adding a new fea­ture

So I be­lieve I have cap­tured the work­flow of adding a new fea­ture to OTel. You can check my home­work here:

OpenTelemetry Enhancement Proposal (OTEP) (https://​github.com/​open-teleme­try/​open­teleme­try-spec­i­fi­ca­tion/​tree/​main/​oteps/)

Once the OTEP is ac­cepted, the text goes into the Specification di­rec­tory in the same repo.

After that it seems to go to Semantic con­ven­tions. This seems to be where we get down to the spe­cific de­tails and where most of the long dis­cus­sions seem to live. At this point we’re talk­ing about more or less a per­ma­nent com­mit­ment to this de­sign and where the lock-in process be­comes very hard to change.

Each of the SDKs im­ple­ments the API sur­face that is de­fined in the spec­i­fi­ca­tion. Now some of the SDKs have done 2.0 break­ing changes, so it does seem like the ear­lier “please no 2.0 at all costs” sen­ti­ment has been aban­doned (which I think is smart and good).

Contrib / in­stru­men­ta­tion. This is slightly more mushy. Looks like they should track lat­est API/SDK but each con­trib pack­age may ver­sion in­de­pen­dently so its more flex­i­ble as a de­sign.

Collector + OTLP. The data has to ac­tu­ally go some­where. OTLP (wire pro­to­col) has its own sta­bil­ity life­cy­cle and spec­i­fi­ca­tion (here). Collector com­po­nents have their own sta­bil­ity in their READMEs and as far as I can tell that’s kinda all over the place.

Things I’m not re­ally clear on

It’s un­clear how long the OTEP -> Specification process takes. I’ve looked through the Git his­tory but there does­n’t seem to be any pre­dictable num­ber or cy­cle.

I don’t fully un­der­stand what is the re­la­tion­ship be­tween all these sta­bil­ity com­mit­ments. Does Collector + OTLP group work in lock­step? Can a lan­guage “fall out of scope” if you lag too far be­hind?

Attempting to test it

So be­cause OpenTelemetry is a CNCF pro­ject, I fig­ured it made the most sense to com­pare them to other CNCF pro­jects. My ba­sis for com­par­i­son is Envoy and Prometheus. I have used a hacky Python script I’ve used be­fore for mea­sur­ing the “health” of open-source pro­jects, which is prob­a­bly not the best. However I’ll in­clude a link to the raw data with­out the charts so folks can re­view it and (more than likely) find a prob­lem in what I gen­er­ated.

So we look at 24 months of ac­tiv­ity for Envoy and what we see is a pretty healthy pro­ject. There’s good dis­tri­b­u­tion of au­thors, merg­ers, is­sue closers. phlax is ob­vi­ously pretty im­por­tant to the pro­ject but in gen­eral there’s a good bench of peo­ple to step in if needed. I’ve at­tempted to fil­ter out all the known bot traf­fic.

Let’s com­pare that to one of the OpenTelemetry lan­guages. The ones I have the most pro­fes­sional ex­pe­ri­ence with are Golang and Python, but I hear from a lot of folks in the com­mu­nity that the Ruby and PHP ones strug­gle a lot. This is the PHP one for the same pe­riod.

So we see pretty clearly that there’s way too much con­cen­trated on 2 peo­ple. This is not a healthy open-source pro­ject and they clearly don’t have enough peo­ple to cover the kind of scope OTel needs to cover. Same story with Ruby.

In com­par­i­son the “strongest” OpenTelemetry SDKs in my opin­ion, Golang and Dotnet (although Python is also no slouch) look more healthy.

Golang

So the first is­sue is maybe the least sur­pris­ing. There’s too much con­cen­tra­tion among too few main­tain­ers. Your au­thors should­n’t also be your merg­ers and your is­sue closers. Ideally these tasks should be dis­trib­uted out more evenly.

For what its worth I think the main­tain­ers have done a good job of at­tempt­ing to keep their dis­cus­sions pub­lic. It was very easy for me to find the pub­lic meet­ing notes of the dif­fer­ent groups of main­tain­ers, read through them and see what was go­ing on. I don’t get the sense that these main­tain­ers are try­ing to stop peo­ple from get­ting in­volved as much as the ex­pec­ta­tions of sta­bil­ity have, more or less, frozen the pro­ject in place.

The is­sue is more a clas­sic case of “someone has to pay the main­tain­ers”. The pro­ject is too com­plex for some­one to re­al­is­ti­cally do this as a hobby. I think any pro­ject sign­ing on for such long sta­bil­ity con­tracts can­not turn to the com­mu­nity of hob­by­ists ex­pect­ing as­sis­tance. I can’t join calls and do the things I would be ex­pected to do for a pro­ject of this size and im­por­tance for free. But it also means that the peo­ple do­ing this crit­i­cal work have ex­pec­ta­tions placed on them by their par­ent or­ga­ni­za­tions.

So these SDKs have too few main­tain­ers. But that does­n’t fully ex­plain why it seems to take so long for new fea­tures to get through the stack. My guess for that was that some­where in the process be­tween sub­mis­sion of the new idea and the for­mal­iza­tion of the idea was a long dis­cus­sion that took a mil­lion years.

Conventions about Semantics

So with this level of sur­face area across dif­fer­ent frame­works and lan­guages, it makes sense to con­cen­trate the con­ver­sa­tion about con­ven­tions in one place. That lives here: https://​github.com/​open-teleme­try/​se­man­tic-con­ven­tions

If ven­dor de­bate is caus­ing the slow­down, we should (in the­ory) see this slow­down in PRs here. Then you should see the slow­down ba­si­cally prop­a­gate out. Spoiler alert, I was wrong about this. Big thanks to the OpenTelemetry peo­ple for hav­ing good con­ven­tions on la­bel­ing their PRs which made this much eas­ier.

So if sem­conv is the slow­down, let’s look at the slow­est PRs there.

Yeah some of them are pretty slow, but there are some com­plex top­ics be­ing dis­cussed. However in­ter­est­ingly this slow­down does­n’t re­ally trickle into the SDK/API space, sug­gest­ing that OpenTelemetry is go­ing a good job of keep­ing these con­ver­sa­tions siloed off.

If we look at Python we see that their slow­est PRs aren’t sem­conv re­lated.

In re­al­ity the slow­down for these are the ex­tra re­quired check im­posed by the Approve Public API check which re­quires an­other main­tainer. But that seems ap­pro­pri­ate and takes us back to the ini­tial prob­lem of “not enough main­tain­ers”.

Potential Solutions

So af­ter look­ing at all of this, the pat­tern be­comes clear. A new fea­ture takes a very long time to make it to the end user in OpenTelemetry be­cause they take sta­bil­ity very se­ri­ously, com­bined with a rel­a­tively lim­ited bench of tal­ent to pull from. Once things make it through the en­tire stack, im­ple­ment­ing the API and get­ting that API change through to the end user falls on an over­worked main­tainer pool. So what do we do?

I think one idea worth ex­plor­ing is adding some sort of time-bound beta tier. Basically be­tween the “Experimental” and the “Stable” in the fol­low­ing di­a­gram. The prob­lem is that for end users, due to the ex­tra steps to use Experimental fea­tures, they might as well not ex­ist. 99% of us have no idea when an ex­per­i­men­tal fea­ture is added and we would never en­gage with it. But if I knew the fea­ture would stick around for at least 12 months with­out a re­moval and was more ac­ces­si­ble to me as an end user, it could ac­tu­ally help the pro­ject get more ac­tion­able feed­back.

Basically a fea­ture would go Experimental (pretty low us­age) -> Beta (more ex­posed to the end user than Experimental) -> 12 months -> Removal or Stable.

Now con­fus­ingly Beta ex­ists for Otel but is used for SDKs, not for com­po­nents. Like Rust is a Beta but it seems like Profiles can­not be a Beta. Honestly it’s nearly im­pos­si­ble for me to fig­ure out like what la­bels should ap­ply to what things. I sus­pect no­body re­ally knows. Here’s the ex­pla­na­tion of Beta that I think only ap­plies to SDKs.

Development

Not all pieces of the com­po­nent are in place yet, and it might not be avail­able for users yet. Bugs and per­for­mance is­sues are ex­pected to be re­ported. User feed­back around the UX of the com­po­nent is de­sired, such as for con­fig­u­ra­tion op­tions, com­po­nent ob­serv­abil­ity, tech­ni­cal im­ple­men­ta­tion de­tails, and planned use-cases for the com­po­nent. Configuration op­tions might break of­ten de­pend­ing on how things evolve. The com­po­nent SHOULD NOT be used in pro­duc­tion. The com­po­nent MAY be re­moved with­out prior no­tice.

Alpha

This is the de­fault level: any com­po­nents with no ex­plicit ma­tu­rity level should be as­sumed to be “Alpha”. The com­po­nent is ready to be used for lim­ited non-crit­i­cal pro­duc­tion work­loads, and the au­thors of this com­po­nent wel­come user feed­back. Bugs and per­for­mance prob­lems are en­cour­aged to be re­ported, but com­po­nent own­ers might not work on them im­me­di­ately. The com­po­nen­t’s in­ter­face and con­fig­u­ra­tion op­tions might of­ten change with­out back­ward com­pat­i­bil­ity guar­an­tees. Components at this stage might be dropped at any time with­out no­tice.

Beta

Same as Alpha, but the in­ter­faces (API, con­fig­u­ra­tion, gen­er­ated teleme­try) are treated as sta­ble when­ever pos­si­ble. While there might be break­ing changes be­tween re­leases, com­po­nent own­ers should try to min­i­mize them. A com­po­nent at this stage is ex­pected to have had ex­po­sure to non-crit­i­cal pro­duc­tion work­loads al­ready dur­ing its Alpha phase, mak­ing it suit­able for broader us­age.

Release Candidate

The com­po­nent is fea­ture-com­plete and ready for broader us­age. The com­po­nent is ready to be de­clared sta­ble, it might just need to be tested in more pro­duc­tion en­vi­ron­ments be­fore that can hap­pen. Bugs and per­for­mance prob­lems are ex­pected to be re­ported, and there’s an ex­pec­ta­tion that the com­po­nent own­ers will work on them. Breaking changes, in­clud­ing con­fig­u­ra­tion op­tions and the com­po­nen­t’s out­put, are only al­lowed un­der spe­cial cir­cum­stances. Whenever pos­si­ble, users should be given prior no­tice of the break­ing changes. Stable

The com­po­nent is ready for gen­eral avail­abil­ity. Bugs and per­for­mance prob­lems should be re­ported, and there’s an ex­pec­ta­tion that the com­po­nent own­ers will work on them. Breaking changes, in­clud­ing con­fig­u­ra­tion op­tions and the com­po­nen­t’s out­put, are only al­lowed un­der spe­cial cir­cum­stances. Whenever pos­si­ble, users should be given prior no­tice of the break­ing changes.

Deprecated

Development of this com­po­nent is halted. No new ver­sions are planned, and the com­po­nent might be re­moved from its in­cluded dis­tri­b­u­tions. Note that new is­sues will likely not be worked on ex­cept for crit­i­cal se­cu­rity is­sues. Components that are in­cluded in dis­tri­b­u­tions are ex­pected to ex­ist for at least two mi­nor re­leases or six months, whichever hap­pens later. They also MUST com­mu­ni­cate in which ver­sion they will be re­moved, ei­ther in terms of a con­crete ver­sion num­ber or the date of a re­lease, like: “the first re­lease af­ter 2023 – 08-01”.

Unmaintained

A com­po­nent iden­ti­fied as un­main­tained does not have an ac­tive code owner. Such com­po­nents may have never been as­signed a code owner, or a pre­vi­ously ac­tive code owner has not re­sponded to re­quests for feed­back within 6 weeks of be­ing con­tacted. Issues and pull re­quests for un­main­tained com­po­nents SHOULD be la­beled as such. After 6 months of be­ing un­main­tained, these com­po­nents MAY be dep­re­cated. Unmaintained com­po­nents are ac­tively seek­ing con­trib­u­tors to be­come code own­ers.

In ad­di­tion it is, re­spect­fully, mis­lead­ing to im­ply that Go and Ruby are be­ing main­tained at the same stan­dard. This is­n’t a shot at the Ruby folks — they are do­ing heroic work with what they have. But pre­tend­ing par­ity ex­ists when it does­n’t just cre­ates con­fu­sion and quiet re­sent­ment when a user shows up ex­pect­ing one ex­pe­ri­ence and gets an­other. Being hon­est about main­te­nance tiers would let peo­ple make in­formed choices and might at­tract more help to the other tiers by nam­ing the prob­lem out loud.

Finally I would try to sur­face these prob­lems more openly for OpenTelemetry from the per­spec­tive of “we need more main­tain­ers”. I feel like the peo­ple do­ing this work prob­a­bly knew there was a prob­lem, but it seems like the com­mu­nity at large has no idea that there is a need for frankly more en­gaged ide­ally in­de­pen­dent main­tain­ers and con­trib­u­tors.

OpenTelemetry is a great pro­ject that is do­ing great work. It’s do­ing, frankly, heroic work at this scale with this few peo­ple. But I think in or­der to ac­tu­ally re­place the ven­dor spe­cific SDKs we need to start get­ting a bit more prag­matic about what is re­al­is­tic to do in terms of sta­bil­ity con­tracts and num­ber of lan­guages. I don’t think break­ing changes are as dev­as­tat­ing to the com­mu­nity as these promises im­ply as long as they are com­mu­ni­cated well and I think with this thin of a bench of main­tain­ers, some­thing has to give.

Anyway feel free to check my data for ac­cu­racy and let me know if you find prob­lems!

My fa­ther passed re­cently, and he was twice my age. I am ap­prox­i­mately the same age that he was when I was born, and I am now “the old gen­er­a­tion” - there’s no one left in the gen­er­a­tion above me.

At the same time, I re­cently joined a com­pany that skews younger-than-me. When I joined Google in 2011, I had just turned 30, and was in the main­stream de­mo­graph­ics of Google in 2011. There were a bunch of more se­nior folks, with the very se­nior ones be­ing in their 50s and hav­ing com­pleted stints at Bell Labs. I ad­mired a lot of these “greybeards” (even though this is a sex­ist term - what’s the right fe­male equiv­a­lent? There were a few very se­nior fe­male en­gi­neers that I would love to in­clude).

So per­haps it is nat­ural that I am re­flect­ing on “what were the im­por­tant re­al­iza­tions that I made since my early 20s that had a pro­found im­pact on the way I think about the world”? In some sense: What are the in­sights I had that made me “more ma­ture”, for some pos­i­tive de­f­i­n­i­tion of “mature”?

This post tries to list them.

1. The im­por­tance of un­der­stand­ing your own in­cen­tive struc­ture, and not be­liev­ing every­thing you think.

I re­cently wrote a Twitter thread about the topic. Oppenheimer was very pub­licly guilt-rid­den about the cre­ation of the nu­clear bomb, and von Neumann at some point quipped “some peo­ple pro­fess guilt to claim credit for sin”. In my young years, par­tic­u­larly in sit­u­a­tions when I had 0day that no­body else had, I ag­o­nized about the re­spon­si­bil­ity that comes with hav­ing 0day. Should I fix them? Should I use them for good? Will the world be harmed this way? Or that way?

In the end, it turns out that - while in­di­vid­u­als mat­ter - many ideas have a “time at which they are ripe”, and the ac­tions of the in­di­vid­ual mat­ter less than the in­di­vid­ual thinks in that mo­ment. There is also al­most no way to pre­dict the ways in which what you do im­pacts the broader world.

If you were asked: “Would it be good if this 0day was used to ap­pre­hend a ter­ror­ist?” you would prob­a­bly say “this is good”. If you were asked “would it be good if this 0day is used to ar­rest some­one and then tor­ture and wa­ter­board him 183 times?”, you would prob­a­bly say “this is bad”. So if your 0day was used to cap­ture KSM, it is prob­a­bly good? Or bad? Things get very com­pli­cated very quickly.

Is clos­ing 0days good for so­ci­ety, be­cause it makes every­thing safer? Or is it en­abling op­pres­sion, be­cause buggy sys­tems are eas­ier to by­pass?

There are no good an­swers, and your own in­cen­tive struc­ture will greatly in­flu­ence how you choose your be­liefs. In the end, peo­ple want to be the he­roes of their own story, and at the same time they have basal needs for recog­ni­tion, for ma­te­r­ial goods, etc. - so they will try to con­struct a nar­ra­tive that al­lows them to sat­isfy their basal needs while also re­main­ing the hero of their saga.

Anxiety about the im­pact of your work is self-flat­ter­ing, and you have to rec­og­nize it as such, and keep it in check - it’s sugar for your ego, but his­tory will largely route around you, be­cause while in­di­vid­ual de­ci­sions mat­ter in spe­cific sit­u­a­tions, the over­all flow of his­tory is less sen­si­tive to the in­di­vid­ual than the in­di­vid­ual thinks. The broader les­son, though, is: Do not be­lieve every­thing you think. Examine your own in­cen­tive struc­tures care­fully. Ask your­self what al­ter­na­tive nar­ra­tives for your be­hav­ior and be­liefs could be, es­pe­cially if they con­tra­dict the nar­ra­tive of the heroic saga you’re con­struct­ing for your­self. Carefully weigh­ing the ques­tion “how might I be the vil­lain in this story?” is an im­por­tant and valu­able skill.

Similarly, meta-cog­ni­tion - just ob­serv­ing your own thoughts in a de­tached man­ner, and then be­ing able to in­ter­pret, an­a­lyze, and con­tex­tu­al­ize them with re­gards to your own in­cen­tive struc­tures, is a great skill to cul­ti­vate.

2. Monocausal de­ter­min­ism is an il­lu­sion, and largely does not ex­ist out­side of com­puter de­bug­ging.

The mono­causal de­ter­min­ism that young com­puter en­thu­si­asts get used to is an il­lu­sion that gen­er­a­tions of elec­tri­cal and process en­gi­neers spent their lives per­fect­ing and main­tain­ing. It is be­cause of these en­gi­neers that com­puter sci­en­tists could largely get away with­out prob­a­bil­i­ties or any em­pir­i­cal ground­ing in the past. There is an ar­gu­ment that you have so many nat­ural sci­en­tists that crossed over into AI be­cause CS ed­u­ca­tion was for a long time too fo­cused on rea­son­ing within the de­ter­min­is­tic mono­causal il­lu­sion.

The re­al­ity is: Computing ma­chines are phys­i­cal de­vices, which in­cludes wear & tear, dif­fer­ences in qual­ity be­tween items, and “probabilistically de­ter­min­is­tic be­hav­ior”, e.g. it’ll ap­pear de­ter­min­is­tic most of the time if not shaken too much. If pushed a bit - be it tem­per­a­ture, volt­age, elec­tro­mag­netic fields, or even rapid mem­ory ac­cesses to ad­ja­cent DRAM rows - de­ter­min­ism has a ten­dency to go out of the win­dow, the il­lu­sion col­lapses, and we’re deal­ing with a very dif­fer­ent beast.

FWIW - this also makes me won­der about model align­ment, be­cause even a per­fectly aligned model will be sub­ject to ran­dom bit flips in in­fer­ence, and it’s hard for me to imag­ine that you can main­tain any rea­son­able guar­an­tees in the pres­ence of bit flips to in­op­por­tune val­ues at in­op­por­tune times.

The real world is one where very few things that hap­pen have a sin­gle rea­son, and very few truly de­ter­min­is­tic trans­mis­sion mech­a­nisms. Everything is prob­a­bilis­tic, and every­thing is mul­ti­causal.

Measurement noise is real, ex­per­i­ment de­sign is dif­fi­cult.

Interestingly, if you think about this care­fully, you also re­al­ize that the sci­en­tific method is a clas­si­fier that is in­ten­tion­ally bi­ased against ac­cept­ing some­thing as true - so that we only ac­cept things as true that are be­yond any rea­son­able doubt true.

A some­what fas­ci­nat­ing coro­lary of this is that there ex­ists a large class of true things that will never be sci­en­tif­i­cally shown as true.

3. The di­chotomy be­tween rea­son and emo­tion is a cul­tural con­struct, and nei­ther grounded in neu­ro­science nor in logic.

With some dig­ging, it turns out that the west­ern be­lief that rea­son and emo­tion are two ends of a spec­trum is a purely cul­tural con­struct, as is the be­lief that “higher-order” rea­son needs to reign in “basal” emo­tions, or that “emotions” in­trude on “rationality”.

In most non-west­ern cul­tures, achiev­ing in­te­gra­tion be­tween ra­tio­nal de­lib­er­a­tion and im­pulses and emo­tions is more com­mon, and it turns out that this is much closer to the bi­o­log­i­cal re­al­ity.

From a neu­ro­science per­spec­tive, it is clear that emo­tional val­u­a­tion is part of a larger de­ci­sion-mak­ing ma­chin­ery that tends to not func­tion prop­erly if the emo­tional val­u­a­tion com­po­nent is dam­aged or re­moved. There is also a large com­po­nent where things that your brain strug­gles to ar­tic­u­late ver­bally are trans­mit­ted via emo­tions, as well as ac­tual feed­back from your sen­sory or­gans in your body. Fun trivia: Your gut’s en­teric ner­vous sys­tem con­tains as many neu­rons as the en­tire cere­bral cor­tex of a dog. Your body also for­ward-de­ploys neu­rons in your mus­cles and ex­trem­i­ties, as a form of la­tency op­ti­miza­tion. Your body is feed­ing you ex­tra in­for­ma­tion, and most of this shows up in the shape of emo­tions.

Which brings us to the log­i­cal ar­gu­ment why at­tempt­ing to “remove” emo­tions from de­ci­sion-mak­ing is a bad idea: Clearly, hav­ing the abil­ity of lever­ag­ing more in­for­ma­tion for de­ci­sion-mak­ing will im­prove the qual­ity of de­ci­sions. Attempting to elim­i­nate a par­tic­u­lar source of in­for­ma­tion al­most cer­tainly makes the qual­ity of your de­ci­sions worse.

This is not to say one should act on im­pulse alone, but it is cer­tain that in­te­grat­ing the full spec­trum of in­for­ma­tion - which in­cludes emo­tions - in your de­ci­sions is a wise idea.

I am sure that if I think more care­fully, I will come up with more in­sights, but these three are im­por­tant enough that they show up in my life with as­ton­ish­ing reg­u­lar­ity.

Hope this is help­ful to some­one.

Meta’s busi­ness can be boiled down to four words that be­gin with the let­ter H: hook, hold, har­vest, hide, ac­cord­ing to a lawyer who is pros­e­cut­ing the world’s largest so­cial me­dia com­pany.

The owner of Facebook and Instagram “hooks” in users, “holds” them on its plat­forms for as long as pos­si­ble, “harvests” their data and then “hides” the truth from the pub­lic, she ar­gued.

“Meta’s busi­ness model worked es­pe­cially well for kids,” said Megan O’Neill, a lawyer for the state of California.

Her ac­cu­sa­tion opened the block­buster trial against the US tech com­pany on Tuesday in Oakland, California, just north of Meta’s head­quar­ters in Silicon Valley. California has joined 28 other US states in su­ing the £1tn ($1.36tn) com­pany for al­legedly de­sign­ing ad­dic­tive prod­ucts that lead to chil­dren be­ing harmed.

Eight ju­rors heard from O’Neill and at­tor­neys for Meta this week, along with tes­ti­mony from for­mer em­ploy­ees and a psy­chol­o­gist. The law­suit cen­ters on al­le­ga­tions that the com­pany vi­o­lated US fed­eral child pri­vacy laws and state-level con­sumer pro­tec­tion laws by col­lect­ing data on chil­dren un­der the age of 13 with­out parental per­mis­sion. Over the course of the trial, the jury is ad­di­tion­ally ex­pected to hear from Meta CEO Mark Zuckerberg and Instagram CEO Adam Mosseri.

The threat to Meta is ex­is­ten­tial. If the com­pany is found li­able, dam­ages could be as high as $200bn — an amount equiv­a­lent to the com­pa­ny’s 2025 an­nual rev­enue. The states are also ask­ing that Meta be forced to change the de­sign of its prod­ucts to make them safer for chil­dren, which could have per­ma­nent ef­fects on the com­pa­ny’s busi­ness model and how its so­cial me­dia plat­forms op­er­ate.

Meta has de­nied all al­le­ga­tions. Liza Crenshaw, a spokesper­son for the com­pany, said: “Rather than stick­ing to the facts or the law, the states have in­stead de­cided to chase an out­landish pay­out.”

During open­ing state­ments, Paul Schmidt, an at­tor­ney for Meta, said there is “no dis­pute” peo­ple can strug­gle with so­cial me­dia, but that Meta had “come up with tools to try and ad­dress that”. He added the com­pany does not al­low chil­dren un­der the age of 13 to reg­is­ter for ac­counts on its so­cial net­works and that it had dis­abled more than 1m ac­counts of those young users.

The trial is ex­pected to last six to eight weeks. The pro­ceed­ings will be led by at­tor­neys for the states of California, Colorado, Kentucky and New Jersey. The ju­ry’s role is ad­vi­sory, which means they will give rec­om­men­da­tions to the pre­sid­ing judge, Judge Yvonne Gonzalez Rogers, who will make the fi­nal de­ci­sion on the ver­dict and dam­ages.

Meta faces thou­sands of sim­i­lar US law­suits brought by fam­i­lies, school dis­tricts and other at­tor­neys gen­eral. The com­pany lost the first two of those cases to go to trial in March. In the first, the com­pany was or­dered to pay nearly $1bn to the state of New Mexico for al­low­ing child sex­ual ex­ploita­tion on its plat­forms; and in the sec­ond, it was found li­able for de­lib­er­ately de­sign­ing ad­dic­tive prod­ucts that hooked one young woman and was or­dered to pay her more than $4m.

af­ter newslet­ter pro­mo­tion

The star wit­ness to take the stand in the tri­al’s first week was Arturo Béjar, a safety en­gi­neer at Meta who worked there in two sep­a­rate stints be­tween 2009 and 2021. Since leav­ing, Béjar has been an out­spo­ken critic of the com­pany, tes­ti­fy­ing be­fore a US Senate com­mit­tee and serv­ing as an ex­pert wit­ness in other cases that in­volve so­cial me­di­a’s harm to chil­dren.

In Oakland, Béjar tes­ti­fied that his mo­ti­va­tion for pur­su­ing so­lu­tions for harms to chil­dren was his own teenage daugh­ter’s treat­ment on Instagram. He said she re­ceived un­wanted sex­ual ad­vances and pho­tos of male gen­i­tals as well as misog­y­nis­tic in­sults. Later, she told her fa­ther that re­port­ing these abuses through Instagram’s es­tab­lished processes was ei­ther in­ef­fec­tive or not pos­si­ble.

“Meta is tak­ing a ‘don’t ask, don’t tell’ strat­egy” when it comes to child safety, Béjar tes­ti­fied.

Béjar said that his job of­ten in­cluded brief­ing Zuckerberg and that he had spo­ken with the CEO more than 100 times in the course of his work.

During Béjar’s tes­ti­mony, at­tor­neys for the gov­ern­ment showed the jury an email he sent Zuckerberg in 2021, which out­lined a sur­vey he had con­ducted of teens’ ex­pe­ri­ences on Instagram. The re­sults showed 51% of users said “yes” to hav­ing bad or harm­ful ex­pe­ri­ences within the pre­vi­ous seven days and that con­tent was taken down only 0.02% of the time.

Béjar tes­ti­fied he sent that data to Zuckerberg be­cause, “in my ex­pe­ri­ence, when Mark makes some­thing a pri­or­ity, moun­tains move.”

“Did he ever re­spond to you?” the at­tor­ney asked.

“No,” Béjar replied. “I did­n’t hear back from him.”

Meta fought to bar Béjar from tes­ti­fy­ing at the trial, fil­ing a se­ries of mo­tions to strike his ex­hibits and pre­vent him from tak­ing the stand, all of which were re­jected. In an email to re­porters on Wednesday, Meta con­tin­ued to hound him. The com­pa­ny’s state­ment said Béjar’s tes­ti­mony was not cred­i­ble or re­li­able be­cause he over­in­flated his role at the com­pany and took credit for work he did­n’t do.

After Béjar’s tes­ti­mony wrapped, the jury heard recorded de­po­si­tions from Elena Davis and Natalie Troxel — both for­mer user ex­pe­ri­ence re­searchers for Meta. Jean Twenge, a psy­chol­ogy pro­fes­sor at San Diego State University, also briefly took the stand, with tes­ti­mony sched­uled to con­tinue next week.

Aug 22, 2026•5 min read

by­Suran­jan Das

pop cul­ture­mu­sicphi­los­o­phynot se­ri­ous

The cen­tral moral ques­tion posed in the 2015 song “Sorry” by Justin Bieber, i.e., “Is it too late to say sorry?” is ac­tu­ally not a moral ques­tion at all when viewed through a Kantian lens. The ques­tion in it­self con­tains the an­swer: yes, it is too late to say sorry, not in the sense that has any­thing to do with the na­ture of time, but rather the form of the ques­tion it­self shows that the song fun­da­men­tally mis­un­der­stands what an apol­ogy is.

When we con­sider the ques­tion’s im­pli­ca­tions, Bieber is not ask­ing “What do I owe to the per­son I have wronged?” Instead, the ques­tion pri­mar­ily asks whether say­ing sorry is still worth any­thing. Utilizing a lan­guage of strat­egy in­stead of duty, such as “Could some­one call a ref­eree?” or “one more shot at for­give­ness” fur­ther in­di­cates that the ques­tion might not be asked in good faith.

If one’s apol­ogy be­comes a means to­wards an end, then it is a hy­po­thet­i­cal im­per­a­tive. You ex­pect to gain for­give­ness by us­ing the apol­ogy. A moral oblig­a­tion, how­ever, does not de­pend on whether the de­sired con­se­quence re­mains avail­able. The song’s ques­tion of whether it is too late shows that Bieber mea­sures the value of this apol­ogy by what it can ac­com­plish.

One’s ad­mis­sions of wrong­do­ing do not change this ei­ther. He claims “I made those mis­takes maybe once or twice”, and then ad­mits, “maybe a cou­ple of hun­dred times”. Therefore, Bieber can­not claim ig­no­rance of the fact that he has wronged some­one. However, the song does not seem fo­cused on this. It con­tin­ues, “So let me re­deem my­self tonight” and “I just need one more shot at sec­ond chances.” The con­cern thus shifts im­me­di­ately from the wrong done to the ap­par­ent con­se­quences suf­fered by the wrong­doer. But moral re­demp­tion is not restor­ing one’s pref­er­ences.

If one was act­ing from duty, their maxim would be sim­pler: I have done wrong; there­fore I ought to ac­knowl­edge it. Whether for­give­ness is pos­si­ble is ir­rel­e­vant, and the same goes for whether he gets an­other chance or not. If, in this form, the apol­ogy is valu­able only be­cause it can se­cure a sec­ond chance or re­demp­tion, then it is not a moral duty; it is merely a po­ten­tially use­ful strat­egy.

Further in the song, Bieber clar­i­fies that he’s “missing more than just your body”. This is the real rea­son for which the apol­ogy is be­ing ren­dered. He is not con­fronting the wrong he com­mit­ted; in­stead, he is con­fronting the loss of some­thing he de­sired. His at­tempt to qual­ify his in­ten­tion by claim­ing that his feel­ings of miss­ing en­com­pass “more than just your body” does not solve this prob­lem ei­ther. It just tells us that he val­ues the af­fec­tion, per­haps com­pan­ion­ship, and the gen­eral re­la­tion­ship it­self. But the cat­e­gor­i­cal im­per­a­tive re­quires treat­ing an­other per­son as an end in it­self, not as a means to sat­isfy one’s de­sires. Thus, we can ask our­selves: would the apol­ogy still be ren­dered if Bieber knew for sure there is no sec­ond chance or re­demp­tion? If not, then the apol­ogy was ren­dered not be­cause of the wrong done in the first place, but to re­store what the wrong­doer lost.

In later verses, the lyrics seem to re­in­force this sus­pi­cion. The line “I’ll take every sin­gle piece of the blame if you want me to” is morally re­veal­ing. If one con­sid­ers them­selves re­spon­si­ble for a wrong done, that re­spon­si­bil­ity can not de­pend on the vic­tim’s re­quest. One ei­ther takes the re­spon­si­bil­ity in good faith, or they do not at all. Furthermore, the song states “there is no in­no­cent one in this game for two” which again re­veals that in­stead of of­fer­ing a moral apol­ogy, the song seems to con­sider wrong­do­ing as a con­test in which the guilt of the other party is enough to di­min­ish their own. But an­other per­son’s mis­con­duct can­not al­ter the maxim by which he acted.

Finally, the lyrics “can we both say the words and for­get this?” high­light that the ob­jec­tive is clo­sure rather than moral ac­knowl­edg­ment. Bieber qual­i­fies his po­si­tion again with “I’m not just tryna get you back on me” but the de­nial does lit­tle to re­solve the prob­lem once more. One’s moral worth ab­solutely can­not be es­tab­lished just by de­clar­ing the pu­rity of their mo­tives. The maxim must be ex­am­ined in­stead. If the maxim is “When I have wronged some­one and fear los­ing them, I will apol­o­gize in the hope of re­gain­ing them,” then the apol­ogy re­mains hy­po­thet­i­cal, and more im­por­tantly, fun­da­men­tally self-in­ter­ested.

Thus, the song’s cen­tral ques­tion iron­i­cally re­veals the very moral fail­ure it seems to try to re­pair. “Is it too late?” in this con­text trans­lates to “is an apol­ogy still ca­pa­ble of get­ting what I want?” Once an apol­ogy is eval­u­ated by whether it can se­cure for­give­ness and rec­on­cil­i­a­tion, it ceases to be an ac­tion grounded in duty. So, yes, it is too late to say sorry, as the song con­ceives it.

This, how­ever, does not stop Bieber from do­ing what moral­ity re­quires: ac­cept­ing the wrong with­out de­mand­ing any­thing in re­turn. Instead of ask­ing whether it was too late to say sorry, the moral thing to do would have been to just say it.

However, even though the ques­tion has a clear ob­jec­tive moral an­swer, we can­not at­tribute any moral judg­ment to Bieber as a hu­man be­ing; we can only an­swer the moral ques­tion as posed in the song. If one were to con­sider an­other in­ter­pre­ta­tion of the song it­self, the ques­tion posed might not be in a moral con­text, but rather as a rhetor­i­cal de­vice, i.e., the artist knows the song is not talk­ing about a sin­cere apol­ogy, but rather it is point­ing out that we can of­ten say sorry in­sin­cerely just be­cause we miss more than their body. This can also ex­plain the in­ten­tional de­ci­sion to pair an up­beat dance track with lyrics that, on the sur­face, seem to be about some­thing much more se­ri­ous.

Note: You can find the pub­lic ver­sion his­tory of my ar­ti­cles on my github: link.