European Citizens’ Initiative
10 interesting stories served every morning and every evening.
10 interesting stories served every morning and every evening.
European Citizens’ Initiative
“But I already have a website on Substack,” you argue.
No, no, Substack is just a distribution tool to amplify your website. It should not be your digital home.
In the last few years, I’ve noticed a pattern of writers leaving their websites to make Substack their digital home.
Now, it’s kinda okay if they have bought a domain and linked it to Substack. (Meaning, it’s better than nothing.)
Rachel from Conscious Living is a good example. This way, Substack more or less functions like a content management system (CMS) for you.
However, compared to other CMS it’s very limited, such as the ability to manage your SEO and customize your pages to add more features, but I digress. If you just want a fuss-free platform, this is one way to get it and Substack’s conditions for domains are very reasonable and cost-efficient. As I will explain later, this could change on a dime without warning.
However, there are some writers who are saying: “Hey readers, I’m now writing on Substack, so head on over there (and ignore my website)!”
Some writers do have a website, but link to their Substacks, calling them their “blogs”. If your Substack has a domain name they own, it’s okay, but if it’s xx.substack.com, Substack is saying “All your content are belong to us”.
In conclusion: Writers, don’t do this. It’s short-sighted and unwise and can derail your long-term visibility on the Internet.
The siren call of convenience
Every few years, the internet convinces writers that a new digital paradise has arrived. First, it was social media like Facebook. Then blogging networks like Tumblr. Then it was Medium. More recently, it’s been Substack.
Platforms promise us an eager audience, built-in monetization, a smooth user interface, and a supportive community. As a writer who just wants to focus on writing, it’s incredibly tempting to hand over the keys to our creative kingdoms and let these portals handle everything. (Believe me, I gave in at one point. For years, I just stopped blogging altogether and even gave up a domain that had high traffic! But I got back in 2012 and never left.)
However, this is the truth that has not changed since the dawn of the Internet: When you build your audience entirely on someone else’s platform, you aren’t a homeowner. You are a tenant. Or worse, a digital sharecropper.
And corporate landlords always change the rules eventually. It’s not personal, it’s just business.
The illusion of the safe space
It’s easy to feel secure when a platform is in its golden era. But we’ve watched the downfalls of Twitter, the policy shifts of Reddit, and the changing tides of algorithmic networks. Relying blindly on a centralized portal not owned by you means your life’s work can alter overnight based entirely on a corporate boardroom decision.
When I looked at how fragile our digital ecosystems really are, I realized I needed a space that wouldn’t go “poof” because a company needed to please its investors or shareholders. This realization completely changed my approach, pushing me to protect my content by learning to blog the IndieWeb way.
Your writing needs a permanent homebase—a domain that you own and control. Full stop.
Moving from “renting” to syndicating
The biggest pushback I hear from writers is: “But my website doesn’t have an audience! Substack does.”
But you don’t have to completely abandon social media or platforms like Substack to protect your autonomy (personally, I prefer the word sovereignty but it does sound a tad dramatic).
You just need to change the order of operations. Instead of publishing directly to a portal, you can shift your mindset to POSSE: Publish (on your) Own Site, Syndicate Elsewhere. (I explain the POSSE/PESOS method in an older post.)
By treating your website as the definitive source of truth and using platforms simply as distribution pipes, you get the best of both worlds. I dug deep into this shift when I committed to being an imperfect gardener of my digital garden, exploring how a less market-y way of presenting my content online let me share my wild garden of thoughts without dancing to the algorithm.
A reality check on platform hype
If you are still holding out hope that Substack is “different” from the social platforms that came before it, let’s look at the numbers and behaviors behind the marketing copy.
After spending a significant amount of time observing the platform ecosystem firsthand, I wrote a brutally honest takeaway in What I learned from one year of Substack. The network effects are real, but so is the pressure to conform to what the platform’s ecosystem favors.
This post, by the way, desperately needs to be updated because things have gotten much, much worse since I wrote it.
When you hand your content over to a platform, you have to conform to their rules and their localized biases. For those of us writing from outside the dominant US-centric echo chambers, platform algorithms heavily prioritize specific western narratives, making it incredibly tough for localized or minority voices to be seen unless they conform.
I wrote about this exact frustration recently in Linkblog: Dwelling on the Internet, highlighting how algorithmic complacency forces us into homogenized bubbles.
The flip side — the writers who refused to leave their websites
Each time there’s a new drama on some platform, and writers are shaking their sabers and declaring that they will leave for yet another social media platform they don’t control, I think about writers like John Scalzi.
As of date, John scalzi has been blogging on https://whatever.scalzi.com/ for 28 years!
This sci-fi novelist has maintained a single independent website continuously for nearly three decades; this makes him one of the longest-running, most consistent original bloggers on the internet. Imagine the amount of digital footprint on that website! Unbroken by time or platforms.
(Specifically, he uses wordpress.com like I do, as we both don’t want to bother with the pain of setting up your own self-hosted wordpress website and just want the folks at Automattic to do it.)
He blogs in the classic Indieweb way, though I doubt he is even aware he’s doing it. He treats his social media channels such as X or Bluesky as a way to amplify his website. All roads lead back to https://whatever.scalzi.com/, and this is something I wish every single writer would do.
He wrote recently in Various & Sundry, 6/3/26:
this site acts as my own institutional memory, if I post something about it here it constitutes an official record. I mean, all the posts I ever placed on the former Twitter are now entirely lost to time, since I have gone in and purged my entire timeline there. This site, however, endures. — John Scalzi
this site acts as my own institutional memory, if I post something about it here it constitutes an official record. I mean, all the posts I ever placed on the former Twitter are now entirely lost to time, since I have gone in and purged my entire timeline there. This site, however, endures. — John Scalzi
Breaking free from platform blues
Trying to adapt your presence across various platforms in an ever-shifting digital landscape is exhausting. One minute a platform is a writer’s darling; the next, it’s being boycotted. Railing against a platform’s focus shift or the presence of (long sigh) Nazis is a useless endeavor.
As I noted in Linkblog March 12, 2026: Platform blues, chasing platform purity is an illusion. Tech will change, corporate algorithms will continue to prioritize profit over human connection, and platforms will continue to cycle through hype and decline.
The antidote to this exhaustion isn’t moving to the next shiny new app. It’s anchoring your work on an independent website with open distribution channels like RSS. It also means ruthlessly using platforms as distribution channels. When one collapses or you prefer to just move, it’s easy to just change strategies because your digital home remains unchanged.
Use platforms to find your readers, but bring them back to your house. It’s time to stop digital sharecropping on rented land.
Featured photo is by vivek vk on Unsplash
@openai/codex-security is a CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities in your code. Scan repositories, review changes, track findings over time, and run security checks in CI.
Documentation
Quick start
Requires Node.js 22 or later, Python 3.10 or later, and access to Codex Security.
npm install @openai/codex-security npx codex-security login npx codex-security scan .
For CI, set OPENAI_API_KEY instead of signing in.
If both a ChatGPT sign-in and an API key are available, interactive scans ask which credential to use. CI and other noninteractive scans keep the existing API-key precedence. Select a credential explicitly when needed:
npx codex-security scan . –auth chatgpt npx codex-security scan . –auth api-key
To make your ChatGPT sign-in the automatic default, unset any configured API keys:
unset OPENAI_API_KEY CODEX_API_KEY
Scan history is stored in the Codex Security workbench state directory. If that directory cannot be written, set CODEX_SECURITY_STATE_DIR to a writable directory outside the repository.
TypeScript SDK
import { CodexSecurity } from “@openai/codex-security”;
const security = new CodexSecurity(); const result = await security.run(”.“);
console.log(result.reportPath); await security.close();
For installation, authentication, scan options, and CI setup, see the official documentation.
The Kimi K3 architecture figure for yesterday’s big open-weight model release, along with some observations and thoughts.
Yes, it looks relatively complicated, but it’s essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now)
Yes, it looks relatively complicated, but it’s essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now)
The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it’s already very crowded, but that’s essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention.
The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it’s already very crowded, but that’s essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention.
Kimi K3’s overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details).
Kimi K3’s overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details).
The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost.
The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost.
Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know.
Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know.
Kimi K3 now also has native multimodal support, which is great!
Kimi K3 now also has native multimodal support, which is great!
There are several other interesting training tidbits in the technical report, but that’s it from the architecture front so far. A really great release overall.
Source: website version of my Substack note.
Read Next
A Few Notable Open-Weight Models This Week
Short note on the architectures of six new open-weight models, including Nanbeige 4.2, Laguna S 2.1, Motif-3-Beta, Solar Open 2, Antares 1B, and BTL-3.
Correction for Listing 6.5 in Build a Reasoning Model From Scratch
Short correction note for the random seed in Listing 6.5 on page 198 of Build a Reasoning Model From Scratch.
Inkling: A New Open-Weight 975B MoE with a Few Surprises
Short note on Thinking Machines Lab’s 975B Inkling model, including benchmarks, sparse MoE design, short convolutions, RMSNorm, and position bias.
A lightweight userscript that adds Hacker News discussions to any article.
HNewhere detects Hacker News stories, loads comments into a sidebar, and lets you browse discussions without leaving the page.
Install
Install HNewhere
Requires a userscript manager such as Tampermonkey, Violentmonkey, or Userscripts.
Features
Opens HN discussions beside articles
Automatically detects matching Hacker News stories
Tracks links opened from Hacker News
Resizable sidebar
Collapsible comments
Preserves sidebar width
Reply links open directly to HN
Install
Install a userscript manager:
Userscripts (Safari) Tampermonkey Violentmonkey
Install a userscript manager:
Userscripts (Safari)
Tampermonkey
Violentmonkey
Install or paste HNewhere.user.js
Install or paste HNewhere.user.js
Visit an article with a Hacker News discussion.
Visit an article with a Hacker News discussion.
Requirements
Browser with userscript support
Access to:
Hacker News API HN Algolia search API
Hacker News API
HN Algolia search API
License
MIT
A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣q⟩\lvert q\rangle is a column vector, ⟨k∣\langle k\rvert is a row vector, ⟨k∣q⟩\langle k\rvert q\rangle is a number, and ∣v⟩⟨k∣\lvert v\rangle\langle k\rvert is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space.
Modern linear attention variants are complex, and a upon first glance it is not so easy to see what they are designed to achieve. For reference here is the state update equation for Kimi Delta Attention (KDA):
S~t=St−1Diag(αt)\widetilde S_t = S_{t-1}\operatorname{Diag}(\alpha_t) ∣v^t⟩=S~t∣kt⟩\lvert\widehat v_t\rangle = \widetilde S_t\lvert k_t\rangle ∣et⟩=βt(∣vt⟩−∣v^t⟩)\lvert e_t\rangle = \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right) St=S~t+∣et⟩⟨kt∣S_t = \widetilde S_t+\lvert e_t\rangle\langle k_t\rvert ∣ot⟩=St(dk−1/2∣qt⟩)\lvert o_t\rangle = S_t\left(d_k^{-1/2}\lvert q_t\rangle\right)
The reason they are so difficult to understand is that this is the latest in a family of linear attention variants that have been developed over the last few years and the complexity of them has inevitably ballooned such that from the outside the latest variants appear inaccessible.
In this post we are going to walk through the DeltaNet family of linear attention variants, two of which are used by the latest Qwen and Kimi model families, and show how you might have arrived at the same equations by asserting simple things about your hidden state.
That is the route we will take:
softmax attention → linear attention → DeltaNet → Gated DeltaNet → KDA
Only after deriving KDA will we turn to the recurrent and chunkwise Triton programs that execute it.
1. Begin with quadratic attention
For a query at token tt, ordinary causal softmax attention is
ati=exp (s⟨ki∣qt⟩)∑j≤texp (s⟨kj∣qt⟩),s=dk−1/2,∣ot⟩=∑i≤tati∣vi⟩.\begin{aligned} a_{ti} &= \frac{ \exp\!\left(s\langle k_i\rvert q_t\rangle\right) }{ \sum_{j\leq t} \exp\!\left(s\langle k_j\rvert q_t\rangle\right) }, \qquad s=d_k^{-1/2},\\ \lvert o_t\rangle &= \sum_{i\leq t}a_{ti}\lvert v_i\rangle. \end{aligned}
Every attention weight is a scalar. It measures the similarity between one key and one query, then softmax turns all of the scores for that query into a distribution. The output is a weighted sum of value vectors.
Over a sequence of length TT, there are T2T^2 key-query pairs. During autoregressive inference we can cache the keys and values instead of recomputing them, but the cache still grows with the sequence and every new query still has to inspect the entire history.
The obstacle to rearranging this computation is the softmax. Its denominator depends jointly on the current query and every earlier key. So, for the moment, remove it.
1.1 Remove the softmax
For clarity, absorb the constant scale ss into the query. The deliberately bare version of attention is then
∣ot⟩=∑i≤t⟨ki∣qt⟩∣vi⟩.\lvert o_t\rangle = \sum_{i\leq t} \langle k_i\rvert q_t\rangle \lvert v_i\rangle.
The scalar inner product can move to the right:
∣ot⟩=∑i≤t∣vi⟩⟨ki∣qt⟩=(∑i≤t∣vi⟩⟨ki∣)∣qt⟩.\begin{aligned} \lvert o_t\rangle &= \sum_{i\leq t} \lvert v_i\rangle \langle k_i\rvert q_t\rangle\\ &= \left( \sum_{i\leq t} \lvert v_i\rangle\langle k_i\rvert \right) \lvert q_t\rangle. \end{aligned}
Everything that depends on the past can now be collected into one matrix of a fixed size V×KV \times K:
St=∑i≤t∣vi⟩⟨ki∣\boxed{ S_t = \sum_{i\leq t} \lvert v_i\rangle\langle k_i\rvert }
and attention becomes a recurrent write followed by a read:
St=St−1+∣vt⟩⟨kt∣,∣ot⟩=St∣qt⟩.\boxed{ \begin{aligned} S_t &= S_{t-1} + \lvert v_t\rangle\langle k_t\rvert,\\ \lvert o_t\rangle &= S_t\lvert q_t\rangle. \end{aligned} }
The identity
(∣v⟩⟨k∣)∣q⟩=⟨k∣q⟩∣v⟩\left(\lvert v\rangle\langle k\rvert\right)\lvert q\rangle = \langle k\rvert q\rangle\lvert v\rangle
is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state StS_t.
This is linear in sequence length rather than quadratic: scan the tokens once, updating the same dv×dkd_v\times d_k state at every step. We have paid for that efficiency by discarding softmax’s normalization and selectivity. More sophisticated linear-attention methods use feature maps and normalizers, but this unadorned form exposes the memory problem that motivates DeltaNet.
1.2 Addition is not assignment
Suppose we write a pair ∣vt⟩⟨kt∣\lvert v_t\rangle\langle k_t\rvert and immediately query the new state with that same key:
St∣kt⟩=(St−1+∣vt⟩⟨kt∣)∣kt⟩=St−1∣kt⟩+∣vt⟩⟨kt∣kt⟩⏟1=St−1∣kt⟩+∣vt⟩.\begin{aligned} S_t\lvert k_t\rangle &= \left( S_{t-1} + \lvert v_t\rangle\langle k_t\rvert \right) \lvert k_t\rangle\\ &= S_{t-1}\lvert k_t\rangle + \lvert v_t\rangle \underbrace{\langle k_t\rvert k_t\rangle}_{1}\\ &= S_{t-1}\lvert k_t\rangle+\lvert v_t\rangle. \end{aligned}
The write does not make the memory return ∣vt⟩\lvert v_t\rangle. It adds ∣vt⟩\lvert v_t\rangle to whatever the memory already returned.
If the old state already produced the correct value, the additive write makes the new state produce twice that value. More generally, keys are not mutually orthogonal, so every write can interfere with previous writes. Linear attention has given us a compact associative memory, but its update behaves like += when what we want is closer to =.
2. DeltaNet: write the error, not the value
DeltaNet replaces the unconditional linear-attention write with a delta-rule correction. There are two useful ways to derive it.
2.1 Derivation one: demand that the write can be read back
Before writing token tt, ask the memory what it currently associates with the new key:
∣v^t⟩=St−1∣kt⟩.\lvert\widehat v_t\rangle = S_{t-1}\lvert k_t\rangle.
If we want the memory to return ∣vt⟩\lvert v_t\rangle, we should not add the whole value. We should add only the difference:
∣vt⟩−∣v^t⟩.\lvert v_t\rangle-\lvert\widehat v_t\rangle.
Introduce a learned write strength βt∈[0,1]\beta_t\in[0,1] and define
∣et⟩=βt(∣vt⟩−St−1∣kt⟩).\lvert e_t\rangle = \beta_t \left( \lvert v_t\rangle - S_{t-1}\lvert k_t\rangle \right).
Then write this error at the current key:
St=St−1+∣et⟩⟨kt∣.\boxed{ S_t = S_{t-1} + \lvert e_t\rangle\langle k_t\rvert. }
Now immediately read the same key:
St∣kt⟩=St−1∣kt⟩+∣et⟩⟨kt∣kt⟩=(1−βt)St−1∣kt⟩+βt∣vt⟩.\begin{aligned} S_t\lvert k_t\rangle &= S_{t-1}\lvert k_t\rangle + \lvert e_t\rangle \langle k_t\rvert k_t\rangle\\ &= (1-\beta_t)S_{t-1}\lvert k_t\rangle + \beta_t\lvert v_t\rangle. \end{aligned}
When βt=1\beta_t=1, the result is exactly ∣vt⟩\lvert v_t\rangle. Smaller βt\beta_t moves the old prediction partway towards the target.
The correction is also local in key space. For any query ∣x⟩\lvert x\rangle orthogonal to the current key,
⟨kt∣x⟩=0⟹(St−St−1)∣x⟩=∣et⟩⟨kt∣x⟩⏟0=0.\langle k_t\rvert x\rangle=0 \quad\Longrightarrow\quad (S_t-S_{t-1})\lvert x\rangle = \lvert e_t\rangle \underbrace{\langle k_t\rvert x\rangle}_{0} =0.
So the rank-one write changes the response in the selected key direction while leaving every orthogonal direction alone.
2.2 Derivation two: take one step on reconstruction loss
The same update falls out of an online learning objective. Treat the current key-value pair as one training example for the linear map SS:
Lt(S)=12∥S∣kt⟩−∣vt⟩∥22.\mathcal L_t(S) = \frac12 \left\| S\lvert k_t\rangle-\lvert v_t\rangle \right\|_2^2.
Its gradient with respect to the state is
∇SLt(S)=(S∣kt⟩−∣vt⟩)⟨kt∣.\nabla_S\mathcal L_t(S) = \left( S\lvert k_t\rangle-\lvert v_t\rangle \right) \langle k_t\rvert.
This is visibly an outer product: a value-space prediction error times the key bra at which that error was observed. Take one gradient-descent step of size βt\beta_t from St−1S_{t-1}:
St=St−1−βt∇SLt(St−1)=St−1−βt(St−1∣kt⟩−∣vt⟩)⟨kt∣=St−1+βt(∣vt⟩−St−1∣kt⟩)⟨kt∣.\begin{aligned} S_t &= S_{t-1} - \beta_t\nabla_S\mathcal L_t(S_{t-1})\\ &= S_{t-1} - \beta_t \left( S_{t-1}\lvert k_t\rangle-\lvert v_t\rangle \right) \langle k_t\rvert\\ &= S_{t-1} + \beta_t \left( \lvert v_t\rangle-S_{t-1}\lvert k_t\rangle \right) \langle k_t\rvert. \end{aligned}
This is exactly the update we got by requiring immediate reconstruction. The two interpretations are the same:
as a memory operation, βt\beta_t controls how strongly to replace the old association;
as online learning, βt\beta_t is the step size;
as linear algebra, the change is a rank-one outer product.
2.3 The DeltaNet state transition
Expanding the error exposes DeltaNet as a structured state transition plus a new input:
St=St−1+βt(∣vt⟩−St−1∣kt⟩)⟨kt∣=St−1(I−βt∣kt⟩⟨kt∣)+βt∣vt⟩⟨kt∣.\begin{aligned} S_t &= S_{t-1} + \beta_t \left( \lvert v_t\rangle-S_{t-1}\lvert k_t\rangle \right) \langle k_t\rvert\\ &= S_{t-1} \left( I-\beta_t\lvert k_t\rangle\langle k_t\rvert \right) + \beta_t\lvert v_t\rangle\langle k_t\rvert. \end{aligned}
For a unit key, I−βt∣kt⟩⟨kt∣I-\beta_t\lvert k_t\rangle\langle k_t\rvert has eigenvalue 1−βt1-\beta_t in the current key direction and eigenvalue 11 in every orthogonal direction. It removes the old association along the current key before adding the new one.
DeltaNet fixes the write. It does not yet fix the lifetime of the state.
3. Gated DeltaNet: sometimes old information should disappear
The linear state compresses the whole history into one matrix. A read
St∣q⟩=∑i≤t⟨ki∣q⟩∣vi⟩S_t\lvert q\rangle = \sum_{i\leq t} \langle k_i\rvert q\rangle\lvert v_i\rangle
cannot choose to skip an individual old token after that token has been folded into StS_t. Every stored direction that overlaps the query contributes. The delta rule can correct the state around the current key, but stale information in other directions remains available and can distort future reads.
We therefore need a way to forget the old state before using it. Let αt∈[0,1]\alpha_t\in[0,1] be a learned scalar retention gate:
S~t=αtSt−1.\widetilde S_t = \alpha_t S_{t-1}.
Run the same delta rule against this gated state:
S~t=αtSt−1,forget,∣v^t⟩=S~t∣kt⟩,predict,∣et⟩=βt(∣vt⟩−∣v^t⟩),correct,St=S~t+∣et⟩⟨kt∣,write.\boxed{ \begin{aligned} \widetilde S_t &= \alpha_tS_{t-1}, &&\text{forget},\\ \lvert\widehat v_t\rangle &= \widetilde S_t\lvert k_t\rangle, &&\text{predict},\\ \lvert e_t\rangle &= \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right), &&\text{correct},\\ S_t &= \widetilde S_t+\lvert e_t\rangle\langle k_t\rvert, &&\text{write}. \end{aligned} }
This is Gated DeltaNet. The order matters: forget first, predict from the retained state, then correct that prediction. If we predicted before forgetting, the error would describe a different memory from the one we update.
Expanding the recurrence gives
St=αtSt−1(I−βt∣kt⟩⟨kt∣)+βt∣vt⟩⟨kt∣.S_t = \alpha_tS_{t-1} \left( I-\beta_t\lvert k_t\rangle\langle k_t\rvert \right) + \beta_t\lvert v_t\rangle\langle k_t\rvert.
The delta rule gives targeted replacement; the scalar gate gives global erasure. They solve different problems and are complementary.
But αt\alpha_t still makes one decision for the entire matrix. The model must retain or forget every key channel at the same rate.
4. Kimi Delta Attention: forget each channel independently
Kimi Delta Attention replaces Gated DeltaNet’s scalar retention with a vector αt∈[0,1]dk\alpha_t\in[0,1]^{d_k}. Put the vector on the diagonal:
Dt=Diag(αt)∈Rdk×dk.D_t = \operatorname{Diag}(\alpha_t) \in\mathbb R^{d_k\times d_k}.
Our state maps keys to values, so the key channels are the columns of SS. Right-multiplication applies a different retention factor to every one:
S~t=St−1Dt.\widetilde S_t = S_{t-1}D_t.
Everything else is the delta rule we have already derived:
S~t=St−1Dt,forget each key channel,∣v^t⟩=S~t∣kt⟩,predict,∣et⟩=βt(∣vt⟩−∣v^t⟩),correct,St=S~t+∣et⟩⟨kt∣,write,∣ot⟩=St(s∣qt⟩),s=dk−1/2,read.\boxed{ \begin{aligned} \widetilde S_t &= S_{t-1}D_t, &&\text{forget each key channel},\\ \lvert\widehat v_t\rangle &= \widetilde S_t\lvert k_t\rangle, &&\text{predict},\\ \lvert e_t\rangle &= \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right), &&\text{correct},\\ S_t &= \widetilde S_t+\lvert e_t\rangle\langle k_t\rvert, &&\text{write},\\ \lvert o_t\rangle &= S_t(s\lvert q_t\rangle), \qquad s=d_k^{-1/2}, &&\text{read}. \end{aligned} }
That is KDA. Compared with Gated DeltaNet, the conceptual change is only the promotion
αt⟶Dt=Diag(αt).\alpha_t \quad\longrightarrow\quad D_t=\operatorname{Diag}(\alpha_t).
The effect is substantial: one channel can be cleared while another is retained.
4.1 Why the transition is diagonal-plus-low-rank
Expand the KDA correction:
St=St−1Dt+βt(∣vt⟩−St−1Dt∣kt⟩)⟨kt∣=St−1Dt(I−βt∣kt⟩⟨kt∣)⏟At+βt∣vt⟩⟨kt∣.\begin{aligned} S_t &= S_{t-1}D_t + \beta_t \left( \lvert v_t\rangle - S_{t-1}D_t\lvert k_t\rangle \right) \langle k_t\rvert\\ &= S_{t-1} \underbrace{ D_t \left( I-\beta_t\lvert k_t\rangle\langle k_t\rvert \right) }_{A_t} + \beta_t\lvert v_t\rangle\langle k_t\rvert. \end{aligned}
The key-space transition is
At=Dt−βtDt∣kt⟩⟨kt∣=Dt−∣bt⟩⟨at∣,\begin{aligned} A_t &= D_t-\beta_tD_t\lvert k_t\rangle\langle k_t\rvert\\ &= D_t-\lvert b_t\rangle\langle a_t\rvert, \end{aligned}
where
∣bt⟩=Dt∣kt⟩,⟨at∣=βt⟨kt∣.\lvert b_t\rangle=D_t\lvert k_t\rangle, \qquad \langle a_t\rvert=\beta_t\langle k_t\rvert.
So AtA_t is a diagonal matrix minus a rank-one matrix: a diagonal-plus-low-rank, or DPLR, transition. “DPLR” describes the dk×dkd_k\times d_k transition acting on key space. The memory state itself is still the dv×dkd_v\times d_k matrix StS_t.
The full journey can now be summarized compactly:
The implementation usually stores gt=logαtg_t=\log\alpha_t with gt≤0g_t\leq0, then obtains the retention factors as exp(gt)\exp(g_t). In the transposed dk×dvd_k\times d_v layout used by the reference code, the recurrence is only five lines:
state = state * g_t.exp().unsqueeze(-1) prediction = einsum(“bhkv,bhk->bhv”, state, k_t) residual = beta_t.unsqueeze(-1) * (v_t - prediction) state = state + einsum(“bhk,bhv->bhkv”, k_t, residual) output = einsum(“bhk,bhkv->bhv”, q_t * scale, state)
See the official naive_recurrent_kda reference.
Half-Life has finally landed for PowerPC based Macintosh computers 28 years after it’s original release! Half-Life is a story driven first-person shooter, that follows scientist Gordon Freeman, who is a theoretical physicist trying to survive and escape the Black Mesa Research Facility after a failed experiment opens a portal to an alien dimension.
The game was originally planned to be released for Mac OS 9 by Valve in 1999, but was cancelled shortly before launch. Valve didn’t bring Half-Life to Mac OS X until 2013 which was well into the intel based CPU era, and now we finally have a release for PowerPC based machines.
This port has been accomplished by GitHub user doctashay using a fork of Xash3D FWGS, (a re-implementation of the GoldSrc engine). It’s playable from start to finish, includes multiplayer support, a demo of Uplink, along with downloads for Blue Shift and Opposing Force.
This release supports G3 and G4 PowerPC based computers running Mac OS 9.0 or later. Performance heavily depends on the GPU present in your machine, iMacs, iBooks etc. may struggle with performance you have less than 8Mb VRAM.
This release also includes:
Half-Life
Half-Life
Half-Life: Blue Shift
Half-Life: Blue Shift
Half-Life: Opposing Force
Half-Life: Opposing Force
This is a huge achievement for the Macintosh gaming community, and doctashay deserves considerable credit for the work involved in bringing Half-Life to the PowerPC platform.
Exploring the tools of the trade
July 2026
Ahh, the demo scene - a digital art subculture, a motley gang of creative nerds, and a favourite pastime of mine. So much amazing art, music and code has been produced by sceners, and anyone who comes into contact with the scene might start to wonder exactly how - apart from hours of dedicated grind, of course.
The scene has a long and storied tradition of building its own tools. Sometimes from scratch, sometimes by stealing ideas or even code from existing offerings. In a combination of teenage inexperience, old habits, a penchant for experimenting and a desire to make things look cool, this has resulted in some rather peculiar user interfaces.
A few of them are presented below. Most of them are for the Amiga, but several other platforms are represented as well.
Elite Sinus Producer
Not wanting to be accused of clickbaiting, let’s start off with one of the main attractions: Elite Sinus Producer (sinus here is to be understood as sine), made by Ipec Elite for the Amiga. Demos are usually described as “real time”, which is true in the sense that they’re (almost) never just animation players, and that demo effects are produced, frame by frame, from code. However, to achieve seemingly impossible technical feats, extensive “cheating” is employed. The most common cheat is probably the so called precalc, meaning that instead of doing complex maths on a 7 MHz (or even slower) CPU, lookup tables are utilized. A plethora of tools for creating such lookup tables exist. This is one of them.
When first starting Elite Sinus Producer, the user is met with this menu. Upon pressing an F-key to make a selection, the corresponding menu option is highlighted and a sample of a cuckoo clock is played. Loudly.
Here I’ve selected the “Flower” option. Pretty, no? This can then be saved to disk as (presumably) assembly source code. Any sprite would look nice when moving along this path!
There’s also a handy help screen, which is so outlandish I had to grab a short film clip of it: Part of the background consists of moving blue raster bars, and the other part flashes between red and cyan. This obviously helps immensely when reading the text, displayed in a font designed with nothing but legibility in mind.
Text Based Interfaces
Plenty of scene related tools are either command line utilities or predominantly text based. First out, we have the assemblers. There was a wide variety of assemblers for the Amiga, but the scene always favoured Seka, Asm-One and other derivatives of the same concept. There were so many different versions and hacks (Trash’m-one, for example) that the sprawling family tree rivals that of Unix systems.
Here’s Seka 2.0, and as we can see, it’s based on a commercial assembler. This type of assembler always asks the user for the size of the working memory to be allocated. They then enter a command line mode, which can be used to examine RAM memory and CPU registers, and load source files into the editor proper.
Here’s AsmOne in one of its many incarnations. It’s quite similar to Seka (in fact, it’s “Seka-Updated”), but has more built-in commands and presumably other improvements as well (I’ve never been much of a coder). Interface-wise, it opens its own screen, as opposed to running in a window on the default Workbench screen.
What if you found a piece of cool music or graphics in, say, a game? What if you wanted to steal some samples, or a sprite, or perhaps just save an entire tune for easy listening? Then you’d need a ripper - a tool for hunting through your computer’s memory, looking for remnants of such data after quitting the game. There were tons of various such rippers for the Amiga. Here’s Multi-Ripper, which has a fairly representative set of features.
Here’s another type of ripper, specifically written to look for Seka assembly sources in memory after the computer had crashed. The Amiga, like most other home computers, had no memory protection, and demo coding is a notoriously crash-prone activity. Saving often was common practice, but even seasoned coders sometimes messed up and forgot. With a bit of luck, the code could be extracted from memory after a warm reboot.
Here’s another type of sine precalculator, called The Sinus Creator. I have no idea if the numbers I’ve entered in the screenshot make sense, but the two-window text interface is interesting. Of course, the result can be saved as a Seka source file.
Music Trackers
Demo music has historically been made in trackers. Far from traditional notation, a tracker lets the musician enter tones along with various modifiers and effects in something that’s more reminiscent of a programming editor rather than regular composing. They also let the user manage instruments, whether sampled or synthesized.
Sample-based trackers on the Amiga have an even more sprawling family tree than that of Amiga assemblers, but they all originate from Karsten Obarski’s commercial Ultimate Soundtracker from 1987. Being commercial and thus costing money, it was soon picked apart by sceners, which resulted in NoiseTracker, which was then revamped into ProTracker, which in turn exists in so many various versions and re-hashed hacks that it’s nigh impossible to keep track (heh) of. The sprawl is even sprawlier than that of Amiga assemblers!
SoundMonitor 1.0 for the Commodore 64 (by Chris Huelsbeck) isn’t strictly speaking a scene release, and wasn’t called “tracker”. However, the interface (with one “track” per available sound channel) is what inspired the previously mentioned Ultimate Soundtracker, and is thus included here for posterity and correctness.
NoiseTracker by the Swedish duo Mahoney and Kaktus wasn’t the first, but it was immensely popular and came to define the tracker experience for years to come. Its legacy lives on just not in Protracker on the Amiga, but on several other platforms as well.
Here’s Protracker’s file picker. Just like in NoiseTracker above, it’s accessed by clicking the “Disk Op.” button in the rather dense interface. It’s hard to describe what a strange experience this UI delivers, because it’s almost - but not quite - intuitive to someone used to more mainstream Amiga programs. It’s easy to misclick, misunderstand or just plain miss things. To illustrate its idiosyncratic design, take note of the vertical EXIT button, which quits back to the main menu. It’s conveniently placed between the up and down arrows used for scrolling the purple file and directory listing.
Here’s Digicomposer 1.0 for the Atari ST/e, which in turn builds on Noisetracker for the Atari. The interface is clearly more than just inspired by its Amiga counterpart. It’s but one of a whole menagerie of trackers for the Atari ST, some “digi” (sample-based) like this one, some for YM-chip based music.
Fasttracker II for MS-DOS is iconic in its own right. With support for Gravis UltraSound and other advanced PC sound hardware, it has features for 16-bit samples, bizarre amounts of sound channels, and even lets the user play a simple version of Snake if they so desire.
Abyss’ Highest Experience is a chiptune tracker for the Amiga, intended to sound like the Commodore 64′s SID chip. The user interface is an interesting crossover between Soundtracker and a more modern, Workbench 2.0-like look.
I don’t know much about Megatizer for the Atari ST, but it sure does look cool!
JamCrackerPro for the Amiga eschewed the traditional tracker UI and opted for a system-friendly, multi-window interface.
Disk Copiers
In order to distribute demos (and pirated software), disk copying was a common activity on the scene. Commodore provided a disk copier in AmigaOS, but it wasn’t always up to the task of copying demos and games that bypassed the file system by writing directly to the tracks of a floppy. Thus, special software was needed!
Like SoundMonitor, X-Copy isn’t strictly a scene release. Although originally written by sceners, it was released commercially and then heavily pirated on the scene. The prominently featured grids contain one square for each track on the disk, displaying the status for copying that particular part of the floppy. When a copy was finished (which could take some time), the program helpfully played a little “boing” sound.
X-Copy was released in many versions during the Amiga heydays. Here’s X-Copy 3.0, apparently in an illicit variety distributed by the cracking group Paradox.
Personally, I preferred D-Copy, mostly because I thought the user interface looked cool (and I still do). It is, as far as I know, a bona fide non-commercial scene product.
Other Tools
A short article like this can only ever scratch the surface of the numerous demo scene tools ever created. Here are but a few more various tools and interfaces, selected for being interesting and/or representative of their kind.
This is Titanics Cruncher for the Amiga. A cruncher uses asymmetrical compression of executable files. This saves space on disk, with the tradeoff being longer loading times, due to the decrunching (decompression) of the executable upon running it. Several other crunchers exist, and on a variety of platforms.
Before the Internet, there was the Bulletin Board System, or BBS. Sceners were usually interested in Elite BBS:es, meaning ones that offered pirated software for download. There were few, if any, Elite BBS:es without cool ANSI (or PETSCII, on the C64) graphics, for example in animated menu screens. Thus, many various ANSI editors popped up for different platforms. This is Digital Intelligence’s Ansi-Editor v2.4 for the Amiga, and it has a very peculiar user interface. The toolbar at the bottom displays the currently active colours, but you can’t select them from there. Click all you want, to no effect - you have to use the pull-down menu to actually pick one.
Before Twitter, Facebook, IRC and even widespread access to modems and BBS:es, scroll texts were the communication medium of choice for the scene. Cool scrollers required cool fonts, and it could be helpful with a dedicated font (or charset) editor. Here’s Charedit by Escape, for MS-DOS.
Home computers often came with custom or otherwise esoteric hardware. The Atari Falcon, for example, sported a Motorola 56001 digital signal processor. DSPdit by tSCc (short for The Sirius Cybernetics Corporation) is an editor and assembler made specifically for DSP56k programming. It uses the standard GEM toolkit, which gives it a very clean and professional look.
The first computer virus for the Amiga was a bootblock virus, infecting the boot sector of floppy disks. As such, it could potentially ruin games and other floppies with custom bootblocks. It was constructed by the Swiss Cracking Association, SCA. The virus spread like wildfire and perhaps SCA was plagued by a guilty conscience, because later they also produced the very first virus killer for the Amiga - designed to counter their own virus. The mega-mighty interface is easy enough to grasp!
As mentioned above, plenty of Amiga demos were so called trackmos, meaning they didn’t use the file system, but rather stored data straight on the tracks of a floppy disk. This soon resulted in various tooling becoming available for creating such trackmos. Mostly it was just code shared between sceners, but there were also complete software suites, like TrackmoDOS by Poison of NOVA. Among other things, it came with this orthodox file manager for writing and deleting files to a trackmo floppy. Being a scener tool, the GUI of course sports some rather spiffy colour gradients.
This is RAW, which isn’t really a tool, but rather a disk magazine, or diskmag. A diskmag is just that - a periodical published on one or more floppy disks. Most of the articles were usually scene related, although some mags branched out and featured everything from short stories and poems to essays about history and politics. RAW was one of the most popular Amiga diskmags during the early 1990s, and as we can see, the UI was shiny and textured long before Frutiger Aero was a thing. It even came with a built-in palette editor (pictured), allowing the user to customize the text colours.
This is FuckPaint, a pixel painter for the Atari Falcon. It is included here solely on the merit of its name, and its equally classy “Analizer” tool.
Although not at all a scene production, Deluxe Paint must be mentioned here. To the best of my knowledge, the scene never produced a pixel painter for the Amiga (except later ports of Grafx2), presumably because nobody saw the need for one. I don’t think a single Amiga user existed that didn’t have a copy of this software, either bought separately, bundled with the machine or just pirated. It was also completely dominant on the PC. There were other Amiga pixel painters, such as Brilliance and Personal Paint, but their combined market share was a mere fraction of this giant, which also dominated graphics creation for games well into the the mid-1990s.
That’s enough demo scene interfaces for one helping. For those still wanting more, I recommend this gallery of utildisk menus.
Take care and happy hacking!
If you managed to put Tailscale on a jailbroken Kindle before it updated too far ahead, you got something pretty great, even if it wasn’t the full Tailscale experience. But good things come to those who wait (or dig around on GitHub).
Open-source developers have improved the Tailscale experience on one of the weakest computers you own. If your Kindle is jailbroken, an updated version of Mitanshu Sukhwani’s Tailscale implementation offers a few new things:
Tailscale SSH enabled by default, so you don’t have to enable USBnetworking SSH and its very obvious default user/password
A proxy mode that lets apps like KOReader to reach other nodes on your tailnet, like a Calibre/OPDS or Wallabag server
A full TUN mode that, on some Kindles, can make Tailscale networking work at the device level
Let’s dig into each one and how to set them up. As before, this is community code working on a very unofficial device state; bring your patience along.
Tailscale on a Kindle, now with proxies
The last time we wrote about Tailscale on a Kindle, the client was basic, but it worked. The Kindle showed up on your tailnet, complete with a green dot in the web admin console. You could reach the Kindle by its Tailscale IP address. You could even SSH into the Kindle over Tailscale, which was handy for further tinkering.
But “reachable via Tailscale” is not the same as “routing all incoming and outgoing traffic across your tailnet,” it turns out. Tailscale on a jailbroken Kindle is typically forced to run in userspace mode, which means it cannot use the device’s own network routing layer, known as TUN mode. You could start Tailscale, and then start an app like KOReader, but when you tried to connect to another Tailscale device, like your Calibre server at 100.x.y.z, it would go like this:
KOReader (or any app) asks the Kindle’s root OS how to reach 100.x.y.z
The Kindle, lacking Tailscale routing, cannot reach that Tailscale IP address
KOReader drops the connection
An update to the Kindle KUAL app by greywolf1499 provides different modes that work around this. Now, when you try to reach another Tailscale IP address on your Kindle, it can go like this:
You start Tailscale in proxy mode
You set up KOReader or another app’s proxy settings to connect to 127.0.0.1:1055
KOReader tells the proxy it wants to reach 100.x.y.z
Tailscale’s daemon tailscaled, listening on port 1055, routes the connection through Tailscale
E-books, articles, and other data flows between your Tailscale-running Kindle and other Tailscale devices
This Tailscale proxy offers two modes, SOCKS5 and HTTP CONNECT, for apps that may prefer one or the other. This opens up a good bit more utility for your more-connected Kindle.
What you can do with a proxied Kindle
A few wild ideas, depending on how dug in you want to get:
Calibre or Wallabag servers, as mentioned
Audiobookshelf connection through KOReader
Use Readest to track reading progress across devices
Linking KOReader’s RSS reader to a a self-hosted feed server
Accessing minimalist dashboards and web pages in the (pretty bad) Kindle browser
Using a Bluetooth keyboard and the kterm app to SSH into tailnet devices
Is that last one all that practical? Not really. But is there a pleasant warmth, knowing that you’ve added the least likely thin client to your what-if kit? For some types, types I know quite well: yes.
The Tailscale plugin for KOReader (with Kobo and PocketBook support)
If you don’t really need any Tailscale powers outside the highly capable KOReader app, check out this Tailscale KOReader plugin. It doesn’t make your Kindle accessible over your tailnet, like the KUAL-based app. But it does automatically create the proxy interfaces that are needed for reaching your content servers from your KOReader-running Kindle—or your Kobo device, or your PocketBook.
I haven’t been able to really try this extension out myself; my 11th-generation standard Kindle doesn’t play well with it at the moment. It’s been “Tested on Kindle PW5/PW6, Kobo, and PocketBook”—it’s nice to see Tailscale come to some other KOReader-friendly devices, too.
Installation is not too hard, at least if you made it this far into jailbreaking already. You copy the plugin into KOReader’s plugins directory, trigger an “Install/Update Tailscale” from KOReader’s menu, copy a Tailscale key into a directory, then toggle Tailscale on in the KOReader menu. From there, you configure KOReader with one of its proxy addresses (127.0.0.1:1055 for SOCKS5, :1056 for HTTP CONNECT), then give other plugins the Tailscale IP addresses you need to reach.
Victoria Riley Barnett’s repository notes that the plugin works great with a SyncThing plugin for KOReader. KOReader is like its own separate OS for jailbroken Kindles at this point,
So now you’ve got a lot more options and weird projects available to you, through this already quite-strange little slab. If you’ve worked up a weirdly useful Tailscale setup on your Kindle, Kobo, or other e-paper device, we’d love to hear about it. Let us know on Reddit, Discord, Bluesky, X, Mastodon, or LinkedIn.
As a member of the Zig core team, one of the most impactful projects I’ve been involved with is the implementation of incremental compilation into the Zig compiler. This feature allows the compiler to detect which individual functions and declarations have changed since a project was last built, recompile only that code, and directly patch the resulting bytes into the output binary, making the rebuild extremely fast.
The Zig project has been working towards this feature for a long time, and over the last few release cycles, it has finally gone from a proof-of-concept quality feature to one which is viable for real-world projects and which most of the Zig core team makes daily use of.
Today, using Zig’s incremental compilation, you can make changes to real, complex applications in a matter of milliseconds.
But don’t just take my word for it! Here’s a simple video (no audio) demonstrating me using Zig to quickly make and test some changes to Fizzy, a pixel editor application. The initial build takes around 5 seconds, and then every time I make a change, a rebuild completes in 50 – 70ms.
For this demo, I had to upgrade Fizzy to Zig’s master branch. This is because while Zig 0.16.0 does have support for incremental compilation, it is missing some important linker features which have since been implemented. This means that if you prefer to stick to tagged releases of Zig, you likely won’t be able to try this out until 0.17.0 drops; sorry!
If you’re already convinced and just want to know how to use this, great! Head on down to the last section of this post to find out. But perhaps you’re understandably skeptical that this is applicable to most projects, or, like me, you just enjoy learning how stuff like this works. For all of you folks, let’s dig into the details!
Processing Source Files
The Zig compiler’s pipeline can be split up into a few different parts, which we’ll look at in order. The first part works at the granularity of entire source files, and basically consists of running the following process in a loop:
Read in a source file from disk
Parse that file into an AST
Convert that AST into a format named “ZIR” using a pass named “AstGen”
If you’re curious, ZIR (Zig Intermediate Representation) is an untyped SSA-form IR—but don’t worry if you have no idea what that means, because it won’t really matter here. All we care about is that we’re converting an entire source file into a different format.
While AstGen runs, it learns about all Zig imports (@import(“foo.zig”)) in the source file, so we can repeat this entire process on all of the imported files. So by running this process in a loop, we will ultimately discover every Zig source file in the compilation, and will convert them all to ZIR.
This part of the pipeline actually has several useful properties:
The processing run on each file is a pure function of that file’s contents, involving no shared or external state
Parse and AstGen are both quite fast on their own: on my laptop, running them both over the entire src/ directory of the Zig compiler (with no parallelism at all) takes around 920ms
Thanks to Zig’s usage of data-oriented design patterns, ZIR can be trivially written to and read from disk with one writev/readv system call—there is no “serialization” step.
These properties have two nice consequences.
Firstly, assuming one “task” per source file, this entire process is embarrassingly parallel. That means we can trivially run it on a thread pool by queuing up a task every time we discover a new source file from an import—the only shared state (which we’ll just protect with a mutex) is a hash set keeping track of which file paths we have already seen.
Secondly, and arguably even more importantly, these properties make it very straightforward to implement incremental compilation for this part of the pipeline. All we need to do is cache each source file’s generated ZIR on disk, and only rebuild it when we detect that the file changed.
Both of these optimizations have been enabled by default in Zig for years—they are battle-tested and make this part of the pipeline near-instantaneous in most cases. If you’re using Zig, you can see how fast this is using the progress output on stderr—when it says “AST Lowering”, this part of the pipeline is running. I’d guess that a lot of Zig users only even notice that happening the very first time they run the compiler (because on its first run the compiler needs to do this work for the entire Zig standard library and compiler_rt).
Okay, so, we made this part fast! That’s great, but the bad news is that this was the easy part—lots of compilers can already do this kind of caching. From here, things will get trickier.
Semantic Analysis
The next part of the pipeline is arguably the most important: semantic analysis. This includes both type checking and comptime evaluation.
The job of semantic analysis is essentially to “interpret” the ZIR we produced earlier, emitting compile errors (such as type errors) along the way; and, for runtime functions, building another intermediate representation (Analyzed Intermediate Representation, or AIR for short) which can be sent on to later parts of the pipeline.
Before we move forward, a quick terminology clarification. A “container-level declaration” is the Zig equivalent of what other languages call a “top-level declaration”. That term is inaccurate in Zig, because container-level declarations do not have to be at the top level syntactically, but the concept is the same. If I say “container-level declaration”, I basically mean “a function, global constant, or global variable”.
Semantic analysis is the most difficult part of the compiler to handle incrementally. Perhaps unsurprisingly then, this is where language design starts to matter a lot: while I am pretty confident that most modern languages could support incremental compilation similar to how we do, certain design decisions can make that much more difficult. Zig has had its design tweaked over the years (sometimes controversially) specifically so that it is easier to support fast incremental compilation.
The name of the game here is to split up your compilation into a bunch of pieces which you can mostly analyze independently of one another, and, crucially, where the dependencies that do exist between those pieces can be easily modeled in a dependency graph.
In the Zig compiler, we call these pieces “analysis units”, or I might sometimes just say “unit” for short. I’m going to ever so slightly simplify things here and tell you that the Zig compiler has four different kinds of analysis unit:
The layout (size, alignment, etc) of a struct or union type.
The type of a container-level declaration.
The value of a container-level const declaration.
The body of a runtime function.
During semantic analysis of a particular unit, we populate a set of other units which this unit depends on. Let’s look at a basic example:
var global_0: u32 = 123; const global_1: u32 = 456;
pub fn foo(cond: bool) u32 { if (cond) { return global_0; } else { return global_1; } }
Here’s what happens when we analyze the body of the function foo:
Because the argument cond is not comptime-known, we semantically analyze both branches of the if
Take a pointer to global_0, in preparation to load from itAdd dependency: type of global_0
Add dependency: type of global_0
Load global_0 at runtime, because it is var so does not have a comptime-known value
Take a pointer to global_1, in preparation to load from itAdd dependency: type of global_1
Add dependency: type of global_1
Load global_1 at compile time, because it has a comptime-known valueAdd dependency: value of global_1
Add dependency: value of global_1
So we end up with this function body depending on the types of global_0 and global_1, and the value of global_1 (since that’s comptime-known). This tells the compiler that if the type of global_0 or global_1 changes, or the comptime-known value of global_1 changes, the function should be re-analyzed.
Dependencies on the body of a runtime function are impossible (at least in the simplified view I’m presenting here). This means that function body analysis units can only have “outgoing” edges in the dependency graph (i.e. they may depend on other units, but other units do not depend on them).
Dependencies on the value of a const declaration only arise due to Zig’s ability to use those at comptime. If not for that language feature, dependencies on the value of a declaration would be impossible, just as it is impossible to depend on the body of a runtime function.
Dependencies on a type’s layout arise, in short, from having values of that type, or from needing to know something about the type’s layout. I’m not going to discuss this any further here, because it’s a bit complicated and quite specific to Zig’s type system, but it’s not fundamentally different.
Okay, so, we’ve told the compiler about when re-analysis of one thing needs to also trigger re-analysis of another thing. However, there’s one more puzzle piece here—source code dependencies. By itself, this dependency graph is useless: what do we actually do when the user asks for a recompile (what we call an “incremental update”)? We don’t know the first thing to re-analyze!
To solve this problem, we track dependencies of analysis units, not only on other units, but also on pieces of source code. In the cases we’ve looked at so far, these are all really simple: in the snippet above, the unit “type of global_0” depends on the source code of global_0, the units “type of global_1” and “value of global_1” both depend on the source code of global_1, and the unit “body of foo” depends on the source code of foo. Whenever any byte of source code in the given region is modified, the dependent analysis unit will be marked as “outdated” and re-analyzed.
Note that in reality, things can get more complicated than each unit depending on one piece of source code. For example, an inline function call in Zig performs semantic inlining, which means that it essentially triggers semantic analysis of a different piece of code but in the caller’s analysis unit. Therefore, inline function calls introduce dependencies from the caller’s analysis unit on the source code of the callee.
Of course, we still need to be able to figure out which regions of source code have changed since an incremental update. For this, ZIR contains hashes for specific “interesting” regions of source code (e.g. the entire source code for each container-level declaration), and those hashes are what you are actually depending on. If the source code changes, the hash changes, and that’s easy for the compiler frontend to detect.
Okay, that was a lot of explaining—now let’s look at some pretty pictures! Here’s some Zig source code:
const lucky_number = 42;
const S = struct { x: u32 };
fn getSomething() S { return .{ .x = lucky_number }; }
fn testLuck(x: u32) void { if (x == lucky_number) { // do something } }
export fn entry() void { const result = getSomething(); testLuck(result.x); }
…and here’s its dependency graph (with some redundant edges removed for legibility). The nodes on the right represent the source code which has been hashed, while the remaining nodes are all analysis units.
Now, let’s say we change the first line of the file to read const lucky_number = 43;. First, the compiler lowers the new ZIR for this file. It maps declarations from the old ZIR to the new ZIR based on the declaration names, and compares the source hashes associated with each declaration. In this case, it successfully maps every declaration, and it sees that one source hash changed—the one associated with const lucky_number. Next, it looks at the dependency graph to find everything which depends on that source hash. In this case, it only finds one direct dependency:
So, the compiler re-analyzes the value of the lucky_number declaration. If our change to the line had been a no-op (e.g. we just added some whitespace), then it would determine that the value did not change, and stop here. But in this case, the value did change! Therefore, the compiler continues this process, by next considering any analysis units which depend on the value of lucky_number, of which there are two:
The compiler analyzes those two units—the bodies of testLuck and getSomething. There are no dependencies on these units (since they’re function bodies), so the semantic analysis loop stops here. However, semantic analysis of those functions does generate new AIR, which brings us neatly to our next topic: code generation.
Code Generation
Code generation, sometimes called “codegen” for short, is the stage in the compiler pipeline where AIR from semantic analysis is converted to something resembling machine instructions. Codegen doesn’t quite emit machine instructions yet—instead it’s something called MIR (Machine Intermediate Representation)—but there is almost a 1 – 1 mapping between MIR instructions and machine instructions. There are separate codegen implementations for each target architecture (x86_64, aarch64, etc).
A nice thing about codegen is that just like the whole-file processing earlier, it is an embarrassingly parallel task (at least in builds where you aren’t doing inter-function optimizations like inlining). There is no state shared between code generation of different functions, so we can have a queue of pending functions whose AIR needs converting to MIR, and process that queue across arbitrarily many threads. There’s just one small gotcha, which is that we need to be careful to cap the size of that queue, because if codegen is ever running behind semantic analysis for any reason, the size of the queued-up AIR can add up fast!
In terms of incremental compilation, this phase of the pipeline is actually as simple as it gets, because AIR and MIR both exist at the granularity of individual functions, which is the same granularity incremental compilation works at. This means that there is no need for the compiler to cache AIR or MIR at all! The AIR is thrown away as soon as code generation is done, and the MIR will be thrown away right after it’s consumed by our next stop: the linker.
Linking
Incremental linking is kind of a difficult problem, and I suspect is a big reason that no other major toolchain supports this kind of incremental compilation yet. General-purpose incremental linkers aren’t really a thing at the moment, and though Wild was originally conceptualized as one, that project seems to have shifted its focus firmly towards cold-link performance over the past couple of years, with no explicit timeframe for incremental linking.
David Lattimore, the creator of Wild, has a blog post discussing some of the difficulties of incremental linking. One of those is diffing input objects to figure out what actually changed on an update. However, when you control the entire compilation pipeline, a simpler design presents itself which neatly sidesteps that entire problem: tightly integrating the linker with the compiler.
To begin with, let’s just look at how the linker might work without incremental compilation. Because linking involves a lot of shared state, our linker is entirely single-threaded (maybe we’ll look into multi-threaded linking in the future, but for now we’re keeping things simple). When the linker receives MIR from codegen, it first needs to convert that MIR into the actual machine code. This logic is specific to the codegen backend, but we can’t run it until now because it requires cooperation with the linker. That’s because while emitting machine code, the codegen backend generates relocations—basically, instructions for the linker to overwrite certain parts of the code with specific addresses or values (for instance the address of another symbol). The linker needs to save all of these relocations internally, so we need to be on the linker thread for this.
After generating the machine code and associated relocations, we reserve space for that machine code in the output section (usually .text). We save the machine code in a buffer, save the relocations to apply later, and do some miscellaneous bookkeeping work, such as adding a symbol table entry.
For a non-incremental linker, this would be the end of the story. At the end of compilation, we would assign addresses to every section, write out everything we reserved space for, and apply all of the relocations. Incremental linking is a bit trickier—writing the machine code to the file, assigning addresses, and applying relocations, all ideally needs to happen before we know the full contents of the binary, and we need to be able to update those things later.
A lot of the complexity here is actually just in moving things around. For example, if we want to add a function to the .text section, but there isn’t enough space, we need to expand that section. But the section might be surrounded by other sections, which we can’t just overwrite, so we’ll need to move something—either the .text section itself, or one of the surrounding sections. In doing so, we’re going to change not only file offsets but also virtual addresses of everything we move—this means we’ll need to update symbol table addresses, re-apply relocations, etc. That’s a lot to keep track of! (There’s also a similar problem for segments, one level up.)
To solve this problem, Jacob Young introduced a nifty abstraction into the Zig compiler called link.MappedFile. It memory-maps the output file, but more importantly tracks a tree of “nodes” in that file. The root node covers the entire file, and child nodes refer to specific regions within their parent node. The API user can add nodes, or grow a node to a given size—in both cases, if there is not space in the parent to trivially perform the operation, MappedFile deals with moving other nodes around to make space. Whenever it resizes or moves a node, the implementation sets a “dirty” flag on that node, so that at some point the linker implementation can detect this and apply any necessary fixups, e.g. re-applying relocations whose target moved.
Right now, MappedFile has fairly primitive logic for node allocation, so sometimes makes suboptimal decisions—but because we’ve abstracted it behind a neat little API, we can improve it independently going forward.
To get to incremental linking, then, we need only slightly change the process I described earlier. After we finish emitting machine code, we create a node in the mapped file, large enough to hold the code—or if this function already existed, we just resize the existing node—and we copy the machine code into it. Allocating this node in the file might (in rare cases) need to move some other stuff in the file around, in which case the appropriate “dirty” flags are set on those nodes. We always set the “dirty” flag for the function’s node itself, so that its relocations will be applied at some point.
Because of the pending relocations, we probably don’t have a valid binary right now—but that’s okay! When the linker thread is next idle (i.e. its work queue is empty), or at the end of compilation if the linker thread remains busy until then, we’ll check all of those “dirty” flags and clean up after ourselves. This could involve work such as assigning new virtual addresses, updating the section headers and program headers, updating addresses in the symbol table, and re-applying relocations.
It might sound like that “fixup” work is expensive. Sometimes, it can be—if you’re creating a dynamic executable and the PLT has to move, that can take a moment, because there are usually a lot of relocations targeting the PLT. However, most of the time, we don’t need to move anything! By using exponential growth factors on nodes (similar to how dynamic data structures like ArrayList work), we amortize this cost and make it extremely rare in reality (at the cost of a slightly increased binary size, which isn’t usually a major concern during development). This design means that you might very occasionally see one update run slightly slower than usual (maybe a few hundred milliseconds?), but I’ve not personally hit this a single time, despite using incremental compilation with this linker near-daily for the past couple of months.
Flush
Okay, we’ve made it to the end, and kept everything incremental along the way. Files were lowered to ZIR with a simple per-file cache; semantic analysis of declarations kept track of a dependency graph to figure out what might have changed; code generation re-ran only for updated functions; and our linker wrote new code into the file without changing any other bytes. We just have a few more loose ends to tie up.
Firstly, because of how Zig’s “lazy analysis” feature interacts with incremental compilation, we need to do a graph traversal to figure out which functions/declarations/etc are actually referenced. It’s possible that something was referenced on a previous incremental update (so we compiled it), but has since become unreferenced, which means we need to ignore any compile errors it emitted, not perform symbol exports from it, etc. There are probably some optimizations you can do here, but at least right now, we just traverse the full reference graph on every update. We can get away with this even on big projects, because computers are really fast!
Once we’ve figured out what’s referenced, we can tell the linker every global symbol which is exported from Zig code, so that it can add any necessary entries to the symbol table. We will also report compile errors if there are any, and some other miscellaneous tasks like that. Finally, we call the linker’s flush function, whose job is just to do any remaining linking work before the file is closed. If the linker has any MappedFile node still marked as “dirty”, we’ll need to handle that, but otherwise we want to do as little work as possible—remember, anything we do here is going to happen on every update, so we want to keep it pretty much O(1). Therefore, all that the ELF linker really does here is write out the .dynamic section, and write the entry field in the ELF header.
We then close the file, and the compilation is complete!
Tracing an Update
Explanations are cool and all, but we can actually see this happening. Tracy is a real-time profiler—it’s designed for games, but you can integrate it into anything. The Zig compiler has optional Tracy integration, enabled using a build flag. (I guess incremental updates kinda resemble frames in a video game if you squint?)
This can occasionally be useful for various compiler performance analysis, but I actually find it really cool to use for incremental compilation, because we can see the different parts of the compiler pipeline clear as day. Let’s take a look at the Tracy output for a change to Fizzy, much like the changes in the video from earlier. This particular update took 37ms (a little faster than the ones we saw in the video), but at first I’m going to zoom in on the first 6ms or so of this 37ms update—I’ll explain why later.
At the start we can see a flurry of activity across all threads—that’s the thread pool doing all of the per-file work. Although we only changed one file, the compiler doesn’t assume that, and instead checks every source file. There’s one small optimization here, which is that because we already know which source files were in the compilation on the last update, we can guess that those files will all still be reachable and so check for changes to all of them. That just means we don’t need to wait for the first file to be processed so that we can discover its imports.
Next we see a good chunk of time (around 1ms) in computeAliveFiles. This function is traversing the graph of file imports to assign every file to a Zig “module” (because it’s possible for a file to move from one module to another between updates). We also use this import traversal to check whether all source files are, in fact, still in the compilation. If any are not—because all imports of them were removed—then we’ll basically just ignore those files for the rest of this update.
Then we have another millisecond in updateZirRefs. This function is responsible for correlating the old and new ZIR of any changed files, and updating all internal references to ZIR instructions to refer to the instruction’s index in the new ZIR rather than its index in the old ZIR. This is a fairly simple task, but the current implementation involves iterating every ZIR instruction we hold a reference to at all and completely rebuilding a hash map’s metadata. This can probably be optimized.
Now we’re done with the single-threaded per-file stuff, and we can finally get onto the meat and potatoes of the pipeline: semantic analysis, codegen, and linking. The “sema_loop” zone contains all of the time spent in semantic analysis—around 1.2ms. We then see that function’s AIR get picked up by codegen on a different thread (the green zones named runCodegenInner), which runs for around 240us. The resulting AIR is picked up by the linker thread and emitted to the binary—this linking work (the purple emitFunction zone and the little green zones next to it) takes around 170us. Overall, this entire part of the pipeline—which by far dominates cold builds—comes in at around 1.6ms for this update. Not bad!
After that’s all done, we’re onto flush. The little purple zone at the bottom-right is a small bit of linking work the frontend requests during flush: regenerating a lookup table which we use to implement Zig’s @errorName builtin. That takes around 50us, and brings us to the end of the 6ms region I’ve zoomed in on. That means it’s finally time to zoom out and see what the remaining 31ms are…
Basically all of the remaining time is spent in one function, resolveReferencesInner. Remember a bit earlier I mentioned doing a graph traversal during flush, to determine which Zig declarations are referenced? Well, that’s this function’s job! It’s not inefficient by any means, but that graph is kinda big, so it’s perhaps unsurprising that it starts to matter when we’re trying to go fast.
On the one hand, this seems pretty silly, so much so that I seriously considered trying to improve it before putting out this blog post (after all, a 7ms time is more impressive than a 37ms time). This is amplified when you consider that the reference graph didn’t actually change here. So the vast majority of the duration of this incremental update is being spent figuring out that a graph didn’t change!
But actually, I think this is really cool, because it shows how much efficiency is still left to squeeze out. This 30ms zone is realistically not a big issue, but we can get rid of it nonetheless—firstly by avoiding recomputing this data when the reference graph is unchanged, but also by only recomputing what we need to when the references do change (that problem is called “dynamic single-source shortest path” and is a fairly well-studied problem in graph theory).
Basically, we’re far from done on the performance front! If you want to keep up with what we’re doing in the future, you might consider adding the Zig devlog to your RSS reader, or checking out the release notes when new versions of Zig are released.
Okay, I’ve been rambling about compilers for long enough; let me actually show you how to use this thing. I’ll assume you have a Zig project with a build script, and that it compiles on a recent master branch build of Zig (or, if you’re reading this after Zig 0.17.0 releases, that’ll also work.)
At the time of writing, this will only really work if you target x86_64-linux, because our other code generation and linker backends are not mature enough yet. The majority of the Zig core team runs Linux on x86_64, so by focusing on it first, we’ve sped up our workflows, meaning it’ll be faster for us to add support for other targets—which is now top priority!
The bad news is that right now, this isn’t zero-effort. Eventually it will be—we’ll cache all of the compiler state to disk and automatically reload the last saved state when you run zig build, so incremental compilation will just happen automatically—but we’re not quite there yet. However, the good news is that using it today requires very little work!
The short version is that you just need to run this command:
To add this web app to your iOS home screen tap the share button and select "Add to the Home Screen".
10HN is also available as an iOS App
If you visit 10HN only rarely, check out the the best articles from the past week.
Visit pancik.com for more.