To use the Mastodon web ap­pli­ca­tion, please en­able JavaScript. Alternatively, try one of the na­tive apps for Mastodon for your plat­form.

In November, I got fed up and said screw it, I’m in­stalling Linux. Since that ar­ti­cle was pub­lished, I have dealt with one mi­nor cat­a­stro­phe af­ter an­other. None of that has any­thing to do with Linux, mind you. It just meant I did­n’t in­stall it on my desk­top un­til Sunday evening.

My goal here is to see how far I can get us­ing Linux as my main OS with­out spend­ing a ton of time futz­ing with it — or even much time re­search­ing be­fore­hand. I am not look­ing for more high-main­te­nance hob­bies at this stage. I want to see if Linux is a wingable al­ter­na­tive to Microsoft’s in­creas­ingly an­noy­ing OS.

Honestly? So far it’s been fine. Many things I ex­pected to be dif­fi­cult — like get­ting my Nvidia graph­ics card work­ing prop­erly — were per­fectly straight­for­ward. A few things I thought would be sim­ple weren’t. And I’ve run into one very funny is­sue with a gam­ing mouse that only works in games. But I’ve been able to use my Linux setup for work this week, I played ex­actly one video game, and I even printed some­thing from my ac­cursed printer.

I picked CachyOS rather than a bet­ter-known dis­tro like Ubuntu be­cause it’s op­ti­mized for mod­ern hard­ware, and I had heard that it’s easy to in­stall and set up for gam­ing, which is one of the rea­sons I’d stuck with Windows for this long. After back­ing up my Windows im­age some­time in December (close enough), I fol­low the in­stal­la­tion in­struc­tions in the Cachy wiki and down­load the CachyOS live im­age to a Ventoy USB drive, plug it into my PC, re­boot into the BIOS to dis­able Secure Boot, re­boot again into the Ventoy boot­loader, and launch the CachyOS disk im­age.

First chal­lenge: My mouse but­tons don’t work. I can move the cur­sor, but can’t click on any­thing. I try plug­ging in a mouse (without un­plug­ging the first one), same deal. Not a ma­jor is­sue; I can get around fine with just the key­board. Maybe this is just an is­sue with the live im­age.

I launch the in­staller and am thrust into analy­sis paral­y­sis. An op­er­at­ing sys­tem needs lots of lit­tle pieces to work — stuff you don’t even think of as in­di­vid­ual com­po­nents if you use Mac or Windows. How do you boot into the OS? What runs the desk­top en­vi­ron­ment? How are win­dows drawn? What’s the file sys­tem? Where do you get soft­ware up­dates? In Mac and Windows, all those de­ci­sions are made for you. But Linux is fun­da­men­tally dif­fer­ent: The core of the OS is the ker­nel, and every­thing else is kind of up to you. A dis­tro is just some­body’s idea of what pieces to use. Some, like Pop_OS! and Mint, aim for sim­plic­ity and make all those choices for you (though you can still change them if you want). But Cachy is based on Arch, a no­to­ri­ously DIY dis­tro, and be­fore I do any­thing else, I have to pick one of four boot­load­ers. I pick Limine, for rea­sons I can’t re­call.

Next, I need to fig­ure out where to in­stall it. On the rec­om­men­da­tion of Will Smith from the Dual Boot Diaries pod­cast — from whom the “an op­er­at­ing sys­tem is a bunch of pieces” thing above is largely cribbed — I in­stall Cachy on a dif­fer­ent phys­i­cal drive from Windows, since Windows up­dates tend not to care if they over­write other boot­load­ers.

I have a 4TB stor­age drive with just over a ter­abyte of data on it, so I shrink that par­ti­tion down to 2TB us­ing the in­staller’s man­ual par­ti­tion­ing in­ter­face, then (following the guide) make a 2GB boot par­ti­tion and a root par­ti­tion us­ing the btrfs file sys­tem. The guide says it needs at least 20GB, so I go big and make it 100GB. This will cause a mi­nor prob­lem later.

Then I’m on the Cachy desk­top, and my mouse but­tons still aren’t work­ing. Swapping USB ports does­n’t do any­thing. Plugging in my track­ball does­n’t fix it ei­ther. I fi­nally try un­plug­ging the mouse, which makes the track­ball work nor­mally. My gam­ing mouse is an an­cient Mad Catz Cyborg RAT 7; it turns out this is a known is­sue. I de­fer edit­ing con­fig­u­ra­tion files for now and just keep the mouse un­plugged.

There are lots of ways to in­stall apps on Linux. Sometimes you can just down­load them from a com­pa­ny’s web­site, or you get them from your dis­tro’s of­fi­cial repos­i­to­ries, or GitHub, or wher­ever. There’s no of­fi­cial app store for Linux, but there are at least three pro­jects aim­ing to pro­vide uni­ver­sal Linux apps: Flatpak, AppImage, and Snap. Neat! Commence hodge­podg­ing.

Cachy has a one-click gam­ing pack­age in­stall that in­cludes the Proton com­pat­i­bil­ity layer, Steam, and Heroic (a launcher for Epic, GOG, and Amazon). I fig­ure I ought to try one game. Then I re­mem­ber that my root par­ti­tion is only 100GB. I re­boot back into the Cachy live im­age and use the Parted util­ity to in­crease it to 1TB, then make a sec­ond btrfs par­ti­tion in the re­main­ing space. I re­boot, log into Epic and GOG, and start down­load­ing The Outer Worlds, a game from 2019 I’ve been play­ing a bit lately. It runs fine with Proton, and I can even sync my saves from the cloud. I play it for a few min­utes with my track­ball, re­mem­ber I hate gam­ing on a track­ball, and plug my gam­ing mouse back in. It works fine as long as I’m in the game, but out­side the game, mouse clicks stop work­ing again. It makes sense — the bug is on the desk­top, not in games — but it’s very funny to have a gam­ing mouse that only works for gam­ing.

The biggest is­sue I’ve had so far is Minecraft: Bedrock Edition. For some rea­son, Microsoft has­n’t pri­or­i­tized mak­ing a Linux ver­sion of Bedrock. Java Edition works fine in Linux, but I play Minecraft with my kids, and they’re on Bedrock Edition on their iPads. There’s sup­posed to be a way to run the Android app with MCPE Launcher, but I could­n’t get it to work. There’s also a pro­ject to get the Windows ver­sion run­ning on Proton, which will be my next step.

I’m well aware this is the hon­ey­moon phase. And us­ing Linux for less than a week is­n’t ex­actly a flex. Many peo­ple use Linux. And I haven’t even tried do­ing any­thing par­tic­u­larly dif­fi­cult, or play­ing a game that came out this decade. But so far it’s been a much eas­ier tran­si­tion than ex­pected, and a qui­eter ex­pe­ri­ence over­all. My OS is­n’t try­ing to change my browser or search en­gine to make some share­holder happy some­where. It’s not nudg­ing me to try some bull­shit AI fea­ture.

Will I go crawl­ing back to ma­cOS or Windows the first time I have to edit a batch of pho­tos? Possibly! I’ll def­i­nitely boot back into Windows — or pull out a Chromebook — to play Minecraft with my kids, if I can’t get it run­ning on Linux. And I don’t think I’ll ever be able to use Linux ex­clu­sively; my job as a re­views ed­i­tor means I have to stay fa­mil­iar with as many op­er­at­ing sys­tems as pos­si­ble. (This is a good way to drive your­self nuts.)

Oh My Zsh is still get­ting rec­om­mended a lot. The main prob­lem with Oh My Zsh is that it adds a lot of un­nec­es­sary bloat that af­fects shell startup time.

Since OMZ is writ­ten in shell scripts, every time you open a new ter­mi­nal tab, it has to in­ter­pret all those scripts. Most likely, you don’t need OMZ at all.

Here are the tim­ings from the de­fault setup with a few plu­g­ins (git, zsh-au­to­sug­ges­tions, zsh-au­to­com­plete) that are usu­ally rec­om­mended:

And that’s only for prompt and a new shell in­stance, with­out ac­tu­ally mea­sur­ing the git plu­gin and vir­tual env plu­g­ins (which are of­ten used for Python). Creating a new tab takes some time for your ter­mi­nal, too. It feels like a whole sec­ond to me when open­ing a new tab in a folder with a git repos­i­tory.

My work­flows in­volve open­ing and clos­ing up to hun­dreds of ter­mi­nal or tmux tabs a day. I do every­thing from the ter­mi­nal. Just imag­ine that open­ing a new tab in a text ed­i­tor would take half a sec­ond every time.

Once in a while, it also checks for up­dates, which can take up to a few sec­onds when you open a new tab.

I see no rea­son in fre­quent up­dates for my shell con­fig­u­ra­tion. Especially, when a lot of third-party plu­g­ins are get­ting up­dates too. Why would you want you shell to fetch up­dates?

My ad­vice is to start sim­ple and only add what you re­ally need.

Here is the min­i­mal Zsh con­fig­u­ra­tion that works well as a start­ing point:

It’s an al­ready pretty good setup with com­ple­tions!

Some de­tails about this con­fig­u­ra­tion:

* HISTSIZE and SAVEHIST set the size of your his­tory.

* au­tocd al­lows you to change di­rec­to­ries with­out typ­ing cd.

You also want to cus­tomize your prompt. For prompts, I’m us­ing star­ship which is a fast and min­i­mal prompt packed into a sin­gle bi­nary.

The very old way of do­ing this in Oh My Zsh was to use plu­g­ins and cus­tom themes. With star­ship, it’s very sim­ple and easy now. It re­places git, vir­tual en­vi­ron­ment and lan­guage spe­cific plu­g­ins.

Here is my con­fig for star­ship:

Because cloud ser­vices are avail­able glob­ally, I’ve dis­abled them. I don’t want them to be dis­played on every prompt, since this adds vi­sual noise.

Here is how my prompt looks like now:

This pro­ject uses both Python and Rust, they are high­lighted in the prompt. When you run a com­mand, it also shows how long it took to ex­e­cute.

To en­able it, add the fol­low­ing line to your .zshrc:

A lot of peo­ple use zsh-au­to­sug­ges­tions plu­gin for his­tory search. I find it dis­tract­ing, be­cause it shows all sug­ges­tions as you type.

Instead, I pre­fer us­ing fzf binded to Ctrl+R for search­ing his­tory. It gives an in­ter­ac­tive fuzzy search.

To en­able it, add the fol­low­ing lines to your .zshrc:

After these changes, the startup should look as fol­lows:

For Vim users, I also sug­gest en­abling Vim mode in Zsh. It makes edit­ing com­mands much faster.

It works the same way as in Vim. By de­fault, zle (the li­brary that reads the shell in­put) uses Emacs key­bind­ings.

After switch­ing from OMZ a year ago, it only took me a few days to get used to the new work­flow. If you still miss­ing some of the plu­g­ins, you can al­ways load them man­u­ally.

Some peo­ple won­der why I open so many tabs. I use tmux and a ter­mi­nal-based ed­i­tor (helix). In tmux, I have pop­ups for lazy­git and yazi file man­ager. Every time I need to check git his­tory or browse files, I just open them. They open on top of the cur­rent ses­sion as an over­lay. You can view them as win­dows in IDEs.

I also use tem­po­rary splits to quickly run the code/​tests and see the out­put. They count as sep­a­rate shell ses­sions. I want to see code and out­put side by side, but I don’t need it all the time.

Effective in 2026, to align with our trunk sta­ble de­vel­op­ment model and en­sure plat­form sta­bil­ity for the ecosys­tem, we will pub­lish source code to AOSP in Q2 and Q4. For build­ing and con­tribut­ing to AOSP, we rec­om­mend uti­liz­ing an­droid-lat­est-re­lease in­stead of aosp-main. The an­droid-lat­est-re­lease man­i­fest branch will al­ways ref­er­ence the most re­cent re­lease pushed to AOSP. For more in­for­ma­tion, see Changes to AOSP.

Learn about the fea­tures added in Android 16 QPR2 and how to im­ple­ment them on your de­vices.

Android fea­ture launch flags en­sure that the AOSP de­vel­op­ment branch is sta­ble for every­one. Contributors to AOSP can use fea­ture launch flags to make sure only tested code is ex­e­cuted.

Read about the Android Open Source Project (AOSP) and learn how to de­velop, cus­tomize, and test your de­vices.

Learn how to set up your en­vi­ron­ment, down­load the AOSP source, build Android, and make con­tri­bu­tions.

Find out how Android in­cor­po­rates in­dus­try-lead­ing se­cu­rity fea­tures to keep the Android plat­form and ecosys­tem safe.

From the ba­sic build­ing blocks of an Android de­vice to the more in­tri­cate ad­vanced fea­tures, learn about all the ways you can cus­tomize the Android OS.

Ensure your users have a co­her­ent ex­pe­ri­ence as they use your de­vice along­side other Android de­vices in the ecosys­tem.

Learn how to de­velop and cus­tomize the Android Automotive plat­form, which runs di­rectly on in-ve­hi­cle hard­ware.

Read about cre­at­ing con­fig­urable vir­tual Android de­vices, man­ag­ing de­vices on a cor­po­rate net­work, and de­liv­er­ing live con­tent to de­vices through Android TV.

Help pro­tect your de­vice by check­ing the lat­est Android Security Bulletins. Also see the Android Automotive, Chromecast, Wear OS, Pixel, and Pixel Watch bul­letins.

Learn how to set up your en­vi­ron­ment, down­load the source, and start con­tribut­ing to AOSP through this tu­to­r­ial.

Learn how the pieces fit to­gether, from the ker­nel to the HALs to up­dat­a­ble sys­tem com­po­nents.

Make sure your de­vices work to­gether and are con­nected through Bluetooth, NFC, Wi-Fi, and tele­phony.

When it comes to per­for­mance, it’s hard to find users who are com­pletely sat­is­fied with Windows 11. Many be­lieve it’s filled with un­nec­es­sary fea­tures that are dif­fi­cult to dis­able.

According to a re­cent test by YouTuber TrigzZolt, Windows 11 might ac­tu­ally be the slow­est ver­sion of Windows re­leased in the past 25 years — yes, even slower than the of­ten-crit­i­cized Windows Vista.

Microsoft pro­motes Windows 11 as its fastest op­er­at­ing sys­tem ever, but real-world tests tell a dif­fer­ent story. In a de­tailed com­par­i­son, TrigzZolt bench­marked sev­eral Windows ver­sions: XP, Vista, 7, 8.1, 10, and 11. The re­sults showed that the newest ver­sion, Windows 11, per­formed the worst over­all.

The tests cov­ered sys­tem boot time, lap­top bat­tery per­for­mance, ap­pli­ca­tion launch speed, mem­ory man­age­ment, and video edit­ing ca­pa­bil­i­ties. The bench­marks were run on a Lenovo ThinkPad X220 — a model not of­fi­cially com­pat­i­ble with Windows 11 — which could have af­fected some re­sults. However, since the in­com­pat­i­bil­ity mainly re­lates to hard­ware se­cu­rity rather than per­for­mance, the im­pact was likely mi­nor.

Among all the tested sys­tems, Windows 11 was the slow­est to start up, took the longest to open sim­ple ap­pli­ca­tions like Paint and File Explorer, and per­formed poorly when han­dling video edit­ing in OpenShot. It also showed the high­est RAM us­age when idle. Even with­out ac­tive ap­pli­ca­tions, Windows 11 con­sumes sig­nif­i­cantly more mem­ory due to its many back­ground ser­vices and al­ways-on fea­tures. Recently added AI func­tions have also con­tributed to slow­ing it down.

The only ar­eas where Windows 11 per­formed bet­ter were file trans­fer speed and stor­age space man­age­ment for sys­tem ap­pli­ca­tions. Overall, the con­clu­sion was clear — Microsoft needs to lighten Windows 11, in­stead of fill­ing it with more and more fea­tures that few users ac­tu­ally use.

ANALYSIS From May’s cy­ber­at­tack on the Legal Aid Agency to the Foreign Office breach months later, cy­ber in­ci­dents have be­come in­creas­ingly com­mon in UK gov­ern­ment.

The scale ex­tends far be­yond these high-pro­file cases: the NCSC re­ports that 40 per­cent of at­tacks it man­aged be­tween September 2020 and August 2021 tar­geted the pub­lic sec­tor, a fig­ure ex­pected to grow.

Given this threat land­scape, why does the UK’s flag­ship Cyber Security and Resilience (CSR) Bill ex­clude both cen­tral and lo­cal gov­ern­ment?

Sir Oliver Dowden, for­mer dig­i­tal sec­re­tary and cur­rent shadow deputy PM, led calls in the House of Commons this week urg­ing Labour to re­think its stance on ex­clud­ing cen­tral gov­ern­ment from the Cyber Security and Resilience (CSR) Bill.

“I would just urge the min­is­ter, as this bill passes through Parliament, to look again at that point, and I think there is a case for putting more strin­gent re­quire­ments on the pub­lic sec­tor in or­der to force min­is­ters’ minds on that point.”

The CSR bill was an­nounced days into Sir Keir Starmer’s tenure as Prime Minister, aim­ing to pro­vide an es­sen­tial re­fresh of the coun­try’s heav­ily out­dated NIS 2018 reg­u­la­tions.

It pro­posed to bring man­aged ser­vice providers into scope, as was sched­uled in 2022 be­fore those plans fell by the way­side, and dat­a­cen­ters, among many other as­pects.

Parallels can be drawn with the EU’s NIS2. However, the CSR bil­l’s scope is nar­rower, ex­clud­ing pub­lic au­thor­i­ties, un­like the EU’s equiv­a­lent reg­u­la­tory re­fresh.

Ian Murray, min­is­ter of state across two gov­ern­ment de­part­ments and re­spon­si­ble, in part, for data pol­icy and pub­lic sec­tor re­form, thanked Dowden for his sug­ges­tions and promised to take them on board.

In re­spond­ing to the shadow deputy PM, Murray also pointed to the Government Cyber Action Plan, which it launched hours be­fore the CSR bill was set for a sec­ond read­ing in the Commons.

This plan will os­ten­si­bly hold gov­ern­ment de­part­ments to equal se­cu­rity stan­dards as the CSR bill… just with­out any of the le­gal oblig­a­tions.

Cynics may see it as a tool to quell any crit­i­cisms of the bil­l’s scope not ex­tend­ing to cen­tral gov­ern­ment, all with­out mak­ing any hard se­cu­rity com­mit­ments.

As Dowden noted in the Commons on Tuesday, cy­ber­se­cu­rity is a mat­ter that is of­ten de­pri­or­i­tized quickly in gov­ern­ment. “I wel­come the min­is­ter’s com­ments about the oblig­a­tion on the pub­lic sec­tor. However, I would cau­tion him that, in my ex­pe­ri­ence, cy­ber­se­cu­rity is one of those things that min­is­ters talk about but then other pri­or­i­ties over­take it. And the ad­van­tage of leg­isla­tive re­quire­ments is that it forces min­is­ters to think about it.”

“I do think that more pres­sure needs to be brought to bear on min­is­ters in terms of their ac­count­abil­ity for cy­ber­se­cu­rity. I fear that if we don’t put this into pri­mary leg­is­la­tion, it’s some­thing that can slip fur­ther and fur­ther down min­is­ters’ in-trays. Whilst [some] min­is­ters may have a de­sire to ad­dress it, other, more press­ing, im­me­di­ate prob­lems dis­tract their at­ten­tion.”

One could ar­gue that if the gov­ern­ment is se­ri­ous about hold­ing it­self to the same stan­dards as the crit­i­cal ser­vice providers in scope of the CSR bill, it would just bring it­self and lo­cal au­thor­i­ties also into scope.

Neil Brown, di­rec­tor at British law firm de­coded.le­gal, told The Register: “The ar­gu­ment is that gov­ern­ment de­part­ments will be held to stan­dards equiv­a­lent to those set out in the bill, and so do not need to be in­cluded. This does not fill me with con­fi­dence.

“If the gov­ern­ment is go­ing to hold it­self to stan­dards equiv­a­lent to those set out in the bill, then it has noth­ing to fear from be­ing in­cluded in the bill since, by de­f­i­n­i­tion, it will be com­pli­ant.”

Labour MP Matt Western, who also chairs the National Security Strategy joint com­mit­tee, sug­gested that the CSR bill would not be a cure-all, but the first of many pieces of be­spoke leg­is­la­tion the gov­ern­ment will pass to im­prove na­tional se­cu­rity.

This sug­gests the gov­ern­ment is con­sid­er­ing spe­cific leg­is­la­tion to shore up pub­lic sec­tor se­cu­rity fur­ther down the line. Perhaps this is wish­ful think­ing.

Brown told us “separate leg­is­la­tion does not sound like a ter­ri­ble idea,” and notes that ex­ist­ing UK tele­coms law is sep­a­rated for ef­fect.

The Telecommunications (Security) Act 2021 and the Product Security and Telecommunications Infrastructure Act 2022, for ex­am­ple, both seek to im­prove se­cu­rity in the telco space, but tar­get dif­fer­ent or­ga­ni­za­tions. Security re­quire­ments of­ten dif­fer be­tween types of or­ga­ni­za­tion, so po­ten­tially re­serv­ing a pub­lic sec­tor-spe­cific cy­ber­se­cu­rity bill could be the way to go.

Ministers’ plans also in­clude a pro­vi­sion in the bill to in­tro­duce new leg­isla­tive amend­ments as needed, to meet the de­mands of a rapidly shift­ing cy­ber­se­cu­rity land­scape, leav­ing be­hind the Brexit-related hin­drances that de­layed the pre­vi­ous NIS up­dates in the first place.

However, the like­li­hood of be­ing able to de­liver on ef­fec­tive leg­isla­tive amend­ments at pace is un­cer­tain.

Arguably, if the gov­ern­ment wanted to do it cor­rectly, it would carry out a com­pre­hen­sive (and lengthy) in­dus­try con­sul­ta­tion be­fore push­ing any amend­ments through the two Houses, an­other typ­i­cally ar­du­ous process.

Whether this way of it­er­at­ing on ex­ist­ing law could bal­ance speed with com­pre­hen­sive­ness in unan­swered.

For Brown, the ap­proach taken by Labour — to leg­is­late in smaller steps — seems like the smarter choice.

“My pref­er­ence is to leg­is­late lit­tle and of­ten, it­er­at­ing as needed, rather than try­ing to cre­ate one piece of leg­is­la­tion which is all things to all peo­ple,” he says. “Legislation in­evitably en­tails com­pro­mise, and of­ten re­flects the di­ver­gent in­ter­ests of nu­mer­ous in­ter­ested par­ties (including lob­by­ing groups) — I look, for in­stance, at the Online Safety Act 2023. Smaller bills/​acts, more tar­geted in scope, re­spond­ing to a clearly-ar­tic­u­lated prob­lem state­ment, seems more sen­si­ble to me.

“As to whether the CSR would re­sult in a bet­ter out­come than NIS2, I’m afraid I do not know.”

Given the scale of the cy­ber threat fac­ing the UK’s pub­lic sec­tor, fail­ing to ac­count for this in the CSR bill could open the gov­ern­ment up to in­tense scrutiny.

The National Audit Office’s re­port into UK gov­ern­ment se­cu­rity im­prove­ments in January 2025 laid bare the sorry state of its sys­tems. Of the 72 most crit­i­cal sys­tems run by var­i­ous de­part­ments, 58 were re­viewed; au­di­tors found a litany of se­cu­rity flaws across them and noted a stag­ger­ingly slow pace at which the is­sues were be­ing ad­dressed.

That is not an as­sess­ment which goes hand-in-hand with a pub­lic sec­tor free from reg­u­lar cy­ber­at­tacks.

Each time a cen­tral au­thor­ity, ar­m’s-length body, lo­cal coun­cil, or NHS trust is com­pro­mised, the gov­ern­men­t’s de­ci­sion not to in­clude the pub­lic sec­tor within the scope of the CSR bill hands the op­po­si­tion an­other op­por­tu­nity to ques­tion its com­mit­ment to cy­ber­se­cu­rity.

Labour does, at least, have some ammo to fire back if this sce­nario were to ever be­come re­al­ity, with the Conservatives hav­ing failed to en­act the cy­ber­se­cu­rity rec­om­men­da­tions from its 2022 con­sul­ta­tion, de­spite hav­ing had more than two years to do so.

Even with the gov­ern­men­t’s Cyber Action Plan, its re­luc­tance to bring the pub­lic sec­tor into the scope of its flag­ship cy­ber leg­is­la­tion fails to in­spire any con­fi­dence that it has se­ri­ous am­bi­tions to im­prove se­cu­rity in this prob­lem area. ®

We dis­cov­ered that en­tire sec­tions de­scrib­ing do­mes­tic U. S. in­tel­li­gence fa­cil­i­ties were de­lib­er­ately re­moved from two pub­lished doc­u­ments, while equiv­a­lent for­eign fa­cil­i­ties re­mained vis­i­ble. The ev­i­dence ex­ists in an un­ex­pected place - the PDF meta­data of doc­u­ments pub­lished by The Intercept in 2016, and by The Intercept and the Australian Broadcasting Corporation in a 2017 col­lab­o­ra­tive in­ves­ti­ga­tion. To our knowl­edge, this is the first time this in­for­ma­tion has been re­vealed pub­licly. The re­moved sec­tions re­veal the op­er­a­tional des­ig­na­tions and cover name struc­ture for do­mes­tic U.S. NRO Mission Ground Stations.

Using PDF analy­sis tools, we found hid­den text em­bed­ded in the meta­data ver­sion­ing of two doc­u­ments pub­lished along­side in­ves­tiga­tive sto­ries about NSA satel­lite sur­veil­lance fa­cil­i­ties. These meta­data ar­ti­facts prove that ear­lier ver­sions of the doc­u­ments con­tained de­tailed de­scrip­tions of do­mes­tic U. S. ground sta­tions that were sys­tem­at­i­cally scrubbed be­fore pub­li­ca­tion (not just redacted with black boxes, but with text com­pletely re­moved).

What was pub­lished from the Snowden doc­u­ments:

What was re­moved from the pub­lished doc­u­ments:

The fa­cil­i­ties them­selves are not un­known. “Aerospace Data Facility” at Buckley Space Force Base is pub­licly ac­knowl­edged as a National Reconnaissance Office (NRO) Mission Ground Station. “Classic Wizard Reporting and Testing Center” at Naval Research Laboratory is pub­licly ac­knowl­edged, though its des­ig­na­tion as a Mission Ground Station is less clear. What’s NOT pub­lic (until now) is the spe­cific op­er­a­tional des­ig­na­tions used in clas­si­fied net­works: “Consolidated Denver Mission Ground Station (CDMGS)” and “Potomac Mission Ground Station (PMGS).” The Snowden doc­u­ments prove these are de­lib­er­ate cover names (not just al­ter­na­tive ter­mi­nol­ogy) and show ex­actly what’s clas­si­fied and what’s not.

The first PDF doc­u­ment ti­tled “Menwith satel­lite clas­si­fi­ca­tion guide” has two ver­sions in the file meta­data: an older one and a newer one. The re­moved in­for­ma­tion ex­ists in the ear­lier ver­sion, and is com­pletely re­moved in the sec­ond, pub­lished ver­sion. This is not stan­dard redac­tion with black boxes - the text was com­pletely deleted from the vis­i­ble doc­u­ment while re­main­ing em­bed­ded in the PDF’s in­ter­nal ver­sion his­tory.

Screenshot from the first ver­sion of the doc­u­ment, con­tain­ing the hid­den text (sections 5.1.5.2 - 5.1.5.6).

Screenshot from the sec­ond ver­sion of the doc­u­ment, where the text is re­moved.

5.1.5.2 (U) Facility Name: Formally iden­ti­fied as the ,Mission Support Facility(MSF) also re-

ferred as the Classic Wizard Reporting and Testing Center (CWRTC).

5.1.5.3 (S//TK) Cover Story: The fact of a cover story is S//TK, the cover story it­self is un­clas-

si­fied.

5.1.5.4 (U) Software de­vel­op­ment, main­te­nance, test­ing, and com­mu­ni­ca­tions sup­port to a

world-wide Navy com­mu­ni­ca­tions and re­port­ing sys­tem.

5.1.5.5 (U) Associations:

1. The term Potomac Mission Ground Station (PMGS)=S//TK

2. The term Classic Wizard Reporting and Testing Center

(CWRTC)=UNCLASSIFIED

3. The term Naval Research Laboratory=UNCLASSIFIED

4. The fact that CWRTC is the cover name for the PMGS=S//TK

5. The fact that CWRTC is a com­mu­ni­ca­tions and data re­lay lo­ca­tion for the

US=UNCLASSIFIED (no as­so­ci­a­tion w/​NRO)

6. The fact that PMGS is lo­cated on the NRL=S//TK

7. The fact that the NRO has a MGS lo­cated on the NRL=S//TK

8. The fact that the CWRTC is lo­cated on the NRL=UNCLASSIFIED (no as­so­ci­a­tion

w/​NRO)

9. CWRTC as­so­ci­ated w/​NRO=S//​TK

10. Association of NRO, CIA, or NSA per­son­nel with the CWRTC=S//TK

11. Association of CWRTC with other NRO MGS=S//TK

12. Association of MSF with the NRO=S//TK.

13. Association of CWRTC with the ADF=UNCLASSIFIED (no as­so­ci­a­tion w/​NRO)

5.1.5.6 (U) Visitors: CWRTC is housed within build­ings 259 and 260 on the Naval Research

Laboratory in Southwest Washington, DC.

The sec­ond doc­u­ment “NRO SIGINT Guide for Pine Gap” also has two ver­sions in the file meta­data: an older one and a newer one.

Screenshot from the first ver­sion of the doc­u­ment, con­tain­ing the hid­den text (section 5.1.2).

Screenshot from the sec­ond ver­sion of the doc­u­ment, where the text is re­moved.

5.1.2 (S//TK) Consolidated Denver Mission Ground Station (CDMGS)

5.1.2.1 (U) Facility Name: Aerospace Data Facility (ADF)

5.1.2.2 (S//TK) Cover Story: The fact of a cover story is S/TK, the cover story it­self is un­classi-

fied.

In the “hidden ver­sion” of the doc­u­ment “Menwith satel­lite clas­si­fi­ca­tion guide” sec­tion 5.1.5 de­scribes a fa­cil­ity for­mally iden­ti­fied by its cover name “Mission Support Facility (MSF),” also re­ferred to as the “Classic Wizard Reporting and Testing Center (CWRTC).” The clas­si­fied op­er­a­tional des­ig­na­tion is Potomac Mission Ground Station (PMGS).

Public cover story: “Software de­vel­op­ment, main­te­nance, test­ing, and com­mu­ni­ca­tions sup­port to a world-wide Navy com­mu­ni­ca­tions and re­port­ing sys­tem.”

The doc­u­ment ex­plic­itly states that “the fact of a cover story is S//TK” - mean­ing even ac­knowl­edg­ing that CWRTC is a cover name (rather than the fa­cil­i­ty’s real iden­tity) is clas­si­fied Secret/Talent Keyhole.

The clas­si­fi­ca­tion guide pro­vides a de­tailed break­down of what’s pub­lic and what’s se­cret:

The fact that CWRTC is a cover name for PMGS = S//TK (SECRET)

Association of NRO, CIA, or NSA per­son­nel with CWRTC = S//TK (SECRET)

This lay­ered clas­si­fi­ca­tion al­lows the fa­cil­ity to op­er­ate un­der a pub­lic iden­tity while keep­ing its ac­tual in­tel­li­gence des­ig­na­tion and func­tion se­cret.

In the “hidden ver­sion” of the doc­u­ment “NRO SIGINT Guide for Pine Gap” sec­tion 5.1.2 de­scribes a fa­cil­ity iden­ti­fied by the cover name “Aerospace Data Facility (ADF)”. The clas­si­fied op­er­a­tional des­ig­na­tion is “Consolidated Denver Mission Ground Station (CDMGS).”

Public ac­knowl­edg­ment: Publicly known as “Aerospace Data Facility-Colorado.” For ex­am­ple ADF-C is openly de­scribed in Wikipedia as an NRO fa­cil­ity for “command and con­trol of re­con­nais­sance satel­lites.”

What’s NOT pub­lic: The op­er­a­tional des­ig­na­tion “Consolidated Denver Mission Ground Station” and the fact that “Aerospace Data Facility” is a cover name for it.

As with PMGS, the clas­si­fi­ca­tion guide shows “the fact of a cover story is S//TK” - the de­cep­tion it­self is clas­si­fied.

Both hid­den fa­cil­i­ties ap­pear in an or­ga­ni­za­tional clas­si­fi­ca­tion table in the Pine Gap guide. The table’s vi­sual struc­ture re­veals which names re­fer to the same fa­cil­i­ties through de­lib­er­ate for­mat­ting. Thicker bor­ders sep­a­rate dis­tinct fa­cil­ity groups, while thin­ner bor­ders sep­a­rate columns within each group (this might be a bit hard to see from the screen­shot/​doc­u­ment). Each fa­cil­ity ap­pears mul­ti­ple times - once un­der its clas­si­fied op­er­a­tional des­ig­na­tion, and again un­der its un­clas­si­fied cover name(s):

For each fa­cil­ity, the real op­er­a­tional des­ig­na­tion (CDMGS, PMGS, HMGS, AMGS) has all agency as­so­ci­a­tions clas­si­fied S//TK (Secret). But the cover names have NSA and NGA as­so­ci­a­tions marked as un­clas­si­fied (U) or For Official Use Only (U//FOUO).

This cre­ates a clas­si­fi­ca­tion sys­tem where you can pub­licly ac­knowl­edge in­tel­li­gence work us­ing san­i­tized fa­cil­ity names, while the ac­tual op­er­a­tional des­ig­na­tions re­main se­cret.

The do­mes­tic fa­cil­i­ties use two cover names each (ADF + FSD for Denver; MSF + CWRTC for Potomac), while for­eign fa­cil­i­ties use only one. This sug­gests lay­ered op­er­a­tional se­cu­rity for fa­cil­i­ties on U. S. soil - pos­si­bly to nav­i­gate Congressional over­sight, le­gal re­stric­tions on do­mes­tic in­tel­li­gence op­er­a­tions, or dif­fer­ent au­di­ences re­quir­ing dif­fer­ent lev­els of plau­si­ble de­ni­a­bil­ity.

There is a de­lib­er­ate pat­tern in these two clas­si­fi­ca­tion guides: de­tailed sec­tions de­scrib­ing spe­cific U. S. fa­cil­i­ties were re­moved, while equiv­a­lent for­eign fa­cil­ity sec­tions were pub­lished in­tact. The 2016 Menwith Hill guide had its PMGS sec­tion (5.1.5) com­pletely re­moved. The 2017 Pine Gap guide had its CDMGS sec­tion (5.1.2) com­pletely re­moved. Both guides re­tained their de­tailed de­scrip­tions of for­eign fa­cil­i­ties, in­clud­ing op­er­a­tional des­ig­na­tions, cover sto­ries, and vis­i­tor pro­to­cols.

U. S. fa­cil­i­ties weren’t en­tirely ab­sent from the pub­lished doc­u­ments. The Pine Gap clas­si­fi­ca­tion table shows CDMGS, PMGS, ADF, and other U.S. fa­cil­ity des­ig­na­tions along­side for­eign fa­cil­i­ties, re­veal­ing the struc­ture of the Mission Ground Station net­work. Other pub­lished doc­u­ments from both in­ves­ti­ga­tions men­tion U.S. fa­cil­i­ties. What was specif­i­cally re­moved were the de­tailed clas­si­fi­ca­tion guide sec­tions that would have ex­plained these U.S. fa­cil­i­ties the same way Menwith Hill and Pine Gap were ex­plained.

PDF meta­data pro­vides foren­sic ev­i­dence of the edit­ing process. The Pine Gap clas­si­fi­ca­tion guide shows time­stamps from July 31, 2017, three weeks be­fore pub­li­ca­tion. Two ver­sions were cre­ated min­utes apart us­ing Nitro Pro 8, a com­mer­cial PDF ed­i­tor: ver­sion 1 at 13:48:54 (containing the CDMGS sec­tion) and ver­sion 2 at 13:50:48 (with CDMGS re­moved). The Intercept and ABC pub­lished iden­ti­cal PDFs with the same meta­data ar­ti­facts, in­di­cat­ing the edit­ing was done once and the same file shared be­tween or­ga­ni­za­tions.

The Intercept, as holder of the Snowden archive, likely han­dled tech­ni­cal doc­u­ment prepa­ra­tion for pub­li­ca­tions. The Menwith Hill clas­si­fi­ca­tion guide, pub­lished solely by The Intercept in 2016, shows more thor­ough meta­data san­i­ti­za­tion but the same ed­i­to­r­ial pat­tern - do­mes­tic fa­cil­ity sec­tions re­moved while for­eign equiv­a­lents re­main.

We con­tacted Ryan Gallagher, the jour­nal­ist who led both in­ves­ti­ga­tions, to ask about the ed­i­to­r­ial de­ci­sion to re­move these sec­tions. After more than a week, we have not re­ceived a re­sponse.

The next part will be a tech­ni­cal deep-dive into PDF meta­data across the pub­lished Snowden doc­u­ments. We found that many doc­u­ments con­tain mul­ti­ple ver­sions in their meta­data, re­veal­ing the ed­i­to­r­ial redac­tion process: vis­i­ble NSA agents’ user­names that were later re­moved, screen­shots that were later redacted, and sur­veil­lance data that went through mul­ti­ple rounds of redac­tion. We’ll also doc­u­ment cases of failed redac­tions - in­clud­ing one where redacted text re­mained fully copy­able, pre­vi­ously re­ported only by a Polish cy­ber­se­cu­rity blog.

You can ex­tract ver­sions from a PDF file for ex­am­ple with a pdfres­ur­rect tool (pdfresurrect -w file­name.pdf).

You can down­load the doc­u­ment ver­sions di­rectly here:

This ar­ti­cle has been re­viewed ac­cord­ing to Science X’s editorial process

and poli­cies. Editors have high­lighted the fol­low­ing at­trib­utes while en­sur­ing the con­tent’s cred­i­bil­ity:

This ar­ti­cle has been re­viewed ac­cord­ing to Science X’s editorial process

and poli­cies. Editors have high­lighted the fol­low­ing at­trib­utes while en­sur­ing the con­tent’s cred­i­bil­ity:

Dorota Skowronska-Krawczyk sits in her of­fice, eyes fixed on the com­puter mon­i­tor in front of her. “You see it move its eye,” says the UC Irvine as­so­ci­ate pro­fes­sor of phys­i­ol­ogy and bio­physics, point­ing to an im­age of a Greenland shark slowly drift­ing through the murky Arctic Ocean. “The shark is track­ing the light—it’s fas­ci­nat­ing.”

The video shows the longest-liv­ing ver­te­brate in the world—long, thick, gray body; small head; and short, rounded snout—with opaque eyes that ap­pear life­less, ex­cept for the par­a­site latched to one of its eye­balls. Scientists have long sus­pected the large species to be func­tion­ally blind, given the fre­quent pres­ence of the par­a­site and its ex­cep­tion­ally dim and ob­structed vi­sual en­vi­ron­ment.

Now, new re­search from Skowronska-Krawczyk on Greenland shark vi­sion—co-au­thored by University of Basel, Switzerland re­searchers Walter Salzburger and Lily G. Fogg, who worked on the evo­lu­tion­ary as­pect of the study—is chal­leng­ing what we know about ag­ing, vi­sion and longevity.

Published in Nature Communications, her find­ings sug­gest that a DNA re­pair mech­a­nism en­ables these sharks—some of which live for 400 years—to main­tain their vi­sion over cen­turies with no signs of reti­nal de­gen­er­a­tion and that they are well adapted to ex­treme low-light con­di­tions.

Skowronska-Krawczyk, who gleans in­sights into the mol­e­c­u­lar mech­a­nisms of ag­ing by study­ing processes that con­trol age-re­lated eye dis­eases, at­trib­utes her in­ter­est in the vi­sual sys­tem of the Greenland shark to a 2016 re­search pa­per by John Fleng Steffensen pub­lished in the jour­nal Science.

“One of my take­away con­clu­sions from the Science pa­per was that many Greenland sharks have par­a­sites at­tached to their eyes—which could im­pair their vi­sion,” she says. “Evolutionarily speak­ing, you don’t keep the or­gan that you don’t need. After watch­ing many videos, I re­al­ized this an­i­mal was mov­ing its eye­balls to­ward the light.”

This left Skowronska-Krawczyk want­ing to learn more.

The Greenland sharks used in her co-study were caught be­tween 2020 and 2024 us­ing sci­en­tific long lines off the coast of the University of Copenhagen’s Arctic Station on Disko Island, Greenland. Steffensen, pro­fes­sor of ma­rine bi­ol­ogy at the University of Copenhagen, and col­leagues Peter G. Bushnell, who teaches at Indiana University South Bend, and Richard W. Brill, who’s based at the Virginia Institute of Marine Science, dis­sected and pre­served the eye­balls in a fix­a­tive so­lu­tion for ex­am­i­na­tion.

Emily Tom, a UC Irvine Ph. D. stu­dent and physi­cian-sci­en­tist in train­ing who works in Skowronska-Krawczyk’s lab, re­calls re­ceiv­ing the box that held a fixed eye­ball.

“I opened the pack­age, and there was a gi­ant, 200-year-old eye­ball sit­ting on dry ice just star­ing back at me,” the 28-year-old says with a laugh. “We’re used to work­ing with mouse eye­balls, which are the size of a pa­paya seed, so we had to fig­ure out how to scale up to a base­ball-sized eye­ball. Luckily, Dorota is very hands-on, both in her men­tor­ing style and in the lab—which you don’t see a lot of with pro­fes­sors.”

Tom then let the eye­ball de­frost. “The lab smelled like a fish mar­ket,” she says.

She em­pha­sizes that it was a care­ful bal­ance of not let­ting it thaw too much be­cause once tis­sue sam­ples reach room tem­per­a­ture, they be­gin to de­grade. Her role in­volved his­to­log­i­cal and vi­sion-spe­cific analy­ses of the eye­ball, find­ing no signs of cell death, and re­veal­ing that rhodopsin (a pro­tein es­sen­tial for vi­sion in dim light) in the shark reti­nas re­mains ac­tive and is tuned to de­tect blue light.

“Not a lot of peo­ple are work­ing on sharks, es­pe­cially shark vi­sion,” Tom says. “We can learn so much about vi­sion and longevity from long-lived species like the Greenland shark, so hav­ing the funds to do re­search like this is very im­por­tant.”

For Skowronska-Krawczyk, the find­ings open the door to dis­cov­er­ing new ap­proaches to avoid­ing age-re­lated vi­sion loss and erad­i­cat­ing eye dis­eases such as mac­u­lar de­gen­er­a­tion and glau­coma—and to more ques­tions about how vi­sion evolves, the mech­a­nisms that help keep tis­sues alive and healthy for many years, and how to ap­ply this knowl­edge to hu­mans.

She notes that with fed­eral re­search fund­ing un­der threat, fu­ture sup­port for her stud­ies is a con­cern, but she be­lieves that “we will pre­vail.”

“What I love about my work is that we are the first in the world to see re­sults—at the fore­front, find­ing new mech­a­nisms, rules and dis­cov­er­ies,” Skowronska-Krawczyk says, look­ing over at the paused shark on the screen. “Then, be­ing able to share this joy with stu­dents—that’s the best part of it.”

A few months ago, users started re­port­ing that Ghostty was con­sum­ing ab­surd amounts of mem­ory, with one user re­port­ing 37 GB af­ter 10 days of up­time. Today, I’m happy to say the fix has been found and merged. This post is an overview of what caused the leak, a look at some of Ghostty’s in­ter­nals, and some brief de­scrip­tions of how we tracked it down.1

The leak was pre­sent since at least Ghostty 1.0, but it is only re­cently that pop­u­lar CLI ap­pli­ca­tions (particularly Claude Code) started pro­duc­ing the cor­rect con­di­tions to trig­ger it at scale. The lim­ited con­di­tions that trig­gered the leak are what made it par­tic­u­larly tricky to di­ag­nose.

The fix is merged and is avail­able in tip/​nightly re­leases, and will be part of the tagged 1.3 re­lease in March.

To un­der­stand the bug, we first need to un­der­stand how Ghostty man­ages ter­mi­nal mem­ory. Ghostty uses a data struc­ture called the

PageList

to store ter­mi­nal con­tent. PageList is a dou­bly-linked list of mem­ory pages that store the ter­mi­nal con­tent (characters, styles, hy­per­links, etc.).

The un­der­ly­ing “pages” are not sin­gle vir­tual mem­ory pages

but they are a con­tigu­ous block of mem­ory aligned to page bound­aries and com­posed of an even mul­ti­ple of sys­tem pages.2

These pages are al­lo­cated us­ing mmap. mmap is­n’t par­tic­u­larly fast, so to avoid con­stant syscalls, we use a mem­ory pool. When we need a new page, we pull from the pool. When we’re done with a page, we re­turn it to the pool for reuse.

The pool uses a stan­dard size for pages. Think of it like buy­ing stan­dard-sized ship­ping boxes: most things peo­ple ship fit in a stan­dard box, and hav­ing a stan­dard box comes with var­i­ous ef­fi­cien­cies.

But some­times ter­mi­nals need more mem­ory than a stan­dard page pro­vides. If a set of lines has many emoji, styles, or hy­per­links, we need a larger page. In these cases, we al­lo­cate a non-stan­dard page

di­rectly with mmap, by­pass­ing the pool en­tirely. This is typ­i­cally a rare sce­nario.

When we “free” a page, we ap­ply some sim­ple logic:

If the page is : re­turn it to the pool

If the page is > stan­dard size: call mun­map to free it

This is the core back­ground for ter­mi­nal mem­ory man­age­ment in Ghostty, and the idea it­self is sound. A logic bug around an op­ti­miza­tion is what pro­duced the leak, as we’ll see next.

There’s one more back­ground de­tail we need to cover to un­der­stand the bug: scroll­back prun­ing.

Ghostty has a scroll­back-limit con­fig­u­ra­tion that caps how much his­tory is re­tained. When you hit this limit, we delete the old­est pages in the scroll­back buffer to free up mem­ory.

But this of­ten hap­pens in a su­per hot path (e.g. when out­putting large amounts of data quickly), and al­lo­cat­ing and free­ing mem­ory pages is ex­pen­sive, even with the pool. Therefore, we have an op­ti­miza­tion:

reuse the old­est page as the newest page when we reach the limit.

This op­ti­miza­tion works great. It re­quires zero al­lo­ca­tions and uses only some quick pointer ma­nip­u­la­tions to move the page from the front to the back of the list. We do some meta­data cleanup to “clear” the page but oth­er­wise leave the pre­vi­ous mem­ory in­tact.

It’s fast and em­pir­i­cally speeds up scroll­back-heavy work­loads sig­nif­i­cantly.

During the scroll­back prun­ing op­ti­miza­tion, we al­ways

re­sized our page back to stan­dard size. But we did­n’t re­size the un­der­ly­ing mem­ory al­lo­ca­tion it­self, we only noted the re­size in the meta­data. The un­der­ly­ing mem­ory was still the large non-stan­dard mmap al­lo­ca­tion, but now the PageList thought it was stan­dard sized.

Eventually, we’d free the page un­der var­i­ous cir­cum­stances (e.g. when the user closes the ter­mi­nal, but also other times). At that point, we’d see the page mem­ory was within the stan­dard size, as­sume it was part of the pool, and we would never call mun­map on it. A clas­sic leak.

This all seems pretty ob­vi­ous, but the is­sue is that non-stan­dard pages are rare by de­sign. The goal of our de­sign and op­ti­miza­tions is that stan­dard pages are the com­mon case and pro­vide a fast-path. Only very spe­cific sce­nar­ios pro­duce non-stan­dard pages and they’re usu­ally not pro­duced in large quan­ti­ties.

But the rise of Claude Code

changed this. For some rea­son, Claude Code’s CLI pro­duces a lot of multi-code­point grapheme out­puts which force Ghostty to reg­u­larly use non-stan­dard pages. Additionally, Claude Code uses the pri­mary screen and pro­duces a sig­nif­i­cant amount of scroll­back out­put. These things com­bined to­gether cre­ated the per­fect storm to trig­ger the leak in huge quan­ti­ties.

The fix is con­cep­tu­ally sim­ple: never reuse non-stan­dard pages. If we en­counter a non-stan­dard page dur­ing scroll­back prun­ing, we de­stroy it prop­erly (calling mun­map) and al­lo­cate a fresh stan­dard-sized page from the pool.

The core of the fix is in the snip­pet be­low, but some ex­tra work was needed to fix up some other bits of ac­count­ing we have:

We could’ve also reused the non-stan­dard page and just re­tained the large mem­ory size, but un­til we have data that shows oth­er­wise, we’re still op­er­at­ing un­der the as­sump­tion that stan­dard pages are the com­mon case and it makes sense to re­set back to a stan­dard pooled page.

Other users have rec­om­mended more com­plex strate­gies (e.g. main­tain­ing some met­rics on how of­ten non-stan­dard pages are used and ad­just­ing our as­sump­tions ac­cord­ingly), but more re­search is needed be­fore mak­ing those changes. This change is sim­ple, fixes the bug, and aligns with our cur­rent as­sump­tions.

As part of the fix, I added sup­port for vir­tual mem­ory tags on ma­cOS pro­vided by the Mach ker­nel. This lets us tag our PageList mem­ory al­lo­ca­tions with a spe­cific iden­ti­fier that shows up in var­i­ous tool­ing.

Now when de­bug­ging mem­ory on ma­cOS, Ghostty’s PageList mem­ory shows up with a spe­cific tag in­stead of be­ing lumped in with every­thing else. This made it triv­ial to iden­tify the leak, as­so­ci­ate it with the PageList, and also ver­ify that the fix worked by ob­serv­ing the tagged mem­ory be­ing prop­erly freed.

We do a lot of work in the Ghostty pro­ject to find and pre­vent mem­ory leaks:

* In de­bug builds and unit tests, we use leak-de­tect­ing Zig al­lo­ca­tors.

* The CI runs val­grind on our full unit test suite on every com­mit

to find more than just leaks, such as un­de­fined mem­ory us­age.

* We reg­u­larly run the ma­cOS GUI via ma­cOS Instruments to look for

leaks par­tic­u­larly in the Swift code­base.

* We run every GTK-related PR us­ing Valgrind (the full GUI) to look

for leaks in the GTK code­path that is­n’t unit tested.

This has worked re­ally well to date, but un­for­tu­nately it did­n’t catch this par­tic­u­lar leak be­cause it only trig­gers un­der very spe­cific con­di­tions that our tests did­n’t re­pro­duce. The merged PR in­cludes a test that does re­pro­duce the leak to pre­vent re­gres­sions in the fu­ture.

This was the largest known mem­ory leak in Ghostty to date, and the only re­ported leak that has been con­firmed by more than a sin­gle user. We’ll con­tinue to mon­i­tor and ad­dress mem­ory re­ports as they come in, but re­mem­ber that re­pro­duc­tion is the key to di­ag­nos­ing and fix­ing mem­ory leaks!

Big thanks to @grishy who fi­nally got me a re­li­able re­pro­duc­tion so I could an­a­lyze the is­sue my­self. Their own analy­sis reached the same con­clu­sion as mine, and the re­pro­duc­tion let me ver­ify both our un­der­stand­ings in­de­pen­dently.

Thanks also to every­one who re­ported this is­sue with de­tailed di­ag­nos­tics. The com­mu­ni­ty’s analy­sis, es­pe­cially around the foot­print out­put and VM re­gion count­ing, gave me im­por­tant clues that pointed to­ward the PageList as the cul­prit.