In 2024, Apple signed a deal with Taboola to serve ads in its app, no­tably Apple News. John Gruber, writ­ing in Daring Fireball said at the time:

If you told me that the ads in Apple News have been sold by Taboola for the last few years, I’d have said, “Oh, that makes sense.” Because the ads in Apple News — at least the ones I see1 — al­ready look like chum­box Taboola ads. Even worse, they’re in­cred­i­bly rep­e­ti­tious.

I use Apple News to keep up on top­ics that I don’t find in sources I pay for (The Guardian and The New York Times). But there’s no way I’m go­ing to pay the ex­or­bi­tant price Apple wants for Apple News+ — £13 — be­cause, while you get more pub­li­ca­tions, you still get ads.

And those ads have got­ten worse re­cently. Many if not most of them look like and prob­a­bly are scams. Here are a few ex­am­ples from Apple News to­day.

Here are three ads that are scammy; the first two were clearly gen­er­ated by AI, and the third may have been cre­ated by AI.

Why are they scams? When I searched do­main in­for­ma­tion for the do­mains, I found that they were reg­is­tered very re­cently.

This re­cent reg­is­tra­tion does­n’t nec­es­sar­ily mean they are scams, but they don’t in­spire much con­fi­dence.

Here’s one ex­am­ple. This ad from Tidenox, whose web­site says I am re­tir­ing, show­ing a photo of an el­derly woman, who says, “For 26 years, Tidenox has been port of your jour­ney in cre­at­ing earth and com­fort at home.” The im­age of the re­tir­ing owner is prob­a­bly made by AI. (Update: some­one on Hacker News pointed out the partly masked Google Gemini logo on the bot­tom right. I had­n’t spot­ted that, in part be­cause I don’t use any AI im­age gen­er­a­tion tools.)

These fake “going out of busi­ness ads” have been around for a few years, and even the US Better Business Bureau warns about them, as they take peo­ples’ money then shut down. Does Apple care? Does Taboola care? Does Apple care that Taboola serves ads like this? My guess: no, no, and no.

Note the reg­is­tra­tion date for the tide­nox.com do­main. It’s nowhere near 26 years old, and it’s reg­is­tered in China:

Shame on Apple for cre­at­ing a hon­ey­pot for scam ads in what they con­sider to be a pre­mium news ser­vice. This com­pany can­not be trusted with ads in its prod­ucts any more.

Think of your data­base like your home. Your home has a liv­ing room, bed­room, bath­room, kitchen, and garage. Each room serves a dif­fer­ent pur­pose. But they’re all un­der the same roof, con­nected by hall­ways and doors. You don’t build a sep­a­rate restau­rant build­ing just be­cause you need to cook. You don’t con­struct a com­mer­cial garage across town just to park your car.

That’s what Postgres is. One home with many rooms. Search, vec­tors, time-se­ries, queues—all un­der one roof.

But this is ex­actly what spe­cial­ized data­base ven­dors don’t want you to hear. Their mar­ket­ing teams have spent years con­vinc­ing you to “use the right tool for the right job.” It sounds rea­son­able. It sounds wise. And it sells a lot of data­bases.

Let me show you why it’s a trap and why Postgres is the bet­ter choice in 99% of cases.

You’ve heard the ad­vice: “Use the right tool for the right job.”

Sounds wise. So you end up with:

Congratulations. You now have seven data­bases to man­age. Seven query lan­guages to learn. Seven backup strate­gies to main­tain. Seven se­cu­rity mod­els to au­dit. Six sets of cre­den­tials to ro­tate. Seven mon­i­tor­ing dash­boards to watch. And seven things that can break at 3 AM.

And when some­thing does break? Good luck spin­ning up a test en­vi­ron­ment to de­bug it.

Here’s a dif­fer­ent idea: Just use Postgres.

This is­n’t just about sim­plic­ity. AI agents have made data­base sprawl a night­mare.

Think about what agents need to do:

With one data­base? That’s a sin­gle com­mand. Fork it, test it, done.

With seven data­bases? Now you need to:

* Make sure they’re all at the same point in time

* Spin up seven dif­fer­ent ser­vices

* Tear down seven ser­vices when you’re done

This is vir­tu­ally im­pos­si­ble with­out a ton of R&D.

And it’s not just agents. Every time some­thing breaks at 3 AM, you need to spin up a test en­vi­ron­ment to de­bug. With six data­bases, that’s a co­or­di­na­tion night­mare. With one data­base, it’s a sin­gle com­mand.

In the AI era, sim­plic­ity is­n’t just el­e­gant. It’s es­sen­tial.

The myth: Specialized data­bases are far su­pe­rior at their spe­cific tasks.

The re­al­ity: Sometimes they’re mar­gin­ally bet­ter at a nar­row task. But they also bring un­nec­es­sary com­plex­ity. It’s like hir­ing a pri­vate chef for every meal. Sounds lux­u­ri­ous, but it adds ex­pense, co­or­di­na­tion over­head, and cre­ates prob­lems you did­n’t have be­fore.

Here’s the thing: 99% of com­pa­nies don’t need them. The top 1% have tens of mil­lions of users and a large en­gi­neer­ing team to match. You’ve read their blog posts about how amaz­ing Specialized Database X works for them. But that’s their scale, their team, their prob­lems. For every­one else, Postgres is more than enough.

Here’s what most peo­ple don’t re­al­ize: Postgres ex­ten­sions use the same or bet­ter al­go­rithms as spe­cial­ized data­bases (in many cases).

These aren’t wa­tered-down ver­sions. They’re the same/​bet­ter al­go­rithms, bat­tle-tested, open source, and of­ten de­vel­oped by the same re­searchers.

* pgvec­torscale: 28x lower la­tency than Pinecone at 75% less cost

* pg_­textsearch: The ex­act same BM25 rank­ing that pow­ers Elasticsearch

Beyond the AI/agent prob­lem, data­base sprawl has com­pound­ing costs:

Cognitive load: Your team needs SQL, Redis com­mands, Elasticsearch Query DSL, MongoDB ag­gre­ga­tion, Kafka pat­terns, and InfluxDB’s non-na­tive SQL workaround. That’s not spe­cial­iza­tion. That’s frag­men­ta­tion.

Data con­sis­tency: Keeping Elasticsearch in sync with Postgres? You build sync jobs. They fail. Data drifts. You add rec­on­cil­i­a­tion. That fails too. Now you’re main­tain­ing in­fra­struc­ture in­stead of build­ing fea­tures.

SLA math: Three sys­tems at 99.9% up­time each = 99.7% com­bined. That’s 26 hours of down­time per year in­stead of 8.7. Every sys­tem mul­ti­plies your fail­ure modes.

These ex­ten­sions aren’t new. They’ve been pro­duc­tion-ready for years:

* JSONB: Since 2014 (11 years). As fast as MongoDB, with ACID.

Over 48,000 com­pa­nies use PostgreSQL, in­clud­ing Netflix, Spotify, Uber, Reddit, Instagram, and Discord.

What this means: Building a RAG app used to re­quire Postgres + Pinecone + Elasticsearch + glue code.

Now? Just Postgres. One data­base. One query lan­guage. One backup. One fork com­mand for your AI agent to spin up a test en­vi­ron­ment.

Here’s all you need:

Below are work­ing ex­am­ples for each use case. Skip to what you need.

What you get: The ex­act same BM25 al­go­rithm that pow­ers Elasticsearch, di­rectly in Postgres.

This is what Elasticsearch re­quires a sep­a­rate plu­gin for. In Postgres, it’s just SQL.

What you get: pgvec­torscale uses the DiskANN al­go­rithm (from Microsoft Research), achiev­ing 28x lower p95 la­tency and 16x higher through­put than Pinecone at 99% re­call.

Now every INSERT/UPDATE au­to­mat­i­cally re­gen­er­ates em­bed­dings. No sync jobs. No drift. No 3 AM pages.

* Prometheus: Great for met­rics, not your ap­pli­ca­tion data

What you get: Automatic time par­ti­tion­ing, com­pres­sion up to 90%, con­tin­u­ous ag­gre­gates. Full SQL.

For AI ap­pli­ca­tions, you of­ten need both key­word search and se­man­tic search:

Try that with Elasticsearch + Pinecone. You’d need two API calls, re­sult merg­ing, fail­ure han­dling, and dou­ble la­tency.

In Postgres: one query, one trans­ac­tion, one re­sult.

Remember the home anal­ogy? You don’t build a sep­a­rate restau­rant just to cook din­ner. You don’t con­struct a com­mer­cial garage across town just to park your car. You use the rooms in your home.

That’s what we’ve shown you here. Search, vec­tors, time-se­ries, doc­u­ments, queues, caching—they’re all rooms in the Postgres home. Same al­go­rithms as the spe­cial­ized data­bases. Battle-tested for years. Used by Netflix, Uber, Discord, and 48,000 other com­pa­nies.

So what about that 99%?

For 99% of com­pa­nies, Postgres han­dles every­thing you need. The 1%? That’s when you’re pro­cess­ing petabytes of logs across hun­dreds of nodes, or you need Kibana’s spe­cific dash­boards, or you have ex­otic re­quire­ments that gen­uinely ex­ceed what Postgres can do.

But here’s the thing: you’ll know when you’re in the 1%. You won’t need a ven­dor’s mar­ket­ing team to tell you. You’ll have bench­marked it your­self and hit a real wall.

Until then, don’t scat­ter your data across seven build­ings be­cause some­one told you to “use the right tool for the right job.” That ad­vice sells data­bases. It does­n’t serve you.

Start with Postgres. Stay with Postgres. Add com­plex­ity only when you’ve earned the need for it.

In 2026, just use Postgres.

All these ex­ten­sions are avail­able on Tiger Data. Create a free data­base in min­utes:

No need for spe­cial­ized data­bases, just use Postgres.

LinkedIn silently probes for 2,953 Chrome ex­ten­sions on every page load.

This repos­i­tory doc­u­ments every ex­ten­sion LinkedIn checks for and pro­vides tools to iden­tify them.

The com­plete list of ex­ten­sions with names and Chrome Web Store links:

Fetches ex­ten­sion names from Chrome Web Store with Extpose fall­back for re­moved/​un­avail­able ex­ten­sions.

# Fetch all ex­ten­sions

node fetch_ex­ten­sion_­names.js

# Fetch a sub­set (useful if rate lim­ited)

node fetch_ex­ten­sion_­names.js –offset 0 –limit 500

node fetch_ex­ten­sion_­names.js -o 500 -l 500

# Show help

node fetch_ex­ten­sion_­names.js –help

Test script that processes the first 3 ex­ten­sions with ver­bose out­put.

node test_fetch.js

* ~22% found via Extpose fall­back (removed or un­avail­able on Chrome Web Store)

There have been a lot of com­plaints about both the com­pe­tency and the logic be­hind the lat­est Epstein archive re­lease by the DoJ: from cen­sor­ing the names of co-con­spir­a­tors to cen­sor­ing pic­tures of ran­dom women in a way that makes in­di­vid­u­als look guiltier than they re­ally are, for­get­ting to redact cre­den­tials that made it pos­si­ble for all of Reddit to log into Epstein’s ac­count and tram­ple over all the ev­i­dence, and the com­plete in­ep­ti­tude that re­sulted in most of the lat­est batch be­ing cor­rupted thanks to in­cor­rectly con­verted Quoted-Printable en­cod­ing ar­ti­facts, it’s safe to say that Pam Bondi’s DoJ did not put its best and bright­est on this (admittedly gar­gan­tuan) un­der­tak­ing. But the most damn­ing ev­i­dence has all been thor­oughly redacted… has­n’t it? Well, maybe not.

I was think­ing of writ­ing an ar­ti­cle on the man­gled quoted-print­able en­cod­ing the day this lat­est dump came out in re­sponse to all the mis­in­formed mus­ings and con­jec­tures that were lit­ter­ing so­cial me­dia (and my dilly-dal­ly­ing cost me, as some­one beat me to the punch), and spent some time search­ing through the lat­est archives look­ing  for some SMTP head­ers that I could use in the ar­ti­cle when I came across a cu­ri­ous ar­ti­fact: not only were the emails badly transcoded into plain text, but also some bi­nary at­tach­ments were ac­tu­ally in­cluded in the dumps in their over-the-wire Content-Transfer-Encoding: base64 for­mat, and the un­lucky in­tern that was as­signed to the doc­u­ments in ques­tion did­n’t re­al­ize the sig­nif­i­cance of what they were look­ing at and did­n’t see the point in cen­sor­ing seem­ingly mean­ing­less page af­ter page of hex con­tent!

Just take a look at EFTA00400459, an email from cor­re­spon­dence be­tween (presumably) one of Epstein’s as­sis­tants and Epstein lackey/​co-con­spir­a­tor Boris Nikolic and his friend, Sam Jaradeh, invit­ing them to a ████████ ben­e­fit:

Those hex char­ac­ters go on for 76 pages, and rep­re­sent the file DBC12 One Page Invite with Reply.pdf en­coded as base64 so that it can be in­cluded in the email with­out break­ing the SMTP pro­to­col. And con­vert­ing it back to the orig­i­nal PDF is, the­o­ret­i­cally, as easy as copy-and-past­ing those 76 pages into a text ed­i­tor, strip­ping the lead­ing > bytes, and pip­ing all that into base64 -d > out­put.pdf… or it would be, if we had the orig­i­nal (badly con­verted) email and not a par­tially redacted scan of a print­out of said email with some shoddy OCR ap­plied.

If you tried to ac­tu­ally copy that text as dig­i­tized by the DoJ from the PDF into a text ed­i­tor, here’s what you’d see:

You can ig­nore the EFTA00400459 on the sec­ond line; that (or some vari­ant thereof) will be in­ter­spersed into the base64 text since it’s stamped at the bot­tom of every page to iden­tify the piece of ev­i­dence it came from. But what else do you no­tice? Here’s a hint: this is what proper base64 looks like:

Notice how in this sam­ple every­thing lines up per­fectly (when us­ing a mono­spaced font) at the right mar­gin? And how that’s not the case when we copied-and-pasted from the OCR’d PDF? That’s be­cause it was­n’t a great OCR job: ex­tra char­ac­ters have been hal­lu­ci­nated into the out­put, some of them not even le­gal base64 char­ac­ters such as the , and [, while other char­ac­ters have been omit­ted al­to­gether, giv­ing us con­tent we can’t use:1

> pb­paste \

| string match -rv ‘EFTA’ \

| string trim -c ” >” \

| string join “” \

| base64 -d >/dev/null

base64: in­valid in­put

I tried the eas­i­est al­ter­na­tive I had at hand: I loaded up the PDF in Adobe Acrobat Pro and re-ran an OCR process on the doc­u­ment, but came up with even worse re­sults, with spaces in­jected in the mid­dle of the base64 con­tent (easily fix­able) in ad­di­tion to other char­ac­ters be­ing com­pletely mis­read and butchered — it re­ally did­n’t like the cramped mono­space text at all. So I thought to do it man­u­ally with tesser­act, which, while very far from state-of-the-art, can still be use­ful be­cause it lets you do things like limit its out­put to a cer­tain sub­set of char­ac­ters, con­strain­ing the field of valid re­sults and hope­fully co­erc­ing it into pro­duc­ing bet­ter re­sults.

Only one prob­lem: tesser­act can’t read PDF in­put (or not by de­fault, any­way). No prob­lem, I’ll just use im­agemag­ick/​ghost­script to con­vert the PDF into in­di­vid­ual PNG im­ages (to avoid fur­ther gen­er­a­tional loss) and pro­vide those to tesser­act, right? But that did­n’t quite work out, they seem (?) to try to load and per­form the con­ver­sion of all 76 sep­a­rate pages/​png files all at once, and then nat­u­rally crash on too-large in­puts (but only af­ter tak­ing for­ever and gen­er­at­ing the 76 (invalid) out­put files that you’re forced to sub­se­quently clean up, of course):

> con­vert -density 300 EFTA00400459.pdf \

-background white -alpha re­move \

-alpha off out.png

con­vert-im6.q16: cache re­sources ex­hausted `/tmp/magick-QqXVSOZutVsiRcs7pLwwG2FYQnTsoAmX47′ @ er­ror/​cache.c/​Open­Pix­el­Cache/​4119.

con­vert-im6.q16: cache re­sources ex­hausted `out.png’ @ er­ror/​cache.c/​Open­Pix­el­Cache/​4119.

con­vert-im6.q16: No IDATs writ­ten into file `out-0.png’ @ er­ror/​png.c/​Mag­ickP­NGEr­rorHan­dler/​1643.

So we turn to pdftoppm from the pop­pler-utils pack­age in­stead, which does in­deed han­dle each page of the source PDF sep­a­rately and turned out to be up to the task, though in­cred­i­bly slow:

> pdftoppm -png -r 300 EFTA00400459.pdf out.png

After wait­ing the req­ui­site amount of time (and then some), I had files out-01.png through out-76.png, and was ready to try them with tesser­act:

for n in (printf “%02d\n” (seq 1 76))

tesser­act out-$n.png out­put-$n \

–psm 6 \

-c tessed­it_char_whitelist=’>’ABCDE­FGHIJKLMNOPQRSTU­VWXYZ­abcde­fghijklmnopqrstu­vwxyz0123456789+/= \

-c load­_sys­tem_­dawg=0 \

-c load­_fre­q_­dawg=0

end

The above fish-shell com­mand in­structs tesser­act(1) to as­sume the in­put is a sin­gle block of text (the –psm 6 ar­gu­ment) and limit it­self to de­cod­ing only le­gal base64 char­ac­ters (and the lead­ing > so we can prop­erly strip it out there­after). My orig­i­nal at­tempt in­cluded a lit­eral space in the valid char whitelist, but that gave me worse re­sults: the very badly kerned base64 has sig­nif­i­cant ap­par­ent spac­ing be­tween some ad­ja­cent char­ac­ters (more on this later) and that caused tesser­act to both in­cor­rectly in­ject spaces (bad but fix­able) and also pos­si­bly af­fect how it han­dled the char­ac­ter af­ter the space (worse).

Unfortunately, while tesser­act gave me slightly bet­ter out­put than ei­ther the orig­i­nal OCR’d DoJ text or the (terrible) Adobe Acrobat Pro OCR re­sults, it too suf­fered from poor recog­ni­tion and gave me very in­con­sis­tent line lengths… but it also suf­fered from some­thing that I did­n’t re­ally think a heuris­tic-based, al­go­rithm-dri­ven tool like tesser­act would suc­cumb to, as it was more rem­i­nis­cent of how first-gen­er­a­tion LLMs would be­have: in a few places, it would only read the first dozen or so char­ac­ters on a line then leave the rest of the line blank, then pick up (correctly enough) at the start of the next line. Before I saw how gen­er­ally use­less the OCR re­sults were and gave up on tesser­act, I fig­ured I’d just man­u­ally type out the rest of the line (the aborted lines were easy enough to find, thanks to the mono­spaced out­put), and that was when I ran into the real is­sue that took this from an in­ter­est­ing chal­lenge to be­ing al­most mis­sion im­pos­si­ble.

I men­tioned ear­lier the bad kern­ing, which tricked the OCR tools into in­ject­ing spaces where there were sup­posed to be none, but that was far from be­ing the worst is­sue plagu­ing the PDF con­tent. The real prob­lem is that the text is ren­dered in pos­si­bly the worst type­face for the job at hand: Courier New.

If you’re a font en­thu­si­ast, I cer­tainly don’t need to say any more — you’re prob­a­bly al­ready shak­ing with a mix of PTSD and rage. But for the ben­e­fit of every­one else, let’s just say that Courier New is… not a great font. It was a dig­i­ti­za­tion of the ven­er­a­ble (though cer­tainly prim­i­tive) Courier font­face, com­mis­sioned by IBM in the 1950s. Courier was used (with some tweaks) for IBM type­writ­ers, in­clud­ing the IBM Selectric, and in the 1990s it was “digitized di­rectly from the golf ball of the IBM Selectric” by Monotype, and shipped with Windows 3.1, where it re­mained the de­fault mono­space font on Windows un­til Consolas shipped with Windows Vista. Among the many is­sues with Courier New is that it was dig­i­tized from the Selectric golf ball “without ac­count­ing for the vi­sual weight nor­mally added by the type­writer’s ink rib­bon”, which gives its char­ac­ter­is­tic “thin” look. Microsoft ClearType, which was only en­abled by de­fault with Windows Vista, ad­dressed this ma­jor short­com­ing to some ex­tent, but Courier New has al­ways strug­gled with gen­eral read­abil­ity… and more im­por­tantly, with its poor dis­tinc­tion be­tween char­ac­ters.

While not as bad as some type­writer-era type­faces that ac­tu­ally reused the same sym­bol for 1 (one) and l (ell), Courier New came pretty close. Here is a com­par­i­son be­tween the two fonts when ren­der­ing these two char­ac­ters, only con­sid­er­ably en­larged:

The com­bi­na­tion of the two faults (the ane­mic weights and the even less dis­tinc­tion be­tween 1 and l as com­pared to Courier) makes Courier New a ter­ri­ble choice as a pro­gram­ming font. But as a font used for base64 out­put you want to OCR? You re­ally could­n’t pick a worse op­tion! To add fuel to the fire, you’re look­ing at SVG out­lines of the fonts, metic­u­lously con­verted and pre­serv­ing all the fine de­tails. But in the Epstein PDFs re­leased by the DoJ, we only have low-qual­ity JPEG scans at a fairly small point size. Here’s an ac­tual (losslessly en­coded) screen­shot of the DoJ text at 100% — I chal­lenge you to tell me which is a 1 and which is an l in the ex­cerpt be­low:

It’s not that there is­n’t any dif­fer­ence be­tween the two, be­cause there is. And some­times you get a clear gut feel­ing which is which — I was mid­way through man­u­ally typ­ing out one line of base64 text when I got stuck on iden­ti­fy­ing a one vs ell… only to re­al­ize that, at the same time, I had con­fi­dently tran­scribed one of them ear­lier that same line with­out even paus­ing to think about which it was. Here’s a zoomed-in view of the scanned PDF: you can clearly see all the JPEG DCT ar­ti­facts, the color fring­ing, and the smear­ing of char­ac­ter shapes, all of which make it hard to prop­erly iden­tify the char­ac­ters. But at the same time, at least in this par­tic­u­lar sam­ple, you can see which of the high­lighted char­ac­ters have a straight serif lead­ing out the top-left (the mid­dle, pre­sum­ably an ell) and which of those have the slight­est of strokes/​feet ex­tend­ing from them (the first and last, pre­sum­ably ones). But whether that’s be­cause that’s how the orig­i­nal glyph ap­peared or it’s be­cause of how the im­age was com­pressed, it’s tough to say:

But that’s get­ting ahead of my­self: at this point, none of the OCR tools had ac­tu­ally given me us­able re­sults, even ig­nor­ing the very im­por­tant ques­tion of l vs 1. After hav­ing been let down by one open source of­fer­ing (tesseract) and two com­mer­cial ones (Adobe Acrobat Pro and, pre­sum­ably, what­ever the DoJ used), I made the very ques­tion­able choice of writ­ing a script to use yet an­other com­mer­cial of­fer­ing, this time Amazon/AWS Textract, to process the PDF. Unfortunately, us­ing it di­rectly via the first-party tool­ing was (somewhat) of a no-go as it only sup­ports smaller/​shorter in­puts for di­rect use; longer PDFs like this one need to be up­loaded to S3 and then use the async work­flow to start the recog­ni­tion and poll for com­ple­tion.

Amazon Textract did pos­si­bly the best out of all the tools I tried, but its out­put still had ob­vi­ous line length dis­crep­an­cies — al­beit only one to two char­ac­ters or so off on av­er­age. I de­cided to try again, this time blow­ing up the in­put 2x (using near­est neigh­bor sam­pling to pre­serve sharp edges) as a workaround for Textract not hav­ing a tun­able I could set to con­fig­ure the DPI the doc­u­ment is processed at, though I wor­ried all in­puts could pos­si­bly be prescaled to a fixed size prior to pro­cess­ing once more:2

> for n in (printf “%02d\n” (seq 01 76))

con­vert EFTA00400459-$n.png -scale 200% \

EFTA00400459-$n”_2x”.png; or break

end

> par­al­lel -j 16 ./textract.sh {} ::: EFTA00400459-*_2x.png

These re­sults were no­tably bet­ter, and I’ve in­cluded them in an archive, but some of the pages scanned bet­ter than oth­ers. Textract does­n’t seem to be 100% de­ter­min­is­tic from my brief ex­pe­ri­ence with it, and their fea­tures page does make vague or un­clear men­tions to “ML”, though it’s not ob­vi­ous when and where it kicks in or what it ex­actly refers to, but that could ex­plain why a cou­ple of the pages (like EFTA00400459-62_2x.txt) are con­sid­er­ably worse than oth­ers, even while the source im­ages don’t show a good rea­son for that di­ver­gence.

With the Textract 2x out­put cleaned up and piped into base64 -i (which ig­nores garbage data, gen­er­at­ing in­valid re­sults that can still be us­able for foren­sic analy­sis), I can get far enough to see that the PDF within the PDF (i.e. the ac­tual PDF at­tach­ment orig­i­nally sent) was at least par­tially (de)flate-encoded. Unfortunately, PDFs are bi­nary files with dif­fer­ent forms of com­pres­sion ap­plied; you can’t just use some­thing like strings to ex­tract any us­able con­tent. qpdf(1) can be (ab)used to de­com­press a PDF (while leav­ing it a PDF) via qpdf –qdf –object-streams=disable in­put.pdf de­com­pressed.pdf, but, pre­dictably, this does­n’t work when your in­put is gar­bled and cor­rupted:

> qpdf –qdf –object-streams=disable re­cov­ered.pdf de­com­pressed.pdf

WARNING: re­cov­ered.pdf: file is dam­aged

WARNING: re­cov­ered.pdf: can’t find startxref

WARNING: re­cov­ered.pdf: Attempting to re­con­struct cross-ref­er­ence table

WARNING: re­cov­ered.pdf (object 34 0, off­set 52): un­known to­ken while read­ing ob­ject; treat­ing as string

WARNING: re­cov­ered.pdf (object 34 0, off­set 70): un­known to­ken while read­ing ob­ject; treat­ing as string

WARNING: re­cov­ered.pdf (object 34 0, off­set 85): un­known to­ken while read­ing ob­ject; treat­ing as string

WARNING: re­cov­ered.pdf (object 34 0, off­set 90): un­ex­pected >

WARNING: re­cov­ered.pdf (object 34 0, off­set 92): un­known to­ken while read­ing ob­ject; treat­ing as string

WARNING: re­cov­ered.pdf (object 34 0, off­set 116): un­known to­ken while read­ing ob­ject; treat­ing as string

WARNING: re­cov­ered.pdf (object 34 0, off­set 121): un­known to­ken while read­ing ob­ject; treat­ing as string

WARNING: re­cov­ered.pdf (object 34 0, off­set 121): too many er­rors; giv­ing up on read­ing ob­ject

WARNING: re­cov­ered.pdf (object 34 0, off­set 125): ex­pected en­dobj

WARNING: re­cov­ered.pdf (object 41 0, off­set 9562): ex­pected end­stream

WARNING: re­cov­ered.pdf (object 41 0, off­set 8010): at­tempt­ing to re­cover stream length

WARNING: re­cov­ered.pdf (object 41 0, off­set 8010): un­able to re­cover stream data; treat­ing stream as empty

WARNING: re­cov­ered.pdf (object 41 0, off­set 9616): ex­pected en­dobj

WARNING: re­cov­ered.pdf (object 41 0, off­set 9616): EOF af­ter en­dobj

qpdf: re­cov­ered.pdf: un­able to find trailer dic­tio­nary while re­cov­er­ing dam­aged file

Between the in­con­sis­tent OCR re­sults and the prob­lem with the l vs 1, it’s not a very en­cour­ag­ing sit­u­a­tion. To me, this is a prob­lem beg­ging for a (traditional, non-LLM) ML so­lu­tion, specif­i­cally lever­ag­ing the fact that we know the font in ques­tion and, roughly, the com­pres­sion ap­plied. Alas, I don’t have more time to lend to this chal­lenge at the mo­ment, as there are a num­ber of things I set aside just in or­der to pub­lish this ar­ti­cle.

So here’s the chal­lenge for any­one I can suc­cess­fully nerd­snipe:

* Can you man­age to recre­ate the orig­i­nal PDF from the Content-Transfer-Encoding: base64 out­put in­cluded in the dump? It can’t be that hard, can it?

* Can you find other at­tach­ments in­cluded in the lat­est Epstein dumps that might also be pos­si­ble to re­con­struct? Unfortunately, the con­trac­tor that de­vel­oped the full-text search for the Department of Justice did a pretty crappy job and full-text search is prac­ti­cally bro­ken even ac­count­ing for the bad OCR and wran­gled quoted-print­able de­cod­ing (malicious com­pli­ance??); nev­er­the­less, search­ing for Content-Transfer-Encoding and base64 re­turns a num­ber of re­sults — it’s just that, un­for­tu­nately, most are use­lessly trun­cated or only the SMTP head­ers from Apple Mail cu­ri­ously ex­tracted.

I have up­loaded the orig­i­nal EFTA00400459.pdf from Epstein Dataset 9 as down­loaded from the DoJ web­site to the Internet Archive, as well as the in­di­vid­ual pages loss­lessly en­coded to WebP im­ages to save you the time and trou­ble of con­vert­ing them your­self. If it’s of any use to any­one, I’ve also up­loaded the very-much-in­valid Amazon Textract OCR text (from the loss­lessly 2x’d im­ages), which you can down­load here.

Oh, and one fi­nal hint: when try­ing to fig­ure out 1 vs l, I was able to do this with 100% ac­cu­racy only via trial-and-er­ror, de­cod­ing one line of base64 text at-a-time, but this only works for the plain-text por­tions of the PDF (headers, etc). For ex­am­ple, I started with my best guess for one line that I had to type out my­self when try­ing with tesser­act, and then was able to (in this case) de­duce which par­tic­u­lar 1s or ls were flipped:

> pb­paste

SW5mbzw8L01sbHVzdHJhdG9yIDgxIDAgUj4+L1Jlc29lcmNlczw8L0NvbG9yU3BhY2U8PC9DUzAG

> pb­paste | base64 -d

Info<>/Resoerces<

> # which I was able to cor­rect:

> pb­paste

SW5mbzw8L0lsbHVzdHJhdG9yIDgxIDAgUj4+L1Jlc291cmNlczw8L0NvbG9yU3BhY2U8PC9DUzAG

> pb­paste | base64 -d

Info<>/Resources<

…but good luck get­ting that to work once you get to the flate-com­pressed sec­tions of the PDF.

I’ll be post­ing up­dates on Twitter @mqudsi, and you can reach out to me on Signal at mqudsi.42 if you have any­thing sen­si­tive you would like to share. You can join in the dis­cus­sion on Hacker News or on r/​net­sec. Leave a com­ment be­low if you have any ideas/​ques­tions, or if you think I missed some­thing!

Deck, Andrew. “A new bill in New York would re­quire dis­claimers on AI-generated news con­tent.” Nieman Journalism Lab. Nieman Foundation for Journalism at Harvard, 5 Feb. 2026. Web. 6 Feb. 2026.

Deck, A. (2026, Feb. 5). A new bill in New York would re­quire dis­claimers on AI-generated news con­tent. Nieman Journalism Lab. Retrieved February 6, 2026, from https://​www.nie­man­lab.org/​2026/​02/​a-new-bill-in-new-york-would-re­quire-dis­claimers-on-ai-gen­er­ated-news-con­tent/

Deck, Andrew. “A new bill in New York would re­quire dis­claimers on AI-generated news con­tent.” Nieman Journalism Lab. Last mod­i­fied February 5, 2026. Accessed February 6, 2026. https://​www.nie­man­lab.org/​2026/​02/​a-new-bill-in-new-york-would-re­quire-dis­claimers-on-ai-gen­er­ated-news-con­tent/.

{{cite web

| url = https://​www.nie­man­lab.org/​2026/​02/​a-new-bill-in-new-york-would-re­quire-dis­claimers-on-ai-gen­er­ated-news-con­tent/

| ti­tle = A new bill in New York would re­quire dis­claimers on AI-generated news con­tent

| last = Deck

| first = Andrew

| work = [[Nieman Journalism Lab]]

| date = 5 February 2026

| ac­cess­date = 6 February 2026

| ref = {{harvid|Deck|2026}}

Your web browser does not sup­port this video. The Waymo World Model: A New Frontier For Autonomous Driving SimulationThe Waymo Driver has trav­eled nearly 200 mil­lion fully au­tonomous miles, be­com­ing a vi­tal part of the ur­ban fab­ric in ma­jor U.S. cities and im­prov­ing road safety. What rid­ers and lo­cal com­mu­ni­ties don’t see is our Driver nav­i­gat­ing bil­lions of miles in vir­tual worlds, mas­ter­ing com­plex sce­nar­ios long be­fore it en­coun­ters them on pub­lic roads. Today, we are ex­cited to in­tro­duce the Waymo World Model, a fron­tier gen­er­a­tive model that sets a new bar for large-scale, hy­per-re­al­is­tic au­tonomous dri­ving sim­u­la­tion. Your web browser does not sup­port this video.Sim­u­la­tion of the Waymo Driver evad­ing a ve­hi­cle go­ing in the wrong di­rec­tion. The sim­u­la­tion ini­tially fol­lows a real event, and seam­lessly tran­si­tions to us­ing cam­era and li­dar im­ages au­to­mat­i­cally gen­er­ated by an ef­fi­cient real-time Waymo World Model.

Simulation is a crit­i­cal com­po­nent of Waymo’s AI ecosys­tem and one of the three key pil­lars of our ap­proach to demon­stra­bly safe AI. The Waymo World Model, which we de­tail be­low, is the com­po­nent that is re­spon­si­ble for gen­er­at­ing hy­per-re­al­is­tic sim­u­lated en­vi­ron­ments.The Waymo World Model is built upon Genie 3—Google DeepMind’s most ad­vanced gen­eral-pur­pose world model that gen­er­ates pho­to­re­al­is­tic and in­ter­ac­tive 3D en­vi­ron­ments—and is adapted for the rig­ors of the dri­ving do­main. By lever­ag­ing Genie’s im­mense world knowl­edge, it can sim­u­late ex­ceed­ingly rare events—from a tor­nado to a ca­sual en­counter with an ele­phant—that are al­most im­pos­si­ble to cap­ture at scale in re­al­ity. The mod­el’s ar­chi­tec­ture of­fers high con­trol­la­bil­ity, al­low­ing our en­gi­neers to mod­ify sim­u­la­tions with sim­ple lan­guage prompts, dri­ving in­puts, and scene lay­outs. Notably, the Waymo World Model gen­er­ates high-fi­delity, multi-sen­sor out­puts that in­clude both cam­era and li­dar data.This com­bi­na­tion of broad world knowl­edge, fine-grained con­trol­la­bil­ity, and multi-modal re­al­ism en­hances Waymo’s abil­ity to safely scale our ser­vice across more places and new dri­ving en­vi­ron­ments. In the fol­low­ing sec­tions we show­case the Waymo World Model in ac­tion, fea­tur­ing sim­u­la­tions of the Waymo Driver nav­i­gat­ing di­verse rare edge-case sce­nar­ios.Most sim­u­la­tion mod­els in the au­tonomous dri­ving in­dus­try are trained from scratch based on only the on-road data they col­lect. That ap­proach means the sys­tem only learns from lim­ited ex­pe­ri­ence. Genie 3’s strong world knowl­edge, gained from its pre-train­ing on an ex­tremely large and di­verse set of videos, al­lows us to ex­plore sit­u­a­tions that were never di­rectly ob­served by our fleet.Through our spe­cial­ized post-train­ing, we are trans­fer­ring that vast world knowl­edge from 2D video into 3D li­dar out­puts unique to Waymo’s hard­ware suite. While cam­eras ex­cel at de­pict­ing vi­sual de­tails, li­dar sen­sors pro­vide valu­able com­ple­men­tary sig­nals like pre­cise depth. The Waymo World Model can gen­er­ate vir­tu­ally any scene—from reg­u­lar, day-to-day dri­ving to rare, long-tail sce­nar­ios—across mul­ti­ple sen­sor modal­i­ties.Your web browser does not sup­port this video.Sim­u­la­tion: Driving on the Golden Gate Bridge, cov­ered in light snow. Waymo’s shadow is vis­i­ble in the front cam­era footage.

Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Sim­u­la­tion: Driving on a street with lots of palm trees in a trop­i­cal city, strangely cov­ered in snow.

Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Sim­u­la­tion: The lead­ing ve­hi­cle dri­ving into the tree branches.

Your web browser does not sup­port this video.Sim­u­la­tion: Driving be­hind a ve­hi­cle with pre­car­i­ously po­si­tioned fur­ni­ture on top.

Your web browser does not sup­port this video.Sim­u­la­tion: A mal­func­tioned truck fac­ing the wrong way, block­ing the road.

Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.In the in­ter­ac­tive view­ers be­low, you can im­mer­sively view the re­al­is­tic 4D point clouds gen­er­ated by the Waymo World Model.Interactive 3D vi­su­al­iza­tion of an en­counter with an ele­phant.

The Waymo World Model of­fers strong sim­u­la­tion con­trol­la­bil­ity through three main mech­a­nisms: dri­ving ac­tion con­trol, scene lay­out con­trol, and lan­guage con­trol.Dri­ving ac­tion con­trol al­lows us to have a re­spon­sive sim­u­la­tor that ad­heres to spe­cific dri­ving in­puts. This en­ables us to sim­u­late “what if” coun­ter­fac­tual events such as whether the Waymo Driver could have safely dri­ven more con­fi­dently in­stead of yield­ing in a par­tic­u­lar sit­u­a­tion.Coun­ter­fac­tual dri­ving. We demon­strate sim­u­la­tions both un­der the orig­i­nal route in a past recorded drive, or a com­pletely new route. While purely re­con­struc­tive sim­u­la­tion meth­ods (e.g., 3D Gaussian Splats, or 3DGS) suf­fer from vi­sual break­downs due to miss­ing ob­ser­va­tions when the sim­u­lated route is too dif­fer­ent from the orig­i­nal dri­ving, the fully learned Waymo World Model main­tains good re­al­ism and con­sis­tency thanks to its strong gen­er­a­tive ca­pa­bil­i­ties.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Scene lay­out con­trol al­lows for cus­tomiza­tion of the road lay­outs, traf­fic sig­nal states, and the be­hav­ior of other road users. This way, we can cre­ate cus­tom sce­nar­ios via se­lec­tive place­ment of other road users, or ap­ply­ing cus­tom mu­ta­tions to road lay­outs.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Lan­guage con­trol is our most flex­i­ble tool that al­lows us to ad­just time-of-day, weather con­di­tions, or even gen­er­ate an en­tirely syn­thetic scene (such as the long-tail sce­nar­ios shown pre­vi­ously).Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Dur­ing a scenic drive, it is com­mon to record videos of the jour­ney on mo­bile de­vices or dash­cams, per­haps cap­tur­ing piled up snow banks or a high­way at sun­set. The Waymo World Model can con­vert those kinds of videos, or any taken with a reg­u­lar cam­era, into a mul­ti­modal sim­u­la­tion—show­ing how the Waymo Driver would see that ex­act scene. This process en­ables the high­est de­gree of re­al­ism and fac­tu­al­ity, since sim­u­la­tions are de­rived from ac­tual footage.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Your web browser does not sup­port this video.Some scenes we want to sim­u­late may take longer to play out, for ex­am­ple, ne­go­ti­at­ing pas­sage in a nar­row lane. That’s harder to do be­cause the longer the sim­u­la­tion, the tougher it is to com­pute and main­tain sta­ble qual­ity. However, through a more ef­fi­cient vari­ant of the Waymo World Model, we can sim­u­late longer scenes with dra­matic re­duc­tion in com­pute while main­tain­ing high re­al­ism and fi­delity to en­able large-scale sim­u­la­tions.🚀  Long roll­out (4x speed play­back) on an ef­fi­cient vari­ant of the Waymo World ModelYour web browser does not sup­port this video.Nav­i­gat­ing around in-lane stop­per and fast traf­fic on the free­way.

Your web browser does not sup­port this video.Your web browser does not sup­port this video.Dri­ving up a steep street and safely nav­i­gat­ing around mo­tor­cy­clists.

Your web browser does not sup­port this video.By sim­u­lat­ing the “impossible”, we proac­tively pre­pare the Waymo Driver for some of the most rare and com­plex sce­nar­ios. This cre­ates a more rig­or­ous safety bench­mark, en­sur­ing the Waymo Driver can nav­i­gate long-tail chal­lenges long be­fore it en­coun­ters them in the real world.

The Waymo World Model is en­abled by the key re­search, en­gi­neer­ing and eval­u­a­tion con­tri­bu­tions from James Gunn, Kanaad Parvate, Lu Liu, Lucas Deecke, Luca Bergamini, Zehao Zhu, Raajay Viswanathan, Jiahao Wang, Sakshum Kulshrestha, Titas Anciukevičius, Luna Yue Huang, Yury Bychenkov, Yijing Bai, Yichen Shen, Stefanos Nikolaidis, Tiancheng Ge, Shih-Yang Su and Vincent Casser.We thank Chulong Chen, Mingxing Tan, Tom Walters, Harish Chandran, David Wong, Jieying Chen, Smitha Shyam, Vincent Vanhoucke and Drago Anguelov for their sup­port in defin­ing the vi­sion for this pro­ject, and for their strong lead­er­ship and guid­ance through­out.We would like to ad­di­tion­ally thank Jon Pedersen, Michael Dreibelbis, Larry Lansing, Sasho Gabrovski, Alan Kimball, Dave Richardson, Evan Birenbaum, Harrison McKenzie Chapter and Pratyush Chakraborty, Khoa Vo, Todd Hester, Yuliang Zou, Artur Filipowicz, Sophie Wang and Linn Bieske for their in­valu­able part­ner­ship in fa­cil­i­tat­ing and en­abling this pro­ject.We thank our part­ners from Google DeepMind: Jack Parker-Holder, Shlomi Fruchter, Philip Ball, Ruiqi Gao, Songyou Peng, Ben Poole, Fei Xia, Allan Zhou, Sean Kirmani, Christos Kaplanis, Matt McGill, Tim Salimans, Ruben Villegas, Xinchen Yan, Emma Wang, Woohyun Han, Shan Han, Rundi Wu, Shuang Li, Philipp Henzler, Yulia Rubanova, and Thomas Kipf for help­ful dis­cus­sions and for shar­ing in­valu­able in­sights for this pro­ject.

After be­ing in­ter­rupted mul­ti­ple times by an an­noy­ing con­sole win­dow that would pop up pe­ri­od­i­cally on my new gam­ing PC, I man­aged to track the of­fend­ing ex­e­cutable down to AMD’s AutoUpdate soft­ware.

In my anger, I de­cided to pun­ish this soft­ware by de­com­pil­ing it to fig­ure out how it worked, and ac­ci­den­tally dis­cov­ered a triv­ial Remote Code Execution (RCE) vul­ner­a­bil­ity in the process.

The first thing I found, is that they store their up­date URL in the pro­gram’s app.con­fig, al­though its a lit­tle odd that they use their “Develpment” URL in pro­duc­tion, it uses HTTPS so its per­fectly safe.

The real prob­lem starts when you open up this URL in your web browser, and re­alise that all of the ex­e­cutable down­load URL’s are us­ing HTTP.

This means that a ma­li­cious at­tacker on your net­work, or a na­tion state that has ac­cess to your ISP can eas­ily per­form a MITM at­tack and re­place the net­work re­sponse with any ma­li­cious ex­e­cutable of their choos­ing.

I was hop­ing that AMD per­haps had some form of cer­tifi­cate val­i­da­tion to en­sure that it could not down­load & run any un­signed ex­e­cuta­bles, how­ever a quick look into the de­com­piled code re­vealed that the AutoUpdate soft­ware does no such val­i­da­tion and im­me­di­ately ex­e­cutes the down­loaded file.

After find­ing this is­sue, I thought it was worth re­port­ing to AMD since it seemed to be a pretty se­vere is­sue.

However it turned out to be con­sid­ered “out of scope”, re­sult­ing in AMD not con­sid­er­ing this to be a vul­ner­a­bil­ity.

If you liked this blog, you can read an­other of my write-ups here: 1.4 Billion ex­posed user records via in­se­cure Firebase in­stances in top Android apps

I was an early em­ployee at CircleCI. I have used, in anger, nearly every CI sys­tem that has ever ex­isted. Jenkins, Travis, CircleCI, Semaphore, Drone, Concourse, Wercker (remember Wercker?), TeamCity, Bamboo, GitLab CI, CodeBuild, and prob­a­bly a half dozen oth­ers I’ve mer­ci­fully for­got­ten. I have mass-tested these sys­tems so that you don’t have to, and I have the scars to show for it, and I am here to tell you: GitHub Actions is not good. It’s not even fine. It has mar­ket share be­cause it’s right there in your repo, and that’s about the nicest thing I can say about it.

Buildkite is what CI should feel like. But first, let me tell you what CI should not feel like.

Before I get into it: if you’re a Nix shop, take a look at Garnix. It eval­u­ates your flake, fig­ures out what needs build­ing, and builds it. No YAML. No pipeline con­fig­u­ra­tion. It just looks at your flake.nix and does the right thing. Sometimes the best CI con­fig­u­ra­tion is no CI con­fig­u­ra­tion.

Most shops are not Nix shops. This post is for the rest of you. I’m sorry.

Let me start with the most vis­ceral thing, the thing that will sound like I’m ex­ag­ger­at­ing but I am not.

Your build fails. You get a red X on your pull re­quest. You click through to see what hap­pened. This is where the or­deal be­gins.

First you land on the checks sum­mary page, which shows you a list of work­flow runs. Maybe one failed. Maybe three failed. You click the one that looks rel­e­vant. Now you’re on the work­flow run page, which shows you a list of jobs. You click the failed job. Now you’re on the job page, which shows you a list of steps, all col­lapsed. You click the step that failed. The page hitches. You scroll. There is a pause, a held breath, and then the logs ap­pear, slowly, like a man­u­script be­ing re­vealed one line at a time to a sup­pli­cant who has not yet proven wor­thy.

That’s three or four clicks just to see the er­ror, and every one of them loads a new page with its own load­ing spin­ner, and none of them are fast. You are nav­i­gat­ing a bu­reau­cracy. You are fill­ing out forms at the DMV of CI.

And then the log viewer it­self. I have used every CI sys­tem known to man, and the GitHub Actions log viewer is the only one that has crashed my browser. Not once. Repeatedly. Reliably. Open a long build log, try to search for an er­ror, and Chrome will look you in the eye and die. This is the log viewer for the most pop­u­lar CI sys­tem in the world. This is the tool you are ex­pected to use to un­der­stand why your build failed. It can­not sur­vive con­tact with its own out­put.

With large logs (and if you have a real build sys­tem, you have large logs) you of­ten can’t even scroll to the bot­tom. The page just gives up. The scroll­bar is there, tech­ni­cally, but it’s dec­o­ra­tive. It’s a sug­ges­tion. You drag it down and the page chokes and stut­ters and even­tu­ally you re­al­ize you’re not go­ing to get there this way. So you down­load the raw log ar­ti­fact and open it in a text ed­i­tor like it’s 2003 and you’re read­ing Apache ac­cess logs on a shared host­ing box. Except it’s 2025 and this is a prod­uct made by one of the rich­est com­pa­nies on earth.

And when you’re done, when you’ve fi­nally found the er­ror, processed your grief, and want to go back to the pull re­quest that started this whole or­deal, you hit the back but­ton. Does it take you to the PR? No. It takes you to some other page in the GitHub Actions UI. A sum­mary page, maybe. Or a dif­fer­ent run. Or a page you don’t rec­og­nize. The back but­ton in the GitHub Actions UI is a roulette wheel. You will land some­where. It will not be where you wanted to go. You will click the back but­ton again. You will land some­where else. Eventually you give up and type the PR URL from mem­ory or go find it in your browser his­tory, which is now 80% GitHub Actions URLs, a fact that will haunt you when you look at it later.

So the logs have be­trayed you, or per­haps they sim­ply could not be made to ap­pear at all. Now be­gins the sec­ond rit­ual: the de­bug­ging.

You push a com­mit. You wait. A run­ner picks it up. You watch logs scroll. Something fails. The er­ror looks like some­one fed a stack trace through a pa­per shred­der and then set the shred­der on fire. You add a run: env step to see what’s go­ing on. You push again. You wait again. A twenty-minute feed­back loop for a one-line change. You do this four­teen times. This is your af­ter­noon now. You had plans. You were go­ing to go out­side. The af­ter­noon be­longs to the CI now. It has al­ways be­longed to the CI. You are only now per­ceiv­ing this truth.

There is some­thing de­vo­tional about the ex­pe­ri­ence. You ap­proach the logs. You make your of­fer­ing of clicks and pa­tience. The page con­sid­ers your re­quest. Sometimes it grants you the knowl­edge you seek. Sometimes it takes your browser tab in­stead, a small sac­ri­fice, con­sumed, gone. You open a new tab. You try again. This is the rit­ual. You did not choose it. The work con­tin­ues.

Every CI sys­tem even­tu­ally be­comes “a bunch of YAML.” I’ve been through the five stages of grief about it and emerged on the other side, di­min­ished but func­tional. But GitHub Actions YAML is a spe­cial breed. It’s YAML with its own ex­pres­sion lan­guage bolted on, its own con­text ob­ject model, its own string in­ter­po­la­tion rules, and a scat­ter­ing of gotchas that will slowly hol­low you out as a per­son. Each gotcha leaves a mark. The marks do not fade.

Have you ever tried to con­di­tion­ally set an en­vi­ron­ment vari­able based on which branch you’re on? Have you done the ${{ }} ex­pres­sion dance, mis­quoted some­thing, and then waited four min­utes for a run­ner to spin up just to dis­cover your string got eaten? Of course you have. We all have. We have all stared at a diff that changes one char­ac­ter in a YAML ex­pres­sion and thought “I went to col­lege for this.”

The ex­pres­sion syn­tax has the qual­ity of a lan­guage that grew in the dark, un­su­per­vised. It be­gan as a con­ve­nience. It ac­creted fea­tures. Somewhere along the way it crossed a thresh­old, and now it ex­ists in a lim­i­nal space- too com­plex to be con­fig­u­ra­tion, too con­strained to be a proper lan­guage. You learn its gram­mar not from doc­u­men­ta­tion but from fail­ure, each er­ror mes­sage a koan that points to­ward un­der­stand­ing but does not pro­vide it.

Ah yes, the GitHub Actions Marketplace. The npm of CI. A bazaar of com­mu­nity-main­tained ac­tions of vary­ing qual­ity, most of which are shell scripts with a Dockerfile and a dream.

Every time you type uses: some-stranger/​cool-ac­tion@v2, you’re hand­ing a stranger ac­cess to your repo, your se­crets, and your build en­vi­ron­ment. Yes, you can pin to a SHA. Nobody does. And even if you do, you’re still run­ning opaque code you did­n’t write and prob­a­bly haven’t read, in a con­text where it has ac­cess to your GITHUB_TOKEN and what­ever else you’ve stuffed in there. Every ac­tion you add is an­other set of house keys you’re hand­ing to some­one you’ve never met and hop­ing for the best.

The Marketplace has the en­ergy of a night mar­ket in a city you don’t know, where every stall sells some­thing that claims to solve your prob­lem. Some of them do. Some of them have other in­ten­tions. You can­not tell which is which from the out­side. You can only in­stall them and see what hap­pens. This is called “dependency man­age­ment.” We have nor­mal­ized it. The nor­mal­iza­tion does not make it safe.

With GitHub Actions, you’re rent­ing Microsoft’s run­ners. They’re slow, they’re re­source-con­strained, you can’t cus­tomize them in any mean­ing­ful way, and you’re at the mercy of GitHub’s ca­pac­ity plan­ning. Need a beefy ma­chine for a big build? You can pay for a larger run­ner, at prices that will get you a cal­en­dar in­vite from fi­nance ti­tled “we need to talk,” and you still don’t con­trol the en­vi­ron­ment.

You know how I know GitHub’s run­ners are bad? Because there’s an en­tire cot­tage in­dus­try of com­pa­nies whose sole prod­uct is “GitHub Actions, but the run­ners don’t suck.” Namespace, Blacksmith, Actuated, Runs-on, BuildJet. There are at least half a dozen star­tups that ex­ist purely to solve the prob­lem of GitHub Actions be­ing slow. Their pitch is, es­sen­tially, “keep your work­flows, we’ll just make them not take for­ever.” The fact that this is a vi­able busi­ness model, that mul­ti­ple com­pa­nies can sus­tain them­selves on the premise that the de­fault com­pute for the world’s most pop­u­lar CI sys­tem is in­ad­e­quate, tells you every­thing you need to know.

Now, to be fair, you can bring your own run­ners to GitHub Actions. Self-hosted run­ners ex­ist. You can set up your own ma­chines, in­stall Nix, con­fig­ure your en­vi­ron­ment ex­actly how you want it. And this does solve the com­pute prob­lem. Your builds will be faster. Your caches will be warm. But you’ll still be writ­ing GitHub Actions YAML. You’ll still be fight­ing the ex­pres­sion syn­tax and the per­mis­sions model and the mar­ket­place and the log viewer that crashes your browser. You’ve up­graded the en­gine but you’re still dri­ving the car that catches fire when you turn on the ra­dio.

I think the peo­ple who orig­i­nally built GitHub Actions were prob­a­bly well-in­ten­tioned. They prob­a­bly cared about de­vel­oper ex­pe­ri­ence. But this is a Microsoft prod­uct now, and Microsoft is where am­bi­tious de­vel­oper tools go to be­come en­ter­prise SKUs. The orig­i­nal en­gi­neers have long since been re­orged into other di­vi­sions or ground down into prod­uct man­agers. The vi­sion, if there was one, is en­tombed now. But if you press your ear to the floor dur­ing a par­tic­u­larly slow build, you can still hear its heart beat­ing, faintly, be­neath the floor­boards.

Things that seem small but ac­cu­mu­late. Each one is sur­viv­able. Together they form a com­pelling case for sim­ply walk­ing into the sea. The sea does not have YAML. The sea does not re­quire a GITHUB_TOKEN.

The ac­tions/​cache ac­tion is an ex­er­cise in fu­til­ity. Cache keys are con­fus­ing, cache misses are silent, and cache evic­tion is opaque. You will spend more time de­bug­ging caching than you save by hav­ing a cache.

Reusable work­flows can’t be nested be­yond a cer­tain depth, can’t ac­cess the call­ing work­flow’s con­text cleanly, and live in YAML files that are im­pos­si­ble to test in iso­la­tion. At some point you re­al­ize you’re writ­ing a dis­trib­uted sys­tem in YAML and you have to sit down and think about the choices that led you here. The think­ing changes you. You do not get the old ver­sion of your­self back. That per­son did­n’t know what a work­flow_­call trig­ger was. That per­son was happy.

The GITHUB_TOKEN per­mis­sions model is a maze. per­mis­sions: write-all is a ham­mer, fine-grained per­mis­sions are a puz­zle, and the in­ter­ac­tion be­tween repos­i­tory set­tings, work­flow set­tings, and job-level set­tings will make you want to lie down on the floor. I once spent an en­tire day on to­ken per­mis­sions. I will never get that day back. It’s gone. I could have learned to paint. I could have called my mother. I could have mass-tested a new CI sys­tem. Anything.

Concurrency con­trols are blunt. Cancel in-progress runs on the same branch? Sure, one line. Anything more nu­anced? No. The sys­tem does not wish to dis­cuss nu­ance. The sys­tem has other con­cerns.

Secrets can’t be used in if con­di­tions. This means you can’t do things like if: se­crets. DEPLOY_KEY != ‘’ to con­di­tion­ally run a step based on whether a se­cret is con­fig­ured. GitHub does­n’t want se­cret val­ues leak­ing into logs via ex­pres­sion eval­u­a­tion, which is a rea­son­able se­cu­rity con­cern. But the prac­ti­cal re­sult is that you can’t write work­flows that grace­fully de­grade when op­tional se­crets aren’t pre­sent. Instead you need awk­ward workarounds like set­ting a non-se­cret en­vi­ron­ment vari­able that flags whether the real se­cret ex­ists. It’s one of those de­ci­sions that makes per­fect sense in a se­cu­rity re­view and makes you want to scream when you’re ac­tu­ally try­ing to write a work­flow that works in both forks and the main repo.

At some point in every en­gi­neer’s CI jour­ney, a temp­ta­tion pre­sents it­self.

“What if I just wrote bash scripts?” the voice whis­pers. “What if I stopped fight­ing the CI sys­tem and just run:’d a big shell script that does every­thing? I could run it lo­cally. I could test it. I’d be free.”

I un­der­stand the ap­peal. I have felt it my­self, late at night, af­ter the fourth failed work­flow run in a row. The de­sire to burn down the YAML tem­ple and re­turn to the sim­ple hon­est earth of #!/bin/bash and set -euo pipefail. To cast off the chains of mar­ket­place ac­tions and reusable work­flows and just write the damn com­mands. It feels like lib­er­a­tion. It is not.

Here’s what ac­tu­ally hap­pens. Your bash script works. You feel clever. You tell your cowork­ers about it. Then the script grows. It ac­quires con­di­tion­als. It ac­quires func­tions. It ac­quires ar­gu­ment pars­ing. It ac­quires a sec­ond script that it sources. Someone adds er­ror han­dling. Someone else adds log­ging. Someone (and this per­son should be stopped, but never is) adds “just a lit­tle bit of par­al­lelism.”

Three months later you have 800 lines of bash that reim­ple­ments job par­al­lelism with wait and PID files, has its own retry logic built on a for loop and sleep, and parses its own out­put to de­ter­mine suc­cess or fail­ure. The script has be­come self-aware. There’s a race con­di­tion in the cleanup trap that only man­i­fests on Linux ker­nel 6.x, and you are the only per­son who un­der­stands the script, and you are on va­ca­tion, and your phone is ring­ing.

You have not es­caped CI. You have built a CI sys­tem. It’s just worse than every other CI sys­tem, be­cause it’s writ­ten in bash, and no­body can fol­low it, and it has no test frame­work, and shellcheck is scream­ing into the void, and the void is also writ­ten in bash.

Bash is fine for glue. Bash is fine for “run these four com­mands in or­der.” Bash is not a build sys­tem. Bash is not a test har­ness. The fact that it can be co­erced into im­per­son­at­ing both is not a rec­om­men­da­tion. It’s a warn­ing sign. You are not sim­pli­fy­ing. You are mov­ing com­plex­ity from a place with guardrails to a place with none. The com­plex­ity will not be grate­ful for its new free­dom. The com­plex­ity will use its free­dom to make your life worse.

There are other ways to live. Buildkite is what CI should feel like. Not joy­ful, noth­ing about CI is joy­ful, but tol­er­a­ble. A tool that un­der­stands its pur­pose and does not fight you. Let me tell you how the other half lives.

Buildkite’s log viewer is just a web page that shows you logs and does­n’t crash. I re­al­ize that’s a low bar. It’s a bar that GitHub Actions trips over and falls face-first into the mud, gets up, slips again, and some­how sets the mud on fire.

The ter­mi­nal out­put ren­der­ing is ex­cel­lent. Build logs look like ter­mi­nal out­put, be­cause they are ter­mi­nal out­put. ANSI col­ors work. Your test frame­work’s fancy for­mat­ting comes through in­tact. You’re not squint­ing at a web UI that has eaten your es­cape codes and ren­dered them as mo­jibake. This sounds mi­nor. It is not mi­nor. You are read­ing build logs dozens of times a day. The ex­pe­ri­ence of read­ing them mat­ters in the way that a com­fort­able chair mat­ters. You only no­tice how much it mat­ters af­ter you’ve been sit­ting in a bad one for six hours and your back has filed a for­mal com­plaint.

Annotations let your build steps write rich Markdown out­put (test fail­ure sum­maries, cov­er­age re­ports, de­ploy links,) right into the build page. You don’t have to dig through log out­put to find the thing you care about. The in­for­ma­tion comes to you. After years of fight­ing GitHub Actions’ col­lapsi­ble log groups and won­der­ing which of the sev­en­teen nested folds con­tains the ac­tual er­ror mes­sage, this feels like step­ping out of a cave into sun­light.

And de­bug­ging? Buildkite does­n’t make CI de­bug­ging fun. Nothing does. Nothing can. It is one of the ir­re­ducible suf­fer­ings of our craft. But be­cause the agent runs on your in­fra­struc­ture, you can SSH into the box. You can look at what’s ac­tu­ally hap­pen­ing. You can re­pro­duce the en­vi­ron­ment lo­cally be­cause you built the en­vi­ron­ment. You are still mor­tal, but at least you have tools.

Buildkite has YAML too, but the dif­fer­ence is that Buildkite’s YAML is just de­scrib­ing a pipeline. Steps, com­mands, plu­g­ins. It’s a data struc­ture, not a pro­gram­ming lan­guage cos­play­ing as a con­fig for­mat. When you need ac­tual logic? You write a script. In a real lan­guage. That you can run lo­cally. Like a hu­man be­ing with dig­nity and a will to live.

This is the bound­ary the bash zealots were ac­tu­ally look­ing for: not “put every­thing in bash,” but “put the or­ches­tra­tion in con­fig and the logic in code.” Buildkite re­spects this bound­ary. GitHub Actions blurs it un­til you can no longer tell where con­fig­u­ra­tion ends and pro­gram­ming be­gins, and then the pro­gram­ming hap­pens in a lan­guage that can’t do ba­sic arith­metic with­out ${{ }} and a prayer.

With Buildkite, the agent is a sin­gle bi­nary that runs on your ma­chines. Your cloud, your on-prem boxes, your weird cus­tom hard­ware. You con­trol the in­stance types, the caching, the lo­cal stor­age, the net­work. Run agents on a fleet of mas­sive EC2 in­stances with NVMe dri­ves and 20 gigs of Docker layer cache. Run them on a Raspberry Pi. It does­n’t care. The fastest CI is the one with a warm cache on a big ma­chine that you con­trol.

You don’t see a cot­tage in­dus­try of “Buildkite, but faster.” You don’t need one. You just run big­ger ma­chines.

GitHub Actions will never give you this. GitHub Actions will give you a mass-pro­duced Ubuntu VM with the emo­tional warmth of a hos­pi­tal wait­ing room.

I can hear the ob­jec­tion form­ing: “But I don’t want to run my own CI in­fra­struc­ture. I just want to push code and have tests run.”

Fair. Running your own agents is not for every­one. If you’re main­tain­ing a small open source li­brary in your spare time, you should­n’t have to spin up EC2 in­stances and man­age au­toscal­ing groups. GitHub Actions’ free tier for pub­lic re­pos is gen­uinely valu­able for the OSS ecosys­tem, and I’m not here to tell a solo main­tainer they need to set up Terraform to run their unit tests.

This post is mostly aimed at teams run­ning pro­duc­tion sys­tems at busi­nesses, places where you’re al­ready man­ag­ing in­fra­struc­ture, where CI time is mea­sured in en­gi­neer­ing hours lost per week, where the build that takes 45 min­utes is cost­ing you real money in both com­pute and salary. If that’s you, the over­head of run­ning Buildkite agents pays for it­self quickly. If you’re pub­lish­ing a 200-line npm pack­age on your week­ends, the cal­cu­lus is dif­fer­ent and that’s fine.

In Buildkite, pipeline steps are just data. You can gen­er­ate them.

Your pipeline YAML can con­tain a step that runs a script, and that script can emit more pipeline steps. Dynamically. At run­time. Based on what­ever logic you want: which files changed, what day of the week it is, the phase of the moon, whether the build gods are feel­ing mer­ci­ful.

So you write a script that looks at your monorepo, fig­ures out what changed, and up­loads ex­actly the right set of steps for the things that need build­ing and test­ing. No hard­coded ma­trix. No if: con­tains(github.event.pul­l_re­quest…) spaghetti. Just a pro­gram that out­puts steps.

GitHub Actions has ma­trix strate­gies and if con­di­tions and reusable work­flows and all sorts of mech­a­nisms that try to ap­prox­i­mate this. They’re all worse. They’re de­clar­a­tive in the worst sense: you’re de­clar­ing things in a lan­guage that is­n’t pow­er­ful enough to ex­press what you mean, so you end up build­ing a Rube Goldberg ma­chine out of YAML and re­gret. The ma­chine grows. You feed it. It does not thank you.

I’ll be hon­est: Buildkite’s plu­gin sys­tem is struc­turally pretty sim­i­lar to the GitHub Actions Marketplace. You’re still pulling in third-party code from a repo. You’re still trust­ing some­one else’s work. I won’t pre­tend there’s some magic ar­chi­tec­tural dif­fer­ence that makes this safe.

The real dif­fer­ence is nar­rower than I’d like: Buildkite plu­g­ins tend to be thin shell hooks rather than en­tire Docker im­ages with their own run­time, so there’s less sur­face area to hide things in, and you can usu­ally read the whole plu­gin in a few min­utes. More im­por­tantly, they run on your in­fra­struc­ture, so at least the blast ra­dius is some­thing you con­trol. It’s not a solved prob­lem. It’s a smaller prob­lem.

Buildkite has cus­tom emoji. You can put a lit­tle :parrot: or :docker: or your team’s cus­tom emoji next to your pipeline steps. This is, ob­jec­tively, a friv­o­lous fea­ture. It is also one of my fa­vorite things about Buildkite, be­cause it tells you some­thing about the peo­ple who built it. They thought about what it feels like to use their prod­uct. They knew that CI is a slog and that a small dumb thing like a cus­tom emoji next to your de­ploy step can make the slog a frac­tion more bear­able.

GitHub Actions would never do this. GitHub Actions is a prod­uct de­signed by a com­mit­tee that has never once asked “but is it de­light­ful?” and it shows.

Buildkite is built by peo­ple who clearly use CI every day and have thought hard about what makes it tol­er­a­ble. The re­sult is a tool that, if not ex­actly joy­ful, at least does not make you want to lie down on the floor. In this in­dus­try, that’s high praise.

Yeah. GitHub Actions won by be­ing the de­fault, not by be­ing good. It’s free for pub­lic re­pos, it’s built into the plat­form every­one al­ready uses, and it’s Good Enough. It’s the Internet Explorer of CI. It ships with the thing. People use it be­cause switch­ing costs are real and life is fi­nite and we’re all just try­ing to ship code and go home.

If you’re a small team with a sim­ple app and straight­for­ward tests, it’s prob­a­bly fine. I’m not go­ing to tell you to rip it out.

But if you’re run­ning a real pro­duc­tion sys­tem, if you have a monorepo, if your builds take more than five min­utes, if you care about sup­ply chain se­cu­rity, if you want to ac­tu­ally own your CI: look at Buildkite.

I’ve been do­ing this for a long time. I’ve watched CI sys­tems come and go. I’ve helped build one. The pat­tern is al­ways the same: the CI sys­tem that wins mar­ket share is never the one that’s best at be­ing a CI sys­tem. It’s the one that’s eas­i­est to start us­ing.

GitHub Actions is the eas­i­est CI to start us­ing. Buildkite is the best CI to keep us­ing. And in the long run (assuming your CI has­n’t al­ready ground you into a fine paste, as­sum­ing the YAML has­n’t hol­lowed you out en­tirely, as­sum­ing there’s still some­one left who re­mem­bers what it was like be­fore,) that’s what mat­ters.

If your CI works and you’re happy, great, keep go­ing. But if you’ve got that nag­ging feel­ing that it’s fight­ing you more than help­ing you: you’re not the prob­lem. The tool­ing is. There are other ways to live.

Software is a sta­tic list of in­struc­tions, which we are con­stantly chang­ing.