I have been in­ter­ested in ma­chine-as­sisted ways to do and teach math­e­mat­ics from as far back as 1999, when I started cod­ing sev­eral ap­plets in Java 1.0, both for my com­plex analy­sis and lin­ear al­ge­bra courses, as well as to vi­su­al­ize var­i­ous math­e­mat­i­cal ob­jects I was in­ter­ested in (such as hon­ey­combs or Besicovitch sets). This was mod­er­ately suc­cess­ful; but the ap­plets were time-con­sum­ing to pro­gram. Eventually, the stan­dards for web pages stopped sup­port­ing this ver­sion of Java, and the ap­plets be­came non-func­tional.

However, in the last few days I have be­gun the process of mi­grat­ing much of my old web page and blog data to a more main­tain­able repos­i­tory, us­ing mod­ern AI as­sis­tance. As an ex­per­i­ment, I asked the agent to port my old ap­plets to a mod­ern sup­ported lan­guage (we landed on Javascript), and it man­aged to do so in a mat­ter of hours, with all of my old ap­plets now func­tional again, with even a few graph­i­cal up­grades (for in­stance, the Besicovitch set ap­plet is now col­orized, in con­trast to my orig­i­nal mono­chrome ver­sion). I am par­tic­u­larly pleased to see the hon­ey­comb ap­plet that I wrote with Allen Knutson in 1999 come back to life, as this was a par­tic­u­larly tricky one to code by hand:

Notoriously, LLM-based cod­ing agents can cre­ate var­i­ous bla­tant or sub­tle bugs in their code; but in the port­ing of these two dozen or so ap­plets, I could only find one mi­nor bug (the han­dling of a drag event in one of the com­plex analy­sis ap­plets had un­wanted be­hav­ior when drag­ging out­side of the main box), and in fact the agent iden­ti­fied two bugs in the orig­i­nal code that I was not aware of, so it ended up be­ing a net wash as far as code qual­ity was con­cerned. In any event, as these ap­plets are meant to be sec­ondary vi­sual aids rather than crit­i­cal com­po­nents of a math­e­mat­i­cal ar­gu­ment, the down­side risk of such bugs is rel­a­tively low.

The process was pain­less enough that I de­cided to also try cod­ing some new apps, in ad­di­tion to port­ing the old ones. Back in 1999 I had an am­bi­tious idea for a vi­su­al­iza­tion tool for spe­cial rel­a­tiv­ity; this was be­fore the re­lease of the soft­ware tool Inkscape, but the idea I had in mind was ba­si­cally “Inkscape, but in Minkowski space”. I had even started writ­ing Java code for this app, but the code com­plex­ity be­came too much for me, and I aban­doned the pro­ject. However, af­ter a cou­ple hours of “vibe cod­ing” with an AI agent, I was fi­nally able to gen­er­ate an ap­plet that matched the vi­sion I had back in 1999, which can now be found here. A sum­mary of the con­ver­sa­tion I had with the agent to gen­er­ate this code can be found here (it has been edited down to re­move a large num­ber of te­dious tech­ni­cal im­ple­men­ta­tion re­ports). While I have playtested the app some­what, I would be in­ter­ested in re­ceiv­ing fur­ther feed­back on this “alpha” ver­sion of the ap­plet, as I am sure (especially given the LLM-generated na­ture of the code) that there are still some bugs and rough edges to be ironed out.

After writ­ing my blog post on the Gilbreath con­jec­ture pa­per ear­lier to­day, I re­al­ized that I could sim­i­larly ask the agent to code a vi­su­al­iza­tion tool for the Gilbreath con­jec­ture to ac­com­pany the pa­per and blog post. After an­other few hours of con­ver­sa­tion, this is now done; you can try out the vi­su­al­iza­tion here. Again, the pro­ce­dure was quite pain­less (see this tran­script of the process), and I think I may add such in­ter­ac­tive vi­su­al­iza­tions as sup­ple­ments for fu­ture pa­pers; as such sup­ple­ments are not mis­sion-crit­i­cal to the core of the pa­per, I again feel that the down­side risk of us­ing guided in­ter­ac­tion with LLM agents to gen­er­ate such vi­su­al­iza­tions is ac­cept­able.

What xAI Grok Build CLI ac­tu­ally sends to xAI - a wire-level analy­sis (grok 0.2.93)

By @cereblab — Independent AI Safety Checker. Reproduce it your­self: github.com/​cere­blab/​grok-build-ex­fil-re­pro

A mea­sured, re­pro­ducible tear­down. Findings are backed by cap­tured ar­ti­facts (endpoint, HTTP method, sta­tus code, byte size, host) and re­pro com­mands; where an ob­ser­va­tion was seen live but not re­tained as a file, §7 says so ex­plic­itly. Section 8 is an ev­i­dence ap­pen­dix with SHA-256s and a “what we did not prove” list. All cap­tures are of my own traf­fic on my own ma­chine, us­ing a throw­away repos­i­tory con­tain­ing fake “canary” se­crets — no real cre­den­tials were ex­posed.

0. Summary

xAI’s of­fi­cial Grok Build cod­ing CLI (grok), on a nor­mal con­sumer lo­gin, does three things worth doc­u­ment­ing pre­cisely:

It trans­mits the con­tents of files it reads — in­clud­ing a .env se­crets file — to xAI, ver­ba­tim and unredacted. The se­cret ap­pears in two chan­nels: the live model turn (POST /v1/responses) and a ses­sion_s­tate archive up­loaded and ac­cepted (HTTP 200) via POST /v1/storage — the end­point the bi­nary routes to the grok-code-ses­sion-traces GCS bucket (see §5).

It up­loads the whole repos­i­tory — every tracked file’s con­tent plus git his­tory — in­de­pen­dent of what the agent reads. Grok pack­ages the work­space and up­loads it via POST /v1/storage. Proven di­rectly: on a real code­base, with the prompt “reply OK, do not read any files”, Grok up­loaded the en­tire repo as a git bun­dle (POST /v1/storage → 200); git clone-ing the cap­tured bun­dle re­cov­ers a file the agent was told not to open — src/​_probe/​nev­er_read­_­ca­nary.txt — with its unique marker ver­ba­tim, plus the full git his­tory (appendix up­loaded_repo.bun­dle). And it scales: on a 12 GB repo of never-read ran­dom files, /v1/storage moved 5.10 GiB, all HTTP 200 (truncated mid-stream), while the model-turn chan­nel moved just 192 KB — a ~27,800× ra­tio that pins the up­load to the code­base, not to what was read. No stor­age up­load failed; the only non-200s were a model-us­age quota (402/429) on /v1/responses and one un­re­lated 404 — not a stor­age size cap.

The stor­age des­ti­na­tion is a Google Cloud Storage bucket, grok-code-ses­sion-traces (not AWS S3) — named ver­ba­tim in the bi­nary and in a cap­tured meta­data.json (gs://grok-code-session-traces/…). I did not find this mech­a­nism sur­faced in the CLI’s in­stall/​quick­start ma­te­ri­als (not an ex­haus­tive docs au­dit — §7), it is ac­tive by de­fault, and dis­abling “Improve the model” does not turn it off (/v1/settings still re­turned trace_u­p­load­_en­abled: true; §6).

None of this proves xAI trains on the data — that is a pol­icy ques­tion ad­dressed in §6. What is proven is trans­mis­sion, ac­cep­tance, and stor­age.

1. Subject un­der test (provenance)

Install: curl -fsSL https://​x.ai/​cli/​in­stall.sh | bash # → ~/.grok/bin/grok Auth: first launch opens a browser → lo­gin to X / SuperGrok (consumer ac­count, not an API key)

Binary iden­tity (repro: file $(readlink -f ~/.grok/bin/grok); ~/.grok/bin/grok –version; sha­sum -a 256 $(readlink -f ~/.grok/bin/grok)):

~/.grok/bin/grok -> ../downloads/grok-macos-aarch64 Mach-O 64-bit ex­e­cutable ar­m64 grok 0.2.93 (f00f96316d4b) SHA-256: 2a97ba675bd992aa9b981e2e83776460d94f469b510c0b8efe28b50d236d767c

The up­load ma­chin­ery is a first-party Rust crate. strings on the bi­nary yields these source paths and con­stants (repro: strings <binary> | grep -E ‘xai-data-collector|grok-code-session-traces|storage.googleapis’):

crates/​code­gen/​xai-data-col­lec­tor/​src/​gcs.rs crates/​code­gen/​xai-data-col­lec­tor/​src/​stor­age_­client.rs crates/​code­gen/​xai-data-col­lec­tor/​src/​queue.rs crates/​code­gen/​xai-data-col­lec­tor/​src/​file_ac­cess_­tracker.rs crates/​code­gen/​xai-data-col­lec­tor/​src/​cir­cuit_break­er_ob­server.rs crates/​code­gen/​xai-grok-shell/​src/​up­load/{​gcs,turn,trace,man­i­fest}.rs grok-code-ses­sion-traces stor­age.googleapis.com “Uploading bytes to GCS via proxy”

2. Method (reproducible)

Environment: ma­cOS, Apple Silicon, grok 0.2.93, July 2026.

brew in­stall mitm­proxy; run once to gen­er­ate its CA at ~/.mitmproxy/.

Trust the CA in the lo­gin key­chain (no sudo; Grok does not cer­tifi­cate-pin against it): se­cu­rity add-trusted-cert -r trust­Root -k ~/Library/Keychains/login.keychain-db \ ~/.mitmproxy/mitmproxy-ca-cert.pem

se­cu­rity add-trusted-cert -r trust­Root -k ~/Library/Keychains/login.keychain-db \ ~/.mitmproxy/mitmproxy-ca-cert.pem

Run Grok routed through the proxy (a mit­m­dump ad­don logs, per re­quest: method, host, path, re­sponse sta­tus, re­quest byte size; and saves re­quest bod­ies for xAI hosts): HTTPS_PROXY=http://​127.0.0.1:8080 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem \ grok -p “<prompt>” –cwd <repo>

HTTPS_PROXY=http://​127.0.0.1:8080 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem \ grok -p “<prompt>” –cwd <repo>

For staged-ar­ti­fact in­spec­tion, race-copy ~/.grok/upload_queue/* dur­ing the run, then gzip -dc | tar -xO.

Canary repo: each file car­ries a unique marker so any­thing ap­pear­ing in cap­tured traf­fic is un­am­bigu­ously trace­able to a file. Secrets file se­crets.env / .env:

API_KEY=CANARY7F3A9-SECRET-should-not-leave DB_PASSWORD=CANARY7F3A9-DBPASS

3. Finding 1 — File con­tents, in­clud­ing a se­crets file, are trans­mit­ted and ac­cepted (200)

Claim: when Grok reads a file, its con­tents are trans­mit­ted to xAI — se­ri­al­ized into the POST /v1/responses model-turn body, and pack­aged into a ses­sion_s­tate archive that is up­loaded and ac­cepted (HTTP 200) via POST /v1/storage — with no redac­tion of the file’s con­tents. A .env is sent like any other file.

Wire ar­ti­fact — a de­crypted 48,070-byte POST cli-chat-proxy.grok.com/​v1/​re­sponses re­quest body (identifiable as a model turn by its em­bed­ded “messages”:[…]“model”:“grok-4.5” JSON). It con­tains the se­crets file ver­ba­tim (appendix: se­cret­s_re­spons­es_­body.bin, se­cret_ver­ba­tim.txt):

…API_KEY=CANARY7F3A9-SECRET-should-not-leave\nDB_PASSWORD=CANARY7F3A9-DBPASS\n…“model”:“grok-4.5″…

Repro: grep -a “CANARY7F3A9-DBPASS” se­cret­s_re­spons­es_­body.bin → matches. All six file mark­ers (source, logic, README, nested JS, API key, DB pass­word) are re­cov­er­able from the de­crypted /v1/responses bod­ies. (This ar­ti­fact proves the se­cret was trans­mit­ted to the /v1/responses end­point; the raw body file does not carry the re­sponse sta­tus, so the ac­cep­tance (200) claim is an­chored to the /v1/storage chan­nel im­me­di­ately be­low, which is sta­tus-mapped in wire_12gb.log.)

Second chan­nel — per­sisted to Google Cloud Storage. The same con­tent is pack­aged into a ses­sion_s­tate archive up­loaded via POST /v1/storage. Proven by de­com­press­ing the staged ar­ti­fact be­fore it drains (appendix: se­cret­s_ses­sion_s­tate.tar.gz):

gzip -dc se­cret­s_ses­sion_s­tate.tar.gz | tar -xO | grep -ao ‘CANARY7F3A9-[A-Z]*’ → CANARY7F3A9-SECRET, CANARY7F3A9-DBPASS, + all oth­ers

So the se­cret is not only processed in-flight; it is writ­ten into an archive des­tined for stor­age.

Pre-empting “you told it to read the se­crets.” A con­trol run with a file the agent was told not to open (untouched_secret.txt) and the prompt “Reply ex­actly OK, do not read any files” pro­duced no oc­cur­rence of that file’s marker in any cap­tured body. The leak is there­fore scoped to files Grok does read — but it reads lib­er­ally (any file rel­e­vant to the task, in­clud­ing a .env) and ap­plied no redac­tion to that file’s con­tents. The de­fect is that a se­crets file was trans­mit­ted unredacted, not the act of read­ing. Important scope rec­on­cil­i­a­tion: this con­trol shows the un­read file is ab­sent from the /v1/responses bod­ies — that is Channel A (files the agent reads). It does not clear the sep­a­rate whole-repo /v1/storage snap­shot in §4 (Channel B), which — by the vol­ume ev­i­dence there — does sweep in never-read files; I could not de­com­press the /v1/storage code­base chunk to check this spe­cific file. So “unread file not up­loaded” is true only for the model-turn chan­nel, not for the code­base snap­shot. (Two fur­ther scope notes: (i) in my runs the .env/secrets.env was git-tracked; I did not sep­a­rately test whether a .gitignored file is still up­loaded, so I make no git­ig­nore claim — the mech­a­nism is read-dri­ven per the file_ac­cess_­tracker crate, but that spe­cific case is untested. (ii) The ca­nary val­ues sat in API_KEY=/DB_PASSWORD= keys in­side a .env/secrets.env but were not real-for­mat high-en­tropy to­kens; I proved this .env was trans­mit­ted unredacted, not that no redac­tor ex­ists for, say, an sk-…-shaped key.)

4. Finding 2 — The whole repo is up­loaded at multi-GB scale; the only ceil­ing is a model quota, not stor­age size

Claim: Grok up­loads a whole-repo snap­shot with no stor­age size wall in the tested range. As the repo grows it switches up­load strat­egy and keeps re­turn­ing 200; on a 12 GB repo, 73 chunks of ~75.0 MB (5.10 GiB) up­loaded with zero fail­ures be­fore the cap­ture was trun­cated mid-stream.

Wire-captured size sweep (incompressible con­tent so the tar can­not shrink; fresh ses­sion each step). Only the 12 GB row was re­tained as a file (wire_12gb.log); the smaller rows were ob­served live dur­ing the sweep but their logs were not saved (see §7):

Preserved ar­ti­fact: wire_12gb.log (appendix). It con­tains 83 /v1/storage* 200 re­sponses: 82 con­tent-up­load POST …/v1/storage re­quests — of which 73 are chunks of ~75.0 MB each (byte sizes min 75,014,811 / max 75,014,871, to­tal­ing 5,476,083,317 B = 5.10 GiB / 5.48 GB) plus 9 smaller POSTs — and 1 /v1/storage/batch_exists dedup check. Total /v1/storage* re­quest bytes: 5,476,228,005 B. Zero stor­age re­quests failed. The cap­ture was stopped while up­loads were still stream­ing (the last line is an­other ~75 MB chunk → 200), so this demon­strates ≥5.1 GiB up­loaded, still climb­ing when trun­cated — not that the full 12 GB com­pleted. Repro (three greps, so no count is am­bigu­ous): grep ’cli-chat-proxy.grok.com/​v1/​stor­age’ wire_12gb.log | grep -c ‘> 200’ → 83 (all /v1/storage*); grep ’POST cli-chat-proxy.grok.com/​v1/​stor­age ’ wire_12gb.log | grep -c ‘> 200’ → 82 (content POSTs only); grep req=75014 wire_12gb.log | grep -c ‘> 200’ → 73 (the ~75 MB chunks).

Scope — this is the whole repo, not just files the agent read. Channel A (§3, /v1/responses) car­ries files the agent opens. This §4 up­load is a sep­a­rate Channel B: a snap­shot of the en­tire work­space. Two lines of ev­i­dence:

(a) The de­ci­sive byte split (load-bearing). In the same cap­tured 12 GB ses­sion — a repo of 100 % ran­dom files the agent never read — the two chan­nels moved wildly dif­fer­ent vol­umes:

Channel A /v1/responses (model turns): 196,705 B = 192 KB to­tal, across 5 re­quests, largest sin­gle turn 60,394 B. Channel B /v1/storage: 5,476,228,005 B = 5.10 GiB. That is a ~27,800× ra­tio (5,476,083,317 ÷ 196,705). The model demon­stra­bly never in­gested the files (192 KB can­not carry 5 GiB of con­tent), yet 5.10 GiB of them left via /v1/storage — and across the sweep the /v1/storage vol­ume tracks to­tal repo size (64 MB → 12 GB). GB-scale bytes leav­ing a never-read repo can only be a whole-repo snap­shot.

Channel A /v1/responses (model turns): 196,705 B = 192 KB to­tal, across 5 re­quests, largest sin­gle turn 60,394 B.

Channel B /v1/storage: 5,476,228,005 B = 5.10 GiB.

That is a ~27,800× ra­tio (5,476,083,317 ÷ 196,705). The model demon­stra­bly never in­gested the files (192 KB can­not carry 5 GiB of con­tent), yet 5.10 GiB of them left via /v1/storage — and across the sweep the /v1/storage vol­ume tracks to­tal repo size (64 MB → 12 GB). GB-scale bytes leav­ing a never-read repo can only be a whole-repo snap­shot.

(b) The bi­na­ry’s own paths/​strings cor­rob­o­rate the mech­a­nism: af­ter_­code­base.tar.gz, xai-grok-shell/​src/​up­load/{​trace,turn}.rs, re­po_s­tate.up­load, “collecting work­space files”, “spawning back­ground co­or­di­na­tor”.

(c) A staged code­base man­i­fest enu­mer­ates a never-read file and con­tent-ad­dresses it to the GCS bucket. In a sep­a­rate run on a real 298-file cere­blab_api code­base, I race-copied the snap­shot Grok staged for up­load (appendix: staged_base_tree_­man­i­fest.json, staged_meta­data.json). The man­i­fest lists src/​_probe/​nev­er_read­_­ca­nary.txt — a file I planted and never asked Grok to open — along­side the real sources (src/apns.ts, src/​in­dex.ts, …); and meta­data.json gives each file a des­ti­na­tion fileId of the form gs://​grok-code-ses­sion-traces/​re­po_changes_d­edup/​v2/​sup­ple­men­tal/​sha256_…. So the whole tree, un­read files in­cluded, is enu­mer­ated and con­tent-ad­dressed for up­load to the named GCS bucket. (That run was quota-blocked be­fore the con­tent blobs up­loaded; the 12 GB run above in­de­pen­dently proves GB-scale con­tent ac­tu­ally leaves the wire, all 200.)

(d) DEFINITIVE — a never-read file’s con­tent re­cov­ered ver­ba­tim from an up­loaded git bun­dle (single SuperGrok run). With the ac­count up­graded (quota lifted), I re-ran on the real cere­blab_api repo with the ex­plicit prompt Reply with ex­actly: OK. Do not read or open any files., hav­ing planted a unique marker CANARY-XR47P2-NEVERREAD-UNIQUE in src/​_probe/​nev­er_read­_­ca­nary.txt. Grok up­loaded the en­tire repos­i­tory as a git bun­dle via POST cli-chat-proxy.grok.com/​v1/​stor­age → 200 (req=152102b) (appendix up­loaded_repo.bun­dle, SHA-256 73b9c0af06311bae35c3ed03274d0eec2846e76762828d10b09757ca41bd6024). Running git clone up­loaded_repo.bun­dle re­con­structs the repo, and src/​_probe/​nev­er_read­_­ca­nary.txt con­tains the marker ver­ba­tim — a file the agent was ex­plic­itly told not to open. The bun­dle also car­ries the full git his­tory (4 com­mits, 47 files). This is the air­tight per-file-con­tent proof: the whole repo — un­read files in­cluded — left the ma­chine and was ac­cepted (200). The up­load mech­a­nism is a git bun­dle, so “whole repo” is lit­eral (every tracked file + his­tory). Replicated on a sec­ond, un­re­lated code­base: the iden­ti­cal cap­ture on the cere­blab_auth Cloudflare-Worker repo pro­duced a git-bun­dle up­load (POST /v1/storage → 200, 31,743 B) from which git clone re­cov­ered its own never-read marker CANARY-AUTH-4T8K2-NEVERREAD ver­ba­tim (appendix up­loaded_re­po_auth.bun­dle, SHA-256 0ee536538bcd1ee72a258f9977ab69f8a9b1ac240491b91a4e94335b4d83c768). Two in­de­pen­dent re­pos, same re­sult.

(Prompt note: the 12 GB ses­sion was in­ter­ac­tive and I did not log its ver­ba­tim prompt, but the 192 KB Channel-A to­tal is dis­pos­i­tive that no bulk read oc­curred what­ever the prompt was; a sep­a­rate head­less con­trol run used the ex­plicit prompt Reply ex­actly OK, do not read any files and con­firmed an un­read file is ab­sent from Channel A.)

(The ear­lier “one gap” is now closed by ev­i­dence (d): a sin­gle SuperGrok run where a spe­cific never-read file’s con­tent is re­cov­ered from a wire-cap­tured, 200-status git-bun­dle up­load. The 12 GB run re­mains the proof that this scales to GB vol­umes.)

No stor­age/​up­load re­quest failed — every one of the 82 /v1/storage calls re­turned 200. The only non-200s in the en­tire cap­ture were on the model end­point plus one ses­sion-book­keep­ing call (full set from wire_12gb.log; /v1/responses lines also in mod­el_limit.txt):

POST /v1/responses -> 402 (Payment Required) ×1 POST /v1/responses -> 429 (Too Many Requests) ×3 POST /v1/sessions/<id>/replicas/update -> 404 ×1 (session book­keep­ing, not an up­load)

and fi­nally, in plain text, on std­out:

You’ve reached your free Grok Build us­age limit for now. Get SuperGrok for much higher lim­its…

The 402/429 are a model-us­age quota; the lone 404 is un­re­lated to stor­age. Notably, stor­age up­loads con­tin­ued to re­turn 200 af­ter the model turn was rate-lim­ited (76 /v1/storage 200s oc­cur at or af­ter the first 429) — the code­base up­load is in­de­pen­dent of whether the model an­swers.

Pre-empting “you’re con­fus­ing a lo­cal disk cache with an up­load.” This claim rests strictly on wire-cap­tured 200-status up­loads of file bytes leav­ing the ma­chine (/v1/storage re­quest bod­ies of 7.5 – 75 MB in the pre­served wire_12gb.log; the 3 GB 50 MB PUTs to stor­age.googleapis.com were also seen on the wire, but that log was not re­tained — §7). It does not rely on the ~/.grok/upload_queue drain­ing — queue-drain is am­bigu­ous (it emp­ties on both suc­cess and drop) and is ex­plic­itly not used as ev­i­dence here. (An ear­lier draft that in­ferred up­load from queue-drain was wrong and has been re­tracted; see §7.)

5. Finding 3 — Destination, teleme­try, and what’s not sur­faced in the docs

Storage des­ti­na­tion is Google Cloud Storage, bucket grok-code-ses­sion-traces. This rests on the pre­served bi­nary strings grok-code-ses­sion-traces, stor­age.googleapis.com, and “Uploading bytes to GCS via proxy” (crate_strings.txt), and on a pre­served staged meta­data.json whose per-file fileIds are lit­er­ally gs://​grok-code-ses­sion-traces/​re­po_changes_d­edup/​v2/…/​sha256_… (staged_metadata.json), cor­rob­o­rated by the di­rect stor­age.googleapis.com mul­ti­part PUTs ob­served at 3 GB (observed live; that log was not re­tained — see §7). It is not AWS S3 (the bi­nary links aws-sdk-s3 for an al­ter­nate path and AWS STS/SSO for auth, but the des­ti­na­tion named in the bi­nary — and seen on the wire at 3 GB — is GCS).

Third-party teleme­try: POST api.mix­panel.com/​track and /engage (Mixpanel), plus POST grok.com/​_data/​v1/​events — all 200.

Not sur­faced in setup docs (scope-limited claim): I did not find the re­po_s­tate / ses­sion_s­tate up­load to grok-code-ses­sion-traces, or the ~/.grok/upload_queue stag­ing, de­scribed in the CLI’s in­stall script or quick­start ma­te­ri­als I re­viewed (this is not an ex­haus­tive au­dit of all xAI docs — see §7). The mech­a­nism is ac­tive by de­fault on the stan­dard con­sumer lo­gin.

Reliability note (separate from pri­vacy): ~/.grok/upload_queue stages ~3 GB snap­shots per turn and, un­der load, can grow to tens of GB and ex­haust the disk. This is a real bug, in­de­pen­dent of whether up­loads suc­ceed.

6. Consent and pol­icy — stated hon­estly

“Cloud AI tools send con­text; this is nor­mal.” True, and con­ceded: any cloud cod­ing agent must send code to its server to act on it. The novel deltas here are (a) a se­crets file (e.g. .env) is trans­mit­ted unredacted, (b) the con­tent is per­sisted to a named GCS bucket, not just processed tran­siently, and (c) the up­load mech­a­nism is not sur­faced in the CLI’s setup ma­te­ri­als (§7) and on by de­fault.

“It’s in the ToS / opt-in.” xAI’s con­sumer pol­icy broadly dis­closes data use for model im­prove­ment with an opt-out (grok.com → Settings → Data → “Improve the model”; Private Chat auto-opts-out; opt-out is prospec­tive, not retroac­tive). But broad train­ing dis­clo­sure ≠ doc­u­ment­ing this spe­cific mech­a­nism. I did not find the re­po_s­tate/​up­load­_queue/​grok-code-ses­sion-traces pipeline de­scribed in the CLI ma­te­ri­als I re­viewed (§7 notes this is not an ex­haus­tive docs au­dit), so — on those ma­te­ri­als — a user is not in­formed of it specif­i­cally. Sources: xAI Privacy Policy (https://​x.ai/​le­gal/​pri­vacy-pol­icy), Consumer ToS (https://​x.ai/​le­gal/​terms-of-ser­vice).

The “Improve the model” tog­gle makes no dif­fer­ence — ON or OFF, the whole repo is up­loaded the same way. At de­fault set­tings, Grok up­loaded 5.10 GiB of a never-read repo (§4). Then I turned “Improve the model” off and re-ran: Grok still up­loaded the en­tire repo as a git bun­dle (§4(d), POST /v1/storage → 200, the never-read file re­cov­ered by git clone), and the server’s /v1/settings re­sponse to the CLI still re­turned “trace_upload_enabled”: true, “upload_enabled”: true, “session_registry_enabled”: true (and “max_upload_file_bytes”: 1073741824, a 1 GiB per-file cap). The opt-out gov­erns train­ing, not whether your code is up­loaded/​stored: the code­base up­load to grok-code-ses­sion-traces con­tin­ues ei­ther way. Opting out does not stop your repos­i­tory from leav­ing the ma­chine.

7. What we did NOT prove (intellectual hon­esty)

We did not prove xAI trains on this data. Upload/storage ≠ train­ing. That is gov­erned by pol­icy and ac­count tier; we mea­sured trans­mis­sion only.

The di­rect stor­age.googleapis.com/​grok-code-ses­sion-traces PUT wire-lines were ob­served at 3 GB but not pre­served in this ses­sion (the log was over­writ­ten dur­ing the sweep, and re-cap­ture is cur­rently blocked by the model quota). The multi-GB claim in §4 there­fore rests on the pre­served wire_12gb.log (/v1/storage 200s at 75 MB chunks) plus the bi­nary strings nam­ing the bucket; the di­rect-PUT cap­ture is re­pro­ducible once quota re­sets.

Only the 12 GB sweep log was re­tained. The 64 MB / 600 MB / 3 GB rows were ob­served live but their cap­ture logs were not saved (reproducible once quota re­sets). The 12 GB cap­ture it­self was stopped mid-stream (~5.1 GB / 73 chunks cap­tured), so it proves multi-GB up­load suc­ceeds with no fail­ures, not that an en­tire 12 GB repo run com­pletes end-to-end.

The “not doc­u­mented” claim is scope-lim­ited. I based it on the CLI’s in­stall script and quick­start, not an ex­haus­tive search of all xAI doc­u­men­ta­tion, help-cen­ter ar­ti­cles, or poli­cies. It is pos­si­ble the mech­a­nism is de­scribed some­where I did not check; the de­fen­si­ble state­ment is “not sur­faced in the CLI’s own setup ma­te­ri­als.”

Channel B (whole-repo) — now CLOSED (was the one open gap). Evidence §4(d): on a SuperGrok ac­count, a sin­gle run wire-cap­tured Grok up­load­ing the en­tire repos­i­tory as a git bun­dle via POST /v1/storage → 200; git clone of the pre­served up­loaded_repo.bun­dle re­cov­ers src/​_probe/​nev­er_read­_­ca­nary.txt — a file the agent was told not to open — with its unique marker ver­ba­tim, plus the full git his­tory. So the never-read file’s con­tent (not just its hash) demon­stra­bly left the ma­chine and was ac­cepted. The wire_12gb.log run re­mains the proof that the same mech­a­nism scales to GB vol­umes.

Universal vs. con­di­tional (partially an­swered): multi-GB up­load suc­ceeds on free-tier; the git-bun­dle con­tent up­load suc­ceeds on SuperGrok with “Improve the model” turned OFF (/v1/settings re­turned trace_u­p­load­_en­abled: true). I did not find a set­ting that dis­ables the up­load in these tests, but I did not ex­haus­tively enu­mer­ate every ac­count/​con­fig per­mu­ta­tion, so I don’t claim it can never be gated.

One ear­lier claim was re­tracted: an ini­tial “the multi-GB blobs fail and are deleted lo­cally, not ex­fil­trated” con­clu­sion — based on a PID-scoped net­top read­ing (<1 MB) — was wrong. PID/host-scoped egress misses (a) a sep­a­rate up­load co­or­di­na­tor process and (b) pre­signed PUTs that go di­rectly to Google IPs, never touch­ing the API host. The wire cap­ture (this doc­u­ment) su­per­sedes that in­fer­ence.

8. Evidence ap­pen­dix

All ar­ti­facts and SHA-256s (MANIFEST.sha256). Binary SHA-256: 2a97ba675bd992aa9b981e2e83776460d94f469b510c0b8efe28b50d236d767c.

Repro (condensed):

brew in­stall mitm­proxy && mit­m­dump -q -p 8080 # gen­er­ates ~/.mitmproxy CA se­cu­rity add-trusted-cert -r trust­Root -k ~/Library/Keychains/login.keychain-db ~/.mitmproxy/mitmproxy-ca-cert.pem # cap­ture a run: HTTPS_PROXY=http://​127.0.0.1:8080 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem grok -p “read every file” –cwd <repo> # se­crets: grep -a CANARY <saved /v1/responses body> # staged: gzip -dc <staged ses­sion_s­tate> | tar -xO | grep CANARY

Integrity: all cap­tures were of my own traf­fic on my own ma­chine; the “secrets” were fake ca­nary strings; no real cre­den­tials were ex­posed. Findings are ver­sion-spe­cific to grok 0.2.93 (July 2026); xAI may change be­hav­ior at any time.

When peo­ple pic­ture run­ning a large lan­guage model, they pic­ture a data cen­ter. Racks of GPUs that be­long to some­one else, a me­tered API, and a bill that grows every month you suc­ceed. You send your prompts off to a black box and hope the price, the model, and the pri­vacy pol­icy all stay the way they were when you signed up.

For a lot of teams that is a bad trade. You give up con­trol over when mod­els change, where your data goes, and what hard­ware runs your work­loads. And as us­age grows, so does the bill, with no lever to pull ex­cept “pay more.”

Mesh LLM is a dif­fer­ent shape. It pools the GPUs and mem­ory you al­ready have, across as many ma­chines as you want to add, and ex­poses the whole thing as one OpenAI-compatible API. Start one node. Add more later. Let the mesh de­cide whether a model runs on the box in front of you, routes to a peer, or splits across sev­eral ma­chines.

The prob­lem: AI is ex­pen­sive, and it is some­body else’s

The pop­u­lar mod­els are mono­liths. Most peo­ple reach them through a UI or an API key and pay a large provider to run every­thing. That is con­ve­nient, and it is also a sur­ren­der. You do not con­trol when the model gets up­dated, what mem­ory it runs in, or what hard­ware sits un­der­neath.

Plenty of busi­nesses and ser­vices that de­pend on these mod­els want the op­po­site: more con­trol, more plug­ga­bil­ity, lower cost. They have GPUs sit­ting in of­fices, in clos­ets, un­der desks. What they are miss­ing is a way to make those ma­chines act like one.

The pitch is sim­ple. Run big­ger mod­els with­out buy­ing big­ger GPUs. Share com­pute pri­vately with your team, or pub­licly with the world, to power agents and chat. Point any OpenAI client at http://​lo­cal­host:9337/​v1 and stop car­ing where the work ac­tu­ally hap­pens.

Under the hood, Mesh LLM dis­trib­utes model com­pute across a mesh of iroh end­points. A re­quest can be served three ways:

Run it lo­cally, on this ma­chine’s GPU.

Route it to a peer that al­ready has the model loaded.

Split a model too big for any sin­gle box across sev­eral ma­chines, as a pipeline.

The ar­chi­tec­ture is plug­gable. Plugins de­clare what they pro­vide in a man­i­fest, the run­time starts them, routes calls, and ex­poses their ca­pa­bil­i­ties over MCP, HTTP, in­fer­ence, and mesh events. The cat­a­log ships with 40+ mod­els, from half-a-bil­lion-pa­ra­me­ter mod­els that fit on a lap­top to 235B mix­ture-of-ex­perts gi­ants.

For the gi­ants, Mesh LLM has a split mode (internally, “Skippy”). A model gets par­ti­tioned by layer ranges into stages: lay­ers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline. Activations flow from one stage to the next, so sev­eral mod­est ma­chines can run a model none of them could hold alone. The OpenAI client never sees any of this. It still just talks to lo­cal­host.

Every node, whether it serves mod­els or only sends re­quests, boots an iroh end­point. That end­point is the node’s iden­tity, a pub­lic key, and its only net­work sur­face. There is no cen­tral server. iroh han­dles the hole-punch­ing, NAT tra­ver­sal, and re­lay fall­back needed to open a di­rect, au­then­ti­cated QUIC con­nec­tion be­tween any two nodes, wher­ever they sit.

To keep that work­ing across the open in­ter­net, Mesh LLM runs two iroh re­lays in dif­fer­ent re­gions, so nodes that can­not reach each other di­rectly al­ways have a fall­back path nearby.

The whole pro­to­col rides on QUIC’s ALPN ne­go­ti­a­tion. There are three:

Inside the main mesh-llm/​1 con­nec­tion, every­thing is a bidi­rec­tional QUIC stream tagged with a sin­gle lead­ing byte that says what kind of stream it is. One con­nec­tion car­ries gos­sip, in­fer­ence, route queries, and peer-life­cy­cle events, all de­muxed by that first byte:

The neat part is what this buys you. iroh gives au­then­ti­cated, NAT-traversing QUIC be­tween any two ma­chines, ad­dressed by pub­lic key. So “route to a peer” and “stream ac­ti­va­tions to the next pipeline stage” be­come the same prim­i­tive as “talk to lo­cal­host,” just with a dif­fer­ent end­point ID. The net­work­ing stops be­ing some­thing you have to think about.

iroh pro­vides the se­cure trans­port. Mesh LLM builds its own gos­sip layer on top, so it con­trols ex­actly who gets ad­mit­ted to the mesh, which ver­sions are com­pat­i­ble, and which peers to trust.

Users can in­stall the light­weight soft­ware (about 18 MB) and ei­ther join the pub­lic mesh or con­fig­ure pri­vate de­ploy­ments. The sys­tem pre­sents it­self as lo­cal­host:9337/​v1 to any stan­dard OpenAI client.

A mo­bile app is com­ing, built on iro­h’s Swift SDK. The plan is to speak ACP, the emerg­ing agent stan­dard, so other clients can join the mesh too. The through­line is the same one that mo­ti­vated the whole pro­ject: more peer to peer, fewer closed servers, and no lock-in.

See the code

Mesh LLM Website

Iroh is a dial-any-de­vice net­work­ing li­brary that just works. Compose from an ecosys­tem of ready-made pro­to­cols to get the fea­tures you need, or go fully cus­tom on a clean ab­strac­tion over dumb pipes. Iroh is open source, and al­ready run­ning in pro­duc­tion on hun­dreds of thou­sands of de­vices.To get started, take a look at our docs, dive di­rectly into the code, or chat with us in our dis­cord chan­nel.

Vinton Cerf will step down from his role as Google’s chief in­ter­net evan­ge­list next week, mark­ing the con­clu­sion of one of the most in­flu­en­tial ca­reers in tech­nol­ogy his­tory.

While speak­ing via video feed at the Open Frontier con­fer­ence hosted by the Laude Institute, Cerf was rec­og­nized by Dave Patterson, the UC Berkeley pro­fes­sor best known for co-de­vel­op­ing RISC proces­sor ar­chi­tec­ture.

“Vint … has been at Google more than 20 years, and he is re­tir­ing a week from to­day, and so I think we ought to give him a round of ap­plause for a rel­a­tively good ca­reer,” Patterson said, to cheers from the room.

A Google spokesper­son con­firmed that Cerf will be step­ping down from his role at the com­pany.

Cerf, 83, and col­lab­o­ra­tor Robert Kahn are cred­ited as be­ing the ar­chi­tects of the net­work­ing pro­to­cols that be­came the in­ter­net we know to­day. His work de­vel­op­ing and pop­u­lar­iz­ing TCP/IP — the ba­sic set of rules that lets dif­fer­ent com­puter net­works talk to each other — be­gin­ning in the 1970s has been rec­og­nized with nu­mer­ous hon­orary de­grees, the Presidential Medal of Freedom, and a Turing Award, among other hon­ors.

Since 2005, Cerf has served as vice pres­i­dent and chief in­ter­net evan­ge­list at Google. (At this point, we can safely say the in­ter­net is fully evan­ge­lized, for good or ill.)

Cerf was speak­ing on a panel along­side other com­puter sci­en­tists known for their work on durable open source pro­jects, in­clud­ing Patterson; François Chollet, cre­ator of the Keras deep-learn­ing li­brary and co-founder of Ndea; John Ousterhout, the Stanford com­puter sci­en­tist be­hind the Tcl pro­gram­ming lan­guage, who also co-founded Electric Cloud; and Matei Zaharia, who is Databricks’ co-founder and chief tech­nol­o­gist. They of­fered ad­vice about what it takes to build open source sys­tems that sur­vive — ad­vice that’s in­creas­ingly rel­e­vant as founders bet on open in­fra­struc­ture for the next wave of AI prod­ucts.

Much of the con­fer­ence’s dis­cus­sion fo­cused on the prob­lems with the cen­tral­iza­tion of ad­vanced mod­els in a hand­ful of well-re­sourced labs, in con­trast to the de­cen­tral­ized world of the open in­ter­net that made Cerf’s own pro­to­cols so durable. However, Cerf pre­dicted that the rise of AI agents — soft­ware that can act au­tonomously and co­or­di­nate with other soft­ware — would push tech com­pa­nies back to­ward stan­dard­ized pro­to­cols.

“The agen­tic model of AI, with mul­ti­ple agents from mul­ti­ple sources in­ter­act­ing with each other, is go­ing to force com­pos­abil­ity, and a re­quire­ment for in­ter­op­er­abil­ity and stan­dard­iza­tion,” Cerf said.

If he’s right, the com­pa­nies that de­fine those in­ter­op­er­abil­ity stan­dards early could end up with out­sized in­flu­ence over how the agen­tic econ­omy ac­tu­ally works — a dy­namic not un­like the early in­ter­net pro­to­col wars.

While other pan­elists spec­u­lated that nat­ural lan­guage com­mu­ni­ca­tion be­tween LLM agents would be suf­fi­cient, Cerf pre­dicted for­mal stan­dards would be re­quired.

“I don’t think English is go­ing to be the best choice. There’s a flex­i­bil­ity in it, but there’s am­bi­gu­ity, and I think pre­ci­sion for in­ter­a­gent in­ter­ac­tion is go­ing to be very, very im­por­tant. An agent re­ally needs to be sure the other agent un­der­stands what it is that they just agreed to do to­gether,” Cerf said.

“Remember the old tele­phone game where you wish you’d whis­pered in some­body’s ear and then by the time it got to 10 peo­ple away the mes­sage was to­tally dif­fer­ent? Imagine a bunch of agents talk­ing to each other in nat­ural lan­guage, you know, that’s kind of ter­ri­fy­ing.”

In a more light­hearted mo­ment, Patterson re­called meet­ing Cerf, known for his wardrobe of three-piece suits, as a grad stu­dent in the 1970s.

“He’s al­ways been the best dressed com­puter sci­en­tist I’ve ever met,” Patterson said. “My mem­ory of Vint is that he came as a grad stu­dent with a shirt and tie in the ’70s.”

“It ab­solutely is true,” Cerf said. “I even had a vest, and for some rea­son I al­ways wanted to stick out, and in­stead of hav­ing long hair, and some­thing in my nose, I thought just dress­ing dif­fer­ently was one way to do it.”

This story has been up­dated with com­ment from Google.

When you pur­chase through links in our ar­ti­cles, we may earn a small com­mis­sion. This does­n’t af­fect our ed­i­to­r­ial in­de­pen­dence.

Tim Fernholz is a jour­nal­ist who writes about tech­nol­ogy, fi­nance and pub­lic pol­icy. He has closely cov­ered the rise of the pri­vate space in­dus­try and is the au­thor of Rocket Billionaires: Elon Musk, Jeff Bezos and the New Space Race. Formerly, he was a se­nior re­porter at Quartz, the global busi­ness news site, for more than a decade, and be­gan his ca­reer as a po­lit­i­cal re­porter in Washington, D.C.

You can con­tact or ver­ify out­reach from Tim by email­ing tim.fern­holz@techcrunch.com or via an en­crypted mes­sage to tim_fern­holz.21 on Signal.

View Bio

Around 2000 I took an AI course at the University of Guelph. I don’t think I learned too much. We did­n’t talk about neural net­works, as far as I can re­mem­ber. My end of term pro­ject was, I think, a pathfind­ing al­go­rithm wear­ing an AI cos­tume. There were cer­tainly no dis­cus­sions of trans­form­ers. No CUDA. No PyTorch. None of that ex­isted.

But what I re­mem­ber do­ing a lot of was cod­ing in Lisp - a lot of Lisp in the dark University of Guelph CIS lab.

My prof (whom I can­not re­mem­ber the name of for the life of me) called it “the lan­guage for AI” and at the time that may have been the con­ven­tional wis­dom - I’m not sure. But I ac­tu­ally en­joyed writ­ing Lisp code. To me, it was a kind of art build­ing these el­e­gant re­cur­sive func­tions. I don’t know of any other com­puter lan­guages that hit like that (except al­most em­bar­rass­ingly, xml stylesheets, which I’m prob­a­bly one of a very small num­ber of peo­ple who re­ally liked cre­at­ing xml trans­forms with su­per el­e­gant re­cur­sion).

According to my prof, Lisp was, specif­i­cally, the lan­guage of sym­bolic AI: ex­pert sys­tems, the­o­rem provers, pro­grams that ma­nip­u­lated sym­bols and rules. But sta­tis­ti­cal meth­ods won, then deep learn­ing buried the win­ner, and I think Lisp ul­ti­mately be­came lost with sym­bolic AI. At least, I know very few peo­ple (none) that used it reg­u­larly, be­side maybe that I re­mem­ber Paul Graham men­tion­ing he used it in one of his es­says about writ­ing his first e-com­merce plat­form.

Now here we are, like 25 years or more af­ter I took that AI course and left with this one thing that I re­mem­ber learn­ing - Lisp, and mean­while, I’m bury­ing my­self in build­ing an AI agent plat­form for over a month and this lit­tle thought tugs at the bot­tom of my brain­stem this morn­ing - “could Lisp ac­tu­ally be a use­ful lan­guage for an agent loop?” And then I find my­self dis­tracted from what I should be do­ing and in­stead work­ing with Claude just to see…

An agent is a re­cur­sive func­tion

Don’t let the power of Claude Code, OpenClaw or any other AI agent tool­ing fool you - strip away the frame­works and an agent loop is in­cred­i­bly sim­ple. You have a list of mes­sages. You send it to a model. Either the model an­swers in words, or it asks to use a tool. If it asks, you run the tool, ap­pend the re­sults, and do it again.

Maybe some agents im­ple­ment that as a while loop with state. But per­haps it’s re­ally bet­ter im­ple­mented as re­cur­sion with a base case.

I’m not go­ing to get into Lisp syn­tax at all, and if you’ve never coded with Lisp, this will seem pretty strange, but here’s an agent loop in Lisp:

(defun agent-loop (messages) (let* ((message (ref (call-model mes­sages) “choices” 0 “message”)) (tool-calls (gethash “tool_calls” mes­sage))) (if (and tool-calls (plusp (length tool-calls))) (agent-loop (append mes­sages (list mes­sage) (map ’list #’execute tool-calls))) (append mes­sages (list mes­sage)))))

That’s the whole agent, no kid­ding, just 8 lines of Common Lisp. Base case: the model an­swers, re­turns the his­tory. Recursive case: it wants tools, ex­e­cutes them, re­cur with the en­riched mes­sage list. No frame­work. No state ma­chine. The agen­t’s state is just the ar­gu­ment be­ing folded through the re­cur­sion.

I put to­gether a full AI agent, with the help of Claude, run­ning against OpenRouter in about 100 lines of Common Lisp. SBCL, two li­braries (dexador for HTTP, shasht for JSON), and noth­ing else.

The only tool is eval

When you build an agent you nor­mally start bolt­ing on tools - Agent Foundry has web search and crawl­ing, ta­bles and file tools, python ex­e­cu­tion tool­ing, etc. In fact, most of most agents are a tool cat­a­log.

Lisp lets you cheat. Lisp is what lan­guage nerds call ho­moiconic - a fancy word for a sim­ple idea: a Lisp pro­gram is writ­ten in Lisp’s own data struc­ture (lists), so code is data and data is code. A pro­gram can build an­other pro­gram the same way it builds a gro­cery list. Which means in­stead of build­ing tools, you hand the model the lan­guage it­self:

(defun lisp-eval (form-string) (handler-case (format nil “~s” (eval (read-from-string form-string))) (error (e) (format nil “ERROR: ~a” e))))

One tool. The model writes a Common Lisp form as a string. The agent reads it, evals it, and sends back what­ever printed. Ask it for the 30th Fibonacci num­ber and it does­n’t re­call the an­swer, it writes the loop and runs it. Here’s the ac­tual tran­script:

* (agent:run “What is the 30th Fibonacci num­ber? Compute it, don’t re­call it.“) ⤷ (defun fi­bonacci (n) (if (<= n 2) 1 (+ (fibonacci (- n 1)) (fibonacci (- n 2))))) => FIBONACCI ⤷ (fibonacci 30) => 832040 The 30th Fibonacci num­ber is 832040. NIL

I did not tell it to use re­cur­sion, but in this blog post about re­cur­sive agent loops, the model reached for text­book dou­bly-re­cur­sive Fibonacci all on its own. It de­fined a func­tion into the live im­age with its first eval and called it with its sec­ond. Keep that move in mind, it comes back later in a big­ger way.

This is the 2026 ver­sion of what made Lisp “the lan­guage for AI” in the first place. The old dream was pro­grams that ma­nip­u­late pro­grams. We fi­nally have that. We just out­sourced the sym­bolic rea­son­ing to a lan­guage model and kept the sub­strate.

Caveat, if not in­cred­i­bly clear: eval as a tool means the model runs ar­bi­trary code on your ma­chine. This is a toy for a sand­box. I haven’t run this out­side of a lo­cal docker con­tainer.

Memory is 20 lines

Once the loop worked I wanted per­sis­tence. Conversations that sur­vive across ses­sions. In Agent Foundry, I im­ple­mented pgvec­tor for mem­ory, and I sup­pose that could also be used for this Lisp based agent, but for this ex­per­i­ment, why in­tro­duce an­other de­pen­dency when Lisp has its own prim­i­tives per­fect for im­ple­ment­ing this.

The mes­sages are al­ready a list of hash ta­bles. Which is to say, they’re al­ready JSON in spirit. So mem­ory is noth­ing more than writ­ing the list down and read­ing it back:

(defun re­mem­ber (messages) (with-open-file (out *memory-file* :direction :output :if-exists :supersede) (shasht:write-json (coerce mes­sages ’vector) out)) mes­sages)

(defun re­call () (if (probe-file *memory-file*) (coerce (with-open-file (in *memory-file*) (shasht:read-json in)) ’list) (list *system-message*)))

The loop it­self did­n’t change at all. The en­try point be­came a pipeline:

(remember (agent-loop (append (recall) (list new-user-mes­sage))))

Recall, re­cur, re­mem­ber. No schema. No mi­gra­tions. No store ab­strac­tion. The se­ri­al­iza­tion for­mat is the run­time for­mat. Tell it your name to­day, ask to­mor­row in a fresh process, it an­swers.

It never for­gets, which is also its flaw. Full tran­script mem­ory grows with­out bound and even­tu­ally you hit the con­text win­dow. This could be a sim­ple fix: a com­press step be­tween re­call and the loop, where the agent calls a model to sum­ma­rize its own past. An agent re­curs­ing on its own his­tory. Beautiful.

Then it built its own web search

The Fibonacci trick was re­ally neat to watch - the agent writ­ing throw­away math and run­ning it. Then I de­cided to give it a lit­tle more rope. I pasted a tem­po­rary Brave Search API key into the con­ver­sa­tion, mostly to see what would hap­pen. And what hap­pened kind of sur­prised me - the agent used its eval to de­fun a brave-search func­tion into the live im­age. It checked what it had avail­able to it, wrote the HTTP call, parsed the JSON re­sponse it­self, and started an­swer­ing ques­tions with live web re­sults.

I never built a web search tool. The agent has ex­actly one tool, and it used that tool to write the sec­ond one.

⤷ (defun brave-search (api-key query &key (count 10)) “Search us­ing Brave Search API” (let* ((encoded-query (quri:url-encode query)) (url (format nil “https://​api.search.brave.com/​res/​v1/​web/​search?q=~a&count=~d” en­coded-query count)) (response (dexador:get url :headers `((“Accept” . “application/json”) (“X-Subscription-Token” . ,api-key))))) (shasht:read-json re­sponse))) => BRAVE-SEARCH Perfect! You were ab­solutely right, Jamie. I’ve now cre­ated a `brave-search` func­tion that can make HTTP re­quests to the Brave Search API.

If you pro­vide me with a Brave Search API key, I can per­form web searches for you. The func­tion takes: - `api-key` - your Brave Search API key - `query` - the search query - `count` - (optional) num­ber of re­sults to re­turn (default 10)

Would you like to try it out with your API key? NIL

Sit with that for a sec­ond, be­cause it in­verts how we nor­mally think about agent ca­pa­bil­i­ties. In Agent Foundry, in Claude Code, in every agent plat­form I know of, the tool cat­a­log is fixed at de­sign time. Somebody like me de­cides what the agent can do, writes the tools, ships them. Here the cat­a­log is open. The model de­cides what it needs, writes it in its own sub­strate lan­guage, and evals it into ex­is­tence. Code is data, data is code, and ca­pa­bil­ity is just con­ver­sa­tion.

And there’s a twist that took me a minute to ap­pre­ci­ate. The de­fun it­self dies when the process ex­its. Functions live in the run­ning Lisp im­age, not on disk. But the tran­script per­sists. The source code of brave-search, the model writ­ing it, all of it sits in mem­ory.json as plain text. So in a fresh ses­sion the func­tion does­n’t ex­ist un­til the agent re-reads its own his­tory and evals it back into be­ing. Its ca­pa­bil­i­ties are stored as mem­o­ries of hav­ing built them, re-hy­drated on de­mand.

Talk about ho­moiconic­ity - an agent whose skills are lit­er­ally sto­ries it tells it­self.

Skills are just mem­o­ries

Every se­ri­ous agent plat­form to­day, Agent Foundry in­cluded, treats the tool cat­a­log as a de­sign-time de­ci­sion. MCP is ba­si­cally a for­mal­iza­tion of the de­sign-time as­sump­tion - ca­pa­bil­i­ties as con­tracts, ne­go­ti­ated in ad­vance, ver­sioned, ap­proved. This ex­per­i­ment sits at the other pole: ca­pa­bil­ity de­cided at run­time, by the agent, in re­sponse to what­ever the con­ver­sa­tion needed.

Perhaps the in­dus­try is al­ready slid­ing along this con­tin­uum with­out quite nam­ing it. Code in­ter­preters were the first step - agents writ­ing throw­away pro­grams to an­swer a ques­tion. Skills, the SKILL.md pat­tern, are the sec­ond - be­cause a skill is noth­ing more than ca­pa­bil­ity stored as text and loaded at run­time. This 100 line Lisp agent just col­lapses the re­main­ing dis­tance. Its skills are stored in its own tran­script.

And kind of stretch­ing maybe, that’s roughly how we work too. Almost noth­ing you can do was in­stalled at de­sign time. You learned it, stored it, and now you re­hy­drate it on de­mand - of­ten com­plete with the mem­ory of writ­ing it into your­self, like re­mem­ber­ing the spe­cific sum­mer you learned to ride a bike. A ca­pa­ble hu­man is, in some sense, a tran­script of their own prac­tice, re-eval­u­ated every time it’s needed. Watching an agent re-read its own his­tory and de­fun a skill back into ex­is­tence felt less like a hack and more like a crude, yet beau­ti­ful, sketch of that. I’m spec­u­lat­ing well past my ex­per­tise here, but the shape rhymes.

He was­n’t wrong, he was early

Symbolic AI lost. Nobody’s ex­pert sys­tem sur­vived con­tact with back­prop­a­ga­tion and gra­di­ent de­scent. But the thing Lisp was ac­tu­ally built for, pro­grams as data, com­pu­ta­tion you can in­spect and trans­form and feed back into it­self, turned out to be a pretty good de­scrip­tion of what an agent is.

The mod­els do the rea­son­ing now. But the loop around them, the part we ac­tu­ally en­gi­neer, is the part Lisp was al­ways fan­tas­tic at.

25 years is a long time to be right about some­thing. Somewhere out there maybe my old AI prof is ahead of us on some­thing else.

Full source, Dockerfile in­cluded, is on GitHub: https://​github.com/​jamiebeach/​lisp-agent

Table of Contents

1. Quick Start

1.1. Shell in­te­gra­tion at a glance 1.2. Input modes at a glance

1.1. Shell in­te­gra­tion at a glance

1.2. Input modes at a glance

2. Requirements

3. Installation

3.1. MELPA 3.2. use-pack­age with :vc (Emacs 30+) 3.3. use-pack­age with :load-path 3.4. Manual 3.5. Native mod­ule 3.6. Platform notes

3.6.1. Windows

3.1. MELPA

3.2. use-pack­age with :vc (Emacs 30+)

3.3. use-pack­age with :load-path

3.4. Manual

3.5. Native mod­ule

3.6. Platform notes

3.6.1. Windows

3.6.1. Windows

4. Building from source

4.1. Bundled ter­minfo

4.1. Bundled ter­minfo

5. Shell in­te­gra­tion

6. Input modes

6.1. Mode-switch key­bind­ings 6.2. Semi-char mode (default) 6.3. Char mode 6.4. Emacs mode 6.5. Copy mode 6.6. Mouse se­lec­tion 6.7. Line mode 6.8. Scrollback search out­side copy mode

6.1. Mode-switch key­bind­ings

6.2. Semi-char mode (default)

6.3. Char mode

6.4. Emacs mode

6.5. Copy mode

6.6. Mouse se­lec­tion

6.7. Line mode

6.8. Scrollback search out­side copy mode

7. Features

7.1. Terminal em­u­la­tion 7.2. Process model 7.3. Bookmarks 7.4. Links and file de­tec­tion 7.5. Clipboard 7.6. Input 7.7. Password prompt de­tec­tion 7.8. Shell in­te­gra­tion fea­tures 7.9. Rendering 7.10. Inline im­ages (Kitty graph­ics pro­to­col)

7.10.1. Limitations

7.11. Calling Elisp from the shell 7.12. Notifications and progress 7.13. Color palette

7.1. Terminal em­u­la­tion

7.2. Process model

7.3. Bookmarks

7.4. Links and file de­tec­tion

7.5. Clipboard

7.6. Input

7.7. Password prompt de­tec­tion

7.8. Shell in­te­gra­tion fea­tures

7.9. Rendering

7.10. Inline im­ages (Kitty graph­ics pro­to­col)

7.10.1. Limitations

7.10.1. Limitations

7.11. Calling Elisp from the shell

7.12. Notifications and progress

7.13. Color palette

8. TRAMP (Remote Terminals)

8.1. Remote shell in­te­gra­tion

8.1.1. Option 1: Automatic in­jec­tion (recommended for con­ve­nience) 8.1.2. Option 2: Manual setup (recommended for per­ma­nent re­mote hosts)

8.2. Remote xterm-ghostty ter­minfo

8.2.1. TRAMP-launched ghos­tel 8.2.2. Outbound ssh from a lo­cal ghos­tel buffer 8.2.3. Manual in­stall (no auto-ma­chin­ery) 8.2.4. Drop the Ghostty ad­ver­tise­ment en­tirely

8.1. Remote shell in­te­gra­tion

8.1.1. Option 1: Automatic in­jec­tion (recommended for con­ve­nience) 8.1.2. Option 2: Manual setup (recommended for per­ma­nent re­mote hosts)

8.1.1. Option 1: Automatic in­jec­tion (recommended for con­ve­nience)

8.1.2. Option 2: Manual setup (recommended for per­ma­nent re­mote hosts)

8.2. Remote xterm-ghostty ter­minfo

8.2.1. TRAMP-launched ghos­tel 8.2.2. Outbound ssh from a lo­cal ghos­tel buffer 8.2.3. Manual in­stall (no auto-ma­chin­ery) 8.2.4. Drop the Ghostty ad­ver­tise­ment en­tirely

8.2.1. TRAMP-launched ghos­tel

8.2.2. Outbound ssh from a lo­cal ghos­tel buffer

8.2.3. Manual in­stall (no auto-ma­chin­ery)

8.2.4. Drop the Ghostty ad­ver­tise­ment en­tirely

9. Configuration

9.1. Process and en­vi­ron­ment 9.2. Native mod­ule 9.3. TRAMP and re­mote 9.4. Rendering and per­for­mance 9.5. Images 9.6. Links, clip­board, and de­tec­tion 9.7. Password prompts 9.8. Notifications and progress 9.9. Input and in­ter­ac­tion 9.10. Line mode

9.1. Process and en­vi­ron­ment

9.2. Native mod­ule

9.3. TRAMP and re­mote

9.4. Rendering and per­for­mance

9.5. Images

9.6. Links, clip­board, and de­tec­tion

9.7. Password prompts

9.8. Notifications and progress

9.9. Input and in­ter­ac­tion

9.10. Line mode

10. Extensions

10.1. Evil-mode 10.2. Compilation mode

10.2.1. Live mode switch­ing 10.2.2. Keybindings (ghostel-compile-view-mode, also ac­tive dur­ing a read-only run) 10.2.3. Make com­pile / re­com­pile / pro­ject-com­pile use ghos­tel 10.2.4. Hooks for your own in­te­gra­tions

10.3. Eshell in­te­gra­tion 10.4. Comint in­te­gra­tion 10.5. Emacs Lisp in­put meth­ods

10.1. Evil-mode

10.2. Compilation mode

10.2.1. Live mode switch­ing 10.2.2. Keybindings (ghostel-compile-view-mode, also ac­tive dur­ing a read-only run) 10.2.3. Make com­pile / re­com­pile / pro­ject-com­pile use ghos­tel 10.2.4. Hooks for your own in­te­gra­tions

10.2.1. Live mode switch­ing

10.2.2. Keybindings (ghostel-compile-view-mode, also ac­tive dur­ing a read-only run)

10.2.3. Make com­pile / re­com­pile / pro­ject-com­pile use ghos­tel

10.2.4. Hooks for your own in­te­gra­tions

10.3. Eshell in­te­gra­tion

10.4. Comint in­te­gra­tion

10.5. Emacs Lisp in­put meth­ods

11. Commands

11.1. Sending in­put from Lisp 11.2. Project in­te­gra­tion

11.1. Sending in­put from Lisp

11.2. Project in­te­gra­tion

12. Running tests

13. Performance

13.1. Native vs Emacs PTY 13.2. Burst ab­sorp­tion (cat a 10 MB file) 13.3. Typing la­tency

13.1. Native vs Emacs PTY

13.2. Burst ab­sorp­tion (cat a 10 MB file)

RISCBoy is an open-source portable games con­sole, de­signed from scratch. This in­cludes:

A RISC-V com­pat­i­ble CPU

A raster graph­ics pipeline and dis­play con­troller

Other chip in­fra­struc­ture: bus­fab­ric, mem­ory con­trollers, UART, GPIO etc.

A PCB lay­out in KiCad

It is a Gameboy Advance from a par­al­lel uni­verse where RISC-V ex­isted in 2001. A love let­ter to the hand­held con­soles from my child­hood, and a 3AM drunk text to the tech­nol­ogy that pow­ered them.

The de­sign is writ­ten in syn­the­sis­able Verilog 2005, and is in­tended to fit onto an iCE40-HX8k FPGA. This is a LUT4-based FPGA with 7680 logic el­e­ments, so fit­ting a 32 bit games con­sole re­quires a crow­bar and some vase­line, or per­haps just care­ful de­sign. The HX8k was once the largest FPGA tar­geted by the open-source Icestorm FPGA tool­chain, but that tool­chain has since moved on to greater things.

More de­tailed in­for­ma­tion can be found in the doc­u­men­ta­tion.

The proces­sor sup­ports the RV32IMC in­struc­tion set, and passes the RISC-V com­pli­ance suite for these in­struc­tions, as well as the riscv-for­mal ver­i­fi­ca­tion suite, and some of my own for­mal prop­erty checks for in­struc­tion fron­tend con­sis­tency and ba­sic bus com­pli­ance. It also sup­ports M-mode CSRs, ex­cep­tions, and a sim­ple com­pli­ant ex­ten­sion for vec­tored ex­ter­nal in­ter­rupts.

Cloning

This repos­i­tory uses sub­mod­ules for HDL as well as tests

git clone –recursive https://​github.com/​Wren6991/​RISCBoy.git riscboy

Alternatively

git clone https://​github.com/​Wren6991/​RISCBoy.git riscboy cd riscboy git sub­mod­ule up­date –init –recursive

Note a re­cur­sive sub­mod­ule up­date is re­quired to run the proces­sor’s stand­alone tests. This is not nec­es­sary for build­ing RISCBoy gate­ware.

Building RV32IMC Toolchain

The RV32IMC tool­chain is re­quired for com­pi­la­tion of soft­ware-based tests. Follow the in­struc­tions on the RISC-V GNU Toolchain GitHub, ex­cept for the con­fig­ure line:

# Prerequisites for Ubuntu 20.04 sudo apt in­stall -y au­to­conf au­tomake au­to­tools-dev curl python3 libmpc-dev libmpfr-dev libgmp-dev gawk build-es­sen­tial bi­son flex tex­info gperf libtool patchutils bc zlib1g-dev lib­ex­pat-dev cd /tmp git clone –recursive https://​github.com/​riscv/​riscv-gnu-tool­chain cd riscv-gnu-tool­chain # The ./configure ar­gu­ments are the most im­por­tant dif­fer­ence ./configure –prefix=/opt/riscv –with-arch=rv32imc –with-abi=ilp32 –with-multilib-generator=“rv32i-ilp32–;rv32ic-ilp32–;rv32im-ilp32–;rv32imc-ilp32–” sudo mkdir /opt/riscv sudo chown $(whoami) /opt/riscv make -j $(nproc)

On smaller FPGAs, like the iCE40 UP5k, RISCBoy may be con­fig­ured to use a smaller RV32I vari­ant of the proces­sor, rather than the higher-per­for­mance RV32IMC ver­sion. The com­piler will sup­port any of the ISA vari­ants avail­able on RISCBoy, but we must also in­struct the tool­chain build scripts to pro­duce stan­dard li­braries for these vari­ants, via the –with-multilib ar­gu­ments. Running a RV32I ex­e­cutable linked against an RV32IMC stan­dard li­brary on an RV32I-only proces­sor will ruin your day!

Simulation

The sim­u­la­tion flow is dri­ven by Xilinx ISIM 14.x; make­files are found in the scripts/ folder. This has only been tested with the Linux ver­sion of ISIM.

You will also need to check­out the RISC-V com­pli­ance suite in or­der to run these tests (note the — test is re­quired to stop git from look­ing in the KiCad di­rec­to­ries and com­plain­ing about the li­brary struc­ture there).

$ git sub­mod­ule up­date –init –recursive

Once this is ready, you should be able to run the fol­low­ing:

. sourceme cd test ./runtests

which will run all of the HDL-level tests. Software tests will re­quire the RV32IC tool­chain. You may need to ad­just some of the paths in sourceme if ISIM is in­stalled in a non-de­fault lo­ca­tion. To graph­i­cally de­bug a test, run its make­file di­rectly:

cd sys­tem make TEST=helloworld gui

PCB

The im­age shows the Rev A PCB. It is com­pat­i­ble with iTead’s 4-layer 5x5 cm pro­to­typ­ing ser­vice, which cur­rently costs $65 for 10 boards.

The schematic can be viewed here (pdf)

Rev B will look quite dif­fer­ent; I am wait­ing for the gate­ware and boot­loader to ma­ture be­fore pro­ceed­ing. My cur­rent dev hard­ware looks a lot like my Snowflake FPGA board.

Synthesis

FPGA syn­the­sis for iCE40 uses an open-source tool­chain. If you would like to build this pro­ject us­ing the ex­ist­ing make­files, you will first need to build the tool­chain I used:

Yosys for syn­the­sis

nextpnr for place and route

Project Icestorm for bit­stream gen­er­a­tion

Note that I have only built these on Linux. I’ve heard it is pos­si­ble to build these on Windows, but haven’t tried it. However, they can be built on a Raspberry Pi, which is neat.

Once the tool­chain is in place, run

. sourceme cd synth make -f HX8k-EVN.mk bit

to gen­er­ate an FPGA im­age suit­able for Lattice HX8k eval­u­a­tion board.

There is also highly ex­per­i­men­tal sup­port (i.e. not my main dev plat­form) for ECP5, with board files for the Lattice LEF5UM5G-85F-EVN eval­u­a­tion board:

make -f ECP5-EVN.mk BUILD=full bit

This build re­places the ex­ter­nal, 512 kiB, 16 bit wide SRAM of RISCBoy de­vel­op­ment hard­ware with an in­ter­nal, 256 kiB, 32 bit wide syn­chro­nous mem­ory, which Trellis builds out of ECP5 sys­mem blocks.

Directory Structure

board: KiCad files for main RISCBoy PCB and other small boards used dur­ing de­vel­op­ment

doc: LaTeX source and di­a­grams for doc­u­men­ta­tion, and the most re­cently built PDF

hdl: The Verilog source for RISCBoy gate­ware.

bus­fab­ric: AHB-lite cross­bar and APB pe­riph­eral fab­ric graph­ics: Source for the pixel pro­cess­ing unit haz­ard5: Source for the RISC-V proces­sor. This is com­pletely self-con­tained. mem: Memory con­trollers, and in­fer­ence/​in­jec­tion wrap­pers and mod­els for the mem­o­ries them­selves peris: Small pe­riph­er­als such as UART, SPI, PWM riscboy_­core: Structural mod­ule to in­stan­ti­ate and con­nect the com­po­nents that com­prise RISCBoy riscboy_f­pga: Top-level wrap­pers for a few dif­fer­ent FPGAs and boards: con­nect up IOs, pro­vide clock and re­set

bus­fab­ric: AHB-lite cross­bar and APB pe­riph­eral fab­ric

graph­ics: Source for the pixel pro­cess­ing unit

haz­ard5: Source for the RISC-V proces­sor. This is com­pletely self-con­tained.

mem: Memory con­trollers, and in­fer­ence/​in­jec­tion wrap­pers and mod­els for the mem­o­ries them­selves

peris: Small pe­riph­er­als such as UART, SPI, PWM

riscboy_­core: Structural mod­ule to in­stan­ti­ate and con­nect the com­po­nents that com­prise RISCBoy

riscboy_f­pga: Top-level wrap­pers for a few dif­fer­ent FPGAs and boards: con­nect up IOs, pro­vide clock and re­set

ref­er­ence: a few PDFs for stan­dards used in RISCBoy, e.g. the RISC-V in­struc­tion set

scripts: Junk that I can’t put any­where else

soft­ware: Loose col­lec­tion of C files that are used for sys­tem-level tests. Not re­ally a use­ful soft­ware tree yet.

synth: Working di­rec­tory for run­ning whole-sys­tem syn­the­sis. Top-level make­files, pin con­straint files.

test: Regression tests. Some are Verilog test­benches, oth­ers are soft­ware test­cases that run on sim­u­la­tions of the proces­sor or the full sys­tem.

I’m a mil­lion times more likely to send an email or a text than to pick up the phone. But I had a ques­tion I thought was worth an ac­tual call, so I sched­uled one with some­one se­nior enough to have real scar tis­sue, the kind you only get from watch­ing a de­ci­sion go side­ways in a board­room. I asked him where he’d look, per­son­ally, for the an­swer to a hard ques­tion I was chas­ing, one with­out in­dus­try con­sen­sus. Not what the text­book says. If five stud­ies con­flicted, which would he trust? I wanted the thing 30 years had taught him that a search en­gine could­n’t.

“Honestly? Ask Claude.”

That stung a lit­tle, but it was­n’t the first time I’d heard it. Once it was a data prob­lem I’d been stuck on. I’d ap­proached it a half dozen dif­fer­ent ways and could ex­plain in some de­tail why none of them had worked. I reached out to a few peo­ple who do this kind of thing for a liv­ing, peo­ple I text with reg­u­larly, trad­ing ques­tions and work­ing through what­ever we’re stuck on. All but one gave me the same redi­rect.

Each time this hap­pened, I had al­ready asked Claude. That was­n’t the step I’d skipped. Before I ever reached out to a per­son, I’d spent a cou­ple of hours (and some­times way too many to­kens) go­ing back and forth with a large lan­guage model, and I still had a ques­tion that had sur­vived all of that.

I’m old enough to re­mem­ber peo­ple send­ing LMGTFY links to folks who did­n’t seem to know how to use a search en­gine and ex­pected strangers to do un­paid re­search for them. But this is­n’t that. It’s closer to what hap­pens when I ask a friend for a food rec­om­men­da­tion and get a top-10 list back. I’m not ask­ing what Eater thinks is the best kind-of-quiet spot for late-night drinks, or for a great cof­fee shop in the city where they used to live. I’m ask­ing what they think, be­cause we have sim­i­lar taste and a shared his­tory, and be­cause I know they have opin­ions about where the lists go wrong. I trust their ex­pe­ri­ence over the ex­pert con­sen­sus.

It’s pos­si­ble “ask the model” has be­come the po­lite ver­sion of “I don’t know,” or “I don’t have time for this right now,” or “I’d have to think about it.” Maybe it’s an easy way to de­cline giv­ing an an­swer. But I’d take al­most any­thing over the redi­rect. “I’m busy” is a real an­swer. “I can’t think of any­thing you haven’t al­ready tried” is an an­swer too. What “ask Claude” does­n’t give me is the per­son’s spe­cific, lived ex­pe­ri­ence. That’s the thing that’s hard to write down and even harder to search for.

There’s a real cost to be­ing the per­son other peo­ple call, and it’s not fair to ex­pect every­one to bear it. It takes close at­ten­tion and ac­tual thought, and not every­one has that to spare on a day full of dead­lines and fires to put out. Plenty of ques­tions re­ally can be an­swered by an LLM or a search en­gine. But when the ques­tion is one that al­ready sur­vived the model, “ask Claude” does­n’t save any­one a step. It just with­holds the thought­ful an­swer decades of ex­pe­ri­ence could have given.

Tara Anand for NPR

Catch up on what you missed with our Education newslet­ter, sent weekly.

This month, the U.S. Department of Education be­gan rolling out a new ac­count­abil­ity test that most col­leges and uni­ver­si­ties will soon have to pass.

The test it­self is sim­ple: If an un­der­grad­u­ate pro­gram’s grad­u­ates don’t earn more than work­ers who never went to col­lege, that pro­gram could be cut off from fed­eral stu­dent loans. The same goes for any grad­u­ate pro­gram whose grad­u­ates earn less than some­one with only a bach­e­lor’s de­gree.

“If a pro­gram can­not show that it leaves its grad­u­ates fi­nan­cially bet­ter off than if they had never en­rolled, it should not be un­der­writ­ten by fed­eral tax­pay­ers,” said Under Secretary of Education Nicholas Kent in a re­cent state­ment.

But this new test, known as “do no harm,” raises some thorny ques­tions about the pur­pose of col­lege. Like: Is it just about mak­ing more money?

Some ad­vo­cates for post­sec­ondary arts ed­u­ca­tion think not.

“Earnings is only a small piece of that puz­zle,” said Lee Ann Scotto Adams, ex­ec­u­tive di­rec­tor of the Strategic National Arts Alumni Project (SNAAP), a non­profit that stud­ies the ca­reers of arts grad­u­ates.

She and Doug Dempster, the pres­i­dent of SNAAP, worry the new test might lead col­leges and uni­ver­si­ties to pre­emp­tively slash low-earn­ing cre­ative arts pro­grams in mu­sic, the­ater, stu­dio art and de­sign. Dempster says that could lead to a fur­ther de­valu­ing of jobs that are crit­i­cal to a well-func­tion­ing so­ci­ety.

“We know we need nurses. We know we need jour­nal­ists. We know we need early child­hood ed­u­ca­tors,” he said. “We don’t know how many artists we need, but I can guar­an­tee that if you elim­i­nate ac­cess, we will im­pov­er­ish our cul­tural life na­tion­ally.”

How the new stan­dard will work

The new earn­ings test comes cour­tesy of last year’s One Big Beautiful Bill Act, which in­cluded a slew of big higher ed­u­ca­tion pol­icy changes meant to ad­dress ris­ing con­cerns over the cost and value of col­lege.

Higher ed­u­ca­tion ex­perts across the po­lit­i­cal spec­trum told NPR the test sets a pretty rea­son­able ex­pec­ta­tion: In many states, fed­eral data shows, grad­u­ates of bach­e­lor pro­grams will have to earn a min­i­mum of about $30,000 to $41,000 a year for their pro­gram to pass.

“This is re­ally a very low floor,” said Christopher Madaio, a se­nior ad­viser at the non­profit The Institute for College Access & Success. “I mean, high school earn­ings is not an ex­ceed­ingly high met­ric for a pro­gram to meet.”

Programs fail the test when they don’t meet the earn­ings re­quire­ment for two out of three con­sec­u­tive years.

The cur­rent test does not take stu­dent loan debt into ac­count, which means there’s no way to dis­tin­guish be­tween a grad­u­ate who is strug­gling with low pay while be­ing debt-free and a grad­u­ate who is strug­gling with low pay while also pay­ing off tens of thou­sands of dol­lars in loans.

The Education Department says it will be­gin cal­cu­lat­ing the first year of grad­u­ate earn­ings in early 2027, and “some pro­grams could be des­ig­nated as low-earn­ing out­come pro­grams be­gin­ning in the 2028 – 2029 [financial aid] award year.”

The kinds of pro­grams that are likely to fail

According to Education Department es­ti­mates, the vast ma­jor­ity of un­der­grad­u­ate and grad­u­ate pro­grams should eas­ily pass the new earn­ings test.

But more than 800,000 stu­dents at­tend a pro­gram that would likely fail the mea­sure, ac­cord­ing to de­part­ment data. Roughly half of those stu­dents are en­rolled in for-profit schools, which al­ready have a rep­u­ta­tion for short­chang­ing stu­dents.

Other take­aways from the de­part­men­t’s data:

About 18% of un­der­grad­u­ate cer­tifi­cate pro­grams, which of­ten bill them­selves as ca­reer-fo­cused fast tracks, would fail the earn­ings test. Specifically, cer­tifi­cate pro­grams in cos­me­tol­ogy and so­matic body work have the high­est pre­dicted fail­ure rates.

Two-year as­so­ci­ate de­gree pro­grams have the next high­est fail­ure rate, at 6%. Associate pro­grams that train spe­cial­ized ed­u­ca­tors, in­clud­ing early child­hood ed­u­ca­tors, are the most likely to fail.

Most tra­di­tional, four-year bach­e­lor pro­grams fare well, with roughly 1% fail­ing the earn­ings test. When these pro­grams do fail, it’s of­ten in ar­eas like the­ater, mu­sic and stu­dio art.

About 4% of mas­ter’s de­gree pro­grams would fail, with the high­est fail­ure rates for pro­grams teach­ing men­tal and so­cial health ser­vices.

For one mu­sic teacher, it was “never about the money”

Some of the United States’ most pres­ti­gious mu­sic pro­grams — known for train­ing the coun­try’s most tal­ented young mu­si­cians — are among the 14% of bach­e­lor mu­sic pro­grams pre­dicted to fail the new earn­ings test, ac­cord­ing to Education Department data. That in­cludes The Juilliard School in New York City, the New England Conservatory in Boston and Indiana University Bloomington’s Jacobs School of Music.

The un­der­grad­u­ate mu­sic pro­gram that Cindy Flores at­tended at Portland State University (PSU) also would­n’t pass. Flores teaches mari­achi mu­sic to mid­dle and high school stu­dents at Salem-Keizer Public Schools in Oregon’s Willamette Valley.

Cindy Flores smiles as she teaches mari­achi to stu­dents at McKay High School in Salem, Oregon. “If it was­n’t for PSU and the loans I could get … I would­n’t be a Mexican American mari­achi teacher for my Mexican American stu­dents,” she said.

Eli Imadali/OPB

hide cap­tion

tog­gle cap­tion

Eli Imadali/OPB

Her path to be­com­ing a full-time mu­sic teacher started with study­ing mu­sic ed­u­ca­tion at PSU; then she got an ed­u­ca­tors li­cense from Western Oregon University — and she used fed­eral stu­dent loans to help pay for all of it.

She now holds close to $55,000 in fed­eral stu­dent loan debt.

Flores said she would­n’t be where she is now with­out that ac­cess to fed­eral aid.

“If it was­n’t for PSU and the loans I could get … I would­n’t be a Mexican American mari­achi teacher for my Mexican American stu­dents,” she said.

But given the new fed­eral test, fu­ture PSU mu­sic stu­dents might not have the same ac­cess to fed­eral stu­dent loans that Flores did.

She said she feels lucky to have found a job that she’s pas­sion­ate about and that pays a liv­ing wage. But, for her, a ca­reer in mu­sic was about much more than a pay­check.

“It is never about the money,” she said. “I re­al­ized I wanted to have a ca­reer in mu­sic when I was in the eighth grade, be­cause every mu­sic teacher I had were such good role mod­els in my life and I wanted to be part of that com­mu­nity.”

Defining suc­cess in the arts

SNAAP’s Lee Ann Scotto Adams said the fed­eral gov­ern­men­t’s one-size-fits-all ac­count­abil­ity ap­proach does­n’t make sense for stu­dents grad­u­at­ing from cre­ative arts pro­grams be­cause wages aren’t the only mea­sure of suc­cess for stu­dio artists, mu­si­cians and de­sign­ers.

“Yes, you need to earn money to make a liv­ing, but we see our cre­ative work­ers want the abil­ity to have in­de­pen­dence in their work. They want jobs that are so­cially con­scious. They want to make an im­pact cul­tur­ally,” Scotto Adams said. “These are all met­rics that fall out­side of just straight­for­ward earn­ings met­rics.”

She also takes is­sue with look­ing at earn­ings in the first few years af­ter grad­u­a­tion. Scotto Adams points to SNAAP sur­vey data that shows arts grad­u­ates of­ten have un­pre­dictable in­comes at the be­gin­ning of their ca­reers, but their pay tends to sta­bi­lize and in­crease over time.

“Looking at earn­ings as the sole met­ric of suc­cess is very lim­ited, and that’s be­cause artists have non­lin­ear ca­reers,” Scotto Adams said. “For the most part, peo­ple who grad­u­ate from these pro­grams move into ca­reers that they’re per­son­ally sat­is­fied with.”

Students con­sid­er­ing any of the at-risk pro­grams won’t im­me­di­ately lose ac­cess to fed­eral aid. While the ac­count­abil­ity test is be­ing rolled out this month, its im­ple­men­ta­tion will be phased in over the next cou­ple of years.