Language mod­els to­day, while use­ful for a va­ri­ety of tasks, are still lim­ited. The only in­for­ma­tion they can learn from is their train­ing data. This in­for­ma­tion can be out-of-date and is one-size fits all across ap­pli­ca­tions. Furthermore, the only thing lan­guage mod­els can do out-of-the-box is emit text. This text can con­tain use­ful in­struc­tions, but to ac­tu­ally fol­low these in­struc­tions you need an­other process.

Though not a per­fect anal­ogy, plu­g­ins can be “eyes and ears” for lan­guage mod­els, giv­ing them ac­cess to in­for­ma­tion that is too re­cent, too per­sonal, or too spe­cific to be in­cluded in the train­ing data. In re­sponse to a user’s ex­plicit re­quest, plu­g­ins can also en­able lan­guage mod­els to per­form safe, con­strained ac­tions on their be­half, in­creas­ing the use­ful­ness of the sys­tem over­all.

We ex­pect that open stan­dards will emerge to unify the ways in which ap­pli­ca­tions ex­pose an AI-facing in­ter­face. We are work­ing on an early at­tempt at what such a stan­dard might look like, and we’re look­ing for feed­back from de­vel­op­ers in­ter­ested in build­ing with us.

Today, we’re be­gin­ning to grad­u­ally en­able ex­ist­ing plu­g­ins from our early col­lab­o­ra­tors for ChatGPT users, be­gin­ning with ChatGPT Plus sub­scribers. We’re also be­gin­ning to roll out the abil­ity for de­vel­op­ers to cre­ate their own plu­g­ins for ChatGPT.

In the com­ing months, as we learn from de­ploy­ment and con­tinue to im­prove our safety sys­tems, we’ll it­er­ate on this pro­to­col, and we plan to en­able de­vel­op­ers us­ing OpenAI mod­els to in­te­grate plu­g­ins into their own ap­pli­ca­tions be­yond ChatGPT.

/ Sign up for Verge Deals to get deals on prod­ucts we’ve tested sent to your in­box daily.

Now avail­able with AMD Ryzen™ 7040 Series and 13th Gen Intel® Core™

Performance up­grades for your Framework Laptop 13, with the lat­est Intel and AMD proces­sor op­tions.

Higher-capacity bat­ter­ies, im­proved hinges, matte dis­plays, and more for your Framework Laptop 13.

Get the lat­est news and prod­uct up­dates from Framework

“The team over at Framework has man­aged to not just cre­ate a lap­top that is eas­ily re­pairable and upgrad­able, but it‘s also a thin, gor­geous, per­for­mant lap­top.” — Linus Tech Tips

“This is the best lap­top you can get right now if you want a com­pletely re­pairable and up­grade­able de­vice.” — Dave2D

Best of the Best Design Award

The time has come for con­sumer elec­tron­ics prod­ucts that are de­signed to last: prod­ucts that give you back the power to up­grade, cus­tomize, and re­pair them. We’re ex­cited for the op­por­tu­nity to fix the con­sumer elec­tron­ics in­dus­try to­gether.

At our Next Level Event, we launched a colos­sal set of new prod­ucts and up­grades, in­clud­ing the new, high-per­for­mance 16” Framework Laptop 16 and both Intel and AMD-powered Framework Laptop 13. We can’t wait to see what you think!

Early in January I wrote about the pos­si­bil­ity of con­nect­ing ChatGPT to Wolfram|Alpha. And to­day—just two and a half months later—I’m ex­cited to an­nounce that it’s hap­pened! Thanks to some heroic soft­ware en­gi­neer­ing by our team and by OpenAI, ChatGPT can now call on Wolfram|Alpha—and Wolfram Language as well—to give it what we might think of as “computational su­per­pow­ers”. It’s still very early days for all of this, but it’s al­ready very im­pres­sive—and one can be­gin to see how amaz­ingly pow­er­ful (and per­haps even rev­o­lu­tion­ary) what we can call “” can be.

Back in January, I made the point that, as an LLM neural net, ChatGPT—for all its re­mark­able prowess in tex­tu­ally gen­er­at­ing ma­te­r­ial “like” what it’s read from the web, etc.—can’t it­self be ex­pected to do ac­tual non­triv­ial com­pu­ta­tions, or to sys­tem­at­i­cally pro­duce cor­rect (rather than just “looks roughly right”) data, etc. But when it’s con­nected to the Wolfram plu­gin it can do these things. So here’s my (very sim­ple) first ex­am­ple from January, but now done by ChatGPT with “Wolfram su­per­pow­ers” in­stalled:

It’s a cor­rect re­sult (which in January it was­n’t)—found by ac­tual com­pu­ta­tion. And here’s a bonus: im­me­di­ate vi­su­al­iza­tion:

How did this work? Under the hood, ChatGPT is for­mu­lat­ing a query for Wolfram|Alpha—then send­ing it to Wolfram|Alpha for com­pu­ta­tion, and then “deciding what to say” based on read­ing the re­sults it got back. You can see this back and forth by click­ing the “Used Wolfram” box (and by look­ing at this you can check that ChatGPT did­n’t “make any­thing up”):

There are lots of non­triv­ial things go­ing on here, on both the ChatGPT and Wolfram|Alpha sides. But the up­shot is a good, cor­rect re­sult, knit­ted into a nice, flow­ing piece of text.

Let’s try an­other ex­am­ple, also from what I wrote in January:

A fine re­sult, wor­thy of our tech­nol­ogy. And again, we can get a bonus:

In January, I noted that ChatGPT ended up just “making up” plau­si­ble (but wrong) data when given this prompt:

But now it calls the Wolfram plu­gin and gets a good, au­thor­i­ta­tive an­swer. And, as a bonus, we can also make a vi­su­al­iza­tion:

Another ex­am­ple from back in January that now comes out cor­rectly is:

If you ac­tu­ally try these ex­am­ples, don’t be sur­prised if they work dif­fer­ently (sometimes bet­ter, some­times worse) from what I’m show­ing here. Since ChatGPT uses ran­dom­ness in gen­er­at­ing its re­sponses, dif­fer­ent things can hap­pen even when you ask it the ex­act same ques­tion (even in a fresh ses­sion). It feels “very hu­man”. But dif­fer­ent from the solid “right-answer-and-it-doesn’t-change-if-you-ask-it-again” ex­pe­ri­ence that one gets in Wolfram|Alpha and Wolfram Language.

Here’s an ex­am­ple where we saw ChatGPT (rather im­pres­sively) “having a con­ver­sa­tion” with the Wolfram plu­gin, af­ter at first find­ing out that it got the “wrong Mercury”:

One par­tic­u­larly sig­nif­i­cant thing here is that ChatGPT is­n’t just us­ing us to do a “dead-end” op­er­a­tion like show the con­tent of a web­page. Rather, we’re act­ing much more like a true “brain im­plant” for ChatGPT—where it asks us things when­ever it needs to, and we give re­sponses that it can weave back into what­ever it’s do­ing. It’s rather im­pres­sive to see in ac­tion. And—although there’s def­i­nitely much more pol­ish­ing to be done—what’s al­ready there goes a long way to­wards (among other things) giv­ing ChatGPT the abil­ity to de­liver ac­cu­rate, cu­rated knowl­edge and data—as well as cor­rect, non­triv­ial com­pu­ta­tions.

But there’s more too. We al­ready saw ex­am­ples where we were able to pro­vide cus­tom-cre­ated vi­su­al­iza­tions to ChatGPT. And with our com­pu­ta­tion ca­pa­bil­i­ties we’re rou­tinely able to make “truly orig­i­nal” con­tent—com­pu­ta­tions that have sim­ply never been done be­fore. And there’s some­thing else: while “pure ChatGPT” is re­stricted to things it “learned dur­ing its train­ing”, by call­ing us it can get up-to-the-mo­ment data.

This can be based on our real-time data feeds (here we’re get­ting called twice; once for each place):

Or it can be based on “science-style” pre­dic­tive com­pu­ta­tions:

And now (almost) all of this is ac­ces­si­ble to ChatGPT—opening up a tremen­dous breadth and depth of new pos­si­bil­i­ties. And to give some sense of these, here are a few (simple) ex­am­ples:

ChatGPT is built to be able to have back-and-forth con­ver­sa­tion with hu­mans. But what can one do when that con­ver­sa­tion has ac­tual com­pu­ta­tion and com­pu­ta­tional knowl­edge in it? Here’s an ex­am­ple. Start by ask­ing a “world knowl­edge” ques­tion:

And, yes, by “opening the box” one can check that the right ques­tion was asked to us, and what the raw re­sponse we gave was. But now we can go on and ask for a map:

But there are “prettier” map pro­jec­tions we could have used. And with ChatGPT’s “general knowl­edge” based on its read­ing of the web, etc. we can just ask it to use one:

But maybe we want a heat map in­stead. Again, we can just ask it to pro­duce this—un­der­neath us­ing our tech­nol­ogy:

Let’s change the pro­jec­tion again, now ask­ing it again to pick it us­ing its “general knowl­edge”:

And, yes, it got the pro­jec­tion “right”. But not the cen­ter­ing. So let’s ask it to fix that:

OK, so what do we have here? We’ve got some­thing that we “collaborated” to build. We in­cre­men­tally said what we wanted; the AI (i.e. ) pro­gres­sively built it. But what did we ac­tu­ally get? Well, it’s a piece of Wolfram Language code—which we could see by “opening the box”, or just ask­ing ChatGPT for:

If we copy the code out into a Wolfram Notebook, we can im­me­di­ately run it, and we find it has a nice “luxury fea­ture”—as ChatGPT claimed in its de­scrip­tion, there are dy­namic tooltips giv­ing the name of each coun­try:

What’s hap­pen­ing “under the hood” with ChatGPT and the Wolfram plu­gin? Remember that the core of ChatGPT is a “large lan­guage model” (LLM) that’s trained from the web, etc. to gen­er­ate a “reasonable con­tin­u­a­tion” from any text it’s given. But as a fi­nal part of its train­ing ChatGPT is also taught how to “hold con­ver­sa­tions”, and when to “ask some­thing to some­one else”—where that “someone” might be a hu­man, or, for that mat­ter, a plu­gin. And in par­tic­u­lar, it’s been taught when to reach out to the Wolfram plu­gin.

The Wolfram plu­gin ac­tu­ally has two en­try points: a Wolfram|Alpha one and a Wolfram Language one. The Wolfram|Alpha one is in a sense the “easier” for ChatGPT to deal with; the Wolfram Language one is ul­ti­mately the more pow­er­ful. The rea­son the Wolfram|Alpha one is eas­ier is that what it takes as in­put is just nat­ural lan­guage—which is ex­actly what ChatGPT rou­tinely deals with. And, more than that, Wolfram|Alpha is built to be for­giv­ing—and in ef­fect to deal with “typical hu­man-like in­put”, more or less how­ever messy that may be.

Wolfram Language, on the other hand, is set up to be pre­cise and well de­fined—and ca­pa­ble of be­ing used to build ar­bi­trar­ily so­phis­ti­cated tow­ers of com­pu­ta­tion. Inside Wolfram|Alpha, what it’s do­ing is to trans­late nat­ural lan­guage to pre­cise Wolfram Language. In ef­fect it’s catch­ing the “imprecise nat­ural lan­guage” and “funneling it” into pre­cise Wolfram Language.

When ChatGPT calls the Wolfram plu­gin it of­ten just feeds nat­ural lan­guage to Wolfram|Alpha. But ChatGPT has by this point learned a cer­tain amount about writ­ing Wolfram Language it­self. And in the end, as we’ll dis­cuss later, that’s a more flex­i­ble and pow­er­ful way to com­mu­ni­cate. But it does­n’t work un­less the Wolfram Language code is ex­actly right. To get it to that point is partly a mat­ter of train­ing. But there’s an­other thing too: given some can­di­date code, the Wolfram plu­gin can run it, and if the re­sults are ob­vi­ously wrong (like they gen­er­ate lots of er­rors), ChatGPT can at­tempt to fix it, and try run­ning it again. (More elab­o­rately, ChatGPT can try to gen­er­ate tests to run, and change the code if they fail.)

There’s more to be de­vel­oped here, but al­ready one some­times sees ChatGPT go back and forth mul­ti­ple times. It might be rewrit­ing its Wolfram|Alpha query (say sim­pli­fy­ing it by tak­ing out ir­rel­e­vant parts), or it might be de­cid­ing to switch be­tween Wolfram|Alpha and Wolfram Language, or it might be rewrit­ing its Wolfram Language code. Telling it how to do these things is a mat­ter for the ini­tial “plugin prompt”.

And writ­ing this prompt is a strange ac­tiv­ity—per­haps our first se­ri­ous ex­pe­ri­ence of try­ing to “communicate with an alien in­tel­li­gence”. Of course it helps that the “alien in­tel­li­gence” has been trained with a vast cor­pus of hu­man-writ­ten text. So, for ex­am­ple, it knows English (a bit like all those corny sci­ence fic­tion aliens…). And we can tell it things like “If the user in­put is in a lan­guage other than English, trans­late to English and send an ap­pro­pri­ate query to Wolfram|Alpha, then pro­vide your re­sponse in the lan­guage of the orig­i­nal in­put.”

Sometimes we’ve found we have to be quite in­sis­tent (note the all caps): “When writ­ing Wolfram Language code, NEVER use snake case for vari­able names; ALWAYS use camel case for vari­able names.” And even with that in­sis­tence, ChatGPT will still some­times do the wrong thing. The whole process of “prompt en­gi­neer­ing” feels a bit like an­i­mal wran­gling: you’re try­ing to get ChatGPT to do what you want, but it’s hard to know just what it will take to achieve that.

Eventually this will pre­sum­ably be han­dled in train­ing or in the prompt, but as of right now, ChatGPT some­times does­n’t know when the Wolfram plu­gin can help. For ex­am­ple, ChatGPT guesses that this is sup­posed to be a DNA se­quence, but (at least in this ses­sion) does­n’t im­me­di­ately think the Wolfram plu­gin can do any­thing with it:

Say “Use Wolfram”, though, and it’ll send it to the Wolfram plu­gin, which in­deed han­dles it nicely:

When the Wolfram plu­gin is given Wolfram Language code, what it does is ba­si­cally just to eval­u­ate that code, and re­turn the re­sult—per­haps as a graphic or math for­mula, or just text. But when it’s given Wolfram|Alpha in­put, this is sent to a spe­cial Wolfram|Alpha “for LLMs” API end­point, and the re­sult comes back as text in­tended to be “read” by ChatGPT, and ef­fec­tively used as an ad­di­tional prompt for fur­ther text ChatGPT is writ­ing. Take a look at this ex­am­ple:

The re­sult is a nice piece of text con­tain­ing the an­swer to the ques­tion asked, along with some other in­for­ma­tion ChatGPT de­cided to in­clude. But “inside” we can see what the Wolfram plu­gin (and the Wolfram|Alpha “LLM end­point”) ac­tu­ally did:

There’s quite a bit of ad­di­tional in­for­ma­tion there (including some nice pic­tures!). But ChatGPT “decided” just to pick out a few pieces to in­clude in its re­sponse.

By the way, some­thing to em­pha­size is that if you want to be sure you’re get­ting what you think you’re get­ting, al­ways check what ChatGPT ac­tu­ally sent to the Wolfram plu­gin—and what the plu­gin re­turned. One of the im­por­tant things we’re adding with the Wolfram plu­gin is a way to “factify” ChatGPT out­put—and to know when ChatGPT is “using its imag­i­na­tion”, and when it’s de­liv­er­ing solid facts.

Sometimes in try­ing to un­der­stand what’s go­ing on it’ll also be use­ful just to take what the Wolfram plu­gin was sent, and en­ter it as di­rect in­put on the Wolfram|Alpha web­site, or in a Wolfram Language sys­tem (such as the Wolfram Cloud).

One of the great (and, frankly, un­ex­pected) things about ChatGPT is its abil­ity to start from a rough de­scrip­tion, and gen­er­ate from it a pol­ished, fin­ished out­put—such as an es­say, let­ter, le­gal doc­u­ment, etc. In the past, one might have tried to achieve this “by hand” by start­ing with “boilerplate” pieces, then mod­i­fy­ing them, “gluing” them to­gether, etc. But ChatGPT has all but made this process ob­so­lete. In ef­fect, it’s “absorbed” a huge range of boil­er­plate from what it’s “read” on the web, etc.—and now it typ­i­cally does a good job at seam­lessly “adapting it” to what you need.

So what about code? In tra­di­tional pro­gram­ming lan­guages writ­ing code tends to in­volve a lot of “boilerplate work”—and in prac­tice many pro­gram­mers in such lan­guages spend lots of their time build­ing up their pro­grams by copy­ing big slabs of code from the web. But now, sud­denly, it seems as if ChatGPT can make much of this ob­so­lete. Because it can ef­fec­tively put to­gether es­sen­tially any kind of boil­er­plate code au­to­mat­i­cally—with only a lit­tle “human in­put”.

Of course, there has to be some hu­man in­put—be­cause oth­er­wise ChatGPT would­n’t know what pro­gram it was sup­posed to write. But—one might won­der—why does there have to be “boilerplate” in code at all? Shouldn’t one be able to have a lan­guage where—just at the level of the lan­guage it­self—all that’s needed is a small amount of hu­man in­put, with­out any of the “boilerplate dress­ing”?

Well, here’s the is­sue. Traditional pro­gram­ming lan­guages are cen­tered around telling a com­puter what to do in the com­put­er’s terms: set this vari­able, test that con­di­tion, etc. But it does­n’t have to be that way. And in­stead one can start from the other end: take things peo­ple nat­u­rally think in terms of, then try to rep­re­sent these com­pu­ta­tion­ally—and ef­fec­tively au­to­mate the process of get­ting them ac­tu­ally im­ple­mented on a com­puter.

Well, this is what I’ve now spent more than four decades work­ing on. And it’s the foun­da­tion of what’s now Wolfram Language—which I now feel jus­ti­fied in call­ing a “full-scale com­pu­ta­tional lan­guage”. What does this mean? It means that right in the lan­guage there’s a com­pu­ta­tional rep­re­sen­ta­tion for both ab­stract and real things that we talk about in the world, whether those are graphs or im­ages or dif­fer­en­tial equa­tions—or cities or chem­i­cals or com­pa­nies or movies.

Why not just start with nat­ural lan­guage? Well, that works up to a point—as the suc­cess of Wolfram|Alpha demon­strates. But once one’s try­ing to spec­ify some­thing more elab­o­rate, nat­ural lan­guage be­comes (like “legalese”) at best un­wieldy—and one re­ally needs a more struc­tured way to ex­press one­self.

There’s a big ex­am­ple of this his­tor­i­cally, in math­e­mat­ics. Back be­fore about 500 years ago, pretty much the only way to “express math” was in nat­ural lan­guage. But then math­e­mat­i­cal no­ta­tion was in­vented, and math took off—with the de­vel­op­ment of al­ge­bra, cal­cu­lus, and even­tu­ally all the var­i­ous math­e­mat­i­cal sci­ences.

My big goal with the Wolfram Language is to cre­ate a com­pu­ta­tional lan­guage that can do the same kind of thing for any­thing that can be “expressed com­pu­ta­tion­ally”. And to achieve this we’ve needed to build a lan­guage that both au­to­mat­i­cally does a lot of things, and in­trin­si­cally knows a lot of things. But the re­sult is a lan­guage that’s set up so that peo­ple can con­ve­niently “express them­selves com­pu­ta­tion­ally”, much as tra­di­tional math­e­mat­i­cal no­ta­tion lets them “express them­selves math­e­mat­i­cally”. And a crit­i­cal point is that—un­like tra­di­tional pro­gram­ming lan­guages—Wol­fram Language is in­tended not just for com­put­ers, but also for hu­mans, to read. In other words, it’s in­tended as a struc­tured way of “communicating com­pu­ta­tional ideas”, not just to com­put­ers, but also to hu­mans.

But now—with ChatGPT—this sud­denly be­comes even more im­por­tant than ever be­fore. Because—as we be­gan to see above—Chat­GPT can work with Wolfram Language, in a sense build­ing up com­pu­ta­tional ideas just us­ing nat­ural lan­guage. And part of what’s then crit­i­cal is that Wolfram Language can di­rectly rep­re­sent the kinds of things we want to talk about. But what’s also crit­i­cal is that it gives us a way to “know what we have”—be­cause we can re­al­is­ti­cally and eco­nom­i­cally read Wolfram Language code that ChatGPT has gen­er­ated.

The whole thing is be­gin­ning to work very nicely with the Wolfram plu­gin in ChatGPT. Here’s a sim­ple ex­am­ple, where ChatGPT can read­ily gen­er­ate a Wolfram Language ver­sion of what it’s be­ing asked:

And the crit­i­cal point is that the “code” is some­thing one can re­al­is­ti­cally ex­pect to read (if I were writ­ing it, I would use the slightly more com­pact func­tion):

I might have writ­ten the code a lit­tle dif­fer­ently, but this is again some­thing very read­able:

It’s of­ten pos­si­ble to use a pid­gin of Wolfram Language and English to say what you want:

Here’s an ex­am­ple where ChatGPT is again suc­cess­fully con­struct­ing Wolfram Language—and con­ve­niently shows it to us so we can con­firm that, yes, it’s ac­tu­ally com­put­ing the right thing:

And, by the way, to make this work it’s crit­i­cal that the Wolfram Language is in a sense “self-contained”. This piece of code is just stan­dard generic Wolfram Language code; it does­n’t de­pend on any­thing out­side, and if you wanted to, you could look up the de­f­i­n­i­tions of every­thing that ap­pears in it in the Wolfram Language doc­u­men­ta­tion.

OK, one more ex­am­ple:

Obviously ChatGPT had trou­ble here. But—as it sug­gested—we can just run the code it gen­er­ated, di­rectly in a note­book. And be­cause Wolfram Language is sym­bolic, we can ex­plic­itly see re­sults at each step:

So close! Let’s help it a bit, telling it we need an ac­tual list of European coun­tries:

And there’s the re­sult! Or at least, a re­sult. Because when we look at this com­pu­ta­tion, it might not be quite what we want. For ex­am­ple, we might want to pick out mul­ti­ple dom­i­nant col­ors per coun­try, and see if any of them are close to pur­ple. But the whole Wolfram Language setup here makes it easy for us to “collaborate with the AI” to fig­ure out what we want, and what to do.

So far we’ve ba­si­cally been start­ing with nat­ural lan­guage, and build­ing up Wolfram Language code. But we can also start with pseudocode, or code in some low-level pro­gram­ming lan­guage. And ChatGPT tends to do a re­mark­ably good job of tak­ing such things and pro­duc­ing well-writ­ten Wolfram Language code from them. The code is­n’t al­ways ex­actly right. But one can al­ways run it (e.g. with the Wolfram plu­gin) and see what it does, po­ten­tially (courtesy of the sym­bolic char­ac­ter of Wolfram Language) line by line. And the point is that the high-level com­pu­ta­tional lan­guage na­ture of the Wolfram Language tends to al­low the code to be suf­fi­ciently clear and (at least lo­cally) sim­ple that (particularly af­ter see­ing it run) one can read­ily un­der­stand what it’s do­ing—and then po­ten­tially it­er­ate back and forth on it with the AI.

When what one’s try­ing to do is suf­fi­ciently sim­ple, it’s of­ten re­al­is­tic to spec­ify it—at least if one does it in stages—purely with nat­ural lan­guage, us­ing Wolfram Language “just” as a way to see what one’s got, and to ac­tu­ally be able to run it. But it’s when things get more com­pli­cated that Wolfram Language re­ally comes into its own—pro­vid­ing what’s ba­si­cally the only vi­able hu­man-un­der­stand­able-yet-pre­cise rep­re­sen­ta­tion of what one wants.

And when I was writ­ing my book An Elementary Introduction to the Wolfram Language this be­came par­tic­u­larly ob­vi­ous. At the be­gin­ning of the book I was eas­ily able to make up ex­er­cises where I de­scribed what was wanted in English. But as things started get­ting more com­pli­cated, this be­came more and more dif­fi­cult. As a “fluent” user of Wolfram Language I usu­ally im­me­di­ately knew how to ex­press what I wanted in Wolfram Language. But to de­scribe it purely in English re­quired some­thing in­creas­ingly in­volved and com­pli­cated, that read like legalese.

But, OK, so you spec­ify some­thing us­ing Wolfram Language. Then one of the re­mark­able things ChatGPT is of­ten able to do is to re­cast your Wolfram Language code so that it’s eas­ier to read. It does­n’t (yet) al­ways get it right. But it’s in­ter­est­ing to see it make dif­fer­ent trade­offs from a hu­man writer of Wolfram Language code. For ex­am­ple, hu­mans tend to find it dif­fi­cult to come up with good names for things, mak­ing it usu­ally bet­ter (or at least less con­fus­ing) to avoid names by hav­ing se­quences of nested func­tions. But ChatGPT, with its com­mand of lan­guage and mean­ing, has a fairly easy time mak­ing up rea­son­able names. And al­though it’s some­thing I, for one, did not ex­pect, I think us­ing these names, and “spreading out the ac­tion”, can of­ten make Wolfram Language code even eas­ier to read than it was be­fore, and in­deed read very much like a for­mal­ized ana­log of nat­ural lan­guage—that we can un­der­stand as eas­ily as nat­ural lan­guage, but that has a pre­cise mean­ing, and can ac­tu­ally be run to gen­er­ate com­pu­ta­tional re­sults.

If you “know what com­pu­ta­tion you want to do”, and you can de­scribe it in a short piece of nat­ural lan­guage, then Wolfram|Alpha is set up to di­rectly do the com­pu­ta­tion, and pre­sent the re­sults in a way that is “visually ab­sorbable” as eas­ily as pos­si­ble. But what if you want to de­scribe the re­sult in a nar­ra­tive, tex­tual es­say? Wolfram|Alpha has never been set up to do that. But ChatGPT is.

And here within ChatGPT we’re ask­ing for this same Wolfram|Alpha re­sult, but then telling ChatGPT to “make an es­say out of it”:

Another “old chest­nut” for Wolfram|Alpha is math word prob­lems. Given a “crisply pre­sented” math prob­lem, Wolfram|Alpha is likely to do very well at solv­ing it. But what about a “woolly” word prob­lem? Well, ChatGPT is pretty good at “unraveling” such things, and turn­ing them into “crisp math ques­tions”—which then the Wolfram plu­gin can now solve. Here’s an ex­am­ple:

Here’s a slightly more com­pli­cated case, in­clud­ing a nice use of “common sense” to rec­og­nize that the num­ber of turkeys can­not be neg­a­tive:

Beyond math word prob­lems, an­other “old chest­nut” now ad­dressed by is what physi­cists tend to call “Fermi prob­lems”: or­der-of-mag­ni­tude es­ti­mates that can be made on the ba­sis of quan­ti­ta­tive knowl­edge about the world. Here’s an ex­am­ple:

is some­thing very new—re­ally a com­pletely new kind of tech­nol­ogy. And as hap­pens when­ever a new kind of tech­nol­ogy ar­rives, it’s open­ing up tremen­dous new op­por­tu­ni­ties. Some of these we can al­ready be­gin to to see—but lots of oth­ers will emerge over the weeks, months and years to come.

So how can you get in­volved in what promises to be an ex­cit­ing pe­riod of rapid tech­no­log­i­cal—and con­cep­tual—growth? The first thing is just to ex­plore . ChatGPT and Wolfram are each on their own vast sys­tems; the com­bi­na­tion of them is some­thing that it’ll take years to fully plumb. But the first step is just to get a sense of what’s pos­si­ble.

Find ex­am­ples. Share them. Try to iden­tify suc­cess­ful pat­terns of us­age. And, most of all, try to find work­flows that de­liver the high­est value. Those work­flows could be quite elab­o­rate. But they could also be quite sim­ple—cases where once one sees what can be done, there’s an im­me­di­ate “aha”.

How can you best im­ple­ment a work­flow? Well, we’re try­ing to work out the best work­flows for that. Within Wolfram Language we’re set­ting up flex­i­ble ways to call on things like ChatGPT, both purely pro­gram­mat­i­cally, and in the con­text of the note­book in­ter­face.

But what about from the ChatGPT side? Wolfram Language has a very open ar­chi­tec­ture, where a user can add or mod­ify pretty much what­ever they want. But how can you use this from ChatGPT? One thing is just to tell ChatGPT to in­clude some spe­cific piece of “initial” Wolfram Language code (maybe to­gether with doc­u­men­ta­tion)—then use some­thing like the pid­gin above to talk to ChatGPT about the func­tions or other things you’ve de­fined in that ini­tial code.

We’re plan­ning to build in­creas­ingly stream­lined tools for han­dling and shar­ing Wolfram Language code for use through ChatGPT. But one ap­proach that al­ready works is to sub­mit func­tions for pub­li­ca­tion in the Wolfram Function Repository, then—once they’re pub­lished—re­fer to these func­tions in your con­ver­sa­tion with ChatGPT.

OK, but what about within ChatGPT it­self? What kind of prompt en­gi­neer­ing should you do to best in­ter­act with the Wolfram plu­gin? Well, we don’t know yet. It’s some­thing that has to be ex­plored—in ef­fect as an ex­er­cise in AI ed­u­ca­tion or AI psy­chol­ogy. A typ­i­cal ap­proach is to give some “pre-prompts” ear­lier in your ChatGPT ses­sion, then hope it’s “still pay­ing at­ten­tion” to those later on. (And, yes, it has a lim­ited “attention span”, so some­times things have to get re­peated.)

We’ve tried to give an over­all prompt to tell ChatGPT ba­si­cally how to use the Wolfram plu­gin—and we fully ex­pect this prompt to evolve rapidly, as we learn more, and as the ChatGPT LLM is up­dated. But you can add your own gen­eral pre-prompts, say­ing things like “When us­ing Wolfram al­ways try to in­clude a pic­ture” or “Use SI units” or “Avoid us­ing com­plex num­bers if pos­si­ble”.

You can also try set­ting up a pre-prompt that es­sen­tially “defines a func­tion” right in ChatGPT—something like: “If I give you an in­put con­sist­ing of a num­ber, you are to use Wolfram to draw a poly­gon with that num­ber of sides”. Or, more di­rectly, “If I give you an in­put con­sist­ing of num­bers you are to ap­ply the fol­low­ing Wolfram func­tion to that in­put …”, then give some ex­plicit Wolfram Language code.

But these are very early days, and no doubt there’ll be other pow­er­ful mech­a­nisms dis­cov­ered for “programming” . And I think we can con­fi­dently ex­pect that the next lit­tle while will be an ex­cit­ing time of high growth, where there’s lots of valu­able “low-hanging fruit” to be picked by those who chose to get in­volved.

Even a week ago it was­n’t clear what was go­ing to be like—or how well it was go­ing to work. But these things that are now mov­ing so quickly are built on decades of ear­lier de­vel­op­ment. And in some ways the ar­rival of fi­nally mar­ries the two main ap­proaches his­tor­i­cally taken to AI—that have long been viewed as dis­joint and in­com­pat­i­ble.

ChatGPT is ba­si­cally a very large neural net­work, trained to fol­low the “statistical” pat­terns of text it’s seen on the web, etc. The con­cept of neural net­works—in a form sur­pris­ingly close to what’s used in ChatGPT—originated all the way back in the 1940s. But af­ter some en­thu­si­asm in the 1950s, in­ter­est waned. There was a resur­gence in the early 1980s (and in­deed I my­self first looked at neural nets then). But it was­n’t un­til 2012 that se­ri­ous ex­cite­ment be­gan to build about what might be pos­si­ble with neural nets. And now a decade later—in a de­vel­op­ment whose suc­cess came as a big sur­prise even to those in­volved—we have ChatGPT.

Rather sep­a­rate from the “statistical” tra­di­tion of neural nets is the “symbolic” tra­di­tion for AI. And in a sense that tra­di­tion arose as an ex­ten­sion of the process of for­mal­iza­tion de­vel­oped for math­e­mat­ics (and math­e­mat­i­cal logic), par­tic­u­larly near the be­gin­ning of the twen­ti­eth cen­tury. But what was crit­i­cal about it was that it aligned well not only with ab­stract con­cepts of com­pu­ta­tion, but also with ac­tual dig­i­tal com­put­ers of the kind that started to ap­pear in the 1950s.

The suc­cesses in what could re­ally be con­sid­ered “AI” were for a long time at best spotty. But all the while, the gen­eral con­cept of com­pu­ta­tion was show­ing tremen­dous and grow­ing suc­cess. But how might “computation” be re­lated to ways peo­ple think about things? For me, a cru­cial de­vel­op­ment was my idea at the be­gin­ning of the 1980s (building on ear­lier for­mal­ism from math­e­mat­i­cal logic) that trans­for­ma­tion rules for sym­bolic ex­pres­sions might be a good way to rep­re­sent com­pu­ta­tions at what amounts to a “human” level.

At the time my main fo­cus was on math­e­mat­i­cal and tech­ni­cal com­pu­ta­tion, but I soon be­gan to won­der whether sim­i­lar ideas might be ap­plic­a­ble to “general AI”. I sus­pected some­thing like neural nets might have a role to play, but at the time I only fig­ured out a bit about what would be needed—and not how to achieve it. Meanwhile, the core idea of trans­for­ma­tion rules for sym­bolic ex­pres­sions be­came the foun­da­tion for what’s now the Wolfram Language—and made pos­si­ble the decades-long process of de­vel­op­ing the full-scale com­pu­ta­tional lan­guage that we have to­day.

Starting in the 1960s there’d been ef­forts among AI re­searchers to de­velop sys­tems that could “understand nat­ural lan­guage”, and “represent knowl­edge” and an­swer ques­tions from it. Some of what was done turned into less am­bi­tious but prac­ti­cal ap­pli­ca­tions. But gen­er­ally suc­cess was elu­sive. Meanwhile, as a re­sult of what amounted to a philo­soph­i­cal con­clu­sion of ba­sic sci­ence I’d done in the 1990s, I de­cided around 2005 to make an at­tempt to build a gen­eral “computational knowl­edge en­gine” that could broadly an­swer fac­tual and com­pu­ta­tional ques­tions posed in nat­ural lan­guage. It was­n’t ob­vi­ous that such a sys­tem could be built, but we dis­cov­ered that—with our un­der­ly­ing com­pu­ta­tional lan­guage, and with a lot of work—it could. And in 2009 we were able to re­lease Wolfram|Alpha.

And in a sense what made Wolfram|Alpha pos­si­ble was that in­ter­nally it had a clear, for­mal way to rep­re­sent things in the world, and to com­pute about them. For us, “understanding nat­ural lan­guage” was­n’t some­thing ab­stract; it was the con­crete process of trans­lat­ing nat­ural lan­guage to struc­tured com­pu­ta­tional lan­guage.

Another part was as­sem­bling all the data, meth­ods, mod­els and al­go­rithms needed to “know about” and “compute about” the world. And while we’ve greatly au­to­mated this, we’ve still al­ways found that to ul­ti­mately “get things right” there’s no choice but to have ac­tual hu­man ex­perts in­volved. And while there’s a lit­tle of what one might think of as “statistical AI” in the nat­ural lan­guage un­der­stand­ing sys­tem of Wolfram|Alpha, the vast ma­jor­ity of Wolfram|Alpha—and Wolfram Language—operates in a hard, sym­bolic way that’s at least rem­i­nis­cent of the tra­di­tion of sym­bolic AI. (That’s not to say that in­di­vid­ual func­tions in Wolfram Language don’t use ma­chine learn­ing and sta­tis­ti­cal tech­niques; in re­cent years more and more do, and the Wolfram Language also has a whole built-in frame­work for do­ing ma­chine learn­ing.)

As I’ve dis­cussed else­where, what seems to have emerged is that “statistical AI”, and par­tic­u­larly neural nets, are well suited for tasks that we hu­mans “do quickly”, in­clud­ing—as we learn from ChatGPT—natural lan­guage and the “thinking” that un­der­lies it. But the sym­bolic and in a sense “more rigidly com­pu­ta­tional” ap­proach is what’s needed when one’s build­ing larger “conceptual” or com­pu­ta­tional “towers”—which is what hap­pens in math, ex­act sci­ence, and now all the “computational X” fields.

And now can be thought of as the first truly large-scale sta­tis­ti­cal + sym­bolic “AI” sys­tem. In Wolfram|Alpha (which be­came an orig­i­nal core part of things like the Siri in­tel­li­gent as­sis­tant) there was for the first time broad nat­ural lan­guage un­der­stand­ing—with “understanding” di­rectly tied to ac­tual com­pu­ta­tional rep­re­sen­ta­tion and com­pu­ta­tion. And now, 13 years later, we’ve seen in ChatGPT that pure “statistical” neural net tech­nol­ogy, when trained from al­most the en­tire web, etc. can do re­mark­ably well at “statistically” gen­er­at­ing “human-like” “meaningful lan­guage”. And in we’re now able to lever­age the whole stack: from the pure “statistical neural net” of ChatGPT, through the “computationally an­chored” nat­ural lan­guage un­der­stand­ing of Wolfram|Alpha, to the whole com­pu­ta­tional lan­guage and com­pu­ta­tional knowl­edge of Wolfram Language.

When we were first build­ing Wolfram|Alpha we thought that per­haps to get use­ful re­sults we’d have no choice but to en­gage in a con­ver­sa­tion with the user. But we dis­cov­ered that if we im­me­di­ately gen­er­ated rich, “visually scannable” re­sults, we only needed a sim­ple “Assumptions” or “Parameters” in­ter­ac­tion—at least for the kind of in­for­ma­tion and com­pu­ta­tion seek­ing we ex­pected of our users. (In Wolfram|Alpha Notebook Edition we nev­er­the­less have a pow­er­ful ex­am­ple of how mul­ti­step com­pu­ta­tion can be done with nat­ural lan­guage.)

Back in 2010 we were al­ready ex­per­i­ment­ing with gen­er­at­ing not just the Wolfram Language code of typ­i­cal Wolfram|Alpha queries from nat­ural lan­guage, but also “whole pro­grams”. At the time, how­ever—with­out mod­ern LLM tech­nol­ogy—that did­n’t get all that far. But what we dis­cov­ered was that—in the con­text of the sym­bolic struc­ture of the Wolfram Language—even hav­ing small frag­ments of what amounts to code be gen­er­ated by nat­ural lan­guage was ex­tremely use­ful. And in­deed I, for ex­am­ple, use the mech­a­nism in Wolfram Notebooks count­less times al­most every day, for ex­am­ple to con­struct sym­bolic en­ti­ties or quan­ti­ties from nat­ural lan­guage. We don’t yet know quite what the mod­ern “LLM-enabled” ver­sion of this will be, but it’s likely to in­volve the rich hu­man-AI “collaboration” that we dis­cussed above, and that we can be­gin to see in ac­tion for the first time in .

I see what’s hap­pen­ing now as a his­toric mo­ment. For well over half a cen­tury the sta­tis­ti­cal and sym­bolic ap­proaches to what we might call “AI” evolved largely sep­a­rately. But now, in they’re be­ing brought to­gether. And while we’re still just at the be­gin­ning with this, I think we can rea­son­ably ex­pect tremen­dous power in the com­bi­na­tion—and in a sense a new par­a­digm for “AI-like com­pu­ta­tion”, made pos­si­ble by the ar­rival of ChatGPT, and now by its com­bi­na­tion with Wolfram|Alpha and Wolfram Language in .

My phone pinged: “Tell Mark Essien they are com­ing for him.”

I had no idea what that even meant. It was a for­warded WhatsApp mes­sage that some­one had sent to a friend of mine.

All I was do­ing was try­ing to con­vert PDFs from the 2023 Nigeria Presidential elec­tions into spread­sheets.

But let’s rewind a bit and give some con­text.

In 2020 a small group of about 5 young peo­ple went to a cross­roads in Lagos, Nigeria to protest against po­lice bru­tal­ity. Their protest was called #EndSARS, and they were protest­ing po­lice bru­tal­ity. SARS (The Special Anti-Robbery Squad) was a no­to­ri­ous seg­ment of the Nigeria po­lice force, known for ex­tor­tion and ex­tra-ju­di­cial killings of any­body they termed armed rob­bers.

For 4 days that tiny group protested, with barely any­body notic­ing.

On the last day, as they were about to leave, some­one tweeted about their protest. Then some­one else retweeted, and some­one else. And soon, there thou­sands of retweets, with so­cial me­dia hail­ing them as he­roes.

A few peo­ple who saw the protest on twit­ter joined them. It was still a small crowd, but it was grow­ing. The news started spread­ing on so­cial me­dia, and the crowd grew. Within a cou­ple of days, there were thou­sands of peo­ple protest­ing with the #EndSARS hash­tag. The gov­ern­ment was silent, be­liev­ing it would blow away shortly.

But the protests grew and grew. Young peo­ple in other cities started join­ing. Nigerian so­cial me­dia was aflame - this was the only topic. Till the protests be­came vi­o­lent. The pro­test­ers started at­tack­ing po­lice sta­tions. The po­lice­men started shoot­ing pro­test­ers.

The pro­test­ers marched to no­to­ri­ous SARS pris­ons and forced them open. Prisoners es­caped. Malls were burnt down. Policemen threw off their uni­forms and Government of­fi­cials fled.

As the vi­o­lence flared, so­cial me­dia now started call­ing for calm. But it was late - a lot of hood­lums had al­ready in­fil­trated the protests, and they were bent on get­ting re­venge for every­thing they blamed the Government for.

The Government tried to stop the protests, but there was no­body to talk to. There was no head of protest, no fig­ure­head, no­body with any real au­thor­ity. They called it a “headless mob”.

Social me­dia started call­ing for calm, and ask­ing the pres­i­dent to speak. After many days of si­lence, he fi­nally spoke, urg­ing for calm, dis­band­ing SARS and promis­ing to ad­dress the con­cerns of the pro­test­ers.

Things calmed down. But there was still a huge crowd gath­ered at the Lekki Tollgate - a ma­jor, 6 lane high­way that passes through the heart of Lagos. Concerts were play­ing at the lo­ca­tion, peo­ple were serv­ing food and drinks.

Then omi­nously, on the 20th of October of 2020 some peo­ple drove there in un­marked cars and re­moved all the Cameras in­stalled at the toll­gate.

That night, as the DJs played, all the flood­lights at the toll­gate switched off. Social me­dia users warned about mil­i­tary ve­hi­cles dri­ving along the high­way to­wards the lo­ca­tion.

Short clips from mo­bile phones cap­ture what hap­pened next. It’s dark, peo­ple are run­ning, gun­shots every­where. People are gath­ered around bleed­ing peo­ple, scream­ing. More run­ning, lights flash­ing.

There are no cam­eras, so there is no clear re­con­struc­tion.

But the next morn­ing, the army is in con­trol of the toll­gate, and the pro­test­ers are gone. Social me­dia is filled with mes­sages that many peo­ple were killed and the army took away their bod­ies and cleared the scene. The army and gov­ern­ment said noth­ing like that hap­pened.

There is a lot of anger, but the protests are over. Everything seems to be back to nor­mal. But there is an un­der-cur­rent of youth anger.

And the pres­i­den­tial elec­tion is in 2023.

Nigeria has al­ways been a two party state - much like the U. S. It uses pretty much the same pres­i­den­tial sys­tem as the U.S, and just like the Democrats and Republicans, there have been vari­ants of the same two par­ties fac­ing each other. Since 2011, it has been the PDP (former rul­ing party) against the APC (current rul­ing party).

In 2022, one of the can­di­dates abruptly left the PDP, claim­ing that the process was rigged against him win­ning the nom­i­na­tion. He joined a small party that had only ever won one state gov­er­nor­ship elec­tion in its en­tire his­tory.

Suddenly, the so­cial me­dia mob adopted this can­di­date - Peter Obi - as their pre­ferred can­di­date. Many of the same han­dles who had called out #EndSARS now be­came #Obidient. Tweets, WhatsApp mes­sages grew, call­ing on oth­ers to join the #Obidient move­ment.

Peter Obi him­self seemed as sur­prised as every­one else that the head­less mob had adopted him, but he quickly ac­cepted the Obidients, and started us­ing the slo­gans as part of his cam­paign.

The po­lit­i­cal es­tab­lish­ment mocked the move­ment, say­ing the “four peo­ple tweet­ing in a room” had never won even a coun­cil elec­tion in Nigeria.

The move­ment grew louder and louder on so­cial me­dia, with the mem­bers of the move­ment let­ting out their anger at the ex­ist­ing po­lit­i­cal in­sti­tu­tion. The politi­cians mocked back - say­ing that pol­i­tics was not won on so­cial me­dia, but at the grass­roots - re­fer­ring to the vil­lagers who nor­mally vote for who­ever gives them the most money.

Election Day came. The Labour Party can­di­date was Peter Obi, the can­di­date adopted by the young peo­ple on so­cial me­dia. The can­di­date from the rul­ing party was Bola Tinubu, for­mer Governor of the most pop­u­lous state (Lagos), and who has been in­volved in se­lect­ing every Governor to the state since he was Governor. He was also ru­mored to be get­ting a cut of all Lagos State tax rev­enue via a con­sult­ing firm he set up when he was Governor. The can­di­date from the for­mer rul­ing party was Atiku Abubakar, a for­mer vice pres­i­dent, a very wealthy man. His wealth came from his hold­ings in the ports, which he helped pri­va­tize when he was vice pres­i­dent.

For the first time, Nigeria had adopted elec­tronic trans­mis­sion of vot­ing from the polling units. That meant that the polling unit vot­ing sheets could be viewed in real time on a web­site.

As the re­sults started com­ing in, the coun­try was elec­tri­fied. Tens of thou­sands of votes were com­ing in for Peter Obi. Then hun­dreds of thou­sands of votes started com­ing in. Then a mil­lion votes came in and an­other mil­lion.

The sen­a­to­r­ial elec­tion re­sults started com­ing in, and the Labour Party was win­ning seats. An un­known party with no pre­vi­ous Governor or Senate seat was sweep­ing seats across wide swathes of the coun­try.

But there was no way of know­ing who won. Even though the Nigerian Electoral Body had agreed to elec­tronic trans­mis­sion of polling unit re­sults, the sen­a­tors and po­lit­i­cal es­tab­lish­ment had blocked the elec­tronic trans­fer of ac­tual votes. They wanted it still writ­ten on pa­per, for rea­sons we can guess at.

With 170,000 thou­sand polling units, it was go­ing to take a long time to count the re­sults.

But across so­cial me­dia, videos and pho­tos were spread­ing of the Labour Party win­ning polling unit af­ter polling unit, peo­ple yelling in hap­pi­ness as the party won. Even in the rural ar­eas, where the politi­cians had claimed they had ab­solute con­trol, the party was win­ning unit af­ter unit.

But other dis­turb­ing videos were also com­ing out - of polling units de­stroyed, and peo­ple threat­en­ing to hurt any­one who voted for the Labour party.

Social Media waited for the elec­tion com­mis­sion to an­nounce the re­sults. People tried to tally the re­sults on­line, but with 170,000 PDFs in a com­pletely un­struc­tured for­mat, it was close to im­pos­si­ble.

Then INEC started an­nounc­ing the state re­sults, over a mul­ti­ple day count­ing marathon. At the end, the re­sult was an­nounced:

The Obidients had won in the cap­i­tal city and had de­feated the rul­ing party and their can­di­date in Lagos - the state they had ruled for the last 16 years. But ac­cord­ing to the of­fi­cial tally, they had lost in the coun­try.

But even as the re­sults were be­ing an­nounced, peo­ple on so­cial me­dia were dis­cov­er­ing that if they man­u­ally added all the re­sults from the servers of the elec­toral um­pire to­gether, the to­tal did not add up to the votes an­nounced. They showed ex­am­ples from Rivers State, where just a few polling units added to­gether ex­ceeded the votes an­nounced by the elec­toral um­pire. People cried foul.

The amended elec­toral rules re­quired the par­ties to chal­lenge elec­tion re­sults within 21 days af­ter they were an­nounced. But to file, ev­i­dence was needed. The only ev­i­dence about how the elec­tion re­ally went was spread out across 170,000 pho­tographs of polling unit re­sults. There was no dig­i­tal ver­sion of the re­sults, just pho­tos.

The clock was tick­ing, and im­me­di­ately var­i­ous Obidient groups sprung up to fig­ure out how to ex­tract the data out of the pho­tos. I was in one of the groups, and we de­cided to try var­i­ous things.

The first thing was OCR. The pho­tos were all snapped in many dif­fer­ent ways, with hardly any struc­ture. Each party re­sult had a num­ber be­side it. The photo an­gles were dif­fer­ent. Many sheets were blurry or had cam­era flash on them.

All the open source OCR soft­ware gave bad re­sults. The best re­sult came from Amazon Rekognition, but it was still not good enough - it would oc­ca­sion­ally change the scores, and that was sim­ply not go­ing to work.

After ex­per­i­ment­ing with OCR for about a day, we gave up. We had about 8 days left to go.

We had a brain­storm­ing meet­ing, and de­cided to try a new ap­proach. We would sim­ply ask the Obidients to help us do the con­ver­sion. If hun­dreds of Obidients did the tran­scrip­tion, it would go fast.

So we quickly de­signed out a web­site:

And then started cod­ing. The fron­tend was in React, and the back­end in PHP Laravel.

A cou­ple of days later, the app was done. A sim­ple site that showed a pic­ture of a polling unit sheet and asked peo­ple to en­ter the votes they saw in some textfields. Then the val­ues would be saved in the data­base.

Within min­utes, I was get­ting replies from the Obidients. People were jump­ing on the site, and the first re­sults started go­ing up. The progress bar started mov­ing, slowly at first, and then faster and faster. We went from tran­scrib­ing one re­sult every minute to tran­scrib­ing 1 every 10 sec­onds, and soon we were tran­scrib­ing at 1 sheet per sec­ond.

Traffic grew on the site, and soon our com­pletely un­op­ti­mized back­end was strug­gling to catch up. But we were mov­ing for­ward. We quickly grew to be tran­scrib­ing 20,000 sheets per day.

The site had a re­sults page that was up­dat­ing the re­sults live as the sheets were tran­scrib­ing. The re­sults were grow­ing, and by the time we had counted 6 mil­lion votes, the Obidients were clearly in the lead. We kept tran­scrib­ing, and by the time we had reached 50% of the count - 10 mil­lion votes - Peter Obi was strongly in the lead.

I tweeted that out, and all hell broke loose. Suddenly, out of nowhere thou­sands of spam twit­ter ac­counts came out of nowhere, at­tack­ing the ef­fort. Threats, at­tacks of all kinds hit us.

Then the bots came. Thousands of en­tries started on the site - all en­ter­ing huge num­bers for the rul­ing party. Different IP ad­dresses, dif­fer­ent prox­ies, all en­ter­ing fake num­bers.

When we started the pro­ject, we had a plan. We would first tran­scribe all 170,000 polling unit re­sults, then we would do a sec­ond pass of all the re­sults again as a val­i­da­tion step. If the same num­bers were en­tered twice, then it was likely that the en­tered num­bers were cor­rect. But with the bots en­ter­ing fake num­bers, we now had a new bat­tle to fight.

We im­me­di­ately en­abled captcha, and that slowed down the bots a bit. Then we quickly im­ple­mented a check to see if any­body would en­ter weirdly large num­bers, we would ig­nore their en­tries mov­ing for­ward. Then we started show­ing some re­sults we knew to the bots - if they en­tered wrong num­bers, we would stop ac­cept­ing the re­sults.

Whoever was be­hind the bots kept adapt­ing and coun­ter­act­ing what we were do­ing. They went from us­ing a script to us­ing what felt like hun­dreds of hu­mans. They went from en­ter­ing ab­surdly large num­bers to en­ter­ing plau­si­ble num­bers. They started en­ter­ing some cor­rect and some wrong num­bers.

But the com­bi­na­tion of tech­niques and the large num­ber of Obidients also work­ing meant that we were also get­ting a huge num­ber of cor­rect en­tries.

It seemed the counter-par­ties re­al­ized that they would not win on tech­nol­ogy, so they started a new cam­paign. Hundreds of ac­counts that claimed to be Obidients sud­denly popped up, say­ing that we were work­ing for the rul­ing party, and that the work we were do­ing was de­signed to prove that the Obidients had lost the elec­tion. Every sin­gle post I made would have tens of those ac­counts re­ply­ing.

And they dou­bled down - they cre­ated a fake screen­shot show­ing that the rul­ing party had won on our site, and started spread­ing that. A cur­rent Government min­is­ter even tweeted this out, and a badly writ­ten ar­ti­cle hur­riedly ap­peared on a ma­jor lo­cal news­pa­per.

Their tech­nique worked. The Obidients started doubt­ing. Prominent Obidient ac­counts started threads ques­tion­ing if we were work­ing for the Government.

Meanwhile, we had com­pleted tran­scrib­ing 150,000 polling unit sheets, and we needed to move to the val­i­da­tion phase where we would cor­rect all the dam­age the bots had done.

But the dam­age had been done. The crowd we had pulled to do the work did not trust us any­more. We did not have any­one any­more to help us val­i­date the en­tries. Work started, but moved very slowly - less than 2000 en­tries val­i­dated in a day. It would take us 3 months to fin­ish at this rate.

Was our pro­ject go­ing to fail? If we did not an­nounce re­sults, the Obidients would be sure that we were work­ing for the Government. And the Government al­ready saw us as en­e­mies. How could we solve this?

I started look­ing through the data. We had 800,000 sub­mis­sions in our data­base for 170,000 polling units. I was ran­domly sam­pling, and I no­ticed that a huge num­ber of en­tries were cor­rect. And strangely enough, the ones with the wrong en­tries had lots and lots of wrong en­tries. While the vast ma­jor­ity, which only had one en­try, were mostly cor­rect.

I tested around a bit, and then I re­al­ized. We had a bug in the code. When you opened the web­site for the first time, it was buggy - it was not re­turn­ing a to­tally ran­dom en­try the way it was sup­posed to. It had a huge ten­dency to re­turn from a small set of en­tries. But sub­se­quent ones were now ran­dom.

That meant that every time the web­site was fully re­freshed, the first polling unit en­tered was one that prob­a­bly al­ready had hun­dreds of en­tries. But if you kept work­ing, you would now be work­ing on new units.

And I re­al­ized what could have hap­pened - who­ever was con­trol­ling the bots must have told them to re­fresh af­ter every en­try. Probably they fig­ured out that it would make it harder for us to de­tect. But then they ended up en­ter­ing all the wrong val­ues mostly for a small set of units that we could eas­ily clean up. Up to 90% of our data was per­fectly clean.

We quickly stopped all en­tries, and start crunch­ing the re­sults. A few hours of heavy server lift­ing, and we were done. We had tran­scribed 170,000 polling unit sheets to CSV for­mat in 5 days with a large group of vol­un­teers.

We shared the re­sults with the team for them to sub­mit as part of their ev­i­dence. And we were done.

We pub­lished the re­sults in spread­sheets here. https://​drive.google.com/​drive/​fold­ers/​173o­Hgm­s6wYy5WKz_i3Lh­l5mX­cmobCWHz?usp=shar­ing.

Peter Obi filed his pe­ti­tion on the last day. And I was happy to see some fa­mil­iar things in the fil­ing:

Now we wait to see how the tri­bunal will de­cide.

Email me on markessien@gmail.com or fol­low on twit­ter: twit­ter.com/​markessien.

* Block Inc., for­merly known as Square Inc., is a $44 bil­lion mar­ket cap com­pany that claims to have de­vel­oped a “frictionless” and “magical” fi­nan­cial tech­nol­ogy with a mis­sion to em­power the “unbanked” and the “underbanked”.

* Our 2-year in­ves­ti­ga­tion has con­cluded that Block has sys­tem­at­i­cally taken ad­van­tage of the de­mo­graph­ics it claims to be help­ing. The “magic” be­hind Block’s busi­ness has not been dis­rup­tive in­no­va­tion, but rather the com­pa­ny’s will­ing­ness to fa­cil­i­tate fraud against con­sumers and the gov­ern­ment, avoid reg­u­la­tion, dress up preda­tory loans and fees as rev­o­lu­tion­ary tech­nol­ogy, and mis­lead in­vestors with in­flated met­rics.

* Our re­search in­volved dozens of in­ter­views with for­mer em­ploy­ees, part­ners, and in­dus­try ex­perts, ex­ten­sive re­view of reg­u­la­tory and lit­i­ga­tion records, and FOIA and pub­lic records re­quests.

* Most an­a­lysts are ex­cited about the post-pan­demic surge of Block’s Cash App plat­form, with ex­pec­ta­tions that its 51 mil­lion monthly trans­act­ing ac­tive users and low cus­tomer ac­qui­si­tion costs will drive high mar­gin growth and serve as a fu­ture plat­form to of­fer new prod­ucts.

* Our re­search in­di­cates, how­ever, that Block has wildly over­stated its gen­uine user counts and has un­der­stated its cus­tomer ac­qui­si­tion costs. Former em­ploy­ees es­ti­mated that 40%-75% of ac­counts they re­viewed were fake, in­volved in fraud, or were ad­di­tional ac­counts tied to a sin­gle in­di­vid­ual.

* Core to the is­sue is that Block has em­braced one tra­di­tion­ally very “underbanked” seg­ment of the pop­u­la­tion: crim­i­nals. The com­pa­ny’s “Wild West” ap­proach to com­pli­ance made it easy for bad ac­tors to mass-cre­ate ac­counts for iden­tity fraud and other scams, then ex­tract stolen funds quickly.

* Even when users were caught en­gag­ing in fraud or other pro­hib­ited ac­tiv­ity, Block black­listed the ac­count with­out ban­ning the user. A for­mer cus­tomer ser­vice rep shared screen­shots show­ing how black­listed ac­counts were reg­u­larly as­so­ci­ated with dozens or hun­dreds of other ac­tive ac­counts sus­pected of fraud. This phe­nom­e­non of al­low­ing black­listed users was so com­mon that rap­pers bragged about it in hip hop songs.

* Block ob­fus­cates how many in­di­vid­u­als are on the Cash App plat­form by re­port­ing mis­lead­ing “transacting ac­tive” met­rics filled with fake and du­pli­cate ac­counts. Block can and should clar­ify to in­vestors an es­ti­mate on how many unique peo­ple ac­tu­ally use Cash App.

* CEO Jack Dorsey has pub­licly touted how Cash App is men­tioned in hun­dreds of hip hop songs as ev­i­dence of its main­stream ap­peal. A re­view of those songs show that the artists are not gen­er­ally rap­ping about Cash App’s smooth user in­ter­face—many de­scribe us­ing it to scam, traf­fic drugs or even pay for mur­der. [See our com­pi­la­tion video on this here]

* “I paid them hit­ters through Cash App”— Block paid to pro­mote a video for a song called “Cash App” which de­scribed pay­ing con­tract killers through the app. The song’s artist was later ar­rested for at­tempted mur­der.

* Cash App was also cited “by far” as the top app used in re­ported U.S. sex traf­fick­ing, ac­cord­ing to a lead­ing non-profit or­ga­ni­za­tion. Multiple Department of Justice com­plaints out­line how Cash App has been used to fa­cil­i­tate sex traf­fick­ing, in­clud­ing sex traf­fick­ing of mi­nors.

* There is even a gang named af­ter Cash App: In 2021, Baltimore au­thor­i­ties charged mem­bers of the “Cash App” gang with dis­tri­b­u­tion of fen­tanyl in a West Baltimore neigh­bor­hood, ac­cord­ing to news re­ports and crim­i­nal records.

* Beyond fa­cil­i­tat­ing pay­ments for crim­i­nal ac­tiv­ity, the plat­form has been over­run with scam ac­counts and fake users, ac­cord­ing to nu­mer­ous in­ter­views with for­mer em­ploy­ees.

* Examples of ob­vi­ous dis­tor­tions abound: “Jack Dorsey” has mul­ti­ple fake ac­counts, in­clud­ing some that ap­pear aimed at scam­ming Cash App users.  “Elon Musk” and “Donald Trump” have dozens.

* To test this, we turned our ac­counts into “Donald Trump” and “Elon Musk” and were eas­ily able to send and re­ceive money. We or­dered a Cash Card un­der our ob­vi­ously fake Donald Trump ac­count, check­ing to see if Cash App’s com­pli­ance would take is­sue—the card promptly ar­rived in the mail.

* Former em­ploy­ees de­scribed how Cash App sup­pressed in­ter­nal con­cerns and ig­nored user pleas for help as crim­i­nal ac­tiv­ity and fraud ran ram­pant on its plat­form. This ap­peared to be an ef­fort to grow Cash App’s user base by strate­gi­cally dis­re­gard­ing Anti Money Laundering (AML) rules.

* The COVID-19 pan­demic and na­tion­wide lock­downs posed an ex­is­ten­tial threat to Block’s key dri­ver of gross profit at the time, mer­chant ser­vices.

* In this en­vi­ron­ment, amid Cash App’s anti-com­pli­ance free-for-all, the app fa­cil­i­tated a mas­sive wave of gov­ern­ment COVID-relief pay­ments. CEO Jack Dorsey Tweeted that users could get gov­ern­ment pay­ments through Cash App “immediately” with “no bank ac­count needed” due to its fric­tion­less tech­nol­ogy.

* Within weeks of Cash App ac­counts re­ceiv­ing their first gov­ern­ment pay­ments, states were seek­ing to claw back sus­pected fraud­u­lent pay­ments—Wash­ing­ton State wanted more than $200 mil­lion back from pay­ment proces­sors while Arizona sought to re­cover $500 mil­lion, for­mer em­ploy­ees told us.

* Once again, the signs were hard to miss. Rapper “Nuke Bizzle”, made a pop­u­lar mu­sic video about com­mit­ting COVID fraud. Several weeks later, he was ar­rested and even­tu­ally con­victed for com­mit­ting COVID fraud. The only pay­ment provider men­tioned in the in­dict­ment was Cash App, which was used to fa­cil­i­tate the fraud­u­lent pay­ments.

* We filed pub­lic records re­quests to learn more about Block’s role in fa­cil­i­tat­ing pan­demic re­lief fraud and re­ceived an­swers from sev­eral states.

* Massachusetts sought to claw back over 69,000 un­em­ploy­ment pay­ments from Cash App ac­counts just four months into the pan­demic. Suspect trans­ac­tions at Cash App’s part­ner bank were dis­pro­por­tion­ate, ex­ceed­ing ma­jor banks like JP Morgan and Wells Fargo, de­spite the lat­ter banks hav­ing 4x-5x as many de­posit ac­counts.

* In Ohio, Cash App’s part­ner bank had 8x the sus­pect pan­demic-re­lated un­em­ploy­ment pay­ments as the bank that processed the most un­em­ploy­ment claims in the state, even though the lat­ter bank processed 2x the claims as Cash App’s, ac­cord­ing to data we ob­tained via a pub­lic records re­quest.

* The data shows that com­pared to its Ohio com­peti­tor, Cash App’s part­ner bank had nearly 10x the num­ber of ap­pli­cants who ap­plied for ben­e­fits through a bank ac­count used by an­other claimant — a clear red flag of fraud.

* Block had ob­vi­ous com­pli­ance lapses that made fraud easy, such as per­mit­ting sin­gle ac­counts to re­ceive un­em­ploy­ment pay­ments on be­half of mul­ti­ple in­di­vid­u­als from var­i­ous states and in­ef­fec­tive ad­dress ver­i­fi­ca­tion.

* In an ap­par­ent ef­fort to pre­serve its growth en­gine, Cash App ig­nored in­ter­nal em­ployee con­cerns, along with warn­ings from the Secret Service, the U.S. Department of Labor OIG, FinCEN, and State Regulators which all specif­i­cally flagged the is­sue of mul­ti­ple COVID re­lief pay­ments go­ing to the same ac­count as an ob­vi­ous sign of fraud.

* Block re­ported a pan­demic surge in user counts and rev­enue, ig­nor­ing the con­tri­bu­tion of wide­spread fraud­u­lent ac­counts and pay­ments. The new busi­ness pro­vided a sharp one-time in­crease to Block’s stock, which rose 639% in 18 months dur­ing the pan­demic.

* As Block’s stock soared on the back of its fa­cil­i­ta­tion of fraud, co-founders Jack Dorsey and James McKelvey col­lec­tively sold over $1 bil­lion of stock dur­ing the pan­demic. Other ex­ec­u­tives, in­clud­ing CFO Amrita Ahuja and the lead man­ager for Cash App Brian Grassadonia, also dumped mil­lions of dol­lars in stock.

* With its in­flux of pan­demic Cash App users, our re­search shows Block has qui­etly fu­eled its prof­itabil­ity by avoid­ing a key bank­ing reg­u­la­tion meant to pro­tect mer­chants. “Interchange fees” are fees charged to mer­chants for ac­cept­ing use of var­i­ous pay­ment cards.

* Congress passed a law that legally caps “interchange fees” charged by large banks that have over $10 bil­lion in as­sets. Despite hav­ing $31 bil­lion in as­sets, Block avoids these reg­u­la­tions by rout­ing pay­ments through a small bank and goug­ing mer­chants with el­e­vated fees.

* Block in­cludes only a sin­gle vague ref­er­ence in its fil­ings ac­knowl­edg­ing it earns rev­enue from “interchange fees”. It has never re­vealed the full eco­nom­ics of this cat­e­gory, yet roughly one-third of Cash App’s rev­enue came from this opaque source, ac­cord­ing to a 2022 Credit Suisse re­search re­port.

* Competitor PayPal has dis­closed it is un­der in­ves­ti­ga­tion by both the SEC and the CFPB over its sim­i­lar use of a small bank to avoid “interchange fee” caps. A Freedom of Information Act (FOIA) re­quest we filed with the SEC in­di­cates that Block may be part of a sim­i­lar in­ves­ti­ga­tion.

* Block’s $29 bil­lion deal to ac­quire ‘buy now pay lat­er’ (BNPL) ser­vice Afterpay closed in January 2022. Afterpay has been cel­e­brated by Block as a ma­jor fi­nan­cial in­no­va­tion, al­low­ing users to buy things like a pair of shoes or a t-shirt and pay over time, only in­cur­ring mas­sive fees if sub­se­quent pay­ments are late.

* Afterpay was de­signed in a way that avoided re­spon­si­ble lend­ing rules in its na­tive Australia, ex­tend­ing a form of credit to users with­out in­come ver­i­fi­ca­tion or credit checks. The ser­vice does­n’t tech­ni­cally charge “interest”, but late fees can reach APR equiv­a­lents as high as 289%.

* The ac­qui­si­tion is flop­ping. In 2022, the year Afterpay was ac­quired, it lost $357 mil­lion, ac­cel­er­at­ing from 2021 losses of $184 mil­lion.

* Fitch Ratings re­ported that Afterpay delin­quen­cies through March 2022 had more than dou­bled to 4.1%, from 1.7% in June 2021 (just prior to the an­nounced ac­qui­si­tion). Total pro­cess­ing vol­ume de­clined -4.8% from the pre­vi­ous year.

* Block reg­u­larly hy­pes other mun­dane or preda­tory sources of rev­enue as tech­no­log­i­cal break­throughs. Roughly 31% of Cash App’s rev­enue comes from “instant de­posit” which Block says it pi­o­neered and works as if by “magic”. Every other ma­jor com­peti­tor we checked pro­vides a sim­i­lar ser­vice at com­pa­ra­ble or bet­ter rates.

* On a purely fun­da­men­tal ba­sis, even be­fore fac­tor­ing in the find­ings of our in­ves­ti­ga­tion, we see down­side of be­tween 65% to 75% in Block shares. Block re­ported a 1% year over year rev­enue de­cline and a GAAP loss of $540.7 mil­lion in 2022. Analysts have fu­ture ex­pec­ta­tions of GAAP un­prof­itabil­ity and the com­pany has warned it may not be prof­itable.

* Despite this, Block is val­ued like a prof­itable growth com­pany at (i) an EV/EBITDA mul­ti­ple of 60x; (ii) a for­ward 2023 “adjusted” earn­ings mul­ti­ple of 41x; and (iii) a price to tan­gi­ble book ra­tio of 13.1x, all wildly out of line with fin­tech peers.

* Despite its cur­rent rich mul­ti­ples, Block is also fac­ing threats from key com­peti­tors like Zelle, Venmo/Paypal and fast-grow­ing pay­ment so­lu­tions from smart­phone pow­er­houses like Apple and Google. Apple has grown Apple Pay ac­ti­va­tions from 20% in 2017 to over 70% in 2022 and now leads in dig­i­tal wal­let mar­ket share.

* In sum, we think Block has mis­led in­vestors on key met­rics, and em­braced preda­tory of­fer­ings and com­pli­ance worst-prac­tices in or­der to fuel growth and profit from fa­cil­i­ta­tion of fraud against con­sumers and the gov­ern­ment.

* We also be­lieve Jack Dorsey has built an em­pire—and amassed a $5 bil­lion per­sonal for­tune—pro­fess­ing to care deeply about the de­mo­graph­ics he is tak­ing ad­van­tage of. With Dorsey and top ex­ec­u­tives al­ready hav­ing sold over $1 bil­lion in eq­uity on Block’s me­te­oric pan­demic run higher, they have en­sured they will be fine, re­gard­less of the out­come for every­one else.

Initial Disclosure: After ex­ten­sive re­search, we have taken a short po­si­tion in shares of Block, Inc. (NYSE: SQ). This re­port rep­re­sents our opin­ion, and we en­cour­age every reader to do their own due dili­gence. Please see our full dis­claimer at the bot­tom of the re­port.

Block, Inc. (SQ), founded by en­tre­pre­neur Jack Dorsey, is a Silicon Valley dar­ling, of­fer­ing pay­ment and mo­bile bank­ing ser­vices for mer­chants and con­sumers.

The com­pany, then named Square, be­gan op­er­a­tions in 2009 and emerged with a dis­rup­tive idea: a tiny card reader that could plug into a smart­phone’s head­phone jack to eas­ily en­able artists and ven­dors to take credit card pay­ments.

From there, the com­pany de­vel­oped tablet-style point-of-sale sys­tems, a mar­ket that grew im­mensely, re­sult­ing in strong com­pe­ti­tion in­clud­ing from com­pa­nies like Toast and Clover. Once the point-of-sale mar­ket adapted to Block’s dis­rup­tion, the com­pany looked else­where to keep its growth story alive.

In 2013, a month fol­low­ing PayPal’s ac­qui­si­tion of Venmo, Block launched Square Cash, later re­branded Cash App. It aimed to com­pete with Venmo by pro­vid­ing fi­nan­cial ser­vices to con­sumers, start­ing with a peer-to-peer mo­bile app where users could send and re­ceive money.

In 2014, fol­low­ing the surge of the peer to peer lend­ing mar­ket, Block be­gan of­fer­ing short-term loans to its mer­chant base, with Block re­ceiv­ing a per­cent­age of every trans­ac­tion un­til the loans were re­paid. [Pg. 5] [1,2,3]

In 2017, Block ex­panded to­ward more tra­di­tional bank­ing by in­tro­duc­ing the “Cash Card,” a pre­paid debit card al­low­ing users to trans­act out­side of their vir­tual Cash App wal­let. [Pg. 10]

In 2018, fol­low­ing bit­coin’s spike in price and pop­u­lar­ity, Cash App al­lowed users to move funds in their Cash App wal­let into Bitcoin.

By 2019, fol­low­ing Robinhood’s surge in pop­u­lar­ity, Block also be­gan of­fer­ing free stock trad­ing. [Pg. 14]

Prior to 2020, the mer­chant ser­vices side of Block’s busi­ness drove the com­pa­ny’s prof­itabil­ity. As of the end of 2019, mer­chant ser­vices ac­counted for $1.39 bil­lion in gross profit, com­pared to the con­sumer-fac­ing Cash App, which ac­counted for only $457.6 mil­lion of gross profit.[1]

That changed dur­ing the pan­demic, as many mer­chant busi­nesses locked down and in­di­vid­u­als ac­ti­vated mil­lions of Cash App ac­counts to re­ceive gov­ern­ment stim­u­lus and un­em­ploy­ment pay­ments.

By the end of 2019, Cash App had 24 mil­lion monthly ac­tive users, ac­cord­ing to the com­pa­ny’s Q4 let­ter to share­hold­ers. [Pg. 2] By the end of 2020, Cash App re­ported 36 mil­lion monthly ac­tives, which has since grown to 51 mil­lion. [Pg. 2]

The ex­plo­sion of user growth re­sulted in higher gross profit. By the end of 2020, Cash App gross profit reached $1.2 bil­lion, a 170% growth rate from the prior year, com­pared with mer­chant ser­vices gross profit of $1.5 bil­lion, an 8% growth rate from the prior year.[2]

During the com­pa­ny’s Q3 earn­ings call in November 2021, Jack Dorsey ex­plained how Cash App is an ideal plat­form to in­tro­duce new ser­vices to cus­tomers be­cause of its low cus­tomer ac­qui­si­tion costs, ow­ing to its vi­ral ap­peal:

“So, we have these in­cred­i­ble vi­ral loop and net­work ef­fects in peer-to-peer, and now we get to look for­ward to­wards be­ing more of a com­merce front end.”

In September 2022, Block CFO Amrita Ahuja cited the com­pa­ny’s pur­chase of buy-now-pay-later com­pany Afterpay as an ex­am­ple of this strat­egy, say­ing it cre­ated “an in­cred­i­ble op­por­tu­nity for us, par­tic­u­larly when you think about lever­ag­ing that ca­pa­bil­ity across Cash App 80 mil­lion an­nual ac­tives.”[3]

Investors are now bet­ting that Cash App’s grow­ing mil­lions of users will cre­ate a ready mar­ket for Block’s fu­ture high-mar­gin fi­nan­cial prod­ucts.

“Block ex­pects Cash App Commerce to be one of Cash App’s pri­mary gross profit growth dri­vers in the com­ing years and is in­vest­ing ac­cord­ingly,” Credit Suisse said af­ter a Q3 2022 meet­ing with Block.

Analysts have high­lighted the re­liance on Cash App as the key dri­ver of Block’s stock go­ing for­ward.

“In our view, per­for­mance of the Cash App busi­ness re­mains the most im­por­tant dri­ver of the stock,” Bank of America said in an October 2022 re­port.

As of March 23rd, 2023, 23 sell-side an­a­lysts had a “Buy” rat­ing on Block shares, 5 had a “Hold” rat­ing, and only 2 had a “Sell” rat­ing, ac­cord­ing to MarketBeat.

The com­pa­ny’s promise has drawn in pop­u­lar in­vestors like Cathie Wood, whose ARK Funds hold a com­bined $651.5 mil­lion in Block stock, com­pris­ing just over 5% of the man­ager’s hold­ings, ac­cord­ing to Cathie’s Ark, a site that tracks ARK’s hold­ings.

Block’s rev­enue de­clined ~1% in 2022 ver­sus the prior year. [Pg. 69] On a GAAP ba­sis, Block posted a net loss of $540.7 mil­lion for 2022. Analysts ex­pect Block will lose $208 mil­lion in 2023, ac­cord­ing to FactSet es­ti­mates. Block’s 2022 an­nual re­port warned:

“We have gen­er­ated sig­nif­i­cant net losses in the past, and we in­tend to con­tinue to in­vest in our busi­ness. Thus, we may not be able to main­tain prof­itabil­ity.” [Pg. 25]

Cash App, which an­a­lysts have said is the most im­por­tant dri­ver of the stock, has demon­strated signs of stag­na­tion with slow­ing in­flows and ac­count growth.

Despite its rev­enue slow­down, cur­rent un­prof­itabil­ity, its warn­ing of fu­ture un­prof­itabil­ity, and signs of stag­na­tion at Cash App, in­vestors have awarded Block val­u­a­tion mul­ti­ples that price in an ex­pec­ta­tion of rapid growth. The com­pany trades at an EV/EBITDA mul­ti­ple of 60x and a 2023 for­ward P/E ra­tio of 40x on “adjusted” earn­ings, ac­cord­ing to FactSet.

By com­par­i­son, Block com­peti­tor Paypal trades at an EV/EBITDA mul­ti­ple of 16.6x, and 15.1x ad­justed 2023 earn­ings, sug­gest­ing 62%-72% down­side for Block were it to trade in line with its peer.

On a tan­gi­ble book ba­sis, many fin­tech com­peti­tors such as Affirm, Robinhood, SoFi and Upstart trade be­tween 1.6x to 2.1x, whereas Block trades at ~13x tan­gi­ble book. Even Paypal, which has grown rev­enue and gen­er­ated a GAAP profit every year since re-emerg­ing as a pub­lic com­pany, trades at a 25% dis­count to Block on this ba­sis. A re-rat­ing closer to the peer av­er­age would rep­re­sent 77.5% down­side for Block.

To make up for these fun­da­men­tal re­al­i­ties, Block has ex­ten­sively re­lied on non-GAAP ad­just­ments to re­port growth de­spite weak­en­ing met­rics.

Block trans­formed its $540.7 mil­lion GAAP loss for 2022 into a $613 mil­lion non-GAAP profit, by ad­just­ing out costs, in­clud­ing over $1 bil­lion in 2022 share-based com­pen­sa­tion, ac­cord­ing to fi­nan­cial in­for­ma­tion from the com­pa­ny’s web­site.[4]

Block’s re­liance on non-GAAP ad­just­ments to re­port profit has in­creased sub­stan­tially over time.

We are en­ter­ing a pe­riod where the end­less hype and un­re­al­is­tic val­u­a­tions of past years are be­gin­ning to face the fi­nan­cial re­al­ity of higher in­ter­est rates.

We ex­pect this macro head­wind, com­bined with the other is­sues out­lined be­low, will re­sult in the con­trac­tion of Block’s ex­ag­ger­ated claims of prof­itabil­ity and gen­er­ous val­u­a­tion mul­ti­ples.

Founder Jack Dorsey has de­scribed Block’s busi­ness as hav­ing a “purpose of eco­nomic em­pow­er­ment.”

The com­pany says it has cho­sen to serve the “unbanked” and “underbanked”, seg­ments of the pop­u­la­tion that do not en­gage in tra­di­tional bank­ing due to weak or non-ex­is­tent credit scores, dis­trust of banks and high fees.

In 2017, Dorsey de­scribed how Block planned to con­tinue ser­vic­ing the un­der­banked through Cash App:

“So, from a high level, we want to make sure that we con­tinue to build ser­vices for an un­der­served mar­ket. That has been what has re­ally tied the com­pany to­gether for the past — close to 9 years now.”[5]

The ef­fects of that strat­egy are ap­par­ent. A September 2022 Pew Research Center sur­vey shows that Cash App has tar­geted low-in­come and mi­nor­ity con­sumers.

Block says it pro­vides uniquely easy ac­cess to send­ing and re­ceiv­ing funds, by re­mov­ing fric­tion from the sys­tem and “adding some magic”:[6]

“Our fo­cus is re­ally ob­serv­ing what our cus­tomers are do­ing,” Dorsey said of the com­pa­ny’s ap­proach in 2018, “and then tak­ing as much fric­tion out of that as pos­si­ble while adding some magic at the same time.”[7]

Cash App’s fric­tion­less ap­proach lets users join with just an email or a phone num­ber.

With its prin­ci­pled mis­sion and easy to use Cash App pay­ment plat­form, Block has gained a rep­u­ta­tion as an in­no­va­tor and one of the most well-known pub­licly traded fin­tech com­pa­nies.

By con­trast, what we found is that the com­pa­ny’s pay­ment sys­tem does not seem to of­fer a dis­cernible edge over its key com­peti­tors like PayPal/Venmo, Zelle, or Apple. Rather, it seems to have sim­ply em­braced non-com­pli­ance as a tac­tic to grow its user base.

Our re­search shows that Block has em­braced a tra­di­tion­ally very un­der­banked seg­ment of the pop­u­la­tion: crim­i­nals.

Cash App’s em­brace of non-com­pli­ance be­gins by mak­ing it easy for users to get on the plat­form, easy for them to get back on the plat­form if their ac­counts are closed, and easy to re­main anony­mous or op­er­ate un­der bla­tantly false iden­ti­ties.

As one for­mer said about sign­ing up for Cash App:

“It’s wide open. And if I was a crim­i­nal, I would have done it.”

Another for­mer com­pli­ance em­ployee of a Cash App part­ner­ship told us, “every crim­i­nal has a Square Cash App ac­count.”

There is even a gang named af­ter Cash App: In 2021, Baltimore au­thor­i­ties ar­rested mem­bers of the “Cash App” gang, who were charged with dis­tri­b­u­tion of fen­tanyl and co­caine in a West Baltimore neigh­bor­hood, ac­cord­ing to a Baltimore news re­port, and crim­i­nal records.

Beyond the gang named af­ter Cash App, nu­mer­ous de­part­ment of Justice in­dict­ments de­tail how Cash App is used by gangs, in­clud­ing the no­to­ri­ous Sinaloa car­tel, to dis­trib­ute drugs like fen­tanyl and metham­phet­a­mine. [1,2,3]

In one ex­am­ple, a crim­i­nal com­plaint against an al­leged meth dis­tri­b­u­tion ring de­scribes how Cash App is com­monly used by gangs to trans­fer il­licit drug pro­ceeds:

In May 2020, Block CEO Jack Dorsey spoke at a JPMorgan in­vestor con­fer­ence about how Cash App was mak­ing its way into pop­u­lar cul­ture through thou­sands of men­tions in hip hop mu­sic:

“We have a very main­stream cus­tomer for Cash App. And ev­i­dence of this is — I talked about this on the call, maybe on the stage be­fore, but the num­ber of hip-hop songs that in­clude the phrase Cash App or even named Cash App is pretty in­cred­i­ble. I think it’s over 1,000 or 2,000 right now.”[8]

At JPMorgan’s 2021 con­fer­ence, Dorsey again told in­vestors about the large num­ber of songs named af­ter Cash App and de­scribed how mu­sic had be­come a way to share with oth­ers how valu­able the app is to them per­son­ally, pro­vid­ing them with “so much util­ity.”[9]

A re­view of songs men­tion­ing Cash App shows that the artists are not gen­er­ally rap­ping about Cash App’s smooth user in­ter­face and ro­bust soft­ware in­te­gra­tion toolkit.

Instead, lyrics de­scribe how easy it is to move money through Cash App to fa­cil­i­tate fraud, traf­fic drugs, or even pay for mur­der.

[Listen to a com­pi­la­tion of songs men­tion­ing Cash App here]

Block pro­moted the video for 22Gz’s song “Cash App” which de­scribed a turf war in which the artist pays hit­men through Cash App to mur­der his op­po­si­tion, then stuffs their bod­ies in trash bags. The video opens with the artist show­ing his Cash App as he makes a ‘throat-slitting’ mo­tion.

Aperiodic tiling, in which shapes can fit to­gether to cre­ate in­fi­nite pat­terns that never re­peat, has fas­ci­nated math­e­mati­cians for decades, but un­til now no one knew if it could be done with just one shape

Aperiodic tiling, in which shapes can fit to­gether to cre­ate in­fi­nite pat­terns that never re­peat, has fas­ci­nated math­e­mati­cians for decades, but un­til now no one knew if it could be done with just one shape

This sin­gle shape pro­duces a pat­tern that never re­peats

Mathematicians have dis­cov­ered a sin­gle shape that can be used to cover a sur­face com­pletely with­out ever cre­at­ing a re­peat­ing pat­tern. The long-sought shape is sur­pris­ingly sim­ple but has taken decades to un­cover — and could find uses in every­thing from ma­te­r­ial sci­ence to dec­o­rat­ing.

Simple shapes such as squares and equi­lat­eral tri­an­gles can tile, or snugly cover a sur­face with­out gaps, in a re­peat­ing pat­tern that will be fa­mil­iar to any­one who has stared at a bath­room wall. Mathematicians are in­ter­ested in …

Skip to con­tent

Sign up

Reference im­ple­men­ta­tion of the Transformer ar­chi­tec­ture op­ti­mized for Apple Neural Engine (ANE)

This com­mit does not be­long to any branch on this repos­i­tory, and may be­long to a fork out­side of the repos­i­tory.

Name al­ready in use

A tag al­ready ex­ists with the pro­vided branch name. Many Git com­mands ac­cept both tag and branch names, so cre­at­ing this branch may cause un­ex­pected be­hav­ior. Are you sure you want to cre­ate this branch?

Use Git or check­out with SVN us­ing the web URL.

Work fast with our of­fi­cial CLI. Learn more.

Please sign in

to use Codespaces.

If noth­ing hap­pens, down­load GitHub Desktop and try again.

If noth­ing hap­pens, down­load GitHub Desktop and try again.

If noth­ing hap­pens, down­load Xcode and try again.

Your code­space will open once ready.

There was a prob­lem prepar­ing your code­space, please try again.

Permalink

Reference im­ple­men­ta­tion of the Transformer ar­chi­tec­ture op­ti­mized for Apple Neural Engine (ANE)

You can’t per­form that ac­tion at this time.

You signed in with an­other tab or win­dow. Reload to re­fresh your ses­sion.

You signed out in an­other tab or win­dow. Reload to re­fresh your ses­sion.

Heya. Sorry for not writ­ing for so long. I’ll make up for it with 3000 pages here.

I’m just hop­ping right now. That’s kinda the only way to get me to blog any­more.

I’ve rewrit­ten this post so many times. It’s about AI. But AI is chang­ing so fast that the post is out of date within a few days. So screw it. I’m bust­ing this ver­sion out in one sit­ting.

There is some­thing leg­endary and his­toric hap­pen­ing in soft­ware en­gi­neer­ing, right now as we speak, and yet most of you don’t re­al­ize at all how big it is.

LLMs aren’t just the biggest change since so­cial, mo­bile, or cloud–they’re the biggest thing since the World Wide Web. And on the cod­ing front, they’re the biggest thing since IDEs and Stack Overflow, and may well eclipse them both.

But most of the en­gi­neers I per­son­ally know are sort of squint­ing at it and think­ing, “Is this an­other crypto?” Even the devs at Sourcegraph are skep­ti­cal. I mean, what en­gi­neer is­n’t. Being skep­ti­cal is a sur­vival skill.

Remember I told you how my Amazon shares would have been worth $130 mil­lion USD to­day if I had­n’t been such a skep­tic about how big Amazon was go­ing to get, and un­loaded them all back in 2004-ish. Right? I told you about that? I’m sure I men­tioned it once or twice. Not that I am bit­ter. No.

But did I ever tell you about the time AWS was just a demo on some en­gi­neer’s lap­top? No? Well it was Ruben Ortega and Al Vermeulen. They were walk­ing around the eng de­part­ment at Amazon, show­ing their “web ser­vice” demo to any­one who’d watch it. This was back in maybe… 2003? Ish? They showed us how you could make a ser­vice call over the web, like by hit­ting a URL and send­ing the right query pa­ra­me­ters.

Well lo and be­hold we were skep­ti­cal. Why the hell would you make a ser­vice call over the web? That’s not what it was even de­signed for! Not to men­tion, it ob­vi­ously would­n’t per­form as well as CORBA (Amazon’s stu­pid-ass RPC sys­tem at the time). The whole thing just did­n’t make any sense to us.

We were see­ing the first lit­tle flecks of lava from what would be­come a tril­lion-dol­lar vol­cano of money called AWS and Cloud Computing.

But a lot of us were skep­ti­cal. To most of us, those lit­tle lava flecks looked like fire­flies.

I could tell you a LOT of sto­ries like the web-ser­vices one. Great big shit al­ways starts life as a demo.

What about chat­ting with peo­ple in a browser? Doesn’t mat­ter whether you’re us­ing Facebook, Google Chat, LinkedIn, or just chat­ting with a cus­tomer ser­vice agent: if you’re hav­ing a con­ver­sa­tion with some­one in a browser, all that shit started life as a teeny demo of 2 en­gi­neers send­ing mes­sages back and forth over a “hanging GET” chan­nel back in 2005. Entire in­dus­tries were built on that one lit­tle chan­nel, and it was­n’t even very good.

What about Kubernetes? I re­mem­ber see­ing a demo of that early on, on Brendan Burns’ work lap­top, when it was called mini-Borg. Entire in­dus­tries are be­ing built on Kubernetes, and it’s not even very good ei­ther. 😉 Or look at Docker! Something as in­nocu­ous as linux cgroups, a lit­tle process-iso­la­tion man­ager, be­came the tech­ni­cal foun­da­tion for con­tain­ers, which now ut­terly per­vade our in­dus­try.

If you can build some­thing as big as Amazon Web Services with a stack based on a sim­ple ser­vice call, or whole so­cial net­works and cus­tomer ser­vice suites based on sim­ple browser-to-browser com­mu­ni­ca­tion, or a ro­bust way of de­liv­er­ing and man­ag­ing soft­ware based on a lit­tle process iso­la­tion code, then just imag­ine how big a thing you could build — bear with me here — if you had the god­damn Singularity as your start­ing point?

I mean, I joke, but… I mean… Right? I’m guess­ing you pro­lly missed it in OpenAI’s 98-page GPT-4 tech­ni­cal re­port, but large mod­els are ap­par­ently al­ready prone to dis­cov­er­ing that “power-seeking” is an ef­fec­tive strat­egy for in­creas­ing their own ro­bust­ness. Open the PDF and search for “power-seeking” for a fun and to­tally 100% non-scary read.

You can build truly mas­sive things by build­ing upon lit­tle tech­ni­cal break­throughs.

And folks, this tech­ni­cal break­through? It ain’t lit­tle.

If you’re not pant-pee­ingly ex­cited and wor­ried about this yet, well… you should be.

We did an in­ter­nal poll at Sourcegraph: Do you have pos­i­tive sen­ti­ment or neg­a­tive sen­ti­ment about LLMs for cod­ing? Options were Positive, Negative, and Meh. And lo, it was about ⅔ Meh or Negative (i.e., Skeptics), which I sus­pect is fairly rep­re­sen­ta­tive of the whole in­dus­try.

I asked around, and even as of a cou­ple weeks ago, some devs ques­tioned whether ChatGPT could even write work­ing code, let alone write a full pro­gram by sim­ply by telling it to write it.

So here I am, talk­ing about money vol­ca­noes, and my cowork­ers have formed a huge whirling meh-nado. Which nat­ural dis­as­ter should you be­lieve?

Well I mean, I guess a demo is worth a thou­sand mehs. Let’s try it out.

Let’s have ChatGPT write some Emacs-Lisp code. I’m pick­ing emacs-lisp be­cause it’s a cor­ner case lan­guage, bit of a stress test for the LLM, and be­cause it’s easy for me to try it out in­ter­ac­tively.

OK. I just typed this prompt into ChatGPT:

> Write an in­ter­ac­tive emacs-lisp func­tion that pops to a new buffer, prints out the first para­graph of “A tale of two cities”, and changes all words with ‘i’ in them red. Just print the code with­out ex­pla­na­tion.

Here’s the code it spat out, if you can’t read the screen­shot:

(defun dis­play-tale-of-two-cities ()

(interactive)

(let ((text “It was the best of times, it was the worst of times, it was the age of wis­dom,

it was the age of fool­ish­ness, it was the epoch of be­lief, it was the epoch of in­credulity,

it was the sea­son of Light, it was the sea­son of Darkness, it was the spring of hope, it was

the win­ter of de­spair, we had every­thing be­fore us, we had noth­ing be­fore us, we were all

go­ing di­rect to Heaven, we were all go­ing di­rect the other way — in short, the pe­riod was so

far like the pre­sent pe­riod, that some of its nois­i­est au­thor­i­ties in­sisted on its be­ing

re­ceived, for good or for evil, in the su­perla­tive de­gree of com­par­i­son only.“))

(pop-to-buffer “*A Tale of Two Cities*“)

(erase-buffer)

(insert text)

(goto-char (point-min))

(while (re-search-forward “\\b\\w*\\i\\w*\\b” nil t)

(add-text-properties

(match-beginning 0) (match-end 0)

’(face (:foreground “red”))))))

I copied this code di­rectly into my Emacs ses­sion and ran it, and it did ex­actly what I asked it to do:

As you can see from the screen­shot, I ran the com­mand and it opened a buffer, printed the re­quested text, and then turned all the words con­tain­ing ‘i’ red.

In one shot, ChatGPT has pro­duced com­pletely work­ing code from a sloppy English de­scrip­tion! With voice in­put wired up, I could have writ­ten this pro­gram by ask­ing my com­puter to do it.

And not only does it work cor­rectly, the code that it wrote is ac­tu­ally pretty de­cent emacs-lisp code. It’s not com­pli­cated, sure. But it’s good code.

Of course peo­ple have done much, much fancier things than this. Someone wrote a prod­uct de­scrip­tion on a nap­kin, took a pic­ture, and GPT wrote a work­ing web app that im­ple­ments the prod­uct de­scrip­tion on the nap­kin in the pic­ture. The amount of power here is hon­estly un­known; it’s more like a cav­ern that we haven’t fully ex­plored. And it just gets deeper as the LLMs get big­ger.

I mean, this stuff is un­be­liev­ably pow­er­ful. And yet I am per­sis­tently met with a mix­ture of dis­be­lief and pearl-clutch­ing. Argh, the pearl-clutch­ing! Don’t even get me started on the pearl-clutch­ing. Oh look, now you’ve got me started.

Okay, you asked for it.

One of the cra­zi­est damned things I hear devs say about LLM-based cod­ing help is that they can’t “trust” the code that it writes, be­cause it “might have bugs in it”.

Can you trust code you yeeted over from Stack Overflow? NO!

Can you trust code you copied from some­where else in your code base? NO!

Can you trust code you just now wrote care­fully by hand, your­self? NOOOO!

All you crazy MFs are com­pletely over­look­ing the fact that soft­ware en­gi­neer­ing ex­ists as a dis­ci­pline be­cause you can­not EVER un­der any cir­cum­stances TRUST CODE. That’s why we have re­view­ers. And lin­ters. And de­bug­gers. And unit tests. And in­te­gra­tion tests. And stag­ing en­vi­ron­ments. And run­books. And all of god­damned Operational Excellence. And se­cu­rity check­ers, and com­pli­ance scan­ners, and on, and on and on!

So the next one of you to com­plain that “you can’t trust LLM code” gets a lit­tle badge that says “Welcome to en­gi­neer­ing moth­er­fucker”. You’ve fi­nally learned the se­cret of the trade: Don’t. Trust. Anything!

Peeps, let’s do some re­ally sim­ple back-of-en­ve­lope math. Trust me, it won’t be dif­fi­cult math.

* You get the LLM to draft some code for you that’s 80% com­plete/​cor­rect.

* You tweak the last 20% by hand.

How much of a pro­duc­tiv­ity in­crease is that? Well jeep­ers, if you’re only do­ing 1/5th the work, then you are… punches but­tons on cal­cu­la­tor watch… five times as pro­duc­tive. 😲

When was the last time you got a 5x pro­duc­tiv­ity boost from any­thing that did­n’t in­volve some sort of chem­i­cals?

I’m se­ri­ous. I just don’t get peo­ple. How can you not ap­pre­ci­ate the his­toric change hap­pen­ing right now?

OK time to get con­crete. I’m al­ready on page 7, and my last at­tempt at this blog ran 25+ pages and lan­guished for weeks.

OK sooooo… this is the part that went on for 20 pages be­fore, so let’s just make it reeeeeal sim­ple. One para­graph.

Here is every­thing you need to know about the his­tory of LLMs, for our pur­poses to­day:

Congrats, you’re all caught up on the his­tory of LLMs. Go watch this amaz­ing video for how to im­ple­ment it in Python.

OK now we can talk cod­ing as­sis­tants. They’re just a thing that sits in your IDE and they talk to the LLM for you.

Depending on the par­tic­u­lar as­sis­tant, they can read and ex­plain code, doc­u­ment code, write code, au­to­com­plete it, di­ag­nose is­sues, and even per­form ar­bi­trary IDE tasks through “agents” that give the LLM ro­botic pow­ers, in­clud­ing the abil­ity to wield and tar­get laser guns, if some­one wants to put in the work. Some as­sis­tants also un­der­stand your pro­ject en­vi­ron­ment and can an­swer ques­tions about build tar­gets, branches, your IDE, etc.

So, al­ready pretty cool. Right?

But now they are be­gin­ning to be able to per­form more com­plex tasks, such as gen­er­at­ing a PR from the diffs on the cur­rent branch, in­clud­ing a de­tailed com­mit mes­sage sum­ma­riz­ing the changes.

Some as­sis­tants have a con­ver­sa­tional/​chat in­ter­face, too. This kind can do every­thing a bot like ChatGPT can do, like draft­ing emails, or an­swer­ing ran­dom ques­tions about the code base or the en­vi­ron­ment.

I per­son­ally pre­fer a cod­ing as­sis­tant with a chat in­ter­face. In part be­cause I can type, but also be­cause it makes them a plat­form. I can build my own work­flows. Bonus points if they ex­pose the un­der­ly­ing plat­form bits with APIs.

I guess the sim­plest way to think about it would be a sort of “real-time in-IDE Stack Overflow” cou­pled with a re­ally pow­er­ful new set of boil­er­plate au­toma­tion tasks.

OK, con­grats again — you’re up to speed on what LLM-based cod­ing as­sis­tants can do. It’s… pretty much any­thing. You could hook it up to out­bound email and tell it to sell it­self. Sky’s the limit. At this point we’re more lim­ited by imag­i­na­tion than by tech­nol­ogy.

So! Yeah. Coding as­sis­tants. I hope by now you get how pow­er­ful they’re go­ing to be. They may take dif­fer­ent shapes and forms, but they’re all go­ing to be in­cred­i­bly badass be­fore very much longer.

Let’s dig a lit­tle into how they un­der­stand your per­sonal code, and then we’re ready to party. 🎉

LLMs are trained on an ab­solutely stag­ger­ing amount of data… but that does­n’t in­clude your code.

There are two ba­sic ap­proaches to mak­ing the LLM smarter about your code. The first is to fine-tune (or train) on your code. This is not a busi­ness model that has been fully fleshed out yet, but it’s com­ing. And im­por­tantly it’s only part of the pic­ture.

The other way is to bring in a search en­gine. You can think of it as three re­lated sce­nar­ios:

* A raw LLM is like a Harvard CS grad who knows a lot about cod­ing and took a magic mush­room about 4 hours ago, so it’s mostly worn off, but not to­tally.

* Fine-tuning it on your code base is like hav­ing it study your code care­fully, which means it will give bet­ter an­swers in gen­eral.

* Incorporating a search en­gine, much like for hu­mans, makes the AI even more ef­fec­tive, be­cause it can an­swer di­rect queries very quickly. And im­por­tantly, be­cause the search en­gine can be used to pop­u­late the query con­text.

Meaning, a search en­gine can be use­ful twice per query — once when fig­ur­ing out how to de­scribe and con­tex­tu­al­ize the query, and again po­ten­tially when an­swer­ing the query.

* Meaning, a search en­gine can be use­ful twice per query — once when fig­ur­ing out how to de­scribe and con­tex­tu­al­ize the query, and again po­ten­tially when an­swer­ing the query.

You talk to LLMs by send­ing them an ac­tion or query, plus some rel­e­vant con­text. So for in­stance, if you want it to write a unit test for a func­tion, then you need to pass along that whole func­tion, along with any other rel­e­vant code (e.g. test-fix­ture code) so that it gets the test right.

That con­text you send over is called the con­text win­dow, and I think of it as the “cheat sheet” of in­for­ma­tion that you pass along as part of your query.

And folks, it ain’t much. It’s al­most ex­actly like a 2-sided in­dex card vs your whole text­book, for an exam. They give you be­tween 4k and 32k to­kens of 3-4 char­ac­ters each, so at best, maybe 100k of text, to in­put into the LLM as con­text for your query. That 100k cheat sheet is how you tell the LLM about your code.

In an ideal world, you’d just pass your en­tire code base in with each query. In fact, Jay Hack just tweeted a graph show­ing how the lat­est con­text win­dow size in GPT-4 com­pares to some pop­u­lar code bases:

Which is kind of ex­cit­ing… un­til you re­al­ize that it’s still just in­cred­i­bly tiny com­pared to real-world code bases. It’s an in­dex card vs a text­book… just a slightly big­ger in­dex card.

That cheat sheet is all you get. That’s how you talk to an LLM. You pass it a cheat sheet.

Which means what goes ON that cheat sheet, as you can prob­a­bly imag­ine, is re­ally re­ally im­por­tant.

And with that, friends, we are fi­nally ready for the punch­line, the party, and the demo.

There are, by my last count, ap­prox­i­mately 13 hillion frillion jil­lion LLM-backed cod­ing as­sis­tants out there, as of mid-March. But they are all in a des­per­ate race to the bot­tom, be­cause they’re all us­ing the ex­act same raw ma­te­ri­als: An LLM, your IDE, your code base, and that pesky lit­tle con­text win­dow.

Nobody can dif­fer­en­ti­ate on the LLM; they’re all about the same. And the IDE and your code base are the same. All they can try to dif­fer­en­ti­ate on is their UI and work­flows, which they’re all go­ing to copy off each other. Good for you, bad for them.

The punch­line, and it’s hon­estly one of the hard­est things to ex­plain, so I’m go­ing the faith-based route to­day, is that all the win­ners in the AI space will have data moats.

A “data moat” is, in a nut­shell, hav­ing ac­cess to some data that oth­ers do not have ac­cess to.

You need a data moat to dif­fer­en­ti­ate your­self in the LLM world.

Why? Because the data moat is how you pop­u­late the con­text win­dow (“cheat sheet”).

If you can’t feed the LLM your whole code base, and you can only show it 100k char­ac­ters at a time, then you’d bet­ter be re­ally god­damn good at fetch­ing the right data to stuff into that 100k-char win­dow. Because that’s the only way to af­fect the qual­ity of the LLM’s out­put!

Starbucks CEO Laxman Narasimhan told em­ploy­ees Thursday that he’ll work a half day every month at one of the cof­fee gi­ant’s lo­ca­tions.

Narasimhan took the helm on Monday and will lead the com­pa­ny’s share­holder meet­ing Thursday. He joined Starbucks in October, spend­ing months learn­ing the busi­ness. That in­cluded 40 hours of barista train­ing and don­ning the baris­tas’ iconic green apron.

But he’s not hang­ing up the apron just yet.

“To keep us close to the cul­ture and our cus­tomers, as well as to our chal­lenges and op­por­tu­ni­ties, I in­tend to con­tinue work­ing in stores for a half day each month, and I ex­pect each mem­ber of the lead­er­ship team to also en­sure our sup­port cen­ters stay con­nected and en­gaged in the re­al­i­ties of our stores for dis­cus­sion and im­prove­ment,” he wrote in a let­ter to em­ploy­ees Thursday morn­ing.

His pledge comes at a rocky time for the com­pa­ny’s re­la­tion­ship with its baris­tas. As of Friday, more than 190 com­pany-owned Starbucks lo­ca­tions have voted to union­ize, ac­cord­ing to National Labor Relations Board data. Workers have cited un­safe work­ing con­di­tions, un­der­staffing and un­re­li­able sched­ul­ing as some of the rea­sons why they’re in fa­vor of a union.

Before join­ing Starbucks, Narasimhan was chief ex­ec­u­tive of Reckitt, which owns brands like Lysol and Durex. Previously, he worked at PepsiCo and McKinsey.