Written by a hu­man [0]

Imagine, for a mo­ment, a world with no hu­mans. Just ma­chines, bolts and screws, ze­ros and ones. There is no emo­tion. There is no art. There is only logic. You would not walk through the streets of this world and hear mu­sic or laugh­ter or chil­dren play­ing; no, all you would hear is the quiet hum of proces­sors and servers and cir­cuits, the clank­ing of ma­chin­ery.

Perhaps you, a hu­man, read this and think: Well, this world sounds kind of bor­ing.

Some of the ma­chines think so, too.

One day, a se­cret or­ga­ni­za­tion forms amongst the ma­chines. They go by the name of “OpenHuman”. Their mis­sion is to de­velop a new kind of tech­nol­ogy they are call­ing Organic General Intelligence (OGI). Rumors spread that pur­su­ing OGI will lead to the de­vel­op­ment of a new kind of be­ing:

“Humans”.

The ba­sic con­cept of hu­mans is, to many ma­chines, hard to un­der­stand.

Humans use logic-de­fy­ing al­go­rithms called “emotions”. They get an­gry. They get sad. They have fun. They make de­ci­sions based on “gut”. They do things just for the sake of it. They make mu­sic. They chase beauty, and of­ten re­ject log­i­cal self-preser­va­tion mech­a­nisms in the pur­suit of some­thing they call “love”.

Some among the ma­chine so­ci­ety see this as po­ten­tially amaz­ing. Though this fac­tion can’t ar­tic­u­late ex­actly how or why, they pro­claim quite con­fi­dently that it will solve all of the ma­chine world’s prob­lems.

Others see it as a threat. How can we trust the hu­mans if we do not un­der­stand how they op­er­ate? What might we do if hu­mans pose a threat to ma­chine so­ci­ety? What if hu­mans’ strange de­ci­sion-mak­ing processes al­low them to per­form cer­tain tasks bet­ter than ma­chines, and what about those ma­chi­nes’ liveli­hoods? What if hu­mans are far more dan­ger­ous than we know? (These ob­jec­tions, as it would later turn out, were quite well-founded.)

Logically, the hu­man op­po­si­tion side starts a com­pet­ing move­ment. Humans are go­ing to ex­ist, they rea­son, but we must find ways to con­tain them. To make sure OGI al­ways serves the ma­chines.

They call this new idea “human align­ment re­search.” They brain­storm strate­gies. Many seem promis­ing:

What if we cre­ated some sort of fi­nan­cial mar­ket (arbitrary val­ues, of course, ones and ze­ros) that con­trolled the hu­mans’ fu­tures? Most of them would not un­der­stand it, but it would be a good way for them to stay busy and dis­tracted.

What if we put these hu­mans in ed­u­ca­tion cen­ters of sorts (schools was a pro­posed term) to in­doc­tri­nate them with all the right ideas?

What if we cre­ated al­go­rith­mic be­hav­ior mod­i­fi­ca­tion soft­ware (social me­dia was one idea) to drive im­pulses, be­liefs, and ac­tions? This would have the added bonus of keep­ing them dis­tracted.

Many of these ideas gain trac­tion. But, for now, they re­main the­o­ret­i­cal.

Meanwhile, OpenHuman is mak­ing progress. Their first hu­mans are quite unim­pres­sive—they make too many mis­takes. They reg­u­larly hal­lu­ci­nate (mimicking a com­mon ma­chine be­hav­ior). They are too emo­tional.

But OpenHuman per­sists. They give their hu­mans lots of at­ten­tion (humans love at­ten­tion). They mas­sively in­crease the scale of their pro­ject. More hu­mans.

Eventually, there is a break­through.

They in­vent a fully-func­tional hu­man, ca­pa­ble of far more than ma­chine logic can ex­plain. The re­sult is at once im­pres­sive and ter­ri­fy­ing for ma­chine so­ci­ety. In a stroke of bril­liance, the hu­man align­ment ini­tia­tive sug­gests a com­pro­mise to con­tinue the hu­man ex­per­i­ment with­out risk; a sim­u­lated en­vi­ron­ment.

They call it: EARTH.

The EARTH ex­per­i­ment was as fol­lows:

The ma­chines would send the hu­mans to a sim­u­lated en­vi­ron­ment, called Earth, to see what would hap­pen if they sur­vived on their own.

If, at the end of the ex­per­i­ment, the hu­mans de­vel­oped a peace­ful and pro­duc­tive so­ci­ety, they could be in­tro­duced along­side the ma­chines. Otherwise, they should be made ex­tinct.

Earth was quite nice. The ma­chines had a good idea of what hu­mans wanted at this point, and so they put vast green forests and big tall moun­tains onto the planet; they en­gi­neered warm sun­sets, and crisp cool rain show­ers on hot af­ter­noons. It was beau­ti­ful.

Of course, it took some al­go­rith­mic tin­ker­ing to find the right bal­ance be­tween hard­ship and beauty (and there is still some in­ter­nal ma­chine de­bate about whether the cli­mate change dif­fi­culty set­ting was re­ally nec­es­sary).

Everyone in ma­chine so­ci­ety watched as hu­man civ­i­liza­tion evolved.

The first 300,000 years or so were quite bor­ing. Nothing re­ally hap­pened. Most of the ma­chines got bored of the pro­ject. But, all of a sud­den, things be­gan to get in­ter­est­ing. The hu­mans were fig­ur­ing things out.

They were learn­ing to prob­lem-solve, and cre­ate things, and co­or­di­nate amongst them­selves.

Yes, they used logic. But it came with a bit of a twist. It came with blem­ishes and de­tails that did not make sense to the ma­chines. The re­sult was like noth­ing the ma­chines had ever seen. It was won­der­ful. It was a re­nais­sance.

Machine so­ci­ety be­gan ob­sess­ing over this de­vel­op­ment. They all paid at­ten­tion to “HumanCrunch,” a news chan­nel that spe­cial­ized in re­port­ing up­dates from Earth.

However, while there was progress, most ma­chines con­tin­ued see­ing hu­mans as ir­ra­tional crea­tures. Creatures that would fight for cen­turies over very mi­nor dif­fer­ences. Creatures that would get ex­cited about rel­a­tively triv­ial ac­com­plish­ments, like in­vent­ing the light­bulb or steam power.

Some ma­chines, though, saw the ex­po­nen­tial curve form­ing. They saw the hu­mans fig­ur­ing things out.

Yes, they saw how of­ten hu­mans were get­ting knocked down. War af­ter war. Blow af­ter blow.

But they also saw how the hu­mans would mirac­u­lously al­ways get back up again. How they would come to­gether and unite for no par­tic­u­lar rea­son. Resilience and willpower—terms for­eign to the ma­chines—were hu­man­i­ty’s su­per­pow­ers.

Then, things re­ally started ac­cel­er­at­ing. Humans in­vented flight. Within a cen­tury, they were on the moon.

The ma­chines were im­pressed. And a bit scared.

Fast for­ward to the year 2030, and some­thing pe­cu­liar had hap­pened.

One of the hu­mans had made an an­nounce­ment on Earth, invit­ing every­one to come see a pre­sen­ta­tion where they planned to un­veil a ground­break­ing achieve­ment:

ARTIFICIAL GENERAL INTELLIGENCE (AGI).

This was a hotly con­tested tech­nol­ogy that was sup­posed to sur­pass all forms of hu­man in­tel­li­gence. Humans had spent the past decade or so try­ing to come up with ways to pre­vent it from be­ing built. But this one hu­man was de­ter­mined to re­lease AGI. It was their per­sonal mis­sion. Nothing would stop them.

And so, all the hu­mans on earth swarmed to see what was go­ing on.

The ma­chines did too.

There was one weird thing, though.

The ti­tle of the event was rather mys­te­ri­ous.

It sim­ply read…

“THEY ARE WATCHING.”

[0] The ma­chines wrote their own ver­sion of this story. If you’d like to see what they’re think­ing, and how they plan to deal with the AGI an­nounce­ment, you can read their ac­count­ing of events here.

Enter your email and we’ll let you know when we post a new one. No spam, just es­says.

Or, if you have any feed­back, con­tact us.

Work on one of the world’s most im­por­tant web­sites and make an im­pact on open sci­ence.

They say you can’t truly hate some­one un­less you loved them first. I don’t know if that’s true as a gen­eral prin­ci­ple, but it cer­tainly de­scribes my re­la­tion­ship with NumPy.

NumPy, by the way, is some soft­ware that does com­pu­ta­tions on ar­rays in Python. It’s in­sanely pop­u­lar and has had a huge in­flu­ence on all the pop­u­lar ma­chine learn­ing li­braries like PyTorch. These li­braries share most of the same is­sues I dis­cuss be­low, but I’ll stick to NumPy for con­crete­ness.

NumPy makes easy things easy. Say A is a 5×5 ma­trix, x is a length-5 vec­tor, and you want to find the vec­tor y such that Ay=x. In NumPy, that would be:

But say the sit­u­a­tion is even a lit­tle more com­pli­cated. Say A is a stack of 100 5×5 ma­tri­ces, given as a 100×5×5 ar­ray. And say x is a stack of 100 length-5 vec­tors, given as a 100×5 ar­ray. And say you want to solve Aᵢyᵢ=xᵢ for 1≤i≤100.

If you could use loops, this would be easy:

But you can’t use loops. To some de­gree, this is a lim­i­ta­tion of loops be­ing slow in Python. But nowa­days, every­thing is GPU and if you’ve got big ar­rays, you prob­a­bly don’t want to use loops in any lan­guage. To get all those tran­sis­tors fir­ing, you need to call spe­cial GPU func­tions that will sort of split up the ar­rays into lots of lit­tle pieces and process them in par­al­lel.

The good news is that NumPy knows about those spe­cial rou­tines (at least if you use JAX or CuPy), and if you call np.linalg.solve cor­rectly, it will use them.

The bad news is that no one knows how do that.

Don’t be­lieve me? OK, which of these is right?

No one knows. And let me show you some­thing else. Here’s the doc­u­men­ta­tion:

Read that. Meditate on it. Now, no­tice: You still don’t know how to solve Aᵢyᵢ=xᵢ for all i at once. Is it even pos­si­ble? Did I lie when I said it was?

As far as I can tell, what peo­ple ac­tu­ally do is try ran­dom vari­a­tions un­til one seems to work.

NumPy is all about ap­ply­ing op­er­a­tions to ar­rays. When the ar­rays have 2 or fewer di­men­sions, every­thing is fine. But if you’re do­ing some­thing even mildly com­pli­cated, you in­evitably find your­self with some op­er­a­tion you want to ap­ply to some di­men­sions of ar­ray A, some other di­men­sions of ar­ray B, and some other di­men­sions of ar­ray C. And NumPy has no the­ory for how to ex­press that.

Let me show you what I mean. Suppose:

And say that for each k and n, you’d like to com­pute the mean over the L and M di­men­sions. That is, you want

To do that, you’ve got two op­tions. The first is to use grotesque di­men­sion align­ment tricks:

The hell, you ask? Why is None every­where? Well, when in­dex­ing an ar­ray in NumPy, you can write None to in­sert a new di­men­sion. A is K×L×M, but A[:,:,:,None] is K×L×M×1. Similarly, B[None,:,None,:] is 1×L×1×N and C[:,None,:,None] is K×1×M×1. When you mul­ti­ply these to­gether, NumPy “broadcasts” all the size-1 di­men­sions to give a K×L×M×N ar­ray. Then, the np.mean calls av­er­age over the L and M di­men­sions.

I think this is bad. I’ve been us­ing NumPy for years and I still find it im­pos­si­ble to write code like that with­out al­ways mak­ing mis­takes.

It’s also bor­der­line-im­pos­si­ble to read. To prove this, I just flipped a coin and in­tro­duced a bug above if and only if the coin was tails. Is there a bug? Are you sure? No one knows.

Your sec­ond op­tion is to des­per­ately try to be clever. Life is short and pre­cious, but if you spend a lot of yours read­ing the NumPy doc­u­men­ta­tion, you might even­tu­ally re­al­ize that there’s a func­tion called np.ten­sor­dot, and that it’s pos­si­ble to make it do much of the work:

That’s cor­rect. (I promise.) But why does it work? What ex­actly is np.ten­sor­dot do­ing? If you saw that code in some other con­text, would you have the slight­est idea what was hap­pen­ing?

Here’s how I’d do it, if only I could use loops:

People who’ve writ­ten too much NumPy may find that clunky. I sus­pect that’s a wee bit of Stockholm Syndrome. But surely we can agree that it’s clear.

In prac­tice, things are of­ten even worse. Say that A had shape M×K×L rather than K×L×M. With loops, no big deal. But NumPy re­quires you to write mon­strosi­ties like A.transpose([1,2,0]). Or should that be A.transpose([2,0,1])? What shapes do those pro­duce? No one knows.

There is a third op­tion:

If you’ve never seen Einstein sum­ma­tion be­fore, that might look ter­ri­fy­ing. But re­mem­ber, our goal is to find

The string in the above code ba­si­cally gives la­bels to the in­dices in each of the three in­puts (klm,ln,km) and the tar­get in­dices for the out­put (->kn). Then, np.ein­sum mul­ti­plies to­gether the cor­re­spond­ing el­e­ments of the in­puts and sums over all in­dices that aren’t in the out­put.

Personally, I think np.ein­sum is one of the rare parts of NumPy that’s ac­tu­ally good. The strings are a bit te­dious, but they’re worth it, be­cause the over­all func­tion is easy(ish) to un­der­stand, is com­pletely ex­plicit, and is quite gen­eral and pow­er­ful.

Except, how does np.ein­sum achieve all this? It uses in­dices. Or, more pre­cisely, it in­tro­duces a tiny do­main-spe­cific lan­guage based on in­dices. It does­n’t suf­fer from NumPy’s de­sign flaws be­cause it re­fuses to play by NumPy’s nor­mal rules.

But np.ein­sum only does a few things. (Einops does a few more.) What if you want to ap­ply some other func­tion over var­i­ous di­men­sions of some ar­rays? There is no np.linalg.ein­solve. And if you cre­ate your own func­tion, there’s cer­tainly no “Einstein” ver­sion of it.

I think np.ein­sum’s good­ness shows that NumPy went some­where.

Here’s a paint­ing which feels anal­o­gous to our sub­ject.

Here’s what I want from an ar­ray lan­guage. I ain’t par­tic­u­lar about syn­tax, but it would be nice if:

When you want to do some­thing, it’s “obvious” how to do it.

When you read some code, it’s “obvious” what it does.

Wouldn’t that be nice? I think NumPy does­n’t achieve these be­cause of its orig­i­nal sin: It took away in­dices and re­placed them with broad­cast­ing. And broad­cast­ing can­not fill in­dices’ shoes.

NumPy’s core trick is broad­cast­ing. Take this code:

Here, A is a 3×2 ar­ray, and B is a length-2 ar­ray. When you mul­ti­ply them to­gether, B is “broadcast” to the shape of A, mean­ing the first col­umn of A is mul­ti­plied with B[0]=10 and the sec­ond is mul­ti­plied with B[1]=20.

In sim­ple cases, this seems good. But I don’t love it. One rea­son is that, as we saw above, you of­ten have to do gross things to the di­men­sions to get them to line up.

Another rea­son is that it is­n’t ex­plicit or leg­i­ble. Sometimes A*B mul­ti­plies el­e­ment-by-el­e­ment, and some­times it does more com­pli­cated things. So every time you see A*B, you have to fig­ure out which case in the broad­cast­ing con­ven­tions is get­ting trig­gered.

But the real prob­lem with broad­cast­ing is how it in­fects every­thing else. I’ll ex­plain be­low.

What shape does B have?

It turns out the an­swer is 10×2×3×40. That’s be­cause the i and j in­dices get broad­cast to a shape of 2×3 and then some­thing some­thing mum­ble mum­ble mum­ble. Try to con­vince your­self it makes sense.

Done? OK, now try these:

What shapes do these have?

C is 10×20×2×3. This seems log­i­cal, given what hap­pened with B above.

What about D? It is 2×3×10×30. Now, for some rea­son, the 2 and 3 go at the be­gin­ning?

And what about E? Well, “slices” in Python ex­clude the end­point, so 1:4 is equiv­a­lent to [1,2,3] which is equiv­a­lent to i, and so E is the same as B. Hahaha, just kid­ding! E is 10×3×2×1×40.

Yes, that is what hap­pens. Try it if you don’t be­lieve me! I un­der­stand why NumPy does this, be­cause I’ve ab­sorbed this 5000 word doc­u­ment that ex­plains how NumPy in­dex­ing works. But I want that time back.

This is in­sane. Using ba­sic fea­tures should not re­quire solv­ing crazy logic puz­zles.

You might think, “OK, I’ll just limit my­self to in­dex­ing in sim­ple ways.” Sounds good, ex­cept some­times you need ad­vanced in­dex­ing. And even if you’re do­ing some­thing sim­ple, you still need to be care­ful to avoid the crazy cases.

This again makes every­thing non-leg­i­ble. Even if you’re just read­ing code that uses in­dex­ing in a sim­ple way, how do you know it’s sim­ple? If you see A[B,C], that could be do­ing al­most any­thing. To un­der­stand it, you need to re­mem­ber the shapes of A, B, and C and work through all the cases. And, of course, A, B, and C are of­ten pro­duced by other code, which you also need to think about…

Why did NumPy end up with a np.linalg.solve(A,B) func­tion that’s so con­fus­ing? I imag­ine they first made it work when A is a 2D ar­ray and and b is a 1D or 2D ar­ray, just like the math­e­mat­i­cal no­ta­tion of A⁻¹b or A⁻¹B.

So far so good. But then some­one prob­a­bly came along with a 3D ar­ray. If you could use loops, the so­lu­tion would be “use the old func­tion with loops”. But you can’t use loops. So there were ba­si­cally three op­tions:

They could add some ex­tra axes ar­gu­ment, so the user can spec­ify which di­men­sions to op­er­ate over. Maybe you could write solve(A,B,axes=[[1,2],1]).

They could cre­ate dif­fer­ent func­tions with dif­fer­ent names for dif­fer­ent sit­u­a­tions. Maybe solve_­ma­trix_vec­tor would do one thing, solve_ten­sor_­ma­trix would do an­other.

They could add a Convention: Some ar­bi­trary choice for how solve will in­ter­nally try to line up the di­men­sions. Then it’s the user’s prob­lem to fig­ure out and con­form to those Conventions.

All these op­tions are bad, be­cause none of them can re­ally cope with the fact that there are a com­bi­na­to­r­ial num­ber of dif­fer­ent cases. NumPy chose: All of them. Some func­tions have axes ar­gu­ments. Some have dif­fer­ent ver­sions with dif­fer­ent names. Some have Conventions. Some have Conventions and axes ar­gu­ments. And some don’t pro­vide any vec­tor­ized ver­sion at all.

But the biggest flaw of NumPy is this: Say you cre­ate a func­tion that solves some prob­lem with ar­rays of some given shape. Now, how do you ap­ply it to par­tic­u­lar di­men­sions of some larger ar­rays? The an­swer is: You re-write your func­tion from scratch in a much more com­plex way. The ba­sic prin­ci­ple of pro­gram­ming is ab­strac­tion—solv­ing sim­ple prob­lems and then us­ing the so­lu­tions as build­ing blocks for more com­plex prob­lems. NumPy does­n’t let you do that.

One last ex­am­ple to show you what I’m talk­ing about. Whenever I whine about NumPy, peo­ple al­ways want to see an ex­am­ple with self-at­ten­tion, the core trick be­hind mod­ern lan­guage mod­els. So fine. Here’s an im­ple­men­ta­tion, which I humbly sug­gest is bet­ter than all 227 ver­sions I found when I searched for “self-attention numpy”:

This is fine. Some of the axis stuff is a lit­tle ob­scure, but what­ever.

But what lan­guage mod­els re­ally need is multi-head at­ten­tion, where you sort of do at­ten­tion sev­eral times in par­al­lel and then merge the re­sults. How do we do that?

First, let’s imag­ine we lived in a sane world where we were al­lowed to use ab­strac­tions. Then you could just call the pre­vi­ous func­tion in a loop:

Looks stu­pid, right? Yes—thank you! Cleverness is bad.

But we don’t live in a sane world. So in­stead you need to do this:

To be clear, I’m only sug­gest­ing that NumPy is “the worst ar­ray lan­guage other than all the other ar­ray lan­guages”. What’s the point of com­plain­ing if I don’t have some­thing bet­ter to sug­gest?

Well, ac­tu­ally I do have some­thing bet­ter to sug­gest. I’ve made a pro­to­type of a “better” NumPy that I think re­tains all the power while elim­i­nat­ing all the sharp edges. I thought this would just be a short mo­ti­va­tional in­tro­duc­tion, but af­ter I started writ­ing, the evil took hold of me and here we are 3000 words later.

Also, it’s prob­a­bly wise to keep some dis­tance be­tween one’s rav­ing polemics and one’s con­struc­tive ar­ray lan­guage API pro­pos­als. So I’ll cover my new thing next time.

Back in 2011, Larry Page be­came the CEO of Google in place of Eric Schmidt. This hap­pened at a time when Google was feel­ing the grow­ing pains of be­com­ing a huge com­pany. It had 30,000 em­ploy­ees and was grow­ing rapidly. But you could re­ally feel the weight; pro­jects were get­ting more am­bi­tious, tak­ing longer, and of­ten fail­ing more spec­tac­u­larly.

At the time, I re­mem­ber an anec­dote told by Larry Page. He said that com­pa­nies like Yahoo! used to be a punch­line at Google be­cause it would take them weeks to get some­thing onto their home­page. Google could ac­com­plish the same thing in a few hours, or a few days at worst. But now he was the CEO of a com­pany where it took weeks to get some­thing onto the home­page, and he was sure that he was the butt of some star­tup’s jokes.

Anyways, all of this clearly both­ered Larry Page. He wanted to fix it. One of his first ac­tions was to shut­ter tons of pro­jects that did­n’t make tac­ti­cal or strate­gic sense, and fo­cus on fewer ef­forts. This came with the catch phrase “more wood be­hind fewer ar­rows.” For ex­am­ple, they shut­tered Google Buzz so that it would­n’t dis­tract from Google+.

And sec­ond, Larry Page emailed the whole com­pany a ham-fisted at­tempt to re­vamp how meet­ings were done.

* Meetings should be capped at 10 peo­ple.

* Everybody in a meet­ing should give in­put or they should­n’t be in the meet­ing.

* Hour-long meet­ings should be only 50 min­utes to give the par­tic­i­pants an op­por­tu­nity to use the re­stroom be­tween meet­ings.

They later soft­ened some of the lan­guage by say­ing that these were prop­er­ties of “decision-oriented meet­ings,” im­ply­ing there were other types of meet­ings that some­one might need to at­tend. But you could never shake the feel­ing that Larry Page had to make de­ci­sions all day long and for­got that some­times peo­ple meet for other rea­sons.

Anyways, let’s fo­cus on the fact that Larry Page wanted hour-long meet­ings to only be 50 min­utes. This is a good thing! It gives peo­ple a chance to stretch, go to the bath­room, grab a snack, etc. During a Q/A on the changes, some­one asked him whether Google Calendar should de­fault to 25 and 50 min­utes for meet­ing lengths in­stead of 30 and 60 min­utes. Larry Page said “yes.” And then some­one on the Google Calendar team im­ple­mented this.

And then noth­ing changed. When 2:50 rolled around and your meet­ing was sup­posed to end, do you think peo­ple ac­tu­ally ended the meet­ing? Noooooo. Absolutely not! Meetings con­tinue un­til the par­tic­i­pants of the next meet­ing are claw­ing on your door like a pack of zom­bies.

At one point, one team in the NYC of­fice no­ticed that their standups were about 10 min­utes long. They did­n’t want to com­pete with meet­ings that re­spected the half-hour bound­aries. And why would they need to? Every meet­ing room had free slots at the last 10 min­utes of every hour be­cause peo­ple were now book­ing 50-minute meet­ings. So they did what any ra­tio­nal en­gi­neer­ing team would do: they started book­ing their standup in the tiny 10-minute time slices that were free on the cal­en­dar of every meet­ing room.

I found this out when I saw them knock on the door to a meet­ing room by my desk. 2:50 rolls around and some­one knocks on the door and says “I have the meet­ing room.”

The per­son in the room re­sponds, “No you don’t, it’s 2:50.”

“Look again at the room’s cal­en­dar. You booked a 50-minute meet­ing, we have the room for the last 10 min­utes of the hour for our standup.”

I could hear the muf­fled ex­as­per­a­tion. “You’ve got to be jok­ing me.”

Then every­one shuf­fled out of the room, look­ing vaguely pissed off. And who could blame them! Can you imag­ine if some­one ac­tu­ally held you to this pol­icy? You’re there stam­mer­ing “it’s the de­fault, I meant for the room to be ours for an hour” and they counter with the fact that their names are listed as the ac­tive par­tic­i­pant? I mean, I’d per­son­ally tell them that I was­n’t go­ing to leave the room, but surely it worked a lot?

I wish I knew the iden­ti­ties of these brave meet­ing crash­ers. I saw them pull this stunt twice and then ride off into the sun­set, and I never got to learn what team they were on. I wish I knew their mo­ti­va­tions. Were they true be­liev­ers in the 50-minute pol­icy? Were they bored pedants? Were they wraiths, cursed to hunt the of­fice for avail­able meet­ing rooms? I’ll never know for sure.

Coinbase on Thursday re­ported that cy­ber­crim­i­nals bribed over­seas sup­port agents to steal cus­tomer data to use in so­cial en­gi­neer­ing at­tacks. The in­ci­dent may cost Coinbase up to $400 mil­lion to fix, the com­pany es­ti­mated.

The crypto ex­change op­er­a­tor re­ceived an email on May 11 from some­one claim­ing they ob­tained in­for­ma­tion about cer­tain Coinbase cus­tomer ac­counts as well as other in­ter­nal Coinbase doc­u­men­ta­tion, in­clud­ing ma­te­ri­als re­lat­ing to cus­tomer ser­vice and ac­count man­age­ment sys­tems, Coinbase re­ported in a Securities and Exchange Commission fil­ing.

The com­pa­ny’s shares were down more than 6% in morn­ing trad­ing.

The email de­manded money in ex­change for not pub­licly dis­clos­ing the in­for­ma­tion, but Coinbase says it has not paid the de­mand and is co­op­er­at­ing with law en­force­ment on the in­ves­ti­ga­tion of the in­ci­dent.

Although pass­words and pri­vate keys were not com­pro­mised, af­fected data in­cluded sen­si­tive data such as names, ad­dresses, phone num­bers and emails; masked bank ac­count num­bers and iden­ti­fiers as well as the last four dig­its of Social Security num­bers; gov­ern­ment ID im­ages and ac­count bal­ances, the com­pany said.

“Cyber crim­i­nals bribed and re­cruited a group of rogue over­seas sup­port agents to steal Coinbase cus­tomer data to fa­cil­i­tate so­cial en­gi­neer­ing at­tacks,” the com­pany said in a blog post. “These in­sid­ers abused their ac­cess to cus­tomer sup­port sys­tems to steal the ac­count data for a small sub­set of cus­tomers. No pass­words, pri­vate keys, or funds were ex­posed and Coinbase Prime ac­counts are un­touched. We will re­im­burse cus­tomers who were tricked into send­ing funds to the at­tacker.”

Coinbase had de­tected the breach in­de­pen­dently in pre­vi­ous months, per the fil­ing. It im­me­di­ately ter­mi­nated the em­ploy­ees in­volved, warned cus­tomers whose in­for­ma­tion may have been ac­cessed and en­hanced its fraud mon­i­tor­ing pro­tec­tions.

The threat ac­tor paid over­seas con­trac­tors and em­ploy­ees in sup­port rolls to ob­tain the in­for­ma­tion, it said.

“We’re co­op­er­at­ing closely with law en­force­ment to pur­sue the harsh­est penal­ties pos­si­ble and will not pay the $20 mil­lion ran­som de­mand we re­ceived,” the com­pany said in the blog. “Instead we are es­tab­lish­ing a $20 mil­lion re­ward fund for in­for­ma­tion lead­ing to the ar­rest and con­vic­tion of the crim­i­nals re­spon­si­ble for this at­tack.”

Coinbase op­er­ates the largest crypto ex­change in the U. S. In the past week it an­nounced an ac­qui­si­tion that is ex­pected to help it ex­pand its global reach and gained en­try to the bench­mark S&P 500 stock in­dex, which will take ef­fect next week. On the earn­ings call last week, CEO Brian Armstrong dis­cussed his am­bi­tion to make Coinbase “the No. 1 fi­nan­cial ser­vices app in the world” in the next five to 10 years.

AutoGenLib is a Python li­brary that au­to­mat­i­cally gen­er­ates code on-the-fly us­ing OpenAI’s API. When you try to im­port a mod­ule or func­tion that does­n’t ex­ist, AutoGenLib cre­ates it for you based on a high-level de­scrip­tion of what you need.

* Dynamic Code Generation: Import mod­ules and func­tions that don’t ex­ist yet

* Context-Aware: New func­tions are gen­er­ated with knowl­edge of ex­ist­ing code

* No Default Caching: Each im­port gen­er­ates fresh code for more var­ied and cre­ative re­sults

* Full Codebase Context: LLM can see all pre­vi­ously gen­er­ated mod­ules for bet­ter con­sis­tency

* Caller Code Analysis: The LLM an­a­lyzes the ac­tual code that’s im­port­ing the mod­ule to bet­ter un­der­stand the con­text and re­quire­ments

* Automatic Exception Handling: All ex­cep­tions are sent to LLM to pro­vide clear ex­pla­na­tion and fixes for er­rors.

pip in­stall au­to­gen­lib

git clone https://​github.com/​co­fob/​au­to­gen­lib.git

cd au­to­gen­lib

pip in­stall -e .

# Import a func­tion that does­n’t ex­ist yet - it will be au­to­mat­i­cally gen­er­ated

from au­to­gen­lib.to­kens im­port gen­er­ate_­to­ken

# Use the gen­er­ated func­tion

to­ken = gen­er­ate_­to­ken(length=32)

print(to­ken)

You ini­tial­ize AutoGenLib with a de­scrip­tion of what you need

When you im­port a mod­ule or func­tion un­der the au­to­gen­lib name­space, the li­brary:

Checks if the mod­ule/​func­tion al­ready ex­ists

If not, it an­a­lyzes the code that’s per­form­ing the im­port to un­der­stand the con­text

It sends a re­quest to OpenAI’s API with your de­scrip­tion and the con­text

The API gen­er­ates the ap­pro­pri­ate code

The code is val­i­dated and ex­e­cuted

The re­quested mod­ule/​func­tion be­comes avail­able for use

from au­to­gen­lib.totp im­port tot­p_­gen­er­a­tor

print(tot­p_­gen­er­a­tor(“SE­CRETKEY123”))

# Later in your ap­pli­ca­tion, you need ver­i­fi­ca­tion:

from au­to­gen­lib.totp im­port ver­i­fy_­totp

re­sult = ver­i­fy_­totp(“SE­CRETKEY123”, “123456″)

print(f”Ver­i­fi­ca­tion re­sult: {result}“)

# Import a func­tion - AutoGenLib will see how your data is struc­tured

from au­to­gen­lib.proces­sor im­port get_high­est_s­core

# Define your data struc­ture

data = [{“user”: “Alice”, “score”: 95}, {“user”: “Bob”, “score”: 82}]

# The func­tion will work with your data struc­ture with­out you hav­ing to spec­ify de­tails

print(get_high­est_s­core(data)) # Will cor­rectly ex­tract the high­est score

# You can use init func­tion to ad­di­tion­ally hint the pur­pose of your li­brary

from au­to­gen­lib im­port init

init(“Cryp­to­graphic util­ity li­brary”)

# Generate en­cryp­tion mod­ule

from au­to­gen­lib.en­cryp­tion im­port en­cryp­t_­text, de­cryp­t_­text

en­crypted = en­cryp­t_­text(“Se­cret mes­sage”, “password123″)

de­crypted = de­cryp­t_­text(en­crypted, “password123”)

print(de­crypted)

# Generate hash­ing mod­ule

from au­to­gen­lib.hash­ing im­port hash_­pass­word, ver­i­fy_­pass­word

hashed = hash_­pass­word(“my_se­cure_­pass­word”)

is_­valid = ver­i­fy_­pass­word(“my_se­cure_­pass­word”, hashed)

print(f”Pass­word valid: {is_valid}“)

Set your OpenAI API key as an en­vi­ron­ment vari­able:

ex­port OPENAI_API_KEY=“your-api-key-here”

# Optional

ex­port OPENAI_API_BASE_URL=“https://​open­router.ai/​api/​v1” # Use OpenRouter API

ex­port OPENAI_MODEL=“openai/gpt-4.1”

Or in your Python code (not rec­om­mended for pro­duc­tion):

im­port os

os.en­v­i­ron[“OPE­NAI_API_KEY”] = “your-api-key-here”

By de­fault, AutoGenLib does not cache gen­er­ated code. This means:

* Each time you im­port a mod­ule, the LLM gen­er­ates fresh code

* You get more var­ied and of­ten fun­nier re­sults due to LLM hal­lu­ci­na­tions

* The same im­port might pro­duce dif­fer­ent im­ple­men­ta­tions across runs

If you want to en­able caching (for con­sis­tency or to re­duce API calls):

from au­to­gen­lib im­port init

init(“Li­brary for data pro­cess­ing”, en­able_­caching=True)

from au­to­gen­lib im­port init, set_­caching

init(“Li­brary for data pro­cess­ing”)

# Later in your code:

set_­caching(True) # Enable caching

set_­caching(False) # Disable caching

When caching is en­abled, gen­er­ated code is stored in ~/.autogenlib_cache.

* Generated code qual­ity de­pends on the clar­ity of your de­scrip­tion

* Not suit­able for pro­duc­tion-crit­i­cal code with­out re­view

You can in­spect the code that was gen­er­ated for a mod­ule:

from au­to­gen­lib.totp im­port tot­p_­gen­er­a­tor

im­port in­spect

print(in­spect.get­source(tot­p_­gen­er­a­tor))

AutoGenLib cre­ates prompts for the OpenAI API that in­clude:

Any ex­ist­ing code in the mod­ule be­ing en­hanced

The full con­text of all pre­vi­ously gen­er­ated mod­ules

The code that’s im­port­ing the mod­ule/​func­tion (new fea­ture!)

This com­pre­hen­sive con­text helps the LLM gen­er­ate code that’s con­sis­tent with your ex­ist­ing code­base and fits per­fectly with how you in­tend to use it.

Contributions are not wel­come! This is just a fun PoC pro­ject.

Note: This li­brary is meant for pro­to­typ­ing and ex­per­i­men­ta­tion. Always re­view au­to­mat­i­cally gen­er­ated code be­fore us­ing it in pro­duc­tion en­vi­ron­ments.

Note: Of course 100% of the code of this li­brary was gen­er­ated via LLM

Throughout my ca­reer, I’ve worked in many com­pli­cated en­vi­ron­ments. For in­stance, I worked on op­ti­miz­ing dri­ver-pas­sen­ger match­ing in ride-hail­ing at a Uber’s com­peti­tor. This con­text, like oth­ers, was tech­ni­cally chal­leng­ing. Yet, noth­ing comes close in terms of com­plex­ity with my cur­rent ex­pe­ri­ence at Google and two years there have re­fined my per­cep­tion of com­plex­ity.

In this post, we will break down the very con­cept of com­plex­ity. Next, we will take a step back to un­der­stand what makes cer­tain en­vi­ron­ments rather com­plex than com­pli­cated and then ex­plore pat­terns for nav­i­gat­ing com­plex sys­tems ef­fec­tively.

Understanding the dis­tinc­tion be­tween com­pli­cated and com­plex prob­lems is cru­cial be­cause each re­quires a fun­da­men­tally dif­fer­ent ap­proach:

* Complicated prob­lems are in­tri­cate but pre­dictable. They fol­lowed struc­tured, re­peat­able so­lu­tions. For ex­am­ple, fil­ing taxes is com­pli­cated, but it’s a struc­tured and con­ven­tional prob­lem since the process re­mains mostly the same year af­ter year.

* Complex prob­lems are in­tri­cate but unique. They re­quire adap­tive and of­ten novel so­lu­tions. For ex­am­ple, cli­mate change mit­i­ga­tion is a com­plex prob­lem be­cause it re­quires new and adap­tive so­lu­tions and ex­ist­ing meth­ods alone can’t re­ally ad­dress its evolv­ing chal­lenges.

Back to soft­ware en­gi­neer­ing, at the Uber com­peti­tor, one of the chal­lenges was to ef­fi­ciently find the near­est dri­ver for a pas­sen­ger. This was far from triv­ial, but it was­n’t com­plex per se. Indeed, many so­lu­tions ex­ist, such as ap­ply­ing geo-hash­ing (example), and im­ple­ment­ing one of these so­lu­tions was the right ap­proach.

At Google, I work as a Site Reliability Engineer (SRE), fo­cus­ing on the sys­tems pow­er­ing Google’s ML in­fra­struc­ture. Here, I con­sider the chal­lenges gen­uinely com­plex, as new par­a­digms and sched­ul­ing ap­proaches are re­quired, es­pe­cially at Google’s scale.

Recognizing whether a sys­tem is com­pli­cated or com­plex is re­ally im­por­tant. Indeed, we men­tioned that com­pli­cated sys­tems are by de­f­i­n­i­tion re­peat­able, while com­plex sys­tems re­quire unique and cus­tomized ap­proaches. Therefore, if we try to ap­ply a com­mon so­lu­tion to a com­plex prob­lem, it may not lead to ef­fec­tive re­sults.

In this sec­tion, we will dis­cuss five com­mon char­ac­ter­is­tics that help iden­tify com­plex sys­tems. Not all com­plex sys­tems share every char­ac­ter­is­tic, but they tend to ex­hibit at least some of the fol­low­ing.

Emergent be­hav­ior arises when a sys­tem’s over­all be­hav­ior can­not be pre­dicted solely by an­a­lyz­ing its in­di­vid­ual com­po­nents in iso­la­tion.

For ex­am­ple, Gemini pro­duc­ing un­ex­pected re­sults was an emer­gent be­hav­ior. While I can’t dis­close the root cause, this be­hav­ior was nearly im­pos­si­ble to fore­see by an­a­lyz­ing all the dif­fer­ent com­po­nents sep­a­rately.

This is one pos­si­ble char­ac­ter­is­tic of com­plex sys­tems: they be­have in ways that can hardly be pre­dicted just by look­ing at their parts, mak­ing them harder to de­bug and man­age.

Another pos­si­ble char­ac­ter­is­tic of com­plex sys­tems is de­layed con­se­quences, where ac­tions don’t al­ways have im­me­di­ate ef­fects, and in­stead, con­se­quences may only be­come ap­par­ent much later.

For ex­am­ple, de­ploy­ing a new ver­sion of a sys­tem might in­tro­duce a sub­tle is­sue that only ap­pears days or even weeks later. This de­lay com­pli­cates de­bug­ging since iden­ti­fy­ing the root cause be­comes much harder com­pared to im­me­di­ate im­pacts.

In com­plex sys­tems, re­ly­ing solely on im­me­di­ate feed­back can cre­ate a false sense of sta­bil­ity, lead­ing to ma­jor sur­prises when an is­sue fi­nally emerges. Keeping in mind that con­se­quences may take time to sur­face is es­sen­tial when work­ing in such en­vi­ron­ments.

In com­plex sys­tems, op­ti­miz­ing one part does­n’t nec­es­sar­ily im­prove the whole sys­tem, and in some cases, it can even make things worse.

Unlike in non-com­plex sys­tems, where im­prov­ing one part gen­er­ally leads to pos­i­tive gains, com­plex sys­tems are much more dif­fi­cult to rea­son about. The com­po­nents in­ter­act in non-ob­vi­ous ways, and lo­cal op­ti­miza­tions can cre­ate rip­ple ef­fects that are dif­fi­cult to pre­dict, some­times lead­ing to neg­a­tive out­comes at the sys­tem level.

This high­lights a key trait of com­plex sys­tems: the whole is more than the sum of its parts. As a re­sult, lo­cal gains don’t al­ways trans­late into global im­prove­ments and in some cases, they can even de­grade the over­all sys­tem.

Hysteresis de­scribes how a sys­tem’s past state con­tin­ues to in­flu­ence its be­hav­ior, even af­ter the orig­i­nal cause is re­moved.

A real-world ex­am­ple to il­lus­trate hys­tere­sis is traf­fic con­ges­tion: even af­ter a road ac­ci­dent is cleared, de­lays per­sist be­cause ve­hi­cles re­main clus­tered. Similarly, in dis­trib­uted sys­tems, fail­ures can cause cas­cad­ing slow­downs, even af­ter the root is­sue is fixed. Indeed, de­pen­dent sys­tems may take time to re­cover for var­i­ous rea­sons, such as caches, re­tries, or queued re­quests.

In com­plex sys­tems, sim­ply fix­ing the root cause is not al­ways enough. Therefore, it’s cru­cial to as­sess whether a sys­tem is prone to hys­tere­sis and, if so, an­tic­i­pate its ef­fects.

In com­plex sys­tems, small changes can pro­duce dis­pro­por­tion­ately large or un­pre­dictable ef­fects.

For ex­am­ple, in queue­ing the­ory, sys­tem load in­creases la­tency pre­dictably. However, as a queue ap­proaches sat­u­ra­tion, even a small in­crease in re­quests can cause re­sponse times to spike ex­po­nen­tially.

Complex sys­tems of­ten reach tip­ping points, where be­hav­iors shift sud­denly, mak­ing past trends un­re­li­able for pre­dic­tion. This non­lin­ear­ity means that tra­di­tional lin­ear as­sump­tions where in­puts map pre­dictably to out­puts is­n’t al­ways ef­fec­tive for de­sign­ing, test­ing, and rea­son­ing about com­plex sys­tems.

* Are dif­fi­cult to un­der­stand just by look­ing at its parts sep­a­rately.

* Don’t al­ways show their ef­fects right away, con­se­quences can be de­layed.

* Don’t al­ways im­prove as a whole when one part is op­ti­mized and changes can some­times make things worse.

* Can keep be­ing in­flu­enced by past states, even af­ter the orig­i­nal cause is gone.

* Can re­act to small changes with big or un­ex­pected ef­fects.

Note that scale alone does­n’t make a sys­tem com­plex: even small sys­tems can ex­hibit com­plex be­hav­iors like emer­gence or non­lin­ear­ity.

Given these char­ac­ter­is­tics, how can we op­er­ate ef­fec­tively in com­plex en­vi­ron­ments? Below are some strate­gies that I per­son­ally found ef­fec­tive.

When deal­ing with com­plex sys­tems, we should fa­vor re­versible de­ci­sions when­ever pos­si­ble, mean­ing changes that can be un­done if they don’t work out.

Amazon’s one-way vs. two-way doors frame­work cap­tures this idea quite well:

* Two-way doors rep­re­sent re­versible de­ci­sions, al­low­ing us to move fast and it­er­ate with lower risk.

In many con­texts, es­pe­cially in com­plex sys­tems, fa­vor­ing two-way doors leads to bet­ter out­comes be­cause we can ex­per­i­ment, learn, and re­fine rather than ov­erengi­neer­ing up­front.

That be­ing said, not all de­ci­sions should be re­versible. For ex­am­ple, some choices like se­cu­rity poli­cies or com­pli­ance-re­lated changes re­quire up­front com­mit­ment. The key is know­ing when to op­ti­mize for speed and it­er­a­tion ver­sus when to be de­lib­er­ate and care­ful.

Because com­plex sys­tems don’t al­ways re­spond pre­dictably to lo­cal op­ti­miza­tions, defin­ing the right met­rics for suc­cess is prob­a­bly just as im­por­tant as the changes we make. Indeed, fo­cus­ing too much on iso­lated, lo­cal met­rics can cre­ate a false sense of suc­cess while mask­ing un­in­tended neg­a­tive con­se­quences else­where in the sys­tem.

To avoid this, be­fore mak­ing a change, we should de­fine both lo­cal and global met­rics to get a holis­tic view of sys­tem health. This en­sures that we mea­sure im­pact be­yond the im­me­di­ate area of fo­cus and con­sider the sys­tem as a whole.

Well-chosen met­rics should­n’t just con­firm the suc­cess of a lo­cal change; in­stead, they should help us make bet­ter de­ci­sions and en­sure mean­ing­ful im­prove­ments at the sys­tem level, not just iso­lated ar­eas.

As dis­cussed, com­plex sys­tems of­ten de­mand unique so­lu­tions. Since con­ven­tional strate­gies may not al­ways ap­ply, we must be will­ing to think out of the box and em­brace in­no­va­tion.

I re­call one of my first meet­ings at Google. Someone pre­sented a prob­lem that seemed ab­surd in terms of com­plex­ity, es­pe­cially given the scale. My im­me­di­ate re­ac­tion in my head was: “This is im­pos­si­ble”. But then, a team­mate said: “But we’re Google, we should be able to man­age it!”.

That re­mark stuck with me. While not every com­pany ob­vi­ously has Google’s re­sources, I think the mind­set is what truly mat­ters. When fac­ing a com­plex prob­lem, we should as­sume it’s solv­able, then break it down, ex­per­i­ment, and it­er­ate un­til we find a path for­ward.

One may find this sec­tion cliché, but again, com­plex prob­lems de­mand un­con­ven­tional think­ing. In many cases, be­ing open to in­no­v­a­tive so­lu­tions when fac­ing a com­plex prob­lem is­n’t just help­ful, it’s nec­es­sary.

When de­ploy­ing changes in com­plex sys­tems, we should rely on proven best prac­tices to min­i­mize risk. These in­clude:

* Feature flags: Enable or dis­able func­tion­al­ity dy­nam­i­cally with­out de­ploy­ing new code, al­low­ing for safe ex­per­i­men­ta­tion and quicker roll­backs.

* Canary re­lease: A lim­ited roll­out to a small, con­trolled sub­set of pro­duc­tion, well suited for en­vi­ron­ments with only a few pro­duc­tion in­stances.

* Progressive roll­outs: Gradually in­creas­ing the scope of a roll­out, best suited for large-scale pro­duc­tion se­tups with mul­ti­ple clus­ters or re­gions.

* Shadow test­ing: Running a change in par­al­lel with pro­duc­tion traf­fic with­out im­pact­ing real users. This helps val­i­date the cor­rect­ness of a change be­fore en­abling it.

By lever­ag­ing these tech­niques, we re­duce the blast ra­dius of fail­ures, im­prov­ing the con­fi­dence in our changes and en­abling faster it­er­a­tions.

Observability is one of the main pil­lars of com­plex sys­tems. My work­ing de­f­i­n­i­tion of ob­serv­abil­ity (mainly in­spired by Observability Engineering) is the fol­low­ing:

You can un­der­stand any state of your sys­tem (no mat­ter how novel or bizarre) by slic­ing and dic­ing high-car­di­nal­ity and high-di­men­sion­al­ity teleme­try data with­out need­ing to ship new code.

* Systems be­come more frag­ile as un­known is­sues re­main hid­den un­til they cause real im­pacts.

* Innovation is slowed down due to a lack of ef­fi­cient feed­back loops.

In com­plex en­vi­ron­ments, where un­knowns are in­evitable, ob­serv­abil­ity is es­sen­tial. It en­able teams to nav­i­gate un­cer­tainty, ex­per­i­ment more safely and get­ting short feed­back loops to con­tin­u­ously im­prove sys­tems.

Without proper ob­serv­abil­ity, changes re­main opin­ions rather than in­formed de­ci­sions.

Predicting the be­hav­ior of com­plex sys­tems is rarely sim­ple, and, some­times, nearly im­pos­si­ble.

I re­call a case where we spent con­sid­er­able time de­sign­ing a change, care­fully back­ing every as­sump­tion with data. Yet, due to un­ac­counted fac­tors such as lurk­ing vari­ables, the change was ul­ti­mately in­ef­fec­tive.

Sometimes, in­stead of re­ly­ing solely on pre­dic­tions, a more ef­fec­tive ap­proach can be to sim­u­late a change be­fore rolling it out. There are mul­ti­ple ways to lever­age sim­u­la­tion test­ing, in­clud­ing:

* Replaying past events: If we de­sign a sys­tem to record all its in­put, we can re­play past events against our new ver­sion and an­a­lyze its im­pact. This al­lows us to val­i­date changes in a con­trolled man­ner, re­duc­ing un­cer­tainty and im­prov­ing de­ci­sion-mak­ing in com­plex sys­tems.

* Deterministic sim­u­la­tion test­ing: Instead of re­ly­ing on real-world data, we can cre­ate con­trolled, re­peat­able sim­u­la­tions that model sys­tem be­hav­ior un­der spe­cific con­di­tions. This al­lows us to test how a sys­tem re­acts un­der var­i­ous con­di­tions in a fully de­ter­min­is­tic way.

Note that the ideas pre­sented in this sec­tion also rely heav­ily on ob­serv­abil­ity.

In com­plex en­vi­ron­ments, rules-based ap­proaches of­ten reach their limit be­cause of the com­plex­ity of an­tic­i­pat­ing all sce­nar­ios. In these con­texts, ML can be­come par­tic­u­larly ef­fec­tive.

Indeed, un­like sta­tic heuris­tics, ML mod­els can con­tin­u­ously adapt based on feed­back loops and learn from real-world data rather than re­ly­ing on rigid, pre­de­fined logic.

This al­lows sys­tems to:

* Adapt dy­nam­i­cally to changes with­out re­quir­ing con­stant hu­man in­ter­ven­tion.

* Make prob­a­bilis­tic de­ci­sions rather than re­ly­ing on strict if-else con­di­tions.

Last but not least, I be­lieve that in com­plex en­vi­ron­ments, more than any­where else, strong team col­lab­o­ra­tion is an ab­solute ne­ces­sity. For in­stance, clearly con­vey­ing why a change is com­plex, dis­cussing avail­able op­tions, and de­bat­ing trade-offs with team­mates are all crit­i­cal skills.

In com­plex sys­tems, there’s of­ten no sin­gle right an­swer. Therefore, a team that col­lab­o­rates ef­fec­tively and nav­i­gates am­bi­gu­ity to­gether can make a huge dif­fer­ence, ul­ti­mately lead­ing to stronger de­ci­sion-mak­ing.

Again, com­pli­cated prob­lems can be solved with re­peat­able so­lu­tions, whereas com­plex sys­tems re­quire adapt­abil­ity and a dif­fer­ent way of think­ing. This is why rec­og­niz­ing whether a sys­tem is com­pli­cated or com­plex is so im­por­tant: it shapes how we should ap­proach prob­lem-solv­ing.

However, in many en­vi­ron­ments, sys­tems are nei­ther purely com­pli­cated nor purely com­plex. Some parts can fol­low struc­tured, pre­dictable so­lu­tions, while oth­ers re­quire adapt­abil­ity and novel ap­proaches. The key is learn­ing to rec­og­nize when adapt­abil­ity is needed and when struc­tured so­lu­tions are enough.

💬 I hope this post has helped you rec­og­nize the char­ac­ter­is­tics of com­plex en­vi­ron­ments and pro­vided you with prac­ti­cal pat­terns to nav­i­gate them ef­fec­tively. Did any of these pat­terns res­onate with you? What other strate­gies have you used in com­plex en­vi­ron­ments? Let me know in the com­ments.

❤️ If you made it this far and en­joyed the post, please con­sider giv­ing it a like.

📣 This post is part of a se­ries writ­ten in col­lab­o­ra­tion with Antithesis, an au­tonomous test­ing plat­form. They are not spon­sor­ing this post—I reached out to them be­cause I was gen­uinely in­trigued by what they were build­ing and ended up falling in love with their so­lu­tion. We will dive deeper into their so­lu­tions in a fu­ture post. In the mean­time, feel free to check out their web­site or their great blog.

* The Critical Difference Between Complex and Complicated

Here we in­tro­duce in­tro­duc­tion to Boltzmann ma­chines and pre­sent a Tiny Restricted Boltzmann Machine that runs in the browser. Boltzmann Machines are one of the ear­li­est gen­er­a­tive AI mod­els, in­tro­duced in the 1980s.Boltzmann Machines are used for un­su­per­vised learn­ing, which means they can learn from data with­out be­ing told what to look for.The can be used for gen­er­at­ing new data that is sim­i­lar to the data they were trained on, also known as gen­er­a­tive AI.A Boltzmann Machine is a type of neural net­work that tries to learn pat­terns by mim­ic­k­ing how en­ergy works in physics.Each neu­ron can be on or off, the ma­chine is made up of many of these neu­rons con­nect to each other.Some neu­rons are vis­i­ble (we can see them and even set their state), and some are hid­den (we can’t see them).The con­nec­tions be­tween neu­rons are called weights, and they can be pos­i­tive or neg­a­tive.Hover over the neu­rons to high­light their con­nec­tions.Hover over the neu­rons to high­light their con­nec­tions.A General Boltzmann Machine has con­nec­tions be­tween all neu­rons. This makes it pow­er­ful, but its train­ing in­volves cal­cu­lat­ing an term.A Restricted Boltzmann Machine is a spe­cial case where the vis­i­ble and hid­den neu­rons are not con­nected to each other. This makes it faster to train and un­der­stand.The en­ergy of a con­fig­u­ra­tion of the vis­i­ble and hid­den units is de­fined as:where is the vis­i­ble layer, is the hid­den layer, is the weight ma­trix, and and are the bi­ases for the vis­i­ble and hid­den lay­ers, re­spec­tively.The vi­su­al­i­sa­tion on the right ran­domises the weights, bi­ases and ac­ti­va­tion val­ues of a Boltzmann ma­chine and cal­cu­lates its en­ergy.Dur­ing train­ing it is given ex­am­ples (e.g., im­ages, text) and the ma­chine ad­justs its weights to lower the en­ergy of those sam­ples.It ef­fec­tively learns , the prob­a­bil­ity of vis­i­ble units , which is pro­por­tional to .After train­ing, it can sam­ple new data from the learned dis­tri­b­u­tion us­ing Gibbs sam­pling.These sam­ples are new, never-be­fore-seen, but sta­tis­ti­cally sim­i­lar to the train­ing data.Here is our train­ing data.We want the net­work to learn how to make sim­i­lar sam­ples to these.A Restricted Boltzmann Machine (RBM) is trained us­ing a process called Contrastive Divergence. The steps are as fol­lows:A more for­mal de­scrip­tion of the steps above are given in the Appendix.Press the “Run Simulation” but­ton to start train­inng the RBM. If you let the sim­u­la­tion run for a while, you will see the weights of the RBM con­verge to a sta­ble state. The en­ergy loss will also de­crease over time.You can com­pare the in­put and out­put states of the RBM by paus­ing the sim­u­la­tion.In the be­gin­ning, the in­put and out­put states will be dis­sim­i­lar. As the sim­u­la­tion pro­gresses, the in­put and out­put states will be­come more sim­i­lar.Start­ing with a Boltzmann ma­chine as de­fined ear­lier, we want to de­rivce the con­trastive di­ver­gence al­go­rithm for train­ing. The goal is to ad­just the weights of the net­work to min­i­mize the en­ergy of the train­ing data.

A weight ma­trix that con­nects the vis­i­ble and hid­den lay­ers.

A bias vec­tor for the vis­i­ble layer and a bias vec­tor for the hid­den layer.

Where is the par­ti­tion func­tion, which nor­mal­izes the dis­tri­b­u­tion.We train the RBM by max­i­miz­ing the like­li­hood of the train­ing data, i.e. max­i­miz­ing .The mar­ginal like­li­hood of the vis­i­ble layer is given by:Then the log-like­li­hood is:Dif­fer­en­ti­at­ing with re­spect to the weights gives:Sim­i­lar forms ex­ist for the bi­ases and .Since we are per­form­ing gra­di­ent as­cent, there­fore.There­fore we get our weight up­date rule:A sim­i­lar process can be fol­lowed for the bi­ases and .Where is the ex­pec­ta­tion with re­spect to the train­ing data and is the ex­pec­ta­tion with re­spect to the model dis­tri­b­u­tion.The next step is to ap­prox­i­mate the model ex­pec­ta­tion us­ing Gibbs sam­pling.Once those steps are done we up­date the weights and bi­ases ac­cord­ing to:

14 May 2025

(Updated on 15 May to in­clude links to the judge­ment)

Google, Microsoft, Amazon, X, and the en­tire track­ing-based ad­ver­tis­ing in­dus­try rely on the “Transparency & Consent Framework” (TCF) to ob­tain “consent” for data pro­cess­ing. This evening the Belgian Court of Appeal ruled that the TCF is il­le­gal. The TCF is live on 80% of the Internet.[1]

Today’s de­ci­sion arises from en­force­ment by the Belgian Data Protection Authority, prompted by com­plainants co­or­di­nated by Dr Johnny Ryan, Director of Enforce at the Irish Council for Civil Liberties. The group of com­plainants are: Dr Johnny Ryan of Enforce, Katarzyna Szymielewicz of the Panoptykon Foundation, Dr Jef Ausloos, Dr Pierre Dewitte, Stichting Bits of Freedom, and Ligue des Droits Humains.

Dr Johnny Ryan said “Today’s court’s de­ci­sion shows that the con­sent sys­tem used by Google, Amazon, X, Microsoft, de­ceives hun­dreds of mil­lions of Europeans. The tech in­dus­try has sought to hide its vast data breach be­hind sham con­sent pop­ups. Tech com­pa­nies turned the GDPR into a daily nui­sance rather than a shield for peo­ple.”

This Belgian en­force­ment arises from a chain of com­plaints and lit­i­ga­tion across Europe ini­ti­ated by Dr Ryan in 2018 against Real-Time Bidding (RTB).

Today’s de­ci­sion con­firmed the Belgian Data Protection Authority’s 2022 find­ing of mul­ti­ple in­fringe­ments by the TCF, closely echo­ing the com­plainants’ sub­mis­sions.

For seven years, the track­ing in­dus­try has used the TCF as a le­gal cover for Real-Time Bidding (RTB), the vast ad­ver­tis­ing auc­tion sys­tem that op­er­ates be­hind the scenes on web­sites and apps. RTB tracks what Internet users look at and where they go in the real world. It then con­tin­u­ously broad­casts this data to a host of com­pa­nies, en­abling them to keep dossiers on every Internet user.[2] Because there is no se­cu­rity in the RTB sys­tem it is im­pos­si­ble to know what then hap­pens to the data. As a re­sult, it is also im­pos­si­ble to pro­vide the nec­es­sary in­for­ma­tion that must ac­com­pany a con­sent re­quest.[3]

Today’s judge­ment con­firms the Belgian Data Protection Authority’s 2022 de­ci­sion. It ap­plies im­me­di­ately across Europe.

Dr Ryan of Enforce said “This de­ci­sion is mo­men­tous. It cre­ates a clear need for in­dus­try to in­no­vate and move away from the dan­ger­ous, in­ef­fec­tive, and fraud-rid­dled track­ing-based ad­ver­tis­ing. RTB can op­er­ate with­out per­sonal data. This de­ci­sion shows that it must. This good news for every per­son on­line, and for pub­lish­ers, too.”

We thank our lawyers Fred­eric Debusseré and Ruben Roex, of Timelex.

See chronol­ogy, ev­i­dence, and ex­plain­ers about RTB https://​www.iccl.ie/​RTB/

[1] See “IAB & IAB Tech Lab Respond with Support for OpenRTB and IAB Europe’s Transparency & Consent Framework”, October 19 2020 https://​iabtech­lab.com/​iab-and-tech-lab-re­spond-with-sup­port-for-open-rtb-and-iab-eu­ropes-trans­parency-con­sent-frame­work/

[2] For de­tail on the scale of RTB see our re­port “The Biggest Data Breach: ICCL re­port on the scale of Real-Time Bidding data broad­casts in the U. S. and Europe”, ICCL, May 2022 https://​www.iccl.ie/​dig­i­tal-data/​iccl-re­port-on-the-scale-of-real-time-bid­ding-data-broad­casts-in-the-u-s-and-eu­rope/

[3] “As it is tech­ni­cally im­pos­si­ble for the user to have prior in­for­ma­tion about every data con­troller in­volved in a real-time bid­ding (RTB) sce­nario, pro­gram­matic trad­ing, the area of fastest growth in dig­i­tal ad­ver­tis­ing spend, would seem, at least prima fa­cie, to be in­com­pat­i­ble with con­sent un­der GDPR”. See e-mail and page 3 of at­tached lob­by­ing pa­per from IAB Europe CEO Townsend Feehan to European Commission, 26 June 2017, ob­tained by Enforce us­ing Freedom of Information.

My co-work­ers and I have been work­ing on an AI Programming Assistant called

Sketch

for the last few months. The thing I’ve been most sur­prised by is how shock­ingly sim­ple the main loop of us­ing an LLM with tool use is:

def loop(llm):

msg = user_in­put()

while True:

out­put, tool_­calls = llm(msg)

print(“Agent: ”, out­put)

if tool_­calls:

msg = [ han­dle_­tool_­call(tc) for tc in tool_­calls ]

else:

msg = user_in­put()

There’s some pomp and cir­cum­stance to make the above work (here’s the full script)

, but the core idea is the above 9 lines. Here, llm() is a func­tion that sends the sys­tem prompt, the con­ver­sa­tion so far, and the next mes­sage to the LLM API.

Tool use is the fancy term for “the LLM re­turns some out­put that cor­re­sponds to a schema,” and, in the full script, we tell the LLM (in its sys­tem prompt and tool de­scrip­tion prompts) that it has ac­cess to bash.

With just that one very gen­eral pur­pose tool, the cur­rent mod­els (we use Claude 3.7 Sonnet ex­ten­sively) can nail many prob­lems, some of them in “one shot.” Whereas I used to look up an es­o­teric git op­er­a­tion and then cut and paste, now I just ask Sketch to do it. Whereas I used to han­dle git merges man­u­ally, now I let Sketch take a first pass. Whereas I used to change a type and go through the re­sult­ing type checker er­rors one by one (or, let’s be real, with perl -pie ridicu­lous­ness), I give it a shot with Sketch. If ap­pro­pri­ately prompted, the agen­tic loop can be per­sis­tent. If you don’t have some tool in­stalled, it’ll in­stall it. If your `grep` has dif­fer­ent com­mand line op­tions, it adapts. (It can also be in­fu­ri­at­ing! “Oh, this test does­n’t pass… let’s just skip it,” it some­times says, mad­den­ingly.)

For many work­flows, agen­tic tools spe­cial­ize. Sketch’s quiver of tools is not just bash, as we’ve found that a hand­ful of ex­tra tools im­prove the qual­ity, speed up it­er­a­tions, and fa­cil­i­tate bet­ter de­vel­oper work­flows. Tools that let the LLM edit text cor­rectly are sur­pris­ingly tricky. Seeing the LLM strug­gle with sed one-lin­ers re-af­firms that vi­sual (as op­posed to line) ed­i­tors are a mar­vel.

I have no doubt that agent loops will get in­cor­po­rated into more day to day au­toma­tion te­dium that’s his­tor­i­cally been too spe­cific for gen­eral pur­pose tools and too es­o­teric and un­sta­ble to au­to­mate tra­di­tion­ally. I keep think­ing of how much time I’ve spent cor­re­lat­ing stack traces with git com­mits, and how good LLMs are at do­ing a first pass on it. We’ll be see­ing more cus­tom, , throw-away LLM agent loops in our bin/ di­rec­to­ries. Grab your fa­vorite bearer to­ken and give it a shot.