Published: July 1, 2020

I.

The smartest per­son I’ve ever known had a habit that, as a teenager, I found strik­ing. After he’d prove a the­o­rem, or solve a prob­lem, he’d go back and con­tinue think­ing about the prob­lem and try to fig­ure out dif­fer­ent proofs of the same thing. Sometimes he’d spend hours on a prob­lem he’d al­ready solved.

I had the op­po­site ten­dency: as soon as I’d reached the end of the proof, I’d stop since I’d “gotten the an­swer”.

Afterwards, he’d come out with three or four proofs of the same thing, plus some ex­pla­na­tion of why each proof is con­nected some­how. In this way, he got a much deeper un­der­stand­ing of things than I did.

I con­cluded that what we call ‘intelligence’ is as much about virtues such as hon­esty, in­tegrity, and brav­ery, as it is about ‘raw in­tel­lect’.

Intelligent peo­ple sim­ply aren’t will­ing to ac­cept an­swers that they don’t un­der­stand — no mat­ter how many other peo­ple try to con­vince them of it, or how many other peo­ple be­lieve it, if they aren’t able to con­vince them selves of it, they won’t ac­cept it.

Importantly, this is a ‘software’ trait & is in­de­pen­dent of more ‘hardware’ traits such as pro­cess­ing speed, work­ing mem­ory, and other such things.

Moreover, I have no­ticed that these ‘hardware’ traits vary greatly in the smartest peo­ple I know — some are re­mark­ably quick thinkers, cal­cu­la­tors, read­ers, whereas oth­ers are ‘slow’. The soft­ware traits, though, they all have in com­mon — and can, with ef­fort, be learned.

What this means is that you can in­ter­nal­ize good in­tel­lec­tual habits that, in ef­fect, “increase your in­tel­li­gence”. ‘Intelligence’ is not fixed.

II.

This qual­ity of “not stop­ping at an un­sat­is­fac­tory an­swer” de­serves some ex­am­i­na­tion.

One com­po­nent of it is en­ergy: think­ing hard takes ef­fort, and it’s much eas­ier to just stop at an an­swer that seems to make sense, than to pur­sue every­thing that you don’t quite get down an end­less, and rapidly pro­lif­er­at­ing, se­ries of rab­bit holes.

It’s also so easy to think that you un­der­stand some­thing, when you ac­tu­ally don’t. So even fig­ur­ing out whether you un­der­stand some­thing or not re­quires you to at­tack the thing from mul­ti­ple an­gles and test your own un­der­stand­ing.

This re­quires a lot of in­trin­sic mo­ti­va­tion, be­cause it’s so hard; so most peo­ple sim­ply don’t do it.

The Nobel Prize win­ner William Shockley was fond of talk­ing about “the will to think”:

Motivation is at least as im­por­tant as method for the se­ri­ous thinker, Shockley be­lieved…the es­sen­tial el­e­ment for suc­cess­ful work in any field was “the will to think”. This was a phrase he learned from the nu­clear physi­cist Enrico Fermi and never for­got. “In these four words,” Shockley wrote later, “[Fermi] dis­tilled the essence of a very sig­nif­i­cant in­sight: A com­pe­tent thinker will be re­luc­tant to com­mit him­self to the ef­fort that te­dious and pre­cise think­ing de­mands — he will lack ‘the will to think’ — un­less he has the con­vic­tion that some­thing worth­while will be done with the re­sults of his ef­forts.” The dis­ci­pline of com­pe­tent think­ing is im­por­tant through­out life… (source)But it’s not just en­ergy. You have to be able to mo­ti­vate your­self to spend large quan­ti­ties of en­ergy on a prob­lem, which means on some level that not un­der­stand­ing some­thing — or hav­ing a bug in your think­ing — both­ers you a lot. You have the drive, the will to know.

Related to this is hon­esty, or in­tegrity: a sort of com­pul­sive un­will­ing­ness, or in­abil­ity, to lie to your­self. Feynman said that the first rule of sci­ence is that you do not fool your­self, and you are the eas­i­est per­son to fool. It is uniquely easy to lie to your­self be­cause there is no ex­ter­nal force keep­ing you hon­est; only you can run the con­stant loop of ask­ing “do I re­ally un­der­stand this?”.

(This is why writ­ing is im­por­tant. It’s harder to fool your­self that you un­der­stand some­thing when you sit down to write about it and it comes out all dis­jointed and con­fused. Writing forces clar­ity.)

III.

The physi­cist Michael Faraday be­lieved noth­ing with­out be­ing able to ex­per­i­men­tally demon­strate it him­self, no mat­ter how te­dious the demon­sta­tion.

Simply hear­ing or read­ing of such things was never enough for Faraday. When as­sess­ing the work of oth­ers, he al­ways had to re­peat, and per­haps ex­tend, their ex­per­i­ments. It be­came a life­long habit—his way of es­tab­lish­ing own­er­ship over an idea. Just as he did count­less times later in other set­tings, he set out to demon­strate this new phe­nom­e­non to his own sat­is­fac­tion. When he had saved enough money to buy the ma­te­ri­als, he made a bat­tery from seven cop­per half­pen­nies and seven discs cut from a sheet of zinc, in­ter­leaved with pieces of pa­per soaked in salt wa­ter. He fixed a cop­per wire to each end plate, dipped the other ends of the wires in a so­lu­tion of Epsom salts (magnesium sul­fate), and watched. (source)Understanding some­thing re­ally deeply is con­nected to our phys­i­cal in­tu­ition. A sim­ple “words based” un­der­stand­ing can only go so far. Visualizing some­thing, in three di­men­sions, can help you with a con­crete “hook” that your brain can grasp onto and use as a model; un­der­stand­ing then has a phys­i­cal con­text that it can “take place in”.

This is why Jesus speaks in para­bles through­out the New Testament — in ways that stick with you long af­ter you’ve read them — rather than just stat­ing the ab­stract prin­ci­ple. “Are not two spar­rows sold for a cent? And yet not one of them will fall to the ground apart from your Father.” can stick with you for­ever in a way that “God watches over all liv­ing be­ings” will not.

Faraday, again, had this qual­ity in spades — the book makes clear that this is partly be­cause he was bad at math­e­mat­ics and thus un­der­stood every­thing through the medium of ex­per­i­ments, and con­trasts this with the French sci­en­tists (such as Ampere) who un­der­stood every­thing in a highly ab­stract way.

But Faraday’s phys­i­cal in­tu­ition led him to some of the most cru­cial dis­cov­er­ies in all of sci­ence:

Much as he ad­mired Ampère’s work, Faraday be­gan to de­velop his own views on the na­ture of the force be­tween a cur­rent-car­ry­ing wire and the mag­netic nee­dle it de­flected. Ampère’s math­e­mat­ics (which he had no rea­son to doubt) showed that the mo­tion of the mag­netic nee­dle was the re­sult of re­pul­sions and at­trac­tions be­tween it and the wire. But, to Faraday, this seemed wrong, or, at least, the wrong way around. What hap­pened, he felt, was that the wire in­duced a cir­cu­lar force in the space around it­self, and that every­thing else fol­lowed from this. The next step beau­ti­fully il­lus­trates Faraday’s ge­nius. Taking Sarah’s four­teen-year-old brother George with him down to the lab­o­ra­tory, he stuck an iron bar mag­net into hot wax in the bot­tom of a basin and, when the wax had hard­ened, filled the basin with mer­cury un­til only the top of the mag­net was ex­posed. He dan­gled a short length of wire from an in­su­lated stand so that its bot­tom end dipped in the mer­cury, and then he con­nected one ter­mi­nal of a bat­tery to the top end of the wire and the other to the mer­cury. The wire and the mer­cury now formed part of a cir­cuit that would re­main un­bro­ken even if the bot­tom end of the wire moved. And move it did—in rapid cir­cles around the mag­net! (source)

Being able to gen­er­ate these con­crete ex­am­ples, even when you’re not phys­i­cally do­ing ex­per­i­ments, is im­po­ratant.

I re­cently saw this strik­ing rep­re­sen­ta­tion of the “bag of words” model in NLP. If you were read­ing this in the usual dry math­e­mat­i­cal way these things are rep­re­sented, and then forced your­self to come up with a vi­su­al­iza­tion like this, then you’d be much fur­ther on your way to re­ally grasp­ing the thing.

Conversely, if you’re not com­ing up with vi­su­als like this, and your un­der­stand­ing of the thing re­mains on the level of equa­tions or ab­stract con­cepts, you prob­a­bly do not un­der­stand the con­cept deeply and should dig fur­ther.

Another qual­ity I have no­ticed in very in­tel­li­gent peo­ple is be­ing un­afraid to look stu­pid.

Malcolm Gladwell on his fa­ther:

My fa­ther has zero in­tel­lec­tual in­se­cu­ri­ties… It has never crossed his mind to be con­cerned that the world thinks he’s an id­iot. He’s not in that game. So if he does­n’t un­der­stand some­thing, he just asks you. He does­n’t care if he sounds fool­ish. He will ask the most ob­vi­ous ques­tion with­out any sort of con­cern about it… So he asks lots and lots of dumb, in the best sense of that word, ques­tions. He’ll say to some­one, ‘I don’t un­der­stand. Explain that to me.’ He’ll just keep ask­ing ques­tions un­til he gets it right, and I grew up lis­ten­ing to him do this in every con­ceiv­able set­ting. If my fa­ther had met Bernie Madoff, he would never have in­vested money with him be­cause he would have said, ‘I don’t un­der­stand’ a hun­dred times. ‘I don’t un­der­stand how that works’, in this kind of dumb, slow voice. ’I don’t un­der­stand, sir. What is go­ing on?’Most peo­ple are not will­ing to do this — look­ing stu­pid takes courage, and some­times it’s eas­ier to just let things slide. It is strik­ing how many sit­u­a­tions I am in where I start ask­ing ba­sic ques­tions, feel guilty for slow­ing the group down, and it turns out that no­body un­der­stood what was go­ing on to be­gin with (often peo­ple mes­sage me pri­vately say­ing that they’re re­lieved I asked), but I was the only one who ac­tu­ally spoke up and asked about it.

This is a habit. It’s easy to pick up. And it makes you smarter.

IV.

I re­mem­ber be­ing taught cal­cu­lus at school and get­ting stuck on the “dy/dx” no­ta­tion (aka Leibniz no­ta­tion) for cal­cu­lus.

The “dy/dx” just looked like a frac­tion, it looked like we were do­ing di­vi­sion, but we weren’t ac­tu­ally do­ing di­vi­sion. “dy/dx” does­n’t mean “dy” di­vided by “dx”, it means “the value of an in­fin­i­tes­i­mal change in y with re­spect to an in­fin­i­tes­i­mal change in x”, and I did­n’t see how you could break this thing apart as though it was sim­ple di­vi­sion.

At one point the proof of the fun­da­men­tal the­o­rem of cal­cu­lus in­volved mul­ti­ply­ing out a poly­no­mial, and along the way you could can­cel out “dy*dx” be­cause “both of these quan­ti­ties are in­fin­i­tes­i­mal, so in ef­fect this can be can­celled out”. This rea­son­ing did not make sense.

The “proof” of the chain rule we were given looked like this.

(Amusingly, you can even get cor­rect re­sults us­ing in­valid math­e­mat­ics, like this. Even though this is clearly in­valid, it does­n’t feel far off the “valid” proof of the chain rule I was taught.)

It turns out that my mis­giv­ings were right, and that the Leibniz no­ta­tion is ba­si­cally just a con­ve­nient short­hand and that you more or less can treat those things “as if” they are frac­tions, but the proof is su­per com­pli­cated etc. Moreover, the Leibniz short­hand is ac­tu­ally far more pow­er­ful and eas­ier to work with than Newton’s func­tions-based short­hand, which is why main­land Europe got way ahead of England (which stuck with Newton’s no­ta­tion) in cal­cu­lus. And then all of the log­i­cal prob­lems did­n’t re­ally get sorted out un­til Riemann came along 200 years later and for­mu­lated cal­cu­lus in terms of lim­its. But all of that went over my head in high school.

At the time, I was in­fu­ri­ated by these in­ad­e­quate proofs, but I was un­der time pres­sure to just learn the op­er­a­tions so that I could an­swer exam ques­tions be­cause the class needed to move onto the next thing.

And since you ac­tu­ally can an­swer the exam ques­tions and me­chan­i­cally per­form cal­cu­lus op­er­a­tions with­out ever deeply un­der­stand­ing cal­cu­lus, it’s much eas­ier to just get by and do the exam with­out re­ally ques­tion­ing the con­cepts deeply — which is in fact what hap­pens for most peo­ple. (See my es­say on ed­u­ca­tion.)

How many peo­ple ac­tu­ally go back and try and un­der­stand this, or other such top­ics, in a deeper way? Very few. Moreover, the ‘meta’ les­son is: don’t ques­tion it too deeply, you’ll fall be­hind. Just learn the al­go­rithm, plug in the num­bers, and pass your ex­ams. Speed is of the essence. In this way, school kills the “will to un­der­stand­ing” in peo­ple.

My coun­ter­vail­ing ad­vice to peo­ple try­ing to un­der­stand some­thing is: go slow. Read slowly, think slowly, re­ally spend time pon­der­ing the thing. Start by think­ing about the ques­tion your­self be­fore read­ing a bunch of stuff about it. A week or a month of con­tin­u­ous pon­der­ing about a ques­tion will get you sur­pris­ingly far.

And you’ll have a se­man­tic men­tal ‘framework’ in your brain on which to then hang all the great things you learn from your read­ing, which makes it more likely that you’ll re­tain that stuff as well. I read some­where that Bill Gates struc­tures his fa­mous “reading weeks” around an out­line of im­por­tant ques­tions he’s thought about and bro­ken down into pieces. e.g. he’ll think about “water scarcity” and then break it down into ques­tions like “how much wa­ter is there in the world?”, “where does ex­ist­ing drink­ing wa­ter come from?”, “how do you turn ocean wa­ter into drink­ing wa­ter”, etc., and only then will he pick read­ing to ad­dress those ques­tions.

This method is far more ef­fec­tive than just read­ing ran­dom things and let­ting them pass through you.

V.

The best thing I have read on re­ally un­der­stand­ing things is the Sequences, es­pe­cially the sec­tion on Noticing Confusion.

There are some mantra-like ques­tions it can be help­ful to ask as you’re think­ing through things. Some ex­am­ples:

But what ex­actly is X? What is it? (h/t Laura Deming’s post)Why must X be true? Why does this have to be the case? What is the sin­gle, fun­da­men­tal rea­son? Do I re­ally be­lieve that this is true, deep down? Would I bet a large amount of money on it with a friend?

VI.

Two para­bles:

First, Ezra Pound’s para­ble of Agassiz, from his “ABC of Reading” (incidentally one of the most un­der­rated books about lit­er­a­ture). I’ve pre­served his quirky for­mat­tingNo man is equipped for mod­ern think­ing un­til he has un­der­stood the anec­dote of Agassiz and the fish:

A post-grad­u­ate stu­dent equipped with ho­n­ours and diplo­mas went to Agassiz to re­ceive the fi­nal and fin­ish­ing touches.

The great man of­fered him a small fish and told him to de­scribe it.

Post-Graduate Student: “That’s only a sun-fish”

Agassiz: “I know that. Write a de­scrip­tion of it.”

After a few min­utes the stu­dent re­turned with the de­scrip­tion of the Ichthus Heliodiplodokus, or what­ever term is used to con­ceal the com­mon sun­fish from vul­gar knowl­edge, fam­ily of Heliichterinkus, etc., as found in text­books of the sub­ject.

Agassiz again told the stu­dent to de­scribe the fish.

The stu­dent pro­duced a four-page es­say.

Agassiz then told him to look at the fish. At the end of the three weeks the fish was in an ad­vanced state of de­com­po­si­tion, but the stu­dent knew some­thing about it. The sec­ond, one of my fa­vorite pas­sages from “Zen and the Art of Motorcycle Maintenance”:

He’d been hav­ing trou­ble with stu­dents who had noth­ing to say. At first he thought it was lazi­ness but later it be­came ap­par­ent that it was­n’t. They just could­n’t think of any­thing to say.

One of them, a girl with strong-lensed glasses, wanted to write a five-hun­dred­word es­say about the United States. He was used to the sink­ing feel­ing that comes from state­ments like this, and sug­gested with­out dis­par­age­ment that she nar­row it down to just Bozeman.

When the pa­per came due she did­n’t have it and was quite up­set. She had tried and tried but she just could­n’t think of any­thing to say.

He had al­ready dis­cussed her with her pre­vi­ous in­struc­tors and they’d con­firmed his im­pres­sions of her. She was very se­ri­ous, dis­ci­plined and hard­work­ing, but ex­tremely dull. Not a spark of cre­ativ­ity in her any­where. Her eyes, be­hind the thick-lensed glasses, were the eyes of a drudge. She was­n’t bluff­ing him, she re­ally could­n’t think of any­thing to say, and was up­set by her in­abil­ity to do as she was told.

It just stumped him. Now he could­n’t think of any­thing to say. A si­lence oc­curred, and then a pe­cu­liar an­swer: “Narrow it down to the main street of Bozeman.” It was a stroke of in­sight.

She nod­ded du­ti­fully and went out. But just be­fore her next class she came back in real dis­tress, tears this time, dis­tress that had ob­vi­ously been there for a long time. She still could­n’t think of any­thing to say, and could­n’t un­der­stand why, if she could­n’t think of any­thing about all of Bozeman, she should be able to think of some­thing about just one street.

He was fu­ri­ous. “You’re not look­ing!” he said. A mem­ory came back of his own dis­missal from the University for hav­ing too much to say. For every fact there is an in­fin­ity of hy­pothe­ses. The more you look the more you see. She re­ally was­n’t look­ing and yet some­how did­n’t un­der­stand this.

He told her an­grily, “Narrow it down to the front of one build­ing on the main street of Bozeman. The Opera House. Start with the up­per left-hand brick.”

Her eyes, be­hind the thick-lensed glasses, opened wide. She came in the next class with a puz­zled look and handed him a five- thou­sand-word es­say on the front of the Opera House on the main street of Bozeman, Montana. “I sat in the ham­burger stand across the street,” she said, “and started writ­ing about the first brick, and the sec­ond brick, and then by the third brick it all started to come and I could­n’t stop. They thought I was crazy, and they kept kid­ding me, but here it all is. I don’t un­der­stand it.”

Neither did he, but on long walks through the streets of town he thought about it and con­cluded she was ev­i­dently stopped with the same kind of block­age that had par­a­lyzed him on his first day of teach­ing. She was blocked be­cause she was try­ing to re­peat, in her writ­ing, things she had al­ready heard, just as on the first day he had tried to re­peat things he had al­ready de­cided to say. She could­n’t think of any­thing to write about Bozeman be­cause she could­n’t re­call any­thing she had heard worth re­peat­ing. She was strangely un­aware that she could look and see freshly for her­self, as she wrote, with­out pri­mary re­gard for what had been said be­fore. The nar­row­ing down to one brick de­stroyed the block­age be­cause it was so ob­vi­ous she had to do some orig­i­nal and di­rect see­ing.

The point of both of these para­bles: noth­ing beats di­rect ex­pe­ri­ence. Get the data your­self. This is why I wanted to an­a­lyze the coro­n­avirus genome di­rectly, for ex­am­ple. You de­velop some ba­sis in re­al­ity by get­ting some first-hand data, and rea­son­ing up from there, ver­sus start­ing with some­body else’s lossy com­pres­sion of a messy, evolv­ing phe­nom­e­non and then won­der­ing why events keep sur­pris­ing you.

People who have not ex­pe­ri­enced the thing are un­likely to be gen­er­at­ing truth. More likely, they’re resur­fac­ing cached thoughts and nar­ra­tives. Reading pop­u­lar sci­ence books or news ar­ti­cles is not a sub­sti­tute for un­der­stand­ing, and may make you stu­pider, by fill­ing your mind with nar­ra­tives and sto­ries that don’t rep­re­sent your own syn­the­sis.

Even if you can’t ex­pe­ri­ence the thing di­rectly, try go­ing for in­for­ma­tion-dense sources with high amounts of de­tail and facts, and then rea­son up from those facts. On for­eign pol­icy, read books pub­lished by uni­ver­sity presses — not The Atlantic or The Economist or what­ever. You can read those af­ter you’ve de­vel­oped a model of the thing your­self, against which you can judge the pop­u­lar nar­ra­tives.

Another thing the para­ble about the bricks tells us: un­der­stand­ing is not a bi­nary “yes/no”. It has lay­ers of depth. My friend un­der­stood Pythagoras’s the­o­rem far more deeply than I did; he could prove it six dif­fer­ent ways and had sim­ply thought about it for longer.

The sim­plest things can re­ward close study. Michael Nielsen has a nice ex­am­ple of this — the equals sign:

I first re­ally ap­pre­ci­ated this af­ter read­ing an es­say by the math­e­mati­cian Andrey Kolmogorov. You might sup­pose a great math­e­mati­cian such as Kolmogorov would be writ­ing about some very com­pli­cated piece of math­e­mat­ics, but his sub­ject was the hum­ble equals sign: what made it a good piece of no­ta­tion, and what its de­fi­cien­cies were. Kolmogorov dis­cussed this in lov­ing de­tail, and made many beau­ti­ful points along the way, e.g., that the in­ven­tion of the equals sign helped make pos­si­ble no­tions such as equa­tions (and al­ge­braic ma­nip­u­la­tions of equa­tions).

Prior to read­ing the es­say I thought I un­der­stood the equals sign. Indeed, I would have been of­fended by the sug­ges­tion that I did not. But the es­say showed con­vinc­ingly that I could un­der­stand the equals sign much more deeply. (link)The pho­tog­ra­pher Robert Capa ad­vised be­gin­ning pho­tog­ra­phers: “If your pic­tures aren’t good enough, you’re not close enough”. (This is good fic­tion writ­ing ad­vice, by the way.)

It is also good ad­vice for un­der­stand­ing things. When in doubt, go closer.

Thanks to Jose-Luis Ricon for read­ing a draft of this es­say.

Follow me on Twitter: @nabeelqu

Tweet

A talk at Hydra, on­line (originally planned to be in Moscow, Russia), 06 Jul 2020

Conflict-free Replicated Data Types (CRDTs) are an in­creas­ingly pop­u­lar fam­ily of al­go­rithms for op­ti­mistic repli­ca­tion. They al­low data to be con­cur­rently up­dated on sev­eral repli­cas, even while those repli­cas are of­fline, and pro­vide a ro­bust way of merg­ing those up­dates back into a con­sis­tent state. CRDTs are used in geo-repli­cated data­bases, multi-user col­lab­o­ra­tion soft­ware, dis­trib­uted pro­cess­ing frame­works, and var­i­ous other sys­tems.

However, while the ba­sic prin­ci­ples of CRDTs are now quite well known, many chal­leng­ing prob­lems are lurk­ing be­low the sur­face. It turns out that CRDTs are easy to im­ple­ment badly. Many pub­lished al­go­rithms have anom­alies that cause them to be­have strangely in some sit­u­a­tions. Simple im­ple­men­ta­tions of­ten have ter­ri­ble per­for­mance, and mak­ing the per­for­mance good is chal­leng­ing.

In this talk Martin goes be­yond the in­tro­duc­tory ma­te­r­ial on CRDTs, and dis­cusses some of the hard-won lessons from years of re­search on mak­ing CRDTs work in prac­tice.

A browser is an in­cred­i­bly com­plex piece of soft­ware. With such enor­mous com­plex­ity, the only way to main­tain a rapid pace of de­vel­op­ment is through an ex­ten­sive CI sys­tem that can give de­vel­op­ers con­fi­dence that their changes won’t in­tro­duce bugs. Given the scale of our CI, we’re al­ways look­ing for ways to re­duce load while main­tain­ing a high stan­dard of prod­uct qual­ity. We won­dered if we could use ma­chine learn­ing to reach a higher de­gree of ef­fi­ciency.

At Mozilla we have around 50,000 unique test files. Each con­tain many test func­tions. These tests need to run on all our sup­ported plat­forms (Windows, Mac, Linux, Android) against a va­ri­ety of build con­fig­u­ra­tions (PGO, de­bug, ASan, etc.), with a range of run­time pa­ra­me­ters (site iso­la­tion, WebRender, multi-process, etc.).

While we don’t test against every pos­si­ble com­bi­na­tion of the above, there are still over 90 unique con­fig­u­ra­tions that we do test against. In other words, for each change that de­vel­op­ers push to the repos­i­tory, we could po­ten­tially run all 50k tests 90 dif­fer­ent times. On an av­er­age work day we see nearly 300 pushes (including our test­ing branch). If we sim­ply ran every test on every con­fig­u­ra­tion on every push, we’d run ap­prox­i­mately 1.35 bil­lion test files per day! While we do throw money at this prob­lem to some ex­tent, as an in­de­pen­dent non-profit or­ga­ni­za­tion, our bud­get is fi­nite.

So how do we keep our CI load man­age­able? First, we rec­og­nize that some of those ninety unique con­fig­u­ra­tions are more im­por­tant than oth­ers. Many of the less im­por­tant ones only run a small sub­set of the tests, or only run on a hand­ful of pushes per day, or both. Second, in the case of our test­ing branch, we rely on our de­vel­op­ers to spec­ify which con­fig­u­ra­tions and tests are most rel­e­vant to their changes. Third, we use an in­te­gra­tion branch.

Basically, when a patch is pushed to the in­te­gra­tion branch, we only run a small sub­set of tests against it. We then pe­ri­od­i­cally run every­thing and em­ploy code sher­iffs to fig­ure out if we missed any re­gres­sions. If so, they back out the of­fend­ing patch. The in­te­gra­tion branch is pe­ri­od­i­cally merged to the main branch once every­thing looks good.

A sub­set of the tasks we run on a sin­gle mozilla-cen­tral push. The full set of tasks were too hard to dis­tin­guish when scaled to fit in a sin­gle im­age.

These meth­ods have served us well for many years, but it turns out they’re still very ex­pen­sive. Even with all of these op­ti­miza­tions our CI still runs around 10 com­pute years per day! Part of the prob­lem is that we have been us­ing a naive heuris­tic to choose which tasks to run on the in­te­gra­tion branch. The heuris­tic ranks tasks based on how fre­quently they have failed in the past. The rank­ing is un­re­lated to the con­tents of the patch. So a push that mod­i­fies a README file would run the same tasks as a push that turns on site iso­la­tion. Additionally, the re­spon­si­bil­ity for de­ter­min­ing which tests and con­fig­u­ra­tions to run on the test­ing branch has shifted over to the de­vel­op­ers them­selves. This wastes their valu­able time and tends to­wards over-se­lec­tion of tests.

About a year ago, we started ask­ing our­selves: how can we do bet­ter? We re­al­ized that the cur­rent im­ple­men­ta­tion of our CI re­lies heav­ily on hu­man in­ter­ven­tion. What if we could in­stead cor­re­late patches to tests us­ing his­tor­i­cal re­gres­sion data? Could we use a ma­chine learn­ing al­go­rithm to fig­ure out the op­ti­mal set of tests to run? We hy­poth­e­sized that we could si­mul­ta­ne­ously save money by run­ning fewer tests, get re­sults faster, and re­duce the cog­ni­tive bur­den on de­vel­op­ers. In the process, we would build out the in­fra­struc­ture nec­es­sary to keep our CI pipeline run­ning ef­fi­ciently.

The main pre­req­ui­site to a ma­chine-learn­ing-based so­lu­tion is col­lect­ing a large and pre­cise enough re­gres­sion dataset. On the sur­face this ap­pears easy. We al­ready store the sta­tus of all test ex­e­cu­tions in a data ware­house called ActiveData. But in re­al­ity, it’s very hard to do for the rea­sons be­low.

Since we only run a sub­set of tests on any given push (and then pe­ri­od­i­cally run all of them), it’s not al­ways ob­vi­ous when a re­gres­sion was in­tro­duced. Consider the fol­low­ing sce­nario:

It is easy to see that the “Test A” fail­ure was re­gressed by Patch 2, as that’s where it first started fail­ing. However with the “Test B” fail­ure, we can’t re­ally be sure. Was it caused by Patch 2 or 3? Now imag­ine there are 8 patches in be­tween the last PASS and the first FAIL. That adds a lot of un­cer­tainty!

Intermittent (aka flaky) fail­ures also make it hard to col­lect re­gres­sion data. Sometimes tests can both pass and fail on the same code­base for all sorts of dif­fer­ent rea­sons. It turns out we can’t be sure that Patch 2 re­gressed “Test A” in the table above af­ter all! That is un­less we re-run the fail­ure enough times to be sta­tis­ti­cally con­fi­dent. Even worse, the patch it­self could have in­tro­duced the in­ter­mit­tent fail­ure in the first place. We can’t as­sume that just be­cause a fail­ure is in­ter­mit­tent that it’s not a re­gres­sion.

The writ­ers of this post hav­ing a hard time.

In or­der to solve these prob­lems, we have built quite a large and com­pli­cated set of heuris­tics to pre­dict which re­gres­sions are caused by which patch. For ex­am­ple, if a patch is later backed out, we check the sta­tus of the tests on the back­out push. If they’re still fail­ing, we can be pretty sure the fail­ures were not due to the patch. Conversely, if they start pass­ing we can be pretty sure that the patch was at fault.

Some fail­ures are clas­si­fied by hu­mans. This can work to our ad­van­tage. Part of the code sher­if­f’s job is an­no­tat­ing fail­ures (e.g. “intermittent” or “fixed by com­mit” for fail­ures fixed at some later point). These clas­si­fi­ca­tions are a huge help find­ing re­gres­sions in the face of miss­ing or in­ter­mit­tent tests. Unfortunately, due to the sheer num­ber of patches and fail­ures hap­pen­ing con­tin­u­ously, 100% ac­cu­racy is not at­tain­able. So we even have heuris­tics to eval­u­ate the ac­cu­racy of the clas­si­fi­ca­tions!

Another trick for han­dling miss­ing data is to back­fill miss­ing tests. We se­lect tests to run on older pushes where they did­n’t ini­tially run, for the pur­pose of find­ing which push caused a re­gres­sion. Currently, sher­iffs do this man­u­ally. However, there are plans to au­to­mate it in cer­tain cir­cum­stances in the fu­ture.

We also need to col­lect data about the patches them­selves, in­clud­ing files mod­i­fied and the diff.  This al­lows us to cor­re­late with the test fail­ure data. In this way, the ma­chine learn­ing model can de­ter­mine the set of tests most likely to fail for a given patch.

Collecting data about patches is way eas­ier, as it is to­tally de­ter­min­is­tic. We it­er­ate through all the com­mits in our Mercurial repos­i­tory, pars­ing patches with our rust-parsep­atch pro­ject and an­a­lyz­ing source code with our rust-code-analy­sis pro­ject.

Now that we have a dataset of patches and as­so­ci­ated tests (both passes and fail­ures), we can build a train­ing set and a val­i­da­tion set to teach our ma­chines how to se­lect tests for us.

90% of the dataset is used as a train­ing set, 10% is used as a val­i­da­tion set. The split must be done care­fully. All patches in the val­i­da­tion set must be pos­te­rior to those in the train­ing set. If we were to split ran­domly, we’d leak in­for­ma­tion from the fu­ture into the train­ing set, caus­ing the re­sult­ing model to be bi­ased and ar­ti­fi­cially mak­ing its re­sults look bet­ter than they ac­tu­ally are.

For ex­am­ple, con­sider a test which had never failed un­til last week and has failed a few times since then. If we train the model with a ran­domly picked train­ing set, we might find our­selves in the sit­u­a­tion where a few fail­ures are in the train­ing set and a few in the val­i­da­tion set. The model might be able to cor­rectly pre­dict the fail­ures in the val­i­da­tion set, since it saw some ex­am­ples in the train­ing set.

In a real-world sce­nario though, we can’t look into the fu­ture. The model can’t know what will hap­pen in the next week, but only what has hap­pened so far. To eval­u­ate prop­erly, we need to pre­tend we are in the past, and fu­ture data (relative to the train­ing set) must be in­ac­ces­si­ble.

Visualization of our split be­tween train­ing and val­i­da­tion set.

We train an XGBoost model, us­ing fea­tures from both test, patch, and the links be­tween them, e.g:

* In the past, how of­ten did this test fail when the same files were touched?

* How far in the di­rec­tory tree are the source files from the test files?

* How of­ten in the VCS his­tory were the source files mod­i­fied to­gether with the test files?

The in­put to the model is a tu­ple (TEST, PATCH), and the la­bel is a bi­nary FAIL or NOT FAIL. This means we have a sin­gle model that is able to take care of all tests. This ar­chi­tec­ture al­lows us to ex­ploit the com­mon­al­i­ties be­tween test se­lec­tion de­ci­sions in an easy way. A nor­mal multi-la­bel model, where each test is a com­pletely sep­a­rate la­bel, would not be able to ex­trap­o­late the in­for­ma­tion about a given test and ap­ply it to an­other com­pletely un­re­lated test.

Given that we have tens of thou­sands of tests, even if our model was 99.9% ac­cu­rate (which is pretty ac­cu­rate, just one er­ror every 1000 eval­u­a­tions), we’d still be mak­ing mis­takes for pretty much every patch! Luckily the cost as­so­ci­ated with false pos­i­tives (tests which are se­lected by the model for a given patch but do not fail) is not as high in our do­main, as it would be if say, we were try­ing to rec­og­nize faces for polic­ing pur­poses. The only price we pay is run­ning some use­less tests. At the same time we avoided run­ning hun­dreds of them, so the net re­sult is a huge sav­ings!

As de­vel­op­ers pe­ri­od­i­cally switch what they are work­ing on the dataset we train on evolves. So we cur­rently re­train the model every two weeks.

After we have cho­sen which tests to run, we can fur­ther im­prove the se­lec­tion by choos­ing where the tests should run. In other words, the set of con­fig­u­ra­tions they should run on. We use the dataset we’ve col­lected to iden­tify re­dun­dant con­fig­u­ra­tions for any given test. For in­stance, is it re­ally worth run­ning a test on both Windows 7 and Windows 10? To iden­tify these re­dun­dan­cies, we use a so­lu­tion sim­i­lar to fre­quent item­set min­ing:

Collect fail­ure sta­tis­tics for groups of tests and con­fig­u­ra­tions

Calculate the “support” as the num­ber of pushes in which both X and Y failed over the num­ber of pushes in which they both run

Calculate the “confidence” as the num­ber of pushes in which both X and Y failed over the num­ber of pushes in which they both run and only one of the two failed.

We only se­lect con­fig­u­ra­tion groups where the sup­port is high (low sup­port would mean we don’t have enough proof) and the con­fi­dence is high (low con­fi­dence would mean we had many cases where the re­dun­dancy did not ap­ply).

Once we have the set of tests to run, in­for­ma­tion on whether their re­sults are con­fig­u­ra­tion-de­pen­dent or not, and a set of ma­chines (with their as­so­ci­ated cost) on which to run them; we can for­mu­late a math­e­mat­i­cal op­ti­miza­tion prob­lem which we solve with a mixed-in­te­ger pro­gram­ming solver. This way, we can eas­ily change the op­ti­miza­tion ob­jec­tive we want to achieve with­out in­va­sive changes to the op­ti­miza­tion al­go­rithm. At the mo­ment, the op­ti­miza­tion ob­jec­tive is to se­lect the cheap­est con­fig­u­ra­tions on which to run the tests.

A ma­chine learn­ing model is only as use­ful as a con­sumer’s abil­ity to use it. To that end, we de­cided to host a ser­vice on Heroku us­ing ded­i­cated worker dynos to ser­vice re­quests and Redis Queues to bridge be­tween the back­end and fron­tend. The fron­tend ex­poses a sim­ple REST API, so con­sumers need only spec­ify the push they are in­ter­ested in (identified by the branch and top­most re­vi­sion). The back­end will au­to­mat­i­cally de­ter­mine the files changed and their con­tents us­ing a clone of mozilla-cen­tral.

Depending on the size of the push and the num­ber of pushes in the queue to be an­a­lyzed, the ser­vice can take sev­eral min­utes to com­pute the re­sults. We there­fore en­sure that we never queue up more than a sin­gle job for any given push. We cache re­sults once com­puted. This al­lows con­sumers to kick off a query asyn­chro­nously, and pe­ri­od­i­cally poll to see if the re­sults are ready.

We cur­rently use the ser­vice when sched­ul­ing tasks on our in­te­gra­tion branch. It’s also used when de­vel­op­ers run the spe­cial mach try auto com­mand to test their changes on the test­ing branch. In the fu­ture, we may also use it to de­ter­mine which tests a de­vel­oper should run lo­cally.

Sequence di­a­gram de­pict­ing the com­mu­ni­ca­tion be­tween the var­i­ous ac­tors in our in­fra­struc­ture.

From the out­set of this pro­ject, we felt it was cru­cial that we be able to run and com­pare ex­per­i­ments, mea­sure our suc­cess and be con­fi­dent that the changes to our al­go­rithms were ac­tu­ally an im­prove­ment on the sta­tus quo. There are ef­fec­tively two vari­ables that we care about in a sched­ul­ing al­go­rithm:

The amount of re­sources used (measured in hours or dol­lars).

The re­gres­sion de­tec­tion rate. That is, the per­cent­age of in­tro­duced re­gres­sions that were caught di­rectly on the push that caused them. In other words, we did­n’t have to rely on a hu­man to back­fill the fail­ure to fig­ure out which push was the cul­prit.

sched­uler ef­fec­tive­ness = 1000 * re­gres­sion de­tec­tion rate / hours per push

The higher this met­ric, the more ef­fec­tive a sched­ul­ing al­go­rithm is. Now that we had our met­ric, we in­vented the con­cept of a “shadow sched­uler”. Shadow sched­ulers are tasks that run on every push, which shadow the ac­tual sched­ul­ing al­go­rithm. Only rather than ac­tu­ally sched­ul­ing things, they out­put what they would have sched­uled had they been the de­fault. Each shadow sched­uler may in­ter­pret the data re­turned by our ma­chine learn­ing ser­vice a bit dif­fer­ently. Or they may run ad­di­tional op­ti­miza­tions on top of what the ma­chine learn­ing model rec­om­mends.

Finally we wrote an ETL to query the re­sults of all these shadow sched­ulers, com­pute the sched­uler ef­fec­tive­ness met­ric of each, and plot them all in a dash­board. At the mo­ment, there are about a dozen dif­fer­ent shadow sched­ulers that we’re mon­i­tor­ing and fine-tun­ing to find the best pos­si­ble out­come. Once we’ve iden­ti­fied a win­ner, we make it the de­fault al­go­rithm. And then we start the process over again, cre­at­ing fur­ther ex­per­i­ments.

The early re­sults of this pro­ject have been very promis­ing. Compared to our pre­vi­ous so­lu­tion, we’ve re­duced the num­ber of test tasks on our in­te­gra­tion branch by 70%! Compared to a CI sys­tem with no test se­lec­tion, by al­most 99%! We’ve also seen pretty fast adop­tion of our mach try auto tool, sug­gest­ing a us­abil­ity im­prove­ment (since de­vel­op­ers no longer need to think about what to se­lect). But there is still a long way to go!

We need to im­prove the mod­el’s abil­ity to se­lect con­fig­u­ra­tions and de­fault to that. Our re­gres­sion de­tec­tion heuris­tics and the qual­ity of our dataset needs to im­prove. We have yet to im­ple­ment us­abil­ity and sta­bil­ity fixes to mach try auto.

And while we can’t make any promises, we’d love to pack­age the model and ser­vice up in a way that is use­ful to or­ga­ni­za­tions out­side of Mozilla. Currently, this ef­fort is part of a larger pro­ject that con­tains other ma­chine learn­ing in­fra­struc­ture orig­i­nally cre­ated to help man­age Mozilla’s Bugzilla in­stance. Stay tuned!

If you’d like to learn more about this pro­ject or Firefox’s CI sys­tem in gen­eral, feel free to ask on our Matrix chan­nel, #firefox-ci:mozilla.org.

Let’s find out how well you know com­put­ers! All of these pro­grams have a vari­able NUMBER in them. Your mis­sion: guess how big NUMBER needs to get be­fore the pro­gram takes 1 sec­ond to run.

You don’t need to guess ex­actly: they’re all be­tween 1 and a bil­lion. Just try to guess the right or­der of mag­ni­tude! A few notes:

* If the an­swer is 38,000, both 10,000 and 100,000 are

con­sid­ered cor­rect an­swers. The goal is to not be wrong by

more than 10x :)

* We know com­put­ers have dif­fer­ent disk & net­work & CPU

speeds! We’re try­ing to get you to tell the dif­fer­ence

be­tween code that can run 10 times/​s and 100000 times/​s. A

newer com­puter won’t make your code run 1000x faster :)

* That said, all this was run on a new lap­top with a fast

SSD and a sketchy net­work con­nec­tion. The C code was com­piled with gcc -O2.

Good luck! We were sur­prised by a lot of these. We’ll be anony­mously col­lect­ing your an­swers, so ex­pect some graphs in the fu­ture! =D

Our rout­ing al­go­rithm prefers paths that go through parks, forests or by wa­ter, and avoids busy roads wher­ever pos­si­ble.

Here’s a few things you can do with Trail Router:

Manually cre­ate your own point-to-point route that prefers na­ture.

Choose whether you’d pre­fer routes that in­volve na­ture, well-lit streets or a lack of hills.

We’re not per­fect! Help im­prove Trail Router by re­port­ing an is­sue here.

A road is un­safe for run­ners ( See FAQ

Thank you for re­port­ing an is­sue!

We will re­view your is­sue and be in touch via email. Please note that cor­rec­tions to map­ping data can take a cou­ple of weeks to fil­ter through.

Dr. Lam ex­plained that “in places like Singapore, we want to keep the win­dows open as much as pos­si­ble” to re­duce the use of car­bon-in­ten­sive air-con­di­tion­ers and to pre­vent buildup of stale air that can pose health risks for some peo­ple.

But with win­dows open, the con­stant din from city traf­fic, trains, jets pass­ing over­head and con­struc­tion equip­ment can rat­tle apart­ments. The Anti-Noise Control Window, as it is called, is the sonic equiv­a­lent of shut­ting a win­dow.

With any sound, the best way to re­duce it is at the source, like a gun’s si­lencer. So the re­searchers treated the win­dow aper­ture it­self as the noise source, be­cause most noise en­ters a room that way.

The sys­tem uses a mi­cro­phone out­side the win­dow to de­tect the re­peat­ing sound waves of the of­fend­ing noise source, which is reg­is­tered by a com­puter con­troller. That in turn de­ci­phers the proper wave fre­quency needed to neu­tral­ize the sound, which is trans­mit­ted to the ar­ray of speak­ers on the in­side of the win­dow frame.

The speak­ers then emit the proper “anti” waves, which can­cel out the in­com­ing waves, and there you have it: near bliss­ful si­lence.

“If you sit in the room, you get that same feel­ing like when you flick on the switch of noise-can­cel­ing ear­phones,” Dr. Lam said, splay­ing his hands to de­note the calm­ing ef­fect.

The sys­tem is best at at­ten­u­at­ing the au­di­ble blasts from the types of steady noise sources found within the op­ti­mal fre­quency range.

Darwin is the Open Source op­er­at­ing sys­tem from Apple that forms the ba­sis for Mac OS X and PureDarwin. PureDarwin is a com­mu­nity pro­ject that aims to make Darwin more us­able (some peo­ple think of it as the in­for­mal suc­ces­sor to OpenDarwin).

One cur­rent goal of this pro­ject is to pro­vide a use­ful bootable ISO/VM of Darwin 10.x

Come Join our fo­rum over at https://​www.pd-devs.org/

See the Wiki for more in­for­ma­tion.

We’ve de­tected that JavaScript is dis­abled in your browser. Would you like to pro­ceed to legacy Twitter?

This is the once-a-week free edi­tion of The Diff, the newslet­ter about in­flec­tions in fi­nance and tech­nol­ogy. The free edi­tion goes out to 8,221 sub­scribers, up 211 week-over-week. This week’s sub­scribers-only posts:

* The UK as a Science Hub is an up­date on Boris Johnson’s plan (or, if you pre­fer, Dominic Cumming’s scheme) to make Britain a sci­en­tific pow­er­house. The out­lines of the plan aren’t new, but the op­por­tu­nity is.

* The Equity Risk Premium at 0% Interest looks at the im­pli­ca­tions of low real rates for tech com­pa­nies. In equi­lib­rium, low rates are good for eq­ui­ties be­cause they raise the pre­sent value of fu­ture cash flows. But an­other way of say­ing this is that, in fi­nan­cial terms, low rates mean the fu­ture hap­pens all at once.

* Globalization: A Toy Story) is a pre­quel to to­day’s note, dis­cussing the his­tory of Hong Kong’s toy in­dus­try. Hong Kong’s toy in­dus­try was ba­si­cally nonex­is­tent in 1945, the biggest in the world by 1972, and con­sis­tently lost share to China from the 80s on­ward. It’s a case study in how glob­al­iza­tion works.

* The Depressing Bull Thesis for Rocket Mortgage is a writeup of Rocket, the largest mort­gage orig­i­na­tor in the US, which re­cently filed to go pub­lic. Fewer red flags than ex­pected, but it’s partly dri­ven by a dire fi­nan­cial bet.

* Why Are Toys Such a Bad Business?

* V-Shaped Recovery is here… just not evenly dis­trib­uted.

Early-stage in­vestors some­times use the heuris­tic that if a prod­uct gets de­rided as a toy, it’s worth in­vest­ing in. That model would have got­ten you into PCs in the 70s, the Internet in the early 90s, so­cial net­works when the good ones were pri­vately-held, cryp­tocur­ren­cies, and drones. The risk is in­vest­ing in ac­tual toy com­pa­nies, which is usu­ally a ter­ri­ble de­ci­sion. Hasbro stock has­n’t done any­thing for half a decade, and Mattel trades where it did in the early 90s. JAKKS Pacific has de­stroyed most of its share­hold­ers’ wealth, and Funko is work­ing on the same.

This is not a new phe­nom­e­non, ei­ther. The biggest toy com­pany in the US in the 50s was Louis Marx & Company, whose founder made the cover of Time. Sales de­clined slightly over the next decade, and faster af­ter that; the com­pany was bank­rupt in 1980. Coleco rode the Cabbage Patch Kids trend in the mid-80s—in 1985, they had the high­est re­turn on eq­uity of any com­pany in the Fortune 500—but they were bank­rupt by 1988.

The record is no bet­ter for re­tail­ers. Toys R Us is bank­rupt, of course, and they fol­lowed FAO Shwarz, KB Toys, Right Start, and Zany Brainy.

The toy in­dus­try has not been kind to in­vestors, at any level.

There are a few rea­sons, and a few rel­e­vant lessons.

First, the toy busi­ness op­er­ates on an an­nual cy­cle. Historically, about 40% of toy sales hap­pen dur­ing the hol­i­day sea­son, and about half of those were in the two weeks be­fore Christmas. (That’s a dated sta­tis­tic, from about twenty years ago. Discretionary re­tail sales in gen­eral have got­ten spikier, since more shop­pers are used to fast, free ship­ping. With Amazon Prime, the Christmas shop­ping sea­son starts on December 22nd or so.) 84% of US toy sales come from China, trans­ported by a mix of ships and air freight, so they need to be or­dered months in ad­vance.

And they have to be mar­keted: while cheap toys can com­pete on price, the higher-mar­gin ones only get sold when there’s an ef­fec­tive ad cam­paign. Mattel cre­ated this model (and over­turned Louis Marx’s price-first ap­proach) when they spent their en­tire net worth on a one-year spon­sor­ship of The Mickey Mouse Club in 1955.

TV ad cam­paigns, too, tend to be pur­chased in ad­vance. About half of TV ad spend­ing is al­lo­cated to the up­fronts—booked March through May to be de­liv­ered by the end of the year.

This locks toy com­pa­nies into a chal­leng­ing bet. Every year, they have to a) pre­dict trends, b) in­vent them, and c) com­mit cap­i­tal to them. All with­out know­ing how the rest of the year will turn out. Since toy trends ex­ist, but don’t last for very long, they have to in­vent new prod­ucts every year—but the tech­no­log­i­cal state of the art does­n’t ad­vance very fast. It has all the volatil­ity of tech, with­out the progress.

A hand­ful of com­pa­nies have made se­ri­ous money in toys, or, rather, in toy-like or toy-ad­ja­cent busi­nesses. The video game in­dus­try has gen­er­ally done well. Disney turns a profit. And Games Workshop has a nice lit­tle busi­ness. (I wrote up in The Diff in April—note that I’ve since sold the stock, just for val­u­a­tion rea­sons) . Lego, too, is a great busi­ness, worth an es­ti­mated $15bn.

What these com­pa­nies have in com­mon is that they es­cape the de­mo­graphic trap that toy man­u­fac­tur­ers are locked into. Every year, there’s a new co­hort of six-year-olds, and they need some­thing that a) did­n’t ex­ist last year, but b) ap­peals to time­less six-year-old sen­si­bil­i­ties. They don’t have much brand loy­alty, be­cause a year later the same toy is a toy for lit­tle kids. Each of these suc­cess­ful com­pa­nies beats that in a dif­fer­ent way:

* Video games’ av­er­age age has trended older over time, so in­stead of mar­ket­ing to more trend-sen­si­tive young peo­ple, they’re mar­ket­ing to more dol­lar-in­sen­si­tive not-so-young peo­ple.

* Disney has a gen­er­a­tional loop, of which toys are a small part. Movies and stream­ing video get kids hooked on Disney char­ac­ters, which can be mon­e­tized at much higher dol­lar val­ues through their parks. (See my writeup here for much more.)

* Games Workshop and Lego have a very healthy prod­uct dy­namic: the ones you al­ready own are an eco­nomic com­ple­ment to the ones you buy. And Lego clearly de­signs their mar­ket­ing around hit­ting two gen­er­a­tions, too: at the Lego Store, $30 Rise of Skywalker-themed Lego sets are at a kids’ eye level. The $800 set based on the orig­i­nal tril­ogy is po­si­tioned at an adult’s eye level.

These com­pa­nies have some­thing else in com­mon: they own their core in­tel­lec­tual prop­erty. Video game pub­lish­ers do make games based on su­per­heroes and sports leagues, and Lego cer­tainly has branded sets, but the core of each busi­ness is IP owned by the com­pany it­self; the li­censed prod­ucts are a lu­cra­tive side busi­ness: it’s much eas­ier for a video game com­pany to re-skin char­ac­ters than for a movie com­pany to start a video game stu­dio. As a case study, one pop­u­lar game was orig­i­nally in­tended to be set in the Game of Thrones uni­verse, but ended up us­ing in-house IP in­stead. Disney, of course, sells toys based on its own char­ac­ters. And while some Lego sets are as­so­ci­ated with out­side brands at the point of pur­chase, they in­evitably end up be­ing fun­gi­ble with other Legos.

Because it’s a hit-dri­ven in­dus­try, toy com­pa­nies that suc­ceed can be im­mensely prof­itable for a while. The prob­lem is that the dif­fer­ence be­tween a cul­tural land­mark and a fad is vis­i­ble af­ter a decade or so, while the de­ci­sion of how much to or­der and how much to spend on mar­ket­ing has to hap­pen every year re­gard­less. So toy com­pa­nies with a hit prod­uct in year N tend to be bank­rupt com­pa­nies writ­ing down the value of their in­ven­tory to ~$0 in year N+3 or N+5.

You should­n’t have to take big risks to make big re­turns. So when data from Citibank shows that art has out­per­formed the S&P by 180% since 2000 with the least volatil­ity of any ma­jor as­set class—we’re in­clined to no­tice.

The ul­tra-wealthy have in­vested in art for cen­turies, to the tune of over $1.7 tril­lion in to­tal value—so why can’t the rest of us?

Masterworks lets any­one in­vest in paint­ings by some of the most suc­cess­ful artists in his­tory like Banksy, Warhol, Basquiat, and more, in just a few clicks. The only catch? There’s cur­rently a back­log of over 25,000 of peo­ple ap­ply­ing for mem­ber­ship, but you can skip the wait­list by sign­ing up to­day.*

… just not evenly dis­trib­uted. And not likely to last. In Japan, Uniqlo ex­pects Japanese sales to be up 25% Y/Y in their August quar­ter, af­ter a 15% de­cline last quar­ter. Their re­gional es­ti­mates are very much virus-dri­ven, with op­ti­mism in China and pes­simism in coun­tries see­ing a sec­ond wave, or, in the US’s case, a 1.5th wave. And world­wide PC ship­ments grew in Q2, mostly due to the one-time build-out of home of­fices and Zoom-based schools. In China, auto sales were up 10% in Q2 ($), and China’s cop­per smelt­ing is also ris­ing ($).

Inventory re­stock­ing used to be a sig­nif­i­cant dri­ver of GDP growth: when the econ­omy slowed, com­pa­nies had too much in­ven­tory on hand, and had to cut jobs to work through the ex­cess. Once they ran out, they had to re­hire fast. Now, sup­ply and de­mand for man­u­fac­tur­ing are lo­cated in dif­fer­ent places (with dif­fer­ent poli­cies), and com­pa­nies are more averse to hold­ing in­ven­tory for long pe­ri­ods, so this model is­n’t as de­scrip­tive or pre­dic­tive as it once was. When the peo­ple get­ting fired are the ones pro­vid­ing de­mand, it’s easy for a re­ces­sion to feed on it­self, and easy for a re­cov­ery to boot­strap it­self, too. When those groups are in dif­fer­ent coun­tries, and when the swings in in­ven­tory are more muted, it’s less of a fac­tor, lead­ing to fewer re­ces­sions but much slower re­bounds.

The Indonesion gov­ern­ment is strapped for cash, and needs to spend heav­ily to mit­i­gate the ef­fects of Covid-19. But the gov­ern­ment is not great at col­lect­ing taxes (taxes are 11-12% of GDP. For com­par­i­son, Mexico and the Netherlands have sim­i­lar-sized economies, and col­lect 16% and 39%, re­spec­tively). But tech com­pa­nies are great at col­lect­ing taxes on on­line com­merce, and tend to charge close to the Laffer peak. So Indonesia is out­sourc­ing tax­a­tion to them ($) by im­pos­ing a 10% value-added tax on large Internet com­pa­nies. Google, Facebook, and Netflix have built their own “tax col­lec­tion” ap­pa­ra­tus, and are bet­ter at catch­ing tax-evaders and charg­ing the right amount. As it turns out, tax-farm­ing was­n’t a ter­ri­ble idea, just a few cen­turies early

In other tax news, Chinese main­lan­ders work­ing in Hong Kong sud­denly owe the main­land’s 45% tax rates rather than Hong Kong’s 15%. China seems to al­ter­nate—on a daily ba­sis—be­tween want­ing Hong Kong to be a fi­nan­cial cen­ter they con­trol and want­ing to use their con­trol to end Hong Kong’s sta­tus as a fi­nan­cial cen­ter.

The dol­lar, by virtue of be­ing the world’s most-used cur­rency, is the cur­rency that least rep­re­sents how cur­ren­cies work. Since it’s a re­serve cur­rency, dol­lars are de­manded by peo­ple who don’t earn them or spend them, but who know they’ll need them, so the US has less con­trol over the value of its money than any other place. To para­phrase John Connally, it’s every­one’s cur­rency but America’s prob­lem.

For ex­am­ple, there’s no way the US could get away with this ($):

Africa’s most pop­u­lous na­tion has long main­tained sev­eral ex­change rates. In ad­di­tion to the in­ter­bank and black-mar­ket rates, there are of­fi­cial rates for con­sumers want­ing dol­lars for school and med­ical fees abroad, for Muslims mak­ing the pil­grim­age to Saudi Arabia, and for peo­ple wish­ing to buy hard cur­rency at ex­change bu­reaux.

That’s a very clever setup. Smaller coun­tries can use a tiered ex­change-rate sys­tem to mod­er­ately en­cour­age or dis­cour­age cer­tain be­hav­iors, or to dole out fa­vors to par­tic­u­lar groups. Dollars are so liq­uid, and used in so many places, that the US has to take a more bi­nary ap­proach, of al­low­ing or ban­ning trans­ac­tions; taxes get routed around.

Alpha Architect has a neg­a­tive view on trea­suries, ar­gu­ing that they’re not a good di­ver­si­fier and that yields are too low to jus­tify own­ing them. The piece goes into de­tail on how trea­suries func­tion as in­sur­ance (not al­ways!), and how they’re mostly owned by price-in­sen­si­tive buy­ers like reg­u­lated in­sur­ance com­pa­nies and cen­tral banks. All true. But the most im­por­tant line in the piece is: “Okay, Treasuries Aren’t Compelling: What Are My Alternatives? Answer: Nothing.” Investments are al­ways ex­pressed in rel­a­tive terms. In an ag­ing world, we should­n’t ex­pect any­thing to be cheap be­cause there’s so much de­mand for sav­ings.

Chinese CSI 300 in­dex dropped 1.8% in the last ses­sion, and it’s now up only 14% since late June. Bloomberg pro­files the wild mar­ket, with plenty of pull quotes from new in­vestors (“There’s no way I can lose,” sounds like a clas­sic bull mar­ket line, but there’s a tinge of des­per­a­tion there). One com­pany, QuantumCTek, rose 1,000% in its IPO ($).

2K games is try­ing to push video game prices above the de facto ceil­ing of $60/copy. Video game prices have been de­clin­ing in real terms, in part be­cause of cheaper man­u­fac­tur­ing and dis­tri­b­u­tion. As the video game in­dus­try gets more ma­ture, pre­dict­ing sales for any given ti­tle gets eas­ier, which en­cour­ages pub­lish­ers to in­vest more in pro­duc­tion. So cost de­fla­tion in one part of the mar­ket is off­set by cost in­fla­tion in an­other.